Spark is not the only engine anymore. Here are the tools replacing it for compaction, analytics, streaming, ML, and lakehouse maintenance — and why the smartest teams use multiple.

Apache Spark has been the default for distributed data processing since roughly 2015. If you needed to transform data at scale, train models, or run SQL across terabytes — you reached for Spark. That was a reasonable decision for a decade.

In 2026, the landscape is different. The lakehouse architecture decoupled storage from compute. Apache Iceberg became the standard table format. And a generation of purpose-built engines emerged — each dramatically better than Spark at one specific thing.

The result: running everything on Spark is now the expensive choice. JVM startup overhead on a 50 MB compaction job. Spark clusters idling between cron-scheduled maintenance windows. Interactive queries taking 8 seconds because Catalyst is planning a 3-row lookup. Data scientists waiting 4 minutes for a notebook cell that Polars finishes in 200 milliseconds.

The modern data platform routes each workload to the engine best suited for it. Spark handles heavy batch ETL. An autonomous control plane like LakeOps handles compaction and table maintenance — 95% faster, no JVM. Trino handles interactive SQL. DuckDB handles local analytics. Flink handles streaming. Each engine reads the same Iceberg tables through the same catalog — that's the multi-engine architecture the format was designed to enable.

This guide covers 9 alternatives — not to replace Spark entirely, but to replace it where it doesn't belong.

1. LakeOps — Autonomous Lakehouse Control Plane

Replaces Spark for: Compaction, table maintenance, snapshot expiration, orphan cleanup, manifest optimization, and all operational work on Iceberg tables.

This is the most impactful entry on this list — and the one most teams discover last, after burning months on Spark maintenance scripts.

Every production Iceberg lakehouse needs continuous maintenance: compaction to merge small files into properly-sized ones, snapshot expiration to trim metadata, orphan cleanup to remove dangling objects from failed writes, manifest optimization to speed up query planning. Without these operations running continuously, tables degrade — queries slow down by 5–12x, storage costs balloon from accumulated garbage, and S3 API bills spike from listing thousands of tiny files.

The traditional approach: Spark. You write compaction jobs using rewrite_data_files. Deploy them on Airflow or cron. Provision EMR clusters or Databricks jobs. Tune JVM heap, executor memory, partition parallelism. When a 1.2 TB table causes an OOM crash at 2 AM, someone gets paged. When query patterns change, nobody updates the sort order. The manual approach doesn't scale past 50 tables.

LakeOps eliminates all of that. It's an autonomous control plane for Apache Iceberg — purpose-built in Rust on Apache DataFusion — that handles the entire maintenance lifecycle without Spark clusters, without JVM overhead, and without human intervention.

What you actually get for compaction

A Rust compaction engine that's 95% faster than Spark. No JVM. No garbage collection pauses. No OOM crashes. In production benchmarks on a 200 GB dataset (600M rows, Parquet, partitioned by date): LakeOps completed binpack compaction in 221 seconds versus 1,612 seconds for Spark and 6,300 seconds for AWS S3 Tables. Peak throughput: 2,522 MB/s. On a 1.2 TB table that crashed Spark with an out-of-memory error, LakeOps finished in 11 minutes. The engine spills to disk gracefully — bounded memory, no OOM regardless of table size.

Query-aware data layout. This is where LakeOps diverges from "just a faster compaction tool." The system continuously observes which columns appear in WHERE, JOIN, and GROUP BY clauses across all engines hitting each table — Spark, Trino, Snowflake, Athena, DuckDB. During compaction, data is physically re-sorted by those columns. Parquet row-group min/max statistics become effective, and engines skip irrelevant data without reading it. In production, sorted tables scan 51% less data than unsorted equivalents — on every query, across every engine, indefinitely. This happens automatically. No manual sort-order configuration.

Event-driven triggers, not cron. Compaction fires when table conditions warrant it — file count thresholds crossed, delete-file ratios exceeded, streaming writes landing. A high-velocity streaming table might compact multiple times per hour. A stable batch table compacts once a week. The system responds to actual state, not arbitrary schedules. After snapshot expiration frees data files, the system re-evaluates whether the remaining file set warrants a compaction pass. After a large batch write lands, newly created small files enter the evaluation queue immediately. No wasted runs, no missed tables.

Sequenced maintenance pipeline. Operations don't run in isolation — they're coordinated so each step's output is the next step's clean input: expire snapshots → remove orphans → compact → rewrite manifests. When compaction runs on a dataset already pruned of expired and orphaned data, the rewrite is tighter — fewer files, cleaner partitions, leaner manifests downstream. One production run removed ~200 TB of orphan data across 324 tables in under 30 minutes — the kind of cleanup that makes subsequent compaction dramatically more efficient.

Self-improving planner. The engine records per-table throughput, partition structure, and memory usage from each run. Subsequent passes execute faster as the planner converges on optimal resource allocation. In production, the same table went from 22 minutes to 18 minutes to 11 minutes across consecutive runs — zero configuration changes.

10-minute setup, no infrastructure changes. LakeOps connects to your existing catalogs (AWS Glue, REST catalogs like Polaris/Nessie/Lakekeeper, S3 Tables) and object storage. No agents to deploy. No data movement. No pipeline changes. Your data stays in your account. Production results across customers: up to 80% total cost reduction, $1.37M saved in 3 months, 786+ tables managed autonomously across 112+ PB.

Benefits over Spark

If you're running Spark primarily for Iceberg table maintenance — compaction, expire_snapshots, remove_orphan_files, rewrite_data_files, rewrite_manifests — you're paying for JVM clusters to do work that a purpose-built engine handles faster and cheaper. LakeOps replaces that entire operational layer. Your Spark clusters can focus on what Spark actually excels at: heavy ETL transformations and large-scale batch processing.

Best suited for: Any team running Apache Iceberg in production that wants to stop managing maintenance infrastructure. Particularly valuable for streaming tables (Kafka/Flink → Iceberg pipelines) where small files accumulate continuously, and for large-scale lakehouses (100+ tables) where manual scripts don't scale.

Learn more: LakeOps Compaction | Platform Overview | Iceberg Lakehouse Optimization

2. Apache Flink — True Stream Processing

Replaces Spark for: Low-latency streaming, complex event processing, stateful stream computation, and continuous Iceberg ingestion.

Spark Structured Streaming is micro-batch. It processes data in small intervals — typically 1–10 seconds at best. For many use cases that's fine. But if you need sub-second event processing, complex windowing with event-time semantics, or exactly-once stateful computation across millions of events per second — Flink is architecturally superior.

Flink 2.0 (supported by Iceberg since version 1.10) processes events with true streaming semantics. Checkpointing provides exactly-once guarantees without the latency penalty of micro-batch boundaries. For Iceberg ingestion specifically, Flink's checkpoint-commit protocol writes data files atomically — each checkpoint produces a clean Iceberg commit with no partial state.

The new Dynamic Iceberg Sink (Flink 1.20+/2.0+) is particularly powerful for production deployments: a single Flink job can ingest from any number of Kafka topics and write to any number of Iceberg tables, automatically handling new topics, schema evolution, and partition changes — without job restarts. Schema changes published to a registry propagate to downstream Iceberg tables automatically.

When to choose Flink over Spark:

• Sub-second end-to-end processing latency is a hard requirement
• Complex event processing with out-of-order events, late arrivals, and watermarks
• High-throughput streaming into Iceberg (100K+ events/second per table)
• Stateful computations — session windows, pattern detection — that must survive failures without data loss
• CDC pipelines from Debezium/Kafka where row-level exactly-once consistency matters
• Multi-table fan-out from a single stream source

When Spark still wins: Mixed batch+streaming on the same cluster, ML training pipelines that also need streaming input, or teams that don't want to operate two engines. Spark 4's improvements to Structured Streaming narrowed the gap for moderate-latency use cases (>1 second).

Important: Flink's streaming writes create the same small-file problem as any continuous writer — every checkpoint interval produces new files. A table receiving events every second accumulates thousands of files per day. Without continuous compaction, query performance degrades rapidly. This is exactly the pattern where an autonomous maintenance layer becomes essential — handling compaction at streaming pace without manual intervention.

3. Trino — Interactive Distributed SQL

Replaces Spark for: Ad-hoc analytics, multi-user BI queries, federated queries across data sources, and interactive SQL exploration.

Trino (formerly PrestoSQL) is a distributed SQL query engine designed for one thing: fast, interactive queries. While Spark's Catalyst optimizer is general-purpose (it handles ETL, ML, streaming, and SQL), Trino's architecture is purpose-built for the pattern that dominates most data platforms: analysts and BI tools running SQL against a lakehouse.

Trino is stateless — it doesn't store data, doesn't manage state between queries, and scales readers independently of writers. For concurrent, multi-user analytics on Iceberg tables, Trino consistently outperforms Spark SQL in query latency. Where Spark takes 5–15 seconds to plan and execute a filtered aggregation, Trino returns in 1–3 seconds.

Recent developments have made Trino even stronger on Iceberg. The connector now supports copy-on-write mode for row-level DELETE, UPDATE, and MERGE operations — eliminating the read-time overhead of merging delete files for read-heavy workloads. For analytics tables with high read/write ratios (dashboards, BI), this provides predictable, low-latency reads without requiring frequent compaction. The connector also supports Iceberg V2 positional deletes, dynamic partition pruning, and deep integration with Iceberg manifest statistics for efficient scan planning.

Key advantages:

• Federation: Query Iceberg, PostgreSQL, MySQL, Elasticsearch, MongoDB, and Kafka from a single SQL statement. No data movement required
• Concurrency: Handles hundreds of simultaneous users without degradation — BI dashboards, Looker, Tableau, Superset all hit Trino directly
• No cold start: Always-on workers eliminate the JVM bootstrap delay that plagues short Spark queries. First query is as fast as the hundredth
• Iceberg-native: Deep integration with Iceberg metadata — predicate pushdown into manifests, file-level min/max pruning, and the full maintenance procedure API (OPTIMIZE, expire_snapshots, remove_orphan_files)
• Cost-based optimizer: Uses table and column statistics to choose join strategies, determine scan order, and manage memory — important for complex multi-join queries at scale

When Spark still wins: Write-heavy ETL, complex multi-stage transformations with intermediate state, ML pipelines, and anything requiring persistent state between query executions. Trino is fundamentally a read-optimized engine — use it for querying, not for building pipelines.

Performance tip: Trino queries benefit enormously from well-maintained tables. Compacted files with proper sort orders let Trino's predicate pushdown skip 60–95% of data. If your Trino queries are slow, the first thing to check isn't the query — it's the physical layout of the underlying Iceberg tables.

4. DuckDB — Single-Node Analytics Without Infrastructure

Replaces Spark for: Local development, CI/CD data validation, notebook exploration, datasets under ~100 GB, and any analytics that doesn't require a cluster.

DuckDB is "SQLite for analytics" — an in-process, embedded OLAP database that runs inside your Python script, your notebook, or your CI pipeline. No cluster. No JVM. No network hops. pip install duckdb and you're running columnar analytics on Iceberg tables in seconds.

For datasets that fit on a single machine — which, with modern hardware (128 GB RAM, NVMe storage), means most of what data teams work with day-to-day — DuckDB eliminates Spark's overhead entirely. Queries that take Spark 8 seconds (between JVM startup, session creation, and query planning) finish in 50 milliseconds on DuckDB.

The v1.5.3 release (May 2026) significantly expanded DuckDB's Iceberg support: full MERGE INTO with merge-on-read semantics for atomic upserts, ALTER TABLE for schema evolution, bucket and truncate partition transforms for flexible data layout, and Iceberg V3 support including binary deletion vectors (Puffin files). DuckDB now covers the most common Iceberg write operations that previously required a Spark cluster — schema changes, upserts, and partitioned writes — all from a single embedded process.

Key advantages:

• Zero infrastructure: Runs in-process. No cluster provisioning, no coordinator nodes, no configuration files. Install is 50 MB
• Full Iceberg support: Reads and writes Iceberg tables directly on S3 via REST catalogs. MERGE INTO, schema evolution, partition pruning, and predicate pushdown
• Arrow integration: Zero-copy data exchange with pandas, Polars, and any Arrow-compatible tool. Results flow directly into your analysis code
• Cost: Free. Open source. No compute credits. Run it on a laptop, in AWS Lambda, in a GitHub Action, in a unit test
• V3 format: Supports reading and writing binary deletion vectors — more compact than V2 positional deletes

Limitations: Single-node by design — it cannot add machines. Very large tables (millions of files, 100+ GB manifests) can still challenge scan planning. For datasets over ~100 GB requiring distributed shuffles, Spark or Trino are the right choice.

Production pattern: DuckDB for development, CI validation, and small-to-medium production workloads. Spark or Trino for the distributed tail. Same Iceberg tables, same catalog, different engines matched to the workload.

5. Polars — Faster DataFrames Without a Cluster

Replaces Spark for: DataFrame transformations under ~50 GB, local ETL development, feature engineering, CI pipelines, and any Python data work that doesn't need distribution.

Polars is a Rust-backed DataFrame library that is categorically faster than PySpark for single-node workloads. No JVM. No serialization between Python and Java. No cluster overhead. Where PySpark spends 3–8 seconds on session creation before processing a single row, Polars finishes the entire job.

On TPC-H benchmarks, Polars outperforms pandas by 10–50x depending on query type. Against PySpark on identical hardware, Polars wins consistently for workloads under 10 GB because it avoids the coordination and serialization overhead that distributed systems impose even when running on a single node. The lazy execution API provides automatic query optimization — predicate pushdown, projection pruning, common subexpression elimination — giving you Catalyst-like planning without the JVM.

At larger scales, the picture shifts. A 2026 benchmark showed Spark becoming the fastest reliable option around 12.7 GB compressed, and by 127 GB compressed Spark was 3.5x faster than DuckDB and Polars failed to complete. The crossover point depends on your hardware and query complexity, but the general rule holds: single-node under 50 GB → Polars. Distributed over 100 GB → Spark.

Key advantages:

• Speed: Rust-native, multi-threaded, vectorized execution on Apache Arrow. 2x faster than DuckDB on small data at low core counts in benchmarks
• Ergonomics: pip install polars. Intuitive expression API with method chaining. No cluster configuration, no environment setup
• Memory efficiency: Apache Arrow columnar format with zero-copy operations. Lazy evaluation avoids materializing intermediate results
• Streaming mode: Handles larger-than-memory datasets via disk spill — though performance degrades once you leave pure in-memory execution
• Ecosystem: Growing integration with cloud platforms. Polars Cloud (beta) aims to bring distributed execution, though unproven at Spark-scale workloads

When Spark still wins: Workloads above 100 GB requiring distributed shuffles. Complex multi-hour fault-tolerant jobs. Integration with enterprise lakehouse platforms (Databricks Unity Catalog, Microsoft Fabric). Spark's ecosystem maturity, hiring pool, and operational tooling remain unmatched for large-scale production.

Practical pattern: Many teams use both — Polars for fast iteration during development and for small/medium production ETL, Spark for the jobs that genuinely need a cluster. Same transformation logic, different execution backends.

6. Apache DataFusion — The Rust Query Engine Substrate

Replaces Spark for: Embedded analytics engines, custom data systems, and the execution layer inside tools like LakeOps, InfluxDB 3.0, and RisingWave.

DataFusion is not a direct end-user competitor to Spark the way Trino or Flink are. It's a query engine library — the Rust-native, Arrow-based execution core that other systems embed. Think of it as "what you build query engines on top of" rather than "what you query data with directly."

Why it matters for this list: DataFusion is the engine behind LakeOps's 95%-faster-than-Spark compaction. It powers InfluxDB 3.0's analytics layer. RisingWave replaced its entire batch execution engine with DataFusion in version 2.8, achieving significant performance improvements on 100 GB TPC-H benchmarks against Iceberg on S3. It represents the architectural shift from JVM-based processing toward Rust+Arrow as the high-performance substrate.

The official iceberg-datafusion crate (part of Apache's iceberg-rust project) provides full Iceberg integration: catalog-backed table providers with automatic metadata refresh, predicate and limit pushdowns, DDL support, INSERT INTO with sort-based clustering to prevent small files, and merge-on-read resolution of positional and equality delete logs. You get native Iceberg table access in Rust — no JVM, no Spark, no external service.

Key advantages:

• Performance: Vectorized execution on Apache Arrow. No JVM overhead, no GC, no serialization between stages. Zero-copy data flow
• Extensibility: Implement custom table providers, UDFs, optimizer rules, and execution strategies. The library is designed to be composed, not just consumed
• Iceberg-native: Full TableProvider integration. Schema discovery, predicate pushdown, DDL, DML, and transactional reading of delete files
• Embeddable: Runs in-process. No separate service to deploy. 50 MB binary, starts instantly
• Community: Apache top-level project. Foundation of Arrow Ballista (distributed), Comet (Spark accelerator), and dozens of production systems

When to reach for DataFusion: If you're building a data product — a compaction engine, a query service, an embedded analytics backend, a CDC processor — and need a fast SQL execution layer without shipping the JVM. If you're a user of data tools rather than a builder, you'll interact with DataFusion through products that embed it (LakeOps for compaction, DuckDB for some operations, InfluxDB for time series, RisingWave for streaming analytics).

7. Ray — Distributed AI and ML Compute

Replaces Spark for: Distributed model training, hyperparameter tuning, LLM serving, reinforcement learning, batch inference, and GPU-heavy workloads.

Spark MLlib was the default for distributed ML for years. In 2026, it's been superseded by Ray for serious AI workloads. The reason is architectural: Spark is a coarse-grained, data-parallel, JVM-based system designed for SQL and ETL. Ray is a fine-grained, task-and-actor system designed for compute-intensive Python workloads on heterogeneous hardware — CPUs and GPUs mixed, fractional allocation, dynamic scaling.

Ray dispatches a task in ~200 microseconds. Spark stage launch takes 1–3 seconds. For workloads made of millions of small tasks — hyperparameter trials, RL rollouts, distributed inference batches — Ray finishes while Spark is still planning. This isn't a marginal improvement; it's a different programming model optimized for a different class of workload.

Key advantages:

• GPU-native: First-class fractional GPU scheduling (num_gpus=0.25). Spark has no equivalent — its barrier execution mode is coarse-grained and inflexible for modern training loops
• Python-first: No JVM interop. Your PyTorch/JAX/TensorFlow training code runs unmodified. Decorate a function with @ray.remote and it distributes
• Integrated AI stack: Ray Train (distributed training with fault-tolerant checkpointing), Ray Tune (ASHA, PBT, Optuna integration for hyperparameter search), Ray Serve (model serving with dynamic batching)
• Databricks integration: Runtime 15+ ships Ray as first-class. Run Spark DataFrame ETL then ray.init() in the same notebook for distributed training
• Streaming batch: Ray Data uses a streaming batch approach where batches flow between stages without full materialization — efficient for heterogeneous CPU/GPU pipelines where each stage has different memory and compute characteristics

When Spark still wins: Tabular data ETL, SQL transformations, feature stores built on structured data, and any workload where Catalyst's query optimizer provides value. Spark remains the standard for data preparation that feeds ML models. The production pattern in 2026: Spark for data prep, Ray for training and serving, shared S3/GCS between them.

8. dbt / SQLMesh — SQL Transformations Without Spark

Replaces Spark for: Data transformations that are expressible in SQL, model dependency management, incremental builds, and analytics engineering workflows.

Many Spark jobs are not doing anything that requires distributed compute. They're running SELECT → JOIN → GROUP BY → INSERT OVERWRITE. They're SQL transformations wrapped in PySpark boilerplate, deployed on Spark clusters, with all the operational overhead that entails — JVM tuning, executor memory, shuffle partitions, cluster provisioning.

dbt and SQLMesh express these transformations as pure SQL, manage dependencies between models, handle incremental logic, and execute against whatever engine makes sense — Snowflake, BigQuery, Trino, DuckDB, or Spark when the scale demands it. The transformation logic is decoupled from the compute engine.

dbt is the established standard with the largest ecosystem: 10,000+ community packages, integrations with every major warehouse and engine, the Semantic Layer for BI tools, and dbt Mesh for multi-team coordination. Most production data platforms run dbt today.

SQLMesh (now a Linux Foundation project after Fivetran donated it in 2026) takes a different architectural approach: virtual data environments that eliminate data duplication during development, column-level semantic change detection that rebuilds only what actually changed, and blue-green deployment safety. On consumption-billed compute (Snowflake, Fabric), SQLMesh's precision saves real money — it doesn't rebuild models that haven't changed.

Key advantages:

• Simplicity: SQL is code. No DataFrame API. No JVM. No cluster management. Version-controlled, testable, documented transformations
• Engine-agnostic: Write once, run on Trino, Snowflake, BigQuery, DuckDB, or Spark. Migrate engines without rewriting logic
• Incremental intelligence: SQLMesh detects at the column level what changed and rebuilds only the affected downstream models. dbt's incremental models require manual implementation but cover most patterns
• Ecosystem: dbt has community packages for almost every common pattern. SQLMesh has virtual environments and automatic cost optimization

When Spark still wins: Transformations requiring Python UDFs, ML feature engineering in the same pipeline, complex stateful logic that SQL can't express, or data volumes that exceed what SQL engines handle natively. For pure SQL transformation workloads — which is 80%+ of most platforms — Spark is overkill.

9. ClickHouse / StarRocks — Real-Time OLAP

Replaces Spark for: Real-time dashboards, high-concurrency analytics, sub-second aggregations on fresh data, and user-facing analytics embedded in applications.

If your use case is "hundreds of concurrent users querying the last hour of data with sub-second response times" — Spark is the wrong tool. It was never designed for that concurrency model or that latency target. A Spark SQL query serving a single dashboard panel takes 3–10 seconds including planning overhead. Multiply that by 200 concurrent users and you need a fleet of clusters.

ClickHouse and StarRocks are columnar OLAP engines built specifically for this access pattern. They maintain materialized views, pre-aggregate common queries, use vectorized execution tuned for analytical scans, and serve results from memory in single-digit milliseconds. Both now integrate with Iceberg as an external table source — meaning they read your lakehouse data while maintaining their own hot layer for real-time queries.

ClickHouse owns the real-time event analytics niche: log analytics, observability, clickstream analysis, and time-series aggregations. Its MergeTree engine is optimized for append-heavy, time-ordered data with automatic background merges. ClickHouse Cloud provides a managed serverless option with separation of storage and compute.

StarRocks excels at high-concurrency, multi-user BI workloads. Its MPP architecture handles hundreds of simultaneous queries with predictable latency. Native Iceberg catalog support (via REST) means it reads your existing Iceberg tables directly — no data copying. StarRocks is a strong choice when you need a lakehouse query accelerator for BI dashboards.

Key advantages:

• Latency: Single-digit millisecond queries on pre-aggregated data. P99 under 100ms on cached queries. Spark can't match this at any cluster size
• Concurrency: Thousands of simultaneous queries without degradation. Purpose-built for multi-tenant, user-facing analytics
• Iceberg integration: StarRocks reads Iceberg via REST catalog natively. ClickHouse supports Iceberg table functions. Both benefit from well-compacted, properly-sorted Iceberg tables
• Real-time ingestion: Streaming inserts from Kafka with automatic materialization. Data is queryable within seconds of arrival
• Materialized views: Pre-compute expensive aggregations. Queries hit pre-built results rather than scanning raw data every time

When Spark still wins: Ad-hoc exploration of raw data (no pre-aggregation possible), complex multi-stage transformations, and workloads where the query patterns aren't predictable enough to optimize for.

The Bigger Picture

The lakehouse architecture made this multi-engine world possible. When data lives in open formats (Iceberg, Parquet) on object storage, accessed through standard catalogs (REST, Polaris, Glue) — any engine can read the same tables. There's no lock-in to one compute layer. That's the whole point of the open lakehouse.

But multi-engine creates a new problem: maintenance. Every engine benefits from well-maintained tables — compacted files, current statistics, sorted data, clean manifests. Trino queries skip more data. DuckDB scans finish faster. ClickHouse reads fewer files. Snowflake scans cost less. And none of them handle this maintenance themselves.

That's the gap a lakehouse control plane fills. It sits across all your engines, learns from all their query patterns, and keeps every table in optimal physical shape — so every engine performs at its ceiling rather than degrading under accumulated technical debt. The compaction, snapshot expiration, orphan cleanup, and layout optimization that used to require Spark clusters and maintenance scripts now runs autonomously, continuously, and at a fraction of the cost.

The best data platforms in 2026 don't run Spark for everything. They run the right engine for each workload, with an operational layer that keeps the underlying tables healthy. That's the architecture to aim for.

Thanks for reading and happy compaction :) 🍻

Jonathan

Learn more: