Every Apache Iceberg table degrades over time. Streaming pipelines produce thousands of small files per hour. Updates and deletes accumulate position delete files that every read must reconcile. Manifests fragment. Snapshots pile up. Without compaction, query planning that once took milliseconds starts taking minutes, and storage costs climb without anyone noticing until the bill arrives.

Compaction — merging small files into larger ones, applying pending deletes, and reorganizing data layout — is the single most impactful maintenance operation for Iceberg tables. But the landscape of tools that perform it has expanded significantly. Some are embedded in query engines, others are cloud-managed services, and a few are purpose-built systems designed to handle compaction as part of a broader operational loop.

This guide compares seven production-grade compaction engines available in 2026, evaluating each on the dimensions that matter at scale: compaction strategy, layout optimization, automation model, cost efficiency, multi-engine awareness, and how well compaction integrates with the rest of the maintenance lifecycle.

What makes a compaction engine good

Before comparing tools, it helps to understand what separates adequate compaction from excellent compaction:

Strategy diversity. Bin-packing (merging small files without reordering) is the baseline. Sort-order compaction physically reorders data by specified columns so Parquet min/max statistics enable aggressive data skipping. Z-order handles multi-dimensional filter patterns. The best engines choose the right strategy per table, not globally.

Layout intelligence. Choosing which columns to sort by is often more impactful than the compaction itself. An engine that understands actual query patterns — which columns appear in WHERE, JOIN, and GROUP BY clauses — can optimize layout for the workload rather than relying on guesswork at table creation time.

Trigger model. Cron-based scheduling wastes compute on tables that don't need compaction and misses tables that do. Event-driven or threshold-based triggers — firing compaction when file count, average size, or delete ratio crosses a boundary — are significantly more efficient.

Maintenance sequencing. Compaction doesn't exist in isolation. Running it before snapshot expiration means rewriting files that are about to be garbage-collected. Running it after orphan cleanup means operating on a clean dataset. The order matters: expire snapshots → clean orphans → compact → rewrite manifests → generate statistics. Getting this wrong wastes compute without improving table health.

Engine efficiency. JVM-based compaction (Spark, Trino) carries garbage collection pauses, executor provisioning overhead, and memory pressure that compounds across hundreds of tables. Native engines avoid these costs entirely.

Conflict awareness. On tables with continuous streaming writes, compaction must coordinate with active writers to avoid commit conflicts. An engine that blindly rewrites files being actively written to will fail, retry, and waste resources.

Here's a quick comparison matrix cheat sheet:

Let's dive in :)

1. LakeOps

LakeOps is a dedicated lakehouse control plane built specifically for Apache Iceberg. Compaction is not a bolt-on feature — it is the core of an intelligent system-level optimization loop that coordinates file optimization with every other maintenance operation across the entire lake.

The platform connects to existing Iceberg catalogs (AWS Glue, Apache Polaris, REST catalogs, S3 Tables, Nessie, Gravitino) and object storage without moving data or changing pipelines. Your data stays in your account; LakeOps reads metadata and query telemetry to drive operation execution and optimization decisions.

The result is full automation with a system-wide view, where each operation is optimized based on actual telemetry data. For example, compaction is done according to each table's actual query patterns — so files are organized in a way that makes queries run up to 12x faster and reduces CPU costs by up to 80%. This is just one example.

What sets it apart

A Rust-based engine built on Apache DataFusion. Unlike every other tool on this list, LakeOps does not use Spark or any JVM-based engine for compaction. The execution engine processes Parquet data through Arrow columnar buffers with bounded memory, lock-free parallelism, zero garbage collection, and no executor provisioning. The performance difference is dramatic: in production benchmarks across 10 tables totaling 5.5 TB, bin-pack compaction completed in 221 seconds versus 1,612 seconds for Spark — 86% faster. File counts dropped from 101K to 19K. On a 1.2 TB table that caused Spark to OOM, the Rust engine completed compaction in 11 minutes.

What this means in practice: less CPU consumed per compaction run, fewer commit conflicts because operations complete faster (shorter compaction windows mean less overlap with concurrent writes), and the ability to compact aggressively without provisioning dedicated Spark clusters. The engine also self-improves — it records per-table throughput, partition structure, and memory usage from each run, so subsequent passes execute faster as the planner converges on optimal resource allocation. In production, the same table went from 22 minutes to 11 minutes across consecutive runs with zero configuration changes.

Query-aware file organization: LakeOps collects telemetry across every connected query engine — Trino, Spark, Snowflake, Athena, DuckDB, Flink — and identifies which columns appear in WHERE, JOIN, and GROUP BY clauses for each table. Compaction then applies the sort order that maximizes data skipping for the actual workload, not a static order defined at table creation. The planner evaluates single-column sort, multi-column sort, and z-order independently per table. As query patterns shift, sort orders adapt on subsequent compaction passes.

Event-driven triggers, not cron. The system continuously monitors per-table structural signals — file count, average file size relative to target, delete-file-to-data-file ratio, manifest depth. Compaction fires only when thresholds are crossed. A streaming table receiving continuous Flink writes may compact multiple times per hour; a weekly batch table compacts once. No wasted runs on idle tables, no missed degradation on active ones.

Sequenced with the full maintenance stack. Compaction runs as part of a coordinated pipeline: (1) snapshot expiration removes stale data → (2) orphan cleanup removes unreferenced files → (3) compaction rewrites the clean, current dataset → (4) manifest optimization consolidates metadata against the new layout → (5) statistics generation produces Puffin files for query planning. Each operation's output becomes the next one's clean input. One production deployment can remove approximately 200 TB of orphan data across 324 tables in under 15 minutes — the kind of cleanup that makes subsequent compaction dramatically cheaper and faster.

Conflict-aware scheduling. On tables with active streaming writers (Flink, Kafka Connect), LakeOps identifies which partitions are being appended to and excludes those files from the compaction plan. This eliminates the most common class of commit conflicts — compaction trying to rewrite files that are simultaneously being modified by ingestion — so compaction runs complete on the first attempt in the vast majority of cases.

Layout Simulations. Changing a sort order rewrites every data file in a table. If the new layout performs worse for the dominant query pattern, you've paid the full rewrite cost with negative returns. LakeOps Layout Simulations test proposed changes by applying the layout on an Iceberg branch, replaying production queries against it, and comparing cost/performance impact before modifying production data.

Lake-wide policies with per-table precision. Compaction rules are declared as policies with trigger conditions, strategy, and scope. Scope follows a specificity hierarchy: table-level overrides namespace, which overrides catalog-wide defaults. Platform teams set baselines for the entire lake; individual teams override for their SLAs. Policies are versioned with a full audit trail.

Delete file handling. Position and equality delete files are physically applied during compaction in the same pass — so readers stop paying reconciliation overhead on every query. Standalone delete file rewriting is also available as a separate operation for addressing delete accumulation independently.

Full observability. Every compaction operation is logged with before/after file counts, duration, throughput, and cost impact. Health classification (Critical/Warning/Healthy) surfaces degradation across the entire lake before users notice.

Proactive alerts at four severity levels cover excessive manifests, partition skew, small-file buildup, and delete ratio drift. At the table level, an Insights engine pinpoints exactly what's wrong — a high alert for 92 manifest files where 50 is the threshold, a warning for partition skew, an early note for small-file accumulation — each linking directly to the affected table.

You can dive into any table see it's current and reccomended setting, propertis, partioning strategy and everything else right on top of your Iceberg tables.

Here for example you can visually see the imapct of a triggered compaction on this production table:

Best for: Teams running hundreds to thousands of Iceberg tables across multiple catalogs and engines who need autonomous, cost-efficient compaction as part of a complete maintenance system — without provisioning Spark clusters or writing custom scheduling logic.

Explore the LakeOps platform →

More resources →

2. Apache Spark (rewrite_data_files)

Apache Spark is the original and most widely used compaction tool for Iceberg. The rewrite_data_files procedure is part of the Iceberg library itself and runs as a Spark SQL call.

How it works

Compaction is invoked manually or via a scheduler:

CALL catalog.system.rewrite_data_files(
    table => 'db.events',
    strategy => 'sort',
    sort_order => 'event_date ASC, user_id ASC',
    options => map('target-file-size-bytes', '536870912')
);

Three strategies are supported: binpack (merge without reordering), sort (global sort by specified columns), and zorder (multi-dimensional clustering). You can scope compaction to specific partitions with a WHERE clause to avoid touching active write partitions.

Strengths

• Maximum flexibility. Every parameter is configurable — target file size, min/max input files, partial progress commits, partition filters. Data engineers have full control over what gets compacted and how.
• Ecosystem maturity. Spark is available everywhere Iceberg runs. EMR, Dataproc, Databricks, standalone clusters — if you have Spark, you have compaction.
• Sorting support. Sort and z-order strategies produce optimized layouts, not just consolidated files.

Limitations

• No automation. Spark does not decide when to compact. You need external orchestration (Airflow, Step Functions, cron) to schedule runs, monitor results, and handle failures.
• JVM overhead. Garbage collection, executor provisioning, and cluster idle time add cost. For a lakehouse running compaction across hundreds of tables daily, Spark cluster costs are significant — typically around $50/TB versus $5/TB for native engines.
• No workload awareness. Sort columns must be specified manually. Spark has no mechanism to observe query patterns and adjust sort order accordingly.
• No maintenance coordination. Compaction runs independently of snapshot expiration, orphan cleanup, and manifest rewriting. Engineers must build sequencing logic themselves.
• Conflict risk. Compaction on tables with active streaming writes frequently produces commit conflicts that require retry logic.

Best for: Teams with existing Spark infrastructure who need full control over compaction parameters and are willing to build orchestration around it. The default choice when no dedicated compaction system is in place.

3. Databricks Predictive Optimization

Databricks Predictive Optimization automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables — including both Delta Lake and Apache Iceberg.

How it works

Predictive Optimization is enabled by default for new Databricks accounts (rollout to existing accounts expected to complete by August 2026). Once enabled, Databricks identifies tables that would benefit from maintenance, queues the operations, and executes them on serverless compute. No manual scheduling required.

For Iceberg tables, OPTIMIZE performs file compaction with optional Liquid Clustering — Databricks' approach to adaptive data layout that replaces static partitioning and z-ordering. The system collects statistics during writes and uses them to determine when and how aggressively to compact.

Strengths

• Fully automated. No DAGs, no cron, no manual scheduling. The system decides when to compact based on table state.
• Liquid Clustering. An intelligent layout strategy that adapts to query patterns over time, replacing the need to manually choose partition columns or sort orders.
• Bundled maintenance. OPTIMIZE, VACUUM, and ANALYZE run as a coordinated set, though the sequencing is managed by Databricks internally.
• Serverless execution. Compaction runs on Databricks-managed compute, so there's no cluster to provision or manage.

Limitations

• Unity Catalog lock-in. Only works for Unity Catalog managed tables. Tables registered in Glue, Polaris, Nessie, or any other catalog are excluded.
• Databricks-only. This is a platform feature, not a standalone tool. You must run your lakehouse on Databricks to benefit.
• Limited multi-engine awareness. Optimization decisions are based on Databricks query patterns. If Trino, Athena, or DuckDB also query your tables, their access patterns are invisible to the optimizer.
• Iceberg support is newer. Predictive Optimization was originally built for Delta Lake. Iceberg support, while GA, has a shorter track record in production.
• Cost opacity. Runs on serverless compute with Databricks pricing, which can be difficult to predict at scale.

Best for: Teams fully committed to the Databricks ecosystem with Unity Catalog as their primary catalog. If you're already on Databricks, this is the path of least resistance for automated compaction.

4. AWS Glue Auto Compaction

AWS Glue provides table optimizers for Iceberg tables registered in the Glue Data Catalog. Auto compaction monitors partitions and merges small files automatically.

How it works

You enable compaction through the Glue console, CLI, or API by configuring a table optimizer with an IAM role. Supported strategies include binpack (default), sort, and z-order. Glue evaluates partitions and triggers compaction based on internal heuristics. The service also offers snapshot retention and orphan file deletion as separate optimizers.

Configuration is per-table — you select the strategy, target file size, and assign an IAM role with the necessary permissions. Catalog-level optimization can apply defaults across all tables in a database.

Strengths

• AWS-native. No additional infrastructure. If your tables are in Glue, enabling compaction is a configuration change.
• Three strategies. Binpack, sort, and z-order are all available, providing layout optimization options beyond simple file merging.
• Bundled with other optimizers. Snapshot retention and orphan file deletion are available alongside compaction as separate table optimizers.
• Catalog-level defaults. You can set compaction policies at the database level so new tables inherit them automatically.

Limitations

• Glue-only. Tables must be registered in the AWS Glue Data Catalog. Tables in Polaris, REST catalogs, or Nessie are not supported.
• No query awareness. Sort columns are specified manually. The optimizer does not observe query patterns to inform layout decisions.
• Limited observability. Monitoring compaction results requires CloudWatch metrics and Glue job logs. There's no unified health dashboard for table compaction status.
• No maintenance sequencing. Compaction, snapshot retention, and orphan deletion run as independent optimizers with no coordination between them.
• AWS lock-in. Tied to the AWS ecosystem entirely.

Best for: AWS-native teams using the Glue Data Catalog who want managed compaction without running Spark clusters. A good default for straightforward AWS Iceberg deployments.

5. Amazon S3 Tables

Amazon S3 Tables provides fully managed Iceberg tables in dedicated S3 table buckets. Compaction is handled automatically as part of the managed service — no configuration, no optimizer setup, no IAM roles to manage.

How it works

When you create a table in an S3 table bucket, AWS manages the complete lifecycle: compaction, snapshot expiration, and orphan file removal run continuously in the background. The service handles everything from file sizing to metadata cleanup. AWS reduced compaction processing fees by up to 90% in July 2025, making it one of the most cost-effective managed options.

Strengths

• Low operational overhead. Compaction is invisible. No configuration, no monitoring, no scheduling. Tables stay healthy automatically.
• Cost-efficient. Reduced processing fees and no Spark cluster costs make this the cheapest compaction option for straightforward workloads.
• Full lifecycle management. Compaction, snapshot expiration, and orphan cleanup are all managed — not just file merging.

Limitations

• S3 table buckets only. Your data must live in dedicated S3 table buckets, not standard S3 buckets. Migrating existing tables requires data movement.
• No sort optimization. Compaction is bin-pack only. There's no sort-order or z-order compaction to optimize data layout for query patterns.
• No configurability. You cannot set target file sizes, choose strategies, or control when compaction runs. The service operates on its own internal schedule and heuristics.
• Limited engine access. While any Iceberg-compatible engine can read from S3 Tables, the catalog (s3tablescatalog) is AWS-specific and not all engines have mature support yet.
• No query awareness. The service has no visibility into how engines are querying the data, so layout optimization is impossible.

Best for: New Iceberg deployments on AWS where simplicity matters more than layout optimization, and where the team is comfortable with the S3 table bucket model.

6. Dremio Automatic Optimization

Dremio automates table maintenance for Iceberg tables in the Dremio Open Catalog (formerly Arctic). The optimization service handles data file compaction, delete handling, clustering, partition evolution, and manifest rewriting.

How it works

Automatic optimization runs on a dedicated engine configured by Dremio, separate from query workloads. The service evaluates file sizes, partition layout, metadata organization, and accumulated row-level deletes. Five operations run continuously: data file compaction, delete file handling (both v2 position deletes and v3 deletion vectors), clustering by specified keys, partition evolution, and manifest rewriting.

You can also run these operations manually via OPTIMIZE TABLE for one-off situations. Incremental processing handles tables in batches to prevent resource exhaustion.

Strengths

• Comprehensive maintenance. Five operations — not just compaction — run as a coordinated set, including delete handling and partition evolution.
• Clustering support. Data can be clustered by specified keys to improve query performance, similar to sort-order compaction.
• Separate compute. Optimization runs on dedicated engines, so it doesn't compete with query workloads.
• Incremental processing. Large tables are processed in batches, avoiding OOM errors and resource exhaustion.
• V3 support. Handles both v2 position deletes and v3 deletion vectors.

Limitations

• Dremio ecosystem only. Available for tables in the Dremio Open Catalog. Tables in Glue, Polaris, or other catalogs are not covered.
• No cross-engine telemetry. Optimization is based on Dremio's view of the workload. If Trino, Spark, or Athena also query the same tables, their patterns are invisible.
• Cloud-only. Automatic optimization is a Dremio Cloud feature. Self-hosted Dremio requires manual OPTIMIZE TABLE commands.
• Limited policy granularity. Per-table tuning options are more limited compared to dedicated optimization platforms.

Best for: Teams using Dremio as their primary lakehouse platform who want maintenance handled without custom tooling. The most complete integrated option among query engines.

7. Apache Flink TableMaintenance

The Flink TableMaintenance API is the newest entry in this space — a native Iceberg maintenance framework that runs compaction, snapshot expiration, and orphan cleanup directly within Flink streaming jobs.

How it works

The API provides a maintenance topology that can be embedded within existing Flink streaming pipelines or deployed as standalone Flink jobs. You build a maintenance graph using TableMaintenance.forTable() or TableMaintenance.forChangeStream() and add operations:

TableMaintenance.forTable(env, tableLoader)
    .add(RewriteDataFiles.builder()
        .targetFileSizeBytes(256 * 1024 * 1024)
        .partialProgressEnabled(true)
        .build())
    .add(ExpireSnapshots.builder()
        .maxSnapshotAgeMs(TimeUnit.DAYS.toMillis(3))
        .build())
    .build();

The framework includes coordinator-based locking (merged in early 2026) to prevent concurrent maintenance operations without requiring external lock services.

Strengths

• Streaming-native. Compaction runs inside the same Flink runtime as your ingestion pipeline, eliminating the need for a separate Spark cluster solely for maintenance.
• Integrated with writes. For Flink-based streaming pipelines, maintenance can be embedded directly in the sink topology, running compaction immediately after commits.
• Multiple operations. Supports compaction, snapshot expiration, and orphan cleanup — not just file merging.
• No external dependencies. With coordinator-based locking, no external lock service (ZooKeeper, DynamoDB) is needed.

Limitations

• Flink-only. The API is Flink-specific. Teams not running Flink need a different compaction solution.
• No query awareness. Sort columns must be specified manually. There's no telemetry or workload analysis to inform layout decisions.
• Operational maturity. The streaming maintenance API is relatively new (GA in 2026). Production track record is shorter compared to Spark procedures or dedicated platforms.
• Limited observability. Monitoring requires Flink metrics and logging infrastructure. No unified dashboard for table health or compaction status.
• No lake-wide coordination. Each Flink job manages its own tables independently. There's no centralized view or policy management across the lakehouse.

Best for: Teams running Flink as their primary streaming engine who want to consolidate ingestion and maintenance in a single runtime, eliminating the Spark dependency for table maintenance.

How to choose

If you need autonomous, cost-efficient compaction across a multi-catalog, multi-engine lakehouse (while still keeping manual and even organiztional wide control— LakeOps. It's the only option that combines a purpose-built Rust engine, query-aware layout optimization from cross-engine telemetry, event-driven triggers, full maintenance sequencing, and lake-wide policy management in a single platform.

If you want maximum manual control and already run Spark — Apache Spark's rewrite_data_files. You accept the operational burden of scheduling, monitoring, and sequencing in exchange for complete flexibility over every parameter.

If you're fully on Databricks — Predictive Optimization. Zero-configuration compaction for Unity Catalog managed tables with Liquid Clustering for adaptive layouts.

If you're AWS-native with Glue Catalog — AWS Glue Auto Compaction. Managed compaction with sort and z-order support, tightly integrated with the AWS ecosystem.

If you want zero-touch simplicity on AWS — S3 Tables. Accept the limitation of bin-pack-only compaction in exchange for a fully managed lifecycle.

If Dremio is your lakehouse engine — Dremio Automatic Optimization. The most complete integrated maintenance among query engines, with five coordinated operations.

If Flink is your primary streaming engine — Flink TableMaintenance. Eliminate the Spark dependency for maintenance by running compaction inside your existing Flink runtime.

The compaction landscape in 2026 reflects a broader shift in how teams operate Iceberg lakehouses. Manual Spark jobs are giving way to automated systems that understand table state, coordinate with other maintenance operations, and optimize layouts based on actual workload patterns. The question is no longer whether to automate compaction — it's whether your compaction engine is smart enough to make the right decisions autonomously, efficient enough to run at scale without breaking the budget, and integrated enough to keep every table healthy without a dedicated maintenance team.

Thanks for reading!

Learn more: