Apache Iceberg gives you ACID semantics, schema evolution, partition evolution, and time travel on object storage. What it does not give you is a pager that explains why a dashboard that ran in three seconds yesterday now hangs for forty. The format is sound. The failure modes are operational.

Every production Iceberg incident we have helped teams debug traces back to the same root: maintenance that did not run, ran at the wrong time, ran in the wrong order, or ran without awareness of the table's current state. Small files accumulate because compaction could not keep up. Manifests fragment because nobody rewrote them after compaction. Storage grows because orphan cleanup never followed snapshot expiration. Writers fail with CommitFailedException because compaction targeted the hot partition. The official Iceberg maintenance docs are explicit about the primitives. The path from symptom to root cause to fix is what most teams invent under pressure at 2 AM.

This guide lays out both debugging routes. First, how a control plane like LakeOps turns Iceberg debugging from a reactive scramble into a proactive observability workflow — with continuous health classification, actionable insights, cross-engine telemetry, and an audit trail for every operation. Then the full manual debugging playbook: the metadata queries, the triage decision tree, and every common incident with diagnosis, fix, and prevention. Along the way we link to deeper LakeOps guides on incident runbooks, small files, orphan cleanup, maintenance automation, and more.

In this article

• Two routes to debugging Iceberg
• Route 1: Observability and a control plane
• Route 2: Manual debugging with metadata queries
• Common incidents: slow queries, planning, conflicts, storage, OOMs
• More incidents: time travel, MOR deletes, schema, wrong results, catalog
• Writers, the safe fix sequence, and choosing your route
• Quick reference cheatsheet

Two routes to debugging Iceberg

Every Iceberg debugging session starts with the same question: what is wrong with this table? How you answer it depends on your tooling.

Route 1: Observability-first debugging. A control plane continuously reads Iceberg metadata across every table in your lake. It classifies health, surfaces insights ranked by severity, logs every maintenance operation with before/after metrics, and collects query telemetry from every connected engine. When something goes wrong, you open a dashboard, see which tables are Critical, drill into the specific signals driving that classification, and either let the system remediate automatically or trigger a targeted fix. Diagnosis takes seconds. You spend your time understanding why the degradation happened, not whether it happened.

Route 2: Manual metadata debugging. You write SQL against Iceberg metadata tables — .files, .snapshots, .manifests, .all_delete_files — and piece together the picture yourself. You cross-reference file counts with snapshot history, check manifest fragmentation, compute delete ratios, and run maintenance procedures one by one in the correct sequence. It works. It scales to a few dozen tables before the operational overhead becomes a full-time job.

Most teams start with Route 2 and graduate to Route 1 when the pain of manual triage exceeds the effort of adopting a control plane. This guide covers both — starting with the observability approach because understanding what healthy debugging looks like makes the manual path more purposeful.

Route 1: debugging with observability and a control plane

LakeOps is a dedicated control plane for Apache Iceberg. It connects to your existing catalogs (AWS Glue, REST/Polaris, Nessie, S3 Tables, Gravitino, Lakekeeper) and query engines (Trino, Spark, Flink, Snowflake, Athena, DuckDB) without moving data or changing pipelines.

LakeOps reads metadata and telemetry, executes maintenance through standard Iceberg APIs, and commits results back through the catalog.

Setup takes roughly ten minutes: point it at your catalog, and it discovers every table, reads metadata, and starts classifying health. Your data stays in your storage account. Then, your dashboard will start shwoing data.

What follows is how each surface in LakeOps maps to a debugging capability — the same capabilities you would otherwise build from scratch with SQL monitors, Airflow DAGs, and custom dashboards.

Autonomous maintenance

Observability tells you what is wrong. Autonomous maintenance is what actually fixes it — and prevents it from recurring.

LakeOps is not a monitoring tool that surfaces alerts and leaves remediation to you. It is an operational control plane that closes the loop: detect degradation, execute the correct fix, verify the outcome, and feed results into the next decision. Most of the incidents described later in this guide — small file explosions, manifest fragmentation, orphan accumulation, snapshot bloat, delete-file read amplification — resolve at the Warning stage before they become user-visible. Here is what the autonomous maintenance layer covers.

Compaction

The core operation — and the highest-leverage one. LakeOps merges small files into optimally sized targets, physically applies pending delete files, and sorts data by the columns your queries actually use. The engine is written in Rust on Apache DataFusion: Arrow columnar buffers, bounded memory, zero garbage collection, lock-free parallelism. Partitions that crash Spark with OutOfMemoryError complete in minutes at roughly $5/TB versus $50/TB on Spark — a 10x cost reduction that makes 4–8 compaction passes per day economically viable instead of one nightly job.

Compaction is also query-aware. LakeOps collects query telemetry from every connected engine and identifies which columns appear in WHERE, JOIN, and GROUP BY clauses per table. During sort compaction it applies the optimal sort order based on actual production access patterns, not a static configuration someone guessed at during table creation. When access patterns shift, the sort order adapts on subsequent passes. The result: Parquet min/max statistics let engines skip entire file groups, cutting scan I/O, reducing CPU, and accelerating query times — without modifying a single query. Layout Simulations let you preview the projected data-skipping improvement before committing.

Compaction is conflict-aware too: it inspects active writer state, targets only cold partitions, and retries only the affected file group on the next cycle if a boundary-case conflict occurs. For full benchmark details, see the compaction tools comparison.

Snapshot expiration

Every Iceberg write creates a snapshot. Without expiration, thousands accumulate within weeks on streaming tables, deepening the metadata tree and dragging down query planning. LakeOps expires snapshots continuously with configurable retention policies — per table, per namespace, or per catalog. Safety windows protect long-running queries from having their active snapshot deleted mid-flight. Tagged snapshots for compliance or audit checkpoints are excluded regardless of age.

Orphan file cleanup

Failed writes, aborted compaction jobs, and crashed pipelines leave data files on object storage that no snapshot references. These orphans are invisible to queries but fully visible to your storage bill — on mature lakes they routinely account for 25–40% of billable storage on affected prefixes. LakeOps detects and removes orphan files after snapshot expiration (so newly dereferenced files are caught in the same sweep), enforces a safe age threshold to protect in-flight writes, and logs every file removed with bytes reclaimed.

Manifest and metadata optimization

Manifests are the index layer between snapshots and data files. After many append and compaction cycles, hundreds of tiny manifests accumulate — each one an Avro file the query planner must open and evaluate. LakeOps rewrites manifests after every compaction cycle, consolidating fragmented entries into a compact index. It also manages position delete file compaction, computes Puffin column statistics for tighter file pruning, and handles metadata.json file cleanup to prevent the metadata layer from growing unbounded.

Smart, sequenced, event-driven execution

The order of operations matters more than most teams realize until they get it wrong. The correct sequence — expire snapshots, remove orphans, compact data files, rewrite manifests, refresh statistics — is enforced as a coordinated pipeline on every cycle. Compacting before expiration rewrites files about to become unreferenced. Cleaning orphans before expiration misses the largest reclaimable set. Rewriting manifests before compaction produces an index that goes stale minutes later. LakeOps encodes this sequence so it cannot be inverted at 3 AM.

Operations are triggered by table state, not cron schedules. LakeOps continuously collects telemetry — file counts per partition, snapshot depth, delete ratios, manifest growth — and uses those signals to decide what to run, when, and on which tables. A streaming table that crosses 500 small files at 3 PM gets compacted at 3 PM. A batch table that loaded once this week and is structurally healthy gets left alone. No wasted runs on idle tables, no 23-hour gaps where a streaming table degrades between nightly jobs. For the full rationale, see Automating Apache Iceberg Table Maintenance.

Multi-engine coordination

Iceberg tables serve Trino, Spark, Snowflake, Athena, DuckDB, and Flink simultaneously. Each engine has its own planning behavior and sensitivity to file layout. LakeOps ingests telemetry from all of them and makes maintenance decisions that account for the full workload — not just the last engine someone tuned for. A sort order that benefits 80% of Trino dashboard queries does not get overridden because a nightly Spark ETL ran last. Multi-engine query routing directs workloads to the right engine by cost, latency, or throughput — and the routing layer feeds back into maintenance prioritization.

Policies and governance

Individual table fixes do not scale. LakeOps policies are declarative rules for compaction targets, snapshot retention, orphan cleanup thresholds, and manifest optimization — scoped at organization, catalog, namespace, or individual table level, cascading with inheritance. Every policy execution is logged with duration, impact, and outcome. Retention management includes GDPR-compliant deletion with full audit trails and coordination with compaction so expired data is physically removed. Cross-catalog enforcement works uniformly across AWS Glue, Polaris, Nessie, and Gravitino.

Agentic AI readiness

AI agents and autonomous pipelines need healthy tables but hit them harder and less predictably. LakeOps provides an MCP (Model Context Protocol) interface with schema discovery, async queries, and PostgreSQL/MySQL/Arrow Flight wire compatibility. Layered guardrails — ReadOnly, CostEstimate, PIIMask, HumanApproval — are configurable per agent session. Agent query telemetry feeds back into compaction and sort-order decisions, so the lake self-optimizes for the workloads actually running, human and machine alike.

The net effect: 10x lower compaction costs, significantly faster queries through data-skipping, and an engineering team that stops writing maintenance scripts and starts treating table health as something the platform guarantees.

Watch:

Health classification — know before users complain

The first thing you see after connecting a catalog is a lake-wide dashboard. Every table is continuously scored and classified into one of three states:

• Healthy — structural indicators within target bounds. File counts, manifest depth, snapshot retention, delete ratios, and sort alignment are all within the ranges you define.
• Warning — degradation is underway. Small files are accumulating, manifests are fragmenting, or snapshots are building up. No user-visible impact yet, but without action it will reach Critical.
• Critical — severe fragmentation, metadata bloat, or structural damage. Planning or scan performance is at risk. This is the state that generates user complaints and 2 AM pages.

Health is computed from the same Iceberg signals that matter in manual debugging: file count and size distribution per partition, manifest count and depth, snapshot accumulation, delete-file ratio, partition skew, and sort-order alignment with real query patterns. The difference is that the control plane evaluates these signals continuously across every table in the lake — not one table at a time when someone asks "why is this slow?"

The dashboard shows aggregate counts, total data volume across catalogs, storage trends, and a summary of recent operations. Platform teams can triage the entire lake in one screen without writing a single query.

Proactive insights — prioritized, actionable alerts

Health classification tells you which tables are sick. Insights tell you why and what to do about it.

LakeOps evaluates tables on a continuous schedule and raises prioritized findings. Each insight is tied to a specific table, a severity level, and a recommended next step. Examples from production:

• CRITICAL — raw_clickstream: 312 partitions exceed file threshold. Query scan amplified 8x.
• HIGH — search_query_logs: excessive manifests (487). Planner latency +2.1s.
• WARNING — payment_transactions: small file ratio 38%. S3 GET costs elevated.
• LOW — early drift you can fix before the next compaction window.

In a manual debugging workflow, you discover these conditions after a user files a ticket. With Insights, you discover them while they are still at Warning — before they escalate to a production incident. Each finding links directly to the affected table with remediation options: fix manually, adjust a policy, or let autonomous maintenance handle it.

Table-level investigation — Explore, Metrics, and Partitions

When you need to drill into a specific table, LakeOps provides the investigation surfaces you would otherwise build with SQL:

Explore view. The full structural picture for any table: records over time across recent snapshots, active file counts, stale file counts, delete file counts, file-size histograms showing the percentage of files in optimal range versus undersized, and snapshot-level growth trends. This is the same data you would pull from .files, .snapshots, and .all_delete_files metadata tables — pre-joined, charted, and updated continuously.

Metrics tab. Position and equality delete tracking for merge-on-read tables. Records distribution across snapshots to spot write-pattern changes. File size distribution showing where compaction would have the most impact.

Partitions view. Per-partition breakdown of file counts, byte distribution, and delete-file concentration. This is the view that answers "which partition is driving the planning timeout?" and "where is the streaming write explosion happening?" — the questions that take 15 minutes of SQL and GROUP BY in a manual workflow.

Every chart and metric in these views maps directly to the manual diagnosis queries you would write. The difference is speed: instead of running four SQL queries per table and eyeballing thresholds, you open one tab and see the answer with historical context.

Events and audit trail — what changed and when

Every maintenance operation — compaction, snapshot expiration, orphan removal, manifest rewrite — is logged lake-wide and per table with duration, impact, and status. The Events view shows a live stream:

• Compact Data Files · customer_orders · 1.24 TB, 16 → 1 files · 4s · OK
• Expire Snapshots · payment_transactions · 12 snapshots expired · 4.6s · OK
• Remove Orphan Files · user_sessions · 847 MB reclaimed, 1,203 files · 1m 12s · OK
• Rewrite Manifests · search_query_logs · 487 → 12 manifests · 2.1s · OK

During incident response, the audit trail answers the question that usually takes an hour of Spark log archaeology: "what maintenance ran on this table, when, and did it succeed?" Filter by catalog, operation type, success or failure. See before/after file and manifest counts on every event. This is the operational log you reference during RCA instead of parsing executor logs and DAG run histories.

Schema changes are also surfaced in the event stream — every modification logged with timestamp and before/after state. When a downstream consumer breaks, you can answer "when did the schema change?" in one lookup instead of a Slack archaeology project.

Cross-engine telemetry — see how every engine uses your tables

Iceberg tables are multi-engine by design. Trino, Spark, Snowflake, Athena, DuckDB, and Flink can all read and write the same tables. The problem for debugging: each engine has its own query UI, its own metrics, and its own view of table health. Correlating a slow Trino dashboard with manifest bloat in an Iceberg table requires stitching three different systems together.

LakeOps ingests query telemetry from every connected engine and provides:

• Field-access analysis — which columns appear in SELECT, FILTER, and JOIN clauses, and how often. This drives query-aware sort optimization during compaction.
• Per-engine query volume and latency trends — see whether Trino queries are slowing while Spark queries are fine on the same table, pointing to engine-specific issues versus table-structure problems.
• Hot tables and cold tables — prioritize maintenance where it matters. A table queried 10,000 times per day needs tighter file health than one queried once a week.

This telemetry loop closes the gap between "the table is structurally healthy" and "queries on this table are actually fast." It also feeds back into compaction: LakeOps analyzes which columns your queries filter, join, and group on, then organizes data files accordingly during sort compaction. The result is predicate pushdown and file pruning tuned to real workloads, not static guesses.

Route 2: manual debugging with metadata queries

Not every team has a control plane on day one. Many teams run a handful of Iceberg tables, manage maintenance with Airflow DAGs or cron jobs, and debug by querying metadata tables directly. This section is a complete manual debugging playbook — the same methodology that works whether you have a control plane or not.

The debugging mental model

When an Iceberg table fails, engineers often start in the wrong place: rewriting the SQL, scaling the Trino cluster, or blaming Spark. Start by asking one question:

Is this an engine problem, a catalog problem, or a table-structure problem?

Engine problems are local. One engine is slow; the others are fine. The explain plan shows a bad join order or a missing predicate pushdown. Fix the query or the engine config.

Catalog problems are coordination failures. Commits hang, metadata pointers disagree, REST timeouts leave ambiguous state, or Glue rate limits stall writers. You debug the catalog path and network, not Parquet.

Table-structure problems are the majority of "Iceberg is broken" pages. Every engine is slow. Planning dominates. File counts explode. Snapshots stack up. Delete files multiply. Storage bills diverge from logical data size. The query is fine. The physical layout and metadata tree are not.

Iceberg's metadata hierarchy is the debugger's map:

• metadata.json — current table pointer and schema
• snapshots — versions of the table over time
• manifest lists — index of manifests for a snapshot
• manifests — file entries with column stats for pruning
• data / delete files — the actual Parquet (or ORC) objects on S3/GCS/ADLS

Every symptom maps to a layer in that tree. Slow scans usually mean too many data files. Slow planning usually means too many manifests or snapshots. Inflated bills usually mean orphans pinned by stale snapshots. Commit storms usually mean two writers validating against the same partition. Once you know which layer is sick, the diagnosis queries are short. In a control plane like LakeOps, health classification does this layer mapping for you — Critical or Warning on a table tells you which layer to investigate, and the Insights tab tells you exactly which signal triggered it.

Triage in five minutes

Use this decision path before you open Spark UI.

1. Scope the blast radius. Is one table affected, one namespace, or the whole lake? One table → structure or writer config. Whole lake → catalog, storage, or a broken maintenance job that touched everything.

2. Check whether all engines agree. If Trino, Spark, and Athena all degrade on the same table, it is not a Trino bug. It is the table. (LakeOps' cross-engine telemetry shows per-engine latency on the same table side by side — saving you the time of testing in each engine manually.)

3. Classify the symptom:

• Latency / timeouts on reads → file count, delete files, or sort staleness
• Long hang before first row → manifests / snapshots (planning)
• Intermittent write failures → commit conflicts
• Bill spike with flat logical size → orphans + snapshot retention
• Maintenance jobs dying → sort compaction OOM on huge partitions
• Time travel / rollback errors → aggressive snapshot expiration
• Downstream "column not found" → schema evolution without coordination

4. Pull four numbers from metadata (queries below): file count and average size per partition, snapshot count, manifest count, delete-to-data ratio. Those four numbers solve most Iceberg pages.

5. Fix in the safe sequence. Expire → orphans → compact → rewrite manifests. Never invert it under pressure. Wrong order wastes compute or risks data loss.

The metadata toolkit

Iceberg exposes diagnosis as SQL against metadata tables. You do not need custom scanners for the first pass. Start here on every incident.

File health

SELECT
  partition,
  COUNT(*) AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 1) AS avg_size_mb,
  SUM(CASE WHEN file_size_in_bytes < 33554432 THEN 1 ELSE 0 END) AS files_under_32mb
FROM catalog.db.affected_table.files
GROUP BY partition
ORDER BY file_count DESC
LIMIT 20;

Healthy partitions: under ~100 files, 256–512 MB average. Warning: 500+ files or averages below ~32 MB. Critical: 1,000+ tiny files. See the small files guide for why file count dominates latency more than data volume.

Snapshot depth

SELECT
  COUNT(*) AS snapshot_count,
  MIN(committed_at) AS oldest_snapshot,
  MAX(committed_at) AS latest_snapshot
FROM catalog.db.affected_table.snapshots;

Streaming tables with 5-minute commits create ~288 snapshots/day. Above ~1,000–2,000 retained snapshots, planning and metadata IO degrade. This is the core of how Iceberg query planning works — and why metadata at scale becomes its own discipline.

Manifest fragmentation

SELECT
  COUNT(*) AS manifest_count,
  ROUND(AVG(length) / 1024, 1) AS avg_manifest_size_kb,
  SUM(added_data_files_count + existing_data_files_count) AS total_file_entries
FROM catalog.db.affected_table.manifests;

A healthy streaming table after regular rewriting: under ~100 manifests. Thousands of tiny manifests is a planning bottleneck, not a data-volume problem.

Recent write vs maintenance activity

SELECT
  committed_at,
  operation,
  summary['added-data-files'] AS files_added,
  summary['deleted-data-files'] AS files_deleted,
  summary['changed-partition-count'] AS partitions_affected
FROM catalog.db.affected_table.snapshots
ORDER BY committed_at DESC
LIMIT 50;

append without intervening replace (compaction) is the smoking gun for small-file explosions. Overlapping append and replace timestamps on the same partitions explain commit conflicts.

Delete file pressure (MOR / CDC)

WITH data AS (
  SELECT partition, COUNT(*) AS data_files
  FROM catalog.db.affected_table.files
  GROUP BY partition
),
deletes AS (
  SELECT partition, COUNT(*) AS delete_files, SUM(record_count) AS delete_records
  FROM catalog.db.affected_table.all_delete_files
  GROUP BY partition
)
SELECT
  d.partition,
  d.data_files,
  COALESCE(del.delete_files, 0) AS delete_files,
  ROUND(COALESCE(del.delete_files, 0) * 100.0 / d.data_files, 1) AS delete_ratio_pct
FROM data d
LEFT JOIN deletes del ON d.partition = del.partition
WHERE COALESCE(del.delete_files, 0) > 0
ORDER BY delete_ratio_pct DESC;

Above ~10% delete-to-data file ratio, reads pay real reconciliation tax. Above ~50%, users feel it. Details in the delete files guide. LakeOps tracks this ratio per partition continuously and triggers compaction when the threshold is crossed.

Debug: queries suddenly slow

Symptom. Queries that ran in 2–5 seconds now take 30–60+. No schema change. No obvious data-volume spike. Dashboards time out. Every engine is affected.

Root cause. Small file explosion. Streaming writers — Flink, Spark Structured Streaming, Kafka Connect Iceberg sink, CDC — produce files sized by checkpoint throughput, not by optimal read size. Each query pays an object-store GET, footer parse, and task setup per file. 200,000 files at 3 MB each is a different world from 2,000 files at 300 MB with the same bytes.

Confirm. Run the file health query. Look for partitions with 1,000+ files averaging under 32 MB. Cross-check snapshots for days of append without replace.

Immediate fix. Emergency binpack on the worst recent partitions:

CALL catalog.system.rewrite_data_files(
  table => 'db.affected_table',
  strategy => 'binpack',
  where => 'event_date >= current_date() - INTERVAL 7 DAYS',
  options => map(
    'target-file-size-bytes', '268435456',
    'min-input-files', '3',
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '20',
    'max-concurrent-file-group-rewrites', '15'
  )
);
CALL catalog.system.rewrite_manifests(
  table => 'db.affected_table'
);

Always enable partial-progress.enabled so a mid-job failure does not throw away hours of rewrite work.

Prevention. Compaction cadence must match write cadence. Streaming tables often need compaction every 1–4 hours, not nightly. Writer-side defenses matter too: 3–5 minute checkpoints, write.distribution-mode=hash, and a target file size of 256–512 MB. For the full degradation timeline, see streaming compaction.

How a control plane helps. LakeOps watches file count and average size per partition continuously. When thresholds trip, compaction fires from table state — not from a cron. The Rust/DataFusion engine makes frequent passes viable at ~$5/TB vs ~$50/TB on Spark, so streaming tables stay healthy instead of oscillating between midnight cleanups.

Debug: planning takes forever

Symptom. Queries hang 30 seconds to minutes before returning any rows. EXPLAIN itself is slow. Every query against the table suffers.

Root cause. Manifest bloat plus deep snapshot retention. Every commit adds at least one manifest. A streaming table at 5-minute commits accumulates thousands of manifests per month. This is why metadata lifecycle maintenance is not optional at scale.

Confirm. Manifest count over ~500, or snapshot count over ~2,000.

Immediate fix. Expire first, then rewrite manifests:

CALL catalog.system.expire_snapshots(
  table => 'db.affected_table',
  older_than => TIMESTAMP '2026-07-12 00:00:00',
  retain_last => 50
);
CALL catalog.system.rewrite_manifests(
  table => 'db.affected_table'
);

On extreme fragmentation (2,000+ manifests), rewriting alone can drop planning from 30+ seconds to under a second. Do not compact first hoping planning will improve — you will rewrite data while leaving the metadata tree fragmented.

Prevention. Rewrite manifests after every compaction cycle. Expire snapshots with a retention window that matches your rollback SLA. Enable metadata file cleanup:

ALTER TABLE db.affected_table SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled' = 'true',
  'write.metadata.previous-versions-max' = '100'
);

How a control plane helps. LakeOps enforces the sequence expire → orphans → compact → rewrite manifests as a pipeline. Manifests cannot silently re-fragment after a successful compaction pass that nobody followed up.

Debug: CommitFailedException storms

Symptom. Spark or Flink jobs fail with CommitFailedException: Cannot commit changes based on stale table metadata. Retries sometimes succeed. Compaction and streaming writers appear to fight each other.

Root cause. Optimistic concurrency. A commit validates against the table state when the operation started. If another writer committed in between, the commit is rejected. The dedicated commit conflicts guide walks through the concurrency model in depth.

Split conflicts while you debug:

• Catalog / metadata conflicts (retriable). Two writers append different partitions; they collide on the metadata pointer. Iceberg refreshes, rebases, and retries.
• Data conflicts (non-retriable). Compaction assumes source files still exist; another operation removed them. The job must restart entirely. This is the expensive class.

Confirm. Snapshot history showing append and replace overlapping on the same partitions within seconds.

Immediate fix.

ALTER TABLE db.affected_table SET TBLPROPERTIES (
  'commit.retry.num-retries' = '10',
  'commit.retry.min-wait-ms' = '200',
  'commit.retry.max-wait-ms' = '30000',
  'commit.retry.total-timeout-ms' = '600000'
);
CALL catalog.system.rewrite_data_files(
  table => 'db.affected_table',
  strategy => 'binpack',
  where => 'event_date < current_date()',
  options => map(
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '10'
  )
);

Prevention. Never compact the active write partition. Use hash write distribution. Keep partial-progress.enabled=true. Sequence maintenance operations — expiration concurrent with compaction is a classic way to turn a retriable race into a full rewrite.

How a control plane helps. LakeOps compaction is conflict-aware by design — it inspects active writer state, targets cold partitions, and retries only the affected group on the next cycle.

Debug: storage growing faster than data

Symptom. Object storage bills climb 30–50% month over month while logical table size is flat. On mature lakes, orphans routinely account for 25–40% of billable storage on affected prefixes.

Root cause. Orphan files from failed writes and snapshot retention keeping superseded data files referenced. See safe orphan cleanup.

Confirm.

SELECT ROUND(SUM(file_size_in_bytes) / 1073741824, 2) AS logical_data_gb
FROM catalog.db.affected_table.files;
CALL catalog.system.remove_orphan_files(
  table => 'db.affected_table',
  older_than => TIMESTAMP '2026-07-05 00:00:00',
  dry_run => true
);

Always dry-run first. Always verify URI schemes (s3:// vs s3a:// vs s3n://) match between metadata and listings — a scheme mismatch can mark an entire table's data as orphaned.

Immediate fix.

CALL catalog.system.expire_snapshots(
  table => 'db.affected_table',
  older_than => TIMESTAMP '2026-07-12 00:00:00',
  retain_last => 100
);
CALL catalog.system.remove_orphan_files(
  table => 'db.affected_table',
  older_than => TIMESTAMP '2026-07-05 00:00:00'
);

Hard safety rule. The orphan older_than threshold must be far enough in the past to protect in-flight writes — Iceberg defaults to 3 days; production teams use 7+ days. Deleting "orphans" that belong to an uncommitted writer corrupts the table silently.

How a control plane helps. LakeOps runs orphan cleanup after expiration in the coordinated pipeline and enforces the safety window by default. The audit trail logs bytes reclaimed per operation — storage debt cannot silently compound for months.

Debug: compaction OOMs

Symptom. Spark compaction dies with OutOfMemoryError. Raising executor memory delays the failure. One giant partition fails every time.

Root cause. Sort (or Z-order) compaction on a massive partition. Global sort needs memory for the whole file group.

Confirm.

SELECT
  partition,
  COUNT(*) AS file_count,
  ROUND(SUM(file_size_in_bytes) / 1073741824, 2) AS partition_size_gb
FROM catalog.db.affected_table.files
GROUP BY partition
ORDER BY partition_size_gb DESC
LIMIT 10;

Immediate fix. Binpack with bounded file groups:

CALL catalog.system.rewrite_data_files(
  table => 'db.affected_table',
  strategy => 'binpack',
  where => 'partition_date = DATE ''2026-07-01''',
  options => map(
    'target-file-size-bytes', '268435456',
    'min-input-files', '5',
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '50',
    'max-file-group-size-bytes', '10737418240'
  )
);

Stabilize with binpack first. If you still need sort layout, run a second bounded sort pass. Strategy selection is covered in the compaction strategies guide.

How a control plane helps. LakeOps' Rust compaction engine uses streaming sort with bounded memory and no JVM. Partitions that OOM Spark complete in minutes without heap tuning.

Debug: time travel and rollback failures

Symptom. FOR SYSTEM_TIME AS OF fails with missing snapshot errors. Occasionally a long-running query fails mid-flight because its snapshot disappeared.

Root cause. Snapshot expiration configured tighter than the rollback / query SLA.

Confirm.

SELECT snapshot_id, committed_at, operation
FROM catalog.db.affected_table.snapshots
ORDER BY committed_at ASC
LIMIT 10;
SHOW TBLPROPERTIES db.affected_table;

Immediate fix. Protect the future (expired snapshots cannot be resurrected):

ALTER TABLE db.affected_table SET TBLPROPERTIES (
  'history.expire.max-snapshot-age-ms' = '604800000',
  'history.expire.min-snapshots-to-keep' = '100'
);
ALTER TABLE db.affected_table CREATE TAG `incident_baseline_2026_07_19`
  AS OF VERSION 847291035
  RETAIN 365 DAYS;

Prevention. Retention must exceed your longest-running query and your incident detection SLA. Tag compliance checkpoints. Document per-table windows. More in the retention policy guide.

How a control plane helps. LakeOps supports configurable retention policies per table, namespace, or catalog. Tagged snapshots are excluded from expiration regardless of age. The observability dashboard surfaces the current retention window for every table, flagging where configuration is too aggressive.

Debug: merge-on-read read amplification

Symptom. UPDATE/DELETE-heavy tables slow over days even when row counts are stable. A query that touched 10 files now reconciles against hundreds of delete files.

Root cause. Merge-on-read writes delete files instead of rewriting data. Without compaction that physically applies deletes, overhead grows linearly. See data quality and table health.

Immediate fix.

CALL catalog.system.rewrite_data_files(
  table => 'db.affected_table',
  strategy => 'binpack',
  where => 'partition_date >= current_date() - INTERVAL 14 DAYS',
  options => map(
    'delete-file-threshold', '1',
    'target-file-size-bytes', '268435456',
    'partial-progress.enabled', 'true',
    'remove-dangling-deletes', 'true'
  )
);

Prevention. Match compaction frequency to mutation rate. Iceberg V3 deletion vectors reduce reconciliation cost but do not remove the need to compact away dead rows.

How a control plane helps. LakeOps tracks delete-file ratios per partition continuously. When the threshold is crossed, compaction fires automatically and physically applies pending deletes. The Insights system surfaces partitions with rising delete ratios at WARNING before they reach CRITICAL.

Debug: schema changes break consumers

Symptom. Downstream Trino queries, dbt models, or BI tools fail after a schema change: missing columns, type mismatches, unexpected nulls.

Root cause. Schema evolution applied directly on main without consumer validation. Additive nullable columns are usually safe; destructive changes are not.

Immediate fix. Rollback if possible, then re-apply via Write-Audit-Publish on a branch:

CALL catalog.system.rollback_to_snapshot(
  table => 'db.affected_table',
  snapshot_id => 847291034
);
ALTER TABLE db.affected_table CREATE BRANCH schema_test_v2 RETAIN 7 DAYS;
CALL catalog.system.fast_forward(
  table => 'db.affected_table',
  branch => 'main',
  to => 'schema_test_v2'
);

Prevention. Treat public columns as an API. Prefer additive evolution. Deeper patterns in the schema evolution guide. LakeOps logs every schema change event with timestamp and before/after state in the table event stream.

Debug: wrong results

Latency pages are loud. Wrong results are quiet — and worse. Treat discrepancies across engines as a data-integrity incident.

Root causes in the wild.

• Equality-delete caching bugs. Executor caches reuse delete records across queries with different projections. Deletes get skipped. Diagnostic: spark.sql.iceberg.executor-cache.delete-files.enabled=false. Upgrade to builds that canonicalize delete schemas by field ID.
• Dangling deletes after rewrite. Use remove-dangling-deletes in rewrite options. Consider rewrite_position_delete_files.
• COW + equality delete filter interactions. COW paths that prune equality delete files based on query filters can resurrect deleted rows.
• Reading the wrong branch / snapshot. Confirm current_snapshot_id in both engines before comparing results.

Debug checklist. Pin both engines to the same snapshot ID. Inspect delete files vs data files for affected partitions. Reproduce with narrow and wide projections. Stop maintenance until you understand the state. Roll back if a pre-incident snapshot remains.

Debug: catalog and commit-state unknowns

Symptom. REST 502/503/504, CommitStateUnknownException during commits. Table appears corrupted: metadata points at manifests the client already cleaned up.

Root cause. Ambiguous commit outcomes. The catalog persisted the new metadata while the client saw a timeout and attempted cleanup. Treating 503 as a clean failure is unsafe when a proxy sits between Spark and a REST catalog.

Debug steps.

• Do not manually delete metadata or data files because a client said the commit failed.
• Check the catalog's current metadata pointer vs object storage. Reconcile before cleanup.
• Treat CommitStateUnknownException as "stop and verify."
• Upgrade Iceberg/engine versions that include safer handling of 503s.

Catalog comparisons (Glue, Polaris, Nessie, Unity, Gravitino, Lakekeeper) are in LakeOps' catalog guide.

Debug the writers, not just the table

Many Iceberg incidents are writer-configuration debt. Before scheduling more compaction, ask whether the pipeline creates structural debt faster than maintenance can repay.

Checkpoint / commit interval. Flink at 1-minute checkpoints manufactures file storms. Move to 3–5 minutes.

Write distribution mode. Set write.distribution-mode=hash for partitioned streaming tables.

ALTER TABLE catalog.db.events SET TBLPROPERTIES (
  'write.distribution-mode' = 'hash',
  'write.target-file-size-bytes' = '268435456'
);

Partition grain. Over-partitioning is a leading cause of small files. Prefer days(ts). Use bucket(N, col) instead of identity transforms on high-cardinality columns. Decision frameworks in the partitioning best practices and partitioning strategies guides.

Delete mode vs mutation rate. MOR is right for high-churn CDC — wrong if you never compact. COW is right when read latency is sacred. Mixing modes without a compaction policy is how delete-ratio pages start.

Fixing writers halves the maintenance burden. Skipping this step turns your control plane into a very expensive mop.

The safe fix sequence

Under pressure, teams compact first because "small files" is the familiar villain. That is often the wrong first move.

The correct maintenance sequence — consistent across the official docs, LakeOps' table health guide, and production incident work — is:

1. Expire snapshots — release metadata references to files nobody needs for time travel
2. Remove orphan files — delete unreferenced objects from storage (after expiration, with 7+ day safety window)
3. Compact data files — merge small files / apply deletes
4. Rewrite manifests — consolidate the metadata index against the final file set
5. Refresh statistics (optional) — so planners see current NDVs and sketches

Why the order matters:

• Compacting before expiration rewrites files that expiration is about to discard — wasted compute.
• Orphan cleanup before expiration misses the largest reclaimable set still pinned by old snapshots.
• Manifest rewrite before compaction produces an index stale minutes later.
• Orphan cleanup with a too-short window deletes in-flight writer files — silent corruption.

Memorize the sequence. Put it in the runbook. Or use a control plane that encodes it for you — LakeOps enforces this sequence as a coordinated pipeline on every maintenance cycle, so humans cannot invert it at 3 AM.

Wrapping up: Choosing your debugging strategy

Both routes work. The question is scale and cost of attention.

Route 2 (manual) fits when: you have fewer than ~50 tables, one or two engines, batch-only writes, and a platform engineer who owns maintenance. The metadata queries in this guide are your on-call playbook. Run them hourly for streaming tables, daily for batch. Set alert thresholds. Document per-table retention. It works.

Route 1 (control plane) fits when: table count crosses 50, you run streaming or CDC pipelines, multiple engines hit the same tables, or engineering time spent babysitting maintenance scripts exceeds the cost of a purpose-built system. The transition is incremental:

1. Connect the catalog — discovery and health classification begin immediately
2. Inspect table health — review which tables are Healthy, Warning, or Critical
3. Run maintenance manually on the worst tables — learn the before/after metrics in the Events tab
4. Enable scheduled policies per namespace — every table in scope inherits automatically
5. Turn on adaptive maintenance for streaming and CDC tables — fully autonomous
6. Keep this debugging playbook for the residual P1s that are not structural

Teams running the control-plane model report order-of-magnitude fewer Iceberg pages — not because the runbook got faster, but because file explosions, manifest storms, and orphan cliffs are resolved while still at Warning. The real end state of debugging well: you still know how to diagnose, but you rarely need to.

For production readiness beyond debugging, see the Iceberg production readiness checklist and the managed Iceberg operating model.

Quick reference cheatsheet

Queries 10x slower → file count / avg size per partition → binpack worst partitions → compact on write cadence; fix writer distribution

Planning takes minutes → manifest + snapshot counts → expire snapshots, then rewrite manifests → rewrite manifests after every compaction

CommitFailedException → overlapping append/replace on same partitions → raise retries; compact only cold partitions → hash distribution; partial progress

Storage >> logical data → logical size vs dry-run orphans → expire, then remove orphans (7-day safety, scheme check) → weekly/daily cleanup after expiration

Compaction OOM → huge partition under sort → binpack with max-file-group-size-bytes → bounded groups or Rust engine

Time travel broken → oldest snapshot vs need → raise retention / tag checkpoints → retention ≥ detection SLA and longest query

MOR getting slower → delete ratio per partition → compact with delete-file-threshold=1 → compaction frequency = mutation rate

Schema broke consumers → metadata log → rollback; re-apply via branch WAP → additive-first contract

Wrong results → pin same snapshot across engines; inspect delete files → stop maintenance; rollback if needed → validate on a branch

Ambiguous commit / REST timeout → catalog pointer vs storage → do not cleanup blindly; verify state → upgrade Iceberg client

Debt returning every day → writer config (checkpoint, distribution, partition grain) → fix writers, then maintenance cadence → automation that triggers on table state

Debugging Iceberg is mostly debugging operations. The format already gave you ACID and time travel. Your job in an incident is to read the metadata tree honestly, fix the sick layer, respect the sequence, and leave the system harder to break than you found it. You can do that table by table with SQL queries and cron jobs — or you can let a control plane like LakeOps do the reading, the sequencing, and the fixing continuously, and step in only when something genuinely novel happens.

For the incident-shaped version of this material, keep the operational runbook bookmarked. For the automation path, start with automating table maintenance. For table-health theory and metrics, see the complete health guide. For the platform that closes the loop, explore lakeops.dev.

Thanks for reading! 🙏 🍺

Jonathan