Every morning, a question sails into a busy port: "Show me last month's sales."

It looks simple. It is not. To answer it, the question has to pass through a warehouse, a logbook, a set of cranes, and a lighthouse — four very different workers who, together, make the whole port run.

None

The Warehouse

The question's first stop is the warehouse — this is Parquet.

Most warehouses stack goods by the order they arrived: box after box, row after row. If you want "all the blue widgets," you walk past everything to find them. Parquet's warehouse works differently. It stores goods by type, in labeled aisles — all the dates in one aisle, all the prices in another, all the customer names in a third. Ask for "the total sales amount," and the query only walks down one aisle instead of the whole building. That's what makes it fast, and why it compresses so well — similar things sitting next to each other squeeze down tighter than a jumbled pile ever could.

None

The Logbook

But a warehouse alone doesn't know its own history. What was in aisle 3 last Tuesday? Did someone add a new column of goods last month? For that, the port relies on its harbor master's logbook — Iceberg.

Iceberg doesn't move anything itself. It just keeps an exact, timestamped record: which files belong to the table right now, what the layout looked like an hour ago, a day ago, before that schema change. Because of this ledger, you can ask the port to show you last Tuesday's warehouse exactly as it stood — a kind of time travel — and the port can safely reorganize its aisles without ever losing track of what's true.

None

The Cranes

Now the goods need to move — from the warehouse onto a ship, or between two ships docked side by side. This is where Arrow works.

Arrow is the port's standardized container and crane system. Every ship, no matter whose fleet it belongs to, uses the same container shape. Nothing needs to be unpacked and repacked at each handoff. A crate slides straight from warehouse to ship, or ship to ship, in one motion. That's Arrow's whole purpose: a shared in-memory format so different tools can pass data to each other instantly, without translation delays.

None

The Lighthouse

One problem remains. This port isn't served by a single fleet — dozens of different ships, from different companies, all need to sail in and query the same warehouse. How do they agree on whose logbook is the real one?

That's Polaris, the lighthouse. It doesn't store cargo or write history — it simply tells every ship, regardless of fleet, exactly where the true table lives and which logbook to trust. With Polaris as the shared reference point, a dozen different engines can all navigate to the same data without stepping on each other or drifting into disagreement.

None

Coming Together

So the morning's question finally gets its answer: Polaris points the query to the right table. Iceberg's logbook confirms which files are current. Parquet's warehouse lets the query scan only the aisles it needs. Arrow's cranes carry the results out, container by container, with no wasted motion. Four very different jobs, one smooth trip through the port.

Technical recap

For readers who want it without the metaphor:

  • Apache Arrow — an in-memory columnar data format enabling zero-copy, cross-language data exchange between systems (e.g., Spark, Pandas, DuckDB) without serialization overhead.
  • Apache Parquet — a columnar file format for storage on disk, optimized for compression and fast analytical (column-scan) queries.
  • Apache Iceberg — a table format layered on top of files like Parquet, providing ACID transactions, schema evolution, partition evolution, and time travel over large analytic datasets.
  • Apache Polaris — an open-source catalog implementing the Iceberg REST catalog spec, giving multiple engines a shared, vendor-neutral source of truth for table metadata and governance.

Together: Arrow moves data fast between tools, Parquet stores it efficiently, Iceberg tracks it reliably over time, and Polaris lets everyone agree on where the truth lives.

References

Further Readings

How I Built a Java Physics Simulation That Publishes Real-Time Kafka Events

Practical guide: stream real-time motion and collision events from a JavaFX + dyn4j physics simulation to Kafka using…

itnext.io

From Physics Events to Real-Time Analytics with Apache Flink

Build a JavaFX and dyn4j physics simulation that streams real-time position and collision events to Analytics Apache…

itnext.io

📣 Call to Action

If you are interested in following along with my journey, I invite you to dive into all the details provided below:

Wagner Franchin - Medium

Read writing from Wagner Franchin on Medium. Software Engineer writing about tech, coding, career and more.

medium.com

Thanks for reading