Introduction
Storing data in object storage is easy. Managing that data like a reliable table is the hard part.
That is the gap Apache Iceberg fills. Instead of treating a data lake as a loose collection of Parquet files and folder conventions, Iceberg adds a real table layer on top of files in S3, ADLS, GCS, or HDFS. The result is a format that gives analytics teams a much more database-like experience: consistent reads, safer writes, schema evolution, partition evolution, and time travel.
If you have ever dealt with Hive-style partitions, brittle metadata, slow table planning, or unsafe concurrent writes, Apache Iceberg is one of the main reasons the modern lakehouse feels more stable and much easier to operate.
What Apache Iceberg Actually Is
Apache Iceberg is an open table format for huge analytic datasets.
That wording matters.
Iceberg is not:
- a query engine
- a data warehouse
- a storage system
- a file format like Parquet
Iceberg sits between your storage layer and your compute engine. It tells engines how to interpret a collection of files as a table, how to track versions of that table, and how to commit changes safely.
A simple way to think about it is this:
| Layer | Example |
|---|---|
| Storage | S3, ADLS, GCS, HDFS |
| Data files | Parquet, ORC, Avro |
| Table format | Apache Iceberg |
| Compute engines | Spark, Trino, Flink, Presto, Hive, Impala |
| Catalog | Hive, REST, Glue, JDBC, Nessie |
This separation is one of Iceberg’s biggest strengths. You can keep data in open file formats while allowing multiple engines to work with the same logical table.
Why Plain Data Lake Files Become Painful
Before formats like Iceberg, many data lakes depended on folder layout and metastore entries to behave like tables.
That approach creates a few predictable problems:
- partition logic leaks into user queries
- renaming or evolving schemas becomes risky
- concurrent writes are difficult to make safe
- scanning large tables becomes expensive when metadata grows out of control
- rolling back bad writes is painful
- engines often disagree on what the table actually looks like
Iceberg was designed to solve exactly these operational and correctness problems.
How Iceberg Works at a High Level
At the physical level, an Iceberg table still stores data in files such as Parquet. What changes is the metadata model.
Iceberg tracks a table through:
- data files that hold records
- manifest files that describe groups of data files
- metadata files that describe the current state of the table
- snapshots that represent committed versions of the table
- catalogs that let engines find and load the table by name
This metadata-first design is why Iceberg can plan scans quickly and support features like time travel, rollback, and safe concurrent updates.
The Features That Make Iceberg Important
1. Schema Evolution Without Surprises
Schema changes are common in analytics systems, but historically they have been risky.
Iceberg was designed so schema evolution can happen without rewriting all data files and without dangerous side effects. You can add, drop, update, and rename columns while preserving correctness.
That matters because schema evolution in a raw file-based lake can easily create subtle bugs, especially when different engines read the same data differently.
2. Hidden Partitioning
Traditional Hive-style tables often force users to understand physical partition columns and include them correctly in queries. That is both inconvenient and error-prone.
Iceberg introduces hidden partitioning. Partition values are derived automatically, and query performance no longer depends on analysts remembering physical layout details. That makes partitioning much easier to change later.
For example, a table may be partitioned by day from a timestamp column, but analysts do not need to manually manage or query an extra event_date column.
3. Partition Evolution
Real systems change over time. A partition strategy that works at 10 GB often does not work at 10 TB.
Iceberg allows you to evolve partition specs as data volume and access patterns change. That means you can fix a poor partitioning decision without rebuilding the entire table from scratch.
4. Time Travel and Rollback
Every write creates a new snapshot of the table.
That gives you two very useful capabilities:
- time travel, where you query the table as it existed at a previous timestamp or snapshot
- rollback, where you reset the table to a known good state after a bad write or bad deployment
For debugging, auditing, reproducibility, and incident recovery, this is a major operational win.
5. Concurrent Writes and Serializable Isolation
One of the hardest parts of using object storage as a data platform is write correctness.
Iceberg addresses this with atomic commits and optimistic concurrency. Multiple writers can operate on the same table, and if a conflict happens, failed writers retry against the new current state instead of silently corrupting the table.
This is one of the key reasons Iceberg feels far more reliable than ad hoc data lake patterns.
6. Multi-Engine Interoperability
Iceberg’s goal is not to lock you into one engine. Its goal is to let multiple engines use the same table definition safely.
That is why Iceberg integrates with engines such as Spark, Trino, Flink, PrestoDB, Hive, and Impala. You keep the table format and metadata model stable while choosing the compute engine that fits the workload.
7. Row-Level Deletes and Modern Table Capabilities
Iceberg’s format versions have expanded what the table format can represent. Version 2 introduced row-level deletes, which made it much more practical for modern incremental pipelines and mutation-heavy analytical workloads. Later format versions extend data types and metadata capabilities even further.
A Small Spark Example
Spark is currently the most feature-rich engine for Iceberg operations, so it is a good place to start.
Configure a local catalog
spark-sql \
--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.1 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.local.type=hadoop \
--conf spark.sql.catalog.local.warehouse=$PWD/warehouse
Create a table
CREATE TABLE local.db.events (
event_id BIGINT,
user_id BIGINT,
event_type STRING,
event_ts TIMESTAMP
)
USING iceberg
PARTITIONED BY (days(event_ts));
Insert and query data
INSERT INTO local.db.events VALUES
(1, 1001, 'login', TIMESTAMP '2026-04-01 09:00:00'),
(2, 1001, 'purchase', TIMESTAMP '2026-04-01 09:05:00');
SELECT event_type, COUNT(*)
FROM local.db.events
GROUP BY event_type;
Inspect snapshots
SELECT * FROM local.db.events.snapshots;
Query an earlier version
SELECT *
FROM local.db.events
TIMESTAMP AS OF '2026-04-01 09:01:00';
That last query is where Iceberg starts to feel very different from a plain file-based lake.
Catalogs Matter More Than Many Teams Expect
Iceberg tables are typically discovered through a catalog. In Spark, official catalog types include:
hivehadooprestgluejdbcnessie
This is an important design point. Iceberg is not only about files and snapshots. It also needs a consistent way for engines to locate tables and commit metadata changes.
For simple experiments, a Hadoop catalog may be enough. In larger organizations, REST, Glue, JDBC, or Nessie-based setups often make more sense depending on governance and platform standards.
Iceberg Maintenance Is Real Maintenance
Iceberg gives you a lot, but it is not maintenance-free.
Because each write creates a snapshot and new metadata, you need to manage table history deliberately. Over time, tables may require:
- snapshot expiration
- old metadata cleanup
- orphan file deletion
- compaction or file rewrite strategies
This is not a weakness. It is the tradeoff for having rich metadata and strong table semantics on top of object storage. The teams that succeed with Iceberg are the ones that treat table maintenance as a normal platform responsibility.
When Iceberg Is a Great Fit
Iceberg is usually a strong choice when you need one or more of these:
- very large analytical tables
- multiple compute engines over the same data
- reliable schema evolution
- safe concurrent writes
- time travel and rollback
- mutation support such as deletes, merges, or incremental updates
- open storage formats without warehouse lock-in
When Iceberg Might Be More Than You Need
Iceberg is not mandatory for every dataset.
If you have a tiny workload, a single engine, and append-only files with minimal governance needs, plain Parquet in object storage may still be enough.
Iceberg starts to pay off when your platform complexity grows and you need table-level guarantees rather than file-level conventions.
Final Thoughts
Apache Iceberg has become one of the foundational technologies behind the modern lakehouse because it solves the part that raw files never solved well: treating distributed files as a dependable table.
Its strongest ideas are simple but powerful:
- tables should support safe evolution
- partitioning should not leak into user experience
- metadata should make planning fast at large scale
- concurrent writes should be safe
- history should be queryable
- storage and compute should stay decoupled
That combination is why Iceberg is not just another storage format. It is a table abstraction designed for real-world analytical systems.
If your team is building a platform on top of object storage and wants database-like behavior without giving up open formats, Iceberg is one of the most important technologies to understand.
Sources
- Apache Iceberg Home - iceberg.apache.org/
- Apache Iceberg Documentation - iceberg.apache.org/docs/latest/
- Apache Iceberg Table Specification - iceberg.apache.org/spec/
- Apache Iceberg Spark Getting Started - iceberg.apache.org/docs/latest/spark-getting-started/
- Apache Iceberg Spark Queries - iceberg.apache.org/docs/latest/spark-queries/
- Apache Iceberg Branching and Tagging - iceberg.apache.org/docs/latest/branching/
- Apache Iceberg Maintenance - iceberg.apache.org/docs/latest/maintenance/
- Apache Iceberg Releases - iceberg.apache.org/releases/