The lake format

A Pondra lake is one folder: a directory on a disk, or a prefix in a bucket. This page says what is in it and who may read or write it, so you can back it up, copy it, or point another tool at it.

Disk and bucket are the same

A lake on a local directory (/data/lake) and a lake in a bucket (s3://my-bucket/lake) have exactly the same layout. The object keys in the bucket are the same relative paths as the files on disk, and the catalog stores only relative paths. So you can copy a lake from disk to a bucket, or back, with any sync tool while its nodes are stopped, and it opens there as it was.

aws s3 sync ./lake s3://my-bucket/lake
pondra serve s3://my-bucket/lake

It works wherever the binary runs: Linux, macOS or Windows, on a local or network directory, S3, Cloudflare R2, MinIO, Google Cloud Storage or Azure. A bucket needs conditional writes.

What is inside

lake/
├── catalog/              Pondra's catalog: a SlateDB key-value store
│   ├── manifest/         which files make up the store right now
│   ├── wal/              recent catalog commits
│   └── compacted/        the catalog's sorted tables
├── cluster/
│   ├── term/             leader election: one small object per term
│   └── alive/            "the leader of this term is still here"
├── inbox/                writes from machines that can't reach the leader
├── log/                  the log tail: bigger flushes, as Arrow IPC + ZSTD
├── files/                the lake's own files (PUT /files/<path>): notebooks, SQL, data, images
└── data/<table>/         one folder per table
    ├── <uuid>.parquet    the table's rows
    ├── _manifests/       the file list, once a table has more than 128 files
    ├── _delta_log/       only if the table publishes Delta
    └── metadata/         only if the table publishes Iceberg

A new lake has a catalog/, a cluster/ and, once you make a table, a data/. The rest appears when it is needed. Views and task outputs are tables too, so they have folders under data/.

Make a table that publishes Delta and Iceberg, write a row, and look:

CREATE TABLE orders (id BIGINT, item VARCHAR, amount DOUBLE) WITH (publish = 'delta,iceberg');
INSERT INTO orders VALUES (1, 'tea', 7.0), (2, 'cake', 4.5);
CHECKPOINT;

find lake/data -maxdepth 2 | sed -E 's/[0-9a-f]{8}-[0-9a-f-]{27}/<uuid>/' | sort

lake/data
lake/data/orders
lake/data/orders/<uuid>.parquet
lake/data/orders/_delta_log
lake/data/orders/metadata

Path	What it holds	Who reads it	How it changes
catalog/	Tables, log segments, producer progress, views, tasks, functions, procedures, sealed secrets, attached lakes	Pondra	New objects only. Old ones are compacted away
cluster/	Which node leads, and that it is alive	Pondra	One new object per election. alive is rewritten every 10 seconds
inbox/	Requests from writers that can’t reach the leader, and the leader’s answers	The leader	Each request is deleted once answered
log/	Rows not yet in Parquet: Arrow IPC with ZSTD	Pondra	Never edited. Deleted once tiered, and older than --retain-secs (or --changelog-secs, if longer)
data/<table>/*.parquet	The table’s rows. LZ4-compressed by default (PONDRA_CODEC=zstd for smaller files)	Anyone	Never edited. Replaced files are deleted after --retain-secs
data/<table>/data/	The table’s rows as another engine wrote them through the Iceberg catalog: its files are the table’s where they are, their rows’ ids and versions from the commit that recorded them	Anyone	As the table’s own files
files/	Whatever was put there	Anyone	Replaced only by a PUT that names the version it replaces (If-Match); nothing else is. Each save is also kept under files/.versions/<path>/<ms>.<who>
data/<table>/_manifests/	The older files’ list, with each file’s column ranges	Pondra	Never edited
data/<table>/_delta_log/	Delta Lake commits and checkpoints	Delta readers	Appended to
data/<table>/metadata/	Iceberg metadata, manifest lists and manifests	Iceberg readers	Appended to. version-hint.text is rewritten

A node also keeps two things outside the lake. Delete either whenever you like.

• The SSD tier, for lakes on object storage only: copies of the objects that never change (all but files/, whose files are replaced in place), in a pondra-cache folder in the temp directory (or --cache-dir).
• Replica files, with --ack replicated: the changes the bucket doesn’t have yet. They are deleted as soon as it has them.

Who can read and write it

Anyone with the binary and access to the folder or bucket can use it. There are three ways in, and other tools get in by three more.

Reader	How it reads	Sees	Writes
A node: pondra serve LAKE (--reader to only read)	The catalog in memory, kept current by the leader’s stream of commits. Hot objects on the local SSD	Every committed write, milliseconds after the acknowledgement	Yes. --reader: no
pondra sql --dir … "…", no node	Opens the catalog in the bucket, reads the log tail and the Parquet files	Every write already in the bucket (in the default --ack durable, every acknowledged write)	CREATE TABLE, INSERT, UPDATE, DELETE: this machine does the work, the leader records it
Another engine, through Delta or Iceberg	The table’s _delta_log/ or metadata/, or a node’s Iceberg REST catalog	The table as of the last tiering round	Appends to an append table, through the REST catalog
Kafka clients, on a node’s --kafka port	A topic per table	Every committed write	Yes
Arrow Flight and ADBC clients, on --flight	Flight SQL, or a table’s log as a stream	Every committed write	Yes
Plain Parquet readers	data/<table>/*.parquet	Every file, including replaced ones in their grace period and, in a keyed table, old versions of a key	Never

A plain Parquet reader can count rows twice: it sees files that a merge has just replaced and, in a keyed table, the old versions of a key. Only the catalog, the Delta log or the Iceberg metadata says which files are current. Use those.

A machine that can reach the bucket but not the leader (another network, another company) still writes: its request goes through inbox/ and the leader records it within a second or so. Several lakes can share one bucket. Each has its own prefix and its own leader, and one lake attaches the others as databases: sales.orders, or sales.eu.orders for schema eu. A write to sales.orders is recorded by the sales lake’s leader.

Where a row is, over time

A row is readable from the first step. It reaches other engines at the second.

1. Acknowledged. The row is in the log: in log/<ms>-<uuid>.seg, or inside the catalog commit for a small write. With the default --ack durable, the acknowledgement comes once the commit is in the bucket, which is one object-store write. With --ack replicated, it comes once the leader and a follower hold the write, and the bucket gets it a moment later.
2. Tiered. Rows that have committed are folded into a Parquet file in data/<table>/, at most every --tier-secs. A table that publishes gets a new Delta version and Iceberg snapshot, written only from state already in the bucket.
3. Merged. Later the row moves into bigger files. Delta records the swap as a remove and an add. Iceberg writes a snapshot without the old files. The old files are deleted after --retain-secs.

Other engines see a row from step 2. Measured with --tier-secs 2, the default when this was measured:

• about 30 ms after the acknowledgement on local disk;
• 3 to 4 seconds on a nearby R2 bucket, and 7 to 10 seconds on a far one, which is a few sequential round trips to the bucket.

Caution

The default is now --tier-secs 10, to write fewer objects. Other engines can be up to that much further behind. Lower it (--tier-secs 0.5) for fresher Delta and Iceberg, at the cost of more object-store writes. Nodes and pondra sql are not affected: they read the log tail.

Small files, compaction and skipping data

Small writes don’t make small objects, on disk or in a bucket.

• Batches. A node puts everything it received into one flush per round. Flushes up to 64 KB (1 MB with replicated acks) ride inside the catalog commit and make no object at all.
• Tiering writes one Parquet file per busy table per round.
• Append tables. Once 8 files are small, they are merged 8 into 1, up to 4 million rows or 64 MB. A table has at most about 8 small files, plus big ones.
• Keyed tables merge differently. Keyed tables explains.
• Table metadata stays small. A table’s catalog entry lists at most 128 files. Past that, all but the newest 64 are sealed into manifests of up to 4,096 files each, behind one list object. A table of a million files commits an entry of about 20 KB, as fast as a table of ten.

Every append-table file’s column ranges are kept, so a query skips whole manifests, then whole files, before it opens a Parquet footer. The rest of the skipping is in the Parquet files:

• partition_by (a column, or the year, month, day or hour of a timestamp) keeps one partition value per file, so a filter on it skips the other partitions.
• cluster_by sorts each file by those columns. Measured with 8 million rows and 100,000 users on local disk: a query for one user ran 6.4 times faster, a range of users 11 times faster, and a full scan 0.6 times as fast (it got slower, because sorted columns compress worse).
• Two or more cluster_by columns order rows along a Hilbert curve, so a filter on any of them skips most row groups.
• Parquet’s own min and max statistics per row group and page apply as usual.

Read it with other engines

A table opts in to being read by other engines. Make it with WITH (publish = 'delta,iceberg'), change it later with ALTER TABLE t SET (publish = 'delta'), or start a node with --publish delta,iceberg to publish every new table. Turning a format off deletes its metadata, so nobody reads a stale copy.

Then point a Delta reader at <lake>/data/<table>, or an Iceberg reader at <lake>/data/<table>/metadata/v<N>.metadata.json (N is in version-hint.text). The table made above works:

import os
import deltalake
import polars as pl
from pyiceberg.table import StaticTable

print(deltalake.DeltaTable("lake/data/orders").to_pandas())
print(pl.read_delta("lake/data/orders"))

version = open("lake/data/orders/metadata/version-hint.text").read().strip()
iceberg = StaticTable.from_metadata(os.path.abspath(f"lake/data/orders/metadata/v{version}.metadata.json"))
print(iceberg.scan().to_arrow().to_pandas())

   id  item  amount
0   1   tea     7.0
1   2  cake     4.5
shape: (2, 3)
┌─────┬──────┬────────┐
│ id  ┆ item ┆ amount │
│ --- ┆ ---  ┆ ---    │
│ i64 ┆ str  ┆ f64    │
╞═════╪══════╪════════╡
│ 1   ┆ tea  ┆ 7.0    │
│ 2   ┆ cake ┆ 4.5    │
└─────┴──────┴────────┘
   id  item  amount
0   1   tea     7.0
1   2  cake     4.5

That needs pip install deltalake polars pyiceberg pandas. On a bucket, add the store’s options: see Other engines. Delta and Iceberg readers in DuckDB, Polars, delta-rs and PyIceberg were checked against Pondra’s own answers, and so was Spark 4. Databricks, Trino and Athena read the same folders through their connectors, but were not tested here.

SELECT item, sum(amount) FROM delta_scan('s3://my-bucket/lake/data/orders') GROUP BY item;
SELECT count(*) FROM iceberg_scan('s3://my-bucket/lake/data/orders/metadata/v42.metadata.json');

Rules for outside readers:

• Write through Pondra. Pondra owns the folder. Write through a node or pondra sql, or append through a node’s Iceberg REST catalog (Spark, PyIceberg, Trino). Never commit to the metadata/ folder behind its back: the catalog is the truth, and a version it didn’t write is overwritten by the next one.
• Latest, plus a little history. 100 Delta versions and 100 Iceberg snapshots are listed. Files replaced by compaction are deleted after --retain-secs, so time travel further back than that fails.
• Types. Numbers, strings, booleans, dates, timestamps, binary, decimals and vector columns publish. INT8 and INT16 publish to Delta only. A table with a column type a format can’t hold isn’t published in that format.
• Keyed tables are published when one file holds one row per key, which is after a full compaction. CHECKPOINT compacts a published keyed table so other engines see it as it is now.
• Iceberg REST appends go to append tables. Deletes, schema changes and writes to keyed tables are refused by name.

Design notes

ADR-009 explains why the native format comes first and Delta and Iceberg are opt-in. The full format description lists every catalog key.