How Pondra works

Pondra is one program. Copies of it share one lake, and the lake holds all the state: the rows, the log of the newest rows, and the catalog that lists them. This page follows a row from the moment you write it to the moment a query reads it, so you know where your data is and what you can count on.

The pieces

Every node takes writes and queries. The leader is one of them. All of them read and write the same lake.

One binary. pondra is the SQL shell, the server and pondra sql. The pip and npm packages carry the same binary. There is no JVM, no Postgres, no ZooKeeper and no Kafka to run beside it.
The lake is a folder, or a prefix in an S3, R2, MinIO, GCS or Azure bucket. Tables are Parquet files plus a log tail: the newest rows, not yet in a Parquet file. The lake format says what is in the folder.
Nodes are running copies of pondra. Every node takes writes and answers queries. Start another on the same bucket and it joins.
The leader is one of the nodes. It puts commits in order and does nothing else: it never touches your data. The nodes elect it through the bucket, so no other service has to run.

Follow a write

You send rows to any node. An INSERT, append from Python or JavaScript, a Kafka producer, a Postgres COPY.
That node batches them. It encodes the batch (Arrow, compressed with ZSTD), runs the materialized views over that table on it, and writes it to the bucket if it is over 64 KB (1 MB with --ack replicated). Smaller batches ride inside the catalog commit.
The leader puts it in order. It drops a retry it has already seen (the same producer and seq), numbers the batch, and commits every node’s batch of that round in one catalog write.
You get the acknowledgement. By default that is when the commit is in the bucket. With --ack replicated it is when two nodes hold it. Every node can read the row within milliseconds.
Later, the row becomes Parquet. Within --tier-secs (10 seconds by default), the nodes fold the new rows into a Parquet file under data/<table>/. Small files are merged, and replaced files are deleted after --retain-secs (60 by default).

Freshness and guarantees has what each step costs, and what is safe when.

Follow a read

A query reads the Parquet files and the log tail together, from one snapshot. A row is queryable the moment it commits: you don’t wait for it to become Parquet. Every node keeps the whole catalog in memory, kept current by the leader’s stream of commits, so a read doesn’t ask the bucket for metadata.

CREATE TABLE orders (id BIGINT, item VARCHAR, qty INT, amount DOUBLE);
INSERT INTO orders VALUES (1, 'tea', 2, 7.0), (2, 'cake', 1, 4.5), (3, 'tea', 1, 3.5);
SELECT count(*) AS orders, sum(amount) AS total FROM orders;

+--------+-------+
| orders | total |
+--------+-------+
| 3      | 15.0  |
+--------+-------+

CHECKPOINT turns every waiting row into Parquet now. The answer doesn’t change, because the query already read those rows from the log tail:

CHECKPOINT;
SELECT count(*) AS orders, sum(amount) AS total FROM orders;

+--------+-------+
| orders | total |
+--------+-------+
| 3      | 15.0  |
+--------+-------+

A node’s /stats counts the rows still waiting in the log:

curl -s localhost:8080/stats | grep -o '"untiered_rows":[0-9]*'

"untiered_rows":0

A process with no node

A machine with the binary and access to the folder or bucket can read, and write, without running a node. pondra sql opens the catalog and reads the log tail and the Parquet files itself. Nothing has to be running for it to work.

pondra sql --dir lake "SELECT count(*) AS orders FROM orders"

+--------+
| orders |
+--------+
| 3      |
+--------+

An INSERT from pondra sql does its own work: it runs the query on that machine and writes the Parquet. The leader only records the files. If nobody leads, pondra sql leads for a moment to record them. Clusters has the details.

The catalog is in the bucket

The catalog is a small key-value store (SlateDB) kept inside the lake, in catalog/. It lists the tables, log segments and files, and holds views, tasks, functions, procedures, secrets (sealed) and the lakes attached. One process, the leader, writes it. Everyone reads it.

So there is no metastore to run and no separate database to back up. The bucket is the only state, and nodes are disposable: a node’s SSD cache holds only copies of objects that are in the bucket.

The leader, and what happens when it goes

A node becomes leader by writing a small object, cluster/term/N, that fails if it already exists. Nodes check on each other over HTTP. If no peer still hears the leader, another node takes over after about 5 seconds (4.5 to 5.3 seconds measured with all nodes on one machine). The catalog fences the old leader, so once it is replaced it can’t write. After a full shutdown, the first node started on the lake leads at once.

The leader rewrites cluster/alive/N every 10 seconds, so a machine outside the cluster can tell a live leader from a dead one.

Queries across nodes

For a big query, every node runs the same plan on its own slice of the biggest table. Small tables and keyed tables are read whole, at the coordinator’s snapshot. There is no driver and no scheduler: the node that got your query coordinates it.

Where a join or a many-group aggregate needs rows that sit on another node, the nodes swap them (a shuffle). A shuffle that doesn’t fit in memory spills to disk.
A step that fails is retried once, then the shuffle runs again without that node.
The exchanges add up in the same order every time, so a query gives the same answer on every run.
A query spreads only when moving its data costs less than the time it saves, judged from the network’s measured speed. Otherwise it runs on the node that got it.

All 22 TPC-H queries give the same answers spread over three nodes as on one.

Where next

The lake formatWhat is in the folder, and who can read it.

Keyed tablesUpserts, deletes, merge tables and TTL.

Freshness and guaranteesWhen a write is safe, and when each reader sees it.

Design notesWhy it was built this way: the decision records.