Design notes
Each design note records one decision and the reasons for it. You don’t need them to use Pondra: the
other pages say what it does. Read them when you want to know why. They are in the repository’s
docs/ folder, and each link below opens
the full text on GitHub. The status is as each note says it.
Some notes were written as proposals and then built on by later ones, and still say “Proposed”.
Where it started
Section titled “Where it started”Why Pondra is one binary on object storage.
| Note | What it decides | Status |
|---|---|---|
| ADR-001 | The first study: a lakehouse on DuckLake, with snapshots published for readers that need only the bucket, and the alternatives it weighed | Proposed |
| ADR-002 | Why one binary instead of Fluss, Flink and Kafka: the streamhouse design and its size and speed budgets | Proposed |
| ADR-003 | Why serverless, and why every node runs the same plan on its share (SPMD) instead of a driver and executors | Proposed |
| ADR-004 | Cluster mode: the leader is elected through the bucket, the old leader is fenced, tasks keep their state in the lake | Accepted, built (round 3) |
| ADR-005 | Every node takes writes and the leader only orders commits; views, push and queries across nodes | Accepted, built (round 4) |
Tables, reads and formats
Section titled “Tables, reads and formats”How rows are stored, read fast, and shared with other engines.
| Note | What it decides | Status |
|---|---|---|
| ADR-006 | Keyed tables as a merge tree, and key lookups as a first-class path | Accepted, built (round 5) |
| ADR-007 | A lake other engines can read (Delta), tiering as rows commit, the SSD tier, and the catalog in every node’s memory | Accepted, built (round 6) |
| ADR-008 | Key lookups without SQL planning, keyed tables read as anti-joins, and the result cache | Accepted, built (round 7) |
| ADR-009 | Native format first, Delta and Iceberg on request, millisecond acknowledgements without S3 Express, writes from any machine, cluster_by |
Accepted, built (round 8) |
Doors: how you get in
Section titled “Doors: how you get in”The protocols, clients and installers.
| Note | What it decides | Status |
|---|---|---|
| ADR-010 | Everyone writes: the bucket inbox, attached lakes, SQL writes, the Postgres protocol, the Python client, MCP and tokens | Accepted, built (round 9) |
| ADR-011 | The Kafka protocol, an Iceberg REST catalog, ALTER TABLE ADD COLUMN, JSON functions and event-time windows |
Accepted, built (round 10) |
| ADR-018 | Install anywhere: a glibc 2.17 Linux binary, pip and npm, the SQL shell, local(), and sums that add up the same in any order |
Accepted, built (round 17) |
| ADR-024 | Nothing to set up: one-line installers, python -m pondra, and rows without pyarrow |
Built (0.22.1) |
Scale and speed
Section titled “Scale and speed”Bigger tables, faster queries, and queries across nodes.
| Note | What it decides | Status |
|---|---|---|
| ADR-012 | Table metadata that stays small, partitions, memory limits and spilling, shuffles between nodes, Arrow Flight | Accepted, built (round 11) |
| ADR-013 | What made one node fast, files and bytes and vectors as column types, models in SQL, and functions of your own | Accepted, built (round 12) |
| ADR-014 | Shuffles that fit on disk, steps that can be retried, and a join order chosen from the catalog | Accepted, built (round 13) |
| ADR-015 | Any query across the nodes, and the same answer every time | Accepted, built (round 14) |
| ADR-016 | Tables split by the ranges of a key, hot keys shared out, and distinct values counted for the join order | Accepted, built (round 15) |
| ADR-017 | Watermarks taken from the data, session windows, and point-in-time joins (ASOF JOIN) |
Accepted, built (round 16) |
A database you can shape
Section titled “A database you can shape”Names, DDL, changing rows, and streams.
| Note | What it decides | Status |
|---|---|---|
| ADR-019 | Schemas and three-part names, DDL in SQL, stored and materialized views, and other lakes attached as databases | Accepted, built (round 18) |
| ADR-020 | UPDATE, DELETE and MERGE on every table, system columns, a change feed, and a cluster never slower than one node |
Accepted, built (round 19) |
| ADR-021 | Far fewer objects in the bucket, a key with partition_by and cluster_by, Hilbert clustering, COPY, stream joins and sliding windows |
Accepted, built (round 20) |
| ADR-022 | Renaming and dropping columns with no file rewritten, views that start full, the latest row by event time, and the DataFrame API designed | Accepted, built (round 21) |
SQL, Python and the outside world
Section titled “SQL, Python and the outside world”One vocabulary, and reading and writing everything else.
| Note | What it decides | Status |
|---|---|---|
| ADR-023 | Frames with Polars’ and PySpark’s names, macros and procedures: SQL and Python as one language | Accepted, built (round 22) |
| ADR-025 | One name, one meaning: SQL’s words are the clients’ names | Built (0.23.0) |
| ADR-026 | Read and write anything: files, Delta and Iceberg tables and Kafka clusters anywhere, and CREATE SECRET |
Accepted, built (round 23) |
| ADR-027 | CREATE FUNCTION in Postgres’s forms, procedures that do anything Python can, warm workers, schedules and a run log |
Accepted, built (round 24) |
| ADR-028 | read_* and write_* everywhere, other engines appending through the Iceberg REST catalog, live queries, temporary tables |
Accepted, built (round 25) |
| ADR-033 | A workspace: SQL, Python and notebook files in the lake, run with parameters from every client, recorded and scheduled | Accepted, built (after round 26) |
| ADR-034 | A console of tabs: notebooks, SQL, Python and data files edited in place, one grid, a performance budget | Accepted, built (round 26) |
Decisions made and not yet built.
| Note | What it decides | Status |
|---|---|---|
| ADR-029 | Anyone’s compute, one catalog: other engines write with their own compute and Pondra only commits their files | Accepted; phase 1 built (round 27), phase 2 next |
| ADR-030 | The web console, serving a folder of lakes as databases, Postgres as dbt and BI tools use it, and this documentation site | Accepted (round 26) |
| ADR-031 | Extensions: DuckDB-style INSTALL and LOAD, as WebAssembly components |
Proposed |
| ADR-032 | Files by name (CREATE EXTERNAL TABLE), one serve command (--lake, --lakes, local or in a bucket), a page’s Python cells sharing variables, the brand in one place, and the console as a core to build on; backward compatibility from 1.0; a server’s catalog (proposed) |
Accepted (round 26), §9 proposed |
Other documents
Section titled “Other documents”| Document | What it is |
|---|---|
prototype-status.md |
The status report. Where Pondra stands, round by round: what it does, every measured number, and what is missing |
roadmap.md |
What is left and in what order |
lake-format.md |
The lake on disk and in the bucket, in full. The lake format is the short version |
comparison-spark-flink-fluss.md |
Head to head with Spark, Flink, Fluss, Lakehouse//RT and the single-node engines, with sources. Pondra compared is the short version |
dataframe-api.md |
The DataFrame API’s design, and the table of names |
README.md |
The order to read the documents in |