Pondra

Pondra is one program that takes in streams of rows, stores them as tables in a folder or a bucket (Parquet files plus a small catalog), and answers SQL. Materialized views, keyed tables and event-time windows keep results current as rows arrive, and other engines can read the tables as Delta Lake or Iceberg. Object storage is the only state, so there is no Kafka, Flink, Spark, JVM or metastore to run.

Start with one machine, grow to many

The same binary that runs on your laptop runs a cluster. Start another copy on the same bucket and it joins the first. There is nothing else to set up: the nodes find each other, and pick a leader, through the bucket.

Every node works

Every node takes writes and answers queries, and runs its share of the streaming tasks and of big queries. Add read-only nodes for dashboards, as many as you like.

Queries spread when it pays

A big query runs on every node at once, each on its slice of the data, with the rows exchanged between them. A node first weighs what spreading would move against the time it would save, so a small query stays on one node.

A node can stop

If the leader stops, another node takes over within seconds, and every write that was acknowledged is kept. Tested by killing leaders during writes, and by crashing nodes at random points.

Fast on one machine, too

Pondra is written in Rust on Apache DataFusion. On one 2-vCPU machine it ran TPC-H’s 22 queries at scale factor 10 from Parquet in 38.0 s, the fastest of the engines measured there (DuckDB: 39.8 s). Performance

Clusters shows how to start one, and how Pondra works explains why it needs no coordinator.

What you can do

Make tables

Append tables, keyed tables that upsert, merge tables, TTL, schemas and views. Tables

Load data

INSERT, exactly-once append, COPY, CSV, Parquet and JSON, pandas, Polars and Arrow. Load data

Query

SQL over HTTP, Postgres drivers and Arrow Flight, joins, parameters and live queries. Query

Read files and other lakes

Parquet, CSV and JSON on S3, GCS, Azure and HTTPS, and other engines’ Delta and Iceberg tables. Files and other lakes

Change rows

UPDATE, DELETE and MERGE, system columns, and a feed of every change. Change rows

Stream

Views kept current on every commit, event-time windows, stream joins and point-in-time joins. Streaming

Use DataFrames

Polars’ method names, or PySpark’s, over one SQL statement. DataFrames, PySpark

Write functions and procedures

SQL and Python functions, stored procedures, and tasks on a schedule. Functions and procedures

Share with other engines

Publish Delta and Iceberg, let Spark or PyIceberg append, and speak the Kafka protocol. Other engines, Kafka

Run a cluster

Several nodes on one bucket, on S3, R2, MinIO, GCS or Azure. Clusters, Object storage

Work with AI and files

Vector search, ai_complete and ai_embed, files in the lake, and an MCP server for agents. AI and files

Work in the browser

Every node serves a console: SQL, Python and Markdown cells, answers that stay live, and notebooks saved in the lake. The console

Connect your tools

dbt, DBeaver, Tableau, Metabase and Excel see a Postgres database, and a server holds many. dbt and BI tools, Many databases

Use it from notebooks, and lock it down

Jupyter with %%sql cells, and read, write and admin tokens on every door. Notebooks, Security

One binary replaces

For the common jobs, pondra takes the place of a stack:

Instead of running	For	Pondra has
Kafka	Ingesting events, topics, consumers	append with exactly-once batches, and a Kafka port: producers, consumers and Debezium change events
Flink	Streaming SQL: windows, joins, deduplication	Materialized views, event-time windows, stream joins, ASOF JOIN and tasks
Spark	Batch SQL, ELT and DataFrames	SQL that can spread over several nodes, pondra.frame, and PySpark’s names in pondra.spark
A metastore (Hive, Polaris, Unity)	The catalog of tables	A catalog inside the bucket, and an Iceberg REST catalog on every node
ZooKeeper	Picking a leader	A leader elected through the bucket

It replaces them for the common jobs, not all of them. Pondra is new: it has a large test suite (crashes, failovers and clients, on local disk and on object storage) and published benchmarks, but it hasn’t run for years in production as they have. Pondra compared says where Spark, Flink, Fluss, DuckDB, Databricks and Snowflake are stronger.

Try it in 30 seconds

Install Pondra with pip (other ways are on the install page), and start the shell on a folder called lake:

pip install pondra
pondra

Then type SQL. This makes a table and a view that keeps its totals current, writes three rows, and reads the totals:

CREATE TABLE events (user VARCHAR, amount BIGINT);
CREATE MATERIALIZED VIEW totals AS SELECT user, sum(amount) AS total FROM events GROUP BY user;
INSERT INTO events VALUES ('ann', 5), ('bob', 3), ('ann', 7);
SELECT * FROM totals ORDER BY user;

+------+-------+
| user | total |
+------+-------+
| ann  | 12    |
| bob  | 3     |
+------+-------+

Write another row and the view has it already:

INSERT INTO events VALUES ('bob', 4);
SELECT * FROM totals ORDER BY user;

+------+-------+
| user | total |
+------+-------+
| ann  | 12    |
| bob  | 7     |
+------+-------+

The folder lake now holds your tables. Stop Pondra, copy the folder to a bucket, and it opens there as it was. How Pondra works explains what is in it, and Your first lake does the same from Python and JavaScript.