Pondra compared

Pondra puts a stream log, a lakehouse and a SQL engine in one binary, to replace Kafka, Flink, Spark and a metastore for the common jobs. It is new, and it doesn’t replace them all. This page puts it next to the tools you may be choosing between, says what each one is, and says where each one is stronger. The numbers are in Performance.

The short answer

Per machine, in the main areas, Pondra measured ahead of Spark and Flink:

• Batch SQL: TPC-H SF1 ran about 12 times faster than Spark 4.2 (4.5 s against 56.1 s for all 22 queries, same answers; 2026-10-01).
• Streaming: Nexmark’s q1, q2, q5, q7 and q11 over 10 million bids in 7.1 s against Flink 2.3’s 16.2 s on 4 cores (14.3 s against 30.4 s on 2), ingest over HTTP included; about 3 times Flink’s throughput on a keyed aggregation.
• Into the lake: a row is in the Delta and Iceberg tables other engines read about 0.4 s after its acknowledgement on local disk (3 to 10 s on R2), where Fluss’s lake tables trail by 3 minutes by default.
• Light: one 118 MB binary that answers about 0.1 s after it starts and idles at about 105 MB, half of it its own code; Spark and Flink take 6 to 7 s and over 530 MB.
• Latency: a change shows up on another node in about 5 ms. Writes are acknowledged in 2 to 4 ms on R2 with --ack replicated.
• Flows: materialized views that follow each other (bronze, silver, gold) move in the commit that wrote the rows, with expectations that keep, drop or refuse rows: no stage lags another, where Databricks’ pipelines, Snowflake’s dynamic tables and chained Flink jobs each lag the stage before (Flows).
• A database’s behaviour: BEGIN … COMMIT as one commit with snapshot isolation, Postgres’s error codes on every door, and key lookups through the Postgres port in about 0.4 ms. pgbench’s own TPC-B script runs with its balances right.

At cluster scale, and in breadth, it is not there yet:

• Scale: Spark and Flink are proven on thousands of machines. Pondra has run on several processes on one machine and on three machines over the internet, and nothing has run on a cluster in one data centre.
• Streaming features: Flink has timers, CEP and a watermark per partition. Pondra doesn’t.
• Ecosystem: Spark and Databricks have hundreds of connectors and mature governance.
• Maturity: Pondra is new: its first release was in 2026. It has a large test suite (crash, failover and chaos tests on one machine and on object storage) and benchmarks, but it hasn’t yet run for years in production as Spark, Flink or Postgres have.

Pondra’s own notes put the realistic claim like this: it can beat them for the common case (small to mid-size clusters, up to terabytes a day, SQL-shaped work) and take far less to run. The gaps are engineering work with a known shape. The biggest open risk is scale-out, and only a multi-machine benchmark can retire it.

At a glance

	What it is	Pick it over Pondra when	Pick Pondra when
Spark	A distributed batch and streaming engine, with DataFrame APIs in Python, Scala, Java and R	You need thousands of machines, Scala or Java jobs, hundreds of connectors, or a managed platform	The work is SQL-shaped, one small cluster does it, and you don’t want a JVM cluster
Flink	A stateful stream processor	You need timers, CEP, Top-N by event time, a watermark per partition or very large keyed state	SQL views, windows, joins and deduplication cover it, and you want no checkpoints and no Kafka underneath
Fluss	Streaming storage: a columnar log and primary-key tables, tiered to a lake	You want its proven scale and its Flink integration	You want the same ideas without ZooKeeper, TabletServers and a Flink tiering job
DuckDB (and Polars, Daft)	An analytical engine in your process	You want an embedded engine with a mature ecosystem, or its own format’s speed	You also need ingest, streaming and serving, or more than one machine
Databricks	A managed lakehouse platform	You want a managed platform, Unity Catalog governance, notebooks, ML and AI tools	You want to run it yourself, cheaply, from one binary
Snowflake	A managed cloud data warehouse	You want a managed warehouse with mature governance	You want open files in your own bucket, and streaming

Spark

Spark is a distributed engine for batch and streaming jobs, with DataFrame APIs in Python, Scala, Java and R.

Where Spark is stronger:

• It is proven on thousands of machines, with shuffles, spilling and skew handling.
• It has hundreds of connectors, notebooks, and Spark Connect clients in several languages.
• Real-Time Mode gives millisecond streaming latency. It became generally available in 4.1, for stateless Scala queries first.
• Scala and Java jobs run as they are. Pondra has no Scala or Java API, and no Spark Connect server.

Where Pondra fits:

• On the same machine, TPC-H SF1 took 4.5 s against Spark 4.2’s 56.1 s (2026-10-01), and a 20-million-row batch ran 2 to 14 times faster, with 293 MB of peak memory against 1.5 GB.
• One binary of about 100 MB, against 485 MB of PySpark and a JVM. It starts a first query in 0.02 to 0.07 s, where Spark takes 4.1 s.
• pondra.spark takes a PySpark job by changing its imports. 55 of 55 test pipelines give PySpark 4’s own answers and column names. PySpark says what differs.
• Views and tasks are declared in SQL and run incrementally, as Spark’s declarative pipelines do.

If you want PySpark and Spark SQL to run unchanged, Sail (LakeSail) is a Rust replacement for Spark built on the same DataFusion engine. It replaces Spark. Pondra replaces the stack around it too: ingest, exactly-once streaming, views, keyed tables, serving and a catalog in the bucket.

Flink

Flink is a stateful stream processor.

Where Flink is stronger:

• Timers, pattern matching (CEP, MATCH_RECOGNIZE), Top-N by event time, a watermark per partition, and very large keyed state. Pondra has none of the first three, and one watermark per source.
• The DataStream API, and years in production at scale.
• Newer SQL: materialized tables with a declared freshness, ML_PREDICT, VECTOR_SEARCH, Flink Agents.

Where Pondra fits:

• Streaming SQL that is part of the same engine: a materialized view is kept current on every commit, filled from the rows already there. Event-time tumbling, sliding and session windows are emitted once. Two streams join with each pair once, ASOF JOIN looks a table up as it was, and order_by deduplicates by event time.
• About 3 times Flink’s throughput on a keyed aggregation of 10 million events (4.2 s end to end, durable, over HTTP, against 13.3 s), and on the Nexmark queries q1, q2, q5, q7 and q11, 10 million bids in 7.1 s against Flink 2.3’s 16.2 s on 4 cores (14.3 s against 30.4 s on 2). Flink generated its input in process, and Pondra took its over HTTP.
• A change is visible on another node in 5 ms. Flink’s exactly-once results become visible at a checkpoint.
• A flow of views (a filter, then an aggregate of it, then a rollup) is one commit: each stage’s rows are written with the rows that caused them, exactly once, with no state to checkpoint between stages. In Flink each stage is an operator or a job whose output a sink commits at a checkpoint.
• No JVM, checkpoints, or Kafka or Fluss underneath.

Pondra’s stream joins run on the leader alone.

Fluss

Apache Fluss is streaming storage for analytics: a columnar log and primary-key tables with changelogs and lookups, whose lake tables a tiering service writes to Paimon, Iceberg or Lance. It became an Apache top-level project in August 2026 and released 1.0 on 2026-09-21. You pair it with Flink.

Where Fluss is stronger:

• It runs at scale: Alibaba reports 3 PB and more, 40 GB/s of ingest and 500,000 queries a second on one table. RedNote, JD.com and Ant Group run it too.
• It is a top-level Apache project at 1.0, with clients in Java, Rust, Python and C++.
• Its integration with Flink is deep: delta joins cut one Alibaba team’s Flink state from about 100 TB to almost nothing.

Where Pondra fits:

• One process to run. Fluss is a CoordinatorServer, TabletServers, ZooKeeper and a Flink tiering job, all on the JVM. Pondra is one binary, and object storage is its only state.
• Like for like, the fast paths match. Pondra’s nodes see a write 10 to 15 ms after the acknowledgement, as Fluss’s TabletServers serve theirs in milliseconds.
• The lake tables don’t. Pondra’s Delta and Iceberg versions trail the acknowledgement by about 0.4 seconds on local disk and 3 to 10 seconds on R2. Fluss’s lake tables trail by 3 minutes by default. Fluss can be set lower, and so can Pondra.
• Pondra already has the Postgres protocol and MCP, which are on Fluss’s roadmap.

Fluss’s roadmap moves toward Pondra’s design: writes straight to S3 with diskless servers, and no ZooKeeper.

DuckDB, Polars and Daft

DuckDB is an analytical database that runs inside your process, on one machine. MotherDuck runs it as a service.

Where DuckDB is stronger:

• It is the reference for single-machine SQL, with a mature ecosystem and extensions.
• With the data in memory, on 4 cores (round 32), DuckDB 1.5’s own format was about 15% ahead on TPC-H SF1 (1.05 s against Pondra’s 1.24 s), and its 2.0 preview more (0.92 s). On ClickBench 2.0’s preview was ahead (6.22 s against 6.58 s) and 1.5 behind (7.18 s).
• It is small, about 20 MB for its command line, against 118 MB for Pondra’s one binary (its server, every door, TLS and Python’s bridge included).

Where Pondra fits:

• Reading Parquet on 4 cores (round 32), Pondra was level with DuckDB 1.5 on TPC-H SF1 (2.16 s against 2.05 to 2.19 s) and ahead of DuckDB 1.5 and 2.0’s preview on ClickBench (10.47 s against 11.10 and 10.89 s).
• Reading Parquet files on 2 vCPUs, Pondra was the fastest of five at SF10 (38.0 s, against DuckDB’s 39.8 s), and level with Polars’ streaming engine at SF1. Polars’ streaming engine took 3.18 s to Pondra’s 3.19 s. Daft took 6.11 s.
• It also ingests, streams, serves and spreads a query over machines.
• It speaks their words. read_parquet, delta_scan, CREATE MACRO and ASOF JOIN work as in DuckDB, and pondra.frame uses Polars’ method names.

DuckDB is also a reader of Pondra’s tables: it reads the Delta and Iceberg tables Pondra publishes, and attaches Pondra’s Iceberg REST catalog.

Databricks

Databricks is a managed lakehouse platform: Spark, Delta Lake, Unity Catalog, SQL warehouses, and ML and AI tools.

Where Databricks is stronger:

• It is a managed platform, with notebooks, jobs, Agent Bricks and Genie, and governance through Unity Catalog (attribute-based access control).
• Lakehouse//RT (beta) is ahead on new, heavy analytical queries at high concurrency. Databricks claims sub-100 ms at 12,000 queries a second on a cluster. Pondra runs a new TPC-H query in 60 to 600 ms on two cores.
• Lakebase gives it a serverless Postgres next to the lakehouse. It has Zerobus for direct ingestion. Its newest items are collected partly from third-party summit recaps, so check them.

Where Pondra fits:

• Point reads and repeated dashboards: a key lookup takes 0.14 to 0.22 ms and 20,000 to 36,000 a second run on two cores (38,900 on four, and 27,600 while writes land), where Lakehouse//RT publishes 10 ms and 12,000 queries a second on a cluster.
• Rows a few milliseconds old, and it takes the writes itself.
• It runs anywhere you can run a binary, with no cloud account: on your disk or in any bucket.
• Spark 4 reads the Delta and Iceberg tables Pondra publishes, and gets Pondra’s answers. Databricks’ own runtime hasn’t been tried.
• Its flows are Databricks’ declarative pipelines in SQL — views following views, expectations (CONSTRAINT … EXPECT (…) ON VIOLATION DROP ROW | FAIL), counted in pondra.expectations — but every stage commits with its source’s rows, where a Databricks pipeline updates its tables on compute of its own, every 10 seconds to a few minutes in continuous mode. History per key (SCD type 2) isn’t built yet.

Snowflake

Snowflake is a managed cloud data warehouse.

Where Snowflake is stronger:

• It is a managed warehouse with mature governance.
• It has added Snowflake Postgres (generally available in February 2026), Openflow for ingestion, Snowpipe Streaming’s high-performance architecture, Cortex AISQL, a managed Iceberg catalog with Iceberg v3, and dynamic tables.

Where Pondra fits:

• database.schema.table names work: a lake is a database, and other lakes attach as databases.
• Tasks with CREATE TASK … SCHEDULE, stored procedures in SQL and Python, and Python functions follow Snowflake’s ideas. The table below sets them side by side.
• The tables are open files in your own bucket, and Iceberg and Delta readers can use them.
• A dynamic table refreshes within its target lag, 60 seconds at least; a Pondra flow’s views move in the commit that wrote the rows.

Pondra hasn’t been measured against Snowflake.

Functions, procedures and tasks

	DuckDB	Snowflake	Databricks	Pondra
SQL functions	CREATE MACRO	SQL UDFs	SQL UDFs	Postgres’s CREATE FUNCTION and DuckDB’s CREATE MACRO, kept in the lake
Python functions	In process	Python UDFs, vectorized UDFs, UDTFs	Python UDFs, pandas UDFs, UDTFs	LANGUAGE python: per row, vectorized, or a table
Stored procedures	None	SQL, Python, Java, Scala	SQL procedures, jobs	SQL and Python, called from SQL, Postgres, Python, JavaScript, the shell and MCP
Schedules	None	Tasks	Jobs	CREATE TASK … SCHEDULE
Who may make one	Anyone	Owner roles	Owner or admin	The admin token

What Pondra doesn’t have yet

Caution

• Evidence at scale. The multi-machine runs so far were on one machine and on three machines over the internet, where a cluster was no faster than one node.
• Governance. Users, roles and grants down to a column, TLS on every door and between nodes, an audit log and quotas are built (Security). Single sign-on, row filters and masking policies aren’t, and Unity Catalog’s and Snowflake’s governance go further.
• Transactions. Snapshot isolation, first committer wins (40001 to the other, to try again) where Postgres would wait on a row’s lock: under heavy contention on one row, most transactions retry. Flight SQL’s own transactions and savepoints aren’t taken.
• Kafka. One partition per topic and no transactions, so Kafka Streams and Flink’s exactly-once Kafka sinks are not covered.
• Other databases. Reading and writing Postgres or MySQL as attached databases, and streaming their changes in natively, are on the roadmap and not built. Debezium events through the Kafka port work.
• BI tools and JDBC. dbt, psql, SQLAlchemy, pgjdbc and psqlODBC are tested against the Postgres port, and so is Npgsql, the driver in Power BI Desktop’s PostgreSQL connector (dbt and BI tools).
• Streaming. No timers, CEP, or Top-N by event time, and one watermark per source.
• Maturity. Pondra is new and hasn’t run in production for years. Chaos tests across several machines, fuzzing and long soak runs are still on the plan.

Where to read more

Performance has every number behind this page. The comparison notes have the item-by-item comparison with Fluss 1.0, what Fluss, Flink, Spark, Databricks and Snowflake are building next, the plan for each gap, and the sources.