Skip to content
PondraPondraPondra

Other engines

A Pondra table can also be a Delta Lake table and an Iceberg table, in the same folder as its Parquet files. Spark, Databricks, DuckDB, Polars, delta-rs, PyIceberg, Trino and Athena read them without knowing Pondra. Engines can also attach the node’s Iceberg REST catalog by URL, make tables there and append to them with their own compute. Use this page to hand a table to a team that uses other tools.

Publish a table

Section titled “Publish a table”

Publishing is per table, in CREATE TABLE. Pondra keeps the Delta log and the Iceberg metadata current as the table changes.

CREATE TABLE events (user VARCHAR, amount BIGINT, ts TIMESTAMP) WITH (publish = 'delta,iceberg');
INSERT INTO events VALUES
  ('ann', 5, '2026-09-29 10:00:00'),
  ('bob', 7, '2026-09-29 10:01:00'),
  ('ann', 2, '2026-09-29 10:02:00');

publish is 'delta', 'iceberg' or both. Change it later with ALTER TABLE events SET (publish = 'delta'); turning a format off deletes its metadata, so nobody reads a stale copy. pondra serve --publish delta,iceberg publishes every new table by default.

The table is a folder at <lake>/data/<table>, on disk or in a bucket:

lake/data/events/
  <uuid>.parquet
  _delta_log/00000000000000000000.json …     the Delta table
  metadata/v1.metadata.json …                the Iceberg table

Other engines see a table as of the last tiering round: about 30 ms after a write on local disk, and a few seconds on object storage. --tier-secs (default 10) is how often a busy table is tiered at most; Pondra’s own queries always see every commit at once. CHECKPOINT tiers every table now, so an engine sees the rows without waiting for the next round:

CHECKPOINT;

Read with Delta readers

Section titled “Read with Delta readers”

These read lake/data/events directly:

import duckdb
duckdb.sql("SELECT user, sum(amount) AS total FROM delta_scan('lake/data/events') GROUP BY user ORDER BY user").fetchall()

A lake in a bucket is the same with the bucket’s path and that engine’s own credentials: delta_scan('s3://my-bucket/lake/data/events').

Read with Iceberg readers

Section titled “Read with Iceberg readers”

Attach the node’s REST catalog and the tables are found by name, with no paths. The namespace default is the lake’s public schema; each other schema, and each attached lake, is a namespace of its own.

import duckdb
con = duckdb.connect()
con.execute("ATTACH 'pondra' AS lake (TYPE iceberg, ENDPOINT 'http://localhost:8080', AUTHORIZATION_TYPE 'none')")
con.execute("SELECT user, sum(amount) AS total FROM lake.default.events GROUP BY user ORDER BY user").fetchall()

You can also point a reader at the table’s metadata file. Its number is in metadata/version-hint.text:

import duckdb
version = open("lake/data/events/metadata/version-hint.text").read().strip()
duckdb.sql(f"SELECT count(*) FROM iceberg_scan('lake/data/events/metadata/v{version}.metadata.json')").fetchall()

With tokens on, give the catalog the read token: load_catalog(…, token="…") in PyIceberg, and in DuckDB CREATE SECRET t (TYPE iceberg, TOKEN '…') with SECRET t in the ATTACH.

Polars’ scan_iceberg and other Iceberg readers that need field ids work too, because the table’s Parquet files carry them.

Append from another engine

Section titled “Append from another engine”

Through the REST catalog, other engines append to a table with their own compute. The files a commit adds become the table’s where the writer put them: the node reads each file’s Parquet footer once, checks it against the table, and records it. It never copies the rows, so an append of any size costs the node about what a 10-row one does (a million rows: 0.01 s of its CPU, against 0.24 s to copy them). The rows get their row ids and versions from the commit, and the Delta copy and every reader see them. So this is Pondra’s catalog, not a second one.

import datetime as dt
import pyarrow as pa
from pyiceberg.catalog import load_catalog


catalog = load_catalog("pondra", type="rest", uri="http://localhost:8080")
rows = pa.table({
    "user": ["cy"],
    "amount": pa.array([9], pa.int64()),
    "ts": pa.array([dt.datetime(2026, 9, 29, 11, 0)], pa.timestamp("us")),
})
catalog.load_table("default.events").append(rows)
SELECT user, amount FROM events ORDER BY ts;
user  amount
ann   5
bob   7
ann   2
cy    9

Spark does the same through the catalog: INSERT INTO lake.default.events SELECT … or df.writeTo("lake.default.events").append().

Each commit lands once: a writer whose snapshot is stale gets 409 and retries on top of the new version, and a commit sent twice is applied once.

Checked, from each file’s footer: its row count is the one its manifest says, its columns have the table’s types, a NOT NULL column holds no NULL, and in a partitioned table the file holds one partition value. A file that fails is refused by name, and nothing of the commit is recorded.

Followed as written. Materialized views, streaming tasks, the change feed, /watch and Kafka topics take an append’s rows from its files, in the same commit that records them.

Copied instead, as a bulk INSERT would: an append to a table with a renamed or dropped column (Delta readers match columns by name, and the table’s files keep the names they were written under).

Refused: another engine’s compaction (Pondra merges its tables’ files itself).

Change rows from another engine

Section titled “Change rows from another engine”

Other engines change a table’s rows through the catalog too, both ways Iceberg has:

  • Copy-on-write, Iceberg’s default: Spark’s DELETE, UPDATE and MERGE, PyIceberg’s delete and overwrite. The commit takes the files it rewrote out of the table and adds the new ones, which are recorded as written.
  • Merge-on-read: Spark with the table’s write.delete.mode, write.update.mode and write.merge.mode set to merge-on-read (ALTER TABLE … SET TBLPROPERTIES from Spark, or in Pondra’s SQL). The commit adds position-delete files, which are taken as written: reads skip the rows they name without decoding them. Nothing is rewritten until a file is a tenth deleted; then Pondra’s maintenance rewrites it without them.

Rows in a file the change left alone keep their row ids; rows it rewrote get new ones, since Iceberg v2 sees a change as a delete and an insert. Pondra’s own UPDATE, DELETE and MERGE are published the same way: as Iceberg position-delete files and Delta deletion vectors, not rewritten files.

catalog.load_table("default.events").delete("user = 'cy'")
SELECT user, amount FROM events ORDER BY ts;
user  amount
ann   5
bob   7
ann   2
  • Against the table as Pondra has it. A change that takes files out is refused with 409 when the table has rows the writer couldn’t read in its last published version (rows still in the log, or changed by Pondra’s own UPDATE): the node writes them into files and publishes at once, and the writer reads the table again and runs Iceberg’s own conflict checks before it retries. So a DELETE from Spark never misses a row Pondra took before it. (A change that matches no file at all commits nothing, so rows only in the log escape it: CHECKPOINT first, or change them with Pondra’s SQL.)
  • Pondra’s upkeep never fails a writer. A change planned before Pondra merged or rewrote the files it names (writers retry with the new version, having checked it only rewrote files) is carried over to the files that hold those rows now, by their row ids. A row changed since fails it with 409, as it should.
  • Followed as written: views take the rows a change deletes back, and the change feed shows them as delete rows.
  • Refused by name: a change to a table followed by a view that can’t take rows back (one that keeps a min or max, or has no count(*)) or by a streaming task; one to a table with a renamed column (change those with Pondra’s SQL); and equality-delete files on an append table (Spark writes position deletes).

Several tables at once

Section titled “Several tables at once”

Iceberg’s multi-table transactions (POST /v1/transactions/commit) are one Pondra commit: every table’s change, or none. If one table changed since the writer read it, the whole transaction is refused with 409 and no table changes. A keyed table’s changes go a table at a time.

Keyed tables

Section titled “Keyed tables”

An engine writing to a keyed table (a PRIMARY KEY) writes upserts: each row it appends is its key’s newest version. Its deletes delete keys: a copy-on-write DELETE, a merge-on-read position delete of a key’s current version, and an equality delete on the key columns (what Flink’s upsert mode writes) all become delete markers. These go through the log, like Pondra’s own writes, so views and the change feed see them as upserts and deletes; the node tiers and publishes the table before it answers. A keyed table that combines each key’s rows (order_by, or a view’s GROUP BY) refuses other engines’ changes by name.

Schema changes come through the catalog as ALTER TABLE does them (an admin’s, when tokens are set): Spark’s ALTER TABLE … ADD COLUMN and PyIceberg’s update_schema add a column (at the end, optional), rename one, widen its type or drop it, and no file is rewritten. A required column added, columns reordered, a narrower type or a changed key are refused by name.

Make a table through the catalog

Section titled “Make a table through the catalog”

An engine can make, rename and drop tables there too, as CREATE TABLE, ALTER TABLE … RENAME TO and DROP TABLE do (with an admin token, when tokens are set). The schema’s types become Pondra’s, a required field is NOT NULL, identifier fields are the key, a partition spec is partition_by (one field: the column, or its year, month, day or hour) and a write order is cluster_by. (To write a partitioned table, PyIceberg needs its pyiceberg-core extra: pip install "pyiceberg[pyiceberg-core]".)

from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, LongType, StringType, TimestampType
from pyiceberg.partitioning import PartitionSpec, PartitionField
from pyiceberg.transforms import DayTransform


clicks = catalog.create_table(
    "default.clicks",
    schema=Schema(NestedField(1, "ts", TimestampType()), NestedField(2, "page", StringType()), NestedField(3, "ms", LongType())),
    partition_spec=PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=DayTransform(), name="ts_day")),
)
clicks.append(pa.table({
    "ts": pa.array([dt.datetime(2026, 9, 28, 9), dt.datetime(2026, 9, 29, 9)], pa.timestamp("us")),
    "page": ["/", "/docs"],
    "ms": pa.array([120, 80], pa.int64()),
}))
SELECT page, ms FROM clicks ORDER BY ts;
page   ms
/      120
/docs  80

A table’s layout is published for writers to follow: its partition_by as the Iceberg partition spec (so PyIceberg and Spark write a file per day, here), a cluster_by on one column as its sort order, and a keyed table’s key as its identifier fields.

What other engines see

Section titled “What other engines see”
  • The latest version, plus a little history: 100 Delta versions and 100 Iceberg snapshots. Files replaced by compaction are deleted after --retain-secs (60 by default), so time travel further back than that fails.
  • Column types: integers, floats, strings, booleans, dates, timestamps, binary, decimals and float arrays (vector columns) publish. A table with a column of another type isn’t published in that format.
  • Keyed tables publish every tier round, like append tables: the older versions a round replaces and its delete markers become position deletes (Delta deletion vectors), so readers see one row per key. A keyed table with order_by or merge functions still publishes after each full compaction (CHECKPOINT brings it up to date).
  • Deleted rows are Iceberg position-delete files and Delta deletion vectors. PyIceberg, DuckDB, Polars, Spark and Trino read them, and so does delta-rs’s QueryBuilder; its to_pyarrow_table doesn’t read deletion vectors yet, and refuses the table.
  • Schema changes follow. ADD COLUMN, renames and drops reach Delta (column mapping) and Iceberg (field ids) without rewriting files.
  • A table rename (ALTER TABLE … RENAME TO) keeps the Delta and Iceberg copies readable: the table’s folder stays where it is, and the Iceberg catalog lists the table by its new name.
  • Plain Parquet readers can read data/<table>/*.parquet, but they see replaced files and, for keyed tables, old versions of a key. Only the Delta log, the Iceberg metadata or Pondra itself gives the right answer.

To go the other way, reading other engines’ tables from Pondra, see files and other lakes.