Iceberg REST catalog

Every node serves the Iceberg REST catalog API under /v1, on the same port as HTTP. Point an engine at the node’s URL and it finds the tables that publish Iceberg, reads them, and appends to them. Nothing has to be started: it needs no flag.

Pondra stays the catalog that counts. Other engines read tables and append rows to append tables. Everything else (creating tables, deleting, updating, changing a schema) is done in Pondra’s SQL, and the catalog refuses it by name.

Which tables

A table is in the catalog when it publishes Iceberg:

CREATE TABLE events (id BIGINT, kind VARCHAR, amount DOUBLE) WITH (publish = 'iceberg');
INSERT INTO events VALUES (1, 'view', 0.0), (2, 'buy', 9.5);
CREATE SCHEMA sales;
CREATE TABLE sales.orders (id BIGINT, item VARCHAR) WITH (publish = 'iceberg');

Its first, empty version is published when the table is made, so an engine that appends to a new table finds it. Views are not in the catalog. Set publish on a table that exists with ALTER TABLE … SET (publish = 'iceberg'): see table options.

Namespaces are schemas:

Namespace	Is
default	the lake’s public schema
sales	the lake’s other schema sales
other	the public schema of an attached lake other
other.sales	that lake’s schema sales. On the wire the parts are joined with the character 0x1F, as the REST specification says

Other engines read a table’s files, so they see the rows the node has written to files. New rows sit in the log first and reach the files within --tier-secs (10 seconds by default). Pondra’s own queries see every write at once.

Caution

The table’s Iceberg version follows its files: after an INSERT, another engine may see the old version for up to --tier-secs. POST /tier (with the admin token) writes the files at once, or read with Pondra’s SQL.

Endpoints

Method and path	Meaning
GET /v1/config	the catalog’s settings and the list of endpoints it serves. Clients read it to know what to ask for
GET /v1/namespaces	every namespace
GET, HEAD /v1/namespaces/{ns}	one namespace. 404 NoSuchNamespaceException if there is none
GET /v1/namespaces/{ns}/tables	the tables of a namespace that publish Iceberg
GET, HEAD /v1/namespaces/{ns}/tables/{table}	load a table: metadata-location, the current metadata, and config. 404 NoSuchTableException if there is none
POST /v1/namespaces/{ns}/tables/{table}	commit an append or a change of rows. See appending
POST /v1/transactions/commit	commit changes to several tables of a database at once: all or none. 204
POST /v1/namespaces/{ns}/tables	make a table, as CREATE TABLE … WITH (publish = 'iceberg') does. See making tables
DELETE /v1/namespaces/{ns}/tables/{table}	DROP TABLE, its files with it. 204
POST /v1/tables/rename	ALTER TABLE … RENAME TO, within the table’s namespace. 204

There is no {prefix} in the path. These are refused, each with a 400 BadRequestException whose message names the SQL that does it:

Request	Do this in Pondra’s SQL
POST /v1/namespaces	CREATE SCHEMA
DELETE /v1/namespaces/{ns}	DROP SCHEMA

Errors come in the REST API’s shape, {"error": {"message": …, "type": …, "code": …}}. A client that reads /v1/config knows which of these the catalog serves, and does not send the others.

# write the new rows to files now, so other engines see them
curl -s -X POST localhost:8080/tier > /dev/null
curl -s localhost:8080/v1/namespaces
echo
curl -s localhost:8080/v1/namespaces/default/tables
echo
curl -s localhost:8080/v1/namespaces/sales/tables
echo
curl -s localhost:8080/v1/namespaces/default/tables/events | python3 -c "import json, sys; d = json.load(sys.stdin); print(sorted(d), d['metadata']['format-version'])"

{"namespaces":[["default"],["sales"]]}
{"identifiers":[{"namespace":["default"],"name":"events"}]}
{"identifiers":[{"namespace":["sales"],"name":"orders"}]}
['config', 'metadata', 'metadata-location'] 2

Read with PyIceberg

from pyiceberg.catalog import load_catalog

catalog = load_catalog("pondra", uri="http://localhost:8080", type="rest")
print(catalog.list_namespaces())
print(catalog.list_tables("default"))
table = catalog.load_table("default.events")
print(table.scan().to_arrow().sort_by("id").to_pydict())

[('default',), ('sales',)]
[('default', 'events')]
{'id': [1, 2], 'kind': ['view', 'buy'], 'amount': [0.0, 9.5]}

A table is Iceberg format 2. Its Parquet files carry Iceberg field ids, so readers that need them, such as Polars’ scan_iceberg, read the table too. After ALTER TABLE … RENAME TO, the catalog lists the table under its new name.

Appending

A commit is POST /v1/namespaces/{ns}/tables/{table}, as PyIceberg’s append and Spark’s INSERT INTO and writeTo(…).append() send it. The engine writes Parquet files in the table’s data/{table}/data/ folder and commits a snapshot. The node reads each added file’s Parquet footer (one ranged read), checks it, and the leader records the files as the table’s, where they are, with their rows’ ids, version and time; then it publishes the table’s next version under the engine’s snapshot id. The rows are never read or written again, so the commit’s cost grows with the number of files, not the rows.

• Checked from each footer: the row count is the manifest’s, each column has the table’s type (matched by field id, or by name), a NOT NULL column holds no NULL, and a partitioned table’s file holds one partition value. A file that fails is refused by name with a 400, and nothing of the commit is recorded.
• Followed in the same commit: views and streaming tasks that follow the table take the new rows from the files, and the change feed, /watch and Kafka topics read them there.
• Copied, not recorded as written: when the table has a renamed or dropped column (Delta readers match columns by name). Then the node writes the rows as the table’s own, as a bulk INSERT does, and deletes the engine’s files. A keyed table’s rows go through the log as upserts (below).
• The layout the metadata publishes is the one to follow: the table’s partition_by as its partition spec, a cluster_by on one column as its sort order, a key as its identifier fields.

import pyarrow as pa

table.append(pa.table({"id": pa.array([3], pa.int64()), "kind": ["buy"], "amount": [4.5]}))

SELECT id, kind FROM events ORDER BY id;

[{"id":1,"kind":"view"},{"id":2,"kind":"buy"},{"id":3,"kind":"buy"}]

What a commit may contain:

Part	Taken
add-snapshot with operation append, overwrite or delete	yes: one, or several one after the other (PyIceberg’s overwrite sends a delete’s and an append’s), the last made main
set-snapshot-ref	only for main, and only as a branch that makes the snapshot main
requirements	assert-table-uuid, assert-ref-snapshot-id (on main), assert-current-schema-id, assert-default-spec-id, assert-default-sort-order-id, assert-last-assigned-field-id, assert-last-assigned-partition-id
data files	Parquet, under the table’s data/ folder: the ones the snapshots add, and the table’s files they remove (copy-on-write)
position-delete files	Parquet, under the table’s data/ folder (merge-on-read): each file they name must be the table’s; reads skip those rows
equality-delete files	on a keyed table, by its key’s fields: delete markers for those keys
properties	set-properties and remove-properties alone: ALTER TABLE … SET TBLPROPERTIES, run as the caller (write.delete.mode and the like)
a schema change	add-schema and set-current-schema alone (with the name mapping’s set-properties): each field added, renamed, widened or dropped is that ALTER TABLE, run as the caller

Everything else is refused with a 400 BadRequestException that says why: a snapshot that does replace (another engine’s compaction), a schema change Pondra’s ALTER TABLE can’t make (a required column added, columns reordered, a narrower type, a changed key), other branches and tags, equality deletes on an append table (or on a keyed table’s other columns), a change to a table followed by a view that can’t take rows back (a min or max, no count(*)) or by a streaming task, a change to a table with a renamed column, a change to a keyed table that combines its rows (order_by, merge functions), files outside the table’s folder, and any other update in a commit.

A keyed table (identifier fields) takes an append as upserts, each row its key’s newest version, and its deletes as delete markers: a copy-on-write commit’s files taken out, a position delete of a key’s current version, an equality delete by the key. They go through the log as Pondra’s own writes do, and the node tiers and publishes the table before it answers.

Iceberg’s rules hold. The requirements are checked as one step, under the leader’s lock. A commit made against a table that has changed since the engine read it is answered 409 CommitFailedException, and PyIceberg and Spark retry on top of the new version. A commit that takes files out or deletes rows is also answered 409 when the table holds rows its last published version doesn’t (rows in the log, or rows Pondra changed): the node writes them into its files and publishes first, so the engine’s retry sees them. A file the commit takes out, or deletes rows of, that Pondra merged or rewrote since has the change carried over to the files holding those rows now, by their row ids (a version that only rewrote files is published as a replace, which the engine’s own conflict checks pass over); a row changed since is a 409. A commit sent again with a snapshot id already recorded is answered as done and applied once. A transaction is one commit: a 409 for any of its tables refuses all of them.

Making tables

POST /v1/namespaces/{ns}/tables makes the table as CREATE TABLE would, published as Iceberg (and in whatever else its publish property names), and answers its first version:

Iceberg	Pondra
boolean, int, long, float, double, date, timestamp, timestamptz, string, binary, decimal(p, s), a list of one of them	BOOLEAN, INT, BIGINT, REAL, DOUBLE, DATE, TIMESTAMP, TIMESTAMPTZ, VARCHAR, BYTEA, DECIMAL(p, s), type[]
a required field	NOT NULL
identifier-field-ids	PRIMARY KEY
a partition spec of one field: identity, year, month, day, hour	PARTITION BY col, PARTITION BY day(col)…
a write order of columns, ascending	CLUSTER BY (col, …)

Other types (struct, map, uuid, time, fixed), a spec of more than one field or by bucket or truncate, a descending order and a staged create are refused by name. Making, renaming and dropping a table need an admin token when tokens are set, as the SQL does.

Note

Changes come copy-on-write (Spark’s DELETE, UPDATE and MERGE in Iceberg’s default mode, PyIceberg’s delete and overwrite) or merge-on-read (Spark with the table’s write.delete.mode and the like set to merge-on-read). Pondra publishes deleted rows the same way: Iceberg position-delete files, Delta deletion vectors.

Read with DuckDB

DuckDB’s iceberg extension attaches the node as a catalog.

import duckdb

con = duckdb.connect()
con.sql("INSTALL iceberg; LOAD iceberg;")
con.sql("ATTACH '' AS pondra (TYPE iceberg, ENDPOINT 'http://localhost:8080', AUTHORIZATION_TYPE 'none')")
print(con.sql("SELECT kind, sum(amount) AS total FROM pondra.default.events GROUP BY kind ORDER BY kind").fetchall())

[('buy', 14.0), ('view', 0.0)]

Other engines

Spark 4 with Iceberg 1.10 reads and appends the same way. Trino and Snowflake speak the same API; they are not tested here.

spark-sql \
  --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.pondra=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.pondra.type=rest \
  --conf spark.sql.catalog.pondra.uri=http://node.example.com:8080 \
  -e "INSERT INTO pondra.default.events VALUES (4, 'view', 0.0)"

A lake on object storage needs the engine to reach the bucket too: for PyIceberg on R2 or S3, add py-io-impl set to pyiceberg.io.fsspec.FsspecFileIO and the s3.* properties.

Tokens

With tokens set, the catalog uses the same ones as HTTP, sent as Authorization: Bearer. Reading needs the read token, and a commit needs the write token. A token with too few rights gets 403 with the text this needs more rights than this token's or user's; no token, or a wrong one, 401.

import os
from pyiceberg.catalog import load_catalog

catalog = load_catalog("pondra", uri="https://node.example.com", type="rest", token=os.environ["PONDRA_WRITE_TOKEN"])