Loading the files

Loading the files

What lands in your bucket

<folder> is the folder you chose when setting up the data feed, such as rollerup/, or nothing if you left it empty.

<folder>wash_packages/20261020T140003Z.snapshot-0001.parquet
<folder>wash_packages/20261020T141502Z.parquet
<folder>visits/20261020T140003Z.snapshot-0001.parquet
<folder>visits/20261020T140003Z.snapshot-0002.parquet
<folder>visits/20261020T141502Z.parquet
<folder>_batches/20261020T140003Z.snapshot.json
<folder>_batches/20261020T141502Z.json
<folder>_rollerup_connection_test.json
  • One folder per table. A batch has one file per table, <table>/<batch>.parquet, and only for tables where one of your rows changed.
  • Batch names are UTC times, YYYYMMDDTHHMMSSZ: 20261020T141502Z is 14:15:02 UTC on October 20, 2026. They sort in order, and a later batch always has a later name.
  • Your first snapshot is named after the batch it starts from, <batch>.snapshot. It sorts right after that batch and before every later one. A big table comes in several numbered files, -0001, -0002 and so on.
  • Markers go in _batches/, one per batch or snapshot. Always read file paths from the marker rather than building them from its name.

Markers

A marker is written after all of its files are in place, and lists them:

{
  "batch": "20261020T141502Z",
  "kind": "incremental",
  "format_version": "v1",
  "created_at": "2026-10-20T14:15:02.118Z",
  "files": [
    {
      "table": "prices",
      "path": "rollerup/prices/20261020T141502Z.parquet",
      "rows": 3
    }
  ]
}
Field Description
batch The batch it’s for. For a snapshot, the batch it starts from, which can differ from the marker’s own name. Sort and track markers by their file name, not this field.
kind incremental for a batch of changes, snapshot for a full copy. See Loading below.
format_version The version of this file format: the markers, the metadata columns and how deletes are sent. Currently v1.
created_at When the batch was made, in UTC.
files Every file, with its table, its path in your bucket and how many rows it has.

Only load a batch once its marker exists. A batch’s files arrive one after another. The marker tells you all of them are there, so tables loaded from the same batch stay consistent with each other.

A marker can list no files. That batch had no changes for you, and still tells you the data feed is running.

Columns every file has

Column Type Description
_row_id string The row’s ID, the same as id, but also set on deleted rows, where id is null. Upsert and delete on it.
_deleted boolean true when the row was deleted. Only _row_id and _deleted are set on a deleted row; every other column is null.

After these come the table’s own columns, listed in Tables.

Loading

Keep track of the newest marker you’ve loaded. Then, each time you check your bucket:

  1. List <folder>_batches/ and take every marker with a name after the newest one you’ve loaded, in name order.
  2. For each marker, load its files according to its kind:
    • incremental: for each row, if _deleted is true, delete the row with that _row_id. Otherwise insert it, or replace the row with the same _row_id.
    • snapshot: replace each listed table’s contents with its files. A snapshot lists every table it covers, even one where you have no rows (an empty file), so once you load it, each of those tables has exactly the snapshot’s rows. Leave tables it doesn’t list as they are.
  3. Load a marker’s files in the order it lists them, all in one transaction if you can, then record its name as the newest you’ve loaded.

A few things to expect:

  • A row can appear in several batches. Each time it has its values as of that batch, and the later batch wins.
  • Loads are safe to repeat. A retried delivery writes the same file names with the same rows, so loading the same batch twice gives the same result.
  • Use the marker’s kind, not its name, to decide between upserting and replacing.
  • A marker can list several files for the same table. Load them in the listed order.

Tools that load each file as it lands, such as Snowpipe or Databricks Auto Loader, only append rows. Pointed straight at the bucket, they keep every version of a row, never remove deleted rows, and don’t replace a table on a snapshot. Load into a raw table with them, then apply the steps above from there.

To read a file directly, any Parquet reader works. For example, with DuckDB:

SELECT *
FROM read_parquet('s3://your-bucket/rollerup/prices/20261020T141502Z.parquet');

Knowing it’s running

A batch comes about every 15 minutes, even when nothing changed. If your newest marker is over an hour old, the data feed has stopped. The Data Feed page in the admin portal shows recent deliveries and any errors.

Changes to the columns

The columns follow the public GraphQL API, so they change the same way:

  • New columns can appear at any time. Make sure your loader tolerates columns it doesn’t know, by adding them or ignoring them.
  • A new column is null on rows that haven’t changed since it was added. Rows aren’t resent when a column is added: each row gets the column the next time it changes.
  • Removing a column or changing its type follows the API’s deprecation process, with the same notice API clients get.
  • The file format itself (markers, metadata columns, how deletes are sent) is versioned by format_version. It only changes with notice.
Previous Setting up the data feed Next Tables