Snapshots
Snapshots
A snapshot is a full copy of one or more tables. Load it like any other marker (see Loading the files): replace each table it lists with its files.
Your first snapshot
A new data feed starts with a snapshot of every table, named after the batch it starts from: <batch>.snapshot. Batches follow from there. A big snapshot can take a few runs to write, and its marker only appears once every file is there.
Snapshots of some tables
When RollerUp adds a table to the data feed, you get a snapshot of just that table, then its changes in batches. Until then, your batches leave it out.
That snapshot’s marker is named after the newest batch you already have, <batch>.snapshot~<time>, so it sorts after everything you’ve loaded and before your next batch. Its files are named after the batch the snapshot started from, which can be an earlier one, so read their paths from the marker.
If batches went out while the snapshot was being written, a second marker, <batch>.snapshot~<time>~catchup (kind incremental), comes right after it with those tables’ changes from those batches. It lists each table’s files from several batches, oldest first. Its batch is the newest batch you already have.
Loading a snapshot can leave a row in one of its tables briefly older than it is in RollerUp. Loading the batches after it brings every such row back in line.
Moving your bucket
Changing the endpoint, bucket or folder keeps the same data feed, so the new location only gets batches from then on, plus fresh schema files. It doesn’t get your earlier data. To get a full copy at the new location, delete the data feed on the Data Feed page in the admin portal and set it up again instead. Replacing the key doesn’t change where files go.
If delivery stops
RollerUp keeps undelivered batches for about 6 days. If it can’t write to your bucket for that long, for example because the key was revoked, the data feed stops. It can’t be turned back on: to start again, delete the data feed on the Data Feed page and set it up again, which begins with a fresh snapshot of every table. Loading it replaces each table, which also removes rows deleted while you weren’t receiving batches.