Data feed
Data feed
The data feed delivers your RollerUp data as files to a storage bucket you own: AWS S3, or any S3-compatible storage. RollerUp writes a full copy of each table once, then about every 15 minutes writes the rows that changed. The files are Parquet, one folder per table, with the same tables and columns as the GraphQL API. If you know the API, you know the files.
When to use it
Use the data feed when you want all of your data, not just what you ask for: every row, without paging through the API or running into rate limits. What you do with the files is up to you, such as loading them into another system.
Use the GraphQL API when you need live data for a specific customer, car or visit, or when you need to make changes.
| Data feed | GraphQL API | |
|---|---|---|
| What you get | Every row of every table, then the rows that changed | What you ask for, when you ask |
| How often | New files about every 15 minutes | Whenever you ask |
| Can make changes | No | Yes |
| Rate limits | None | Cost-based |
| Good for | Getting all your data into another system | Apps, integrations, lookups and automations |
The files work with anything that reads Parquet from S3-compatible storage: Snowflake, BigQuery, Amazon Redshift, Databricks, ClickHouse, DuckDB, and most ELT and BI tools.
What it isn’t
- Not a change history. A batch has each changed row as it is when the batch is made, not every change in between. If a row changes three times between batches, the next batch has it once, with its values after the third change. Keep your own history if you need one.
- Not nested. Each table is flat: a nested object comes as its ID (a price’s
productasproductId), and lists aren’t included. Some fields, such as ones that are expensive to work out, may be left out, and some tables only hold some rows, such as finished visits. See Tables. - One-way. Once the files are in your bucket, anything you do to them only changes things on your side.
How it works
- You create a bucket and a key that can write to it, and a company admin enters them in the RollerUp admin portal. See Setting up the data feed.
- RollerUp writes a test file to check the key, then a full snapshot of every table.
- From then on, about every 15 minutes, RollerUp writes a batch of the rows that changed since the last one.
- After each batch’s or snapshot’s files, RollerUp writes a small marker file that lists them. Read a batch only once its marker exists. See Loading the files.
RollerUp only ever writes to your bucket. It never reads, lists or deletes anything there.