Setting up the data feed

Setting up the data feed

A company admin sets up the data feed on the Data Feed page of the RollerUp admin portal, at /admin/data-feed on your company’s admin portal address. Any S3-compatible storage works: AWS S3, Cloudflare R2, Backblaze B2, Google Cloud Storage (with HMAC keys), Wasabi or SeaweedFS.

1. Create a bucket, or pick a folder in an existing one, such as rollerup/.
2. Create an access key that can write objects there, with as little else as your provider allows. Files are uploaded with multipart upload, so allow that too.
3. On the Data Feed page, choose Set up data feed and enter:
   • Endpoint: your provider’s S3 endpoint. Leave it empty for AWS S3.
   • Bucket
   • Folder: every file goes under it. Leave it empty, or end it with a slash, such as rollerup/.
   • Under Advanced, the Region (auto unless your provider needs one) and the URL style: path for R2 and SeaweedFS, virtual-hosted for AWS S3.
   • The access key ID and secret.
4. Choose Test and save. RollerUp writes <folder>_rollerup_connection_test.json to check the settings, and only saves them if it can. If it can’t, you see your provider’s error. You can delete that file.
5. Choose Turn on. The first delivery is a full snapshot of every table, then batches of changes.

RollerUp stores the secret encrypted and never shows it again.

AWS S3 example

An IAM policy for a user or role that can write files under rollerup/ in your-bucket, including replacing ones already there, but can’t read, list or delete them. Use a folder only the data feed writes to:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::your-bucket/rollerup/*"
    }
  ]
}

If the bucket is encrypted with a KMS key (SSE-KMS), also allow kms:GenerateDataKey and kms:Decrypt on that key. S3 needs kms:Decrypt to put a multipart upload’s parts together.

On the Data Feed page, leave the endpoint empty. Under Advanced, enter the bucket’s region (such as us-east-1) and choose Virtual-hosted (AWS S3).

Cloudflare R2 example

1. In the Cloudflare dashboard, go to R2, then Manage API tokens, and create an API token.
2. Give it Object Read & Write permission, and limit it to your bucket. R2 has no write-only permission; RollerUp still only ever writes.
3. Copy the Access Key ID and Secret Access Key. Cloudflare only shows the secret once.

On the Data Feed page, enter https://<ACCOUNT_ID>.r2.cloudflarestorage.com as the endpoint, with your Cloudflare account ID. For a bucket in the EU jurisdiction, use https://<ACCOUNT_ID>.eu.r2.cloudflarestorage.com. The defaults under Advanced (region auto, path URL style) are right for R2.

Changing your settings

Choose Edit on the Data Feed page to change the bucket, folder, endpoint or key. A new endpoint or bucket needs the key entered again. A new endpoint, bucket or folder only gets changes from then on, not your earlier data: for a full copy there, delete the data feed and set it up again. Changes are tested the same way before they’re saved. To rotate the key, create a new one with your provider, save it here, then delete the old one.

The page also shows recent deliveries and any errors, and lets you turn the data feed off and on again.