File Format & Layout

This page covers everything about how exported files appear in your bucket: the path layout, the file format itself, idempotency, and how to reason about ordering. Per-record schemas are on the Datasets page.

Bucket layout

Snapshot and change-stream files are written to distinct paths:

<your-prefix>/<dataset>/snapshot/<chunk-id>.jsonl.gz
<your-prefix>/<dataset>/cdc/date=YYYY-MM-DD/<chunk-id>.jsonl.gz

For example:

exports/fan_profiles/snapshot/128025bc-ce03-46e4-a010-558c130f188e.jsonl.gz
exports/fan_profiles/cdc/date=2026-07-28/128025bc-ce03-46e4-a010-558c130f188e.jsonl.gz

The Hive-style date= partition under cdc/ is recognized by most warehouse loaders (BigQuery external tables, Athena, Snowflake external stages, Databricks, etc.) and lets you prune ongoing change-stream scans to specific UTC dates. Snapshot files remain together under the dataset's snapshot/ prefix.

File format

• Encoding: UTF-8.
• Compression: gzip, a single concatenated gzip stream per file.
• Format: NDJSON, one JSON object per line, \n-terminated. A trailing newline is not guaranteed.
• Atomicity: every file is a whole-object write; there are no partial uploads. A retry after a transient failure may replace an existing object at the same key with an equivalent payload, but a given object key, once visible, is always a complete, valid file.

Because every visible file is complete and named deterministically, you can list the bucket and process new objects with no risk of reading a partial file.

Idempotency and ordering

Every record carries an id field that is deterministic within a delivery: replays of the same record during at-least-once retry edge cases always produce the same id. Treat id as the primary key when loading into a warehouse so duplicate files collapse on insert.

File ordering across partitions is not guaranteed. Use the event-time field inside each record to reason about chronological order, not the filename timestamp:

The filename timestamp is when the file was written, not when the underlying event occurred.

Snapshot vs change stream

The three state datasets — fan_profiles, fan_identity_graph, and fan_profile_merges — are delivered in two phases:

• Snapshot: an initial point-in-time set of every existing row at the moment the destination was enabled (or the moment a snapshot rerun was requested). Files land under <your-prefix>/<dataset>/snapshot/. All records in a snapshot represent state, not changes.
• Change stream: every subsequent insert / update / delete from that point forward. Files land under <your-prefix>/<dataset>/cdc/date=YYYY-MM-DD/.

The snapshot for a dataset completes before its steady-state change stream begins flowing for that dataset, so when you point a loader at a freshly-enabled destination you'll see the snapshot land first, then change-stream files arrive on a rolling basis.

fan_track_events is change-stream only; it has no snapshot phase. Track events are append-only behavioral signals (SDK telemetry and Rover message-lifecycle events), so there is no historical state to replay; files for this dataset begin arriving as new events occur after the destination is enabled.

You generally do not need to distinguish snapshot records from change-stream records when loading: an upsert by id works for either within a single export run. The one dataset where the distinction shows up in the payload is fan_identity_graph, where change-stream records carry a meaningful operation field (create / update / delete); snapshot records are always emitted as operation: "create".

Common envelope

Every record across every dataset carries these fields. Per-dataset sections on the Datasets page describe additional fields.

Records also carry a per-dataset event-time field (updatedAt, occurredAt, or timestamp); see each dataset's schema on the Datasets page.