File Format & Layout

This page covers everything about how exported files appear in your bucket: the path layout, the file format itself, idempotency, and how to reason about ordering. Per-record schemas are on the Datasets page.

Bucket layout

Every record is written to a path of the form:

<your-prefix>/exports/<dataset>/date=YYYY-MM-DD/hour=HH/<dataset>-YYYYMMDDTHHmmssZ-part-<token>.ndjson.gz

The Hive-style date= / hour= partitions are recognized by most warehouse loaders (BigQuery external tables, Athena, Snowflake external stages, Databricks, etc.) and let you prune scans to specific time windows without listing every object.

File format

• Encoding: UTF-8.
• Compression: gzip, a single concatenated gzip stream per file.
• Format: NDJSON, one JSON object per line, \n-terminated. A trailing newline is not guaranteed.
• Atomicity: every file is a whole-object write; there are no partial uploads. A retry after a transient failure may replace an existing object at the same key with an equivalent payload, but a given object key, once visible, is always a complete, valid file.

Because every visible file is complete and named deterministically, you can list the bucket and process new objects with no risk of reading a partial file.

Idempotency and ordering

Every record carries an id field that is deterministic within a delivery: replays of the same record during at-least-once retry edge cases always produce the same id. Treat id as the primary key when loading into a warehouse so duplicate files collapse on insert.

File ordering across partitions is not guaranteed. Use the event-time field inside each record to reason about chronological order, not the filename timestamp:

The filename timestamp is when the file was written, not when the underlying event occurred.

Snapshot vs change stream

The three state datasets — fan_profiles, fan_identity_graph, and fan_profile_merges — are delivered in two phases, both of which land at the same exports/<dataset>/... prefix:

• Snapshot: an initial point-in-time set of every existing row at the moment the destination was enabled (or the moment a snapshot rerun was requested). All records in a snapshot represent state, not changes.
• Change stream: every subsequent insert / update / delete from that point forward.

The snapshot for a dataset completes before its steady-state change stream begins flowing for that dataset, so when you point a loader at a freshly-enabled destination you'll see the snapshot land first, then change-stream files arrive on a rolling basis.

fan_track_events is change-stream only; it has no snapshot phase. Track events are append-only behavioral signals (SDK telemetry and Rover message-lifecycle events), so there is no historical state to replay; files for this dataset begin arriving as new events occur after the destination is enabled.

You generally do not need to distinguish snapshot records from change-stream records when loading: an upsert by id works for either within a single export run. The one dataset where the distinction shows up in the payload is fan_identity_graph, where change-stream records carry a meaningful operation field (create / update / delete); snapshot records are always emitted as operation: "create".

Common envelope

Every record across every dataset carries these fields. Per-dataset sections on the Datasets page describe additional fields.

Records also carry a per-dataset event-time field (updatedAt, occurredAt, or timestamp); see each dataset's schema on the Datasets page.