Incremental sync (delta reads)

Status: v1, live on five streams — flights, tasks, waypoints, screens and sites. Additive — nothing on this page changes the behaviour of the existing page-based endpoints.

Every /api/v1 list endpoint pages with ?page / ?pageSize. That is fine for "show me a screen of flights" and useless for "what changed since last time": if a row is created between two page fetches every later page shifts, so a client either re-reads the whole collection or silently misses rows.

This page defines a second, opt-in read mode on the same URLs. You ask for changes since a cursor, you get back the changed rows, the deleted ids, and the next cursor. A poll that finds nothing costs one conditional request and a 304 with no body.

1. At a glance

GET /api/v1/flights?since=0&limit=100
Authorization: Bearer wf_at_…

{
  "data": [ /* full flight DTOs, same shape as the page-based route */ ],
  "deleted": [],
  "sync": { "cursor": "d2YxLmZsaWdodHM…", "hasMore": true, "changeCount": 100 }
}

Then, forever after:

GET /api/v1/flights?since=d2YxLmZsaWdodHM…
Authorization: Bearer wf_at_…
If-None-Match: W/"1.0.418-418.s1.9f2ab1c4"

HTTP/1.1 304 Not Modified
ETag: W/"1.0.418-418.s1.9f2ab1c4"

No body, no rows, no parse. That is the whole point.

2. Which endpoints

No new scopes for the mechanism itself: ?since is gated by exactly the scope the endpoint already requires.

waypoints sets a much smaller limit. Rows are not the same size across streams — a flight DTO is a few hundred bytes, a waypoint collection can carry ten thousand points — so the stream that knows its own row size sets its own cap. Ask for more and it clamps.

sites is global. Its cursors are issued to * rather than to an account, so they are interchangeable between pilots. That leaks nothing: the takeoff database is identical for everybody and is already served to anonymous visitors at /takeoffs. Every other stream is per-user, and its cursor is re-checked against the authenticated principal on every request.

Two endpoints that take ?since are NOT part of this contract. GET /api/v1/thermals?since= is an ISO 8601 build-time filter, because that resource is filtered by bbox and a filtered change feed cannot exist under the no-filters rule below. GET /api/v1/tasks/{id} refuses ?since outright. Passing a cursor to either is an error, not a slow path.

Adding a stream is a registry entry and a DTO, and this contract does not change when it happens — same parameters, same envelope, same statuses, same cursor rules. Build your client against the stream, not against flights.

3. Request

?since=<cursor> switches the endpoint into delta mode. It is the only switch; if it is absent you get today's { data, pagination } response, byte-identical to what you get now.

An out-of-range or unparseable limit is clamped to the default rather than refused — a client bug should slow a sync down, not stop it.

since=now exists for the case where a user connects an account but does not want their back-catalogue pulled down. You get {"data":[],"deleted":[],"sync":{…}} and a cursor that starts tracking from here.

Delta reads take no filters. No ?season, no ?country, no date range. A filtered delta has to represent "this row left your filter" as a deletion, which means the server must remember every client's filter, and a client that changes its filter silently diverges. If you need a subset, filter locally — the DTOs are small.

4. Response

{
  // Rows created or changed since your cursor. Full current state, not a patch.
  // Apply as upsert-by-id.
  "data": [ { "id": "6540…", /* …the same DTO the page-based route returns… */ } ],

  // Rows that no longer exist. Apply as delete-by-id.
  "deleted": [
    { "id": "6512…", "observedAt": "2026-08-20T09:12:44.000Z" }
  ],

  "sync": {
    "cursor": "d2YxLmZsaWdodHM…",  // store this; send it as ?since next time
    "hasMore": false,              // true → fetch again immediately with the new cursor
    "changeCount": 3               // data.length + deleted.length, for logging
  }
}

data carries the complete current DTO, not a diff. A client applying it needs no knowledge of what changed.

observedAt is when the server noticed the row was gone, not when it was deleted. It can lag the real deletion by up to one scan interval (§9). It is there for your logs; do not build logic on it.

There is deliberately no generatedAt / serverTime in the body. The body is a pure function of (stream, epoch, cursor range, schema version). Wall-clock is in the standard Date header. This is what makes the ETag able to actually match — see §7.

Ordering and compaction

• Rows within one response are compacted per id: a flight edited five times since your cursor appears once, with its final state.
• An id never appears in both data and deleted in the same response. A flight created and then deleted since your cursor either does not appear at all (we never observed it existing) or appears only in deleted — never as an upsert you then have to undo.
• Across a hasMore run, an id may appear in more than one response. Apply responses in order; last occurrence wins. Ids are MongoDB ObjectIds and are never reused, so a delete is always terminal for that id.
• Within one response, data and deleted are order-independent.

5. The cursor

Opaque. Not a timestamp. You must treat it as a string, store it verbatim, and never parse, compare, or construct one.

We are telling you what it encodes so you can reason about lifetime, not so you can read it:

Wire format: base64url("1|<stream>|<owner>|<epoch>|<position>[|<seedkey>]|<checksum>"). Roughly 60–90 characters. The checksum catches truncation and copy-paste damage so a mangled cursor is a clean 400, not a wrong answer. It is not a signature: we do not defend against a client forging its own cursor, because the worst it can do is ask for its own data from a different position.

What it survives

The position is an integer maintained in the database. It is not derived from a clock, a process, a build id, or a secret. Therefore:

• Deploys, restarts, rollbacks: safe. Cursors issued before a deploy work after it.
• Clock changes, NTP steps, backdated rows: safe. No timestamp is involved. This is the main reason we did not use a timestamp cursor: a flight uploaded today for a flight flown last year would be invisible to a ?since=<timestamp> client forever.
• Ties: impossible. Every change gets its own position; two changes never share one.
• Database restore from backup: not safe. That is an epoch bump (§6) and every client is told to resync.

What it does not mean

A cursor is not a timestamp and positions are not dense. Gaps in the underlying sequence are normal and carry no meaning. Do not infer "3 changes happened" from two cursors — use changeCount.

6. Status codes

Error bodies use the house shape:

{
  "error": "cursor_expired",
  "error_description": "This cursor is older than the 90-day change-retention window.",
  "recovery": "seed"
}

recovery is "seed" on both 400 and 410 and exists so a client can branch on one field. error is the stable string; error_description is prose and may change.

We never answer an unusable cursor with a full collection and a 200. That is the failure mode nobody can detect: your sync appears to work, costs a full download every time, and hides a real bug. An unusable cursor is always a distinct status.

410 vs 304 — the distinction you asked for

They are opposite instructions and they can never be confused:

• 304 — empty body, ETag present. Your local copy is correct and current. Do nothing.
• 410 — JSON body, error: "cursor_expired". Your local copy may be missing deletions. Reseed and reconcile.

After a 410: how to reconcile

Deletions are the reason a stale cursor is dangerous — an addition you missed arrives in the seed, but a deletion you missed leaves a row in your database forever. So:

1. GET …?since=0, page through with the returned cursors until hasMore is false.
2. Track every id the seed returned.
3. Delete every local row for that stream whose id the seed did not return.
4. Store the final cursor.

Do not skip step 3. It is the whole reason the reseed exists.

Retention

Change history is retained for 90 days. Only deletions age out — a row that still exists is always representable, however old. So 410 means precisely: "you have been away longer than 90 days and a row may have been deleted while you were gone." A client that syncs even monthly will never see one.

7. ETag and If-None-Match

It is per-cursor-position, not per-collection

The ETag identifies the response to this request from this position, not the state of the collection. It is a function of:

(stream, epoch, from-position, to-position, DTO schema version, normalised query)

Store it next to the cursor, as one unit. They advance together or not at all. A per-collection validator would also work for a single client, but it cannot express "the collection moved and your delta is still empty", and two clients at different positions cannot share one. Since you keep cursor and validator in the same store and advance them in the same transaction, per-position is the shape that matches what you already built.

The limit and any future query parameters are folded into the validator, so changing limit between polls correctly produces a 200, not a stale 304.

The DTO schema version is in there too: if we add a field to the flight payload, your validator stops matching and you get a 200 with the fuller rows, even though your position did not move.

It is weak

ETag: W/"1.0.418-418.s1.9f2ab1c4"

W/ is correct and deliberate. We compute the validator from metadata, not by hashing the response bytes, so we assert semantic equivalence, not octet equivalence. RFC 9110 §13.1.2 specifies that If-None-Match uses the weak comparison function, so a weak validator is fully usable for exactly this.

This is also why there is no generatedAt in the body (§4). A per-request-varying field would force a strong validator to never match and the whole mechanism would quietly do nothing — which is the trap you flagged, and we avoided it in the body shape rather than papering over it with W/. Both defences are in place.

Rules

• Send If-None-Match with the stored validator on every delta poll.
• 304 responses carry ETag and no body.
• Cache-Control: private, no-cache — storable, must revalidate. Not no-store; no-store on a conditional endpoint invites clients to drop the validator.
• Vary: Authorization.
• ETag is in Access-Control-Expose-Headers, so browser-based clients can read it.

8. Client algorithm

This is the whole loop. It is written for a store that keeps {cursor, etag} per stream and advances them inside the transaction that applies the rows.

sync(stream):
  state = store.load(stream)              # {cursor, etag} or null

  if state == null:
      cursor = "0"; etag = null           # seed
  else:
      cursor = state.cursor; etag = state.etag

  loop:
      res = GET /api/v1/<stream>?since=<cursor>&limit=100
            with If-None-Match: <etag> if etag != null

      if res.status == 304:  return                      # done, nothing changed
      if res.status == 410:  reseed(stream); return      # §6
      if res.status == 400:  log_error(); reseed(stream); return
      if res.status == 429:  sleep(Retry-After); continue

      transaction:                                        # one transaction
          upsert all res.data by id
          delete all res.deleted by id
          store.save(stream, res.sync.cursor, res.ETag)   # same transaction

      cursor = res.sync.cursor
      etag   = res.ETag
      if not res.sync.hasMore: return

Two properties this gives you, both intentional:

• A crash mid-sync cannot skip data. The cursor only advances in the transaction that applied the rows. Re-running resumes from the last applied page.
• Re-delivery is always safe. Every apply is upsert-by-id or delete-by-id, so a retried or duplicated page is a no-op.

9. Guarantees, and what we do not promise

We guarantee:

• No missed rows. A row created, changed, or deleted between two delta fetches is delivered by a later fetch. Position allocation does not depend on a clock, so a concurrent write cannot land "behind" a cursor you already hold.
• At-least-once delivery. You may receive a row you already have. You will not fail to receive one.
• Deletions are delivered, for 90 days.
• A cursor is either honoured or refused with a distinct status. Never silently reinterpreted.

We do not promise:

• Exactly-once. Apply idempotently.
• Sub-second freshness. Change detection runs when a delta is requested (§10), so freshness is bounded by your own poll interval.
• That every change produces a delta. Changes are compacted; five edits between two polls are one row.
• Cross-stream consistency. Streams advance independently. When a second stream exists, a flight can arrive referencing a site your local copy does not have yet. Join defensively.
• Deltas on joined data. The flight DTO embeds its takeoff site for convenience, but the change signal is derived from the flight document alone. Renaming a takeoff site does not re-emit its flights. Until a sites stream exists, treat embedded site metadata as a snapshot taken when the flight last changed, not as a mirror of the site. This is the layering, not an oversight: hashing the join would re-emit every flight at a site each time somebody fixes its spelling.

10. How change detection works (so you can reason about it)

You do not need this to write a client, but you asked to be able to reason about the cursor, and the mechanism explains the freshness bound.

We-Fly has ~28 code paths that write flights, and no database trigger or change stream available on the deployment. Instrumenting all of them would mean the delta feed silently stops seeing a resource the first time someone adds a write path and forgets — which is the one failure this feature cannot have.

So change detection is not instrumented at the write sites. Instead, when a delta is requested, the server compares a stored fingerprint index against the live collection and materialises whatever differs:

1. Read the fingerprint index for the stream (one small row per entity: id, hash, position).
2. Read the live rows and hash the DTO-relevant fields.
3. Anything whose hash changed, or is absent from the index, becomes an upsert. Anything in the index but absent from live becomes a tombstone.
4. Allocate a contiguous block of positions atomically, one per change.
5. Write the index rows forward-only (a row's position never decreases).

Consequences worth knowing:

• Any writer is caught, including migration scripts, admin tools, and a DBA in mongosh. There is no "we forgot to instrument that path".
• Partial failure is self-healing. The diff is recomputed from durable state each time, so a half-finished materialisation is simply finished by the next one. Nothing is lost, at worst a change is delivered late.
• Freshness is poll-bounded. We materialise on every delta request, so a change is visible to your next poll. There is no background job to fall behind.
• Compaction is structural, not a post-processing step. The index holds exactly one row per entity, so "five edits collapse to one row" is a property of the storage rather than something we remember to do.
• A poll that finds nothing allocates nothing. The position does not move, the validator does not change, and you get your 304. That is load-bearing: if an idle poll bumped the position, the ETag would change every time and 304 would never fire.

11. Worked example

GET /api/v1/flights?since=0&limit=2

{ "data": [ {"id":"a1…"}, {"id":"a2…"} ],
  "deleted": [],
  "sync": { "cursor": "…p=0,k=a2…", "hasMore": true, "changeCount": 2 } }

GET /api/v1/flights?since=…p=0,k=a2…&limit=2

{ "data": [ {"id":"a3…"} ], "deleted": [],
  "sync": { "cursor": "…p=0", "hasMore": false, "changeCount": 1 } }

Seed complete, position 0. The pilot lands, uploads a flight, and deletes an old one.

GET /api/v1/flights?since=…p=0
If-None-Match: W/"1.0.0-0.s1.9f2ab1c4"

{ "data": [ {"id":"a4…"} ],
  "deleted": [ {"id":"a1…","observedAt":"2026-08-20T17:41:02.000Z"} ],
  "sync": { "cursor": "…p=2", "hasMore": false, "changeCount": 2 } }

ETag: W/"1.0.0-2.s1.9f2ab1c4"

Nothing happens for a week. Every poll from here:

GET /api/v1/flights?since=…p=2
If-None-Match: W/"1.0.2-2.s1.9f2ab1c4"

HTTP/1.1 304 Not Modified
ETag: W/"1.0.2-2.s1.9f2ab1c4"

12. Rate limits

Delta reads use the same per-application tier buckets as every other /api/v1 call (see Rate limits). A 304 still consumes one unit — it is cheap for your bytes, not free for our database. Polling every 60 s sits comfortably inside the standard tier.

A 304 poll costs us a stream read, a projected scan of your flights, and a range query that returns nothing. It never builds a DTO. That is why it is worth sending If-None-Match even though the request itself is not free.