Skip to main content

Outlets

Outlets are change propagation from the processor to consumers—not a raw append-only event log like a generic message broker or Kafka topic. The flow publishes into outlets you declare in Spine. What readers see is whether a key (or other slot your pipeline keys on) changed, not an unfiltered stream of every intermediate emission.

If the processing layer would emit the same logical value for a key as what is already current for that key, no new change is propagated on that outlet for that key. Outlets and ingests are both strictly typed in the data product model.

Outlet versions​

A single logical outlet can have multiple outlet versions over time, parallel to ingest versions. Each version pins a data model in Spine; compatibility and rollout follow the same platform rules as for ingests. See the Data Product Manifest, data product lifecycle, and control plane.

Per-key model: upsert, remove, and change messages​

A useful mental model is a key–value view with change propagation: the pipeline upserts or removes a value per key. The outlet layer detects whether a key’s value actually changed; only then does it emit a change message, conceptually in a shape like:

OutletChange — Added(…)
              Changed { previous, current }
              Removed { previous }

The exact Spine types and field names in your product follow your flow. Related aggregate patterns in the handbooks use a similar three-way split—see aggregateBy and AggregateChange in Data product pipelines and the runtime type docs. Added covers a key’s first appearance, Changed when the value differs from the last one, Removed when the key is dropped. If a recomputed value is identical to the current one, consumers do not see another outlet message for that key.

Change detection: hash-based and equality-based​

The platform can decide “did this key’s value change?” in one of two ways:

  • Hash-based — The previous and new values are hashed; a change is reported when the hashes differ. This mode is significantly more performant, but it relies on hash inequality only: if a new value is different but collides with the previous value’s hash (a rare but possible case), the outlet would not treat it as a new change and would not propagate it.

  • Equality-based — The new value is compared to the previous value in full, byte-for-byte (equivalent to comparing the full serialised or logical value). It is slower than hash-based, but it does not share the collision trade-off; any real value change is detected.

Replays​

The platform supports deterministic replays of history with the same processing semantics. If you replay without changing processing logic, the final state is the same and you do not get spurious outlet changes—keys that did not move are not re-notified. If you change the flow, only keys whose value actually differs from what was published before get a new change message; unchanged keys are not reflooded on replay.

Offsets and consumer position​

Outlets are offset-based so you can resume and reason about at-least-once delivery, but arbitrary or random seek offsets are not supported. A reader may start at the beginning of the retained stream, use a named start such as the end of the stream (only new changes from that point) where your client exposes it, or continue from the last offset the platform returned to you on that outlet (or the value you persisted from those messages). Do not pick a numeric offset yourself or reuse a value from another system as if it were a safe cursor—doing so can skip or re-deliver work and corrupt downstream state. See OutletService / StartOutletFetchMessagesRequest in the gRPC document.

Consumers are responsible for storing the last processed offset from platform messages—ideally in their own data store, in the same transaction as the state they apply from outlet events, so retries do not double-apply. Alternatively, a dedicated offset feature on the product side (when your control plane and version expose it) can keep offsets for you. Choose with your SRE and compliance requirements in mind.

Client SDKs and the poll loop​

Normal use is the Data Product client SDKs—TypeScript, Rust, Java, and Python are described under Application integration (check the per-language pages for your toolchain’s coverage). Lower-level gRPC is available to any runtime; the same endpoints and patterns apply—see the gRPC transport ( OutletService, FetchMessages, poll windows, and any wire encoding options for your stack).

Reading an outlet is a long-lived bidirectional stream: you start a read, then poll in a short cadence (commonly a new poll every couple of seconds). While a poll is in progress, new messages are delivered on the response stream as they arrive—not only when the poll window ends. When that poll round finishes (e.g. poll_completed on the wire), you start the next poll immediately so the connection stays live and both sides can detect liveness. The same style of bidi stream and poll rounds appears on the Ingest service side for comparison. Unary APIs cover request/response and query-style access where you model those surfaces on the product.

See also — Ingests · Processor · Correlation IDs · Data Product Manifest · Data product lifecycle · Data product pipelines · Transports: gRPC · Application integration