Ingests

An ingest is how you load data into a data product. It is a typed contract: each ingest is declared in Spine with a message type and a wire-level name; at runtime, producers open an append stream that pipes events into the product’s processor according to the flow.

• Ingests are not REST — Ingestion is a long-lived, bidirectional gRPC path (Ingest service) suited to high throughput, chunking, and stream control. There is no REST append surface today, because the model is inherently streaming in both directions (commands, per-message and per-tx results, reauthentication, and so on). Prefer the Data Product client SDKs; the gRPC reference is for low-level and non-SDK runtimes.

Append path (conceptual) — Producers use an append stream; the processor runs the flow and connects ingests to the rest of the product.

Ingest versions

A single logical ingest can have multiple ingest versions over time. Each version pins a data model (the Spine type bound to that ingest for that version), so you can upgrade a data product and still accept traffic from older producers on an earlier version. Versioning and constraints (what the control plane allows in production) are a platform concern: see the Data Product Manifest, data product lifecycle, and control plane for how registered versions, rollout, and compatibility are managed.

You can read data back for a given ingest version (for example via the FetchMessages RPC on the Ingest service). That capability is for observability, debugging, and operational workflows—it is not the main way to consume a data product in an application, which is through the flow, outlets, and other product surfaces, not an ingest “read” API.

Same logical ingest, different versions — Older and newer producers can keep sending on their registered version while the processor runs one flow.

Ordering guarantee — Order is defined within one ingest version for the successful appends you choose to care about: the stream the processor sees for that version preserves append order. There is no ordering guarantee between different ingests (or, in the general case, between different version streams), and the processor does not define cross-ingest (or cross-version-stream) message order. If you need a total order across several independent feeds, the flow must express that (for example, by merging keyed streams, assigning explicit sequence fields, or using a single ordered ingest; see the processing docs when they cover multi-ingest composition).

Global order at processing time — How streams are interleaved into one data-product-wide order is explained in the processor and related processing material. In brief, a global order for the entire product is established in processing; for each message, that assigned position in the product order is permanent and does not change, including in replay and other re-execution paths.

Ingest-time vs product-wide order — There is no append-time guarantee between separate ingest streams, but the processor still builds one stable product order (see Product-wide order on the Processor page).

Between A and B there is no guaranteed interleaving at append time. G is fixed per message and does not change, including on replay.

Multiple ingests per product — A data product can declare several ingests. Use as many as you need for independent ordering domains: for example, events from two systems that are not required to be globally ordered with each other can go through separate ingests so each stream stays cheap and local.

Typical modeling pattern — In Spine, you describe payloads with structs (record types, each with a fixed set of fields) and enums (a choice of variants; each variant can carry different data from the others). If every message on a stream has the same shape, a single struct is usually enough. If you need several different kinds of event to stay in one total order for a single ingest version’s stream, the usual pattern is one enum and one variant for each event kind that ingest is allowed to carry. Use separate ingests when the traffic is really independent (different partners, no cross-ordering requirement) so you do not force unrelated producers through one big ordered pipe.

Modeling: one ordered sequence vs independent feeds

Two append modes: batch and transactional

Over the append stream, producers work in one of two main modes. Order is based on fully delivered logical messages; for chunked payloads, a message only takes its slot when the last chunk is in (see Chunked messages). In batch mode, the log of accepted work reflects that order, and failure is per message; in transactional mode, per-handle order of complete messages drives the merge as in the transactional section below up to commit.

Message size and how long one message can take

The product model does not cap how large a single message may be, nor how long it may take to finish delivering that one message. A message can be on the order of multiple gigabytes and can take hours to stream end to end. What you can sustain in practice still depends on client and server software, the network path, and your organization’s infrastructure and operational choices (for example timeouts and proxies).

Chunked messages: order is fixed when the last chunk completes

A single logical message can be streamed in several segments on the Ingest append stream (has_more_segments on TransactionMessage and BatchMessage). Order between finished messages in a batch or a transaction (per client_handle) is established when each message is fully received—last chunk in and reconstructed on the server—not when the first chunk was sent. If messages are interleaved on the wire (e.g. A1, B1, A2 …), their ingest order is the order in which the entire A and entire B are complete, not the order of the first segments.

Correlation IDs

Correlation is a cross-cutting concern. See Correlation IDs for the full picture. In short, one CorrelationIds value (a primary id and repeated secondary key–value entries on the Ingest append API) can be attached to each batch and transactional message; you can also use the model in connection with a root or sub-transaction in addition to per-message ids, so more than one correlation and more than one attachment level is expected in nontrivial traces and ops setups.

Concurrency: batches, transactions, and other producers

A transaction on one producer does not block other producers. Many open transactions can run in parallel, and on one long-lived append connection you can have several batches in flight and several open transactions at the same time, within the limits of your client and server configuration and normal resource pressure.

Batch (non-transactional) append

In batch (non-transactional) mode you append messages one after another, possibly grouped in batches for efficiency. The platform keeps order within your append stream for accepted work, but there is no “all or nothing” guarantee across a set of messages:

• If you send a sequence of messages A, B, C in order, it is possible that A is accepted, B fails, and C is accepted. A and C are then available to the product even though B did not make it. There is no automatic rollback of A when B fails.
• You receive a per-message delivery response (success, error, and enough detail to retry or dead-letter) so you know exactly which message failed.

Batch: per-message results along one append order

So “no guaranteed [group] delivery” here means: no cross-message atomic commit, not that the network is allowed to drop data silently. Failed messages are visible in the client API.

A long-lived ingest stream (for example a single gRPC bidi AppendMessages session) can carry batch appends. Depending on the client and registered ingest handles on that stream, you may be able to target more than one logical ingest (or version) in batch mode on the same connection, without conflating their ordering: each ingest / version you write to is still an independent ordering domain. The exact ergonomics (register, handles, and responses) are described in the gRPC ingest reference and your chosen SDK.

Transactional append

Transactional mode is for when you need atomicity across several messages: they are only made visible to the processor on commit of the root transaction, as one unit (all-or-nothing for that root; if a failure aborts the root, none of it is published as a single committed unit).

• Sub-transactions form a tree under a root. In the Ingest append API, each open scope has its own client_handle. Sub-transactions can be aborted on their own so a branch is rolled back while the root is still open. There is no product-level time limit for how long a root may stay open, subject to your own infrastructure and operational policies.

Commit and abort (each client_handle) — A CommitTransaction or AbortTransaction on the append stream always names a single client_handle (one open node in the tree). A sub-transaction is not auto-committed when a parent commits. Every enclosed sub-transaction in a tree must be sealed (commit or abort) before the root merge for that work can finish: the merger walks the root’s ordered list of payloads and branches and waits for each sub-transaction in that list in list order—so sibling branches are folded in the order Start child was ingested on the parent, not in the order a Commit for each sibling may happen on the wire. A CommitTransaction for the root can arrive on the stream while a sub-transaction is still open; the work to apply the root will not complete until later commands on the same bidi stream seal the outstanding sub-transactions the merger is still awaiting (each one is sealed with its own Commit or Abort).

Message order in a tree — The visible sequence is defined by the ingestion order of StartChild, TransactionMessage to the root client_handle, and TransactionMessage to each sub-transaction client_handle on the bidi append stream—not by the order of Commit or Abort. Appends to one client_handle are strict in append order. A Start child is recorded in the parent’s run at the point where that command is processed; a sub-transaction’s messages in its own append order (and nesting deeper branches recursively) form one contiguous block at that place; further parent appends, after that Start child in the parent’s command order, go after that block. Commit/abort only seals each scope after all appends for that node; it does not reorder the ingested structure.

Transactional: tree of work, atomic visibility on commit

Use batch when per-message ack and partial progress are acceptable. Use transactions when a set of changes must appear together or not at all.

Defining ingests in Spine

In Spine (.spine sources under your project’s namespace), you declare a wired ingest with the Ingest keyword, a Spine identifier, and a block that sets the wire name (what producers, tests, and clients use) and the type of each message. Those declarations are the type definition; the Data Product Manifest and CLI tie your tree to a build, and the First data product tutorial shows dataspine check and golden ingests/ / expected/ file naming (001-<name>.json matches the name: string).

Wired declarations use the Type Name { … } form (see Alternate singleton (ingests, outlets, config)); Ingest in the heading is that leading type. See the Spine language overview and Data product pipelines for flow, outlets, and standard helpers such as com.dataspine.interleave.

Example — one ingest, one struct

A single struct and a passthrough flow to an outlet (same pattern as the tutorial):

namespace com.example.orders {
    Struct Order { id: String, total: Integer32 }

    Ingest OrderIngest {
        name: "OrderIngest",
        type: Order,
    }

    Outlet OrderOutlet {
        name: "OrderOutlet",
        type: Order,
    }

    flow = OrderIngest.map(o -> o).publish(OrderOutlet)
}

Example — one ingest, Enum with multiple variants

An enum is the right container when a single ingest must accept several different shapes: add a variant for each event kind. Each variant carries the fields (or other payload) you define for that case. The flow can branch with match (not shown here) or pass through the whole DomainEvent in this minimal pipeline:

namespace com.example.crm {
    Enum DomainEvent {
        ContactCreated { id: String, email: String },
        NoteAdded { id: String, text: String }
    }

    Ingest CrmEvents {
        name: "CrmEvents",
        type: DomainEvent,
    }

    Outlet CrmEventsOutlet {
        name: "CrmEventsOutlet",
        type: DomainEvent,
    }

    flow = CrmEvents.map(e -> e).publish(CrmEventsOutlet)
}

Example — two ingests, same message type, merged in the flow

Multiple ingests (see the Multiple ingests per product section above) for independent sources often use the same struct as type for each ingest but different name values so clients and tests can target the right stream. Merging in product order is a flow concern: here com.dataspine.interleave interleaves two DataStream<T> values of the same T (see interleave and the data product pipelines table for this pattern):

namespace com.example.metrics {
    Struct MetricPoint { name: String, value: String }

    Ingest RegionA {
        name: "RegionA",
        type: MetricPoint,
    }

    Ingest RegionB {
        name: "RegionB",
        type: MetricPoint,
    }

    Outlet Merged {
        name: "Merged",
        type: MetricPoint,
    }

    flow = com.dataspine.interleave(RegionA, RegionB).publish(Merged)
}

Access: SDKs and gRPC

You will almost always go through a Data Product client SDK (TypeScript, Java, Rust, …) so authentication, reauthentication on the long-lived stream, chunking, and version registration are handled for you. Raw gRPC is available for runtimes and languages that do not have a first-party client yet.

See also — Processor (how multiple ingest streams are combined in the flow) · Correlation IDs · Data Product Manifest · Data product lifecycle · Spine: pipelines and ingests in source · Application integration · gRPC transport