Skip to main content

gRPC

This page is the low-level gRPC reference: message shapes and protocol behavior for a deployed data product. For day-to-day integration, use the official client libraries or MCP instead of assembling these messages by hand.

gRPC is the primary path to the Dataspine data plane: HTTP/2, TLS, unary API calls, and bidirectional streaming for ingest and outlet. gRPC-Web is supported for environments that use the browser gRPC-Web stack against the same endpoint.

  • Auth: on every RPC, set metadata authorization: Bearer <jwt>. The server rejects missing or invalid tokens before your application logic runs. Long-lived streams can send a fresh token on the same connection (see Reauthentication) when the JWT rotates.
  • API vs transport errors: for ApiService.UnaryCall, business and validation outcomes are returned inside the ApiResponse message (see ApiStatusCode). gRPC UNAUTHENTICATED and INVALID_ARGUMENT are reserved for envelope problems (e.g. no token, malformed request wrapper), not for per-operation domain errors.
  • Payload encoding: the outer request and response use ApiRequest / ApiResponse with byte payloads; you choose serializers (binary v0 or JSON v0) in the request. Large bodies may be chunked on streaming RPCs; see Chunking.

ApiService​

Defined in api.proto. Single unary RPC:

service ApiService {
  rpc UnaryCall(ApiRequest) returns (ApiResponse) {}
}

ApiRequest​

  • api_operation_id — Uuid for the API operation (assigned when the data product is published).
  • correlation_id — optional Uuid, echoed in the response for tracing.
  • payload — bytes: serialized request body.
  • request_serializer / response_serializer — oneof; each is either Dataspine binary v0 or JSON v0. Only serializer_version = 0 is valid today. Both must be set; otherwise the server returns an invalid-argument style response at the gRPC layer or ApiResponse with INVALID_REQUEST depending on the failure mode.

ApiResponse​

  • status — ApiStatusCode.
  • rate_limiting — optional, when the operation exposes rate-limit metadata (retry_after_seconds, limit, remaining, reset).
  • request_duration_ms — server time spent on the call.
  • response — oneof: success_payload (bytes, when status == SUCCESS, encoded per response_serializer) or error_message (string).

ApiStatusCode​

  • SUCCESS = 0
  • CANCELLED = 1
  • INVALID_REQUEST = 2
  • UNAUTHENTICATED = 3
  • UNAUTHORIZED = 4
  • NOT_FOUND = 5
  • INTERNAL_SERVER_ERROR = 6
  • SERVICE_UNAVAILABLE = 7
  • CONFLICT = 8
  • TOO_MANY_REQUESTS = 9
  • DEADLINE_EXCEEDED = 10
  • API_OPERATION_NOT_DEFINED = 11
  • UNKNOWN_ERROR = 999

UnaryCall returns these in fields of ApiResponse, not as the top-level gRPC status for normal application outcomes. The gRPC call still completes with OK in those cases.

IngestService​

Defined in ingest.proto. All RPCs are streaming.

service IngestService {
  rpc AppendMessages(stream AppendMessageCommand) returns (stream AppendMessageResponse) {}
  rpc AppendPackagedMessages(PackagedAppendMessageCommand) returns (stream AppendMessageResponse) {}
  rpc FetchMessages(stream IngestFetchMessagesRequest) returns (stream IngestFetchMessagesResponse) {}
}

Append commands and responses​

AppendMessages is a long-lived bidirectional stream. The client sends a multiplexed stream of AppendMessageCommand (oneof), including: register ingest version, start/commit/abort transactions, append in transaction or batch, and reauthentication. client_handle and ingest_handle are client-chosen correlation IDs for that single stream; they are not global identifiers.

AppendPackagedMessages sends a batch of the same command messages in one request and still receives a response stream; useful for producers that do not want to own a request half-stream.

AppendMessageResponse is a oneof: transaction result, batch result, per-message result, register-version result, or command_rejected for envelope-level failures.

AppendMessageError includes an AppendMessageErrorKind and message text, e.g. INGEST_NOT_FOUND, TRANSACTION_NOT_FOUND, AUTHORIZATION_DENIED, AUTHENTICATION_EXPIRED, and others (see the proto for the full set).

FetchMessages (ingest)​

Reads previously appended data from a specific ingest version (a niche path; most readers use OutletService). The request stream sends start (with earliest / end / offset and serializer), poll for long-poll windows, and optional reauthenticate. The response stream returns message and optional continuation chunks, poll_completed, or error_code.

OutletService​

Defined in outlet.proto. One bidirectional streaming RPC:

service OutletService {
    rpc FetchMessages(stream OutletFetchMessagesRequest) returns (stream OutletFetchMessagesResponse) {}
}

Request: start (outlet version, start position, serializer), poll, reauthenticate.
Response: message and continuation chunks, poll_completed, error_code. Optional correlation_id on a message links back to the ingested event when the pipeline sets it.

Start position: the first start must use a supported position only—typically earliest (beginning of retained data), a server-defined tail (e.g. end-of-stream for new data only, if your product exposes that shape), or resume with an offset value that the platform previously returned on this outlet read. Arbitrary client-chosen offsets are not a supported way to seek; they can lead to skipped or duplicated changes and corrupt consumer state.

Poll behaviour: While a poll is active, new outlet data is sent on the response stream as it becomes available; it is not held until the end of the poll window. poll_completed only marks the end of that round so the client can issue the next poll.

Reauthentication on long-lived streams​

JWTs expire. On ingest and outlet streams you can send a reauthenticate message on the request stream with a new jwt_token so the server validates the new credential without tearing down the stream and resuming from the last position.

Chunking​

Ingest and outlet may split large payloads. When the primary message has has_next_chunk == true, follow with the listed chunk frames and concatenate payload bytes in order until has_next_chunk == false.

Application integration​

Language SDKs and MCP: Application integration.