Data products and Spine
Two ideas run through this site. They are orthogonal (different concerns) but interlinked (one cannot exist without the other in practice):
- A data product is a deployment and operations unit on the Dataspine platform. It accepts data through ingests, processes it in a stream processor, publishes outlets, exposes key–value and RPC APIs (unary today), and is consumed through Data Product client SDKs and the wire protocols behind them. Organizations own products; the control plane manages versions, access, and deployment. Authentication, observability, and applications (bundling) apply across every product.
- The Spine language (
.spinefiles) is the declarative DSL in which you define the types, Ingest / Outlet / KeyValue and flow for that data product. The compiler and the Data Product Manifest turn sources into what the control plane deploys to the data plane.
Mental model (similar in spirit to how TypeScript splits the language from your project): use Data products when you think about running and operating a deployed unit, and use Spine when you think about authoring definitions. The pages below are the data-product side; the Spine language handbook and specification live under Reference and Resources.
Runtime components
| Concept | Role |
|---|---|
| Ingests | Accept append streams and related producer traffic into the product. |
| Processor | Execute the Spine flow and stream logic between ingest and output surfaces. |
| Outlets | Propagate typed, per-key changes and offset-based reads—not a raw append-only event log. |
| APIs | Expose materialized key–value and RPC operations (fast reads, optional control-plane-style behavior); REST and MCP; subscriptions planned. |
| Data Product client SDKs | TypeScript, Java, Rust (with Python on the roadmap) for the supported client surface. |
| Test framework | Golden JSON tests for the flow: synthetic ingests, expected outlets/APIs, local or pipeline runs. |
| Triggers | Lambda nudge when bound outlets have new activity (control flow only; outlet reads stay poll-based for data). |
Data Product Manifest
The Data Product Manifest (conventionally dataspine.json) is the single source-control file that joins Spine build parameters and product metadata. Read the full page.
Repository layout
A typical Git tree places dataspine.json at the root, Spine sources under a directory you pass as --source-dir, optional type libraries and golden test folders, and build output from compile (often gitignored). Filesystem layout describes a conventional structure and monorepo patterns.
Control plane and lifecycle
The control plane is API-first; the official CLI and automation use the same public platform APIs (no separate proprietary operator protocol) as your own clients. A running product’s data plane (ingest, outlet, API) is what application code calls—also via documented APIs. See Control plane. Day-to-day design → deploy → operate is covered in the Data product lifecycle.
Cross-cutting concerns
| Topic | Why it matters |
|---|---|
| Authentication and authorization | Identity and policy across ingests, outlets, APIs, CLI, and the control plane. |
| Correlation IDs | CorrelationIds on the Ingest gRPC append stream; per-message and transaction-scoped metadata for traces and operations alongside observability. |
| Observability | Logs, metrics, and traces; roll-up at org level where supported. |
| Applications | Bundle multiple data products and attach CORS, extra authN policy, and a shared program context. |
Organizations and applications
Organizations group data products; the hierarchy supports quotas, authorization and ownership (often inherited down the org tree), and org-wide observability. Applications sit on top: they reference one or more products in an org and shape how they are exposed together.
Where to read next
| I want to... | Start here |
|---|---|
| A narrative on design, test, deploy, operate | Data product lifecycle |
| Write Spine and flow | Spine language handbook |
| Low-level grammar / lexer | Spine language (specification) in Resources |
| MCP and typed clients | Application integration |
For concrete CLI commands and (when published) Admin operations, the Data product platform and CLI sections in Reference will link to the right concept page here.
Operations such as creating a data product, bumping a version, managing access, or replaying events are control plane concerns: CLI and platform APIs, cross-linked to Control plane and the relevant component pages above.