Skip to main content

Data product lifecycle

This guide follows the lifecycle of a Dataspine data product (runtime unit) built from Spine (.spine) sources. For how data products and Spine relate, read Data products and Spine; for runtime components see Ingests, Processor, Outlets, APIs, and Client SDKs. Control plane (lifecycle APIs, API-first): Control plane. For Spine as a language see Spine language. CLI entry points: dataspine check and dataspine compile; Data Product Manifest (usually dataspine.json): Data Product Manifest. For VS Code / Cursor and the spinec LSP, see Dev tooling: VS Code and Cursor.

Overview​

Anatomy of a Spine data product​

A minimal product in one namespace usually contains:

PieceRole
namespaceDotted root name for all types in this file (must match the CLI --root-namespace prefix for your project layout).
Struct (and other types)Payload shapes for messages.
Ingest / OutletDeclarations with a wire name and type; inputs and outputs attach here.
flowStream pipeline: start from an ingest, apply operators (.map, .filter, innerJoin, …), end with .publish(...).
Optional APIse.g. KeyValueApi<K, V> published with KeyValueApiCommand.*.

Example minimal pipeline:

namespace com.example.events {
    Struct Event {
        message: String,
    }

    Ingest EventIngest {
        name: "EventIngest",
        type: Event,
    }

    Outlet EventOutlet {
        name: "EventOutlet",
        type: Event,
    }

    flow = EventIngest.map(e -> e).publish(EventOutlet)
}

Phase 1: Design​

  • Scope: Which events or entities enter the product, what transformations apply, and what consumers need (streams, key-value state, joins).
  • Model in Spine: Sketch structs and ingests/outlets first, then the flow. For joins, design matching key types (JoinTrigger, key extractors). For JSON-heavy payloads, consider com.dataspine.json.Value.
  • Libraries: If you rely on external types (timestamps, org-specific models), plan type libraries and import usage; pass them to check / compile with --lib as needed.

Phase 2: Prepare test data​

To see how golden JSON tests fit local work and CI / org validation, read Test framework. Golden-style tests often use a repeatable folder layout you can mirror locally:

your-case/
├── main.spine
├── ingests/
│   ├── 001-EventIngest.json
│   └── 002-EventIngest.json
└── expected/
    ├── 001-EventOutlet.json
    └── 002-EventOutlet.json

Rules:

  • Numeric prefix (001, 002, …): lexicographic order defines the order of ingest steps; after each step, expected outlet snapshots are checked.
  • Filename suffix after the prefix must match the Spine name on the corresponding Ingest or Outlet (e.g. 001-EventIngest.json ↔ name: "EventIngest").

Example ingest and expected outlet for a filter flow = EventIngest.filter(e -> e.value > 10).publish(EventOutlet): first message has value: 5 and is dropped; the second passes through.

001-EventIngest.json:

{
  "category": "low",
  "value": 5
}

002-EventOutlet.json (outlet state after applying ingests through step 002):

{
  "category": "medium",
  "value": 15
}

Phase 3: Implement​

  • Add .spine files under a source directory (often spine-src/). Split files by domain if needed; the CLI loads all .spine files recursively when using --source-dir. For a full repository sketch (dataspine.json, tests, build output), see Filesystem layout.
  • Implement the flow: e.g. .filter, chained .map, innerJoin, indexBy, com.dataspine.interleave, then .publish.
  • Keep wire names on ingests/outlets stable—they are the contract for producers and consumers.
  • Reuse patterns from the Data product pipelines chapter (joins, map chains, key-value APIs, and more).

Richer example with filtering:

namespace com.example.filtered_events {
    Struct Event {
        category: String,
        value: Integer32,
    }

    Ingest EventIngest {
        name: "EventIngest",
        type: Event,
    }

    Outlet EventOutlet {
        name: "EventOutlet",
        type: Event,
    }

    flow = EventIngest.filter(e -> e.value > 10).publish(EventOutlet)
}

Phase 4: Test​

Local validation

dataspine check --source-dir ./spine-src --root-namespace com.example.myproduct

Artifact build

dataspine compile \
  --source-dir ./spine-src \
  --root-namespace com.example.myproduct \
  --out ./build/artifact.json

For JSON-based golden tests, run ingests in order and compare outlet payloads to expected/*.json, and wire the same check into your build or org validation pipeline when it can run the golden suite (see Test framework). Automate the same checks in your CI so behaviour stays pinned when the flow changes.

Phase 5: Deploy​

Deploy steps depend on your environment (build artifact upload, runtime wiring, credentials). Start from a compiled artifact (dataspine compile --out …) and your organization’s deployment process. Use the same --root-namespace and type libraries as in CI.

Phase 6: Monitor and iterate​

  • Monitor consumer lag, error rates, and schema drift against your Spine structs.
  • Iterate: version structs (@version where applicable), extend flow with new operators or ingests, add ingest/outlet names only with compatibility in mind, and extend golden tests under ingests/ / expected/ before changing behavior.

See also​