Skip to main content

First data product tutorial

You will define a minimal data product in the Spine language, validate it with the Dataspine CLI, and optional JSON golden tests using a conventional ingests/ / expected/ layout. This tutorial does not use dataspine.yaml, embedded Rust, or SQL—only .spine sources.

Prerequisites​

Step 1: Project layout​

Create a directory and a spine-src folder:

mkdir -p hello-product/spine-src
cd hello-product

Step 2: Define the product in main.spine​

Create spine-src/main.spine with a com.example namespace:

namespace com.example {
    Struct Event {
        message: String,
    }

    Ingest EventIngest {
        name: "EventIngest",
        type: Event,
    }

    Outlet EventOutlet {
        name: "EventOutlet",
        type: Event,
    }

    flow = EventIngest.map(e -> e).publish(EventOutlet)
}

Wire name strings must stay consistent with JSON test files in later steps.

Step 3: Validate with the CLI​

dataspine check --source-dir ./spine-src --root-namespace com.example

Fix any diagnostics, then produce an artifact:

mkdir -p build
dataspine compile \
  --source-dir ./spine-src \
  --root-namespace com.example \
  --out ./build/artifact.json

See dataspine check and dataspine compile for --in-file, --lib, and auth options.

Step 4: Optional golden-test style data​

To exercise the product with ordered JSON fixtures, add files whose names encode step order and ingest/outlet names:

hello-product/
├── spine-src/
│   └── main.spine
├── ingests/
│   └── 001-EventIngest.json
└── expected/
    └── 001-EventOutlet.json

001-EventIngest.json (payload for the first ingest step):

{
  "message": "hello world"
}

001-EventOutlet.json (outlet payload expected after that step):

{
  "message": "hello world"
}

The numeric prefix sorts steps; the part after - must match name in the Spine declaration (EventIngest, EventOutlet). Run these comparisons in your own CI so regressions are caught when the flow changes.

Step 5: Add a filter​

Replace the flow line with a filtered pipeline. First extend the struct and names if you want to keep tutorial copy-paste small; here we only change flow and add fields to illustrate numeric filtering:

namespace com.example {
    Struct Event {
        category: String,
        value: Integer32,
    }

    Ingest EventIngest {
        name: "EventIngest",
        type: Event,
    }

    Outlet EventOutlet {
        name: "EventOutlet",
        type: Event,
    }

    flow = EventIngest.filter(e -> e.value > 10).publish(EventOutlet)
}

Example multi-step JSON for that filter:

  • 001-EventIngest.json: {"category":"low","value":5} — filtered out
  • 002-EventIngest.json: {"category":"medium","value":15} — passes
  • 002-EventOutlet.json: {"category":"medium","value":15} — outlet after step 002

Re-run dataspine check after each edit.

What you learned​

  • Declaring types, Ingest / Outlet, and a flow in Spine
  • Checking and compiling with dataspine and --root-namespace aligned to your namespace
  • How ingests / expected JSON files are named for golden-style checks

Further reading​