Skip to main content

Internal binary serialization (v1)

This document specifies Dataspine's internal binary encoding of values—the format implemented by the dataspine-serialization library and used by platform runtimes when they exchange binary payloads (as opposed to JSON) on the data plane.

Most developers and data engineers do not read this page. Typed clients and JSON on the wire hide these details. Use this reference when you maintain runtimes or generated code, build a low-level client, or need the exact byte layout for debugging or automation (including coding agents).

  • Scope: the layout of serialized values only (not public API shapes of any specific language).
  • Version: serializer_version = 0 (the only supported version at the time of writing).
  • Framing: this format defines value encodings only; transport-level framing (e.g. message length prefixes) is out of scope.

Core rules​

  • Endianness: all fixed-width integers are little-endian.
  • Floating point: f32/f64 are IEEE-754, written as their little-endian byte representation.
  • Not self-describing: types are not tagged on the wire; the decoder must know the expected type.

Primitive types​

  • u8 / i8: 1 byte
  • u16 / i16: 2 bytes LE
  • u32 / i32: 4 bytes LE
  • u64 / i64: 8 bytes LE
  • f32: 4 bytes LE
  • f64: 8 bytes LE
  • bool: 1 byte
    • 0x00 = false
    • 0x01 = true
    • other values are invalid
  • Uuid: 16 bytes (RFC 4122 byte order as returned by Uuid::as_bytes())
  • () / Void: empty (0 bytes)

Option<T>​

Layout:

  1. tag: u8
    • 0x00 = None
    • 0x01 = Some
  2. payload: only present for Some, encoded as T

Chunked byte sequences: Bytes and String​

Both Bytes and String use the same chunking structure:

  1. declared length: u64 (number of bytes)
  2. chunks: repeated:
    • chunk_len: u16
    • chunk_data: chunk_len bytes

Notes:

  • The declared length is intended as a capacity hint; decoding is terminated by the chunk rules below.

Termination rule​

Let MAX_CHUNK_SIZE = 65535.

Chunk parsing stops when either:

  • A chunk_len of 0 is encountered (explicit terminator), or
  • A chunk_len is strictly less than MAX_CHUNK_SIZE (implicit terminator).

Encoders only need to append an explicit 0u16 terminator when the last real chunk would have size exactly MAX_CHUNK_SIZE (including the empty case).

String encoding​

  • The chunk data is the UTF-8 bytes of the string.
  • The declared length is the byte length (not character count).

Lists: Vec<T> / LinkedList<T>​

Layout:

  1. declared element count: u64
  2. blocks: repeated:
    • block_len: u16
    • block_items: block_len items, each encoded as T

Notes:

  • The declared element count is intended as a capacity hint; decoding is terminated by the block rules below.

Termination rule​

Let MAX_BLOCK_LEN = 65535.

Block parsing stops when either:

  • A block_len of 0 is encountered (explicit terminator), or
  • A block_len is strictly less than MAX_BLOCK_LEN (implicit terminator).

Encoders only need to append an explicit 0u16 terminator when the last real block would have size exactly MAX_BLOCK_LEN (including the empty case).

Maps: IndexMap<K, V>​

Layout:

  1. declared entry count: u64
  2. blocks: repeated:
    • block_len: u8
    • block_entries: block_len repetitions of:
      • K (encoded as K)
      • V (encoded as V)

Notes:

  • The declared entry count is intended as a capacity hint; decoding is terminated by the block rules below.

Termination rule​

Let MAX_BLOCK_LEN = 255.

Block parsing stops when either:

  • A block_len of 0 is encountered (explicit terminator), or
  • A block_len is strictly less than MAX_BLOCK_LEN (implicit terminator).

Encoders only need to append an explicit 0u8 terminator when the last real block would have size exactly MAX_BLOCK_LEN (including the empty case).

Struct entities (derived)​

Structs are encoded as a simple concatenation of their fields, in declaration order:

field_0 || field_1 || ... || field_n

There are no field tags, no presence bits, and no length prefix for structs. Decoding requires knowing the exact struct schema.

Partial projections​

When a struct is serialized with a partial projection, the output is still just the concatenation of the projected fields (in declaration order). The resulting byte stream is not self-describing; decoding requires knowing the same projection out-of-band.

Enum entities (derived)​

Enums are encoded as:

  1. variant tag: u32 (LE)
  2. variant payload: the fields of the chosen variant, concatenated in declaration order (if any)

The tag value is determined by the enum schema (e.g. #[entity(tag = ...)]) and must match between encoder and decoder.

Partial projections​

For enum partial projections, the u32 variant tag is always written, followed by only the projected fields. As with structs, this is not self-describing without the projection.

Value<T> and references​

Value<T> is used to optionally represent a value either inline or as a reference to externally stored bytes.

When references are enabled​

If the serialization context has references enabled (allow_references = true), Value<T> is encoded as:

  1. kind: u8
    • 0x00 = inline T follows
    • 0x01 = Reference follows
  2. payload:
    • if kind == 0x00: encode T
    • if kind == 0x01: encode Reference (see below)

When references are disabled​

If references are disabled (allow_references = false), Value<T> is encoded exactly as T with no leading kind byte.

Reference layout​

Reference is encoded as a struct (field concatenation, declaration order):

  1. id: Uuid (16 bytes)
  2. serializer_version: u16
  3. offset: u64
  4. uses_references: bool
  5. payload_size: u64