Next data service and direct ingestion

The Next dashboard (ui/dashboard) does not read raw runner blobs per widget. It uses a single in-browser Heat data service (HeatDataService in heat-client) that ingests a structured JSON envelope, indexes channels by realm, and serves samples to widgets (timelines, maps, composable charts, KPI panels, and so on) through React hooks.

That envelope is called direct ingestion: data is already in the channel/realm shape the UI expects, without legacy per-widget JSON keys like { "Map": { ... } }.

Why this format exists

Legacy dashboards (ui/legacy) often received one fixed JSON object per widget type from the v1-era pipeline. Each component had its own contract (Map, Timeline, and similar). Adding a second map or a new chart meant new top-level keys and fragile coupling between runners and UI.

The v2 / Next model separates concerns:

Layer	Responsibility
Upstream processing	Runners or platform nodes produce analytics results
`$heat-dataservice`	A shared, versioned description of what data exists (channels, time series, KPIs, events)
Layout	A separate JSON document describing how to render (which component, which channel ids, titles, chart options)

Benefits:

Multiple widgets can bind the same channel (two maps, timeline + composable chart on one series).
Playback and seeking use a common timeMs axis across series, events, and ranges where applicable.
Realms let one dashboard compare runs (trainees, files, scenarios) without duplicating channel ids in layout.
Validation is centralized (C# on the platform path, TypeScript in the app) so integrators get predictable errors.

Platform and runner docs for producing payloads: dashboard-v2 upstream contract, system-tabular-to-dataservice, system-arbex-js.

End-to-end flow

A dashboard-v2 or heat-system-next-dimension node persists a JSON root (often including $heat-dataservice and optional dashboard_users).
The v2 API returns that payload plus layout for the session dimension.
HeatDataService.ingestSessionData detects $heat-dataservice, validates it, and loads channels into the DataStore (keyed by realm:channelId:name).
Each layout column lists channel ids (and widget-specific options). Components sample data at the current playback time.

What is a direct ingestion payload?

A direct ingestion payload is the object under the $heat-dataservice key on the session/dimension JSON root.

Minimal wrapper:


{
  "$heat-dataservice": {
    "version": "1.0",
    "groups": [{ "id": "metrics", "name": "Metrics" }],
    "realms": [
      {
        "name": "default",
        "channels": [
          {
            "id": "heart-rate",
            "name": "Heart rate",
            "groupId": "metrics",
            "shape": "series",
            "data": [
              { "timeMs": 0, "value": 72 },
              { "timeMs": 1000, "value": 75 }
            ]
          }
        ]
      }
    ]
  },
  "dashboard_users": []
}

The UI checks isHeatDirectIngestionData(root) (presence of $heat-dataservice). Types live in direct-ingestion-types.ts.

Direct ingestion contrasts with legacy automatic mapping inside heat-client, where older session shapes are converted via JMESPath-style rules in data-mapping-config.ts. New Next work should always target $heat-dataservice.

Realms

A realm is a namespace for channel data on one dashboard dimension.

Stored as realms[].name in the payload (for example default, pilot-a, file-metrics-csv).
The in-memory store keys samples as realm:channelId:name (see DataStore.getKey).
Layout includes realms: ["default", "pilot-a", ...] and per-widget configuration.defaultRealm / showRealmFilterDropdown.

Why realms exist: they answer “which run of data am I looking at?” without changing layout channel ids. The same layout can show heart-rate for trainee A or trainee B by switching realm. Typical uses:

One CSV upload per realm (realmPerFile on system-tabular-to-dataservice)
A column in tabular data (realmColumn)
Explicit multi-realm payloads from system-arbex-js heat.dataservice builder (spec.realm)

Important: id + name in realm A is not the same data as the same id + name in realm B. Widgets must pass the selected realm into hooks (useDataServiceData, ComposableChartDS, and so on).

Soft cap: platform normalization recommends at most 50 realms per payload; prefer more channels or separate dimensions when cardinality is high.

Groups

Groups (groups[].id, groups[].name) are UI organisation for channel pickers and registry entries. Every channel must reference a valid groupId.

Groups do not partition data the way realms do. They are labels (for example ops, metrics, biometrics) so authors and operators can find channels in complex dashboards.

Channels

A channel is one logical dataset inside a realm: a single id, name, groupId, shape, and data.

Field	Role
`id`	Stable identifier used in layout `configuration.channels` and timeline `channelId`
`name`	Display name (also part of the internal store key)
`groupId`	Links to `groups[].id`
`shape`	Determines `data` structure and how widgets consume it
`data`	Payload for that shape (required, never null)
`metadata`	Optional hints (`units`, `channelType`, column definitions for KPI widgets, and so on)

Channel id must be unique within a realm. Layout references channels by id (and optional URI forms like realm:channelId:name in advanced composable chart bindings).

Channel shapes

`shape`	Meaning	Typical `data`	Widget examples
`series`	Time-varying samples	`[{ timeMs, value }]`	TimelineChart, ComposableChart area/scatter, playback
`timestamps`	Labelled instants	`[{ timeMs, annotation }]`	Timeline markers
`value`	Non-time-series or structured snapshot	`{ value: <any JSON> }`	StatsSummary, BarChart, arbitrary KPIs
`events`	Boolean events over time	`[{ timeMs, occurred }]`	Event lanes, ComposableChart events_lane
`ranges`	Intervals / phases	`[{ startTimeMs, endTimeMs, durationMs }]`	Range lanes, phase bars

Not every shape supports playback interpolation. value channels are usually static for the session; series drives the shared timeline clock.

Additional shapes (flightpath, mapdisplay, responseTime) exist for specialised widgets; the platform contract for system-utils nodes standardises on the five shapes above.

Layout vs data

Layout (layoutConfiguration on the dimension) is stored beside $heat-dataservice, not inside each channel.

Layout	Data (`$heat-dataservice`)
Which React component (`MapDisplay`, `ComposableChart`, …)	What numbers and structures exist
`configuration.channels: ["heart-rate"]`	Channel `id: "heart-rate"` with `shape` and `data`
Titles, tooltips, chart heights, `composableChartItem` slices	Optional `metadata` on channels

This split lets operators change presentation without re-running analytics, and re-run analytics without rewriting layout (as long as channel ids stay stable).

See Next dashboard components for per-widget layout fields.

How the frontend ingests payload

Simplified sequence inside HeatDataService:

Load session/dimension JSON from the v2 API.
If $heat-dataservice is present, run validateDirectIngestionPayload.
convertDirectToChannelData flattens realms[].channels[] into ChannelData[] with realm set on each row.
DataStore.ingest indexes by realm:channelId:name and extracts timestamps for seeking.
ChannelRegistry registers groups for discovery/selection.
Components subscribe via samplers tied to TimelineClock per realm.

Invalid payloads surface a descriptive string from describeDirectIngestionValidationFailure in the client (mirrors server-side checks in HeatSystemNextDimensionPayload on the platform).

Producing payloads in HEAT

Approach	When to use
`dashboard-v2` on dashboard-utils	Integrator runners; you supply `$heat-dataservice` + layout in node config
`heat-system-dataservice-envelope`	Declarative mapping from a parent JSON root (`valueFromBindings`, `seriesFromPath`)
`system-tabular-to-dataservice`	Many CSV uploads on one `input-node`
`system-arbex-js` + `heat.dataservice.buildRoot()`	Custom logic and multi-shape payloads in JavaScript

All paths normalize to the same canonical realms structure before the Next UI loads the dimension.

Next dashboard components , widget list and layout wiring
dashboard-v2 upstream contract , field-by-field producer contract
ComposableChart , multi-slice layout on channels
Anatomy of a dashboard , where dashboards sit in the session pipeline
In-repo types: ui/dashboard/src/lib/data-service/data-lib/heat-client/