Node status report v1 (statusDetails JSON)

Engineering reference for structured statusDetails on node instances. The node lastState enum (Processing, ProcessingFailed, etc.) is unchanged; this document describes the JSON payload stored in statusDetails.

Detection

If statusDetails parses as JSON and v === 1, treat as structured v1.
Otherwise treat as legacy plain text (for example Processing failed;).

Schema


{
  "v": 1,
  "phase": "decode",
  "message": "hydrate-protobuf: SimulationFrame decode",
  "elapsedMs": 45230,
  "memory": { "rssMb": 1842 },
  "progress": { "current": 5000, "total": 12000, "unit": "payloads" },
  "error": null
}

`phase` values

starting
fetch_inputs
decode
write_output
complete
failed

Failure `error` object


{
  "type": "ValueError",
  "message": "invalid protobuf",
  "location": "hydrate_protobuf.py:512"
}

One location frame only (no full stack). Encoders truncate text to fit the 500-character limit used by Arbex and Python runners.

Libraries

Runtime	Path
Python	`runtimes/python/heat_runtime/src/heatruntime/node_status.py`
C#	`runtimes/csharp/HEAT.Common/Runtime/HeatNodeStatusReport.cs`
Cluster Manager	`core/cluster-mgr/client/src/utils/heatNodeStatusReport.ts`

Python usage


from heatruntime.node_status import NodeStatusReporter
 
reporter = NodeStatusReporter(client, task)
reporter.report("fetch_inputs", "hydrate-protobuf: fetching inputs", force=True)
reporter.fail("hydrate-protobuf: no inputs available")

HeatRuntimeClient.fail_node(task, reason=...) wraps plain reasons as phase=failed JSON. Pass status_report= for full control.

UI

Cluster Manager parses v1 JSON and shows a one-line summary (phase, progress, elapsed, memory, message). Legacy strings render unchanged.

Incremental hydrate and json-flatten memory

Incremental hydrate (incrementalHydration: true, lineDelimitedJson: true) appends NDJSON to a temp file and uploads via submit_output_file. Watermark incrementalLastParentOutputId on the node output config tracks the highest parent NodeOutput id processed. When MD5 is unchanged, enableUploadDeduplication (default on) skips re-upload. Set HEAT_RUNTIME_HEATX_CACHE_MAX_MB to clear the Heatx slice cache between batches when RSS exceeds the threshold.
json-flatten NDJSON (jsonInputFormat: ndjson, inputMode: latest) streams the parent blob to disk, scans line-at-a-time, and writes CSV via submit_output_file so ~300MB logical NDJSON stays off the heap. Gzip parent outputs decompress via streaming gzip on the download path.
Scheduler guard (Core API): incremental nodes do not get a new runner task when no parent output id exceeds the watermark, even if ReferenceTime moved after upload.

Adoption

Pilot: hydrate-protobuf and hydrate-protobuf-v2 in core-utils. C# system-utils bulk writer prose progress is a follow-up.