Skip to main content

Run Observability

Per-node execution capture and viewing for workflow runs — the input, output, token usage, cost, timing, and errors for every node of a run.

  • Test runs stream per-node status over the engine-ws WebSocket to the Canvas Run Log panel.
  • Deployed/scheduled runs persist per-node records to node_executions (partitioned by month), read back via GET /runs/:id/nodes.
  • Runs link into a fan-out tree via runs.parent_run_id (GET /runs/:id/tree), so a docs-sync sweep's child proposals group under their orchestration run.

The authoritative registry lives in api/src/runs/registry.ts:

  • runs table (RLS-scoped per organization) stores run_id, org_id, workflow_id, deployment_id, substrate (temporal | in_memory), status, started_by, parent_run_id, target_doc.
  • resolveRun, registerRun, upsertRun (ON CONFLICT DO NOTHING), updateRunStatus and getRunTree are the only callers of this table.
  • node_executions (RANGE-partitioned on started_at) stores one row per (run_id, node_id, started_at) with input/output (capped via capForWire), tokens_in/out, cost_usd, duration_ms, error.
  • recordNodeExecution and getRunNodes are the write/read paths; both are best-effort when orgId is present.

engine/src/temporal/runWorkflow.ts wires the live path:

  • registerRunOnStart (best-effort) populates parent_run_id/target_doc for docs-sync children before the Temporal start.
  • Every node (gate or not) brackets its activity with recordNodeExecution; agent nodes use the 10-minute runNodeActivitySlow proxy.
  • Gate nodes additionally call gateOpened (writes pending_gates + ledger) and markRunStatus on terminal states; all three are wrapped in try/catch so projection failure never aborts the workflow.
  • getRunTree joins runs.parent_run_id with the latest provenance_ledger hil_decision and pending_gates to surface per-child gate state.

The narrow internal door is api/src/runs/internal-runs.controller.ts (/internal/runs):

  • POST :runId/status accepts only terminal statuses (completed/rejected/failed) from Temporal via markRunStatus (localhost + ENGINE_API_SECRET guard, identical to InternalGatesController).
  • POST :runId/register performs the idempotent upsertRun (ON CONFLICT DO NOTHING) for Schedule-started runs and docs-sync children (records parent_run_id/target_doc).
  • POST :runId/node performs the idempotent recordNodeExecution (RLS-scoped, started_at in PK for partition safety) after every node so the durable Run Log is populated for deployed runs.

The public api/src/runs/runs.controller.ts (/runs) exposes three endpoints for any authenticated org member (JWT + OrgContextInterceptor):

  • POST /runs/:runId/register records an in-memory run into the runs table (substrate='in_memory', status='running', started_by from the verified session). The call is idempotent (ON CONFLICT DO NOTHING) so resumes or retries never overwrite the original initiator; workflowId is required in the body. This path exists precisely so the engine can forward the caller's own JWT instead of asserting identity via an engine secret.
  • GET /runs/:id/tree returns the fan-out tree via getRunTree (404 if the parent run is absent or cross-org).
  • GET /runs/:id/nodes returns the durable node_executions rows via getRunNodes (empty array on missing data, never 404).

engine/src/temporal/runActivities.ts implements runNodeActivity (the Temporal twin of the in-memory handlers). It loads org vars + decrypted account tokens inside the activity (never in workflow args), applies the same makeRunVars/redact path, fetches the org URL allowlist only for guarded defIds, runs the shared stage1Handlers[role], measures wall-time durationMs, and returns RunNodeResult with events for lineage. Heartbeats are emitted every 15 s while the handler runs.

In engine/src/temporal/runWorkflow.ts, targetDoc for docs-sync children is derived inside the registerRunOnStart block by a deterministic JSON.parse(input.seed ?? "{}") check for source === "docs-sync" && typeof targetDoc === "string"; the value (or null) is forwarded to the activity so runs.target_doc records the exact document even after the gate closes.