Workflows

Workflows are the execution backbone of SolutionEngine.

A workflow is a node graph that defines how incoming data is transformed, validated, enriched, and delivered.

Workflows are event-driven. They start when a trigger fires, run as isolated executions, and stop when there are no more nodes to process.

The Workflow Message (Data Object) That Flows Between Nodes

In SolutionEngine, nodes don’t pass “raw values” to each other. They pass a workflow message (a JSON object). The data object is simply the data field inside that message.

This is the canonical message shape created at the start of an execution:

{
  "data": {},
  "projectId": "proj_...",
  "workflowId": "wf_...",
  "nodeId": "node_...",
  "executionId": "wf_..._20260101_120000_000001",
  "timestamp": "2026-01-01T12:00:00.000Z",
  "createdAt": "2026-01-01T12:00:00.000Z",
  "metadata": {},
  "environment": {},
  "status": "pending",
  "errors": [],
  "warnings": []
}

What each part is for

• data (the “data object”): the payload you shape as the workflow runs. This is what most nodes read from and write to.
• metadata: optional, node-specific or execution-specific context (safe for non-secret diagnostic info).
• environment: environment-scoped attributes and secrets (primarily used on Edge). It is expected to look like:
  • environment.attribute → non-secret configuration (URLs, feature flags, IDs, etc.)
  • environment.secret → secret values (API keys, tokens, passwords, etc.)
• status, errors, warnings: execution state that nodes can append to. Routing decisions can follow status-specific ports (success, fail, true, false, etc.).

How node output is stored: outputKey

Most nodes return an updated message. The runtime then decides where the node’s output should live inside data using the node’s outputKey config:

• If outputKey is empty (or set to data): the node output replaces the entire data object.
• If outputKey is set (for example apiResponse, result, or data.result): the runtime injects the node output into message.data at that path and keeps the rest of data intact.

Important details:

• outputKey paths support dot notation and array indices (for example result.items[0].label).
• A leading data. prefix is optional. outputKey: "result" and outputKey: "data.result" both write to the same place: message.data.result.
• Some trigger/output nodes intentionally pass the message through without outputKey routing (for example the trigger nodes, Log, and FramePreview).

How templates read from the message (data, attributes, secrets)

Many node configuration fields support {{ ... }} placeholders that resolve against the full workflow message, not just data.

Common examples:

• {{data.userId}} → read from the current data object
• {{metadata.someKey}} → read from message metadata
• {{environment.attribute.apiBaseUrl}} → read a non-secret environment attribute
• {{environment.secret.apiKey}} → read a secret (Edge)

Shorthand aliases also work:

• {{attribute.apiBaseUrl}} is the same as {{environment.attribute.apiBaseUrl}}
• {{secret.apiKey}} is the same as {{environment.secret.apiKey}}

This is the recommended pattern for keeping workflows portable across environments: store URLs, IDs, and feature flags in attributes; store credentials in secrets; and reference them from node configs using templates.

What a Workflow Contains

Every workflow definition includes:

• Workflow metadata: name, project scope, status, version
• Node definitions: configuration for each processing step
• Edges: explicit paths between nodes
• Runtime references: models, buckets, datasources, and environment bindings
• Optional grouping: organizational grouping within a project

At runtime, this definition is loaded and executed in sequence according to node connections and branching conditions.

Lifecycle and States

A workflow can move through these operational states:

• stopped: Defined but not executing
• active: Deployed and ready to process triggers
• error: Entered a failure state due to runtime or configuration issues

Typical lifecycle:

1. Create workflow in a project
2. Build graph and configure nodes
3. Validate with manual or test triggers
4. Deploy to target environment
5. Monitor runs and logs
6. Update and redeploy

Triggering and Execution Model

A workflow execution starts from a trigger node. Common trigger patterns:

• Datasource trigger: external input (camera, MQTT, webhook, stream)
• Manual trigger: controlled test execution from editor
• Generator/time trigger: interval-based execution

Each trigger event creates an independent execution context.

Execution behavior:

• Data moves from node to node through edges
• Each node reads input, applies logic, emits output
• Branches execute based on node conditions
• Failures are recorded with node-level context
• Execution completes when the queue is drained

This model enables deterministic processing and predictable troubleshooting.

How to Connect Nodes in the Editor

Use this flow when designing graphs in the visual editor:

1. Place exactly one trigger node as the entry point.
2. Connect trigger output to validation or transformation nodes first.
3. Add logic nodes (Condition, Switch, Filter) before expensive inference calls.
4. Connect inference outputs to transformation nodes for normalization.
5. End each branch with delivery or persistence nodes.

Connection design rules:

• Keep one clear direction of flow from left to right.
• Avoid hidden coupling between distant branches.
• Keep branch depth manageable so failures are easy to trace.
• Use node descriptions to document assumptions at branch points.

If you need exact per-node configuration fields, continue in the Nodes section.

Node Categories in Practice

Workflows combine categories instead of using isolated node types.

Trigger and Ingestion

• Generator
• Datasource Trigger
• Manual Trigger

Logic and Control

• Condition
• Switch
• Iterator
• Delay/Gate controls

Transformation and Data Shaping

• Expression
• Filter
• Context/Variables
• Mapping and merge operations

AI and Inference

• Run Model
• Vision/model post-processing nodes
• AI-agent related nodes for LLM-driven flows

Network and Delivery

• HTTP Request/Response
• MQTT Publish
• Bucket save nodes (media, metadata, timeseries)
• Log and preview nodes

A production workflow usually combines all five layers.

Deployment to Environments

Deployment binds a workflow to an execution target.

Typical process:

1. Select target environment (cloud or edge)
2. Deploy workflow definition
3. Validate heartbeat and runtime status
4. Observe first execution logs

Deployment checks should confirm:

• Referenced models are available
• Datasource connectivity is valid
• Bucket permissions and paths are correct
• Required secrets/environment variables exist

Failure Handling and Recovery

For robust workflows, design for failure explicitly.

Recommended patterns:

• Guard conditions before expensive model calls
• Branch-specific fallback paths
• Retry logic for external HTTP/MQTT calls
• Dead-letter style persistence to bucket on failure
• Log key identifiers at critical nodes (request IDs, datasource IDs)

Operational recovery playbook:

1. Inspect failed node in execution logs
2. Validate input payload shape at previous node
3. Re-run with Manual Trigger using captured payload
4. Patch workflow and redeploy
5. Verify with a controlled test event

Workflow Design Standards

Use these standards to keep large graphs maintainable:

• Name nodes by intent, not by tool name
• Keep branch depth shallow where possible
• Separate validation, inference, and delivery blocks
• Prefer explicit transformation steps over hidden side effects
• Store intermediate artifacts needed for incident analysis
• Document assumptions in node descriptions

Example Production Pattern

Datasource Trigger
  -> Pre-Validation
  -> Run Model
  -> Confidence Filter
      -> Save Media (Bucket)
      -> Save Metadata (Bucket)
      -> HTTP Alert

Why this works:

• Early validation prevents noisy failures
• Confidence gating improves output quality
• Dual persistence supports analytics and audits
• External alerting creates real-time operational response

Related Pages

• Nodes
• Trigger & Ingestion Nodes
• Transformation Nodes
• Output & Delivery Nodes
• First Workflow