Execution Reliability

Orchka executes workflows on a BullMQ + Redis engine. Every node runs as its own background job, and the engine is built so that a failure, a crash, or a restart loses as little work as possible. This page explains the reliability guarantees and the knobs you control.

Execution model in one paragraph

When a workflow is triggered, the engine creates an execution record, sorts the nodes, and enqueues the first node. Each node job runs the node, publishes a real-time event to Redis, and then chains the next node(s). Step history is streamed to Redis for the live UI and flushed to Postgres at checkpoints and on completion — so the database isn't hit on every step.

Retries

A failed node is retried automatically — but only when retrying is safe. The engine classifies each node type:

Some failures are never retried even for idempotent nodes — for example a missing credential or an invalid workflow. These are marked non-retryable so the job fails fast instead of burning three attempts on an error that can't succeed.

The HTTP Request node's class is chosen from its method: a GET is treated as safe to retry, a POST is not.

Dead-letter table

When a job fails terminally — retries exhausted, or a side-effecting node that never retries — it is recorded in a dead-letter table in Postgres instead of being discarded. Each entry keeps:

• the full job payload (so it can be re-run unchanged),
• which node, workflow, and execution it belonged to,
• the number of attempts made, and the error message + stack,
• a status: PENDING, REQUEUED, or DISCARDED.

This means a failed post or a flaky integration is never silently lost. Because the entries live in Postgres (not Redis), they are durable, queryable, and scoped to your organization — and surfaced through the API:

These are surfaced in the app under Executions → Failed Jobs — a tab next to the execution list, with a badge showing how many failures are still pending, and per-row Retry (with a confirmation) and Discard actions.

Durability: surviving a restart

Redis holds all in-flight job state and the live step history. If Redis were in-memory only, a restart would drop every running workflow. Orchka ships a Redis configured for append-only persistence:

# docker-compose.yml (repo root)
services:
  redis:
    image: redis:7-alpine
    command: ["redis-server", "--appendonly", "yes", "--appendfsync", "everysec"]
    volumes:
      - orchka-redis-data:/data

• appendonly yes — every change is written to the append-only file (AOF).
• appendfsync everysec — the AOF is fsync'd once per second, so a crash loses at most ~1 second of state, with negligible throughput cost.

Periodic checkpoints

Step history is flushed from Redis to Postgres every 25 nodes (and every 25 loop iterations), not only when the workflow finishes. Combined with AOF, a crash mid-run resumes with most of its history already persisted. Checkpointing is idempotent — re-running it never creates duplicate step rows.

No silent truncation on long runs

Step events go to a per-execution Redis stream holding up to 100,000 events — comfortably above any realistic workflow, including a large loop that emits a pair of events per body node per iteration. Streams also give server-assigned ordering, so history is never reassembled out of order.

Durable loops

Loop bodies execute as queued node jobs, one iteration at a time — not inline inside the loop's own job. That means:

• A crash resumes at the current iteration, not back at item 0. The cursor lives in the job payload, which Redis persists.
• A failure never repeats earlier side effects. Previously a retry of the loop job re-ran every completed iteration, so a failure at item 900 of 1000 could post 900 duplicates. Body nodes now retry individually, and the loop itself never retries as a unit.

• Body nodes get the full reliability stack: per-node retry classification, dead-letter parking, and step history (one row per node per iteration).
• Wait inside a loop no longer blocks a worker. It uses the same delayed job mechanism as a top-level Wait, so a long wait inside a loop costs no worker capacity.
• Nested loops and If/Switch inside a loop body work correctly — each iteration re-decides its own branch, and inner loops iterate per outer item.

Parallel branch execution

By default, Orchka executes one node at a time, depth-first within a branch — so independent branches of a workflow run sequentially. An opt-in mode runs independent branches concurrently.

Sequential (default):    1 → 2 → 4 → 3
Parallel  (opt-in):      1 → (2 and 3 at the same time) → 4

Enable it by setting an environment variable for the worker process (the worker reads apps/web/.env.local):

ORCHKA_PARALLEL_BRANCHES=1

When enabled:

• Every node whose dependencies are satisfied is enqueued immediately, so independent branches progress at the same time.
• Shared execution state (each node's output, which nodes have completed, branch decisions) lives in Redis rather than being threaded through job payloads, so a fan-in / Merge node correctly sees the output of every incoming branch.
• A node with multiple parents is enqueued exactly once (atomic claim), and the last branch to finish finalizes the run.

Loops behave identically in both modes: iterations are sequential by definition, so a loop body runs one node at a time either way.

What's still being hardened

The engine's roadmap (packages/workflow-engine/src/ROADMAP.md) tracks the remaining items — notably execution retention and partitioning for very high-volume workspaces.