Scale Architecture

You run at least one main instance — the one with the UI. Workers and extra API replicas are optional and independently scalable. All coordination goes through Redis and Postgres, so no process holds exclusive state.

The three roles

Main instance — synapseorchai/synapse-ai

The main instance is your SynapseOrch install — the combined image that runs the UI and the API server. In scale mode:

• It accepts /api/v2 requests and puts workflow runs and agent chat turns on the Redis queue instead of executing them
• It reads run events from Redis Streams and forwards them to SSE clients
• It still runs everything that is not /api/v2 itself — the UI's runs, Chat, schedules and /api/v1
• For a run a worker ran, it queues the work back to the fleet rather than executing it: an answer to a human step, a Resume from failure, and a dead-letter retry
• It hosts Settings → Scale: connections, the Workers list, recent runs, the live queue and failed jobs
• It runs a reaper that marks a worker offline once it has gone more than 90 seconds without a heartbeat

It reads and writes the same Postgres store as the workers, so there is nothing to push or sync: a definition saved in the UI is what the next job reads.

Workers — synapseorchai/synapse-ai-worker

Workers are the execution tier. Each worker process:

• Takes jobs from the Redis queue with ARQ — workflow runs, resumes after human input, and agent chat turns
• Reads the workflow, its agents and tools, and the tenant's settings from Postgres at the start of each job
• Runs the workflow engine: LLM calls, tool calls, MCP servers, human steps
• Publishes events to a Redis Stream as the run progresses, for SSE subscribers
• Registers itself in the workers table on startup and sends a heartbeat every 30 seconds
• Serves GET /health on port 9000

Workers keep nothing between jobs. Any worker can handle any job, so you scale by starting more worker containers against the same Redis and Postgres.

API server replicas — synapseorchai/synapse-ai-api-server

The API server image runs the FastAPI backend without the UI. Use it when the API tier itself becomes a bottleneck:

• The same V2 API surface as the main instance — it enqueues jobs, reads Postgres, streams SSE from Redis
• No management UI — put a load balancer in front of the main instance and the replicas
• Stateless: any replica can serve any SSE stream for any run

Most deployments do not need dedicated API replicas. Add them only when the API tier (not the workers) is saturated.

Component diagram

Redis

Redis is the coordination backbone.

1. Job queue (ARQ)

There is one queue, a Redis sorted set that every producer writes to and every worker reads from:

synapse:orchestrations

POST /api/v2/orchestrations/{id}/run calls enqueue_job("run_orchestration_job", …), with the run id as the job id. Workers run ARQ's consumer loop and execute up to WORKER_CONCURRENCY jobs at once per process. Jobs carry their tenant, so any worker can run any tenant's job.

2. Event streams (SSE delivery)

Each run and each V2 chat session gets its own Redis Stream. Workers append events as they execute; the API reads the stream and forwards events to SSE clients.

synapse:run:{run_id}:events
synapse:chat:{session_id}:events

Events expire after PUBSUB_EVENT_TTL (default one hour). A client that reconnects with the Last-Event-ID header gets every event after that stream ID. A run's stream keeps its latest 10,000 events; a chat session's, 5,000. GET /api/v2/orchestrations/runs/{run_id}/events returns the stored events as JSON.

3. Cancellation signals

POST /api/v2/orchestrations/runs/{run_id}/cancel sets a cancellation key (one-hour expiry):

synapse:cancel:{run_id}

A worker running that job checks for the key every 2 seconds and cancels the step in progress — it does not wait for the step to finish. The stream receives an orchestration_error event ("Cancelled"), then done, and the run's status becomes cancelled. Chat turns use synapse:cancel:chat:{session_id}. The key outlives the run by an hour, so that a resume queued just before the cancel dies on arrival; replaying the run withdraws it first, unless a resume of that run is still on the queue.

An answer to a human step is not held in Redis: it travels to the worker as an argument of the resume job.

Postgres

Postgres is SynapseOrch's store — the same database the main instance uses for everything. Workers read definitions from it and write execution state to it. The tables that matter for scale mode:

Request lifecycle

1. Client enqueues the run

POST /api/v2/orchestrations/{orch_id}/run
Authorization: Bearer sk-syn-...
Content-Type: application/json

{"message": "Write a competitive analysis of Redis vs Kafka"}

The API:

1. Validates the API key

2. Checks the per-tenant rate limit (429), the tenant's queued-run quota (429, 1,000 queued, running or paused runs by default) and the global queue depth (503)

3. Generates a run_id — run_<12 hex characters>_<milliseconds>

4. Creates a row in orchestration_runs with status = queued

5. Enqueues run_orchestration_job

6. Returns immediately:

   {
     "run_id": "run_3f9a1c2b7d4e_1780499750067",
     "status": "queued",
     "stream_url": "/api/v2/orchestrations/runs/run_3f9a1c2b7d4e_1780499750067/stream",
     "status_url": "/api/v2/orchestrations/runs/run_3f9a1c2b7d4e_1780499750067/status"
   }

2. Client subscribes to the SSE stream

GET /api/v2/orchestrations/runs/{run_id}/stream
Authorization: Bearer sk-syn-...

The API reads the Redis Stream synapse:run:{run_id}:events and yields SSE lines until the run finishes or the client disconnects.

3. Worker picks up the job

A worker takes the job from the queue, marks the run running with its worker id, and publishes:

{"type": "worker_picked_up", "worker_id": "worker-prod-01-1a2b3c4d", "run_id": "run_3f9a1c2b7d4e_1780499750067"}

4. Worker executes steps

For each step, events like these:

{"type": "step_start", "orch_step_id": "step_research", "step_name": "Research Topic", "step_type": "agent"}
{"type": "thinking", "message": "Analyzing your request..."}
{"type": "tool_execution", "tool_name": "scrape_url", "args": {"url": "https://redis.io"}}
{"type": "step_complete", "orch_step_id": "step_research", "step_name": "Research Topic", "duration_seconds": 8.2}

5. Run completes

{"type": "orchestration_complete", "run_id": "run_3f9a1c2b7d4e_1780499750067", "status": "completed", "final_state": {"analysis": "..."}}
{"type": "done"}

The run row is updated with its final status, end time, cost and tokens. The API sees done and closes the SSE response with a final stream_complete event.

6. Webhook delivery (optional)

If the run request included webhook_url, the worker POSTs the final result there once the run ends (not when it pauses). The request carries an X-Synapse-Timestamp header; if you also sent webhook_secret, it carries X-Synapse-Signature-V2: sha256=<hex>, an HMAC-SHA256 of "{X-Synapse-Timestamp}.{body}" (refuse one more than five minutes old), and X-Synapse-Signature, the HMAC of the body alone, for older receivers. Each attempt is signed afresh. Delivery is tried up to 3 times.

Human input — pause and resume

When a workflow reaches a Human step, the run pauses until someone answers. It does not time out.

1. The worker publishes the human input request:

{
  "type": "human_input_required",
  "orch_step_id": "step_approve",
  "prompt": "Please review the draft and decide whether to publish.",
  "fields": [
    {"name": "decision", "type": "string", "label": "Decision"},
    {"name": "notes", "type": "string", "label": "Editor notes"}
  ],
  "run_id": "run_3f9a1c2b7d4e_1780499750067"
}

2. It publishes {"type": "paused"}, checkpoints the run to Postgres, and the job ends.

3. The SSE connection stays open — the client receives paused, but the stream does not close.

4. The human's answer is submitted:

POST /api/v2/orchestrations/runs/{run_id}/resume
Authorization: Bearer sk-syn-...
Content-Type: application/json

{"response": {"decision": "approve", "notes": "Excellent — publish immediately."}}

A string response is wrapped as {"response": "…"} on the way in; a Human step with no fields stores it as the text. A run that is already completed, failed or cancelled returns 409.

5. The API enqueues resume_orchestration_job with the response as its argument, and returns 202.

6. A worker — not necessarily the same one — restores the run from Postgres, writes the answer to the Human step's output key and human_response, and continues from the step after the Human step. Events flow into the same stream.

Several human steps in one workflow repeat this cycle.

An answer given on the canvas, or in the messaging channel a Human step names, takes the same path: the API queues resume_orchestration_job for the fleet rather than resuming the run itself.

Worker lifecycle

Startup

When a worker starts (python worker_main.py):

1. Starts the health server in a background thread on WORKER_HEALTH_PORT (default 9000)
2. Opens its Postgres connection from SCALE_POSTGRES_URL — with no connection pool (NullPool) when PGBOUNCER_MODE=1
3. Starts the built-in tool servers that are the same for every tenant. A tenant's own MCP servers are started when a job needs them and kept in a pool (up to 32 tenants' sets, closed after 15 minutes idle). Local (stdio) MCP servers do not run in scale mode, and neither does browser automation
4. Inserts or updates its row in workers with status = online, its hostname and health address
5. Starts the heartbeat, every 30 seconds
6. Starts consuming the queue

No model keys are loaded at startup: each job reads its tenant's settings from Postgres, and keys are never put into the process environment.

Job processing

Up to WORKER_CONCURRENCY jobs run at once in a worker process, as asyncio coroutines rather than threads:

• Blocking work reduces effective concurrency — CPU-heavy tools and local models tie up the event loop
• For I/O-heavy workloads (LLM API calls, web requests, database queries), WORKER_CONCURRENCY=10 to 20 is a reasonable start
• For CPU-heavy workloads, keep concurrency lower and run more worker instances

Each job has a timeout, WORKER_JOB_TIMEOUT (default 3600 seconds).

Heartbeat

Every 30 seconds each worker updates workers.last_heartbeat and workers.active_jobs and sets its status to online. The main instance's reaper, which runs every 60 seconds, marks any online worker whose last heartbeat is older than 90 seconds as offline.

Graceful shutdown

On SIGTERM (docker stop, pod termination):

1. The worker stops taking jobs and gives the ones it is running WORKER_JOB_COMPLETION_WAIT seconds (45 by default) to finish, then cancels the rest
2. A cancelled run goes back on the queue and resumes from its checkpoint on the next worker that takes it: steps that had finished do not run again. After WORKER_MAX_RETRIES tries the run is failed and put in the dead-letter queue instead, and Retry there resumes it
3. A chat reply that was cut off is not run again: the conversation says it was interrupted, and the person sends the message again if they still want it
4. The worker marks itself offline in workers and closes its MCP sessions and database connections

The step a run was in when it was cancelled runs again, including its model and tool calls. Keep the orchestrator's grace period (stop_grace_period, terminationGracePeriodSeconds) longer than WORKER_JOB_COMPLETION_WAIT, and take a step with side effects into account before stopping workers.

Failures and retries

The API tier holds no execution state: run state lives in Postgres and Redis. Any API replica can serve any SSE stream, and restarting the main instance does not interrupt runs on workers — clients reconnect and replay with Last-Event-ID. On startup the API marks runs that were running as failed, but skips any run whose worker is still online and has beaten within 90 seconds, so a restart does not fail the work a live worker is in the middle of.

Workers hold a job's in-memory state while it runs:

• A worker process is killed (crash, SIGKILL, lost node) — the job stays marked as in progress until that marker expires, WORKER_JOB_TIMEOUT plus 10 seconds later (by default about an hour). Then another worker picks it up and runs it again from the start.
• The job raises an error — the run is marked failed and an orchestration_error event is published. It is not retried.
• A step fails inside the engine — the run fails in the ordinary way, as it would outside scale mode.

Failed jobs that reach the dead-letter queue can be inspected and retried under Failed jobs (DLQ) in Settings → Scale. A retry there, and Resume from failure on a failed or cancelled run that a worker ran, are queued for the workers; both resume the run from its checkpoint.