Telemetry (OpenTelemetry)
Optional OpenTelemetry export to your own collector — off by default. Nothing is sent to us.
Nothing is sent to us. Synapse has no analytics and no usage reporting. It can export traces and metrics to an OpenTelemetry collector you run — off by default, and only when you turn it on.
Turn it on
The Docker images already carry the telemetry packages. Anywhere else, install them into the environment Synapse runs from:
# A one-command, npm or pip install
synapse upgrade --with telemetry
# A source checkout
pip install -r backend/requirements-telemetry.txtsynapse upgrade --with telemetry keeps the version you run, and the packages stay through every later synapse upgrade.
Then set one Synapse switch and restart:
| Setting | Effect |
|---|---|
OTLP_ENDPOINT=http://collector:4317 | On, exporting to that collector |
| The Scale tab's OTLP Endpoint | The same, saved in settings (applies on restart) |
SYNAPSE_TELEMETRY=otlp | On, exporting to OTEL_EXPORTER_OTLP_ENDPOINT |
SYNAPSE_TELEMETRY=off | Off, whatever else is set |
OTEL_SDK_DISABLED=true | Off, whatever else is set |
The standard OTEL_* variables only tune an export one of those switches started: the
protocol (OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf; gRPC by default), headers
(OTEL_EXPORTER_OTLP_HEADERS, for a hosted collector's API key), the sampler
(OTEL_TRACES_SAMPLER), the service name and resource attributes. A machine that already
exports another application's telemetry with OTEL_EXPORTER_OTLP_ENDPOINT set will not
start sending Synapse's there. There is no default endpoint, not even localhost.
METRICS_TOKEN serves Prometheus metrics at GET /api/v2/metrics (a Bearer token) — pull
only; nothing is sent. It needs the same packages.
What you get
- Traces: one span per HTTP request (named by its route template, never its path), per run and per step, per chat turn and agent loop, per model call (tokens and cost, retries as events), per tool call (refused or not, and why as a code), per compaction; database and Redis calls, and outbound HTTP calls. In scale mode a queued job continues the trace of the request that queued it.
- Metrics: runs and chat turns by how they ended, run/step/turn/job durations, model calls, latency and tokens by model, tool calls by outcome, webhook deliveries, dead-lettered jobs, the queue's depth.
- Logs:
SYNAPSE_LOG_FORMAT=jsonwrites JSON lines carrying the trace and span ids and the run, session and request ids.x-request-idis kept (or set to the trace id) and returned on every response.
What is exported — and what never is
A span carries ids, timings, counts, cost, step types and tool names — never a prompt, an
answer, a tool's arguments or result, a file name, a URL, a query string, a database
statement or an error message. Every span is rebuilt from the list below before it leaves
the process; anything else — including whatever an instrumentation library would have
recorded — is dropped. An outbound call's host is kept only for a service on your own
network; any other host is reported as external. An exception keeps its type, never its
message.
| Attribute | What it holds |
|---|---|
synapse.org_id | the workspace (tenant) id; Cloud only — a self-hosted install has one |
synapse.run_id | a workflow run's id |
synapse.root_run_id | the id of the run a nested run was started by |
synapse.session_id | a chat conversation's id |
synapse.agent_id | the agent's id |
synapse.orchestration_id | the workflow's id |
synapse.step.id | a step's id within its workflow |
synapse.step.type | a step's type: agent, llm, tool, parallel, loop, human, … |
synapse.status | how a run, step or turn ended: completed, failed, paused, cancelled |
synapse.trigger | what started a run: run, resume, resume_failed |
synapse.cost_usd | the model cost of a call or a run, in US dollars |
synapse.steps | how many steps a run executed |
synapse.turn | the agent loop's turn number |
synapse.tool.name | the tool's name, as the agent sees it |
synapse.tool.category | where the tool runs: builtin, mcp, http, python, delegate, subtask, builder |
synapse.tool.refused | true when policy, an allowlist or a person refused the call |
synapse.tool.refusal_kind | why, as a code: not_allowed, policy, person, memory, db_allowlist |
synapse.job.name | the queued job's function name (scale mode) |
synapse.job.id | the queued job's id (scale mode) |
synapse.job.try | which attempt at the job this is (scale mode) |
synapse.request_id | the request's x-request-id, or the trace id when none was sent |
synapse.schedule_id | the schedule that started the work |
synapse.surface | where a chat came from: app, api, widget, channel |
synapse.model.bucket | the model, or own-key:<provider> for a model name a user typed |
synapse.tokens.cache_read | prompt tokens served from the provider's cache |
synapse.tokens.cache_write | prompt tokens written to the provider's cache |
synapse.retry.attempt | a model call's retry number (on a retry event) |
synapse.retry.reason | why a model call was retried, as a code (on a retry event) |
synapse.compaction.messages_before | messages in the context before a compaction |
synapse.compaction.messages_after | messages in the context after a compaction |
synapse.cancelled | true when the work was stopped rather than finished |
gen_ai.provider.name | the model provider: openai, anthropic, gemini, bedrock, … |
gen_ai.system | the model provider (the older name for gen_ai.provider.name) |
gen_ai.operation.name | chat |
gen_ai.request.model | the model's name as configured (metrics use the model bucket instead) |
gen_ai.usage.input_tokens | prompt tokens |
gen_ai.usage.output_tokens | completion tokens |
gen_ai.response.finish_reasons | why the model stopped: stop, tool_calls, length, … |
gen_ai.request.max_tokens | the response cap sent to the model |
gen_ai.request.temperature | the sampling temperature sent to the model |
gen_ai.request.top_p | the nucleus-sampling value sent to the model |
http.request.method | GET, POST, … |
http.response.status_code | the HTTP status |
http.route | the route's template, e.g. /api/orchestrations/{orch_id} — never the path |
url.scheme | http or https |
network.protocol.version | the HTTP version |
server.address | an internal service's host name, or 'external' for any other host |
server.port | an internal service's port |
error.type | an error's class name or HTTP status — never its message |
exception.type | an exception's class name (on an exception event) — never its message |
exception.escaped | whether the exception left the span |
db.system | postgresql, sqlite or redis |
db.system.name | postgresql, sqlite or redis (the newer name) |
db.operation | SELECT, INSERT, … or a Redis command — never the statement |
db.operation.name | SELECT, INSERT, … or a Redis command (the newer name) |
db.redis.database_index | the Redis database number |
messaging.system | arq |
messaging.operation.type | send or process |
messaging.destination.name | the queue's name |
rpc.system | aws-api |
rpc.service | the AWS service, e.g. BedrockRuntime |
rpc.method | the AWS operation, e.g. Converse |
cloud.region | the AWS region |
aws.region | the AWS region (the older name) |
aws.request_id | the AWS request id |
retry_attempts | how many times the AWS SDK retried |
Metric dimensions are low-cardinality only — a step type, a status, a job name, a tool
category, a model bucket (a user-typed model name is reported as own-key:<provider>). Never
a run, session or organisation id.
Try it locally
docker run -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/jaeger:2.21.0
OTLP_ENDPOINT=http://localhost:4317 synapse startOpen http://localhost:16686 and search service synapse-api (or synapse-worker), tag
synapse.run_id=<a run's id>.
Environment Variables
Infrastructure configuration — the database, storage, ports and secrets. Everything a request can change lives in Settings instead.
Caching
SynapseOrch has two independent caches that together cut LLM cost without changing what your agents do. Most of it runs automatically — you only need to know where to look in the UI to see savings, and which knob to flip when you want stronger caching on a specific orchestration step.