Caching

The response cache is stored in SynapseOrch's blob store — SYNAPSE_BLOB_DIR/<tenant>/cache/, per tenant, or under <S3_PREFIX>/<tenant>/cache/ in the bucket when object storage is configured. Entries have a 1-hour TTL by default. The prompt cache is held by the provider. Tool results are never cached: every tool call runs.

Prompt cache

The prompt cache piggybacks on each LLM provider's native caching mechanism. SynapseOrch injects cache_control markers (Anthropic, Bedrock) or relies on automatic prefix caching (OpenAI, DeepSeek, Gemini) so that the long, stable parts of your system prompt — agent instructions, tool definitions, RAG context — are reused across turns instead of being re-tokenised every call.

What gets cached: the stable prefix of the system prompt plus the tools block. SynapseOrch splits the system prompt at an internal volatile separator so turn-changing values (current time, turn budget, recent RAG matches) sit after the cache point — they update freely without invalidating the cached prefix.

Minimum prompt length: about 4000 characters (~1000 tokens). Below that, providers ignore the cache marker, so SynapseOrch skips it to avoid paying the write surcharge for ineligible writes.

Cost shape:

In practice this lands at 50–80% cost reduction on long, repeated conversations, after a one-time write surcharge on the first turn.

Viewing cache savings

Go to Runs → Usage. Its Cache analytics section, shown once there is any cache activity, surfaces:

• Saved — the dollar amount SynapseOrch would have paid without the cache.
• Cache reads / Cache writes — the prefix tokens reused and written.
• Response hits — LLM calls served from the response cache (see below), out of all calls, with the short-circuit rate.
• A per-model table — Cache reads, Resp hits and Saved for each model, so you can see which models are actually using the cache.
• Top workflow runs by savings — the runs that saved most.

Disabling the prompt cache

The prompt cache is controlled by the prompt_cache_enabled setting, which is on by default. There is no toggle in the UI, so turn it off through the settings API, from a script signed in as Calling the settings API from a script shows:

curl -b synapse.cookies -X POST http://localhost:3000/api/settings \
  -H "Content-Type: application/json" \
  -d '{"prompt_cache_enabled": false}'

The setting is read per call, so it takes effect on the next turn.

Response cache

The response cache short-circuits an LLM call entirely when an identical (or near-identical) request has been made before. There are two layers:

Exact match — SHA256 of (model, system_prompt, messages, tools). O(1) lookup. If the hash matches, the cached completion is returned without contacting the provider.

Semantic match — embeds the user's message with the Embedding model set in Settings → Models and compares it with earlier entries for the same step, chat model and system prompt, in a ChromaDB collection per step and embedding model. A high similarity threshold (0.95 by default) keeps hits limited to near-identical prompts. With no embedding model set, or when it cannot embed (a missing key, a server that is down), the step uses exact match only, and the backend log says why once: semantic cache off for step '…': no Embedding Model is set — choose one in Settings → Models. Changing the embedding model starts each step's semantic cache afresh.

Important: opt-in per step

The response cache is off by default, and only two workflow step types use it:

Other step types either make no model call (Extract JSON, Transform, Print, If / Else, Switch) or don't use this cache (Tool).

To enable it on an LLM or Evaluator step, open the workflow, click the step, and tick Cache LLM responses under Caching in its panel; Semantic match (fuzzy) and TTL (seconds) appear once it's on, and Similarity ≥ once semantic match is on. The panel shows the same controls on most other step types, where they have no effect; an Agent step's panel says the response cache is off for agent steps. See Step Types → Common fields.

Viewing response cache activity

Usage shows Response hits under Cache analytics, and the per-model table's Resp hits column counts each model's hits, so you can tell where the cache is benefiting.

On-disk layout

The response cache lives in the blob store, under each tenant's cache/ folder. On a self-hosted install:

SYNAPSE_BLOB_DIR/default/cache/
└── responses_exact/         # Exact-match LLM responses

SYNAPSE_BLOB_DIR defaults to ~/.synapse/var/blobs in an installed SynapseOrch and backend/var/blobs in a source checkout; it is /data/blobs in the single-container Docker image and /blobs in Docker Compose. The semantic response cache is a set of ChromaDB collections, one per step and embedding model, kept with the rest of ChromaDB's data in the state directory.

You can clear the exact-match cache entirely by deleting its folder. The usage summary also reports per-namespace entry counts and sizes, as disk_stats.

When caching does NOT help

• Streaming tool-heavy chats where the user message changes every turn — exact match won't hit, and the prompt cache only helps on the stable system prefix.
• Very short system prompts (under ~1000 tokens) — below the provider minimum, so prompt cache is skipped.
• Agent steps in workflow — these intentionally bypass the response cache.

If your hit rate is unexpectedly low, check the per-model table under Usage → Cache analytics: if Cache reads is zero for a model you're using heavily, your system prompt is likely too short, or the model's provider doesn't support cache markers in the way SynapseOrch expects (file a bug if so).