Semantic caching in production: what pgvector actually gets you

Sometime around the third month of running Supergate in production, I noticed a pattern in our logs: roughly 40% of the prompts hitting our LLM gateway were near-duplicates. Not identical — users would rephrase, swap a synonym, reorder clauses — but the semantic intent was the same. We were paying for the same answer, over and over.

The obvious response was to build a cache. The less obvious question was: what does a "cache hit" mean when the key is a paragraph of natural language?

Why traditional caching falls apart

A Redis lookup on the raw prompt string misses the moment someone writes "summarize this document" instead of "give me a summary of this document." Exact-match caching has zero tolerance for paraphrase. In a multi-tenant gateway processing prompts from dozens of applications, our hit rate was under 3%.

Semantic caching flips the model: embed the prompt into a vector and search for neighbors within a distance threshold. If a cached prompt is close enough, return the stored response and skip the upstream LLM call entirely.

The distance threshold problem

The single most important parameter in a semantic cache is the similarity threshold. Set it too loose and you serve stale, incorrect answers. Set it too tight and your hit rate drops to zero. In Supergate, we started with a cosine similarity threshold of 0.95 and tuned it down carefully per tenant. Some run factual Q&A where 0.92 is fine; others run creative generation where even 0.98 produces noticeable quality drift.

const findSemanticMatch = async (
  embedding: number[],
  tenantId: string,
  threshold: number
) => {
  const result = await db.execute(sql`
    SELECT response, 1 - (embedding <=> ${embedding}) AS similarity
    FROM cache_entries
    WHERE tenant_id = ${tenantId}
      AND 1 - (embedding <=> ${embedding}) >= ${threshold}
    ORDER BY embedding <=> ${embedding}
    LIMIT 1
  `);
  return result.rows[0] ?? null;
};

Why pgvector, not Pinecone

Our application already runs Postgres. Adding pgvector is an extension, not a new service. Cache writes, tenant metadata, and usage tracking all happen in the same transaction — no two-phase commits, no eventual consistency on your cache. With an HNSW index, pgvector handles single-digit-millisecond lookups on tables up to a few million rows, which is exactly the scale range where a semantic cache lives.

"The best infrastructure decision is often the one you don't have to make. If your existing database can do the job, adding a new one is not simplicity — it's accidental complexity with a vendor sticker on it."

Cache poisoning and the model-version trap

The most insidious bug in a semantic cache is entries that were correct when written but are no longer valid because the upstream model changed. We namespace every cache entry by tenant ID + model identifier + model version. When a model version changes, old entries naturally stop matching — a predictable cold start instead of a confusing quality regression.

The real win isn't latency

The real win is cost. A 35-40% hit rate on a well-tuned semantic cache cuts LLM API spend by a third. At scale, that's the difference between a sustainable product and a slow bleed. The second win is rate limit headroom — every cache hit is a request that doesn't count against your tokens-per-minute quota.