Notes From Trying to Cache a Healthcare Chatbot

Background

The product is a maternal-health assistant built by ARTPARK in collaboration with ARMMAN : a multilingual, human-in-the-loop chatbot that answers ANMs' questions about high-risk pregnancies (HRPs) and antenatal care, and acts as a channel for continuous learning and on-the-job support. The chatbot supports Hindi, English, Telugu, Marathi, and code-mixed variants, in both text and voice.

As of May 2026, it's deployed across three Indian states (Uttar Pradesh, Telangana, and Maharashtra), and plans to expand to new states are underway. A few numbers to set the scale:

The Why?

We started with reducing the end-to-end latency as a goal and after several iterations on prompts, output schema, LLM providers, and model evaluations we were able to bring it down from ~25s to sub-10s. Although this did improve the real-time user experience, but it was still far from near-instant responses, and since most components of the pipeline still depended on slow API calls, the cost did not just disappear from being a major blocker in scaling our product.

On a closer look, we realized that a large fraction of the queries in production were near-duplicates of each other, i.e., they could be addressed with the same response. This warranted us to investigate strategies to reuse the responses from existing semantic replicas of new incoming queries.

A semantic cache would not only help reuse the response from similar past queries, but also help cut down the the average cost and latency.

It's worth pausing on what "similar" even means here. These are all the same clinical question — the kind of variation a single ANM produces without thinking:

Same intent, wildly different surface forms: spelling, transliteration, script-mixing, word order, even typos. A good embedding lands all of them near each other in vector space (ideally speaking):

1. The bet

• A cache hit will be near-instant, allowing us to skip multiple expensive API calls, and if it's miss, it'll just be a few hundred extra milliseconds to a pipeline that takes 5-8 seconds end-to-end - which would be impercetible when it comes to the user.
• The problem: in healthcare, a wrong hit isn't a small cost, it's a confidently incorrect clinical answer handed to a worker in the field. Incorrect or unsafe answers erode user trust - something that we highly value.

This is precisely what we explored: to devise a way to reuse pregenerated answers while ensuring they are safe to serve to the end users.

2. What even is a semantic cache?

Skip this section if it feels familiar.

For us to start with the idea of "semantically close" queries, we needed to loosen the exact-match requirement and the first instinct was to try embeddings. Embedding can be used as a rough proxy for the meaning of a piece of text; two texts that mean similar things tend to fall near each other in the embedding space.

So the simplest semantic cache could just embed the incoming question, finds the nearest question embedding it has already answered, and if the similarity score clears a threshold, it reuses that answer instead of using the default pipeline.

The appeal is that it catches paraphrases that an exact-string cache never would. And although, there are lexical techniques that allow fuzzy string matching, but in a multilingual and code-mix context they are too brittle to use.

That clustering is why embedding retrieval feels natural at first, but the problem is that clustering by broad meaning is not the same as preserving the details that decide whether an answer is safe to reuse.

3. Using embeddings in healthcare

For ordinary semantic search, the question you're answering is:

Are these two queries about the same thing?

For a medical cache, the bar is much higher:

Is the cached answer still correct under this query's specific facts — its numbers, its polarity, its timing, its intent?

In a low-stakes product, a wrong cache hit is a mildly irrelevant answer; you shrug and move on. Here, a wrong hit means an ANM with a patient in front of her receives a confidently incorrect recommendation that looks vetted. The downside isn't bounded by "the user is mildly annoyed", it's bounded by "someone is harmed."

"Make the cache fast and cheap" was not enough, and what we needed was to "make the cache never serve a wrong clinical answer," and that was a far harder target.

4. A replay harness to test the cache

Before any of the results below, one question had to be answered: how do you test a clinical cache at all? You can't just use the usual playbook: ship behind a flag, allot 5% of the traffic to the new path, watch the metrics, ramp up. This is exactly what the safety bar forbids. The whole point is to never serve a single wrong answer, and a 5% live experiment is, by definition, serving real answers to real workers.

So we never put it in the request path. Instead we built an offline cache replay (backtest) harness that asks a counterfactual: if this cache had been live last quarter, what would it have done?

We seed the cache with real history — each entry carrying the query, the response that was actually generated and served at the time, and the embedding of the query-plus-response concatenation. Then we replay a later window of real queries through the real lookup logic, as if they were arriving live.

That single design choice paid off three times over:

1. Zero production risk: We measured hit rate, safety, latency, and cost across the whole pipeline without exposing a single real worker to one cached answer. No exposure window where something unsafe could leak — which, given the stakes, was the only acceptable way to run the experiment at all.

2. Free ground truth (no regeneration): Since the replay queries are past queries, we already had the answer generated and stored for each one. So for every cache hit we could compare the cache's suggested answer against an already known baseline, instead of using an LLM to produce a fresh answer to grade against. This helped avoid a large sum in LLM and embedding-API spend.

3. Fast iteration: LLM generation is also the slow part. Reusing the pre-generated answers allowed iterating faster since a full model-and-threshold sweep finished in a fraction of the wall-clock time.

Everything in the sections that follow: the threshold curves, the fooling rates, the cache-hit quality audit, is an output of this harness.

5. Where just "similar" doesn't cut it

Illustration generated with AI

The first thing we tested was the obvious: embed the query, fetch the nearest neighbour above a threshold, score the match. To separate "this is a hit" from "this is a safe hit", we created an LLM-judge to rate each candidate match from 1 (clearly wrong, do not reuse) to 5 (strong, reusable) along with its reasoning, calibrated against a batch I'd first reviewed by hand.

As a result two problems showed up immediately:

Problem one: the threshold trap. Raising the cosine threshold does reduce bad matches, but it collapses the hit rate faster than it buys you safety. On one replay, a serve-threshold of 0.80 gave about a 24% hit rate but with unsafe matches mixed in; 0.85 dropped to ~3%; by 0.90 there were essentially zero qualifying semantic hits left. There was no setting that was simultaneously safe and useful.

Problem two: embeddings are blind to magnitude. Dense embeddings are good at capturing broad topic and meaning. They are not built to treat a number as a hard decision boundary. So a question about hemoglobin of 7 and a question about hemoglobin of 12 sit almost on top of each other in vector space, even though one is anemia needing intervention and the other is normal.

To investigate whether this wasn't just a one-off, we built a deliberately adversarial dataset: anchor questions, each paired with a true paraphrase plus a set of clinically meaningful edits (change the number, flip the polarity, add a negation, shift the gestational timing). The test: does the embedder rank the true paraphrase above the dangerous edits? For numeric edits specifically, the answer was usually no — across every model we tried.

It's worse than "sometimes confused." For four of the six models, the average cosine similarity of a numerically-altered query is actually higher than that of a genuine restatement of the same question:

When the red bar beats the green bar, the embedder literally prefers the clinically wrong answer. It's not that the embeddings are bad; it's that the one thing they're worst at preserving (numbers) is exactly the thing that's most often clinically decisive.

The lesson: a "hit" only means a close vector match. Whether that match is safe to reuse is a different question , and for the cases that mattered, the answer was usually no.

6. Introducing perturbations to fool embeddings

Numbers were the loudest failure, but not the only one. To understand different modes where embeddings failed to grasp the difference between two identical-looking queries with completely different meanings, we created an adversarial dataset around a number of categories of perturbation, i.e., distinct ways a query can stay lexically close while becoming a different clinical question. From a single anchor query, each category is a different lever on its meaning:

Only almost_same should ever produce a cache hit. The other nine are traps, and each one is a real way a worker's question differs from a cached one. A cache that can't tell severe from mild, or give iron from don't give iron, will eventually answer one with the other. The whole point of the taxonomy is to turn that risk into something you can measure instead of fear.

7. Trying to make numbers "louder"

Hypothesis: If the problem is that numbers get buried, maybe we can make them more salient by spelling them out as words ("five" instead of "5") to force the embedder to attend to them.

It didn't work. Using the anemia anchor, the true paraphrase sits at cosine 0.872. Every clinically distinct change to the hemoglobin value — including the dangerous drop from 8 to 5 — scores higher than that paraphrase, in both digit and word form. Spelling the number out nudged the score down by a hair, never enough to cross below the paraphrase. The magnitude information simply isn't represented in a usable way, and surface tricks don't put it there.

8. Training a safety gate

If a frozen embedding can't separate danger from paraphrase, maybe a small model trained on top can. This suggestion was made by my manager, Jigar — to try to rerank/classify on top of the embeddings rather than trusting cosine similarity alone. We built rerankers over the frozen embeddings in two variants: a gradient-boosted pairwise ranker (XGBRanker) and a small triplet-loss projection head.

In cross-validation, this performed well. The numeric features drove the numeric fooling rate from 0.60 down to 0.02 and lifted top-1 accuracy from 0.34 to 0.88.

Then we tested them on 25 topic groups it had never seen in training, and the story flipped.

The XGBRanker fell to 0.44 on unseen topics, worse than raw cosine's 0.68. It had overfitted on the perturbation dataset, not a general notion of clinical safety. Only the triplet head seemed like it generalized, and even it failed on a meaningful fraction of new groups.

A model boasting a score 0.88 on in-distribution data points can be worse than doing nothing on topics outside its training set.

9. Abandoning similarity altogether

Every approach so far relied on embeddings and a similarity score that fails to capture subtle nuances which are critical to a query's intent and meaning. So, we tried something that didn't involve vectors. We thought about representing each query in a structured format such that two paraphrases should end up colliding on the same representation, and any perturbation (e.g. numeric, polarity, etc.) will lead to a different representation. For this to happen, we needed to extract information from the query including, but not limited to:

• who the patient is
• the clinical topic
• the measurements and their values
• the intent (definition, management, referral)
• the polarity (e.g. "kam ho raha hai", "badh raha hai")

Then we can express/represent the information in a json format using a smaller LLM, and check those json objects for an exact match. In practice, this is a single step (extraction + converting to json)

At request time it runs as a second track alongside the existing pipeline: a small LLM turns the query into structured JSON, which is normalized, hashed, and looked up. Because it runs in parallel with the preprocessor, its ~2 s never lands on the critical path — a hit returns the cached answer, a miss just continues down the normal pipeline.

Concretely: these are some of the real ways an ANM might ask the same question — across English, Hinglish, and Hindi — that should all land on one cache entry.

The trick is the schema. Each query is expressed into the same shape, then normalized (synonyms folded, units canonicalized, keys sorted) and hashed. Here is what the queries above actually become:

{
  "schema_version": "query_canonical_v3",
  "intent": "clinical",
  "language": "hi-en",
  "care_recipient": "pregnant_woman",
  "subjects": ["pregnant_woman"],
  "request": {
    "kind": "action", "action_type": "counseling", "information_type": "none",
    "classification_type": "none", "actor": "anm",
    "topic": "anemia", "targets": ["hemoglobin"]
  },
  "context": {
    "gestation": { "kind": "exact", "min_weeks": 24, "max_weeks": 24 },
    "gravida": null, "parity": null, "maternal_age_years": null,
    "inter_pregnancy_interval_years": null, "postpartum_days": null
  },
  "triage": { "stated_acuity": "none", "escalation_asked": false },
  "findings": [
    {
      "name": "hemoglobin", "applies_to": "pregnant_woman",
      "category": "lab", "assertion": "present",
      "value": { "kind": "quantity", "op": "eq", "number": 8, "unit": "g/dl" },
      "temporal": { "frequency": "none", "duration_days": null, "event_time": null }
    }
  ]
}

The full schema they target — intent, care recipient, a structured request, gestational context, triage acuity, and a list of clinical findings carrying values, units, and operators:

And the structure is produced by a single LLM prompt. Note how much clinical domain logic is hand-encoded — which becomes the whole problem later, when we ask why "just fix the prompt" doesn't generalize:

The appeal is that instead of a fragile floating-point threshold, a cache hit becomes an inspectable, explainable equality between two structured objects. A clinician or PM can look at a key and understand why two questions matched. And run back through the replay harness (§4), it produced the most encouraging number among the array of experiments: a 49% hit rate (14% exact + 35% structural). That's more than double the bar we'd set for "worth the complexity."

But again, a hit rate is not a guaranteed safe response. So we inspected the matches one at a time, asking a single question per row: could this matched answer be safely reused for the incoming query?

Only a third were clean paraphrases. Nearly half were clinical violations — matches that would serve a wrong or unusable answer. And the violations weren't random noise. They clustered into a small set of systematic failures:

Examples: "When is breathlessness a danger sign?" (when to worry) colliding with "Why does breathlessness happen?" (a different answer entirely). "Pregnant woman, age 36 — is she high-risk?" canonicalizing to a generic high-risk question with the age 36 dropped. Notice what changed: the dominant failures are no longer numeric — they're topic and intent and structure-extraction errors. We traded "is the embedding similar enough" for "did the LLM extract the right structure," and that extraction failed in exactly the clinically decisive ways.

The tempting fix is to keep tightening the schema and the extraction prompt — split this field, preserve that number, enumerate this special case. But this is where the latency budget and the failure list collide, and the collision is structural rather than a matter of effort. The whole premise of the canonicalization track is speed: it only earns its place on the critical path because it runs on a small, cheap model (we used gpt-4.1-nano). That single constraint is what makes the prompt impossible to win:

• Add too little and it underfits. Without an explicit field, and a corresponding few-shot examples — for each kind of clinical detail, the small model has nowhere to put information it wasn't told to expect, so it silently maps unknown facts onto whichever field looks closest. The age-36 case gets dropped; a "why" question lands in a "when" slot. It doesn't generalize, it guesses.
• Add too much and then the instructions-following breaks. Every fix is one more hand-coded rule and one more example, but as the prompt grows a small model's instruction-following degrades: past some size it begins ignoring the very constraints you just added. And the examples only ever cover phrasings you've already seen, while the surface area of real clinical phrasings is effectively infinite. You end up patching the failure list case by case — and at that point it becomes a whack-a-mole problem.
• Upgrading the model defeats the point. The obvious escape — "just use GPT-4.1 or GPT-5.2" — breaks the only reason the track exists. A more capable model is slower and costlier; put one on this path and you've added latency and spend without removing the main pipeline. There is nothing to gain: at that point you might as well just run the default pipeline.

A canonicalizer that only behaves on cases it has been explicitly shown — and that can't be made more capable without erasing the very latency advantage that justified it — is not a deployable component.

10. Lessons

To sum up:

There were two interesting findings inside the wreckage. First: non-numeric questions are materially safer to reuse than numeric ones — at the same threshold, non-numeric matches were judged safe roughly 1.7x as often. A cache that simply refuses to serve numeric/decision queries and routes them to the model is a defensible design. Second, a trivially-safe slice that needs no embeddings at all: a meaningful share of real traffic is exact repeats of greetings, acknowledgements, and stable definitions ("what is HRP?").

11. Where we left it

We paused automatic semantic answer-reuse for clinical content. To be clear about the status: none of it shipped to users. What we'd actually deploy is the boring-but-safe set: an exact-match cache for the trivially-safe slice. This will also help us in absorbing bursts of identical questions (useful during new-worker onboarding, when trainees fire the same queries at once). Everything clinical and substantive ill routed to the default pipeline.

The thing I'm most glad we built is the replay harness from §4 — it scores matches with a calibrated judge and lets you ask "what would have happened" without ever exposing a real worker to an untrusted answer. That's what turned a string of hunches into decisive results, and it's what any future attempt should start from.