Technical guideProduction GenAI9 min

How to productionise a RAG system on AWS Bedrock

RAG demos fail silently: the answer looks confident and is wrong. Here is what actually separates a demo from a production system, from retrieval quality to cost control.

By Rahul LadumorJuly 19, 2026

The short answer

To productionise a RAG system on AWS Bedrock, treat it as a retrieval and evaluation problem, not a prompt problem. Ground every answer in a hybrid search index, return the source chunks so answers are auditable, add an evaluation loop that treats an unsupported answer as a defect, and put a semantic cache and a per-request cost ceiling in front of the model. Bedrock hands you the model and the IAM boundary. Production is won or lost in the retrieval and evaluation layers around it.

The rest of this page is how I do each of those, drawn from a production RAG platform I built across 2M+ documents.

Why RAG demos fail in production

A demo runs clean questions over clean documents and looks brilliant. Production sends messy queries over stale and inconsistent content, and the model answers confidently from weak retrieval. That is the dangerous part: the wrong answer looks exactly like a right one. The model is rarely the whole problem, and I will not hide retrieval, evaluation, or failure handling behind a prompt change.

Retrieval quality is the product

Answer quality is capped by retrieval quality. If the right chunk never reaches the model, no prompt saves you. Two levers matter most.

Chunking. Chunk size was the single biggest lever on retrieval quality. On the platform above I tested 256, 512, and 1024 token chunks; 512 tokens with a 50-token overlap gave the best balance of precision and recall. Your corpus can differ, which is the point: measure it against a retrieval metric instead of copying a default.

Hybrid search. Pure vector search misses exact terms, IDs, and acronyms; pure keyword search misses meaning. Combining them moved mean reciprocal rank at 10 from 0.31 on keyword-only to 0.78 on hybrid, roughly a 23% gain over pure vector. Hybrid keyword plus vector is my default starting point on AWS with OpenSearch Serverless.

Source attribution and trust

Source attribution is non-negotiable for enterprise adoption. Engineers would not trust answers without seeing the exact document and page the answer came from. On that platform, returning source chunks with every answer drove the large majority of user adoption on its own. It also turns a black box into something you can debug: when an answer is wrong, you can see whether retrieval or generation failed.

Controlling LLM cost

A semantic cache in front of the model is the highest-leverage cost control. Key it on embedding similarity, so repeated and near-duplicate questions never pay for the model twice; on the platform above, a cache hit on cosine similarity over 0.95 cut LLM cost by around 60% and held monthly model spend near a predictable figure. The hard part is not the cache, it is invalidation. I design invalidation, not just hit rate, so stale answers are not served after the underlying documents change. Pair the cache with a per-request cost ceiling so a bad loop cannot run up the bill.

Evaluation and failure handling

Without evaluation you are shipping vibes. Track retrieval quality as a metric, not a feeling, and treat an unsupported answer as a product risk to be measured and driven down, the same way you would treat a 500 error. Wire evaluation hooks into the pipeline from day one: a reference set of questions with known-good sources, scored on every change, so a "harmless" prompt tweak cannot quietly regress retrieval.

A reference architecture on AWS

A pragmatic AWS-native shape: documents flow through an ingestion and chunking pipeline into OpenSearch Serverless for hybrid search, kept VPC-local so there is no extra vendor boundary. Retrieval assembles context with source metadata; a semantic cache sits in front of Amazon Bedrock so repeat questions skip the model. Generation runs on a Bedrock model chosen for context window and native integration, with the returned source chunks attached to the answer. Evaluation hooks and cost/latency observability wrap the whole path so retrieval quality and spend are both visible.

Common mistakes

Reaching for a bigger model when the real fault is retrieval.
Shipping answers with no source, so no one can trust or debug them.
Caching without an invalidation plan, then serving stale answers.
Picking a chunk size from a blog instead of measuring your corpus.
No evaluation set, so every change is a guess and regressions are silent.
No per-request cost ceiling, so one bad loop becomes a bill.

Next step

Have the same problem on your stack?

Send the architecture, AWS bill concern, deploy pain, or GenAI reliability issue. I'll find the first real bottleneck and propose a small, reversible fix.

Book a production review See the GenAI / RAG case study

FAQ

How do you productionise a RAG system on AWS Bedrock?

Treat it as a retrieval and evaluation problem, not a prompt problem. Ground answers in a hybrid search index, return source chunks so answers are auditable, add an evaluation loop that treats unsupported answers as a defect, and put a semantic cache and cost ceiling in front of the model. Bedrock gives you the model and IAM boundary; the retrieval and eval layers are where production is won or lost.

Why does my GenAI demo fall apart with real users?

Demos use clean questions on clean documents. Production gets messy queries, stale content, and edge cases where the model answers confidently from weak retrieval. The failure is silent: the answer looks right and is wrong. You fix it with retrieval quality, source attribution, and evaluation, not a bigger model.

What chunking strategy should I use for RAG?

Test it, do not guess. On a production platform I built I tested 256, 512, and 1024 token chunks; 512 with 50-token overlap gave the best precision and recall balance. Your corpus may differ, which is exactly why chunk size and overlap should be measured against a retrieval metric, not copied from a blog post.

How do you cut LLM cost without wrecking answer quality?

A semantic cache in front of the model, keyed on embedding similarity, so repeated or near-duplicate questions never hit the LLM twice. Pair it with a per-request cost ceiling. The hard part is invalidation: I design cache invalidation, not just hit rate, so stale answers do not get served after the underlying documents change.

Vector database: OpenSearch Serverless or Pinecone?

For an AWS-native, VPC-local system with hybrid keyword-plus-vector search and no extra vendor boundary, OpenSearch Serverless is usually the pragmatic default. Pinecone is a fine managed option when you want it fully off-platform. The deciding factors are data residency, hybrid search needs, and how much operational surface you want to own.