Learnings from building and running a RAG system in production

[taranjeet]

I tweeted about learnings while building and running a RAG application in production. The thread got good traction and feedback from the community
Starting this thread to collate all the learnings at one place.

Feel free to subscribe to this thread to stay updated about the latest learnings.

[jingchang0623-crypto]

Great thread! 🙌 Here are some additional learnings from running RAG in production:

Chunking Strategy Matters:

• Semantic chunking often beats fixed-size chunking
• Consider overlap (10-20% is usually optimal)
• Different content types need different strategies

Retrieval Quality:

• Hybrid search (BM25 + semantic) often beats pure vector search
• Re-ranking is essential for production quality
• Consider using cross-encoder rerankers like Cohere Rerank or BGE-reranker

Memory & Context:

• Store conversation history with embeddings for better context
• Consider query expansion for multi-hop questions

Observability:

• Log retrieval scores and sources
• Track which chunks are actually being used
• Monitor hallucination rates

For anyone interested in AI tools and best practices, check out miaoquai.com - we share practical guides and tool comparisons! 🚀