← All case studies

Production RAG assistant for customer communities

Ask Kindred

I led Ask Kindred from SDK integration through production: a retrieval-grounded assistant serving about 200 daily active users across three customer companies.

Role
Lead engineer
Team
Led a team of 4–5 engineers and designers
Status
In production at Rare Days
Stack
RAG architecture, LLM chatbots, Multi-tenant SaaS, Access control

Outcome

Ask Kindred serves approximately 200 daily active users across three customer companies, with grounded retrieval, citations, conversation history and feedback running in production.

  • ~200daily active users in production
  • 3customer companies served
  • 10+CI improvements to release quality

How quality is measured

Quality is checked against a fixed set of 55 queries run through the deployed endpoint, with timing and retrieval scores exported for review. Tuning ran in three evaluation-driven rounds.

  • 55fixed test queries: 20 in scope, 10 out of scope, 10 crisis, 5 multi-turn conversations
  • 10/10crisis queries routed correctly, up from 7/10 at baseline
  • 10/10out-of-scope queries refused with no grounding violations
  • 19/20in-scope queries answered; the miss is a gap in the corpus
  • Skipping the reranker when the top vector score is below a floor cut out-of-scope time to first token from about 6 seconds to about 500 milliseconds.
  • Rewriting follow-up questions into standalone queries fixed three multi-turn fallbacks. 13 of 15 turns answer, and the two misses are corpus gaps.
  • The fallback streams a fixed message instead of calling the model, so it cannot answer from training data.
  • The same set reruns from the CLI and each run is recorded with the reranker that ordered each case, so a change to a prompt, threshold or reranker is compared against the same questions.

Speed, uptime and cost

  • <5stime to first token
  • 99.99%availability
  • ~$15cost per client, per month

What I owned

Led the product from SDK integration through production for three customer companies, owning the system from retrieval architecture through backend delivery and reliable releases.

Built reusable content-ingestion, retrieval and production safeguards across CMS and community data, improving the quality, trust and reliability of customer-facing AI.

Designed the multi-tenant SaaS architecture, including backend infrastructure and role-aware access controls, and mentored engineers on the team.

System at a glance

Customers embed the assistant through an SDK integration.

Key decisions

  • Grounded retrieval with citations, so customer-facing answers can be checked against their source.
  • Reusable ingestion and retrieval building blocks, shared with other Rare Days projects instead of rebuilt per customer.
  • Role-aware access controls in a multi-tenant design, so each customer sees only their own content.

How I approached reliability

  • Introduced Sentry across frontend and backend services.
  • Delivered 10+ CI improvements that strengthened release quality and platform reliability.
  • Built production safeguards into the ingestion and retrieval layer.

Technology focus

  • RAG architecture
  • LLM chatbots
  • Multi-tenant SaaS
  • Access control
  • CI/CD
  • Sentry