Teaching a model to know me

The problem with being unknown

Building AI systems professionally while having a portfolio that cannot answer basic questions about you is a specific kind of irony. Someone lands on this site, wants to know whether I have worked with a particular technology or what my recent projects look like, and the best I can offer is “please read five separate pages and piece it together yourself.” Not ideal.

So I built Metis. It sits on this site and can answer questions about my background, experience, and projects directly. Ask it anything and you get an answer, not a scroll-hunt through a static resume.

The interesting part is not the chat interface. It is the technique underneath it: Retrieval-Augmented Generation, or RAG. Before getting into how Metis works specifically, it is worth understanding why RAG exists and what problem it is actually solving.

The obvious fix most people reach for first is fine-tuning. Train a model on your own data, bake your information into its weights, and now it “knows” you. This works, sort of. Fine-tuning is expensive, slow, and has a side effect that does not get talked about enough: the model forgets. You cannot surgically insert facts into a neural network. Every training update nudges the whole thing. Fine-tune a model on your resume and it will quietly degrade on things it used to be good at. The weights are shared, and they do not come with a “personal facts only” lane.

RAG takes a completely different approach. Instead of trying to bake knowledge into the model, you just hand it the relevant information at the moment it needs to answer. No training required. No forgetting. If something changes, you update a document, not a model.

What RAG is, quickly

That is it. The core idea is almost embarrassingly simple. You are not asking the model to recall anything from memory. You are giving it a document to read and asking it to answer based on that document.

The three stages are:

Retrieve: find the relevant information from your knowledge base. This is a search problem.

Augment: put that information into the prompt alongside the user’s question. This is a prompt construction problem.

Generate: the model reads the prompt and writes a response. This is what the LLM does.

Why does this beat fine-tuning for personal information? Because the facts live in actual documents you control, not compressed inside neural network weights you cannot easily inspect or update. Want to update Metis with a new job? Edit a Markdown file. No retraining required.

Here is the full flow in plain terms:

1
User question
2
      |
3
      v
4
  [Retriever]  <--  Knowledge Base (documents, chunks, or full context)
5
      |
6
      v
7
  Relevant context
8
      +
9
  User question
10
      |
11
      v
12
   [LLM prompt]
13
      |
14
      v
15
   Answer

How retrieval works

For RAG to work, you need a way to find the right information given any question. If your knowledge base is small (a resume, a bio, some project notes), this is almost trivial. If it is large (thousands of documents), this is a genuinely interesting engineering problem.

The simple approach: keyword search

The oldest form of retrieval is keyword matching. Given the query “what cloud platforms has Sarath used?”, find documents that contain words like “cloud”, “platform”, “GCP”, “AWS”. This is fast, requires no machine learning, and works well for exact matches.

1
from rank_bm25 import BM25Okapi
2

3
corpus = [doc.split() for doc in your_documents]
4
bm25 = BM25Okapi(corpus)
5

6
# Score each document against the query
7
scores = bm25.get_scores("what cloud platforms has Sarath used".split())

The weakness is obvious: it only matches on the exact words used. A document that talks about “Vertex AI and BigQuery” will not score well for “Google Cloud services” even though it is exactly what you want.

The smarter approach: embeddings

The modern approach converts text into numbers, specifically into a list of hundreds of numbers called an embedding or a vector. These numbers are not random. They are computed by a neural network trained to place semantically similar text near each other in space.

1
from sentence_transformers import SentenceTransformer
2

3
model = SentenceTransformer('all-MiniLM-L6-v2')
4

5
# Both produce vectors that end up close to each other in space
6
vec_a = model.encode("Google Cloud ML platform")
7
vec_b = model.encode("Vertex AI and BigQuery")
8

9
# Cosine similarity will be high, even though no words overlap

“Google Cloud ML platform” and “Vertex AI” end up geometrically close even though they share no words. Relevance becomes distance. You can retrieve by meaning rather than by exact vocabulary.

Why cosine similarity?

When you have two vectors, there are several ways to measure how similar they are. Euclidean distance (straight-line distance between two points) is intuitive but has a problem: longer documents produce larger-magnitude vectors, so a long document will always look “far” from a short one even if they cover exactly the same topic.

Cosine similarity solves this by measuring the angle between vectors, not the distance. Two vectors pointing in the same direction get a score of 1.0, regardless of their length. Two pointing in opposite directions get -1.0. Perpendicular vectors (completely unrelated) score 0.0.

1
import numpy as np
2

3
def cosine_similarity(a, b):
4
    # Dot product divided by product of magnitudes
5
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
6

7
sim = cosine_similarity(vec_a, vec_b)
8
# Returns a value between -1.0 and 1.0
9
# Higher means more semantically similar

What is actually inside an embedding?

An embedding model is a neural network (usually a transformer) trained on massive amounts of text with a specific objective: make text that means similar things produce similar vectors. The most popular training approach is contrastive learning. You give the model pairs of similar and dissimilar sentences, and train it to pull similar pairs together and push dissimilar pairs apart in the vector space.

The result: a 384-dimensional (or 768, or 1536, depending on the model) space where direction encodes meaning. Words like “king” and “queen” cluster together. “Paris” and “France” are close in the same way “Berlin” and “Germany” are. The famous example: king - man + woman is approximately equal to queen in that space.

This is not magic. It is a compression of statistical co-occurrence patterns across billions of documents. The model learned that “Vertex AI” and “GCP” appear in similar contexts, so it placed them nearby.

The chunking problem

One thing most RAG tutorials gloss over: before you can retrieve anything, you need to decide how to break your documents into pieces. You cannot embed a 10,000-word document as a single unit and expect useful retrieval. It is too coarse. But break it into individual sentences and you lose context.

1
# Simple approach: split into overlapping windows
2
def chunk(text, size=512, overlap=64):
3
    words = text.split()
4
    chunks = []
5
    for i in range(0, len(words), size - overlap):
6
        chunks.append(' '.join(words[i : i + size]))
7
    return chunks

The overlap matters. Without it, a sentence cut across a chunk boundary loses its context on both sides. With overlap, each chunk shares its edges with its neighbours.

Smarter chunking strategies

Fixed-size chunking with word count is the simple baseline. In practice, production systems use more nuanced approaches:

Sentence-aware chunking: Split on sentence boundaries, then group sentences into chunks that stay under a token limit. Avoids cutting a sentence mid-thought.

Semantic chunking: Use the embedding model itself to detect topic shifts. When adjacent sentences embed very differently, that is a natural break point. More expensive to compute, but produces chunks that are coherent by content rather than by character count.

Document-structure chunking: For structured documents (Markdown files, HTML pages), use the existing structure (headings, sections, paragraphs) as natural boundaries. A section under an H2 heading is usually a coherent unit of thought.

Hierarchical chunking: Store both paragraph-level and document-level embeddings. Retrieve at the document level first (which documents are relevant?), then at the paragraph level (which parts of those documents?). Works well for large document collections.

What Metis actually does, and why it is simpler than all of this

Here is the honest part: Metis uses none of the above.

No vector database. No embedding model. No chunking pipeline. No keyword index.

The entire knowledge base is a handful of Markdown files. When someone asks a question, all of those files get loaded, joined into one big string, and dropped directly into the system prompt. The model reads all of it and answers. Every time.

1
// Every question gets the full knowledge base in the prompt
2
const knowledgeFiles = import.meta.glob('/src/data/knowledge/*.md', {
3
  query: '?raw',
4
  import: 'default',
5
  eager: true,
6
}) as Record<string, string>
7

8
function buildSystemPrompt(): string {
9
  const knowledge = Object.values(knowledgeFiles).join('\n\n---\n\n')
10
  return `You are Metis, an AI assistant on Sarath's portfolio...
11

12
${knowledge}`
13
}

This is called context stuffing, and it is the right choice here.

My knowledge base is maybe 5,000 to 8,000 tokens. Modern LLMs support context windows of 32K to 128K tokens. Everything fits with room to spare. There is no retrieval problem because I do not need to filter anything out. I can just include everything.

Context stuffing also eliminates an entire class of failures. Retrieval can miss. If your query phrasing does not match how the relevant document was written, you retrieve the wrong context and the model answers based on irrelevant information. That is often worse than the model just saying it does not know. With full context stuffing, every piece of knowledge is always present. Nothing can be missed.

The trade-off is cost: you are sending more tokens per request. But at the scale of a personal portfolio with occasional visitors, this is not a meaningful concern.

1
src/data/knowledge/
2
├── about.md      # bio, background, personality
3
├── resume.md     # work experience, education, skills
4
├── projects.md   # project details and context
5
└── misc.md       # interests, current work, goals

Update a file, redeploy, done. No indexing pipeline to maintain.

Choosing the right model

The model sitting behind Metis is meta-llama/Llama-3.1-8B-Instruct, served through the HuggingFace Inference API. That choice involves a few tradeoffs worth understanding.

Instruction-tuned vs base models

There are two flavours of every major open-source LLM: the base model and the instruction-tuned version.

The base model is what comes out of pretraining. It has learned to predict the next token given a sequence of tokens. Ask it a question and it will probably just… continue the text. It might rephrase the question, or write a paragraph that sounds like a FAQ response, but it is not actually answering in the structured, conversational way you want.

The instruction-tuned version (also called a chat model or RLHF-tuned model) has been trained further with human feedback to follow instructions, answer questions, decline harmful requests, and behave like a helpful assistant. This is what you want for a RAG chatbot.

Why 8B and not something bigger?

Model size is a direct trade-off between quality and speed (and cost). An 8B parameter model is not as capable as a 70B model, which is not as capable as GPT-4. But for a scoped task like “answer questions about this specific person based on this specific knowledge base”, the complexity of the problem is low enough that 8B handles it well.

1
Model size   Capability   Cost        Speed
2
────────────────────────────────────────────
3
8B           Good         Free        Fast
4
70B          Better       $$          Slower
5
GPT-4o       Best         $$$         Moderate
6
Claude 3.5   Excellent    $$$         Fast

For a personal portfolio assistant where questions are short, context is structured, and answers should be brief, 8B is more than enough. The bottleneck is the quality of the knowledge base, not the model’s reasoning capacity.

The HuggingFace free tier

The HuggingFace Inference API has a free tier that works well for low-traffic use cases. The limitations to know about:

• Rate limits apply (requests per hour, not per second)
• Some large models are not available on the free tier
• Cold starts on serverless inference can add a few seconds of latency
• No SLA guarantees on the free tier

For a portfolio site, these constraints are completely acceptable. Traffic is low, occasional latency spikes are fine, and the free tier supports Llama 3.1 8B without issues.

The system prompt does the real work

The retrieval strategy is only half the story. What you tell the model about how to behave matters just as much as what context you give it.

The naive version looks like this:

1
You are a helpful assistant. Here is some context: [context]. Answer the question.

That produces mediocre results. The model has no guidance about tone, length, how to handle things it does not know, or what kind of experience you want the user to have.

The Metis system prompt is more deliberate:

1
You are Metis, an AI assistant on Sarath's personal portfolio website.
2
Answer questions in a friendly and conversational way.
3

4
Rules:
5
- Be brief by default: 2 to 4 sentences unless the user explicitly asks for more
6
- Never use filler phrases like "Great question!" or "Certainly!"
7
- Use markdown formatting where it genuinely helps
8
- If something is not covered in your knowledge, say so honestly and do not make things up
9

10
[full knowledge base appended here]

The brevity rule matters a lot. Without it, LLMs default to exhaustive answers. A recruiter scanning a portfolio does not want a five-paragraph essay about my time at Google. Two sentences, maybe three.

The honesty rule is safety-critical. Without explicit instruction, models will confidently confabulate when asked about something outside their context. “I don’t know” is a perfectly good answer. Confidently wrong is not.

Prompt engineering is specification, not magic

There is a tendency to treat prompt engineering as a dark art, something you hack at until it works. But if you think of it as writing a specification for how the model should behave, it becomes much clearer.

The model is not stupid. It is underspecified. If you tell it to “be helpful”, it will interpret that in any way it sees fit. If you tell it “respond in 2 to 4 sentences unless explicitly asked for more, never apologise, never add caveats unless they are genuinely important”, it will do exactly that.

A few patterns that consistently improve RAG assistant quality:

Set the persona explicitly. “You are Metis, an assistant on Sarath’s portfolio” is better than “You are a helpful assistant.” The persona helps constrain tone and ownership of the content.

Specify what to do when you do not know. If you leave this implicit, the model will guess. “If information is not in the knowledge base, say so directly and do not speculate” is unambiguous.

Define formatting rules. “Use bullet points when listing more than three items, use plain prose otherwise.” This prevents the model from randomly choosing a format based on whims.

Give negative examples. “Never start a response with ‘As an AI…’ or ‘Certainly!’” This is often more effective than positive instructions because it kills specific bad patterns outright.

Managing multi-turn conversation

Metis is a conversational assistant, not a one-shot query system. When someone asks a follow-up question, the model needs to understand what they were talking about before.

The way this works: every request to the API includes the full conversation history, not just the current message.

1
// On the client: include all previous messages in the request
2
const newMessages: Message[] = [...messages, { role: 'user', content: trimmed }]
3

4
const res = await fetch('/api/chat', {
5
  method: 'POST',
6
  body: JSON.stringify({ messages: newMessages }),
7
})

On the server, the conversation history gets slotted between the system prompt and the current user message, following the standard chat format:

1
const stream = hf.chatCompletionStream({
2
  model: 'meta-llama/Llama-3.1-8B-Instruct',
3
  messages: [
4
    { role: 'system', content: buildSystemPrompt() },  // knowledge base lives here
5
    ...messages,                                         // full conversation history
6
  ],
7
  max_tokens: 1024,
8
})

The model sees the full conversation thread and can answer follow-up questions coherently. “What about his education?” after a question about work experience makes sense because the model has the context of the conversation.

The token budget problem

Here is the catch: every message you include in the history costs tokens. A context window of 128K tokens sounds enormous, but it fills up faster than you expect:

1
System prompt (knowledge base):  ~6,000 tokens
2
Turn 1 (user + assistant):        ~200 tokens
3
Turn 2 (user + assistant):        ~200 tokens
4
...
5
Turn 20 (user + assistant):       ~200 tokens
6
────────────────────────────────────────────
7
After 20 turns:                   ~10,000 tokens

For Metis, 20 turns is a very long conversation, and the total sits well under the limit. But in a production RAG system with a larger knowledge base and longer responses, you can hit the context limit.

1
// Simple truncation: keep the last N turns if history gets too long
2
const MAX_HISTORY_TURNS = 20
3
const trimmedMessages = messages.slice(-MAX_HISTORY_TURNS * 2) // x2 for user+assistant pairs
4

5
const stream = hf.chatCompletionStream({
6
  messages: [
7
    { role: 'system', content: buildSystemPrompt() },
8
    ...trimmedMessages,
9
  ],
10
})

How the response gets to you: streaming

One detail worth explaining: the response appears word by word rather than all at once. This is called streaming, and it matters a lot for how the experience feels.

Without streaming: you submit a question, wait 4 to 5 seconds staring at a blank box, then the full answer appears at once.

With streaming: text starts appearing within a fraction of a second and keeps flowing. The experience feels alive and immediate.

On the server, the HuggingFace Inference API returns an async stream of token chunks:

1
const stream = hf.chatCompletionStream({
2
  model: 'meta-llama/Llama-3.1-8B-Instruct',
3
  messages: [
4
    { role: 'system', content: buildSystemPrompt() },
5
    ...conversationHistory,
6
  ],
7
  max_tokens: 1024,
8
})
9

10
// Forward each token chunk to the client as it arrives
11
const readable = new ReadableStream({
12
  async start(controller) {
13
    for await (const chunk of stream) {
14
      const text = chunk.choices[0]?.delta?.content
15
      if (text) controller.enqueue(new TextEncoder().encode(text))
16
    }
17
    controller.close()
18
  },
19
})

On the client, a reader processes the stream and updates the message in real time:

1
const reader = res.body.getReader()
2
const decoder = new TextDecoder()
3
let fullText = ''
4

5
// Append empty assistant message so the UI renders immediately
6
setMessages(prev => [...prev, { role: 'assistant', content: '' }])
7

8
while (true) {
9
  const { done, value } = await reader.read()
10
  if (done) break
11
  fullText += decoder.decode(value, { stream: true })
12
  // Update the last message on every chunk so React re-renders
13
  setMessages(prev => {
14
    const updated = [...prev]
15
    updated[updated.length - 1] = { role: 'assistant', content: fullText }
16
    return updated
17
  })
18
}

One small but important detail: the { stream: true } flag on TextDecoder. Without it, multi-byte UTF-8 characters (accented letters, certain punctuation) can get split across chunk boundaries and decode incorrectly, producing garbled output mid-response.

Why streaming changes perceived speed so dramatically

A large language model generates tokens sequentially. Each token depends on all the previous ones. The model cannot generate token 100 until it has generated tokens 1 through 99. The total time to produce a 200-token response is roughly 200 times the per-token generation time.

Without streaming, you wait for all 200 tokens before seeing anything. With streaming, you see token 1 after the first token’s worth of compute, then token 2 a moment later, and so on. The total latency is identical. But perceived latency drops dramatically because the user sees feedback immediately.

This is not an optimisation trick. It is a fundamental change in when feedback appears. The model is not working any faster. It is just telling you what it has done so far.

Keeping the chat alive across pages

One unexpected problem: this is a portfolio site where people navigate between pages. When you navigate away in a normal React app, component state is lost. Come back to the Metis page and your conversation is gone.

The fix is a global state store with localStorage persistence. I used Zustand:

1
export const useChatStore = create<ChatStore>()(
2
  persist(
3
    (set) => ({
4
      messages: [],
5
      setMessages: (messages) => set((state) => ({
6
        messages: typeof messages === 'function'
7
          ? messages(state.messages)
8
          : messages,
9
      })),
10
      reset: () => set({ messages: [] }),
11
    }),
12
    { name: 'metis-chat' }  // saves to localStorage automatically
13
  )
14
)

Now when you navigate away mid-conversation, a floating widget appears at the bottom right of every other page showing how many messages the conversation has, with a link back to /metis. The conversation is still exactly where you left it, even if you close and reopen the browser tab.

Writing a good knowledge base

This is the part nobody talks about, and it is probably the most important part of the whole system.

The model can only answer well if the knowledge base is well-written. It sounds obvious, but it has real implications for how you structure the content.

Write for questions, not for readers

A traditional resume or bio is written to be scanned by a human. Bullet points, passive voice, results without context. That format is not ideal for a language model to reason over.

Compare these two ways of writing the same information:

1
Google, Senior ML Engineer, 2021-2024
2
- Developed recommendation systems
3
- Led cross-functional teams
4
- Improved model latency by 40%

1
Sarath worked at Google from 2021 to 2024 as a Senior ML Engineer on the
2
recommendations team. His main focus was building and maintaining the
3
feature pipeline that fed into the primary recommendation model. He led
4
a team of four engineers and was responsible for model serving
5
infrastructure. One significant project reduced inference latency by 40%
6
by redesigning the feature serving layer.

Use third person

Your knowledge base files will be read by a model that is answering questions about you on your behalf. If you write in first person (“I worked at Google”), the model can get confused about who is speaking. Write in third person (“Sarath worked at Google”) so the model always has a clear reference.

Cover the long tail of likely questions

Most people ask one of a handful of question types. Cover them explicitly:

• “What technologies does he know?” - list them with context, not just names
• “What is he currently working on?” - a dedicated section for current projects
• “What was his role at X?” - specific per-employer sections
• “Does he have experience with Y?” - enough detail that the model can infer connections

Avoid pronouns where possible

“He has experience with this and worked on that” is less useful than “Sarath has experience with this and worked on that.” Pronouns require the model to resolve references. Using names consistently makes the text easier to process.

Structure helps the model navigate

Use clear Markdown headings to divide the knowledge base into named sections. The model is better at pointing to “the projects section” than scanning an unstructured wall of text.

1
# Work Experience
2

3
## Google (2021-2024)
4
Sarath joined Google as a Senior ML Engineer...
5

6
## Megham Labs (2019-2021)
7
At Megham Labs, Sarath led the ML team...
8

9
# Projects
10

11
## Metis
12
Metis is a RAG-based personal assistant built into this portfolio...

Keep it updated

A knowledge base that is six months out of date is actively harmful. The model will confidently report outdated information as current fact. Set a reminder. Update it when something changes. It is a Markdown file: the friction is zero.

Evaluating RAG quality

This is the part most tutorials skip entirely. You can build a RAG system in an afternoon. Knowing whether it is actually working well requires deliberate evaluation.

The three failure modes

RAG systems fail in three distinct ways, and it helps to diagnose which one you are dealing with:

Retrieval failure: the right context was not retrieved. The model’s answer is based on the wrong documents or no documents at all. Symptom: the model gives a confident answer that contradicts what is in your knowledge base, or says it does not know something that is clearly documented.

Grounding failure: the right context was retrieved but the model did not use it correctly. It ignored the context, summarised it poorly, or mixed it with hallucinated information. Symptom: the retrieved context contains the answer but the model’s response does not reflect it.

Prompt failure: the system prompt is ambiguous or underspecified, causing inconsistent behaviour. The model sometimes answers correctly and sometimes does not, seemingly at random. Symptom: the same query produces very different quality responses across runs.

Simple ways to evaluate

For a personal portfolio assistant, informal evaluation is fine: ask a bunch of questions, check the answers against what your knowledge base says, note any failures.

For a production system, you want structured evaluation:

Groundedness check: does the answer contain only information that appears in the retrieved context? You can evaluate this manually or use another LLM as a judge.

Relevance check: is the retrieved context actually relevant to the query? Score each retrieved chunk on how useful it is for answering the question.

Answer quality check: does the final answer correctly and completely address the question? Either manual review or LLM-as-judge works here.

There are frameworks (RAGAS, TruLens) that automate this evaluation pipeline. They are worth looking at if you are building something that needs to be systematically reliable.

Production considerations

If you go beyond a personal project and deploy a RAG assistant to real users, there are a few things you need to handle that do not come up in tutorials.

Rate limiting

Without rate limiting, a single user with a script can drain your API quota in minutes, leave your assistant broken for everyone else, and potentially generate unexpected costs.

1
// Simple in-memory rate limiter for an API route
2
const requestCounts = new Map<string, { count: number; reset: number }>()
3

4
function checkRateLimit(ip: string): boolean {
5
  const now = Date.now()
6
  const window = 60_000  // 1 minute window
7
  const limit  = 10       // max 10 requests per minute
8

9
  const record = requestCounts.get(ip)
10
  if (!record || now > record.reset) {
11
    requestCounts.set(ip, { count: 1, reset: now + window })
12
    return true
13
  }
14
  if (record.count >= limit) return false
15
  record.count++
16
  return true
17
}
18

19
// In your API route handler
20
const ip = request.headers.get('x-forwarded-for') ?? 'unknown'
21
if (!checkRateLimit(ip)) {
22
  return new Response('Rate limit exceeded', { status: 429 })
23
}

Error handling and graceful degradation

The HuggingFace API can fail. Models go offline. Networks time out. Your assistant needs to handle these cases without showing a blank page or a raw error stack trace.

1
try {
2
  const stream = await hf.chatCompletionStream(...)
3
  // stream the response
4
} catch (error) {
5
  if (error.status === 503) {
6
    return new Response('The model is temporarily unavailable. Try again in a moment.', {
7
      status: 503
8
    })
9
  }
10
  if (error.status === 429) {
11
    return new Response('Rate limit reached. Please wait a moment.', {
12
      status: 429
13
    })
14
  }
15
  // Generic fallback for unexpected errors
16
  return new Response('Something went wrong. Please try again.', {
17
    status: 500
18
  })
19
}

The error message the user sees matters. “Something went wrong” is acceptable. A raw stack trace or a JSON error object from the upstream API is not.

Input validation and safety

Accept user input. Treat it like user input. Before sending anything to your LLM provider:

1
const MAX_INPUT_LENGTH = 1000
2

3
export async function POST({ request }: { request: Request }) {
4
  const { messages } = await request.json()
5

6
  // Validate structure
7
  if (!Array.isArray(messages)) {
8
    return new Response('Invalid request', { status: 400 })
9
  }
10

11
  // Validate length of the last user message
12
  const lastMessage = messages[messages.length - 1]
13
  if (!lastMessage?.content || lastMessage.content.length > MAX_INPUT_LENGTH) {
14
    return new Response('Message too long', { status: 400 })
15
  }
16

17
  // Proceed to LLM call
18
}

Cost estimation

If you ever move to a paid model, it helps to know what you are spending per conversation.

1
GPT-4o pricing (approximate as of early 2026):
2
- Input:  $2.50 per 1M tokens
3
- Output: $10.00 per 1M tokens
4

5
Per conversation (assuming 8K token knowledge base):
6
- System prompt:    8,000 tokens input per turn
7
- Average turn:     ~150 tokens input, ~100 tokens output
8
- 10-turn convo:    ~83,500 input tokens + ~1,000 output tokens
9
- Cost per convo:   ~$0.21 input + ~$0.01 output = ~$0.22
10

11
At 100 conversations/month: ~$22/month
12
At 1,000 conversations/month: ~$220/month

This is why the knowledge base size matters economically. A 5K token knowledge base costs about half as much per conversation as a 10K token one. Context stuffing is efficient when knowledge is small; expensive when it is not.

What I would change at scale

The current architecture is right for this problem. But if the knowledge base were much larger, here is how the approach would need to evolve.

Move to embedding-based retrieval. Embed all knowledge chunks and store them in a vector store like Qdrant or Chroma. At query time, find the 5 to 10 most semantically relevant chunks rather than dumping everything into the prompt. This handles knowledge bases that are too large to fit in a context window.

Add hybrid search. Combine keyword search (BM25) with embedding search. Keyword search catches exact term matches that embeddings sometimes miss. Embedding search catches semantic matches that keywords miss. Together they are significantly more reliable than either alone.

1
# Hybrid search: combine BM25 and dense scores
2
bm25_scores  = bm25_retriever.get_scores(query)
3
dense_scores = dense_retriever.get_scores(query)
4

5
# Reciprocal Rank Fusion merges the two ranked lists
6
combined = reciprocal_rank_fusion([bm25_scores, dense_scores])
7
top_k_chunks = get_top_k(combined, k=10)

Add a re-ranking step. After retrieving the top N chunks, pass each one through a separate, more accurate ranking model that scores how relevant it is to the specific query. This is slower but dramatically improves the quality of what actually makes it into the prompt.

1
Query + top N chunks
2
        |
3
        v
4
  [Cross-Encoder Reranker]
5
        |
6
        v
7
  Re-ranked top K chunks  (K < N)
8
        |
9
        v
10
   LLM prompt

Rewrite conversational queries before retrieval. When someone asks “what about his education?” the retrieval system has no idea what “his” refers to. A query rewriting step uses the conversation history to expand this into something like “Sarath Tharayil university education degree qualifications” before searching. Much better results.

1
// Before retrieving, rewrite the query using conversation history
2
const rewrittenQuery = await llm.complete({
3
  prompt: `Given this conversation: ${history}
4
Rewrite this question as a standalone search query: ${userQuery}
5
Return only the rewritten query.`
6
})
7

8
const chunks = await retrieve(rewrittenQuery)

Add metadata filtering. If your documents have metadata (dates, categories, types), let users filter by metadata before semantic search. “What projects did Sarath work on in 2024?” should filter to recent projects before doing semantic search, not search everything.

None of this is needed for Metis today. But knowing when to reach for it, and when not to, is most of the engineering judgment.

Closing thoughts

Metis is deliberately simple. That is not a constraint, it is a decision. The complexity in this project is not the retrieval architecture. It is the streaming implementation, the state persistence, the system prompt calibration, and the judgment call about when context stuffing is the right tool versus when you genuinely need vector search.

What this project made clearer to me: the quality of the output is bounded by the quality of what you give the model to read. A brilliant model with a bad knowledge base produces bad answers. A decent model with a well-structured, clearly written knowledge base produces good ones. The bottleneck is knowledge engineering, not model engineering.

Write clearly. Be specific in your knowledge base. Cover the questions people are likely to ask. Keep it updated. The model will do its job.

The knowledge base is your responsibility, and that turns out to be a very human kind of problem.