First applied-AI project
Nov 2024
Retrieval · RAG

RAG API

A document question-answering service over finance, law and biology PDFs. Each request chooses its retrieval algorithm from dense similarity, hybrid rank fusion, MMR and a similarity threshold, and any of them can be followed by a cross-encoder rerank.

Repositories are private. Code walkthrough on request.

retrieval algorithms × rerank on/off, 8 configurations per request: 4 × 2
Cohere v3 embeddings with separate query and document input types: 1024-d
Postgres round trip per search, since each strategy is a SQL function: 1
domain datasets, scoped by a JSONB metadata pre-filter: 3
over-fetch before the cross-encoder keeps k: 2×
from first commit (Nov 11) to production merge (Dec 3, 2024): ~3 wks

01Next.js 15 · shadcn/ui

The app

Three panes show the sessions, the conversation, and the documents each answer used. The Search Settings popover next to the Send button sets the retrieval algorithm and its parameters for the next message.

rag-chatbot (Next.js 15, shadcn/ui), rebuilt in HTML from the source. The retrieval mode is a per-message setting in the popover above the input. Content is illustrative.

02FastAPI · Supabase · Cohere · Together AI

Architecture

FastAPI runs on Render. Supabase holds the vectors, full-text index, chat history, files and auth. Cohere embeds and reranks, and Llama 3.2 on Together AI writes the answer. Each retrieval mode is a Postgres function, so a search is one database round trip. Ingestion ran offline to pre-load the three datasets. Select a flow to trace one request.

Hover or tap any component to trace its connections, or pick a flow to step through a request end to end.

03Interactive

Retrieval modes

This is the app’s Search Settings popover with working controls. Changing a setting updates the request payload, the SQL that runs, and the notes on when that mode works well and when it fails.

Search Settings

Sent as search_options with every message.

Search Type

Number of Results (k)4

Re-rank Results

Full-text weight1.0

Semantic weight1.0

{
  "search_options": {
    "search_type": "hybrid",
    "k": 4,
    "rerank": true,
    "full_text_weight": 1,
    "semantic_weight": 1,
    "rrf_k": 60
  }
}

Hybrid · RRF hybrid_search()

Ranks a keyword arm and a vector arm separately and combines the two rankings with Reciprocal Rank Fusion.

embed(q)hybrid_search(text, q, 8, filter, 1.0, 1.0, 60)8 rowsrerank → top 4prompt

Scorescore(d) = w_ft / (rrf_k + rank_ft(d)) + w_sem / (rrf_k + rank_sem(d))

WITH full_text AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank_cd(
    to_tsvector('english', content),
    websearch_to_tsquery(query_text)) DESC) AS rank_ix
  FROM document_chunks
  WHERE to_tsvector('english', content)
        @@ websearch_to_tsquery(query_text)
    AND metadata @> filter
  LIMIT least(match_count * 2, 50)
), semantic AS (
  SELECT id, row_number() OVER (
    ORDER BY embedding <=> query_embedding) AS rank_ix
  FROM document_chunks WHERE metadata @> filter
  LIMIT least(match_count * 2, 50)
)
SELECT …,
  coalesce(1.0 / (rrf_k + full_text.rank_ix), 0.0) * full_text_weight
+ coalesce(1.0 / (rrf_k + semantic.rank_ix), 0.0) * semantic_weight
    AS similarity
FROM full_text
FULL OUTER JOIN semantic ON full_text.id = semantic.id …

Wins when

Jargon, identifiers, names and numbers mixed with natural language
General knowledge-base search
Chunks found by both arms rise to the top

Fails when

websearch_to_tsquery ANDs every term, so long questions often get no lexical hits and the result is semantic-only
ts_rank_cd is cover-density ranking with no IDF term, which is weaker than BM25
Rank fusion drops score magnitude, so a weak first place counts as much as a strong one

Result size: Up to k rows (max 30)
Cost: Same single embedding plus a full-text pass; still one round trip to Postgres.

ts_rank_cd and cosine similarity are on different scales. RRF combines ranks, so neither score needs normalizing. At rrf_k = 60, a chunk ranked first in both arms scores 2/61 ≈ 0.0328, and a chunk ranked first in one arm scores ≈ 0.0164.

Rerank is a separate stage controlled by one boolean. It can follow any of the four modes.
Mode	Signals	Result size	Extra cost vs similarity	Optimizes	Main failure mode
Similarity	cosine	always k (≤ 30)	none	semantic recall	exact terms; pads the prompt with noise
Hybrid · RRF	ts_rank_cd full-text and cosine, rank-fused	≤ k (≤ 30)	unindexed full-text pass	lexical and semantic recall	AND-semantics full-text falls back to semantic-only
MMR	cosine and chunk-to-chunk cosine	k	O(n²) over ≤ 50 candidates	coverage, diversity	drops relevant near-duplicates; λ needs tuning
Threshold	cosine ≥ τ	0 to k	none	precision, abstention	uncalibrated τ gives an empty or oversized context
+ Rerank	cross-encoder relevance	k	one more API call per search	precision@k	limited by first-stage recall (2k)

04Embeddings · chunking · pgvector

Embeddings, chunking and the vector store

One Supabase project held pgvector, full-text search, chat history, document metadata, file storage and auth. Having the vector index and the text index in the same database is what allowed hybrid search in a single SQL statement.

Embeddings

Cohere embed-english-v3.0, 1,024 dimensions

Chunks are embedded with search_document and questions with search_query. The reranker is the matching rerank-english-v3.0.

Chunking

Page-aware and idempotent

PDFs load as one document per page. A recursive splitter then cuts on paragraph, line, sentence and word boundaries, at roughly 500 to 1,000 characters with 10–20% overlap. Chunks never cross a page, so every citation has an exact page number. Ids are uuid5(document_id, chunk_i), so re-ingesting a document overwrites its chunks.

Vector store

pgvector with a metadata pre-filter

document_chunks holds the content, JSONB metadata, a vector(1024) with an ivfflat cosine index (lists = 100) and a generated tsvector. Every function filters with metadata @> filter before ranking, so top-k is computed inside the chosen dataset.

create table document_chunks (
  id         text primary key,          -- uuid5(document_id, "chunk_i")
  content    text,
  metadata   jsonb not null,            -- document_id, filename, page_number, …
  embedding  vector not null,           -- 1024-d, Cohere v3
  fts        tsvector generated always as (to_tsvector('english', content)) stored
);
create index on document_chunks using ivfflat (embedding vector_cosine_ops) with (lists = 100);

05Prompting · SSE

Generation and citations

On Nov 15 the model returned JSON containing its own citations, parsed with a Pydantic output parser plus cleanup code for invented or empty sources. On Nov 29 the citations started coming from retrieval metadata instead.

Prompt structure

systemDocument-analysis assistant for law, finance and biology. Prefer the provided context, say clearly when it lacks the answer, express uncertainty and never make up information. Answer in Markdown.
historyEvery prior turn in the session, loaded from Postgres.
humanContext: {retrieved chunks}Question: {question}

SSE events, in order

{"status": "Creating session..."}
{"session_id": …, "title": …}
{"status": "Retrieving context..."}
{"token": …} × n
{"content": …, "citations": [{document_id, filename, chunk_index, page_number}]}
{"status": "complete"}

06Decision records

Trade-offs

ADR 01

Retrieval strategies as Postgres functions

Chose: Four SQL functions, chosen per request
Over: Retrieval logic in Python

MMR first ran client-side and pulled embeddings into Python. With every strategy in SQL, each search is one round trip, filtering and fusion run next to the data, and the UI can switch strategies per message.

Trade-off The SQL was applied to Supabase by hand rather than through migrations, so it is harder to version and test.

ADR 02

Rank fusion for hybrid search

Chose: Weighted RRF, k = 60
Over: Normalizing both scores onto one scale

ts_rank_cd and cosine similarity are not comparable. Fusing ranks needs no calibration, and each arm keeps its own weight.

Trade-off Score magnitude is lost, so a barely relevant first place counts as much as a strong one.

ADR 03

Rerank as a separate stage

Chose: One boolean and a 2× over-fetch
Over: A dedicated reranked mode

Any first stage can be combined with the cross-encoder, which gives eight configurations from one field.

Trade-off One more API call per search. Recall is limited to the 2k pool; 3–10× over-fetch is more common.

ADR 04

Citations from retrieval metadata

Chose: DocumentReference built from chunk metadata
Over: LLM-written JSON citations

It removed parse failures and invented sources, and page-level links became deterministic.

Trade-off Attribution is per retrieved chunk rather than per claim.

ADR 05

Everything on Supabase

Chose: Postgres, pgvector, full-text search, Storage and Auth
Over: A dedicated vector database beside a relational one

Vectors, full text, chat history and documents live in one place, hybrid search is one statement, and there is one less system to run.

Trade-off No HNSW or ANN tuning by default. The functions ended up as exact scans, which is fine at demo scale.

ADR 06

Stateless, serverless-ready API

Chose: Lazy clients, per-request services, offline ingestion
Over: Shared in-process state and BackgroundTasks ingestion

Serverless functions can freeze after responding, so background ingestion and cross-request state were removed.

Trade-off It was packaged for Vercel Python and moved to Render the same day. The repo does not record why.

07Known gaps

What I’d fix

The project shipped as a demo in three weeks with no tests or evals. These are the changes I would make first.

01
Lexical arm
websearch_to_tsquery ANDs every term, so natural-language questions often get zero keyword hits. Use OR semantics or BM25, and query the stored fts column through a GIN index instead of recomputing to_tsvector per row.
02
Index-friendly vector SQL
Ordering by a computed similarity column prevents pgvector from using its index. Writing the functions as ORDER BY embedding <=> q LIMIT k lets them use one (HNSW at larger scale). The functions should also stop returning the 1024-d embedding on every row.
03
Greedy MMR
Penalize each candidate only against chunks already selected. The current query penalizes it against every higher-ranked candidate, which demotes a tight cluster as a whole. The top chunk comes first only because its score is NULL and NULLs sort first.
04
Token budget
tiktoken is a dependency but is never imported. History and context should be capped by tokens, especially in threshold mode, where the UI raises k to 100.
05
Generation settings
Lower the temperature from 0.7 toward 0 for document QA, and label each chunk in the context so the model can cite inline.
06
Streaming in the UI
The API streams tokens, but neither frontend renders them, so users wait for the whole answer.

08Not built

Evals I’d add

There were no evals. The trade-offs on this page come from reasoning about the algorithms and were never measured. The deterministic chunk ids make a golden set cheap to maintain. This is the loop I would build.

01
Golden set
50–100 questions per dataset, each labelled with gold chunk ids. Draft them with an LLM from sampled chunks and verify by hand. uuid5 ids keep the labels valid across re-ingestion.
02
Retrieval grid
Run 4 modes × rerank on/off, plus sweeps over λ, w_ft / w_sem / rrf_k and τ. Score recall@k, MRR, nDCG@k and context precision. For threshold mode, also score abstention on unanswerable questions.
03
Generation
Faithfulness of each claim to the retrieved context, answer relevance, and citation precision and recall.
04
Cost and latency
p50 and p95 latency and dollars per query for every configuration, reported next to quality.
05
CI gate
Run the suite before any change to chunking, models or SQL.

09Nov 11 – Dec 3, 2024

Timeline

Nov 11Ingestion, chunking, Cohere embeddings, vector search and rerank.
Nov 13Rebuilt on LangChain. Llama 3.2 11B on Together AI.
Nov 15SSE streaming. Citations returned as LLM-written JSON.
Nov 20–22Search types. MMR ran client-side first and moved into SQL. Hybrid RRF added.
Nov 27Serverless refactor. Ingestion moved offline. Hosting moved from Vercel to Render.
Nov 29Citations built from retrieval metadata.
Dec 2Threshold search.
Dec 3Production merge.

10Dependencies

Stack

Frontend

Next.js 15
React 18
TypeScript
Tailwind
shadcn/ui
react-markdown
Vercel

API

FastAPI
Uvicorn
Pydantic v2
Server-Sent Events
Render

Orchestration

LangChain 0.2
Custom BaseRetriever × 3
RunnableWithMessageHistory

Retrieval

Supabase Postgres
pgvector
Postgres full-text
Cohere embed v3
Cohere rerank v3

Four SQL functions: match_chunks, hybrid_search, mmr_search and threshold_search.

Generation

Together AI
Llama 3.2 11B Instruct Turbo

Data and auth

Supabase Auth (JWT)
Supabase Storage
PDFPlumber
RecursiveCharacterTextSplitter

Engineering portfolio