Vector retrieval drift under corpus growth

Research goal

Enterprise RAG systems keep adding documents. Practitioners often assume dense retrieval quality stays roughly stable if the gold answer document remains in the index. This study asks whether that holds on a controlled enterprise bench, and what metadata can rescue.

1. Measure drift. Hold a fixed primary-200 question bank and gold-pinned ladder; grow the distractor corpus from 5k → 100k (9 points) and track Hit@1 / Hit@5 / Hit@10 / MRR under raw dense search.
2. Measure rescue. Compare the same ladder with lightweight metadata conditioning; report Δ_meta and whether a collapse onset N★(τ = 0.10) appears on this bank.
3. Replicate on a second embedder. Rerun the identical gold-pinned ladder with Amazon Titan Text Embeddings V2 and test whether endpoint decline (5k→100k) is negative for Hit@1, Hit@10, and MRR.
4. Separate embedding drift from ANN error. Exact cosine controls show whether HNSW recall loss — not the embedder — drives apparent OpenAI–Titan gaps.

Protocol constants, seed 42, and headline results: STUDY_PROTOCOL.md. Threats: THREATS_TO_VALIDITY.md.

Primary findings (OpenAI, n = 200)

On text-embedding-3-small with HNSW (m = 16), raw dense Hit@1 falls 0.600 → 0.305 and Hit@10 0.795 → 0.510 as N grows from 5k to 100k. MRR falls 0.658 → 0.365. Metadata Hit@10 lift stays ≥ +0.13 from 10k–100k (N★(τ = 0.10) is undefined on this ladder; last Δ_meta≥0.10 point is 100k).

Log-linear fit for raw Hit@10: a − b·log(N) with a≈1.474, b≈0.086 (erb_full_primary200_to100k_fit.json. Promotion record: CANONICAL_METRICS.json.

Titan V2 — second-embedder replication

Same gold anchors, same scales, same primary-200 questions. Titan model: amazon.titan-embed-text-v2:0. Status: complete (2026-09-23). Raw Hit@1, Hit@10, and MRR all decline 5k→100k on both embedders (all_three_replicate = true).

Exact cosine vs HNSW (reviewer-critical)

On the deployed HNSW curves, Titan’s 5k→100k Hit@10 drop is milder than OpenAI’s (paired bootstrap B = 5000, seed 42: Titan−OpenAI drop ≈ +0.110, 95% CI [+0.040, +0.175]). Under exact cosine search that gap shrinks to ≈ +0.035 [−0.025, +0.100] — statistically indistinguishable.

• Titan exact search returns identical top-10 lists to Titan HNSW for 200/200 questions at 5k, 50k, and 100k (Δ = 0 disagreements).
• OpenAI HNSW loses Hit@10 vs exact by 0.085 at 50k and 0.075 at 100k.
• Takeaway: do not cite “Titan drifts less” as an embedder property without the exact control; most of the HNSW gap is ANN recall loss on the OpenAI index.

Sources: openai_vs_titan_primary200.json, openai_vs_titan_bootstrap.json, exact_control_titan.json, TITAN_A8_STATUS.md.

Static JSON for reviewers

Everything below is served from this GitHub Pages site (no database required). Start with landing_summary.json for headline numbers, then drill into the comparison files.

Full multi-MB sweep dumps (erb_full_primary200_to100k.json, erb_titan_primary200_to100k.json) are included on Pages when present in the deploying clone. Embeddings live in the GitHub Release Postgres dump (~1 GB), not on this static site.

Reproduce (no re-embedding required)

Download the published Postgres dump, restore it, and re-run sweeps against frozen vectors. You do not need to re-embed 100k documents.

1. Clone the repo and create .env from .env.example.
2. Start Postgres + API: docker compose up -d --build vector-drift-postgres vector-drift-api
3. Download erb_tables.dump.gz from the latest GitHub Release into data/backups/.
4. Restore: ./scripts/erb/restore_from_backup.sh data/backups/erb_tables.dump.gz
5. Validate: python3 scripts/erb/validate_study.py
6. Optional OpenAI re-sweep: python /app/scripts/erb/run_scale_sweep.py --primary-questions … (see REPLICATE.md). Titan tables: document_chunks_titan / vector_drift_results_titan.

Step-by-step: REPLICATE.md. L4 audit (LLM-assisted + author spot-check): HUMAN_AUDIT.md.

Cite & links

@misc{peddapudi2026vectordrift,
  title  = {Vector Retrieval Drift under Corpus Growth: A Controlled Empirical
            Study with Metadata Filtering on EnterpriseRAG-Bench},
  author = {Peddapudi, Bhargava Chary},
  year   = {2026},
  note   = {Manuscript submitted to IEEE Access.
            Artifacts: https://pedapudibhargav.github.io/vector-drift-study/}
}

• Source repository
• LaTeX paper (papers/ieee-vector-drift/)
• EnterpriseRAG-Bench: Sun et al., arXiv:2605.05253
• Pages deploy: push to main → workflow Deploy GitHub Pages → branch gh-pages