Open source · MIT · CPU-only

RAG-ready Markdown, without the site chrome.

ChromeRAG turns scraped HTML into clean Markdown for retrieval-augmented generation. It strips navigation, cookie banners, calls to action and footers, can learn a site's repeated template from a few pages, keeps Schema.org metadata as front-matter and tables as key–value rows, and warns you when a page is an unrendered JavaScript shell.

$pip install chromerag Get started See the benchmark

Python ≥ 3.11 No model download Works offline Test suite

Balanced F-score, public benchmark

–

loading…

Retrieval hit@5 (BM25)

–

loading…

Chrome in retrieved context

–

loading…

Quick start

HTML in, Markdown out. ChromeRAG never fetches pages or runs JavaScript: your crawler or headless browser stores the HTML, and ChromeRAG cleans it before chunking and embedding.

# one page → Markdown (front-matter, key–value tables; --json-meta prints diagnostics)
chromerag extract page.html -o page.md --priority coverage

# learn a site's repeated chrome from ≥ 3 pages, then convert the whole folder
chromerag learn pages/ -o site.json --min-pages 3
chromerag batch pages/ -o out/ --chrome-model site.json

# exit code 3 on JavaScript shells, so a pipeline can render them first
chromerag extract page.html -o page.md --fail-on-thin

from chromerag import ChromeRAG, ContentPriority, PipelineConfig

extractor = ChromeRAG(config=PipelineConfig.from_priority(ContentPriority.COVERAGE))
result = extractor.extract(html, url="https://docs.example.com/docs/pricing")

result.markdown       # clean Markdown
result.front_matter   # title, type, dates, breadcrumb from JSON-LD / Microdata
result.warnings       # e.g. the input looks like a JavaScript-rendered shell
result.diagnostics    # what was removed, and why

How it works

Five stages, each a small module. Everything is deterministic and runs on CPU; the vector filter uses feature hashing, so there is nothing to download.

Input-quality gate

Flags thin pages and JavaScript shells before anything is removed.

Schema harvest

JSON-LD and Microdata become YAML front-matter.

Clean + site template

Strips scripts; applies a learned site model if you give it one.

Structural + density prune

Landmarks, link density and a lexical boilerplate filter.

Tables + Markdown

Tables become Row n -> header: value lines.

chromerag learn (optional) finds blocks that repeat, with the same text, on most pages of a site section and look like chrome. At extraction time a match is removed only if it is close to its learned size and does not hold the page's main content.

Public benchmark

Every tool runs on the same stored HTML. Recall is the share of main-content 5-grams kept, noise retention the share of navigation, header and footer 5-grams kept, and F_bal their harmonic balance. The scored pages are chosen from the input HTML only, never from any tool's output.

Method	Recall ↑	Noise ret. ↓	F_bal ↑	Avg tokens

Is the difference real? Paired bootstrap, 95% CI

Mean per-page difference, ChromeRAG coverage minus the baseline, 2,000 resamples. An interval that does not cross the zero line is a significant difference.

Does it help retrieval?

Each tool's output is split into ~200-word chunks and indexed with BM25, as a RAG ingest step would. Queries are page titles, section headings and content passages taken from the input HTML; each has exactly one correct page.

Hit@5 higher is better

Chrome in top-5 context lower is better

Learning a site's template

Site-template chrome elimination (STCE) learns blocks that repeat across pages of one site section. It is evaluated on the benchmark's sites with at least three pages and on a separate crawl of up to 15 pages per documentation site.

Page set	Sites	Scored pages	Recall without → with	Pages changed	Site-repeated text
loading…

Page explorer

Every benchmark page with every tool's score, so you can check individual cases instead of trusting the averages.

Reproduce, limits and tests

Reproduce every number

pip install -e ".[dev,baselines]"
python scripts/revalidate_corpus.py         # offline: recompute every mean from the published per-page scores
python -m poc.run_corpus_comparison         # fetch the public URLs and score every tool (network)
python -m poc.run_retrieval_eval            # BM25 retrieval over each tool's chunks
python -m poc.crawl_site_groups --fetch     # multi-page crawl for the site-template study (network)
python -m poc.run_stce_eval --raw data/stce_crawl

Live pages change, so a fresh fetch will not match the published numbers exactly. The published per-page scores are in data/corpus_comparison_report.json.

Limitations

ChromeRAG does not run JavaScript. It warns about shell pages but cannot recover their content.
The benchmark uses structural anchors, not human-labelled spans. Pages without landmarks are under-represented, and extractors that also use landmarks may have an advantage.
Most benchmark pages are documentation; other page types are too few for separate conclusions.
The boilerplate filter compares lexical hashed vectors, not semantic embeddings, and retrieval is evaluated with BM25 only.
Only HTML is handled. Use other converters for PDF and Office files.

Test suite

Cite

@software{peddapudi_chromerag,
  author  = {Peddapudi, Bhargava Chary},
  title   = {ChromeRAG: Ingest-Time Elimination of Site Template Noise for Enterprise Web RAG},
  url     = {https://github.com/pedapudibhargav/ChromeRAG},
  license = {MIT},
  doi     = {10.5281/zenodo.22970289},
  year    = {2026}
}