Home

Turns a BibTeX bibliography into grounded survey papers, thesis chapters, undergraduate textbook chapters and hands-on tutorials, with every citation traceable back to a paper the bibliography actually holds.

Named for the Hindu god who keeps the ledger of every deed and audits souls against it -- which is what this does to citations. See more.

---

The one rule

Fabricated placeholder references have made it into real papers before. This pipeline is built to make that impossible rather than unlikely:

A citekey may only be used if it appears in your own .bib export and was picked up into the ledger by a real parse of a real PDF.

• How it works
• Quickstart
• The enrichment layer
• Hardware requirements
• Documentation
• Acknowledgements

How it works

Five phases. You own phase 1, the corpus layer owns phase 2, the drafting layer owns 3 through 5, and nothing reaches phase 5 without passing phase 4.

Two properties of that picture do all the work:

• Phase 1 is the only entrance. Citekeys come from your reference manager's BibTeX export. The pipeline never fetches a paper, never invents a citekey, and never renames one.
• Phase 4 is the only exit. src.draft gate sits on the single path between a draft and a rendered document. There is no arrow around it, and a FAIL is treated like a failing test rather than a lint warning.

The loop back from a failed gate goes to drafting, not to you: the skill discards the unsupported claim and writes again. You only get involved in the rarer case where the paper genuinely isn't in the corpus yet -- the dotted arrow back to phase 1.

Eight skills sit behind phase 3, all obeying the same grounding rules: five that write a new draft -- survey, thesis chapter, undergraduate textbook chapter, tutorial, and a heavier multi-perspective deep-research mode -- and three that change one that already exists, because a draft is never revised by re-running the skill that produced it (docs/GENRE.md). The third of those, overlap-reviser, repairs the verbatim overlap a scan found and re-verifies each repair before keeping it. Two more layers sit outside these phases. Enrichment deepens the same corpus with layout-aware parsing, semantic search and topic clustering, and nothing above needs it. Review is what you run afterwards on a finished draft -- provenance, verbatim and coverage reports, all advisory, none of them a gate -- advisory means nothing blocks you, not that borrowed wording is fine to leave once you've found it.

docs/DIAGRAMS.md draws this workflow eleven ways -- by depth, by genre, and in time order -- and is where the figure above comes from.

One thing the corpus layer does not promise

The corpus layer is deterministic in the sense that matters most -- no LLM, no judgement calls, same bibliography in, same citekeys out -- but it is not bit-reproducible with every parser. With the default pdftotext backend it is: parsed text comes back byte-identical, every ledger column stable except the last_synced timestamp.

With the opt-in docling backend and a worker pool, it isn't -- and the instability reaches the quotable passage, so the exact span quoted from a source can change between runs. That is Docling's behaviour under load rather than something this pipeline adds, and it cannot be switched off; serial parsing ([parser].workers = 1, the default) has not been observed to vary. The artifact-by-artifact contract, the measured rates and how little they can pin down are in docs/ARCHITECTURE.md.

Quickstart

# 1. Export Zotero's library: format BibTeX, tick "Export Files", and save
#    it as `bibliography` inside papers/. Zotero writes the .bib plus a
#    companion attachment folder beside it:
#      papers/bibliography.bib
#      papers/bibliography/files/<id>/<name>.pdf
#    Each entry's file field is a path relative to the .bib, so don't
#    rename or move that folder afterwards -- see docs/ZOTERO.md.
#      Ex: file = {Full Text PDF:bibliography/files/16/paper-name.pdf:application/pdf}
mkdir -p papers && cp -r /path/to/your/export/. papers/

cp config.toml.example config.toml

# ...and, only if your field has its own acronyms (DT, FMU, ...) beyond
#    the PDF/CPU/URL/API/HTML every draft already gets: copy the template,
#    point config.toml's [style].acronyms at your copy, and every genre
#    skill drafts from it too. See assets/style/README.md.
# cp assets/style/acronyms.toml.example content/acronyms.toml
# # then edit [style].acronyms in config.toml to content/acronyms.toml

# 2. Install dependencies. scripts/install_full_pipeline.sh is the only
#    install path -- one script for a bare host and for the Docker image,
#    taking stage names as positional arguments:
#      python-deps  creates .venv-full/ and runs `poetry install --with
#                   enrich` into it. The default when no stage is given.
#      os-deps      apt-gets pdftotext, Pandoc, TeX Live, git/curl/unzip.
#                   Needs root; auto-sudo's. Opt-in.
#      dev-deps     pytest + pytest-cov, to run the test suite.
#      all          os-deps + python-deps (NOT dev-deps).
#    Poetry has to exist first -- either install it yourself, as here, or
#    let the os-deps stage do it.
pipx install poetry
bash scripts/install_full_pipeline.sh all

# ...and, only if you want to run the test suite:
# bash scripts/install_full_pipeline.sh dev-deps
# .venv-full/bin/python -m pytest

# 3. Sync the corpus layer from papers/bibliography.bib. A citekey that
#    later drops out of the bib file (a paper removed from your reference
#    manager) is only *reported* by default; re-run with --remove-stale
#    to actually delete its ledger row once you've reviewed the reported
#    list -- not needed on a first run. docs/ZOTERO.md has the full
#    semantics and why the default is to report rather than delete.
source .venv-full/bin/activate
python -m src.corpus sync

# ...and only once you've read the stale list it prints, and agree with it:
# python -m src.corpus sync --remove-stale

# 4. Inspect what it found. Read-only, takes no lock (so it works while a
#    sync is running), and needs no venv.
python -m src.corpus ledger

# 5. Optional, and only when you want it: the enrichment layer -- layout-aware
#    parsing, semantic search and topic clustering over the whole corpus.
#    Nothing else needs it and no skill builds it for you, so skip this on a
#    first run. What it costs and which stage is worth it: "The enrichment
#    layer" below, then docs/RETRIEVAL.md.

# 6. In Claude Code, ask for a draft, e.g.:
#    "write a survey section on digital twin composability"
#    "draft a thesis chapter on runtime verification for autonomous robots"
#    "write a textbook chapter introducing digital twin asset reuse"
#    "write a tutorial that builds a minimal digital twin asset from scratch"
# The matching skill in .claude/skills/ picks this up automatically,
# including its own gate -> references -> render chain (python -m src.draft <verb>)

Every command that chain runs, every way to re-run one by hand, and all three review-layer commands for checking a finished draft against its sources are in docs/CLI.md -- see The full first run, step by step, which walks the whole sequence above and everything that follows it, in order.

The enrichment layer

Everything above works without it. The enrichment layer is a second, optional pass over the same corpus that buys three things: layout-aware parsing that yields quotable passages, semantic search that finds a paper arguing your point in different words, and topic clustering over the whole corpus.

.venv-full/bin/python -m src.enrich --stages docling,embed

It costs real time and disk -- a first full-corpus parse is measured in tens of minutes, and the enrich dependency group is several gigabytes -- so you build it deliberately. No genre skill builds it for you. The skills read what is already there and fall back to the lightweight default when it isn't.

Which stage is worth that cost, and what each one actually answers, is in docs/RETRIEVAL.md. How the stages fit into the rest of the system, including how to call them from your own script or skill, is in docs/ARCHITECTURE.md.

No stage needs an LLM API key -- this repository intentionally has none. Every stage probes its own prerequisites and reports ok, skipped or missing-binary rather than assuming they are present.

Hardware requirements

What the pipeline needs, not what it was developed on. The split below is the one that matters: the corpus layer -- sync, the citation gate, keyword retrieval -- is light enough for any laptop, and the optional enrichment layer is what costs real disk and real time.

Resource	Minimum (corpus layer only)	Recommended (enrichment layer in regular use)
Disk	~1GB	10-20GB+ -- the full venv alone is 6.0GB (torch pulled in twice over via sentence-transformers/docling, plus docling's own layout/OCR models); TeX Live adds several GB more
RAM	~1-2GB	8GB minimum, 16GB+ better. At ~3GB free, Docling on a 17-page PDF pushed the process to 3.6GB RSS and the host swapped 6.3GB -- it finished, just slowly
CPU	1-2 cores	4+ cores -- without a GPU, Docling's layout inference and BERTopic's UMAP/HDBSCAN are CPU-bound, and more cores directly cut wall-clock time
GPU	none needed	none required. If one is present the installer detects it and torch is set up to use it automatically -- worth ~4.7x on the parse
Network	once, for poetry install	also for first-run model downloads (the embedding model, Docling's layout/OCR models)

For a sense of scale at the top end: this project's own bibliography -- 501 PDFs, 13,400 pages, 1.54GB -- parses in about 4 minutes on a 96-core machine with four A40s, against 1h 56m serially on that same host. On ordinary hardware a first full Docling parse is measured in tens of minutes. A second run over an unchanged corpus costs close to nothing either way, because every stage skips what hasn't changed -- which is what makes it safe to put sync on a schedule.

Every measured figure in this project comes from one of two reference machines: the small machine (4 cores, 9.7GB RAM, no GPU) and the multi-GPU machine (96 cores, 251GB RAM, 4x NVIDIA A40 -- the one in the paragraph above). Treat each figure as that machine's, and expect yours to differ. docs/PERFORMANCE.md has their full specifications, what each setting costs, and the two install-time traps worth knowing before you start (a CPU-only host pulling several GB of unused CUDA packages, and a GPU host where torch.cuda.is_available() comes back False).

📖 Research Citation

When Chitragupta is used in academic work, the following reference may be used:

@software{talasila2026chitragupta,
author = {Prasad Talasila},
title = {Chitragupta: An automated research pipeline for literature review and thesis drafting},
year = {2026},
url = {https://github.com/prasadtalasila/chitragupta},
publisher = {GitHub}
}

Documentation

This file is the overview: what the pipeline is, how to get it running, and what it needs. Everything else lives in one document per question, split by what you are doing -- using the pipeline, or working on it. Which of those you are doing can change within a session, and the split follows the task rather than the person; CLAUDE.md is the one-screen router for exactly that.

Using it

Getting started

Document	Answers
SOUL.md	One page: why this exists, the one invariant, and what it refuses to become
CLAUDE.md	One screen: which of the two agent guides applies to the task you are about to start, and the one rule that binds both
AGENTS.md	The rules an agent drafting with this pipeline must follow -- above all, never fabricate a citekey
docs/GENRE.md	Which of the nine skills writes what? How to pick a genre, what each one refuses to do, and why changing an existing draft never goes back through the genre skill
docs/ZOTERO.md	How do I get my library and its PDFs into the shape this expects? Includes the attachment-path trap that silently leaves every entry without a PDF
docs/CLI.md	What commands are there, what flags does each take, and which interpreter does it need?
docs/CONFIG.md	What settings exist, what values does each accept, and what is the default? Starts with a minimal config.toml. Includes [parser].backend, which decides how faithfully your PDFs are read

Understanding the system

Document	Answers
docs/ARCHITECTURE.md	What actually runs, what does each part write, which parts are optional, and why do some commands need the venv?
docs/DIAGRAMS.md	The workflow drawn eleven ways -- six by depth, three by genre, two in an appendix. Pick the one that matches what you already know
docs/LADDERS.md	Where does the pipeline choose between two ways of doing one job? Every ladder it walks for you and every tier you pick yourself, and what the bottom rung costs
docs/RETRIEVAL.md	BM25, embeddings, topic models -- which one answers my question, and which is worth building?
docs/REJECTION.md	Why is turning a source down the judgment this pipeline is most careful about? The reasoning behind a retrieval change that was built and then withdrawn, and what was kept from it
docs/TOKENS.md	Where do a run's tokens actually go, which of them get billed once and which get billed every turn, and how do I measure that without paying for a full run?
docs/DRAFT-ITERATION.md	What does a draft's dossier hold, and how do I change a draft weeks later without re-running the pipeline that produced it?
docs/PROMPTS.md	What does the prompt sent to the model actually contain, layer by layer -- for a single-context genre skill and for the multi-agent deep-research skill -- and why don't the two look the same?

Choosing settings

Document	Answers
docs/PERFORMANCE.md	What does each setting cost? Every measured figure in one place, organised by setting
docs/PDF-PARSER.md	Which PDF backend should I use, why were two dropped, and why was each newer candidate not adopted?

Reading the output

Document	Answers
docs/CITATION-PROVENANCE.md	What does the provenance report say, and how do I read it?
docs/PLAGIARISM.md	How much of a draft's wording came from its sources? What the verbatim overlap/scan checks catch, and -- just as important -- what they cannot see, since these drafts are LLM-written and the tier that catches a genuine restatement does not run everywhere
docs/WRITING-STANDARDS.md	What prose standards do the genre skills follow, and where in the technical-communication literature do they come from?

Working on it

Document	Answers
docs/DESIGN.md	Why does this refuse what it refuses? The hard constraints, the conflict policy when two runs collide, and the failure analysis behind both
docs/REQUIREMENTS.md	What must a grounded long-form writing system do, how does the closed- and open-source landscape stack up against that bar, and where does this pipeline stand against its own requirement set -- what's built, what was measured and declined, and what's left?
docs/PARALLELISM.md	How does the parallel parse actually work, what is each component for, and what is planned next?
docs/GROBID-CITATION-GRAPH.md	A proposal, not a plan. What would it take to build a corpus-internal citation graph, and is it worth a JDK and a long-running service?
docs/AUTO-IMPROVEMENT.md	Unbuilt. If the pipeline assembled its own worklist and attempted the mechanical repairs, what exactly would be built, and what would it have to satisfy? Normative, and carries no argument
docs/AUTO-IMPROVEMENT-RATIONALE.md	Why that loop, and where its line falls: why every quality signal here currently ends in prose a human must act on, what a machine may never repair, and the one documented rule this cannot satisfy without the user's approval
docs/HOUSE-STYLE.md	Why prose is the axis a machine improves best, why a readability score is the wrong target, and which of your preferences should outlive the draft that prompted them
DEVELOPER.md	How do I run the tests, where does everything live, and what is unbuilt?
DOCKER.md	How do I run this in a container?
DEVELOPER-AGENTS.md	The rules an agent changing this repo must follow -- test policy, the local check suite, code standards, commit/PR/release conventions
docs/CODE-STANDARDS.md	What must the code itself look like? The clean-code checklist mapped rule by rule, the two size rules that are machine-checked as a ratchet, why they count statements rather than lines, and why the rest is left to review
docs/INSPIRATION.md	What did this project borrow, and from whom? Every external idea, what was taken, and -- where the licence requires it -- what was deliberately not
docs/EXPORT-ZOTERO-GROUPS.md	Discouraged, and says so. How the one script that reads zotero.sqlite directly recovers collection labels when Better BibTeX cannot, which two project rules it bends to do it, and why you should use Better BibTeX instead

Every prose document ships in the release archive -- everything under docs/, plus SOUL.md, CLAUDE.md, AGENTS.md, DEVELOPER-AGENTS.md and DEVELOPER.md -- as do .claude/'s genre skills. Only this repo's own machinery stays behind: tests/, bench/ (the measurement harness and its raw timings), .github/ and .gitignore.

Acknowledgements

This project borrows from several others -- the deep-research skill's 7-phase method, the clean-code checklist its own code standard is written against, and the harness-engineering reading list behind much of .claude/. Each is credited, with what was taken and what deliberately was not, in docs/INSPIRATION.md.