Developer guide
Material for working on this repository itself, as opposed to using it to
draft content -- test running, the full source layout, and known gaps.
See README.md for the user-facing Quickstart/Configuration/
Architecture docs and DOCKER.md for the container build.
Table of contents
Running tests
| # Install pytest/pytest-cov into the same venv (run python-deps first)
bash scripts/install_full_pipeline.sh dev-deps
# Run the full suite with coverage
.venv-full/bin/python -m pytest --cov=src --cov=scripts --cov-report=term-missing
# Same, on a host without pandoc/TeX Live/poppler: the render tests skip,
# so opt out of the 100% bar (pyproject's fail_under) rather than lower it
.venv-full/bin/python -m pytest --cov=src --cov=scripts --cov-report=term-missing \
--cov-fail-under=0
|
tests/ covers both the corpus layer and src/enrich/* -- the enrich group's
dependencies (docling, chromadb, bertopic,
sentence-transformers) are mocked via sys.modules for fast,
deterministic unit tests, so the
dev-deps group alone is not enough on its own: the enrich group
(python-deps, step 1 of Quickstart) must already be installed too, since
tests/test_bib_reader.py needs bibtexparser and the src/enrich/ test
modules need docling/chromadb/bertopic/sentence-transformers. A handful of tests
(tests/test_feature_workflows.py, the TestRenderReal/TestExtractTextReal
classes elsewhere) run the real pdftotext/pandoc/pdflatex binaries
end to end rather than mocking them, and skip automatically if those
aren't on PATH.
Benchmarking the parser
bench/ measures what a full docling parse of the bib corpus costs on
a given machine, and is deliberately kept out of tests/: it takes a couple of
hours, needs real PDFs and a GPU, and answers a "how long / what's the
bottleneck" question rather than a pass/fail one. It is excluded from the
release zip for the same reason tests/ is.
- docs/PERFORMANCE.md -- what each setting costs,
organised by setting. Ships in the release archive, unlike
bench/
- docs/PARALLELISM.md -- parallel parse design:
architecture, components, and the roadmap
- bench/README.md
-- how to run it, and what each
switch measures
- bench/RESULTS.md
-- the dated measurement record,
newest last, with raw per-run data in
bench/results/. Read its
"Which sections are current" table first: several early conclusions
were overturned by later runs and are kept, marked, rather than deleted
- bench/PARALLELISM-PLAN.md
-- what is still
unknown, and what to measure before changing it
The headline, in the order it was found:
- Parsing all 501 bib PDFs with
docling took ~1.6 hours (later
measured at 1h 56m), with the A40 at ~7% utilization and three CPU
cores of 48 busy. The GPU was worth only 1.79x over CPU-only -- the
work was CPU-bound.
- Turning OCR off (v0.12.0) was worth more than the GPU: 2.08x
serially, 3.91x at 12 workers and 4.79x at 24, since OCR competes
for the same CPU the parallelism needs. (An earlier 2.46x, from a
16-PDF serial sample, is still quoted in older text; it estimated the
serial case only.)
- Parallelising
sync (v1.0.0) was worth 3.60x at four workers.
- That moved the bottleneck onto a single GPU:
AcceleratorDevice.AUTO
resolves to cuda:0 in every worker, so GPU 0 ran at 100% while
GPUs 1-3 idled.
- Giving each worker its own card (v1.1.0) was worth a further 1.62x
on the full corpus -- 528s to 326s at twelve workers. The whole
501-PDF corpus now parses in 5m 10s, against 1h 56m where this
started.
- Per-worker startup (v2.1.0) turned out to be 3.2s of importing torch
and docling plus ~5s of loading Docling's models, and only the first
is shareable between processes. A forkserver pool with those modules
preloaded, started before the bibliography is read, takes a fixed
~1.5-2s off pool startup -- 9.6% of an 8-document run, 2.5% of a
60-document one.
- Measuring the whole corpus instead of extrapolating from a 16-PDF
sample (2026-08-04) found the serial baseline was 55m 30s, not the
~39m every document had quoted -- 41% low. Correcting it showed
12-worker efficiency is 89%, not the 60% previously reported, and that
worker_ceiling()'s cpus // 4 clamp costs 1.41x: 32 workers
beat the 12 it allows.
- Asking whether a quotable passage survives a re-parse (2026-08-07,
bench/repro_check.py) found that ~1% of documents come back with a
different passage text, and -- correcting what this project had
asserted twice -- that two runs of the same configuration are not
exempt either. The artifact-by-artifact contract that came out of it
is in docs/ARCHITECTURE.md.
The lesson worth carrying: every one of those steps was measured, and
seven intermediate conclusions were wrong until the next measurement
corrected them -- including two that sat in the code as stated fact, and
one that had been written into three documents. bench/ exists so that
the next one is checked too.
Writing a script that drives the enrichment layer
src.enrich.docling_parse.parse_corpus and python -m src.corpus sync both use
a worker pool when [parser].workers is above 1, and every start method
they can pick (forkserver or spawn -- see [parser].start_method)
re-imports the calling program's __main__ in each worker. Any script of
your own that calls them must guard its top level:
| if __name__ == "__main__":
main()
|
Without it, every worker re-runs the script on startup and the pool dies
with BrokenProcessPool. src/enrich/__main__.py and src/sync.py
are both guarded already; this only bites ad-hoc scripts, and it bites
immediately rather than subtly.
Repository layout
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188 | README.md the user-facing overview: what this is, the Quickstart, hardware sizing
bench/ parser measurement (dev-only, not shipped) -- see "Benchmarking the parser"
above; corpus.json/sample*.json are generated and gitignored, results/ is
committed evidence
bench_docling.py backend extraction timings, one process
sweep_sync.py the real `python -m src.corpus sync` swept over worker/GPU counts -- the harness
every pool-level figure must come from
run_parallel.py independent-process baseline; answers a different question to sweep_sync.py
make_corpus.py builds the gitignored work lists from your own bib file
repro_check.py compares two parses at three levels (bytes, passage spans, passage texts)
and self-checks its own detector on every run
SOUL.md one page: why this exists, the one invariant, what it refuses to become
AGENTS.md instructions for agents drafting *with* the pipeline -- the citekey
invariant, the four layers, retrieval
DEVELOPER-AGENTS.md instructions for agents changing *this repo* -- install notes, dev
process, commit/PR/release conventions
DEVELOPER.md this file -- test running, repo layout, open questions
DOCKER.md running this repo in a container (docker/Dockerfile)
.github/ CI/release workflows, plus the issue and PR templates GitHub picks up
automatically and RELEASE_TEMPLATE.md (copied by hand)
docs/ reference docs that ship in the release zip -- everything except the
root-level ones above, which stay put because they're what a reader looks
for first. Every file here opens with a status line on the third line,
`Status: **<kind>.** Written <date>.`, so a reader can tell what they are
holding before reading it -- one of:
reference. what exists and how to look it up
how-to. a procedure to follow start to finish
implemented. a built subsystem, described as built
reasoning document. why a decision went the way it did, kept
because the reasoning outlives the decision
measurements. numbers from a real run on a named machine
a proposal, not a plan. nothing here is built; the decision is open
A new document picks one of those rather than inventing a sixth. The date
is when it was written, not when it was last touched -- git already
records revisions, and a hand-maintained "last revised" goes stale
PARALLELISM.md parallel parse design: architecture, components, and the roadmap
PERFORMANCE.md what each config setting costs, measured -- the lookup-oriented companion
to PARALLELISM.md's design doc
ZOTERO.md getting a bib file and its PDFs into the shape this pipeline expects
EXPORT-ZOTERO-GROUPS.md discouraged: recovering collection labels by reading zotero.sqlite
directly, when Better BibTeX cannot produce them -- carries the
warning about which project rules that bends
CLI.md every command, and which interpreter each one needs
CONFIG.md every setting, with config.toml.example reproduced in full
PDF-PARSER.md parser backend tradeoffs, why grobid/markitdown were removed, and why
marker/surya/xberg/unstructured were surveyed and not adopted
GROBID-CITATION-GRAPH.md a proposal, not a plan: what a GROBID stage alongside docling would
buy (a corpus-internal citation graph) and what it would cost
ARCHITECTURE.md what runs, what each part writes, what is optional, and which interpreter
each command needs -- the user-facing companion to DESIGN.md
RETRIEVAL.md BM25 vs embeddings vs topic model: which answers what, and what to build
DESIGN.md architecture and design decisions -- the rationale, not the map
DIAGRAMS.md the workflow drawn eleven ways; the fenced mermaid blocks are the source
diagrams/ the same eleven as standalone files, for use outside this repo
*.mmd mermaid sources with a title line
svg/*.svg rendered exports (mmdc -b white -w 1900). Exports only -- edit the
fenced block in DIAGRAMS.md, then re-render
CITATION-PROVENANCE.md what src/review/citation_provenance.py reports and how to read it
PLAGIARISM.md what src/review/verbatim_check.py's overlap/scan modes catch and don't
(verbatim reuse only, paraphrase is a later tier), the n-gram
fingerprinting technique and its literature sources, and a measured
docling-vs-pdftotext backend comparison
DRAFT-ITERATION.md what a dossier holds, and how a draft is revised weeks later without
re-running the pipeline that produced it
TOKENS.md where a run's tokens go -- the resident/one-shot pools, two worked
examples, and how to measure it without paying for a full run
GENRE.md the nine skills in .claude/: which writes what, how to pick, and what
each one refuses to do
LADDERS.md every automatic fallback chain the code walks, and every tier you pick
yourself -- and what the bottom rung of each costs
WRITING-STANDARDS.md the prose standards the genre skills share, and their sources in the
technical-communication literature
NAME.md where "chitragupta" comes from
logo.svg, logo-dark.svg the README banner, light and dark
LICENSE MIT
assets/ data files the pipeline reads at runtime, tracked and shipped
csl/ieee.csl the CSL style pandoc formats citations with ([render].csl default).
Vendored byte-identical to the CSL project's own release (CC BY-SA 3.0)
so it can be re-fetched and diffed -- do not edit it in place; the one
attribute this project needs is injected into a temp copy at render
time (see assets/csl/README.md and render_output._collapsed_csl)
csl/README.md the vendoring policy, upstream URL and sha256
.github/workflows/ ci.yml (test suite + coverage + poetry check, on push/PR) and release.yml
(on a v* tag: verifies tag matches pyproject.toml's version, builds
scripts/release.py's zip, publishes it to a GitHub Release)
config.toml.example tracked template for the central config -- paths, parser backend, worker
count, embedding model. Copy to config.toml (gitignored, per-host) before
anything imports src.config; see docs/CONFIG.md
papers/ gitignored, per-host data -- not shipped in the repo
bibliography.bib BibTeX export -- source of truth for citekeys/metadata (config.toml's [bib].path default)
bibliography/ the export's companion attachment folder, referenced by each entry's file field
pyproject.toml Poetry config (dependency/lockfile manager only, package-mode = false --
no [build-system], nothing published) + pytest/coverage tool config
poetry.toml project-local Poetry config: virtualenvs.create = false (installs into
whatever venv VIRTUAL_ENV points at, e.g. .venv-full/, instead of Poetry's own)
poetry.lock resolved dependency versions -- regenerate with `poetry lock` after editing pyproject.toml
mkdocs.yml the documentation site published at prasad.talasila.in/chitragupta. `docs_dir: .`
on purpose: the site is this repository as it stands, so every cross-document
link works unchanged and there is no staging step to keep in sync. Built by
.github/workflows/docs.yml from pyproject.toml's optional `docs` group
(`poetry install --only docs`). Read its header before editing
src/ the corpus and drafting layers (sync needs bibtexparser;
citation_gate/references need nothing)
config.py loads config.toml, env var overrides
runlock.py one-writer-at-a-time lock over content/, held by `src.corpus sync` and
`src.enrich` -- the only two commands that write to it;
a dedicated sqlite file, so a killed holder releases it with no
staleness check and readers are never blocked
bib_reader.py parses bibliography.bib -- the only citekey source
ledger.py per-citekey status tracking (content/ledger.sqlite); find_stale/prune_missing
detect/remove rows for citekeys no longer in the bib file. Also persists each
entry's formatting-relevant BibTeX fields (bib_fields, JSON), so references.py
can build a full bibliography entry without reading the bib file itself
pdf_text.py PDF text extraction, dispatched to pdftotext/docling by config.PARSER; also the parse-quality guard
sync.py orchestrates the above -- the corpus layer's `sync` verb; --remove-stale opts into
deleting stale ledger rows (default: report only, see README's "Removing a paper")
corpus.py the corpus layer's single entry point, `python -m src.corpus sync|ledger`.
sync.py and ledger.py carry no __main__ block of their own. Imports the verb
it was given rather than both, so asking for `ledger` never pays for sync's
bibtexparser -- see docs/ARCHITECTURE.md on why that one is not like the
other dispatchers
draft.py the drafting layer's single entry point, `python -m src.draft
gate|dossier|retrieve|references|render`; same rule, same reasons
dedup.py advisory near-duplicate citekey detection (shared DOI/title), called from sync
retrieval.py BM25 search over the corpus layer, backed by a cached term-frequency index.
`search` ranks and returns a snippet -- the best-covering passage for the
query, and the same one every run; `evidence` reads more of one document
when a snippet is not enough to judge it. A lookup, not a stage: see
docs/REJECTION.md for the two-stage read that was built and withdrawn.
`--log` records each call's payload in the dossier
passages.py where a citekey's supporting text comes from (docling sidecar -> form-feed
pages -> pdftotext) and whether it may be quoted -- shared by the consumers
that need to point at part of a source rather than all of it
overlap_index.py disk-cached word n-gram fingerprint index (content/overlap/) for
src/review/verbatim_check.py's overlap and scan modes -- one .fpr file per citekey plus
a merged, binary-searchable corpus-wide index.bin, both keyed by
(pdf_hash, parsed-file stat) so a re-run over an unchanged corpus costs no
re-fingerprinting. Read-only over the corpus layer, no writer lock
citation_gate.py hard citation-verification gate -- the drafting layer must pass this
dossier.py the working state behind a draft (reader, scope, kept evidence, rejected
candidates, steering, revision log) as Markdown under content/dossiers/,
mirroring the draft's path; plus tar.gz backup/restore. Read-only over the
corpus layer, never a gate -- see docs/DRAFT-ITERATION.md
references.py auto-generates a draft's "## References" section from its own cited citekeys,
as numbered IEEE entries ordered by first appearance -- the same order (and
so the same numbers) pandoc's citeproc assigns when the draft is rendered
render_output.py Pandoc/TeX Live rendering + standalone CLI -- stdlib-only, no enrich group
needed, which is why it sits here and not in src/enrich/. `--format md` on a
Markdown draft skips pandoc entirely and emits references.numbered_markdown's
plain numbered copy instead
src/review/ the review layer -- one command, `python -m src.review <aid>`, three aids
__init__.py the layer's shared output contract -- report path (content/review/,
mirroring the draft), the "not a gate" banner, the header, and the
write-md-then-render routine all three aids use. No timestamp, so a
report diffs across revisions
__main__.py the layer's single entry point: one parser, three subcommands, each
wired to its aid's own build_parser()/run(). The aids below carry no
__main__ block of their own -- see docs/ARCHITECTURE.md on why a layer's
command surface stays one level deep
citation_coverage.py `coverage` -- retrieval-candidates-vs-actually-cited report, not a gate
citation_provenance.py `provenance` -- what in each cited source supports the claim citing it,
not a gate (scores claims against passages.py's ladder; see
docs/CITATION-PROVENANCE.md)
verbatim_check.py `verbatim` -- per-citekey overlap, whole-draft x whole-corpus scan, and
page-locating checks against sources
src/enrich/ the enrichment layer (pyproject.toml's "enrich" Poetry group), optional
__main__.py the layer's entry point, `python -m src.enrich --stages …`; orchestrates
the stages below, which carry no __main__ block of their own
corpus.py the enrichment layer's view of the ledger -- one CorpusDoc per bib item,
so every enriched document is citable, keyed by its citekey
docling_parse.py, embed_index.py, topic_model.py
scripts/ dev tooling only -- no layer entry point lives here
install_full_pipeline.sh single staged install path (os-deps/python-deps/dev-deps/all) for host + Docker
release.py bundles a distributable release/chitragupta-<version>.zip, dev files excluded
tests/ pytest suite -- unit tests per module + end-to-end feature tests (see "Running tests")
content/ generated, gitignored (regenerate with sync)
ledger.sqlite, parsed/<citekey>.txt, drafts/, dossiers/, rendered/, review/,
retrieval_index.json, overlap/,
docling/, chroma/, topics.json, topic_embed_cache.json (src/enrich/ outputs)
logs/ gitignored -- pipeline.log, rotated at 5MB x 5 backups. Level from
config.toml's [logging]; relocate with the LOGS_DIR env var
.claude/skills/ drafting layer: survey-writer, thesis-chapter-writer,
textbook-chapter-writer, tutorial-writer, deep-research
.claude/agents/ deep-research's subagents: deep-research-interviewer, deep-research-writer, peer-reviewer
.claude/hooks/ citation_gate_hook.py -- PostToolUse hook, mechanically enforces citation_gate on
every Write/Edit under content/drafts/*.md and *.tex (see AGENTS.md)
.claude/settings.json wires the hook above into the PostToolUse event
docker/ Dockerfile (TeX Live/Pandoc/Poetry) -- unverified end-to-end, see DOCKER.md
|
With [enrich].docling_images on (off by default), the Docling stage writes
each paper's figure bitmaps to content/docling/<doc>_artifacts/ and an
index of them to content/docling/<doc>.figures.json.
Those images are a reading aid, not draft content. Nothing in this
repo inserts them into content/drafts/, and nothing should start doing
so. A figure's copyright belongs to the publisher or the authors, and
citing a paper grants no right to reproduce its figures -- citation_gate
gates citekeys, and there is deliberately no equivalent gate for
images. The ledger also has no license column, so the pipeline genuinely
cannot tell a CC BY paper from an all-rights-reserved one; that judgment
stays with you, per figure.
The supported way to reference a figure is therefore textually, and
each record in <doc>.figures.json carries a ready-to-paste cite
string:
| {
"page": 8,
"caption": "Figure 3. Subdivision of the entry process of a Digital Twin",
"cite": "Figure 3 of [@richstein_characterizing_2024], p.8",
"image": "richstein_characterizing_2024_artifacts/image_000005_....png"
}
|
Two details worth knowing about that cite string:
- The number comes from the caption's own text, never from the
picture's position. Publisher logos and licence badges are pictures
too -- on a real 17-page MDPI paper, 6 of the 13 extracted pictures
were furniture rather than figures -- so the Nth picture is routinely
not the paper's Figure N.
- The number is captured whole, including chapter-scoped forms
(
Fig. 1.1 ... Fig. 1.4, the convention in edited book chapters)
and sub-figure letters (Figure 2a). Matching only the leading
integer would collapse a chapter's four distinct figures onto one
Figure 1 -- a citation pointing at the wrong picture.
- A picture whose caption carries no number is cited by page instead
(
"the figure on p.1 of [@key]"), rather than being given a number
this repo would have to invent. Two panels of one figure (captions
beginning (a) / (b)) therefore share a page-based citation; that
is the fallback behaving correctly, not a collision.
Every figure's cite string is a real [@citekey], because every
document the enrichment layer parses comes from the bib file (see
src/enrich/corpus.py).
A wholly original diagram -- not derived from any source paper's figure
-- is a different case this section doesn't restrict; see
docs/WRITING-STANDARDS.md §10 for the
supported form and why it's plain ASCII rather than Unicode
box-drawing.
Citation provenance
python -m src.review provenance content/drafts/<slug>.md reports, for
every citation in a draft, what in the cited source supports it and where
-- ordered worst match first. It writes
content/review/<the draft's path minus its suffix>.provenance.md plus
.tex/.pdf renders beside it. The report mirrors the draft's own place
under content/drafts/, the same rule rendered/ and dossiers/ follow
(config.mirrored_dir), and src/review/__init__.py owns that contract for all
three review-layer commands.
Run it directly rather than wrapping it in an enrichment stage: that
would have the enrichment layer importing the review layer, and would
make an advisory report wait on sync's write lock.
Advisory, not a gate, deliberately: matching is lexical, so it
cannot tell "the source doesn't say this" from "the source says it in
words I didn't recognise". citation_gate blocks because it checks
something exact (ledger membership); this reports because it doesn't.
Passage quality depends on what has been parsed. With the Docling stage
run, content/docling/<citekey>.passages.json supplies reading-ordered
paragraphs and the report quotes them. Without it, pdftotext output is
used and the report gives a page number without quoting -- on a
two-column paper that text splices two columns onto every line, so any
excerpt would be a collage of two arguments.
Full design rationale, including the measurements behind those choices:
docs/CITATION-PROVENANCE.md.
Open questions and unbuilt features
Running this pipeline on a schedule was the long-standing goal here.
Most of it now exists: a rotating logs/pipeline.log, shared by the
corpus and enrichment layers (see src/logging_setup.py), a pages/s
throughput figure, exit codes an unattended caller can branch
on, and worked cron and systemd units in
docs/CLI.md -- including the
absolute-interpreter-path detail that cron's minimal environment
requires.
One blocker is left, and it is not a coding task. With no continuous
auto-export, bibliography.bib is a manual, point-in-time snapshot: a
scheduled sync re-reads whatever was last exported, so it keeps the
corpus consistent with the bib file but cannot keep the bib file
consistent with your reference manager. A schedule watching only its
mtime does nothing until a human re-exports. Closing that properly means
either a Zotero auto-export plugin (outside this repo) or accepting that
the export stays a deliberate human step.
content/topics.json has no consumer
src/enrich/topic_model.py writes it and nothing reads it -- no module, no
genre skill. survey-writer groups themes by judgement and says so
explicitly ("With a small corpus there's no BERTopic step"). That is
defensible today: clustering is whole-corpus, so assignments are not
stable between runs, and on a small corpus every document legitimately
lands in the outlier topic. If it is ever wired in, survey-writer's
"Cluster by judgment" step is the seam, gated on the file existing and on
there being non--1 assignments -- the same shape the existing skills use
to gate on content/chroma/.