🛠 Developer guide

Material for working on this repository itself, as opposed to using it to draft content -- test running, the full source layout, and known gaps. See README.md for the user-facing Quickstart/Configuration/ Architecture docs and DOCKER.md for running a container; DOCKER-DEVELOPER.md here is this repo's own record of how the two images are built and verified.

🧭 Table of contents

• Running tests
• Benchmarking the parser
• Writing a script that drives the enrichment layer
• Repository layout
• Figures and copyright
• Citation provenance
• Open questions and unbuilt features

🧪 Running tests

# Install pytest/pytest-cov into the same venv (run python-deps first)
bash scripts/install_full_pipeline.sh dev-deps

# Run the full suite with coverage
.venv-full/bin/python -m pytest --cov=chitragupta --cov=scripts --cov-report=term-missing

# Same, on a host without pandoc/TeX Live/poppler: the render tests skip,
# so opt out of the 100% bar (pyproject's fail_under) rather than lower it
.venv-full/bin/python -m pytest --cov=chitragupta --cov=scripts --cov-report=term-missing \
    --cov-fail-under=0

tests/ covers both the corpus layer and chitragupta/enrich/* -- the enrich group's dependencies (docling, chromadb, bertopic, sentence-transformers) are mocked via sys.modules for fast, deterministic unit tests, so the dev-deps group alone is not enough on its own: the enrich group (python-deps, step 1 of Quickstart) must already be installed too, since tests/test_bib_reader.py needs bibtexparser and the chitragupta/enrich/ test modules need docling/chromadb/bertopic/sentence-transformers.

One module is the deliberate exception. tests/test_enrich_real_libraries.py drives the real chromadb through build_index()/search(), and asks the real sentence_transformers/bertopic classes whether they still accept the keywords the fakes accept (#514). The fakes are faithful enough that the expensive failure is the day they quietly stop being; that module is what notices. It does not download an embedding model -- its own docstring has why -- so it stays as fast and as offline as the rest.

A handful of tests run real dependencies end to end rather than mocking them, and skip automatically when the dependency is absent: tests/test_feature_workflows.py and the TestRenderReal/ TestExtractTextReal classes elsewhere probe for the pdftotext/pandoc/pdflatex binaries on PATH; tests/test_enrich_real_libraries.py probes for an importable library instead, which is the same idiom against a different kind of absence.

⚡ Benchmarking the parser

bench/ measures what a full docling parse of the bib corpus costs on a given machine, and is deliberately kept out of tests/: it takes a couple of hours, needs real PDFs and a GPU, and answers a "how long / what's the bottleneck" question rather than a pass/fail one. It is excluded from the release zip for the same reason tests/ is.

• docs/PERFORMANCE.md -- what each setting costs, organised by setting. Ships in the release archive, unlike bench/
• docs/PARALLELISM.md -- parallel parse design: architecture, components, and the roadmap
• bench/README.md -- how to run it, and what each switch measures
• bench/RESULTS.md -- the dated measurement record, newest last, with raw per-run data in bench/results/. Read its "Which sections are current" table first: several early conclusions were overturned by later runs and are kept, marked, rather than deleted
• bench/PARALLELISM-PLAN.md -- what is still unknown, and what to measure before changing it

The headline, in the order it was found:

1. Parsing all 501 bib PDFs with docling took ~1.6 hours (later measured at 1h 56m), with the A40 at ~7% utilization and three CPU cores of 48 busy. The GPU was worth only 1.79x over CPU-only -- the work was CPU-bound.
2. Turning OCR off (v0.12.0) was worth more than the GPU: 2.08x serially, 3.91x at 12 workers and 4.79x at 24, since OCR competes for the same CPU the parallelism needs. (An earlier 2.46x, from a 16-PDF serial sample, is still quoted in older text; it estimated the serial case only.)
3. Parallelising sync (v1.0.0) was worth 3.60x at four workers.
4. That moved the bottleneck onto a single GPU: AcceleratorDevice.AUTO resolves to cuda:0 in every worker, so GPU 0 ran at 100% while GPUs 1-3 idled.
5. Giving each worker its own card (v1.1.0) was worth a further 1.62x on the full corpus -- 528s to 326s at twelve workers. The whole 501-PDF corpus now parses in 5m 10s, against 1h 56m where this started.
6. Per-worker startup (v2.1.0) turned out to be 3.2s of importing torch and docling plus ~5s of loading Docling's models, and only the first is shareable between processes. A forkserver pool with those modules preloaded, started before the bibliography is read, takes a fixed ~1.5-2s off pool startup -- 9.6% of an 8-document run, 2.5% of a 60-document one.
7. Measuring the whole corpus instead of extrapolating from a 16-PDF sample (2026-08-04) found the serial baseline was 55m 30s, not the ~39m every document had quoted -- 41% low. Correcting it showed 12-worker efficiency is 89%, not the 60% previously reported, and that worker_ceiling()'s cpus // 4 clamp costs 1.41x: 32 workers beat the 12 it allows.
8. Asking whether a quotable passage survives a re-parse (2026-08-07, bench/repro_check.py) found that ~1% of documents come back with a different passage text, and -- correcting what this project had asserted twice -- that two runs of the same configuration are not exempt either. The artifact-by-artifact contract that came out of it is in docs/ARCHITECTURE.md.

The lesson worth carrying: every one of those steps was measured, and seven intermediate conclusions were wrong until the next measurement corrected them -- including two that sat in the code as stated fact, and one that had been written into three documents. bench/ exists so that the next one is checked too.

🧠 Writing a script that drives the enrichment layer

chitragupta.enrich.docling_parse.parse_corpus and python -m chitragupta.corpus sync both use a worker pool when [parser].workers is above 1, and every start method they can pick (forkserver or spawn -- see [parser].start_method) re-imports the calling program's __main__ in each worker. Any script of your own that calls them must guard its top level:

if __name__ == "__main__":
    main()

Without it, every worker re-runs the script on startup and the pool dies with BrokenProcessPool. chitragupta/enrich/__main__.py and chitragupta/sync.py are both guarded already; this only bites ad-hoc scripts, and it bites immediately rather than subtly.

📁 Repository layout

README.md                 the user-facing overview: what this is, the Quickstart, hardware sizing
bench/                    parser measurement (dev-only, not shipped) -- see "Benchmarking the parser"
                          above; corpus.json/sample*.json are generated and gitignored, results/ is
                          committed evidence
  bench_docling.py          backend extraction timings, one process
  sweep_sync.py             the real `python -m chitragupta.corpus sync` swept over worker/GPU counts -- the harness
                            every pool-level figure must come from
  run_parallel.py           independent-process baseline; answers a different question to sweep_sync.py
  make_corpus.py            builds the gitignored work lists from your own bib file
  repro_check.py            compares two parses at three levels (bytes, passage spans, passage texts)
                            and self-checks its own detector on every run
SOUL.md                   one page: why this exists, the one invariant, what it refuses to become
AGENTS.md                 instructions for agents drafting *with* the pipeline -- the citekey
                          invariant, the four layers, retrieval
DEVELOPER-AGENTS.md       instructions for agents changing *this repo* -- install notes, dev
                          process, commit/PR/release conventions
DEVELOPER.md              this file -- test running, repo layout, open questions
DOCKER.md                 how do I run this in a container -- the toolchain image and the agent image
DOCKER-DEVELOPER.md       how the toolchain and agent images are built and verified before a release
.github/                  CI/release workflows, plus the issue and PR templates GitHub picks up
                          automatically and RELEASE_TEMPLATE.md (copied by hand)
docs/                     reference docs that ship in the release zip -- everything except the
                          root-level ones above, which stay put because they're what a reader looks
                          for first. Every file here opens with a status line on the third line,
                          `Status: **<kind>.** Written <date>.`, so a reader can tell what they are
                          holding before reading it -- one of:
                            reference.               what exists and how to look it up
                            how-to.                  a procedure to follow start to finish
                            implemented.             a built subsystem, described as built
                            reasoning document.      why a decision went the way it did, kept
                                                     because the reasoning outlives the decision
                            measurements.            numbers from a real run on a named machine
                            a proposal, not a plan.  nothing here is built; the decision is open
                          A new document picks one of those rather than inventing a sixth. Written
                          is when it was authored and never moves afterward -- git already records
                          revisions, and a hand-maintained "last revised" goes stale. A revision
                          substantive enough that a returning reader should know appends a second
                          sentence right after it, `Updated <date>.` -- itself hand-maintained, so
                          it is a pointer worth checking against `git log`, not a guarantee: a
                          suspiciously old Updated date is a reason to check the file's real history,
                          not proof nothing changed since. Most files carry no Updated sentence at
                          all, which reads as "unchanged since it was written," not as an omission
  PARALLELISM.md            parallel parse design: architecture, components, and the roadmap
  PERFORMANCE.md            what each config setting costs, measured -- the lookup-oriented companion
                            to PARALLELISM.md's design doc
  ZOTERO.md                 getting a bib file and its PDFs into the shape this pipeline expects
  EXPORT-ZOTERO-GROUPS.md   discouraged: recovering collection labels by reading zotero.sqlite
                            directly, when Better BibTeX cannot produce them -- carries the
                            warning about which project rules that bends
  CLI.md                    every command, and which interpreter each one needs
  CONFIG.md                 every setting, with config.toml.example reproduced in full
  PDF-PARSER.md             parser backend tradeoffs, why grobid/markitdown were removed, and why
                            marker/surya/xberg/unstructured were surveyed and not adopted
  GROBID-CITATION-GRAPH.md  a proposal, not a plan: what a GROBID stage alongside docling would
                            buy (a corpus-internal citation graph) and what it would cost
  ARCHITECTURE.md           what runs, what each part writes, what is optional, and which interpreter
                            each command needs -- the user-facing companion to DESIGN.md
  RETRIEVAL.md              BM25 vs embeddings vs topic model: which answers what, and what to build
  DESIGN.md                 architecture and design decisions -- the rationale, not the map
  DIAGRAMS.md               the workflow drawn eleven ways; the fenced mermaid blocks are the source
  diagrams/                 the same eleven as standalone files, for use outside this repo
    *.mmd                     mermaid sources with a title line
    svg/*.svg                 rendered exports, by scripts/render_diagrams.py. Exports only -- edit the
                              fenced block in DIAGRAMS.md, then re-render
    svg/sources.json          which .mmd fingerprint each export was rendered from; pinned by a test
  CITATION-PROVENANCE.md    what chitragupta/review/citation_provenance.py reports and how to read it
  PLAGIARISM.md             what chitragupta/review/verbatim_check/'s overlap/scan modes catch and don't
                            (verbatim reuse only, paraphrase is a later tier), the n-gram
                            fingerprinting technique and its literature sources, and a measured
                            docling-vs-pdftotext backend comparison
  DRAFT-ITERATION.md        what a dossier holds, and how a draft is revised weeks later without
                            re-running the pipeline that produced it
  TOKENS.md                 where a run's tokens go -- the resident/one-shot pools, two worked
                            examples, and how to measure it without paying for a full run
  GENRE.md                  the nine skills in .claude/: which writes what, how to pick, and what
                            each one refuses to do
  LADDERS.md                every automatic fallback chain the code walks, and every tier you pick
                            yourself -- and what the bottom rung of each costs
  WRITING-STANDARDS.md      the prose standards the genre skills share, and their sources in the
                            technical-communication literature
  NAME.md                   where "chitragupta" comes from
  logo.svg, logo-dark.svg   the README banner, light and dark
LICENSE                   MIT
assets/                   data files the pipeline reads at runtime, tracked and shipped
  csl/ieee.csl              the CSL style pandoc formats citations with ([render].csl default).
                          Vendored byte-identical to the CSL project's own release (CC BY-SA 3.0)
                          so it can be re-fetched and diffed -- do not edit it in place; the one
                          attribute this project needs is injected into a temp copy at render
                          time (see assets/csl/README.md and render_output._collapsed_csl)
  csl/README.md             the vendoring policy, upstream URL and sha256
.github/workflows/        ci.yml (test suite + coverage + poetry check, on push/PR) and release.yml
                          (on a v* tag: verifies tag matches pyproject.toml's version, builds
                          scripts/release.py's zip, publishes it to a GitHub Release)
config.toml.example       tracked template for the central config -- paths, parser backend, worker
                          count, embedding model. Copy to config.toml (gitignored, per-host) before
                          anything imports chitragupta.config; see docs/CONFIG.md
papers/                   gitignored, per-host data -- not shipped in the repo
  bibliography.bib          BibTeX export -- source of truth for citekeys/metadata (config.toml's [bib].path default)
  bibliography/             the export's companion attachment folder, referenced by each entry's file field
pyproject.toml            Poetry config -- a real, publishable package (name chitragupta-cli, import
                          package chitragupta, [build-system] = poetry-core; docs/PACKAGING.md has
                          the decision and the naming) + pytest/coverage tool config
poetry.toml               project-local Poetry config: virtualenvs.create = false (installs into
                          whatever venv VIRTUAL_ENV points at, e.g. .venv-full/, instead of Poetry's own)
poetry.lock               resolved dependency versions -- regenerate with `poetry lock` after editing pyproject.toml
mkdocs.yml                the documentation site published at prasad.talasila.in/chitragupta. `docs_dir: .`
                          on purpose: the site is this repository as it stands, so every cross-document
                          link works unchanged and there is no staging step to keep in sync. Built by
                          .github/workflows/docs.yml from pyproject.toml's optional `docs` group
                          (`poetry install --only docs`). Read its header before editing
chitragupta/                      the corpus and drafting layers (sync needs bibtexparser;
                          citation_gate/references need nothing)
  config.py                 every setting, as config.NAME; env var overrides
  config_load.py            the config.toml load and the typed getters config.py reads each setting through
  config_enrich.py          the [enrich]/[discover] knobs, re-exported by config.py as config.NAME
  runlock.py                one-writer-at-a-time lock over content/, held by `chitragupta.corpus sync` and
                          `chitragupta.enrich` -- the only two commands that write to it;
                          a dedicated sqlite file, so a killed holder releases it with no
                          staleness check and readers are never blocked
  bib_reader.py             parses bibliography.bib -- the only citekey source
  ledger.py                 per-citekey status tracking (content/ledger.sqlite); find_stale/prune_missing
                          detect/remove rows for citekeys no longer in the bib file. Also persists each
                          entry's formatting-relevant BibTeX fields (bib_fields, JSON), so references.py
                          can build a full bibliography entry without reading the bib file itself
  pdf_text/                 PDF text extraction, dispatched to pdftotext/docling by config.PARSER; also the parse-quality guard
  sync.py                   orchestrates the above -- the corpus layer's `sync` verb; --remove-stale opts into
                          deleting stale ledger rows (default: report only, see README's "Removing a paper")
  corpus.py                 the corpus layer's single entry point, `python -m chitragupta.corpus sync|ledger`.
                          sync.py and ledger.py carry no __main__ block of their own. Imports the verb
                          it was given rather than both, so asking for `ledger` never pays for sync's
                          bibtexparser -- see docs/ARCHITECTURE.md on why that one is not like the
                          other dispatchers
  draft.py                  the drafting layer's single entry point, `python -m chitragupta.draft
                          gate|dossier|retrieve|references|render`; same rule, same reasons
  dedup.py                  advisory near-duplicate citekey detection (shared DOI/title), called from sync
  reranker.py               the cross-encoder loader, cached per model id; shared by enrich's search rerank
                          and discover's ladder, so neither imports the other's privates
  retrieval.py              BM25 search over the corpus layer, backed by a cached term-frequency index.
                          `search` ranks and returns a snippet -- the best-covering passage for the
                          query, and the same one every run; `evidence` reads more of one document
                          when a snippet is not enough to judge it. A lookup, not a stage: see
                          docs/REJECTION.md for the two-stage read that was built and withdrawn.
                          `--log` records each call's payload in the dossier
  passages.py               where a citekey's supporting text comes from (docling sidecar -> form-feed
                          pages -> pdftotext) and whether it may be quoted -- shared by the consumers
                          that need to point at part of a source rather than all of it
  overlap_index.py          disk-cached word n-gram fingerprint index (content/overlap/) for
                          chitragupta/review/verbatim_check/'s overlap and scan modes -- one .fpr file per citekey plus
                          a merged, binary-searchable corpus-wide index.bin, both keyed by
                          (pdf_hash, parsed-file stat) so a re-run over an unchanged corpus costs no
                          re-fingerprinting. Read-only over the corpus layer, no writer lock
  citation_gate.py          hard citation-verification gate -- the drafting layer must pass this
  dossier/                  the working state behind a draft (reader, scope, kept evidence, rejected
                          candidates, steering, revision log) as Markdown under content/dossiers/,
                          mirroring the draft's path; plus tar.gz backup/restore. Read-only over the
                          corpus layer, never a gate -- see docs/DRAFT-ITERATION.md
  reference_entries.py      citekey -> formatted IEEE entry from the ledger, and the refusal of a key it lacks;
                          shared by references.py, evidence_appendix.py and discover
  references.py             auto-generates a draft's "## References" section from its own cited citekeys,
                          as numbered IEEE entries ordered by first appearance -- the same order (and
                          so the same numbers) pandoc's citeproc assigns when the draft is rendered
  render_output/            Pandoc/TeX Live rendering + standalone CLI -- stdlib-only, no enrich group
                          needed, which is why it sits here and not in chitragupta/enrich/. `--format md` on a
                          Markdown draft skips pandoc entirely and emits references.numbered_markdown's
                          plain numbered copy instead
chitragupta/review/                the review layer -- one command, `python -m chitragupta.review <aid>`, six aids
  __init__.py               the layer's shared output contract -- report path (content/review/,
                          mirroring the draft), the "not a gate" banner, the header, and the
                          write-md-then-render routine all six aids use. No timestamp, so a
                          report diffs across revisions
  __main__.py               the layer's single entry point: one parser, six subcommands, each
                          wired to its aid's own build_parser()/run(). The aids below carry no
                          __main__ block of their own -- see docs/ARCHITECTURE.md on why a layer's
                          command surface stays one level deep
  citation_coverage.py      `coverage` -- retrieval-candidates-vs-actually-cited report, not a gate
  citation_provenance.py    `provenance` -- what in each cited source supports the claim citing it,
                          not a gate (scores claims against passages.py's ladder; see
                          docs/CITATION-PROVENANCE.md)
  synthesis.py              `synthesis` -- how many sources each unit of the draft rests on, at the
                          unit its genre binds at (docs/WRITING-STANDARDS.md §11), not a gate
  uncited_prose.py          `uncited` -- which sentences carry no citation at all, the prose-side
                          question `coverage` does not answer. The one aid that reads no corpus
  verbatim_check/           `verbatim` -- per-citekey overlap, whole-draft x whole-corpus scan, and
                          page-locating checks against sources
  figure_layout/            `figure` -- a TikZ figure's own pre-flight defects (occlusion, chaotic
                          routing, illegible type, ...), not a gate -- see docs/TIKZ-STYLE.md
  _blocks.py                what a *block* is -- a table row, a list item, a heading -- in both
                          markups. Shared by provenance and uncited_prose; neither owns it
  _emit.py                  how an aid prints and files its report under --json/--write, shared by every aid
  _registry.py              which module runs each aid -- the one name->module map, keyed by review.AIDS
  _claims.py                which sentences of a draft carry a claim, and which are scaffolding.
                          Split from uncited_prose so the roadmap's C2 can share it
  _units.py                 what a "unit" is per genre, and how each genre treats uncited prose --
                          the two per-genre policy tables, pinned against dossier.GENRES
  _synthesis_render.py      how a synthesis report reads; _uncited_render.py is the same for uncited
chitragupta/enrich/                the enrichment layer (pyproject.toml's "enrich" Poetry group), optional
  __main__.py               the layer's entry point, `python -m chitragupta.enrich --stages …`; orchestrates
                          the stages below, which carry no __main__ block of their own
  corpus.py                 the enrichment layer's view of the ledger -- one CorpusDoc per bib item,
                          so every enriched document is citable, keyed by its citekey
  docling_parse.py, embed_index.py, topic_model.py, topic_seeding.py, topic_converge.py
scripts/                   dev tooling only -- no layer entry point lives here
  install_full_pipeline.sh  single staged install path (os-deps/python-deps/dev-deps/all) for host + Docker
  release.py                 bundles a distributable release/chitragupta-<version>.zip, dev files excluded
tests/                    pytest suite -- unit tests per module + end-to-end feature tests (see "Running tests")
content/                  generated, gitignored (regenerate with sync)
  ledger.sqlite, parsed/<citekey>.txt, drafts/, dossiers/, rendered/, review/,
  retrieval_index.json, overlap/,
  docling/, chroma/, topics.json, topic_seeds.json, topic_set.json,
  topic_embed_cache.json  (chitragupta/enrich/ outputs)
logs/                     gitignored -- pipeline.log, rotated at 5MB x 5 backups. Level from
                          config.toml's [logging]; relocate with the LOGS_DIR env var
.claude/skills/           drafting layer: five that write a draft (survey-writer,
                          thesis-chapter-writer, textbook-chapter-writer, tutorial-writer,
                          deep-research), three that revise one that already exists
                          (draft-reviser, corpus-reviser, agenda-reviser), and one that
                          assembles accepted units into a book (book-assembler)
.claude/agents/           deep-research's subagents: deep-research-interviewer, deep-research-writer, peer-reviewer
.claude/hooks/            citation_gate_hook.py and style_check_hook.py -- PostToolUse hooks,
                          mechanically enforcing the citation gate and the prose style check on
                          every Write/Edit under content/drafts/*.md and *.tex (see AGENTS.md);
                          session_start_hook.py checks the project can draft at all; draft_target.py
                          is the shared "which file did this tool call touch" helper both PostToolUse
                          hooks use; safe_path.py decides whether their children may import
                          chitragupta from the project root (checkout) or not (installed project)
.claude/settings.json     wires the three hooks above into PostToolUse/SessionStart
docker/                   Dockerfile (TeX Live/Pandoc/Poetry), Dockerfile.claude + docker-compose.yml +
                          entrypoint.sh + .env.example (the Claude Code agent container, cpu/gpu profiles)
                          -- neither image is built by CI, see DOCKER-DEVELOPER.md

📐 Figures and copyright

With [enrich].docling_images on (off by default), the Docling stage writes each paper's figure bitmaps to content/docling/<doc>_artifacts/ and an index of them to content/docling/<doc>.figures.json.

Those images are a reading aid, not draft content. Nothing in this repo inserts them into content/drafts/, and nothing should start doing so. A figure's copyright belongs to the publisher or the authors, and citing a paper grants no right to reproduce its figures -- citation_gate gates citekeys, and there is deliberately no equivalent gate for images. The ledger also has no license column, so the pipeline genuinely cannot tell a CC BY paper from an all-rights-reserved one; that judgment stays with you, per figure.

The supported way to reference a figure is therefore textually, and each record in <doc>.figures.json carries a ready-to-paste cite string:

{
  "page": 8,
  "caption": "Figure 3. Subdivision of the entry process of a Digital Twin",
  "cite": "Figure 3 of [@richstein_characterizing_2024], p.8",
  "image": "richstein_characterizing_2024_artifacts/image_000005_....png"
}

Two details worth knowing about that cite string:

• The number comes from the caption's own text, never from the picture's position. Publisher logos and licence badges are pictures too -- on a real 17-page MDPI paper, 6 of the 13 extracted pictures were furniture rather than figures -- so the Nth picture is routinely not the paper's Figure N.
• The number is captured whole, including chapter-scoped forms (Fig. 1.1 ... Fig. 1.4, the convention in edited book chapters) and sub-figure letters (Figure 2a). Matching only the leading integer would collapse a chapter's four distinct figures onto one Figure 1 -- a citation pointing at the wrong picture.
• A picture whose caption carries no number is cited by page instead ("the figure on p.1 of [@key]"), rather than being given a number this repo would have to invent. Two panels of one figure (captions beginning (a) / (b)) therefore share a page-based citation; that is the fallback behaving correctly, not a collision.

Every figure's cite string is a real [@citekey], because every document the enrichment layer parses comes from the bib file (see chitragupta/enrich/corpus.py).

A wholly original diagram -- not derived from any source paper's figure -- is a different case this section doesn't restrict; see docs/WRITING-STANDARDS.md §10 for the supported form and why it's plain ASCII rather than Unicode box-drawing.

📖 Citation provenance

python -m chitragupta.review provenance content/drafts/<slug>.md reports, for every citation in a draft, what in the cited source supports it and where -- ordered worst match first. It writes content/review/<the draft's path minus its suffix>.provenance.md plus .tex/.pdf renders beside it. The report mirrors the draft's own place under content/drafts/, the same rule rendered/ and dossiers/ follow (config.mirrored_dir), and chitragupta/review/__init__.py owns that contract for all six review-layer commands.

Run it directly rather than wrapping it in an enrichment stage: that would have the enrichment layer importing the review layer, and would make an advisory report wait on sync's write lock.

Advisory, not a gate, deliberately: matching is lexical, so it cannot tell "the source doesn't say this" from "the source says it in words I didn't recognise". citation_gate blocks because it checks something exact (ledger membership); this reports because it doesn't.

Passage quality depends on what has been parsed. With the Docling stage run, content/docling/<citekey>.passages.json supplies reading-ordered paragraphs and the report quotes them. Without it, pdftotext output is used and the report gives a page number without quoting -- on a two-column paper that text splices two columns onto every line, so any excerpt would be a collage of two arguments.

Full design rationale, including the measurements behind those choices: docs/CITATION-PROVENANCE.md.

❓ Open questions and unbuilt features

Running this pipeline on a schedule was the long-standing goal here. Most of it now exists: a rotating logs/pipeline.log, shared by the corpus and enrichment layers (see chitragupta/logging_setup.py), a pages/s throughput figure, exit codes an unattended caller can branch on, and worked cron and systemd units in docs/CLI.md -- including the absolute-interpreter-path detail that cron's minimal environment requires.

One blocker is left, and it is not a coding task. With no continuous auto-export, bibliography.bib is a manual, point-in-time snapshot: a scheduled sync re-reads whatever was last exported, so it keeps the corpus consistent with the bib file but cannot keep the bib file consistent with your reference manager. A schedule watching only its mtime does nothing until a human re-exports. Closing that properly means either a Zotero auto-export plugin (outside this repo) or accepting that the export stays a deliberate human step.

🚫 content/topics.json has no consumer

chitragupta/enrich/topic_model.py writes it and nothing reads it -- no module, no genre skill. survey-writer groups themes by judgement and says so explicitly ("With a small corpus there's no BERTopic step"). That is defensible today: clustering is whole-corpus, so assignments are not stable between runs, and on a small corpus every document legitimately lands in the outlier topic. If it is ever wired in, survey-writer's "Cluster by judgment" step is the seam, gated on the file existing and on there being non--1 assignments -- the same shape the existing skills use to gate on content/chroma/.