Institutional Books 1.0 Pipeline

• The Institutional Books 1.0 Pipeline is a multi-stage workflow that retrieves, decrypts, and processes Harvard Library's digitized Google Books into a provenance-rich public-domain dataset.
• It integrates GRIN Transfer outputs with metadata normalization, advanced OCR post-processing, precise language detection, and duplicate identification to ensure quality and comprehensiveness.
• The pipeline preserves complete provenance and applies rigorous quality control and rights filtering, supporting sustainable research and reproducible data curation.

Searching arXiv for the cited papers to ground the article. Institutional Books 1.0 Pipeline designates the retrieval, processing, quality-assessment, and assembly workflow used to turn Harvard Library’s Google Books scans into a provenance-rich public-domain corpus, and, in its revised operational form, the pipeline that consumes GRIN Transfer outputs directly. It spans GRIN retrieval and decryption, metadata normalization, OCR text handling, language identification, topic classification, OCR post-processing, duplicate detection, rights filtering, and dataset packaging. In the release described for Harvard Library, the listed source collection comprised 1,075,899 volumes; 1,004,977 encrypted archives were successfully retrieved and decrypted; and the public-domain release comprised 983,004 volumes totaling approximately 242 billion tokens (Cargnelutti et al., 10 Jun 2025, Daly et al., 14 Nov 2025).

1. Definition and scope

The pipeline’s original stated goal was to analyze, augment, and assemble Harvard Library’s public-domain Google Books collection into the Institutional Books 1.0 dataset. In that formulation, the pipeline performs collection-level analysis and augmentation including language detection, topic classification, post-processing of OCR, and duplicate identification at the collection level. A later revision was made for compatibility with the output format of GRIN Transfer, so that a library can pair the two systems into an end-to-end processing workflow that retrieves, structures, and enhances data available from GRIN (Cargnelutti et al., 10 Jun 2025).

The scope is therefore broader than OCR extraction alone. Institutional Books 1.0 retains original OCR text, produces an alternate post-processed text that is easier to tokenize and segment, aligns bibliographic, source, and generated metadata, records quality metrics, and preserves provenance fields sufficient for filtering, debugging, and audit. It is also explicitly non-destructive in its treatment of duplicates: likely near-duplicate volumes are exposed as metadata rather than removed from the release.

A common simplification is to treat the pipeline as a single conversion script from scans to text. That is incomplete. The operational descriptions emphasize a staged system with acquisition, enrichment, normalization, analysis, and assembly phases, while the dataset report emphasizes sustainable, reproducible processing, precise legal provenance, and per-volume quality signals.

2. Retrieval substrate and operational coupling with GRIN Transfer

In the revised operational arrangement, GRIN Transfer is responsible for retrieval and structuring, and the Institutional Books 1.0 pipeline begins from those artifacts. GRIN Transfer first collects the full “All Books” inventory and core metadata from GRIN, populates a local SQLite tracking DB, and materializes a CSV export. It then requests conversions for volumes not yet in downloadable “file package” form, respecting GRIN’s maximum length of 50,000 queued conversion requests. When a volume reaches the “CONVERTED” state, the sync pipeline probes the predictable encrypted file package URL, compares the ETag, downloads the encrypted package when needed, decrypts it with the institution’s GPG passphrase, optionally unpacks METS XML with MARC21 and collates page-level OCR into JSONL, and uploads decrypted archives and optional OCR artifacts to local or S3-compatible storage. A metadata enrichment phase then adds condition and processing fields not present in the main listing (Daly et al., 14 Nov 2025).

The revised Institutional Books 1.0 pipeline consumes four principal GRIN Transfer outputs directly: the CSV catalog of volume metadata and pipeline execution summaries, per-volume decrypted archives, packaged METS XML with MARC21, and optional OCR JSONL. It expects information produced during GRIN Transfer’s “collection,” “metadata enrichment,” and “sync” phases, and can act on GRIN-specific signals such as ETag versioning and GRIN “state.”

The runbook-level constraints are operationally significant. GRIN imposes a global five queries-per-second rate limit on all requests, conversion readiness must be polled rather than pushed, and the sync pipeline keeps throughput at about four concurrent downloads while one to three workers handle HEAD requests. The report further states that collecting all book metadata for Harvard Library’s collection of over one million volumes could be completed within an hour, whereas the whole download cycle for the initial Harvard run took approximately three months. These conditions explain why Institutional Books 1.0 is described as a durable ingestion pipeline rather than a one-time scrape.

3. Analytical processing and text refinement

After ingestion, the pipeline normalizes and reconciles identifiers using MARC fields such as Control Number, ISBNs, and OCLC Numbers keyed by GRIN barcode, and then runs analysis tasks over normalized metadata and OCR. In the dataset release, language identification uses two signals per volume: bibliographic language derived from MARC 21 and text-level detection via pyfranc on 768-character chunks produced with LangChain’s RecursiveCharacterTextSplitter. Detected per-chunk language is tallied with per-chunk token counts using the o200k_base tokenizer, and a per-language-in-volume minimum of 1,000 tokens is used to suppress noise. The resulting fields are language_gen and language_distribution_gen (Cargnelutti et al., 10 Jun 2025).

Topic classification is performed with a fine-tuned bert-base-multilingual-uncased model trained from subject strings mapped to top-level Library of Congress Classification labels. The report gives a validation accuracy of approximately 96.9% and a benchmark accuracy on 1,000 held-out records of 97.8%, with labels returned for 1,004,511 volumes. This stage compensates for sparse or inconsistent original subject metadata by generating topic_or_subject_gen and an associated confidence score.

OCR post-processing is deliberately conservative. A line-type labeling stage classifies lines into the categories UNKNOWN, NOISE_OR_BROKEN_TEXT, PAGE_NUMBER, RUNNING_HEAD, HEADING_OR_TITLE, PARAGRAPH_CHUNK, PARAGRAPH_END, LOOSE_SENTENCE_OR_LIST_ITEM, and SEPARATOR. Training labels were generated with microsoft/phi-4, and a static embedding classifier was then fine-tuned for three epochs. On the benchmark split, the classifier achieved approximately 71% accuracy, which the report characterizes as sufficient for coarse signal. Inference was applied to public-domain volumes in five major languages—eng, deu, fra, ita, and spa—and heuristics then reassembled sentences and paragraphs, inserted paragraph breaks, and suppressed page numbers and running heads when confidently detected. The reported character removal was minimal, at −0.97% characters on average, while tokenizability improved by an average of +4.6 points across languages and +6.1 across decades.

Duplicate detection is likewise analytic rather than destructive. The pipeline computes Simhash on 7-character shingles, treats volumes with identical Simhash as near-duplicates, and then applies heuristic filters that discard candidate pairs when main detected languages differ or continuous character counts differ by at least 15%. Manual review of 100 random groups yielded approximately 97% precision for near-duplicates. The output is a metadata field, likely_duplicates_barcodes_gen, intended for downstream filtering.

4. Provenance, rights, and quality control

Provenance is central to the pipeline’s design. The dataset report describes the provenance chain as Harvard physical collections to Google Books digitization, then GRIN retrieval by barcode as encrypted .tar.gz packages, then decryption, deposition into a raw bucket and a primary bucket, and finally Institutional Books dataset assembly with generated quality metrics and external rights fields. The revised runbook adds operational provenance at the storage level through ETag annotations, per-volume pipeline state in SQLite, machine-readable execution summaries, and explicit retention of source URL, barcode, dates, and GRIN “state” for downstream auditing (Cargnelutti et al., 10 Jun 2025, Daly et al., 14 Nov 2025).

Rights determination is delegated to HathiTrust metadata rather than inferred from OCR content. Harvard barcodes are mapped to hvd-prefixed HathiTrust identifiers, and rights status is then retrieved from the HathiTrust rights API. Rights data were found for 1,004,497 volumes, or 93.36% of the collection, with reported status counts including pd 786,964 and pdus 196,171, along with a small number of cc-zero records, known copyright records, and unknown records. The public release includes only volumes with rights-cleared status pd, pdus, or cc-zero, excludes volumes with no OCR text, and is distributed under a noncommercial click-through license.

Quality control is multi-layered. OCR quality is represented by both a Google-provided volume-level OCR score and an independently computed OCRoscope score. Across the full collection, the average scores are 88.38 for Google and 88.16 for OCRoscope, with decade-level differences noted in the report. Text analysis metrics include words, bigrams, trigrams, sentences, type-token ratios, average sentence length, character count, continuous character count, and a tokenizability score intended to measure how efficiently o200k_base tokenizes a volume. The reports also note that low tokenizability in English often flags volumes dominated by tables or figures, and that outliers remain, including tables, music notation, and maps.

Two misconceptions are explicitly cautioned against. First, the release is not a release of raw scan images: raw scan images are not yet publicly released, although OCR text and metadata are. Second, rights determinations are jurisdiction-dependent and may contain errors, so downstream users remain responsible for their own legal assessments.

5. Release structure and research use

The released dataset is organized as per-volume records containing OCR text, metadata, and generated analytical fields. Core source fields include barcode_src, title_src, author_src, date fields derived from MARC 21, page_count_src, identifiers_src, and language_src. Generated fields include language_gen, language_distribution_gen, topic_or_subject_gen, ocr_score_gen, likely_duplicates_barcodes_gen, text_analysis_gen, and, for the five post-processed languages, text_by_page_gen alongside the original text_by_page_src. External rights metadata are carried in hathitrust_data_ext (Cargnelutti et al., 10 Jun 2025).

The release is intended to support filtering rather than impose a single canonical subset. The documentation recommends filtering by rights status, by language_gen or dominant language proportion in language_distribution_gen, by date range, by OCR scores, by tokenizability score and sentence length, by near-duplicate flags, and by topic_or_subject_gen. This makes the pipeline especially suitable for downstream model training and evaluation workflows that need reproducible selection criteria.

At collection scale, the dataset has a strong historical and multilingual profile. Bibliographic metadata indicate 241 volume-level languages; text-level detection found 254 main-language assignments and 379 languages overall within volumes. English accounts for approximately 47% of volumes, while token totals exceed one billion in 17 languages. Temporally, 729,604 volumes had valid dates, with 650,979 volumes concentrated in 1820–1920. The report presents this as historically rich, long-form coverage rather than a balanced representation of all time periods or language communities.

A further research-use distinction concerns versioning. The revised pipeline can detect ongoing Google reprocessing through ETag changes, GRIN “state” changes, and updated processing timestamps, allowing periodic re-sync and re-analysis. Institutional Books 1.0 is therefore not only a published dataset but also a maintainable synchronization and enhancement framework.

6. Related methods, ambiguities, and future directions

Institutional Books 1.0 sits within a broader ecosystem of book-processing pipelines, but its emphasis is distinctive. “From Books to Knowledge Graphs” focuses on extracting structured linked data from bibliographies and back-of-book indexes, with RDF mapping and authority reconciliation for AHSS publishing workflows (Kokash et al., 2022). STONYBOOK standardizes large-scale novel annotation into XML with POS, NER, dependency, coreference, character metadata, similarity modeling, and corpus-level literary analytics over 49,207 novels (Pethe et al., 2023). “Constructing Image-Text Pair Dataset from Books” combines OCR, Mask R-CNN, and SegFormer to extract image-caption pairs from digitized books, producing 9,516 image-text pairs from 175 books and thereby addressing a multimodal problem largely outside the current Institutional Books 1.0 release (Okamoto et al., 2023). “A General Pipeline for Digesting Scientific Literature into a Shared Scientific Knowledge Base” contributes a domain-agnostic architecture centered on append-only ledgers, provenance-rich records, similarity profiles, and human-reviewed corpus mining (Black, 11 Jun 2026).

These systems are best understood as adjacent rather than competing. A plausible implication is that Institutional Books 1.0 provides the large-scale OCR, metadata, and provenance substrate on top of which linked-data extraction, XML structural annotation, multimodal figure-caption pairing, or PUK-style knowledge-base construction could be layered. The reports themselves point in this direction. Future work explicitly includes “a second version of our Institutional Books processing pipeline to improve handling of nonstandard formats and non-English languages,” “a separate pipeline to extract and describe images from scanned pages,” structured OCR exports beyond plain-text reflow, richer topic and subject classification, multilingual improvements, and integration with additional library partners (Daly et al., 14 Nov 2025, Cargnelutti et al., 10 Jun 2025).

A separate naming ambiguity warrants note. A plausible source of confusion is that unrelated lakehouse work uses the phrase “Books 1.0” for institution-scale financial pipelines rather than digitized-book corpora (Sheng et al., 2 Feb 2026). Within the Harvard Library context, however, Institutional Books 1.0 consistently denotes a public-domain digitized-books pipeline grounded in GRIN retrieval, MARC-aligned metadata, OCR analysis, provenance-rich assembly, and rights-aware release.