Source: scans/baldessari-sprengel.pdf
Purpose: Extract structured artwork and artist data from a scanned exhibition catalogue into a CSV ready for Wikibase import.
| Stage | Script | Description |
|---|---|---|
| 1 | scripts/ocr_catalogue.py |
Render PDF pages → run Tesseract OCR → save page texts |
| 2 | scripts/extract_artworks.py |
Parse OCR text → extract artwork rows → save CSV |
| 3 | scripts/validate_csv.py |
Validate CSV against data models → quality report |
| 4 | (future) | Generate DTD + XML for Wikibase import |
The output CSV columns align with three Linked Open Exhibition data models: - data-models/item-in-exhibition.csv – artwork fields - data-models/artists-data-model.csv – artist fields - data-models/exhibition.csv – exhibition-level metadata
Note: After Stage 3, review the CSV in a spreadsheet before proceeding to XML generation.
Install dependencies and configure paths. Run this cell first.
In [4]:
import subprocess, sys
# Install OCR and image-processing packages if not already present
packages = ["pdf2image", "pytesseract", "Pillow", "pymupdf"]
for pkg in packages:
try:
__import__(pkg.replace("-", "_").split("[")[0])
except ImportError:
print(f"Installing {pkg} ...")
subprocess.check_call([sys.executable, "-m", "pip", "install", "-q", pkg])
print("All dependencies available.")Installing Pillow ...
All dependencies available.In [5]:
from pathlib import Path
def _find_repo_root() -> Path:
"""Walk up from CWD until a directory containing _quarto.yml is found."""
cwd = Path(".").resolve()
for candidate in [cwd] + list(cwd.parents):
if (candidate / "_quarto.yml").exists():
return candidate
# Fallback: assume notebook lives one directory below the repo root
return cwd.parent
REPO_ROOT = _find_repo_root()
PDF_PATH = REPO_ROOT / "scans" / "baldessari-sprengel.pdf"
OUTPUT_DIR = REPO_ROOT / "output" / "ocr"
CSV_OUTPUT = REPO_ROOT / "output" / "baldessari-sprengel-artworks.csv"
SCRIPTS_DIR = REPO_ROOT / "scripts"
# ---- Exhibition metadata ----
EXHIBITION_TITLE = "Baldessari. Sprengel Museum Hannover"
EXHIBITION_LOCATION = "Sprengel Museum Hannover"
EXHIBITION_START = "" # fill in if known, e.g. "2000-09-17"
EXHIBITION_END = "" # fill in if known
EXHIBITION_ORGANIZER = "Sprengel Museum Hannover"
# ---- OCR settings ----
OCR_LANG = "deu+eng" # Tesseract language(s)
OCR_DPI = 300 # higher DPI = better quality but slower
PAGE_RANGE = None # None = all pages; or e.g. "1-30"
assert PDF_PATH.exists(), f"PDF not found: {PDF_PATH}"
print(f"PDF : {PDF_PATH}")
print(f"Output dir : {OUTPUT_DIR}")
print(f"CSV output : {CSV_OUTPUT}")PDF : C:\git\linked-open-exhibition\scans\baldessari-sprengel.pdf
Output dir : C:\git\linked-open-exhibition\output\ocr
CSV output : C:\git\linked-open-exhibition\output\baldessari-sprengel-artworks.csvEach page is rendered at OCR_DPI, converted to greyscale, then fed to Tesseract. Already-processed pages are skipped (cached in output/ocr/pages/).
Tesseract must be installed separately on your system.
Windows installer: https://github.com/UB-Mannheim/tesseract/wiki
Make sure the language packsdeuandengare included.
This step can take several minutes for a long catalogue.
In [2]:
# Check Tesseract installation before running OCR
import shutil, subprocess, sys
tesseract_bin = shutil.which("tesseract")
if tesseract_bin:
result = subprocess.run(["tesseract", "--version"], capture_output=True, text=True)
print(f"Tesseract found: {tesseract_bin}")
print(result.stdout.splitlines()[0] if result.stdout else result.stderr.splitlines()[0])
else:
print("=" * 60)
print("TESSERACT NOT FOUND ON PATH")
print("=" * 60)
print()
print("This PDF is a pure image scan and requires Tesseract OCR.")
print()
print("Install steps (Windows):")
print(" 1. Download installer from:")
print(" https://github.com/UB-Mannheim/tesseract/wiki")
print(" 2. During install, tick 'German' and 'English' language packs.")
print(" 3. Add the install folder to your PATH, e.g.:")
print(" C:\\Program Files\\Tesseract-OCR")
print(" 4. Restart this notebook kernel (Kernel > Restart).")
print()
print("Alternatively, set the path explicitly in the next cell:")
print(" import pytesseract")
print(" pytesseract.pytesseract.tesseract_cmd = r'C:\\Program Files\\Tesseract-OCR\\tesseract.exe'")============================================================
TESSERACT NOT FOUND ON PATH
============================================================
This PDF is a pure image scan and requires Tesseract OCR.
Install steps (Windows):
1. Download installer from:
https://github.com/UB-Mannheim/tesseract/wiki
2. During install, tick 'German' and 'English' language packs.
3. Add the install folder to your PATH, e.g.:
C:\Program Files\Tesseract-OCR
4. Restart this notebook kernel (Kernel > Restart).
Alternatively, set the path explicitly in the next cell:
import pytesseract
pytesseract.pytesseract.tesseract_cmd = r'C:\Program Files\Tesseract-OCR\tesseract.exe'In [9]:
import os
# Full path to tesseract.exe - adjust if installed elsewhere
TESSERACT_CMD = os.path.join(
os.environ.get("LOCALAPPDATA", r"C:\Users\worthingtons\AppData\Local"),
r"Programs\Tesseract-OCR\tesseract.exe"
)
if os.path.exists(TESSERACT_CMD):
print(f"Tesseract found: {TESSERACT_CMD}")
else:
print(f"WARNING: tesseract.exe not found at {TESSERACT_CMD}")
TESSERACT_CMD = NoneTesseract found: C:\Users\worthingtons\AppData\Local\Programs\Tesseract-OCR\tesseract.exeIn [10]:
import sys
import importlib
sys.path.insert(0, str(SCRIPTS_DIR))
# Force reload so edits to the script file are picked up without restarting the kernel
import ocr_catalogue
importlib.reload(ocr_catalogue)
from ocr_catalogue import ocr_pdf
full_txt_path = ocr_pdf(
pdf_path=PDF_PATH,
output_dir=OUTPUT_DIR,
lang=OCR_LANG,
dpi=OCR_DPI,
page_range=PAGE_RANGE,
tesseract_cmd=TESSERACT_CMD,
)
print(f"\nFull OCR text: {full_txt_path}")[INFO] Opening PDF: C:\git\linked-open-exhibition\scans\baldessari-sprengel.pdf
[INFO] Total pages: 67
[INFO] PDF appears to be an image scan (avg 0 chars/page).
[INFO] Using Tesseract OCR at 300 DPI …
[INFO] Processing 67 page(s) …
[OCR] Page 1/67 … done
[OCR] Page 2/67 … done
[OCR] Page 3/67 … done
[OCR] Page 4/67 … done
[OCR] Page 5/67 … done
[OCR] Page 6/67 … done
[OCR] Page 7/67 … done
[OCR] Page 8/67 … done
[OCR] Page 9/67 … done
[OCR] Page 10/67 … done
[OCR] Page 11/67 … done
[OCR] Page 12/67 … done
[OCR] Page 13/67 … done
[OCR] Page 14/67 … done
[OCR] Page 15/67 … done
[OCR] Page 16/67 … done
[OCR] Page 17/67 … done
[OCR] Page 18/67 … done
[OCR] Page 19/67 … done
[OCR] Page 20/67 … done
[OCR] Page 21/67 … done
[OCR] Page 22/67 … done
[OCR] Page 23/67 … done
[OCR] Page 24/67 … done
[OCR] Page 25/67 … done
[OCR] Page 26/67 … done
[OCR] Page 27/67 … done
[OCR] Page 28/67 … done
[OCR] Page 29/67 … done
[OCR] Page 30/67 … done
[OCR] Page 31/67 … done
[OCR] Page 32/67 … done
[OCR] Page 33/67 … done
[OCR] Page 34/67 … done
[OCR] Page 35/67 … done
[OCR] Page 36/67 … done
[OCR] Page 37/67 … done
[OCR] Page 38/67 … done
[OCR] Page 39/67 … done
[OCR] Page 40/67 … done
[OCR] Page 41/67 … done
[OCR] Page 42/67 … done
[OCR] Page 43/67 … done
[OCR] Page 44/67 … done
[OCR] Page 45/67 … done
[OCR] Page 46/67 … done
[OCR] Page 47/67 … done
[OCR] Page 48/67 … done
[OCR] Page 49/67 … done
[OCR] Page 50/67 … done
[OCR] Page 51/67 … done
[OCR] Page 52/67 … done
[OCR] Page 53/67 … done
[OCR] Page 54/67 … done
[OCR] Page 55/67 … done
[OCR] Page 56/67 … done
[OCR] Page 57/67 … done
[OCR] Page 58/67 … done
[OCR] Page 59/67 … done
[OCR] Page 60/67 … done
[OCR] Page 61/67 … done
[OCR] Page 62/67 … done
[OCR] Page 63/67 … done
[OCR] Page 64/67 … done
[OCR] Page 65/67 … done
[OCR] Page 66/67 … done
[OCR] Page 67/67 … done
[INFO] Full OCR text saved: C:\git\linked-open-exhibition\output\ocr\baldessari-sprengel_ocr_full.txt
Full OCR text: C:\git\linked-open-exhibition\output\ocr\baldessari-sprengel_ocr_full.txtReview a sample page to assess OCR quality before proceeding.
In [4]:
# Show the OCR text for a specific page (change page number as needed)
SAMPLE_PAGE = 5
sample_file = OUTPUT_DIR / "pages" / f"page_{SAMPLE_PAGE:04d}.txt"
if sample_file.exists():
print(f"--- Page {SAMPLE_PAGE} ---")
print(sample_file.read_text(encoding="utf-8"))
else:
print(f"Page {SAMPLE_PAGE} not found. Check that OCR ran successfully.")Page 5 not found. Check that OCR ran successfully.In [5]:
# Show total pages processed
pages_dir = OUTPUT_DIR / "pages"
page_files = sorted(pages_dir.glob("page_*.txt"))
print(f"Pages processed: {len(page_files)}")
# Character-level OCR statistics
total_chars = sum(len(p.read_text(encoding='utf-8')) for p in page_files)
avg_chars = total_chars // len(page_files) if page_files else 0
print(f"Total characters extracted: {total_chars:,}")
print(f"Average characters per page: {avg_chars:,}")
# Flag suspiciously short pages (may be blank or image-only)
short_pages = [p.stem for p in page_files if len(p.read_text(encoding='utf-8')) < 100]
if short_pages:
print(f"\nShort pages (possibly blank/image-only): {short_pages}")Parse the OCR text into structured rows using the Linked Open Exhibition data models.
In [7]:
from extract_artworks import parse_ocr_blocks, write_csv
full_text = full_txt_path.read_text(encoding="utf-8")
exhibition_defaults = {
"exhibition_title" : EXHIBITION_TITLE,
"exhibition_location" : EXHIBITION_LOCATION,
"exhibition_start" : EXHIBITION_START,
"exhibition_end" : EXHIBITION_END,
"exhibition_organizer": EXHIBITION_ORGANIZER,
}
records = parse_ocr_blocks(full_text, exhibition_defaults)
print(f"Extracted {len(records)} candidate artwork records")In [8]:
# Preview as a DataFrame
import pandas as pd
from dataclasses import asdict
df = pd.DataFrame([asdict(r) for r in records])
# Show key columns only for readability
preview_cols = [
"catalogue_no", "artwork_title", "artist_full_name",
"artwork_inception", "medium_material",
"height_cm", "width_cm", "collection", "source_page"
]
df[preview_cols].head(20)In [9]:
# Save the CSV
write_csv(records, CSV_OUTPUT)
print(f"CSV written: {CSV_OUTPUT}")Check completeness and flag uncertain values before manual review.
In [11]:
from validate_csv import validate
validate(CSV_OUTPUT)In [13]:
import pandas as pd
df = pd.read_csv(CSV_OUTPUT, encoding="utf-8-sig")
# Completeness: fraction of non-empty values per column
completeness = (df.replace("", pd.NA).notna().mean() * 100).sort_values(ascending=False)
try:
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(10, 6))
completeness.plot(kind="barh", ax=ax, color="steelblue")
ax.set_xlabel("% rows with value")
ax.set_title("Field Completeness")
ax.axvline(80, color="orange", linestyle="--", label="80% threshold")
ax.legend()
plt.tight_layout()
plt.show()
except ImportError:
print(completeness.to_string())Rows that still carry [?] markers or have empty required fields.
In [15]:
import pandas as pd
df = pd.read_csv(CSV_OUTPUT, encoding="utf-8-sig")
# Rows with uncertain markers
uncertain = df[df.apply(lambda row: row.astype(str).str.contains(r"\[\?\]").any(), axis=1)]
print(f"Rows with [?] markers: {len(uncertain)}")
# Rows missing artwork title
no_title = df[df["artwork_title"].fillna("") == ""]
print(f"Rows missing artwork_title: {len(no_title)}")
# Rows missing artist
no_artist = df[df["artist_full_name"].fillna("") == ""]
print(f"Rows missing artist_full_name: {len(no_artist)}")
print("\n--- Sample rows needing review ---")
review_needed = pd.concat([uncertain, no_title, no_artist]).drop_duplicates()
review_needed[["catalogue_no", "artwork_title", "artist_full_name", "notes", "source_page"]].head(10)Open output/baldessari-sprengel-artworks.csv in a spreadsheet and:
artwork_title, medium_material, collection etc.[?] suffixes from corrected values.artwork_inception, height_cm, width_cm where missing.artist_wikidata_qid and artist_gnd_id for known artists.catalogue_no matches the printed catalogue numbering.Once corrected, re-run Stage 3 (validate) until the quality score is satisfactory.
When the CSV quality is high enough, the following will be produced:
output/artwork-catalogue.dtd) – document type definition for the XML structureoutput/baldessari-sprengel-import.xml) – one <item> per artwork, ready for WikibaseThese will be created in a future notebook (Stage 4).
In [17]:
# Quick summary of current state
import pandas as pd
df = pd.read_csv(CSV_OUTPUT, encoding="utf-8-sig")
print("=== Current CSV Summary ===")
print(f"Total artwork records : {len(df)}")
print(f"Unique artists : {df['artist_full_name'].nunique()}")
print(f"Rows with dimensions : {(df['height_cm'].fillna('') != '').sum()}")
print(f"Rows with medium : {(df['medium_material'].fillna('') != '').sum()}")
print(f"Rows with cat. no. : {(df['catalogue_no'].fillna('') != '').sum()}")
print(f"Rows with collection : {(df['collection'].fillna('') != '').sum()}")