Tombstone Meaning Representations (TMRs)

Updated 6 July 2026

TMRs are formal, PENMAN-encoded directed acyclic graphs that semantically represent tombstone inscriptions beyond mere text transcription.
They integrate normalization by linking entities to WordNet, GeoNames, and HISCO, standardizing names, dates, occupations, and family roles.
The Tomb2Meaning framework directly parses tombstone images into TMRs using multimodal cues, achieving superior accuracy and robustness compared to OCR pipelines.

Tombstone Meaning Representations (TMRs) are formal graph-based meaning representations for encoding the interpreted content of tombstone inscriptions rather than merely transcribing visible text. In the multimodal tombstone-understanding framework Tomb2Meaning, a TMR is the semantic target produced from a tombstone image: a PENMAN-encoded rooted directed acyclic graph representing deceased entities, names, dates and places of birth and death, occupations, family roles, coreference, and externally normalized identifiers such as GeoNames and HISCO codes (Zhang et al., 6 Jul 2025). The representation is motivated by the view that tombstones are historically rich but structurally heterogeneous artifacts whose content should support storage in relational databases or semantic graphs for search, interpretation, and preservation (Zhang et al., 6 Jul 2025).

1. Formal definition and representational substrate

TMRs are explicitly defined as a rooted directed acyclic graph encoded in PENMAN notation. The rationale given for this design is that tombstone inscriptions contain rich relational information, especially among people, dates, locations, and family roles, but usually do not require the kinds of logical machinery common in broad-coverage semantic formalisms, such as heavy use of negation or universal quantification. A DAG is therefore treated as expressive enough while remaining compact. The formalism was introduced in earlier work by Bos et al. (2022), and the multimodal parsing paper adopts it as the output language of its system (Zhang et al., 6 Jul 2025).

The paper gives the following grammar:

TMR ::= "(" VAR " / " CCT ")" | "(" VAR " / " CCT RLS ")"
CCT ::= LEM "." POS "." SNS
RLS ::= REL TMR | REL DAT | REL LIT | REL VAR | RLS
REL ::= " :ent " | " :nam " | " :txt " | ... | " :pod "
POS ::= "n" | "v" | "a" | "r"
SNS ::= "01" | "02" | ... | "99"

Within this grammar, nodes are introduced as (variable / concept ...), variables are unique identifiers consisting of one lowercase letter followed by a number, and concept labels use WordNet synsets of the form lemma.pos.sense, such as male.n.02, constable.n.03, and date.n.05. Edges are role labels written with a colon, such as :dob, :occ, and :rol. Literals are used for date values, GeoNames identifiers, and HISCO occupation codes. Relations may also be inverted with -of, with the stated equivalence

$R(x,y) \equiv R\text{-of}(y,x).$

This formal definition makes TMRs both symbolic and normalized. Lexical meaning is normalized through WordNet, selected entities can be grounded to external knowledge bases, and role labels define the semantic structure (Zhang et al., 6 Jul 2025).

2. Schema, relation inventory, and normalization conventions

The TMR schema is centered on binary relations that encode tombstone content at the level of deceased entities, biographical facts, kinship and social roles, dates, places, occupations, and coreference. The paper lists the following relations: :ent, :dob, :dod, :pob, :pod, :yoc, :moy, :dom, :equ, :pfx, :sfx, :beg, :end, :rol, :tgt, :geo, :occ, :aft, and :bef. The custom PENMAN language definition in the LaTeX source also mentions :hco, :loc, and :nam, and the grammar includes :txt, although :txt is not elaborated in the main text (Zhang et al., 6 Jul 2025).

These relations fall into several functional groups. :ent introduces the deceased entity mentioned on the stone. :dob, :dod, :pob, and :pod encode dates and places of birth and death. :dom, :moy, and :yoc decompose dates into day, month, and year fields. :rol and :tgt encode family or social roles between persons, while :aft and :bef express temporal ordering among roles. :equ marks coreference. :occ, :hco, :loc, and :geo connect occupation mentions and place mentions to normalized identifiers. :pfx and :sfx capture name morphology, and :beg and :end delimit periods (Zhang et al., 6 Jul 2025).

Normalization is a defining characteristic of the representation. Concepts are normalized to WordNet synsets, toponyms to GeoNames identifiers, occupations to HISCO codes, dates to structured date subgraphs, and names or inscription strings to literals. A date, for example, is represented as a subgraph such as

$10^{-4}$ 0

The paper’s example TMR illustrates how surface evidence and interpreted semantics are combined in a single graph:

$10^{-4}$ 1

In this example, the root is tombstone.n.01; :ent introduces a deceased male.n.02; the occupation is normalized as constable.n.03 while retaining the surface inscription :nam "BRIG. TIT. RIJKSVELDW."; the village mention is normalized as village.n.02 with GeoNames code "2747409"; the birth date is structurally decomposed; and a spousal relation is encoded through :rol and :tgt. This is what differentiates a TMR from OCR output: the graph states that a string is an occupation mention, that it maps to a particular occupation concept and HISCO code, that another string is a place name linked to a specific GeoNames entry, and that another inscription fragment expresses a husband relation between two people (Zhang et al., 6 Jul 2025).

3. Tomb2Meaning: direct image-to-TMR parsing

The principal system built around TMRs is Tomb2Meaning (T2M), a multimodal parser that maps a tombstone image directly to a PENMAN-formatted TMR. Its purpose is to improve interpretation, organization, and retrieval of tombstone content by bypassing OCR as the primary representational target and instead generating semantic graphs directly from images (Zhang et al., 6 Jul 2025).

The paper compares T2M against three baselines. The YOLO-OCR pipeline fine-tunes YOLOv5 to detect labeled regions, applies OCR to transcribe text, then performs post-OCR processing and semantic interpretation. The VLM-OCR baseline uses a vision-LLM to perform OCR-like extraction and then a fine-tuned inscription-to-TMR model to map text to a TMR. The deterministic baseline outputs the single training TMR with highest average Smatch against the training set for every input. T2M differs from all three in treating the image as a source of meaning rather than merely a source of character sequences (Zhang et al., 6 Jul 2025).

Operationally, the pipeline begins with high-resolution JPEG photographs that include metadata such as geolocation coordinates and timestamps and are manually reoriented and cropped for consistency. The VLM sees the entire tombstone image, not OCR boxes, and must jointly infer inscription text, layout cues, whether one or multiple persons are mentioned, whether strings correspond to names, dates, locations, occupations, or roles, and implicit relations such as spouse links. The decoder then outputs a well-formed PENMAN string whose concepts must be selected as WordNet synsets and whose literals must be placed in slots such as :nam, :dom, :geo, and :hco (Zhang et al., 6 Jul 2025).

The paper evaluates both direct prompting and fine-tuning. The core instruction is:

<Image> Generate a meaning representation in PENMAN format for this image of a tombstone. Dont give any other text or explanations.

Fine-tuning uses PEFT/LoRA with fp16, batch size 1, cutoff length 1024, learning rate $10^{-4}$ , 10 epochs, a cosine scheduler, and an H100 GPU (Zhang et al., 6 Jul 2025).

Because some TMR fields depend on externally defined identifiers rather than purely visual evidence, T2M also incorporates retrieval-augmented generation. The retrieval module queries GeoNames, HISCO, and WordNet using entities extracted either from the image or from an initial TMR. GeoNames candidates are filtered by the tombstone image’s GPS coordinates; WordNet candidates inconsistent with the target part of speech are dropped; and the retrieved information is inserted back into prompts or fine-tuning instructions. The paper evaluates three integration strategies: Retrieval-in-the-Beginning Augmented Generation (RibAG), Retrieval-in-the-Middle Augmented Generation (RimAG), and Retrieval-in-the-End Augmented Generation (RieAG). RimAG is described as the strongest strategy because an initial TMR already provides tentative structural and semantic cues, making later retrieval more targeted (Zhang et al., 6 Jul 2025).

4. Corpus construction, evaluation methodology, and empirical performance

The dataset used for TMR parsing contains 1,200 high-resolution photographs of tombstones, each manually annotated with a formal meaning representation following TMR conventions. Images are stored in JPEG; metadata includes geolocation coordinates, timestamps, and date information; and the split is a random 50:50 train/test partition. Most inscriptions are Dutch and in serif fonts, but the corpus also contains English, French, German, Spanish, Italian, Indonesian, and Greek inscriptions. Gothic script occurs frequently, and some inscriptions are multilingual or mixed-language (Zhang et al., 6 Jul 2025).

Evaluation is conducted with graph-level and field-level metrics. Smatch compares predicted and gold graphs by converting them to triples and searching for the best variable alignment:

$P = \frac{m}{p}, \quad R = \frac{m}{g}, \quad F_1 = \frac{2PR}{P+R},$

where $m$ is the number of matched triples, $p$ the number of predicted triples, and $g$ the number of gold triples. Micro $F_1$ is used for names, relations, date values, geocodes, HISCO codes, and WordNet synsets:

$\text{Micro } F_1 = \frac{2TP}{2TP + FP + FN}.$

The Ill-Formed Rate (IFR) counts graphs with cyclic dependencies, isolated nodes, or dangling edges to non-existent elements; such graphs receive Smatch 0 and $F_1$ 0 (Zhang et al., 6 Jul 2025).

The main quantitative result is that the proposed approach improves parsing accuracy from an F1 score of 36.1 for traditional OCR-based pipelines to 89.5 for the VLM+RAG framework (Zhang et al., 6 Jul 2025).

System	Smatch
Deterministic baseline	30.17
YOLO-OCR	36.13
VLM-OCR	66.40
Qwen2.5-VL-7B + RimAG	89.50

Among fine-tuned models, the reported Smatch scores are as follows. Llava-v1.6-mistral-7B reaches 78.70 with T2M, 80.07 with RibAG, 88.07 with RimAG, and 80.55 with RieAG. Llama-3.2-vision-11B reaches 80.90, 82.04, 85.88, and 83.20. Qwen2.5-VL-3B reaches 82.06, 84.29, 86.80, and 84.38. Qwen2.5-VL-7B reaches 85.80, 88.13, 89.50, and 88.54. The best overall system is therefore Qwen2.5-VL-7B + RimAG with 89.50 Smatch (Zhang et al., 6 Jul 2025).

Field-level results for that best model show strong internal semantic performance and weaker external grounding performance: F1-Name 84.83, F1-Role 91.32, F1-Date 89.94, F1-Geo 66.58, F1-Hisco 60.33, F1-Synset 90.52, and IFR 2.17. The importance of retrieval is especially visible in external fields. For Qwen2.5-VL-7B, T2M alone yields F1-Geo 10.83 and F1-Hisco 0.00, while T2M + RimAG yields F1-Geo 66.58 and F1-Hisco 60.33 (Zhang et al., 6 Jul 2025).

Few-shot prompting improves over zero-shot, but zero-shot performance is described as effectively unusable, with many models obtaining 0.00 Smatch. For Qwen2.5-VL-7B, one-shot T2M reaches 56.81, six-shot T2M reaches 67.88, six-shot RimAG reaches 70.33, fine-tuned T2M reaches 85.80, and fine-tuned RimAG reaches 89.50. The paper also notes that RAG can improve well-formedness: for Llava, IFR drops from 10.0 under T2M to 0.50 under both RibAG and RimAG (Zhang et al., 6 Jul 2025).

5. Robustness, semantic coverage, and the distinction from OCR

TMRs are designed to capture semantics that flat inscription transcripts do not represent. The paper enumerates the main additions over OCR output: entity typing, semantic normalization through WordNet synsets, GeoNames codes, and HISCO codes, relational structure such as spouse or husband roles and target links, structured date interpretation, multi-person modeling, and integration of image and text evidence. This makes TMRs suitable for database population, search by semantic fields, genealogical interpretation, heritage archiving, and semantic graph storage (Zhang et al., 6 Jul 2025).

This semantic scope is especially important because tombstones often contain decorative or unusual layouts, abbreviations, multiple persons, rare scripts or fonts, damaged text, and mixed textual and symbolic cues. The multimodal parser can therefore use global visual context and learned cross-modal priors to infer that a string is an occupation abbreviation or that two names are connected by a spouse relation, whereas OCR pipelines fragment the problem into detection, transcription, and subsequent reconstruction (Zhang et al., 6 Jul 2025).

The paper evaluates robustness across five challenge subsets: Rare Languages, Rare Font Styles, Coreference, Abbreviations, and Multiple Persons. VLM-based TMR systems consistently outperform OCR baselines on all five. For Abbreviations, Qwen2.5-VL-7B RimAG reaches 90.5 Smatch, compared with 82.1 for standard T2M and 30.9 for the OCR baseline. For Coreference, RimAG exceeds 88, base T2M is around 83–84, and OCR remains below 50. For Rare Languages and Rare Font Styles, RimAG remains above 88 (Zhang et al., 6 Jul 2025).

The paper also studies physical degradation explicitly. Damage is simulated through image fusion via alpha blending, using masks shaped like clouds, slender stripes, and irregular ellipses, generated with noise, Gaussian blur, and morphological transforms, then blended with variable transparency and multiple overlays. Noise levels are zero, low, medium, and high. As noise increases, all models degrade, but VLM-based TMR generation remains much stronger than OCR baselines. Under high noise, Llava still scores above 60 and Qwen2.5-VL-7B remains above 70, whereas OCR-based baselines drop to or below the deterministic baseline. RibAG yields only minor gains under noise, while RimAG and RieAG remain substantially robust (Zhang et al., 6 Jul 2025).

Two misconceptions are explicitly addressed by this design. First, TMRs are not transcripts; they are interpreted semantic objects. Second, the multimodal task is not monocultural OCR: inscriptions vary by language, font, and style, and many relevant facts depend on role structure, normalization, and relation extraction rather than character recovery alone (Zhang et al., 6 Jul 2025).

6. Comparative position, limitations, and research directions

Within the broader landscape of semantic representation, TMRs occupy a deliberately domain-restricted position. The TMR paper argues that tombstone inscriptions usually do not require heavy use of negation or universal quantification, which is why a rooted DAG is taken to be sufficient (Zhang et al., 6 Jul 2025). By contrast, scoped meaning representations based on Discourse Representation Theory introduce explicit scope boxes for negation, modals, quantification, and presupposition triggers, and evaluate them via clause matching over scoped structures (Noord et al., 2018). Similarly, intensionalized AMR proposals address non-veridicality and de re/de dicto ambiguity through translation into simply-typed lambda calculus, a dedicated :content role, and scope nodes with Cooper storage (Williamson et al., 2021). This suggests that TMRs are optimized for biographical and epigraphic structure rather than for broad-coverage truth-conditional scope phenomena.

A similar contrast appears in evaluation. The TMR paper uses Smatch as its main graph-level metric (Zhang et al., 6 Jul 2025). Work on Uniform Meaning Representation evaluation argues that alignment-sensitive node matching can yield more intuitive and interpretable graph comparison when node-word alignment is available, while avoiding the NP-hard search inherent in Smatch (Zeman et al., 27 Mar 2026). A plausible implication is that, if TMR annotation were enriched with explicit inscription-span or image-region anchoring, anchored comparison could improve adjudication and detailed error analysis for competing TMR annotations.

The methodological contrast with neural implicit meaning representations is also informative. Work on pretrained LLMs shows that BART and T5 encode contextual representations that support linear readout of entities’ current properties and relations and can be manipulated with predictable effects on generation, but those representations are implicit, distributed, and approximate (Li et al., 2021). TMRs, in contrast, are explicit, inspectable, and machine-readable graph structures. A comparative reading therefore places TMRs on the symbolic side of the continuum between interpretable semantic records and latent discourse-state representations.

If TMRs are used not only for parsing benchmarks but also for heritage analytics, archival research, or mixed-method historical inference, a further methodological issue arises. Work on meaning representations as scientific instruments argues for scientific usability, emphasizing geometric legibility, interpretability and traceability to linguistic evidence, robustness to non-semantic confounds, and compatibility with regression-style inference over semantic directions (Plisiecki, 10 Mar 2026). A plausible implication is that TMR-based systems should be evaluated not only by graph accuracy but also by how well they preserve interpretable links to inscriptional evidence.

The TMR paper itself notes several limitations. The dataset is mostly Dutch and has relatively uniform layouts, which may limit generalization. Alpha-blended synthetic degradation may not fully capture real erosion, occlusion, or weathering. RAG depends on structured resources such as GeoNames, HISCO, and WordNet, and this may not transfer to all historical or regional settings. External grounding fields remain the hardest part of the representation. Zero-shot generation is poor, and structural failures still occur, as measured by IFR. Proposed future work includes expanding language and layout diversity, integrating retrieval more end-to-end, and extending the approach to other heritage artifacts such as memorial plaques and archival inscriptions (Zhang et al., 6 Jul 2025).