ChartMark: Semantic Chart Annotation Framework

• ChartMark is a language-agnostic grammar that standardizes chart annotations by decoupling semantic intent from the visualization implementation.
• It organizes annotations into key dimensions such as Data, Task, Marker, Target, Operation, and Chart, enabling modularity and cross-platform reuse.
• The toolkit demonstrates practical conversion to Vega-Lite, supporting tasks like highlighting, trend analysis, and chart customization while addressing fragmentation.

ChartMark is a structured, language-agnostic grammar for chart annotations that separates annotation semantics from visualization implementations. It was proposed to address fragmented, non-standardized annotation representations in systems where annotations are either simulated by layering primitive marks or exposed through tool-specific APIs, both of which tightly couple annotation intent to a particular visualization grammar or software stack. In ChartMark, annotation intent, task, data, targets, and operations are represented independently of any specific renderer, while a toolkit demonstrates conversion of ChartMark specifications into Vega-Lite visualizations (Chen et al., 29 Jul 2025).

1. Definition, motivation, and scope

ChartMark was introduced as a response to a recurring problem in chart annotation practice: the semantic purpose of an annotation is often lost once it is translated into concrete graphical operations. Existing ecosystems such as Vega/Vega-Lite and D3 typically simulate annotations by layering primitive marks, while systems such as ggplot2, matplotlib, and ECharts provide ad hoc annotation APIs. In both cases, annotation semantics are bound to implementation details, which hinders cross-platform reuse and complicates systematic exchange and management of annotations (Chen et al., 29 Jul 2025).

The central design principle of ChartMark is a separation between the abstract layer and the implementation layer. The abstract layer encodes the annotation’s semantic intent, chart context, data references, targets, and marker specifications. The implementation layer then realizes those semantics in a concrete target grammar such as Vega-Lite. This division is also tied to accessibility: descriptive intents such as “highlight the max” or “explain the positive correlation” can be preserved explicitly rather than being reduced to low-level visual operations alone (Chen et al., 29 Jul 2025).

The paper characterizes ChartMark through five design considerations. DC1 is “Comprehensive and extensible semantics.” DC2 is “Language independence.” DC3 is “Modular architecture.” DC4 is “Multi-level semantic representation.” DC5 is “Atomic element design.” Together, these establish ChartMark as a semantic representation framework rather than a benchmark, task suite, or model for end-to-end chart understanding (Chen et al., 29 Jul 2025).

A common confusion is to treat ChartMark as interchangeable with recent chart benchmarks or chart-editing datasets. That is not its function. ChartMark formalizes annotation structure; benchmarks such as Chart$\text{M}^3$ focus on multimodal chart editing, ChartAct on dynamic chart understanding, and ChartGen on chart-to-code reconstruction (Yang et al., 25 Jul 2025, Huang et al., 26 May 2026, Kondic et al., 31 May 2025). This suggests that ChartMark occupies an infrastructural layer: it describes annotations in a reusable form that other systems could consume or emit.

2. Semantic architecture and annotation dimensions

ChartMark organizes annotation semantics into a hierarchical framework built around six dimensions: Data, Task, Marker, Target, Operation, Chart (Chen et al., 29 Jul 2025). The representation consists of a base chart plus a modular list of annotations, where each annotation is an independent unit that can be attached to, composed with, or separated from the underlying visualization.

At the chart level, ChartMark stores base chart metadata such as type, axes, and data. Each annotation then contains a task, a data component, and a set of operations. The task specifies the analytic or communicative purpose, including the types reference, highlight, description, summary, trend, encoding. The data component specifies both source and values, with sources drawn from external, derived, internal, none. Operations combine a target and a marker: the target identifies what the annotation acts upon, and the marker specifies what visual element or setting is added or modified (Chen et al., 29 Jul 2025).

The target taxonomy includes data_items, coordinate, chart_element, and annotation. This permits annotations over logical data subsets, spatial regions or intervals, structural chart components such as axes or grids, and previously defined annotations. The marker abstraction is deliberately atomic. It includes constructs such as line and text, but also lower-level settings such as opacity, stroke, and fill. This atomicity is intended to support reusable composition: an annotation can express a high-level task while remaining decomposable into explicit operations (Chen et al., 29 Jul 2025).

This modularity distinguishes ChartMark from systems where annotations are inseparable from a concrete layer stack. A ChartMark annotation list is designed to be pluggable, and annotation units are explicitly separable from base charts. A plausible implication is that this architecture can support workflows in which annotations are stored, transferred, versioned, or transformed independently of the visualization specification used for rendering.

3. Formal grammar and specification

The paper provides a formal specification for the core representation. The top-level object is an annotated chart unit:

$$\mathit{annotatedChart} := (\mathit{chart}, \mathit{annotations})$$

The chart entity is defined as:

$$\mathit{chart} := (\mathit{title}, \mathit{type}, \mathit{x\_data}, \mathit{y\_data}, \mathit{x\_name}, \mathit{y\_name},...)$$

The annotation list and elements are defined as:

$$\mathit{annotations} := [\mathit{annotation}_1, \mathit{annotation}_2, \ldots, \mathit{annotation}_n]$$

$$\mathit{annotation} := (\mathit{id}, \mathit{task}, \mathit{data}, \mathit{operations})$$

The task component is given by:

$$\mathit{task} := (\mathit{type}, \mathit{subType?})$$

$$\mathit{type} := \mathit{reference} \; | \; \mathit{highlight} \; | \; \mathit{description} \; | \; \mathit{summary} \; | \; \mathit{trend} \; | \; \mathit{encoding}$$

The data component is defined as:

$$\mathit{data} := (\mathit{source}, \mathit{values})$$

$$\mathit{source} := \mathit{external} \; | \; \mathit{derived} \; | \; \mathit{internal} \; | \; \mathit{none}$$

$$\mathit{values} := [\mathit{value}_1, \mathit{value}_2, \ldots, \mathit{value}_n]$$

$$\mathit{annotatedChart} := (\mathit{chart}, \mathit{annotations})$$0

Operations are specified as:

$$\mathit{annotatedChart} := (\mathit{chart}, \mathit{annotations})$$1

$$\mathit{annotatedChart} := (\mathit{chart}, \mathit{annotations})$$2

The target component is:

$$\mathit{annotatedChart} := (\mathit{chart}, \mathit{annotations})$$3

$$\mathit{annotatedChart} := (\mathit{chart}, \mathit{annotations})$$4

The marker component is:

$$\mathit{annotatedChart} := (\mathit{chart}, \mathit{annotations})$$5

$$\mathit{annotatedChart} := (\mathit{chart}, \mathit{annotations})$$6

$$\mathit{annotatedChart} := (\mathit{chart}, \mathit{annotations})$$7

These definitions formalize the distinction between high-level purpose and low-level realization. The paper does not define a formal mapping function, but it states that conversion can be conceptualized as a transformation from a ChartMark specification to a visualization specification, where data sources are resolved, targets selected, and one or more visualization layers emitted per operation while preserving the base chart (Chen et al., 29 Jul 2025).

4. Toolkit and conversion workflow

The reference implementation, ChartMark.py, demonstrates conversion from ChartMark specifications to Vega-Lite (Chen et al., 29 Jul 2025). The toolkit is presented through a Jupyter-oriented workflow with utility functions including load_json(), render_original_chart(), display_vegalite(), render_annotations(), and get_supported_chart_types().

The paper gives a high-level conversion outline. A ChartMark JSON instance is parsed and validated, producing an internal AST for the chart and each annotation. A base Vega-Lite specification is then emitted from chart metadata such as chart type, axes, and data fields. If annotations reference prior annotations through target.type = annotation, dependencies are resolved by ordering annotations topologically. Each annotation’s data source is then resolved: internal selects marks through chart-data filters, derived computes aggregates or regressions, external binds textual content or images, and none requires no additional data. For each operation, the target is compiled and the corresponding marker is emitted as one or more Vega-Lite layers or configuration settings (Chen et al., 29 Jul 2025).

The conversion semantics are concrete. A data_items target is translated into a Vega-Lite transform filter or conditional encoding. A coordinate target is translated into auxiliary layers such as rect or rule. A chart_element target is mapped to axis, grid, or legend configurations. A line marker can be rendered as a rule or line, while a text marker is emitted as a text mark. The final specification is the base chart plus the annotation layers (Chen et al., 29 Jul 2025).

The paper’s examples emphasize that ChartMark is not limited to direct highlighting. It can encode mean lines, regression lines, shading of intervals, value labels, legend or grid customization, and composition through annotation references. The toolkit is therefore positioned less as a chart authoring system than as a proof of convertibility: a semantic annotation description can be compiled into an executable visualization specification.

5. Expressiveness and representative annotation patterns

ChartMark’s expressiveness is illustrated through several representative patterns in the paper. One example is a grouped bar chart annotated with a highlight task over selected departments, a description task adding value labels, a summary task drawing a mean line from derived data, and another summary task outlining the maximum bar. In this example, the targets include data_items and coordinate, while markers include opacity changes, strokes, text labels, and a dashed line (Chen et al., 29 Jul 2025).

A second example concerns a scatter plot with a trend task. The summary given in the paper states that task.type = trend is paired with operations whose targets cover data_items across groups, with data.source = derived, and a dashed line marker. An annotation target is then used to attach explanatory text to the trend line, with data.source = external values.content (Chen et al., 29 Jul 2025). This is significant because it shows that ChartMark can represent both quantitative computation and explanatory commentary within a unified annotation structure.

A third class of examples uses coordinate and chart_element targets. The paper describes a bounding-box region target with x:[Jun, Oct], y:[240, 280] and a low-opacity rect shading, a shaded temporal interval for Q2, a horizontal reference rule at y=190, and customized y-axis grid lines with tickCount=6 (Chen et al., 29 Jul 2025). These examples broaden the notion of annotation beyond callouts or labels. In ChartMark, an annotation can modify chart structure, not just decorate existing marks.

The task repertoire reported in the paper includes reference, highlight, description, summary, trend, encoding. The supported data sources are internal, derived, external, none. Targets span logical data subsets, coordinate regions, chart elements, and prior annotations. Markers include line, text, opacity, stroke, fill, and related atomic settings (Chen et al., 29 Jul 2025). This breadth is the basis for the paper’s claim of modularity, flexibility, and reuse.

6. Relation to adjacent chart research

ChartMark addresses a different layer of the chart research stack than current benchmarks for chart understanding, editing, reconstruction, or grounding. ChartGen formulates chart-to-code reconstruction as generating a Python plotting script from a chart image and provides a large multimodal corpus for executable chart recovery (Kondic et al., 31 May 2025). Chart$$\mathit{annotatedChart} := (\mathit{chart}, \mathit{annotations})$$8 studies multimodal chart editing, where a model receives chart images, code, and either text or visual indicators to produce edited code and renderings (Yang et al., 25 Jul 2025). ChartAct evaluates dynamic chart understanding in interactive environments that require actions such as hovering, clicking, dragging, and scrolling to reveal evidence (Huang et al., 26 May 2026). ChartREG++ focuses on chart referring expression grounding with points, boxes, and pixel-accurate masks for fine chart elements (Niu et al., 8 May 2026).

These systems are complementary rather than redundant. ChartMark does not benchmark model accuracy, generate chart code, or define an interactive environment. Instead, it formalizes annotation semantics. This suggests at least three connections. First, ChartMark can provide a reusable representation layer for annotations that may otherwise be embedded directly into tool-specific code. Second, its explicit separation of task, target, and data source could serve as a schema for supervising or evaluating annotation-related outputs in other systems. Third, its preservation of semantic intent may be relevant to accessibility and downstream explanation pipelines (Chen et al., 29 Jul 2025).

The distinction also matters when compared with chart summarization and reasoning benchmarks such as Chart-to-Text, ChartX, and ChartMind. Those works center on summarization, question answering, and chart reasoning under image, OCR, or table inputs (Kantharaj et al., 2022, Xia et al., 2024, Wei et al., 29 May 2025). ChartMark does not replace those tasks; rather, it encodes annotations that could be attached to the resulting visualizations or extracted from them. A plausible implication is that ChartMark could act as a bridge between semantic chart interpretation and renderer-specific annotation output.

7. Practical applicability, limitations, and future directions

The paper demonstrates ChartMark primarily through a toolkit and examples rather than through benchmarks, user studies, or performance evaluations. It explicitly states that the work does not report user evaluations, benchmarks, or performance metrics; practical applicability is shown through examples such as highlighting, labeling, mean lines, trend lines, and grid customization (Chen et al., 29 Jul 2025). The toolkit supports Vega-Lite conversion, and broader support for ECharts and D3 is described as planned future work.

Several limitations are identified. Current support is centered on Vega-Lite conversion. Dynamic or interactive annotations and natural-language-driven authoring are left for future work. The code snippets in the paper are simplified for exposition, and a more complete schema and broader annotation types are expected to evolve. The paper also notes potential threats to validity in the need for comprehensive converters per target grammar and the consistent handling of complex charts such as facets and multi-view layouts (Chen et al., 29 Jul 2025).

These limitations are conceptually important. Because ChartMark’s value depends on language-independent semantics, its long-term viability depends on whether converters can preserve those semantics across different visualization grammars without excessive loss or tool-specific distortion. Likewise, extending from static annotations to interactive or dynamically generated annotations would test whether the current grammar remains sufficiently expressive.

ChartMark’s broader significance lies in standardization. By encoding annotation purpose, data source, targets, and operations separately from rendering details, it proposes a reusable semantic substrate for chart annotation. In contrast to benchmarks that measure model capability on chart QA, chart editing, chart-to-code reconstruction, or dynamic interaction, ChartMark formalizes the annotation object itself (Chen et al., 29 Jul 2025). That separation of concerns is its defining contribution.