GestoBrush: Embodied AR Graffiti Creation

• GestoBrush is a mobile AR prototype that turns smartphones into virtual spray cans, enabling graffiti artists to create expressive, spatially anchored artwork.
• The system combines real-time device tracking, surface scanning, and gesture-based mark making to support both 2D wall and 3D spatial graffiti creation.
• Field evaluations with graffiti artists reveal enhanced creative freedom, immersion, and the ability to bypass legal and material constraints in urban spaces.

GestoBrush is a mobile augmented-reality prototype and design study that turns smartphones into virtual spray cans for graffiti creation through embodied gestures. Developed to investigate how AR can support graffiti artists’ own creative, bodily practices rather than primarily audience-facing experiences, it combines mobile device tracking, real-time rendering, surface anchoring, and gesture-driven mark making in a workflow that begins on a scanned wall and can extend into surrounding three-dimensional space (Chen et al., 6 Sep 2025).

1. Research problem and conceptual scope

GestoBrush is situated in research on graffiti as a spatial, socio-culturally embedded, and bodily practice. The underlying motivation is that graffiti has long “documented the evolving socio-cultural landscapes of urban spaces,” while increasing regulation has constrained legal surfaces, increased penalties, and narrowed where and how artists can work. Within that framing, the system addresses a specific gap: prior AR graffiti research had largely centered on audience engagement, whereas GestoBrush was designed around artists’ embodied creative processes (Chen et al., 6 Sep 2025).

The project formulates two research questions: RQ1, what aspects of current AR graffiti creation can be improved from graffiti artists’ perspective; and RQ2, how embodied AR gestural interactions benefit graffiti artists in AR graffiti creation. Accordingly, GestoBrush functions both as a creative system and as a technology probe. It is not presented as a general-purpose image editor or a purely virtual drawing environment. Rather, it is a mobile AR graffiti tool intended to preserve physical context while allowing artists to bypass the material and legal constraints of physically marking walls (Chen et al., 6 Sep 2025).

A central conceptual distinction in the work is between graffiti as an image and graffiti as an embodied negotiation with space. The paper treats bodily movement, postural engagement, tool handling, and site specificity as constitutive features of practice. This emphasis differentiates GestoBrush from digital workflows that reduce graffiti to finger drawing on flat screens or to decontextualized virtual canvases (Chen et al., 6 Sep 2025).

2. System architecture and interaction pipeline

GestoBrush is implemented as an iOS mobile AR application built with Unity and Xcode. It uses Apple ARKit for real-time 6-DoF device tracking and plane detection, and RealityKit for real-time rendering of graffiti content in the live camera view. The phone is held and moved like a spray can, with a fixed point at the edge of the device defined as the virtual nib from which digital paint is emitted (Chen et al., 6 Sep 2025).

The interaction loop begins with surface scanning. The user scans a target wall or surface, ARKit detects a plane, and the system registers that plane as a 2D canvas. In this initial mode, strokes are laid onto the detected wall. After a 2D layer has been created, the artist can continue drawing off the wall into surrounding 3D space, so that the same tracked device motion generates graffiti that is no longer constrained to a planar support (Chen et al., 6 Sep 2025).

Drawing is gated by a central on-screen button that serves as the spray trigger: a long press starts drawing and release stops it. The system provides two tool modes, graffiti spray and drip mop. Strokes are generated by a triangular mesh algorithm that takes position, size, and color as parameters. These meshes are rendered in the AR scene and anchored to the ARKit world coordinate system so that they remain fixed in place when viewed from different positions later (Chen et al., 6 Sep 2025).

The software architecture is split between a front end and a back end. The front end captures user gestures, generates graffiti as 3D meshes, and renders them in AR. The back end is implemented in Golang, communicates via HTTP, stores graffiti model data, and supports retrieval and reloading across sessions. Persistence is therefore part of the system design rather than an afterthought: the work is not merely ephemeral overlay, but a stored AR artifact tied to a particular environment (Chen et al., 6 Sep 2025).

A plausible formalization of the spatial logic is the one described in the technical account: at each frame, ARKit supplies a device pose in a world coordinate system, the virtual nib is defined in device-local coordinates, and successive nib positions are sampled either on a detected plane for 2D graffiti or directly in world space for 3D graffiti. The paper itself does not foreground this as a mathematical model, but the operational description is consistent with that interpretation (Chen et al., 6 Sep 2025).

3. Embodied interaction and co-design foundations

Embodiment is the organizing design principle of GestoBrush. In the study, embodiment refers to physical engagement with surroundings, creative processes driven by bodily movement and gestures, and a sense of agency and expression through the body. The system is explicitly designed to restore whole-body involvement—arms, shoulders, torso, and movement through space—rather than confining interaction to fingertip gestures on a touchscreen (Chen et al., 6 Sep 2025).

This design direction emerged from a preliminary co-design workshop with five graffiti artists whose experience ranged from two to six years and whose AR familiarity varied from basic to advanced. Participants first tried an existing 2D AR graffiti app and then discussed the shortcomings of current AR tools, the role of bodily gestures in graffiti, and the importance of communication with physical space. Two themes structured the design response: bodily gestures as essential to graffiti, and communication with physical space (Chen et al., 6 Sep 2025).

The first theme concerned the role of rapid, intense movement, including the kinds of full-body action associated with “station throw-ups.” Existing AR apps that rely on finger-based drawing on flat screens were described as eliminating expressive possibilities and the enjoyment of movement. GestoBrush therefore treats the phone as an embodied tool analogous to a spray can or mop, supports two-handed and upper-body movement, and minimizes dependence on small finger gestures (Chen et al., 6 Sep 2025).

The second theme concerned site specificity. Graffiti was described as inseparable from walls, corners, scale, neighborhood, and surrounding visual context. GestoBrush addresses this by requiring wall scanning and environmental recognition, allowing creation first on a wall-bound 2D surface and then in surrounding 3D space. This preserves the sense that the work is anchored in a place rather than floating abstractly (Chen et al., 6 Sep 2025).

A common misconception would be to read GestoBrush as a gesture-recognition system in the command-based sense of HCI. The paper does not present discrete gesture classification, pressure inference, or stylus-like stroke modeling. Interaction is instead based on continuous pose tracking and button-press gating, with expressivity arising from how the body moves the tracked device through the environment (Chen et al., 6 Sep 2025).

4. Evaluation with graffiti artists

The evaluation involved six graffiti artists in Tianjin, China, recruited via Xiaohongshu. Participants had mixed AR literacy and graffiti profiles, with intermediate to advanced graffiti familiarity and a mix of professional and amateur experience. The field session took place on a local street with spontaneous graffiti, chosen for cultural relevance rather than laboratory neutrality (Chen et al., 6 Sep 2025).

The procedure had three phases. A pre-survey collected demographics, graffiti experience, and AR literacy. Participants then received an instructional video and live demonstration and were given 20 minutes to create one AR graffiti work on site. An optional creative prompt asked what AR graffiti they would leave if existing physical graffiti disappeared and they wished to commemorate the place and its cultural memory. The study concluded with midpoint and post-study semi-structured interviews focused on interaction, embodiment, environmental engagement, and comparison with traditional graffiti and 2D screen-based tools (Chen et al., 6 Sep 2025).

Interview data were analyzed using thematic analysis following Clarke and Braun. Two researchers independently open-coded transcripts, refined an initial codebook through further coding, and resolved disagreements through discussion, achieving more than 80% inter-coder reliability. The evaluation was qualitative; the paper does not report standardized usability scales or statistical tests (Chen et al., 6 Sep 2025).

Findings suggested that embodied AR interactions helped artists bypass real-world constraints and explore new artistic possibilities, and that the resulting AR artworks enhanced senses of intuitiveness, immersion, and expressiveness. Participants emphasized several affordances: the ability to “paint” in prohibited or impossible places without physically marking them, the possibility of graffiti in mid-air or sky-like volumes, and the novelty of 3D spatial graffiti as a form with no blind spots and with multi-perspective viewing as part of the compositional process (Chen et al., 6 Sep 2025).

The study also highlighted an important tension. GestoBrush was experienced as both familiar and unfamiliar: the smartphone is a common object, yet in this system it becomes a spatial brush for body-driven 3D painting. That paradoxical familiarity appears to be central to the system’s usability and novelty. It preserves a recognizable handheld tool metaphor while shifting the locus of control from screen contact to spatial action (Chen et al., 6 Sep 2025).

5. Creative affordances, limitations, and cultural implications

The principal creative affordance identified in GestoBrush is the ability to break physical and legal constraints while retaining spatial grounding. Participants reported that, unlike traditional graffiti, AR creation did not require permission, did not depend on wall material, and was not limited to physically reachable or legal surfaces. At the same time, because works are anchored to real locations, the resulting practice retains some of the locative specificity that defines graffiti culture (Chen et al., 6 Sep 2025).

Another major affordance is the transition from 2D wall-bound tagging to 3D spatial graffiti. Artists described being able to compose work not only on walls but also in surrounding volumetric space, with layered combinations of 2D and 3D components. This changes the design problem: the artist must consider how the piece appears from multiple viewpoints and how movement around the work becomes part of its reception. The paper treats this not as a simple extension of tagging into an extra axis, but as an emergent artistic form (Chen et al., 6 Sep 2025).

The study also identifies concrete limitations. Participants reported a learning curve in aligning body movement with 3D stroke placement; tracking instability and drift in bright outdoor spaces or low-feature environments; physical fatigue during full-body interaction; difficulty judging scale and position without strong physical reference surfaces; and a possible loss of contextual meaning when graffiti is detached from a concrete wall and feels “floating.” The absence of tactile feedback from a physical spray can or surface is also implicit in the discussion of limitations (Chen et al., 6 Sep 2025).

These observations motivate several design implications. The paper proposes embodied gestural interaction as a core design principle rather than an add-on, suggests adding multi-sensory feedback and possibly tangible interfaces, and calls for more sophisticated gesture-aware features that remain intuitive. It also proposes stronger environmental sensing, better support for spatial coherence in open space, and future work on collaborative AR graffiti, location-based sharing, and communal AR walls or districts (Chen et al., 6 Sep 2025).

Culturally, GestoBrush is framed as a bridge rather than a replacement. The project acknowledges graffiti as a form of unauthorized expression tied to public and contested space, and positions AR as a way to preserve aspects of graffiti culture under increased regulation. This framing also introduces ethical questions. The paper does not systematize misuse scenarios, but the discussion implies concerns around authenticity, ownership of augmented public space, and the status of AR overlays in relation to existing physical artworks (Chen et al., 6 Sep 2025).

6. Relation to adjacent brush- and gesture-based research

GestoBrush occupies a distinct position within a broader cluster of brush-, stroke-, and gesture-oriented systems. Unlike algorithmic brushstroke renderers, its primary contribution is not image stylization or low-level stroke optimization, but embodied AR creation situated in graffiti practice. A useful contrast is "Parameterized Brushstroke Style Transfer," which represents paintings as collections of quadratic Bézier strokes and optimizes a stroke parameter matrix through a differentiable renderer and VGG-19 style/content losses; that work is stroke-centric, but it addresses neural style transfer rather than site-specific embodied mark making (Meleti et al., 8 Mar 2026).

A second comparison arises with stroke-based geometry editing. "INST-Sculpt: Interactive Stroke-based Neural SDF Sculpting" performs local fine-tuning of a neural SDF from user-drawn strokes defined over a tubular neighborhood around a surface curve, with custom brush profiles and modulation functions; it is gesture-driven, but the target is neural geometry rather than AR graffiti in physical urban space (Rubab et al., 5 Feb 2025). Similarly, "Layered Diffusion Brushes" and "DiffBrush" are brush-driven systems for training-free latent diffusion editing and generation. The former provides layer masks, per-layer prompts, cached latent editing, and approximately 140 ms editing on 512×512 images, while the latter guides color, semantics, and instance placement through latent-space and attention-space manipulations (Gholami et al., 2024, Chu et al., 28 Feb 2025).

Earlier touchless interfaces also illuminate what GestoBrush is not. "Finger-Stylus for Non Touch-Enable Systems" uses a webcam, YCbCr skin filtering, fingertip extraction, and mid-air finger motion to render strokes on screen without touching the display, and "Visual Rendering of Shapes on 2D Display Devices Guided by Hand Gestures" uses Leap Motion trajectories, extended Npen++ features, and HMMs to map gestures to predefined shapes (Chaudhary, 2014, Singla et al., 2018). These systems treat gesture primarily as an input modality for drawing or command invocation. By contrast, GestoBrush embeds gesture within questions of embodiment, site, and graffiti culture (Chen et al., 6 Sep 2025).

Gesture can also serve as supervision rather than as artistic action. "Gesture-based Bootstrapping for Egocentric Hand Segmentation" uses a predefined calibration gesture, a gesture network operating on background subtraction and optical flow, and a personalized appearance network trained from pseudo-labels with uncertainty weighting. In that work, gesture is a mechanism for collecting labels and calibrating perception models, not a medium of expressive production (Zhang et al., 2016).

Taken together, these neighboring systems suggest a broader taxonomy. Brush- and gesture-based research may target optimization in a stroke parameter space, local deformation of neural fields, masked latent manipulation in diffusion models, touchless mid-air drawing, or user-specific perception. GestoBrush differs in centering embodied AR graffiti creation: the smartphone as virtual spray can, the wall as scanned and augmented site, the body as the primary locus of expression, and the resulting work as a persistent, spatially anchored artifact within the cultural logic of graffiti (Chen et al., 6 Sep 2025).