React Native (Expo) Integration

• React Native (Expo) is a framework that combines declarative UI design with a managed toolchain, facilitating efficient asset handling and cross-platform development.
• The DeclarUI pipeline employs computer vision for component segmentation, JSON-based page transition graphs, and multimodal LLMs to generate and refine TypeScript code automatically.
• Iterative compiler-driven optimization ensures high fidelity in navigation and robust code generation, achieving metrics like 96.8% PTG coverage and 98% compilation success.

React Native, in conjunction with Expo, underpins a prominent paradigm in mobile application development, specifically in declarative user interface (UI) construction. Expo functions as a managed toolchain and runtime atop React Native, promoting increased productivity in asset management, code bundling, dependency installation, and native module access for cross-platform apps. Recent research proposes automated pipelines leveraging computer vision (CV), multimodal LLMs (MLLMs), and iterative compiler optimization for the end-to-end generation and refinement of declarative React Native code within Expo projects, as exemplified by the DeclarUI system (Zhou et al., 2024).

1. DeclarUI–Expo Pipeline Overview

DeclarUI operates a four-stage pipeline designed for integration into Expo–React Native workflows:

1. Component Segmentation via Computer Vision: A flat design image (e.g., Figma mockup or app screenshot) is processed using Grounding DINO for zero-shot object detection, yielding bounding boxes $\{b_i\}$. SAM (Segment Anything Model) then generates fine masks $M_i$ for each $b_i$. A wrapper script outputs component PNG masks, bounding-box coordinates, and references, writing these into a local .declarui/ folder. These are encoded as base64 or S3 URIs for subsequent MLLM prompts.
2. Page Transition Graph (PTG) Construction: The PTG is formally denoted as

$PTG = (N, E)$

where

$$N = \{\,n_i=(id_i,\,name_i,\,type_i,\,properties_i)\mid i=1\ldots k\},$$

$$E = \{\,e_j=(id_j,\,ns_j,\,nt_j,\,condition_j,\,action_j)\mid j=1\ldots m\}.$$

Full-screen images and component segmentation details are provided to the MLLM (e.g., Claude-3.5, GPT-4o), which yields a PTG as a JSON file, stored in the project root to drive navigation logic.

1. MLLM Code Generation: A prompt combining screenshots, component lists, PTG JSON, and generation templates specifies the desired Expo/React Native idioms:
   • React Navigation v6 integration
   • StyleSheet.create styling, output to sibling .styles.ts files
   • Require-based asset imports The MLLM outputs TypeScript (TSX) screen components, style modules, and asset links per page.
2. Compiler-Driven Iterative Optimization: AST-based checks extract all navigation.navigate("TargetScreen") calls and compare against PTG edges. Missing transitions trigger incremental correction prompts. Expo project compilation is invoked (expo-cli start --non-interactive or eas build --local --platform android), and TypeScript/Metro errors are iteratively corrected by re-prompting the MLLM, with up to three passes for both navigation and compilation fixes.

This architecture accelerates transition from static UI mockups to functioning Expo projects with navigation, assets, and style modules.

2. Formal Definitions of Key Metrics

DeclarUI evaluates performance via three principal metrics, all rigorously formalized:

• PTG Coverage Rate: For $E_{PTG}$ (PTG edges) and $E_{Impl}$ (actual edges extracted from code),

$$\mathrm{PTG_{coverage}} = \frac{|\,E_{PTG}\cap E_{Impl}\,|}{|\,E_{PTG}\,|}\times 100\%.$$

This quantifies fidelity of navigation logic generation.

• Visual Similarity Score (IoU per component): For each component mask $M_{gt}$ (ground-truth) and $M_{pred}$ (prediction),

$$\mathrm{IoU} = \frac{|\,M_{gt}\cap M_{pred}\,|}{|\,M_{gt}\cup M_{pred}\,|}.$$

Averaging over all components yields an aggregate screen similarity score.

• Compilation Success Rate (CSR): Given $N$ builds attempted and $S$ successful,

$\mathrm{CSR} = \frac{S}{N}\times 100\%.$

This quantifies the robustness of generated code with respect to the Expo toolchain.

Empirical results indicate 96.8% PTG coverage and 98% compilation success rate in Expo–React Native targets, with substantial improvements over previous MLLM approaches.

3. Concrete Integration Patterns in Expo Projects

Defined directory and code patterns facilitate adoption of DeclarUI output within Expo projects:

• Directory Structure:

my-expo-app/ ├─ assets/ │ ├─ declarui/ │ └─ images/ ├─ screens/ │ ├─ HomeScreen.tsx │ ├─ HomeScreen.styles.ts │ ├─ ProfileScreen.tsx │ └─ ProfileScreen.styles.ts ├─ navigation/ │ └─ AppNavigator.tsx ├─ App.tsx ├─ PTG.json └─ app.json

• Navigation Stack Example:

import { createNativeStackNavigator } from '@react-navigation/native-stack'; import { NavigationContainer } from '@react-navigation/native'; import HomeScreen from '../screens/HomeScreen'; import ProfileScreen from '../screens/ProfileScreen'; const Stack = createNativeStackNavigator(); export default function AppNavigator() { return ( <NavigationContainer> <Stack.Navigator initialRouteName="Home"> <Stack.Screen name="Home" component={HomeScreen} /> <Stack.Screen name="Profile" component={ProfileScreen} /> {/* Additional screens generated from PTG.json */} </Stack.Navigator> </NavigationContainer> ); }

• Asset Management: All referenced images are stored under assets/images/, with app.json's assetBundlePatterns configured accordingly.
• Dependency Installation:

expo install react-native-screens react-native-safe-area-context npm install @react-navigation/native @react-navigation/native-stack

This repeatable workflow enables high fidelity, maintainability, and extensibility.

4. Expo-Specific Limitations and Considerations

Several constraints unique to the Expo managed workflow affect DeclarUI integration:

• Custom Native Modules: Only those supported by Expo config-plugins or via expo prebuild are accessible. Unsupported imports from MLLM output necessitate ejection from the managed workflow.
• Asset Size Limitations: Excessive or large images may breach OTA limits; EAS Update or asset streaming is then required.
• Configuration Management: URL-based PTG actions must be enabled via deep linking (scheme and extra.eas.json). Custom fonts require explicit integration with expo-font and preloading.
• Metro Bundler Compatibility: File extension mismatches, missing TSX registration, or usage of unsupported ES-Next syntax (e.g., import type) require manual intervention, such as updates to babel.config.js.
• Prompt Cost Optimization: To reduce MLLM invocation costs and rate-limit impacts, images can be downsampled or limited to component masks, and PTGs can be cached for reuse.

This suggests that Expo users must strategically manage both configuration and asset provisioning in conjunction with DeclarUI.

5. Recommendations for Expo Developers

Practical guidelines enhance the reliability, reproducibility, and maintainability of DeclarUI-driven Expo projects:

• Pipeline Automation: Component segmentation and PTG generation are automatable via pre-commit hooks. CI/CD (e.g., GitHub Actions or EAS Build) is recommended for iterative compilation and navigation integrity checks.
• Prompt Engineering: Employ code snippets and system prompt templates in editors (e.g., VS Code) to enforce idiomatic Expo code generation by MLLMs.
• Version Control: Store PTG.json under source control and re-run DeclarUI (with diff/merge) after mockup updates.
• Regressive Testing: Integrate visual regression tests (e.g., jest-expo-snapshot) to monitor IoU scores for screens pre- and post-change.
• MLLM Auditing: For long-term projects, route MLLM calls through custom microservices for prompt/output logging and future analysis.
• Documentation of Conventions: Include Expo-specific code style guidelines and common idioms in prompt context to improve code output quality.

Adhering to these practices, Expo project teams can leverage the integration of computer vision, MLLMs, and compiler-in-the-loop systems for rapid, high-quality code generation from declarative UI mockups (Zhou et al., 2024).

6. Significance of Declarative UI Code Generation in Expo

DeclarUI demonstrates that combining structured CV, graph-based UI modeling (PTG), and iterative program synthesis yields tangible improvements in UI development efficiency and quality for Expo–React Native workflows. The empirical benchmarks, including a 96.8% PTG coverage rate and a 98% compilation success rate, validate the pipeline’s effectiveness for declarative frameworks. Furthermore, the system’s iterative refinement methodology exemplifies rigorous compiler-in-the-loop validation—an approach increasingly advocated in automatic code generation research.

A plausible implication is that further developments in multimodal modeling and integration of feedback loops may continue to increase generation fidelity and automation in Expo and similar toolchains. The generalizability demonstrated on frameworks such as Flutter and ArkUI suggests transferable methodologies beyond React Native, indicating a wider applicability for design-to-code translation systems in declarative UI development.