SkyScript: Vision, Video, & Sky Model

• SkyScript is a multi-domain resource comprising a vision-language corpus for remote sensing, a drama video benchmark for short drama video generation, and a sky simulation framework for astrophysical studies.
• It employs advanced CLIP-based methods and transformer architectures to achieve superior zero-shot classification and cross-modal retrieval, with improvements up to 25% in key metrics.
• Its applications span remote sensing analysis, automated video synthesis, and astrophysical simulation, driving novel research opportunities despite geographic and modality limitations.

SkyScript refers to three distinct and technically significant systems and datasets in computational imaging, machine learning, and multimodal text/video generation. The term is used for a large-scale vision-language corpus for remote sensing ("SkyScript"), a billion-scale multimodal script–shooting-script benchmark for short drama video generation ("SkyScript-100M"), and, in a separate context, as a synonym for the Python Sky Model (PySM), a full-sky microwave emission simulation framework occasionally referenced as “SkyScript” in technical documentation. Each instantiation provides foundational infrastructure for state-of-the-art research in its respective subfield.

1. Semantically Diverse Vision-Language Dataset for Remote Sensing

SkyScript is an image–text dataset comprising 2.6 million remote-sensing images paired with captions spanning 29,000 distinct semantic tags. Its primary purpose is to enable zero-shot transfer, open-vocabulary classification, and robust retrieval for vision–LLMs (VLMs) in the remote sensing domain, which lacks large, semantically labeled corpora due to annotation cost and limited Internet availability of labeled geospatial imagery (Wang et al., 2023).

Construction Methodology

• Image Source: Ten Google Earth Engine (GEE) collections (RGB only), with ground sample distances (GSD) from 0.1 m/pixel (SWISSIMAGE) to 30 m/pixel (Landsat 8/9).
• Semantic Tag Acquisition: OpenStreetMap (OSM) “key=value” tags are filtered by:
  1. A CLIP text-embedding-based binary logistic regression for visual groundability (F1 = 0.88).
  2. A multi-class logistic regression to estimate the coarsest GSD (0.1–10 m) for detectability (accuracy = 0.53).

• Object/Image/Captions:

  • Two-stage object sampling: random (400,000 Earth 0.01° grid cells) plus targeted enrichment of rare tags.
  • For each object, select the optimal imagery source, tile size (168–300 px, centered on or near object), and discard if coverage is absent.
  • Caption assembly is rule-based, generating single-object (“[object] is [attribute]”) and multi-object (“[object] surrounded by [others]”) descriptions.
• Noise Reduction: Cosine similarity between image–text CLIP embeddings is used for filtering; top 50% retained for the released set.

Dataset Property	Value
Images × Captions	2.6 M image–text pairs
Semantic Tags	29,000 distinct tags
Single/Multi Captions	100,000 single-object, 1.2 M multi-object captions
Geographic Coverage	Global (denser in U.S./Europe)

2. Mathematical and Training Framework

SkyScript models are trained with the standard CLIP image–text contrastive loss: $$L_{\mathrm{ITC}} = -\frac{1}{N} \sum_{i=1}^{N} \left[ \log \frac{\exp(\langle v_i, t_i \rangle / \tau)}{\sum_{j=1}^N \exp( \langle v_i, t_j \rangle / \tau)} + \log \frac{\exp( \langle v_i, t_i \rangle / \tau )}{\sum_{j=1}^N \exp( \langle v_j, t_i \rangle / \tau )} \right]$$ where $\tau$ is a learnable temperature parameter. Architecturally, ViT-B/32 or ViT-L/14 vision encoders (initialized from CLIP/LAION checkpoints) and CLIP-style Transformer language encoders are employed. Batch size is 512, with 20 epochs over four NVIDIA A100 GPUs; any layer freezing reduces zero-shot accuracy, especially in the vision backbone (Wang et al., 2023).

3. Evaluation, Results, and Benchmarks

SkyScript models are evaluated for:

• Zero-shot scene classification on AID, EuroSAT, fMoW, Million-AID, PatternNet, NWPU-RESISC45, RSI-CB256, and a new 70-class in-domain set. Metric: Top-1 accuracy via max cosine similarity between image embeddings and prompt text embeddings.
• Zero-shot fine-grained classification of properties such as roof-shape, road smoothness, and road surface; images collected independently of SkyScript sources.
• Cross-modal retrieval using Recall@K on SkyScript-retrieval (in-domain), RSICD, RSITMD, and UCM-Captions.

Task	CLIP-Original (%)	SkyScript (%)	Δ Accuracy (%)
Avg. Scene Class. (L/14)	53.76	59.93	+6.17
EuroSAT	41.89	52.44	+10.55
Million-AID	57.88	66.40	+8.52
Roof-shape Fine-grained	25.40	35.80	+10.40
Road Surface	42.73	67.73	+25.00
SkyScript Retrieval (in)	2.97 / 1.95	8.53 / 7.73	+5.56 / +5.78

On several tasks, SkyScript-trained VLMs yield improvements exceeding 25% absolute in fine-grained attributes and also surpass domain-supervised baselines in cross-modal retrieval (Wang et al., 2023).

4. Applications, Limitations, and Future Directions

SkyScript supports open-vocabulary remote sensing tasks, including:

• Zero-shot scene and attribute classification across continents.
• Cross-modal retrieval, with no domain tuning required.
• Foundation for remote sensing captioning and text-to-image synthesis.

Limitations include geographic bias (under-representation in non-Western regions), incomplete OSM tagging, rule-based (bag-of-words) caption generation, and lack of multispectral extensions. Ongoing work focuses on LLM-based caption rewriting and incorporation of non-RGB imagery (Wang et al., 2023).

5. SkyScript-100M for Short Drama Video Generation

SkyScript-100M consists of 1 billion script–shooting-script pairs sourced from 6,660 popular short-drama episodes (about 80,000 mini-episodes, 2,000 hours total duration, 10 TB at 720p) (Tang et al., 2024). Key technical features include:

• Keyframe Extraction & Annotation: About 10 million keyframes, capturing shot/scene metadata, 3D character/object coordinates, emotional continua (Plutchik V/A/D), event structure, and highlight scores.
• SkyReels Script Restoration Model: Transformer-based encoder–decoder with cross-modal attention, minimizing a joint loss:

  $$\mathcal{L} = \lambda_{\mathrm{rec}} \mathcal{L}_{\mathrm{rec}} + \lambda_{\mathrm{LM}} \mathcal{L}_{\mathrm{LM}}$$

where $\mathcal{L}_{\mathrm{rec}}$ is an L1 or L2 loss over shot-level predictions and $\mathcal{L}_{\mathrm{LM}}$ is cross-entropy over generated tokens.

Corpus	Pairs (M)	Resolution	# Shots/Episode	Modality Detail
MSR-VTT	10	240p	1.4	Manual, open
HD-VILA-100M	103	720p	31	ASR, open
SkyScript-100M	1,000	720p	~15,000	Manual, short drama; 3D, emo.

• Applications: Script-to-video synthesis, highlight detection, world layout projection, character affinity mining, and counterfactual story generation. SkyScript-100M enables more script-coherent, compositionally faithful, and character-consistent video generations than loosely captioned datasets.
• Limitations: Domain restriction to popular, monolingual content; exclusion of audio and pre-restoration scripts; ~4% annotation loss due to quality control; MLLM-based pre-annotation reliance.
• Future Work: Expansion to multilingual and cross-cultural contexts, integration of audio tracks, dialogue prosody, hierarchical story modeling, and interactive user-feedback loops (Tang et al., 2024).

6. SkyScript as the Python Sky Model (PySM)

In computational astrophysics, SkyScript refers to the PySM codebase, which generates simulated full-sky intensity and polarization maps at microwave frequencies for CMB experiment design (Thorne et al., 2016). Key attributes include:

• Component Modeling: Modular simulation of synchrotron (power laws), thermal dust (modified blackbody), free–free (Draine-law power law), AME (two-component, with optional polarization), and CMB (lensed realizations).
• Customizability: Multiple emission models, bandpasses, beam convolution, Gaussian/log-normal small-scale fluctuations, and user-extensible pipeline.
• Computational Performance: Entire seven-frequency, nside=512, I/Q/U map suite simulated in minutes with multi-threading; modular architecture supports component/model extension.

Component	Parametric Model Reference	Notable Options
Synchrotron	Power law, curvature, lat-steepening	3 model variants
Dust	Single/two-temp MBB, stochastic β	4 model variants
Free-free	Power law, unpolarized	1 model
AME	SpDust templates, global pol. frac.	2 model variants
CMB	Lensed ΛCDM, user input	Any power spectrum

The pipeline enables high-fidelity, physically grounded sky simulations for foreground separation, delensing, and systematic forecasts in future CMB data analysis (Thorne et al., 2016).

7. Synthesis and Significance

SkyScript’s various instantiations represent reference-grade infrastructure in three domains: remote sensing multimodal learning, script-driven text-to-video generation, and astrophysical microwave simulation. Each resource addresses a longstanding bottleneck—be it semantically diverse image–text corpora in remote sensing, richly structured multimodal scripts for video AI, or parametric, extensible sky emission modeling for CMB research.

A plausible implication is that these datasets and tools act as accelerants for foundation model research in both Earth observation (climate, infrastructure, sustainable development) and automated entertainment content creation, as well as calibration/forecasting for next-generation cosmic microwave background missions. Limitations primarily reflect domain and modality selection, annotation capacity, and representational diversity, which are being addressed in ongoing and future extensions (Wang et al., 2023, Tang et al., 2024, Thorne et al., 2016).