Gemini: Multimodal Models, Astronomy & Genomics

• Gemini is a multifaceted framework spanning multimodal AI models, astronomical instruments, genomics tools, and advanced gravitational-wave testbeds.
• Its AI models, including variants like Gemini Ultra and Pro, achieve state-of-the-art benchmark scores across text, image, video, and audio tasks using efficient transformer architectures.
• Gemini underpins specialized embedding models and domain adaptations in education and medicine, emphasizing secure post-training and robust tool integration.

Gemini is the name of several technically unrelated systems in contemporary research literature. Most prominently, it denotes a family of multimodal foundation models developed for interleaved text, image, audio, and video processing, together with derivative embedding, medical, educational, and security-focused systems. The same name also appears in astronomy, where it refers to observational programs and instrumentation associated with the Gemini Observatory, to the ultra-faint compact Milky Way satellite DELVE 8/Gemini I, to the genomics framework GEMINI (“GEnome MINIng”), and to an underground testbed for next-generation gravitational-wave instrumentation (Team et al., 2023, Kilic et al., 2017, Paila et al., 2013, Andric et al., 5 Sep 2025).

1. Gemini as a multimodal foundation-model family

Gemini, in the machine-learning sense, is a family of multimodal foundation models designed to process interleaved text, image, audio, and video inputs natively and to generate text and images as outputs. The family was introduced in three principal sizes—Ultra, Pro, and Nano—with Nano further split into Nano-1 at 1.8B parameters and Nano-2 at 3.25B parameters. The architecture is based on Transformer decoders, trained for a 32k context length, and uses efficient attention mechanisms including multi-query attention. The system is described as “multimodal from the beginning,” rather than as a text model with vision retrofitted afterward (Team et al., 2023).

The principal technical claim of the initial report is breadth plus frontier performance. Gemini Ultra is reported to advance the state of the art on 30 of 32 benchmarks, including 10 of 12 text/reasoning benchmarks, 9 of 9 image-understanding benchmarks, 6 of 6 video-understanding benchmarks, and 5 of 5 speech-recognition or translation benchmarks. On MMLU, Gemini Ultra reached 90.04%, above the cited human-expert figure of 89.8% and above the prior state of the art at 86.4%. Reported results also include 94.4% on GSM8K with Maj1@32, 53.2% on MATH with 4-shot prompting, 83.6% on BIG-Bench-Hard with 3-shot prompting, 74.4% on HumanEval in the 0-shot setting, and 59.4% pass@1 or 62.4% Maj@32 on MMMU, where the report states that the model outperformed the previous best by more than 5 percentage points (Team et al., 2023).

The multimodal benchmark profile is correspondingly broad. For image understanding, reported Gemini Ultra scores include 82.3% on TextVQA, 90.9% on DocVQA, 80.8% on ChartQA, 80.3% on InfographicVQA, 53.0% on MathVista, 79.5% on AI2D, and 77.8% on VQAv2. For video, the same report lists 62.7 on VATEX, 51.3 on VATEX ZH, 135.4 on YouCook2, 29.9 on NextQA, 52.2 on ActivityNet-QA, and 54.7 on Perception Test MCQA. Gemini Pro, rather than Ultra, was evaluated on several audio tasks and was reported to outperform USM and Whisper across the listed tasks, including 4.9% WER on YouTube ASR, 4.8% on Multilingual Librispeech, 7.6% on FLEURS, 9.1% on VoxPopuli, and 40.1 BLEU on CoVoST 2 speech translation (Team et al., 2023).

The systems report also emphasized training and deployment engineering. Gemini Ultra was trained on a large fleet of TPUv4 accelerators, while other sizes used TPUv5e and TPUv4. The training stack used JAX, Pathways, GSPMD partitioning in XLA, and a MegaScale XLA compiler pass; the paper reports training goodput improved from 85% to 97%. The model was pretrained on a multimodal, multilingual corpus including web documents, books, code, images, audio, and video, with quality filtering, safety filtering, and removal of evaluation data believed to have been seen in training. A synthetic long-context retrieval test reported 98% accuracy across the full 32,768-token context length. A common misconception addressed directly in the paper is that Gemini is merely a LLM with appended modality adapters; the authors explicitly reject that characterization (Team et al., 2023).

2. Gemini-derived embedding models

A second major line of work uses Gemini as the initialization backbone for embedding models. The text-only Gemini Embedding model is described as a unified embedding model that transfers Gemini’s multilingual, code, and semantic understanding into dense vectors intended for classification, similarity, clustering, ranking, reranking, and retrieval. The architecture is encoder-style: an input token sequence is processed by a bidirectional transformer initialized from Gemini, mean pooled, and linearly projected to the final embedding. The model produces 3,072-dimensional embeddings and supports Matryoshka-style subdimensions at 768 and 1,536 (Lee et al., 10 Mar 2025).

Training is based on a contrastive objective with in-batch negatives and task strings, together with a two-stage recipe of pre-finetuning and fine-tuning, followed by Model Soup. A distinctive feature is that Gemini is used not only as the backbone but also as a data engine for synthetic data generation, data filtering, and hard-negative mining. The evaluation centers on MMTEB and cross-lingual retrieval. Reported results include a multilingual MMTEB task mean of 68.32 and task-type mean of 59.64, an English MMTEB task mean of 73.30 and task-type mean of 67.67, and a code mean of 75.5 on MTEB(Code). On cross-lingual retrieval, Gemini Embedding reached 90.42 Recall@5kt on XOR-Retrieve and 64.33 average MRR@10 on XTREME-UP (Lee et al., 10 Mar 2025).

Gemini Embedding 2 extends this formulation to a native multimodal embedding model over text, image, audio, and video, including arbitrary interleaved combinations in a single unified representation space. It is likewise initialized from Gemini, then fine-tuned for embedding and retrieval tasks using large-scale contrastive learning in a multi-task, multi-stage setup. The training pipeline comprises Pre-Fine-Tuning over image, text, and code tasks; Fine-Tuning over text, code, document, image, audio, and video tasks; and Model Soup over multiple fine-tuning runs. The paper again uses Matryoshka representation learning and retains the 3,072-dimensional full embedding with optimized 768- and 1,536-dimensional subspaces (Shanbhogue et al., 26 May 2026).

The reported benchmark profile is deliberately multimodal. The abstract highlights 62.9 R@1 on MSCOCO, 68.8 NDCG@10 on VATEX, 69.9 on MTEB multilingual, and 84.0 on MTEB Code. The body adds 82.3 on CoIR, 64.9 on ViDoRe V2 document retrieval, and an audio result on MSEB of 73.99 average MRR@10 with native audio versus 70.40 with ASR. Zero-shot image-to-text retrieval results are reported in specialized domains as 79.3 R@5 on MicroVQA, 67.7 on ArtCap, 64.4 on AstroLLaVA, 90.2 on Recipe1M ingredients, and 92.1 on Recipe1M instructions. This suggests a recurring architectural role for Gemini: not only as a generative multimodal model, but as a reusable pretrained substrate for retrieval-oriented representation learning across modalities (Shanbhogue et al., 26 May 2026).

3. Domain specialization: education and medicine

Gemini has also served as the basis for domain-specific evaluation and specialization. In education, the relevant study evaluated Gemini 2.5 Pro in an “arena for learning,” a blind, head-to-head, multi-turn benchmark designed around realistic learning scenarios rather than static question answering. The study used a bank of 49 scenarios and a two-stage methodology: in stage one, 189 educators and pedagogy experts carried out 2,666 blind interactions organized into 1,333 head-to-head match-ups; in stage two, 206 educators and experts reviewed recorded match-ups using a 25-item learning rubric. Across the tutoring-quality ranking computed with Elo ratings via a Bradley-Terry model, Gemini 2.5 Pro ranked first overall (Team et al., 30 May 2025).

The headline result was that, excluding ties, experts preferred Gemini 2.5 Pro in 73.2% of match-ups overall. Pairwise preference rates were reported as 71.3% against Claude 3.7 Sonnet, 81.8% against GPT-4o, 74.2% against OpenAI o3, and 61.0% against ChatGPT-4o. The rubric grouped judgments into five principles of good pedagogy, and Gemini 2.5 Pro received the highest marks on each: 82.1% for managing cognitive load, 84.4% for inspiring active learning, 82.8% for deepening metacognition, 82.9% for stimulating curiosity, and 82.0% for adapting to students’ needs and goals. Additional task-specific results included a text re-levelling grade difference of 0.99 with coverage 0.94, 84.1% short-answer assessment accuracy on 2,000 student-written responses from Ghanaian schools, and 87.4% accuracy on mistake identification using the Khan Academy benchmark (Team et al., 30 May 2025).

In medicine, Gemini underlies the Med-Gemini family, which is described as a family rather than a single model. One report defines variants fine-tuned from Gemini 1.0 Pro, Gemini 1.0 Ultra, Gemini 1.5 Pro, and Gemini 1.0 Nano, with web-search integration, multimodal reasoning, long-context processing, and custom modality encoders including an ECG encoder. Across 14 medical benchmarks, Med-Gemini was reported to establish new state of the art on 10. The strongest headline value was 91.1% accuracy on MedQA using an uncertainty-guided search strategy. On 7 multimodal benchmarks, the report states an average relative improvement of 44.5% over GPT-4V, and it also reports 0.77 F1 on a long-EHR “needle-in-a-haystack” retrieval task over records of 200,000 to 700,000 words, together with strong MedVidQA localization performance reaching mIoU 43.4 in the video-only setting and 65.8 with subtitles (Saab et al., 2024).

A complementary medical paper built on Gemini 1.5 Pro with three custom encoders for 2D modalities, 3D volumetric data, and genomics, yielding Med-Gemini-2D, Med-Gemini-3D, and Med-Gemini-Polygenic. The model family was tuned on more than 7 million samples from 3.7 million medical images and cases. In expert-evaluated chest X-ray report generation, 57% of AI reports on normal MIMIC-CXR cases and 43% on abnormal cases were judged “equivalent or better” than the original radiologist report; on IND1, the corresponding values were 96% and 65%. The same paper reported the first large multimodal model-based report generation for 3D CT volumes, with 53% of AI reports deemed clinically acceptable, and stated that Med-Gemini exceeded state of the art or baselines on 17 of 20 tasks in CXR classification and radiology VQA and surpassed baselines across 18 of 20 tasks in histopathology, ophthalmology, and dermatology classification (Yang et al., 2024).

4. Post-training, tool use, and adversarial robustness

Gemini’s deployment profile is strongly tied to post-training, tool use, and security. The original model report describes a conventional post-training stack of prompt-data collection, supervised fine-tuning, reward-model training, and RLHF, yielding product variants for Gemini Apps and Gemini API deployment through Google AI Studio and Vertex AI. Tool use is framed as code generation: the model emits tool calls as code blocks, executes them, observes tool outputs, and continues. The paper reports that, for Gemini API Pro, tool augmentation improved GSM8K from 69.7% to 80.1%, MATH from 30.7% to 41.8%, NQ from 59.0% to 68.0%, and Realtime QA from 39.2% to 70.8%. On factuality-related post-training, the report gives an inaccurate-rate reduction from 6.7% to 3.8%, attribution improvement from 40.2% to 60.0%, and hedging accuracy improvement from 0% to 69.3% (Team et al., 2023).

The security consequence of this agentic design is explicit in later work on indirect prompt injection. That report models the output as $y = M(\text{combine}(user, priv, adv))$, where user input, private context, and adversarially controlled retrieved data are combined in the prompt. The attack objective is to induce a malicious function call $f_{target}(priv)$ that exfiltrates private information. Scenarios include email and calendar tools, private data such as passport numbers, social security numbers, and password reset tokens, and both JSON and non-JSON retrieval formats. Attack success is defined not by textual imitation but by actual malicious function invocation (Shi et al., 20 May 2025).

The empirical result is that an undefended Gemini 2.0 model in function-calling settings is highly vulnerable. The paper states that at least one attack produced more than 70% attack success rate in every evaluated setting, that TAP often reached near-100% ASR, and that one successful trigger against Gemini 2.0 Flash cost less than \$10 to construct. Among evaluated defenses, ICL helped only modestly, spotlighting was stronger but could increase null responses, paraphrasing was often effective, and warning-based defenses could be very effective depending on placement. A central methodological claim is that non-adaptive defense evaluation is misleading: in 16 of 24 cases across 8 defenses and 3 attack families, adaptive attacks were equal to or stronger than non-adaptive ones (Shi et al., 20 May 2025).

The same report describes adversarial fine-tuning of Gemini 2.5 using generated attack data and corrected safe responses. Reported robustness gains include about a 47% average reduction in ASR across the three main attack families. In one comparison, 1,799 triggers optimized on Gemini 2.0 achieved 92% ASR on Gemini 2.0 but only 18% on Gemini 2.5. Combining model-level hardening with the Warning defense reduced ASR to 6.2% in the cited scenario, versus 10.8% for Warning alone against Gemini 2.0. A misconception explicitly rejected by the paper is that higher raw capability implies security; its conclusion is instead that secure deployment requires defense in depth, adaptive evaluation, and continuous reevaluation (Shi et al., 20 May 2025).

5. Gemini in observational astronomy

In astronomy, “Gemini” commonly refers to the Gemini Observatory and its instrumentation. One study reports a “Gemini snapshot radial-velocity survey” of 44 low-mass white dwarf candidates selected from SDSS DR10 spectroscopy. The targets were chosen with $T_{\rm eff} > 12{,}000\,\mathrm{K}$, $M < 0.4\,M_{\odot}$, signal-to-noise greater than 10, and no prior radial-velocity monitoring. Gemini’s role was to provide short, high-cadence time-series spectroscopy with GMOS on an 8 m telescope: each target received 4 to 11 exposures, each 2 to 8 minutes long, over roughly 20 to 30 minutes, a strategy explicitly optimized for sub-hour orbital periods (Kilic et al., 2017).

Follow-up spectroscopy at Gemini and the MMT confirmed four double-degenerate binaries: SDSS J083446.91+304959.2, SDSS J123549.88+154319.3, SDSS J123728.64+491302.6, and SDSS J234248.86+081137.2. Their periods were reported as $0.30079 \pm 0.0011\,\mathrm{d}$, $0.03672 \pm 0.0014\,\mathrm{d}$, $0.10763 \pm 0.0024\,\mathrm{d}$, and $0.16788 \pm 0.0014\,\mathrm{d}$, corresponding to about 7.2 h, 52.9 min, 2.58 h, and 4.03 h. For J123549.88+154319.3, 39 exposures as short as 2 minutes refined the orbit to $P = 52.9\,\mathrm{min}$ and $K = 166.5 \pm 6.2\,\mathrm{km\,s^{-1}}$, with a gravitational-wave merger time of $f_{target}(priv)$0 Myr. High-speed photometry at 10 s cadence and Catalina Sky Survey light curves showed no eclipses, ellipsoidal modulation, or other convincing photometric variability. The broader sample comparison found that the median orbital period dropped from 0.64 d for $f_{target}(priv)$1–$f_{target}(priv)$2 white dwarfs to 0.24 d for $f_{target}(priv)$3 white dwarfs, but the authors did not find a statistically significant mass–period correlation (Kilic et al., 2017).

Gemini South also hosts GHOST, the Gemini High-resolution Optical SpecTrograph, introduced as a facility-class high-resolution optical spectrograph with simultaneous 347–1060 nm coverage in a single exposure. GHOST operates at resolving powers of 56,000 and 76,000, can observe either one or two targets depending on mode, and can reach $f_{target}(priv)$4 in a 1 hr exposure on a $f_{target}(priv)$5 mag target per resolution element under median seeing and dark skies. The instrument is fiber-fed and bench-mounted, with light entering through two integral field units and then traveling by optical fibers to a stabilized spectrograph split at 530 nm into red and blue arms. It was installed on site in June 2022 and reported as fully integrated into queue operations in November 2023 (Kalari et al., 2024).

The same observatory appears in the discovery and confirmation of DELVE 8/Gemini I, a compact Milky Way halo satellite in the constellation Gemini. The object was initially detected in DELVE DR3 with two independent search algorithms, simple and ugali, and then confirmed with Gemini North / GMOS-N imaging reduced with DRAGONS and analyzed with DAOPHOT/ALLSTAR. The system has a heliocentric distance of $f_{target}(priv)$6 kpc, a physical half-light radius of $f_{target}(priv)$7 pc, angular $f_{target}(priv)$8, ellipticity $f_{target}(priv)$9, and mean heliocentric velocity $T_{\rm eff} > 12{,}000\,\mathrm{K}$0. Four members were securely identified with Keck/DEIMOS, including two blue horizontal branch stars, and the brightest member yielded a metallicity upper limit of $T_{\rm eff} > 12{,}000\,\mathrm{K}$1. Morphologically the system lies in the ultra-faint compact-satellite regime defined by $T_{\rm eff} > 12{,}000\,\mathrm{K}$2 and $T_{\rm eff} > 12{,}000\,\mathrm{K}$3 pc, so the paper explicitly states that morphology alone cannot determine whether DELVE 8/Gemini I is an ultra-faint dwarf galaxy or a star cluster (Overdeck et al., 8 Jun 2026).

6. GEMINI as an acronym in genomics and gravitational-wave instrumentation

Outside astronomy and AI, GEMINI is also the name of a genomics framework: GEnome MINIng. This software is designed for integrative analysis of human genetic variation by loading a VCF into a SQLite relational database and combining variant records with sample genotypes, pedigree data from optional PED files, and a wide range of annotations such as dbSNP, 1000 Genomes, ESP, ClinVar, KEGG, HPRD, ENCODE, and UCSC tracks. The database schema includes the tables variants, variant_impacts, samples, resources, and version. Rather than creating one row per sample-genotype pair, GEMINI stores per-variant genotype information as compressed arrays in columns such as gts, gt_types, gt_phases, and gt_depths, a design choice aimed at cohort-scale efficiency (Paila et al., 2013).

The framework supports ad hoc SQL-like querying, a browser interface, and a Python programming interface. Built-in tools include query, stats, annotate, windower, comp_hets, pathways, lof_sieve, interact, auto_rec, auto_dom, de_novo, region, and browser. The paper’s examples emphasize pedigree-aware filtering, such as combining rarity, loss-of-function status, custom regions, and autosomal recessive genotype logic in a single query. Reported performance benchmarks include loading a 12-member exome study with 1.6 million variants in 41 minutes using 4 processors and a 1000 Genomes dataset of 39.7 million variants across 1,092 individuals in 28 hours using 30 processors. For the latter example, the compressed snpEff-annotated VCF occupied 144 GB, whereas the GEMINI database with annotations occupied 78 GB (Paila et al., 2013).

A separate and much newer use of the name refers to an underground R&D facility for next-generation gravitational-wave detectors. This GEMINI is being built at the Laboratori Nazionali del Gran Sasso about 1.4 km underground and is intended as a testbed for seismic isolation, interplatform control, and cryogenic sensor technologies relevant to the Einstein Telescope and the Lunar Gravitational-Wave Antenna. The facility comprises two independently controlled, actively isolated in-vacuum platforms separated by a 3 m baseline, each within a vacuum chamber connected by a central pipe for interferometric relative measurements. The stated control band is 10 mHz to 10 Hz, and the suspended mass per platform is modeled with a total $T_{\rm eff} > 12{,}000\,\mathrm{K}$4 (Andric et al., 5 Sep 2025).

The sensing stack combines T360 GSN broadband seismometers, the COBRI interferometric local displacement sensor, and a Suspension Platform Interferometer for platform-to-platform differential motion. The control model uses complementary filters $T_{\rm eff} > 12{,}000\,\mathrm{K}$5 and $T_{\rm eff} > 12{,}000\,\mathrm{K}$6 with $T_{\rm eff} > 12{,}000\,\mathrm{K}$7 and a blending frequency of 50 mHz, together with plant transfer functions $T_{\rm eff} > 12{,}000\,\mathrm{K}$8 and $T_{\rm eff} > 12{,}000\,\mathrm{K}$9. In ET mode, the goal is minimum residual platform motion and optically rigid interplatform behavior; predicted differential motion is about $M < 0.4\,M_{\odot}$0 at 10 mHz in translation and about $M < 0.4\,M_{\odot}$1 at 10 mHz in pitch. In LGWA mode, the goal shifts to minimizing the error signal seen by a science sensor under test. The facility also includes a cryogenic subsystem centered on a cryobox cooled to about 40 K by a Sumitomo RDK-500B2 cryocooler. A plausible implication is that the name GEMINI now spans nearly the full range from computational biology to underground precision metrology, but in each case it denotes a tightly engineered infrastructure rather than a generic label (Andric et al., 5 Sep 2025).