Vicuna-13B: Open-Source 13B LLM

• Vicuna-13B is an open-source large language model with 13 billion parameters built on LLaMA and fine-tuned using ChatGPT conversational data.
• It employs a decoder-only Transformer architecture with iterative self-refinement, achieving near ChatGPT performance in dialogue, reasoning, and creative tasks.
• Designed for cost-efficiency and privacy, Vicuna-13B supports high-quality conversational AI on resource-constrained hardware and diverse research applications.

Vicuna-13B is an open-source LLM comprising 13 billion parameters, developed via instruction tuning of Meta’s LLaMA backbone on conversational data distilled from ChatGPT (ShareGPT). It is architected as a decoder-only Transformer and is designed to closely emulate the conversational and reasoning capabilities of proprietary models while remaining computationally accessible, privacy-preserving, and cost-effective. Vicuna-13B achieves approximately 90% of ChatGPT’s conversational quality and, through recent advances in iterative self-refinement, can match or even surpass ChatGPT on certain problem-solving and generative benchmarks. Its performance, extensibility, and open-source licensing make it a pivotal reference point for both LLM research and deployment in sensitive or resource-constrained environments (Ghosal et al., 2023, Shashidhar et al., 2023).

1. Architectural Foundations and Training Paradigm

Vicuna-13B adopts the LLaMA 13B architecture, implementing a decoder-only Transformer with 13 billion parameters. The model weights were initialized from LLaMA and then fine-tuned on a large corpus of ChatGPT-conversation transcripts collected via ShareGPT. This instruction-following fine-tuning approach imbues Vicuna-13B with dialogue coherence, user intent detection, and multi-turn conversational ability. The model supports a maximum input sequence of 1,280 tokens and is typically executed in bfloat16 precision for efficient edge and server deployment. Post-hoc refinement via lightweight parameter-efficient methods (e.g., LoRA adapters) is frequently leveraged in research and deployment, preserving the core model capacity while allowing for fast adaptation (Ghosal et al., 2023).

2. Instructional and Conversational Data Sources

The initial tuning of Vicuna-13B focuses on user–assistant interaction quality. Its training data comprises anonymized dialogues from the ShareGPT platform, representing a distilled version of interactions with ChatGPT. Empirical results from InstructEval demonstrate that this conversational fine-tuning enables Vicuna-13B to achieve 88.72% (equal-weighted) or 94.53% (Vicuna-weighted) of ChatGPT’s output quality across diverse categories including writing, roleplay, common-sense reasoning, counterfactuals, and knowledge-centric queries (as judged by GPT-4). Notably, scores for writing (101.3%), generic prompts (101.09%), and knowledge (102.29%) indicate near parity or slight outperformance relative to ChatGPT in these areas (Shashidhar et al., 2023).

3. Performance Characteristics and Self-Refinement

Vicuna-13B’s zero-shot and few-shot abilities have been systematically benchmarked using the Vicuna benchmark (9 categories, 80 prompts, GPT-4 as judge) and external evaluations such as ARC, MMLU, HellaSwag, and TruthfulQA. After a single iteration of untargeted, domain-agnostic self-refinement—implemented as a three-step loop (generate → critique → revise) without access to external rubrics—the model exhibits significant relative gains:

• Mean equal-weighted score: 95.62% of ChatGPT post-refinement (+6.90 pp, +7.78% improvement)
• Highest absolute improvements: “common-sense” (+13.71 pp), “counterfactual” (+13.44 pp), “generic” (+13.34 pp)
• Modest gains on “math” (+6.66 pp), stability on technical/creative tasks compared to smaller models (Shashidhar et al., 2023)

The iterative scheme operates entirely offline and avoids oracle bias by excluding task-specific score access, making it suitable for privacy-constrained deployment.

4. Cost-Performance Trade-offs and PeRFICS Evaluation

The Performance, Refinement, and Inference Cost Score (PeRFICS) metric offers a formalism for balancing quality gains against hardware cost. For Vicuna-13B, the PeRFICS formula is given by:

$$\Psi(m) = \frac{ \eta\, e^{\kappa(\alpha\, \mathcal{B}(m) + \beta\, \mathcal{I}(m)) } + \rho\, \mathcal{E}(m) } { e^{\gamma\, \mathcal{C}(m)} + \delta }$$

Where:

• $\mathcal{B}(m)$: Baseline score on Vicuna benchmark (94.53)
• $\mathcal{I}(m)$: Post-refinement improvement (7.61)
• $\mathcal{E}(m)$: External average (53.7)
• $\mathcal{C}(m)$: VRAM requirement (7.41 GB @ 4-bit quantization)

Using these metrics, Vicuna-13B ranks third among tested open-source LLMs (behind Alpasta-30B and Vicuna-7B), indicating strong cost-adjusted performance, particularly in settings constrained to single-GPU (consumer-class) hardware (Shashidhar et al., 2023).

Model	PeRFICS Ranking	VRAM (GB/4-bit)	Baseline (%)	Gain (%)
Alpasta-30B	1	15.8	—	—
Vicuna-7B	2	3.7	—	—
Vicuna-13B	3	7.41	94.53	7.61
Guanaco-65B	4	32.0	—	—
Airoboros-7B	5	3.7	—	—

5. Applications, Limitations, and Comparative Context

Vicuna-13B is well suited for applications requiring high conversational fluency, privacy, and modest computational requirements. Its sub-8 GB VRAM footprint enables deployment on RTX 30/40-series GPUs. The self-refinement process is fully on-device, obviating the need for external API calls and enhancing data confidentiality for domains such as email response and document summarization.

Performance characteristics highlight strengths in creative/open-ended tasks but also reveal slight degradation on precise arithmetic and pure code generation benchmarks compared to domain-specialized models. For complex reasoning, encoder–decoder models (e.g., FLAN-T5 11B) maintain an edge (e.g., DROP 67.2% vs Flacuna 43.6%). Notably, Flacuna—formed by LoRA fine-tuning Vicuna-13B on a curated mix of FLAN, code, and GPT-4 datasets—demonstrates an average 5.1-point lift over base Vicuna-13B on reasoning-intensive tasks, at the expense of some writing/relevance metrics (Ghosal et al., 2023). Gains in models like Flacuna arise from both high-quality instruction data and preservation of chat-derived knowledge via ShareGPT/Alpaca data.

6. Practical Access and Usage Recommendations

Researchers can obtain Vicuna-13B weights and its derivatives under open licenses. Deployment is supported via Hugging Face's model repository and the 🤗 Transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("declare-lab/flacuna-13b-v1.0") model = AutoModelForCausalLM.from_pretrained("declare-lab/flacuna-13b-v1.0", device_map="auto")

For problem-solving tasks and reasoning-centric applications (MMLU, DROP, CRASS), Vicuna-13B and Flacuna offer substantial improvements over naïve instruction tuning. Prompt engineering is recommended for “chat” and extended writing domains, and practitioners are advised to retain base Vicuna or use data-balanced instruction sets when optimizing for writing fidelity.

7. Future Directions and Ongoing Research

Research continues on closing the reasoning gap between decoder-only LLMs and encoder–decoder architectures, as well as on scaling domain-agnostic self-refinement processes. The direct impact of high-quality instruction data and modular, adapter-based fine-tuning schemes (e.g., LoRA) on diverse downstream tasks is an area of active empirical investigation. A plausible implication is that as dataset curation and self-guided improvement loops mature, Vicuna-13B and its descendants will further erode the performance differential with proprietary models (Ghosal et al., 2023, Shashidhar et al., 2023).