Compositional Semantic Parsing with Large Language Models (2209.15003v2)

Abstract: Humans can reason compositionally when presented with new tasks. Previous research shows that appropriate prompting techniques enable LLMs to solve artificial compositional generalization tasks such as SCAN. In this work, we identify additional challenges in more realistic semantic parsing tasks with larger vocabulary and refine these prompting techniques to address them. Our best method is based on least-to-most prompting: it decomposes the problem using prompting-based syntactic parsing, then uses this decomposition to select appropriate exemplars and to sequentially generate the semantic parse. This method allows us to set a new state of the art for CFQ while requiring only 1% of the training data used by traditional approaches. Due to the general nature of our approach, we expect similar efforts will lead to new results in other tasks and domains, especially for knowledge-intensive applications.

• The paper introduces dynamic least-to-most prompting by decomposing parsing tasks into hierarchical subproblems for improved compositional generalization.
• It leverages dynamic exemplar selection and iterative processing to guide syntactic analysis and enhance output accuracy.
• Empirical evaluations on CFQ and COGS benchmarks achieve state-of-the-art accuracies of 95% and 99.2%, using only 1% of typical training data.

Compositional Semantic Parsing with LLMs

The paper "Compositional Semantic Parsing with LLMs" addresses the challenges of compositional generalization in semantic parsing tasks using LLMs. Traditional machine learning models, such as transformers and recurrent neural networks, often struggle with compositional generalization, failing to robustly analyze novel combinations of linguistic structures. Previous methods that aimed to address this issue predominantly relied on bespoke architectures or training regimens, which often were task-specific or had limited adaptability to novel data types.

The authors present an approach that reframes this problem using refined prompting strategies, particularly focusing on least-to-most prompting. This strategy decomposes semantic parsing tasks into sequential subtasks, allowing LLMs to process more manageable components iteratively. The authors emphasize a dynamic version of least-to-most prompting they term "dynamic least-to-most prompting," which further leverages LLMs to perform syntactic parses that guide decomposition and exemplar selection.

Key contributions of this work include:

1. Dynamic Least-to-Most Prompting: This extends traditional least-to-most prompting by incorporating syntactic parsing within prompting strategies, allowing for adaptable parsing across complex linguistic tasks. The approach involves decomposing tasks into hierarchical subproblems, dynamically selecting relevant exemplars tailored to the input structure, and using these subproblems iteratively to guide final task completion.
2. Exemplar Selection and Use: Exemplars are dynamically selected from a pre-sampled pool based on decomposition matches, thereby creating a flexible framework for exemplification that accounts for context-specific nuances in real-world tasks.
3. Iterative Task Processing: By iteratively solving subproblems and composing back the derived outputs, the approach increases understanding and accuracy in semantic parsing. Emphasis is placed on the ability to sequentially build up complex structures, leveraging the strength of LLMs in understanding syntactic and semantic nuances.

The practicality of this approach manifests through empirical evaluations on benchmarks like CFQ and COGS. Notably, the method sets a new state-of-the-art accuracy of 95% on CFQ, utilizing a mere 1% of the data required by conventional methods—a significant reduction in training data utilization. For the COGS benchmark, dynamic least-to-most prompting reaches extreme accuracy levels of 99.2%, exhibiting the method's robust generalization capabilities.

Upon analysis, the findings imply a few significant potential impacts on the direction of AI and NLP research:

• Enhanced Generalization: This work suggests a pathway toward models that exhibit more human-like comprehension of language, paving the way for applications where understanding novel compositions reliably is crucial.
• Reduced Data Dependency: By achieving high performance with minimal data, this paper sets a precedent for data efficiency, valuable in settings where data acquisition is constrained.
• Broad Applicability: Given its general-purpose design, dynamic least-to-most prompting is poised to influence various other domains reliant on LLMs, such as knowledge-base reasoning or robotic command execution.

Future developments may witness deeper integration of decomposition techniques into LLM architectures, alongside investigations into adaptive learning algorithms to fine-tune such prompting methods. Moreover, understanding the interplay between syntactic parsing fidelity and exemplar efficacy could unlock further generalization advances. This research broadly showcases how refined large model techniques can shift the paradigm of language processing tasks beyond mere statistical mimicry, toward models that can reason compositionally akin to human logic.