Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog (2104.04923v1)

Abstract: Semantic parsing using sequence-to-sequence models allows parsing of deeper representations compared to traditional word tagging based models. In spite of these advantages, widespread adoption of these models for real-time conversational use cases has been stymied by higher compute requirements and thus higher latency. In this work, we propose a non-autoregressive approach to predict semantic parse trees with an efficient seq2seq model architecture. By combining non-autoregressive prediction with convolutional neural networks, we achieve significant latency gains and parameter size reduction compared to traditional RNN models. Our novel architecture achieves up to an 81% reduction in latency on TOP dataset and retains competitive performance to non-pretrained models on three different semantic parsing datasets. Our code is available at https://github.com/facebookresearch/pytext

• The paper introduces a non-autoregressive approach that predicts the full semantic parse in a single step, significantly reducing latency.
• It presents a LightConv Pointer model which leverages CNNs for efficient on-device processing in resource-constrained environments.
• Empirical results show up to 81% latency reduction with competitive accuracy on datasets like TOP, DSTC2, and SNIPS compared to autoregressive models.

Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog

The paper "Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog" presents a non-autoregressive (NAR) approach to semantic parsing aimed at improving the efficiency of sequence-to-sequence models, particularly for real-time applications. Semantic parsing is crucial in transforming natural language inputs into structured representations understandable by downstream systems, such as conversational assistants. Traditional autoregressive models, despite their accuracy, face challenges with latency due to their sequential nature, limiting their real-time applicability.

Core Contributions

The paper proposes two main contributions:

1. Non-Autoregressive Generation Scheme: The authors introduce a non-autoregressive approach for semantic parsing using sequence-to-sequence models. This approach predicts the entire semantic parse structure in a single step, enhancing parallelization and significantly reducing latency. This method also adapts the Conditional Masked LLM (CMLM) to retain performance while enabling faster decoding.
2. LightConv Pointer Model: Leveraging convolutional neural networks, the LightConv Pointer architecture is introduced to improve model efficiency further. This model architecture suits on-device applications with limited computational resources by providing latency and model size improvements over RNN-based models.

Technical Insights

• Non-Autoregressive Decoding: By modifying the standard seq2seq model objective, the proposed method predicts the entire target structure at once, eliminating the dependencies between target tokens that hinder parallelization.
• Length Prediction: The authors emphasize target length prediction due to its critical role in accurate semantic parsing, introducing a specialized module to enhance prediction accuracy.
• Convolutional Architecture: Augmenting traditional transformer models with CNNs, the architecture achieves efficient encoding and decoding, crucial for low-resource settings.

Empirical Evaluation

The paper demonstrates the effectiveness of the NAR approach on multiple datasets: TOP, DSTC2, and SNIPS. The non-autoregressive model achieves up to an 81% reduction in latency on the TOP dataset with EM accuracies of 80.23% on TOP, 88.16% on DSTC2, and 80.86% on SNIPS. These results are competitive with existing state-of-the-art models lacking pre-training.

Implications and Future Directions

The proposed NAR approach significantly enhances latency without substantial accuracy trade-offs, paving the way for deploying real-time semantic parsing on edge devices. Future work could further explore integrating pre-trained models like RoBERTa and BART with non-autoregressive decoding to potentially enhance performance further. Additionally, addressing longer parse tree structures and session-based understanding scenarios could enhance the model's applicability across more complex dialog systems.

By addressing key limitations of autoregressive approaches, this research provides a promising direction for future advancements in semantic parsing and broader natural language understanding applications.