BLADE: Benchmarking Language Model Agents for Data-Driven Science

Abstract: Data-driven scientific discovery requires the iterative integration of scientific domain knowledge, statistical expertise, and an understanding of data semantics to make nuanced analytical decisions, e.g., about which variables, transformations, and statistical models to consider. LM-based agents equipped with planning, memory, and code execution capabilities have the potential to support data-driven science. However, evaluating agents on such open-ended tasks is challenging due to multiple valid approaches, partially correct steps, and different ways to express the same decisions. To address these challenges, we present BLADE, a benchmark to automatically evaluate agents' multifaceted approaches to open-ended research questions. BLADE consists of 12 datasets and research questions drawn from existing scientific literature, with ground truth collected from independent analyses by expert data scientists and researchers. To automatically evaluate agent responses, we developed corresponding computational methods to match different representations of analyses to this ground truth. Though LLMs possess considerable world knowledge, our evaluation shows that they are often limited to basic analyses. However, agents capable of interacting with the underlying data demonstrate improved, but still non-optimal, diversity in their analytical decision making. Our work enables the evaluation of agents for data-driven science and provides researchers deeper insights into agents' analysis approaches.

• The paper presents a comprehensive guide to configuring pdfLaTeX for creating professionally formatted academic documents.
• It details practical techniques for setting up document preambles, managing figures, tables, and citations to ensure consistency.
• The paper offers actionable troubleshooting solutions for common compilation issues and outlines future integration with modern text editors.

A Comprehensive Guide to Using PDFLaTeX for Document Preparation

This document offers an in-depth examination of utilizing pdf\LaTeX{} as the principal engine for creating PDF files from \LaTeX{} source documents. It outlines various practical steps and considerations necessary for both reviewing and finalizing documents using pdf\LaTeX{}, while also briefly touching upon Xe\LaTeX{}.

Document Setup and Preamble

The paper begins by asserting the importance of choosing the correct document class and preamble settings to ensure optimal functionality and appearance. The recommended approach initiates with the command:

\documentclass[11pt]{article}

This establishes the foundation for the document. Additional guidance is provided for loading the required style files, specifically noting the distinctions between draft and final document versions.

To align the document with Times Roman fonts, authors are instructed to include respective font packages within the preamble. The paper refers users to the source for examples of multiple authors in header fields, ensuring clarity in multi-author attributions.

Footnotes, Tables, and Figures

A segment of the paper is dedicated to the inclusion and formatting of footnotes, tables, and figures:

• Footnotes: Insertion is facilitated by the \footnote command, an essential element in academic writing.
• Tables: Through an example, it demonstrates the creation and captioning of tables without overriding default caption sizes. This ensures consistency in document formatting.
• Figures: The usage of the graphicx package is recommended for including and manipulating graphics. Ensuring consistency in font usage between figures and the main document is emphasized, contributing to a professional appearance. Figures, both individual and side-by-side, are illustrated with comprehensive examples.

Hyperlinks and Citations

The document provides technical resolutions for common errors encountered during compilation, such as issues arising from hyperlink nesting when using pdf\LaTeX{}. The prescribed solution involves upgrading to a more recent version of \LaTeX{}.

For citations, compatibility with natbib styles is stressed, and various citation commands (\citep, \citet, \citealp, etc.) are outlined with examples. This section ensures that the document adheres to established citation norms, promoting consistency and accuracy in referencing.

References and Bib\TeX Files

In constructing a bibliography, the document provides concise instructions for integrating personal Bib\TeX{} files and references from the ACL Anthology. This ensures a streamlined and efficient process for bibliography management, enabling hyperlinks for DOIs and URLs through the hyperref package.

An example of generating a references section using Bib\TeX{} is provided:

\bibliography{custom}

or integrating multiple sources:

\bibliography{anthology,custom}

Equations and Appendices

Equations are presented with clear labeling and referencing techniques, ensuring that mathematical expressions are both readable and easily referable within the text. An example of such an equation is:

A = \pi r^2

For appendices, a switch in the section numbering is advised using the \appendix command, facilitating the organization of supplementary information.

Implications and Future Directions

This comprehensive guide on the usage of pdf\LaTeX{} for document preparation offers both theoretical insights and practical steps, significantly smoothing the scholarly writing process. By adhering to these guidelines, researchers can ensure their documents are professionally formatted, accurately cited, and ready for publication. Moving forward, the seamless integration of advanced text editors and version control systems with these \LaTeX{} practices could further enhance the document preparation experience for researchers.

The document stands as a fundamental resource for researchers aiming to master the intricacies of \LaTeX{} and pdf\LaTeX{} for academic writing, ensuring high standards in document presentation and functionality.