Learning Task Decomposition to Assist Humans in Competitive Programming

Abstract: When using LMs to solve complex problems, humans might struggle to understand the LM-generated solutions and repair the flawed ones. To assist humans in repairing them, we propose to automatically decompose complex solutions into multiple simpler pieces that correspond to specific subtasks. We introduce a novel objective for learning task decomposition, termed assistive value (AssistV), which measures the feasibility and speed for humans to repair the decomposed solution. We collect a dataset of human repair experiences on different decomposed solutions. Utilizing the collected data as in-context examples, we then learn to critique, refine, and rank decomposed solutions to improve AssistV. We validate our method under competitive programming problems: under 177 hours of human study, our method enables non-experts to solve 33.3\% more problems, speeds them up by 3.3x, and empowers them to match unassisted experts.

• The paper introduces the assistive value (AssistV) to evaluate and enhance decomposed solutions for competitive programming tasks.
• It employs a multi-stage process with critique, refine, and rank models leveraging human feedback to optimize task decomposition.
• Experiments show that non-experts solved 33.3% more problems at 3.3x speed, with GPT-4 outperforming human judgment by 15.6%.

Overview of "Learning Task Decomposition to Assist Humans in Competitive Programming"

This paper addresses a significant challenge in leveraging LMs for complex task resolution: LMs often generate solutions that are difficult for humans to understand and rectify. The authors propose a novel approach to assist humans in repairing such solutions by automatically decomposing complex tasks into manageable subtasks, thereby creating decomposed solutions. The primary contribution of this research is the introduction of an innovative objective called the "assistive value" (AssistV). AssistV evaluates the feasibility and speed with which humans can repair these decomposed solutions.

Research Methodology

The authors constructed a dataset capturing human experiences in repairing various decomposed solutions. By leveraging this dataset, they trained LLMs to critique, refine, and rank these decompositions to maximize their assistive value. The methodology involved a multi-stage process:

1. Critique Model ($\pi_{critique}$): This component learns to predict human critique regarding a decomposition's usefulness.
2. Refine Model ($\pi_{refine}$): This model integrates human feedback to enhance the decomposition's quality.
3. Rank Model ($\pi_{rank}$): This stage selects the decomposition with the highest predicted assistive value.

The authors validated their method using competitive programming as a test environment, recruiting a mix of expert and non-expert Python programmers. The experiments included 177 hours of human study, wherein participants were asked to rectify solutions for competitive coding problems.

Key Findings

The research generated notable insights:

• Increased Problem-Solving Efficiency: The method enabled non-expert programmers to solve 33.3% more problems and did so with a 3.3x speed increase compared to traditional methods.
• Expert Matching: When equipped with the proposed task decomposition approach, non-experts achieved performance akin to unassisted experts.
• Improving Assistance Beyond Human Judgments: While human intuition alone was no better than random guesses in predicting the most assistive decompositions, LLMs like GPT-4, trained with the proposed method, surpassed human judgment, showing a 15.6% improvement over GPT-3.5-Turbo.

Implications and Future Directions

The findings herein have profound implications for developing LLMs that can effectively supplement human capabilities in complex problem-solving scenarios. This work suggests that rather than being mere substitutes or competitors in solving problems, LMs can function as effective collaborators, improving human efficiency through task decomposition.

Moreover, this paper paves the way for further optimization of AI-human collaborative frameworks across a variety of domains. Future research might explore extending the assistive value framework to additional tasks beyond competitive programming, assessing its impact in varied contexts such as software engineering, scientific research, and any domain where complex problem-solving is requisite.

By focusing on human-centric AI design principles, this study highlights a move towards more human-friendly AI systems capable of empowering users by amplifying their problem-solving abilities. Specifically, it underscores the potential for AI systems to learn from human feedback loops, thus continuously evolving to deliver increasingly efficient assistive functions. This direction holds promise for the broader field of AI research, particularly in the development of intelligent systems aimed at enhancing human productivity and adapting to user-specific needs.