Solving Problems of Unknown Difficulty

Abstract: This paper studies how uncertainty about problem difficulty shapes problem-solving strategies. I develop a dynamic model where an agent solves a problem by brainstorming approaches of unknown quality and allocating a fixed effort budget among them. Success arrives from spending effort pursuing good approaches, at a rate determined by the unknown problem difficulty. The agent balances costly exploration (expanding the set of approaches) with exploitation (pursuing existing approaches). Failures could signal either a bad idea or a hard problem, and this uncertainty generates novel dynamics: optimal search alternates between trying new approaches and revisiting previously abandoned ones. I then examine a principal-agent environment, where moral hazard arises on the intensive margin: how the agent explores. Dynamic commitment leads contracts to frontload incentives, which can be counteracted by the presence of learning. The framework reflects scientific discovery, product development, and other creative work, providing insights into innovation and organizational design.

• The paper introduces a model that dynamically balances exploration of new approaches with recall of past ones under uncertain problem difficulty.
• It differentiates from traditional multi-armed bandit frameworks by coupling learning across arms via a latent difficulty parameter, influencing incentive design.
• The findings inform optimal dynamic contracts and strategic search behaviors applicable to scientific research, innovation, and creative exploration.

Solving Problems Under Unknown Difficulty: A Dynamic and Strategic Perspective

Overview and Motivation

"Solving Problems of Unknown Difficulty" (2604.00156) presents a formal model to understand search, experimentation, and incentive design in environments where both the viability of potential approaches and the fundamental problem difficulty are unknown. The framework is motivated by real-world contexts such as scientific research, technological innovation, and entrepreneurship, where agents must engage in costly creative exploration and cannot distinguish between unproductive efforts due to flawed ideas versus inherent hardness.

Crucially, the model departs from classical multi-armed bandit setups by making difficulty a shared latent parameter that endogenously correlates learning across alternative approaches. The agent’s failures are ambiguous—providing only entangled statistical evidence about the quality of attempted approaches and the global rate of progress achievable.

Formal Model and Key Mechanisms

Single-Agent Dynamic Search

An agent sequentially allocates continuous effort among previously brainstormed approaches (arms); novel approaches are generated at a fixed cost. Each new approach is valid with ex-ante probability $\nu_0$ and is only productive under the "easy" global difficulty state ($\lambda = \lambda_E$), which applies uniformly to all arms. With probability $\delta_0$, the problem is "hard" ($\lambda = \lambda_H < \lambda_E$), which means valid arms yield breakthroughs at a much lower rate. The agent discounts future payoffs at rate $r$ and only observes successes and failures (i.e., the process is bandit-like with censored feedback).

The agent faces a strategic breadth-vs-depth allocation: continued exploitation of existing approaches versus incurring costs for expanding the search space. Exploration and belief-updating are highly interdependent over time and across alternatives.

Departure from Standard Bandits

The critical distinction is the endogenous, information-theoretic coupling of beliefs about each arm via the latent state $\theta$ (difficulty). This invalidates indexability and the tractable decomposition present in Gittins-style bandit solutions, leading instead to a correlated, restless-bandit dynamic with high-dimensional beliefs.

Behavioral Implications

Ambiguous failures: A central result is that failures on an arm do not merely reduce its perceived validity but also induce dynamic learning about the difficulty of the underlying problem, which then recursively modifies beliefs about all other arms.

The agent’s optimal policy alternates between (a) exploration—generating and focusing initial effort on new approaches, and (b) recall/task-juggling—redistributing effort to previously abandoned arms as updates on the global difficulty state unfold. This contrasts with classical models (with known difficulty), where abandoned arms are never revisited, and agents do not diversify effort.

Characterization and Dynamics of the Optimal Policy

Benchmark: Known Difficulty

When the global difficulty (success rate) is known, the agent sequentially generates a new approach and explores each to a static threshold $K^*$, after which the arm is abandoned and a new one is brainstormed. There is no recall and no parallelization: effort is concentrated, and each viable approach is explored exhaustively and then never revisited.

The optimal threshold $K^*$ is strictly increasing in the approach cost $c$ and strictly decreasing in the arrival rate $\lambda$. Interestingly, $\lambda = \lambda_E$0 may be non-monotone in the discount rate $\lambda = \lambda_E$1 (see Figure 1):

Figure 1: Non-monotone effect of time pressure on the threshold $\lambda = \lambda_E$2. Example plotted for $\lambda = \lambda_E$3, $\lambda = \lambda_E$4, $\lambda = \lambda_E$5.

Unknown Difficulty: Endogenous Recall and Task-Splitting

When there is aggregate uncertainty over difficulty, the optimal time/effort thresholds for exploration become state-dependent sequences $\lambda = \lambda_E$6, with $\lambda = \lambda_E$7 increasing in the number of brainstormed approaches. The agent spends more cumulative effort on each new approach before switching, reflecting the shifting posterior over difficulty and the decreasing anticipated value of further search as failure persists.

More importantly, recall and task-juggling emerge endogenously: previous arms are optimally revisited and explored in parallel with newer arms, a sharp contrast to models with known difficulty or independent arm rewards.

Beliefs about approach quality and problem tractability evolve in a sophisticated manner, forming non-trivial stochastic paths (see Figure 2):

Figure 2: Belief over quality of old approaches when brainstorming new approaches.

These non-monotonic belief updates drive periods where the agent is more optimistic about previously-abandoned approaches as accumulating failures shift posterior weight toward global difficulty, thus increasing the likelihood assigned to prior arms being valid but merely unlucky in the prior exploitation round.

Continuum Limit and Application to Agency Problems

To enable tractable dynamic game and contract-theoretic analysis, the model develops a limiting "continuum of approaches" framework. Breadth and depth become continuous control variables, with the overall distribution of breakthrough times being an aggregate over possible valid approaches and realization of the common difficulty state.

This limit enables closed-form Euler–Lagrange characterizations and comparative static analysis for agent and principal-agent problems.

Principal-Agent Contracting and the Incentive to Explore

A principal contracts with an agent to incentivize creative search when effort and exploration allocation are unobservable (moral hazard arises on the intensive—not extensive—margin).

Key results:

• Optimal static (time-invariant) contracts induce insufficient creative search: The agent under-explores relative to the social planner, as the cost of broadening search is unobserved and not perfectly contractible.
• Optimal dynamic contracts generally **frontload incentives—contradicting classical dynamic moral hazard theory (which predicts backloaded compensation under extensive margin risk):**
  • The principal offers higher equity shares for early success to induce exploration, dissipating the effectiveness of incentives over time as the marginal impact of exploration diminishes.
  • This produces a monotonically declining agent share (see Figure 3):
  • 

  • Figure 3: The optimal dynamic contract share given to the agent over time, and the share granted in the no-commitment environment. Solution for $\lambda = \lambda_E$8, $\lambda = \lambda_E$9, $\delta_0$0, $\delta_0$1.

• When compared to the static (or no-commitment) contract, the dynamic (commitment) contract sustains greater exploration breadth early on but reverts to less creative activity for long periods without success (see Figure 4):

  Figure 4: The path of exploration in the dynamic contract vs the no-commitment equilibrium. Solution parameters as above.

• Learning-driven backloading: As pessimism accumulates (i.e., as the belief that the problem is impossible increases), the principal is, at times, incentivized to temporarily increase the agent's share (backloading) to encourage continued exploration. This force can induce non-monotonic contracts, especially when failures increasingly signal impossibility rather than mere luck (see Figure 5):

  Figure 5: The optimal contract when learning about impossibility; each color illustrates the contract for a different prior.

Complex non-monotonic paths for the agent's share $\delta_0$2 can arise, featuring both "early tolerance for failure" and eventual reversion to low-incentive backstops as learning saturates (see Figure 6):

Figure 6: Share alpha offered over time.

Theoretical and Practical Implications

Theoretical

• The model introduces an analytically tractable correlated multi-armed bandit structure, extending well beyond classical index and independence-based approaches.
• It emphasizes the importance of restless, non-separable exploration and endogenously-interdependent beliefs, with immediate consequences for the design of algorithms in adaptive search and learning contexts.
• In dynamic contracting, the result that intensive-margin (creative allocation) moral hazard induces frontloaded, not backloaded, incentive provision is a substantial departure from classical repeated agency models.

Practical

• The results have direct application to the design of research and innovation contracts: tolerance for failure, timing of rewards, and dynamic adjustment to accumulated pessimism must be tailored to the structure of ambiguity in feasibility rather than merely effort observability.
• The behavioral insight that optimal search involves dynamic recall and reuse of previously dismissed ideas, and temporary shifts in attention allocation, aligns with empirical patterns observed in scientific discovery and technological breakthroughs.

Conclusion

This work delivers a rich, technically rigorous framework for the study of dynamic experimentation, exploration, and incentive design in uncertain, creative environments. Through the introduction of global uncertainty in difficulty, the analysis predicts dynamic revisiting of old ideas, optimal parallelization, and nuanced, non-monotonic incentive contracts that depart from canonical results in economics and RL.

The continuum model opens multiple directions for future development, including robustness to alternative uncertainty structures, multi-agent and competitive search, and generalizations to possible endogenously evolving difficulty. The implications for algorithmic experimentation, research management, and organizational design are substantial, especially as search problems become increasingly complex, high-dimensional, and strategically coupled.