Value of adaptive human contributions in conversation

Define and measure the value of adaptive human contributions in multi-turn conversations when the evaluator observes the human’s messages but lacks a description of the adaptive policy that produced them, accounting for both the computation used to formulate those inputs and the adaptivity of the human’s strategy.

Background

The paper’s prompt-value framework evaluates a single human input in a fixed context and can be applied sequentially to a realized multi-turn conversation. This sequential accounting values each human message conditional on the transcript at which it arrives and sums the resulting increments, but it does not account for the computation used to formulate the messages or for the adaptivity of the human’s strategy. The unresolved problem is to develop a method that captures the value of adaptive human contributions under the limited-access setting in which an evaluator sees the messages but not the policy generating them. The paper relates this challenge to the interactive-Turing-machine treatment of conversation proposed by Halpern and Pass, while noting that such an approach is less suitable for the paper’s setting.