Transferability of harness rankings across platforms

Determine whether the performance ranking of Mingbird, goose, opencode, and agent-mini transfers from the evaluated Windows 11 integrated-GPU environment to Linux, macOS, discrete-GPU systems, and other memory configurations.

Background

The main evaluation compares four agent harnesses on a single Windows 11 machine with an integrated GPU and 32 GB of shared memory. Because hardware, operating-system behavior, shell semantics, encoding, and path handling can all affect agent execution, the observed ranking may be platform-specific.

The paper identifies agent-mini’s 35B collapse as specifically interacting with Windows shell behavior and notes that Mingbird itself is Windows-first and untested elsewhere. Establishing whether the reported ranking persists across operating systems, GPU types, and memory envelopes would determine the external validity of the benchmark conclusions.

References

We do not know how the ranking transfers to Linux, to discrete GPUs, or to different memory envelopes; agent-mini's 35B collapse is specifically a Windows interaction and would presumably not reproduce on POSIX shells.

— Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks  (2610.02001 - Wang et al., 1 Oct 2026) in Section 5, “Threats to Validity,” paragraph “Single machine and platform”