Agentic Harness Engineering (AHE): Evolving Coding Agent Harnesses with Observability Driven…

Coding agents have made impressive strides on repository-scale tasks and long-horizon terminal workflows. However, much of their real-world performance comes not just from the underlying LLM, but from the harness the collection of tools, prompts, middleware, memory systems, and execution logic surrounding the model.

The Core Problem

• Harnesses are model-specific and need frequent adaptation.
• Trajectories are long and noisy (~10M tokens).
• Edits are hard to attribute and risk regressions.
• Existing self-evolution methods often target only prompts or playbooks, missing the full combinatorial space.

AHE solves this with three matched observability pillars.

The AHE Framework

1. Component Observability (NexAU Harness) The harness is decomposed into seven orthogonal, file-level components in a git-tracked workspace:

• System prompt
• Tool descriptions & implementations
• Middleware
• Skills
• Sub-agent configs
• Long-term memory

Edits are localized, auditable, and revertible. The seed starts minimal (single shell tool) to ensure every addition earns its performance gain.

2. Experience Observability (Agent Debugger) Raw trajectories are distilled into a layered evidence corpus:

• Benchmark-level overview
• Per-task deep-dive reports with root causes
• Links back to raw traces for drill-down

This turns millions of tokens into something an evolve-agent can actually reason over.

3. Decision Observability (Evolve Agent + Manifests) Every proposed edit includes:

• Failure evidence
• Root cause hypothesis
• Targeted fix
• Predicted impact (which tasks should improve/regress)

The next iteration verifies these predictions and reverts failures at file granularity. This creates falsifiable contracts and prevents collapse into trial-and-error.

The Outer Loop (simplified from Algorithm 1):

• Rollout (k=2 per task)
• Clean & attribute prior edits
• Agent Debugger analysis
• Evolve Agent proposes changes + manifest
• Commit, evaluate, keep best-so-far

Results & Visuals

On Terminal-Bench 2 (89 tasks):

AHE reaches 77.0% pass@1 after 10 iterations (from 69.7% seed). It outperforms:

• Codex-CLI (human-designed): 71.9%
• ACE: 68.9%
• TF-GRPO: 72.3%

Cross-Benchmark Transfer (SWE-bench Verified): The frozen AHE harness achieves the highest aggregate success rate while using fewer tokens than the seed and other baselines. Gains are particularly strong on several repos.

Cross-Model Transfer: Consistent +5 to +10+ percentage point gains across alternate base models, suggesting the evolved components encode general engineering knowledge.

Ablations: Gains primarily come from tools, middleware, and long-term memory. System prompt alone can regress structure transfers better than prose strategy.

Why This Matters

• Practical automation of what was manual expert work.
• Transferability reduces the need for per-model/per-benchmark retuning.
• Reliable self-improvement loop via strong observability and falsification.
• Opens the door to continuously evolving agent platforms as base models advance.

Limitations noted in the paper: Non-additive component interactions (stacking good edits has diminishing returns) and the loop being better at fixes than foreseeing regressions.

Implementation Notes (from the GitHub repo)

The framework uses E2B sandboxes for reproducible rollouts, NexAU as the harness substrate, and supports high concurrency experiments. Ready-to-run configs and scripts are provided.

Conclusion

AHE demonstrates that observability is the key enabler for agent-driven harness evolution. By making components, experiences, and decisions fully observable and verifiable, we can let agents autonomously improve their own engineering scaffolding a powerful complement to scaling base models.

This could be as transformative for production agents as advanced prompting techniques were for single shot LLMs.

Read the full paper: arXiv:2604.25850 Explore the code: GitHub repo