Use case Understand a paper

The Objective

The paper writes ECHO as:

L_ECHO = L_GRPO(A) + λ L_Env(O′)

where:
- A = assistant action/command token positions
- O = all observation token positions
- O′ = terminal-output tokens selected for environment prediction
- L_Env = length-normalized cross-entropy on O′
- λ = weight on world-modeling vs policy optimization

Environment objective (§3.1)

The paper defines the environment-prediction loss as length-normalized cross-entropy on terminal-output tokens $\mathcal{O}' \subseteq \mathcal{O}$:

$$L_{\text{Env}}(\theta; \mathcal{O}') = -\frac{1}{Z} \sum_{t \in \mathcal{O}'} \log p_\theta(x_t \mid x_{<t})$$

where $Z = |\mathcal{O}|$ normalizes each sequence by its total observation length (the sum runs over $\mathcal{O}'$). The joint objective is:

$$L_{\text{total}} = L_{\text{GRPO}} + \lambda L_{\text{Env}}$$

$\mathcal{O}'$ is terminal-output tokens only, excluding harness warning prefixes.

Concrete example

Suppose the transcript contains:

• Action: pytest tests/test_api.py
• Observation: failing assertion, traceback, failing test name in terminal output

GRPO uses the final task reward to update the command tokens only.

ECHO also trains on the observation tokens: given prior context and the command, the model should assign high likelihood to the traceback pytest actually returned.

The same pattern applies to ls src (predict returned filenames) and cat config.yaml (predict file contents).