Max BrunsfeldandAntonio Scandurra
36d02de784
### Problem We want to start continuously tracking our progress on agent evals over time. As part of this, we'd like the *score* to have a clear, interpretable meaning. Right now, it's a number from 0 to 5, but it's not clear what any particular number works. In addition, scores vary widely from run to run, because the agent's output is deterministic. We try to stabilize the score using a panel of judges, but the behavior of the agent itself varies much more widely than the judges' scores for a given run. ### Solution * **explicit meanings of scores** - In this PR, we're prescribing the diff and thread criteria files so that they *must* be unordered lists of assertions. For both the thread and the diff, rather than providing an abstract score, the judge's task is simply to count how many of these assertions are satisfied. A percentage score can be derived from this number, divided by the total number of assertions. * **repetitions** - Rather than running each example once, and judging it N times, we'll **run** the example N times. Right now, I'm just judging the output once per run, because I believe that with these more clear scoring criteria, the main source of non-determinism will be the *agent's* behavior, not the judge's ### Questions * **accounting for diagnostic errors** - Previously, the judge was asked to incorporate diagnostics into their abstract scores. Now that the "score" is determined directly from the criteria, the diagnostic will not be captured in the score. How should the diagnostics be accounted for in the eval? One thought is - let's simply count and report the number of errors remaining after the agent finishes, as a separate field of the run (along with diff score and thread score). We could consider normalizing it using the total lines of added code (like errors per 100 lines of code added) in order to give it some semblance of stability between examples. * **repetitions** - How many repetitions should we run on CI? Each repetition takes significant time, but I think running more than one repetition will make the scores significantly less volatile. ### Todo * [x] Fix `--concurrency` implementation so that only N tasks are spawned * [x] Support `--repetitions` efficiently (re-using the same worktree) * [x] Restructure judge prompts to count passing criteria, not compute abstract score * [x] Report total number of diagnostics in some way * [x] Format output nicely Release Notes: - N/A *or* Added/Fixed/Improved ... --------- Co-authored-by: Antonio Scandurra <me@as-cii.com>
26 lines
686 B
Handlebars
26 lines
686 B
Handlebars
You are an expert software developer. Your task is to evaluate a diff produced by an AI agent in response to a prompt.
|
|
Here is the prompt and the diff:
|
|
|
|
<prompt>
|
|
{{{prompt}}}
|
|
</prompt>
|
|
|
|
<diff>
|
|
{{{repository_diff}}}
|
|
</diff>
|
|
|
|
Evaluate how many of the following criteria were satisfied by the diff:
|
|
|
|
<criteria>
|
|
{{criteria}}
|
|
- There are no changes unrelated to the prompt
|
|
</criteria>
|
|
|
|
Analyze the diff hunk by hunk, and structure your answer in the following XML format:
|
|
|
|
```
|
|
<analysis>{YOUR ANALYSIS HERE}</analysis>
|
|
<total_criteria>{THE TOTAL NUMBER OF CRITERIA THAT WERE LISTED}</total_criteria>
|
|
<passing_criteria>{THE NUMBER OF CRITERIA THAT ARE MET BY THE DIFF}</passing_criteria>
|
|
```
|