oak-team/oak-gpui
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

eval: Fine-grained assertions (#29246)

Browse Source 
- Support programmatic examples
([example](https://github.com/zed-industries/zed/blob/17feb260a0919e102ae4e220669c467885bf4b57/crates/eval/src/examples/file_search.rs))
- Combine data-driven example declarations into a single `.toml` file
([example](https://github.com/zed-industries/zed/blob/17feb260a0919e102ae4e220669c467885bf4b57/crates/eval/src/examples/find_and_replace_diff_card.toml))
- Run judge on individual assertions (previously called "criteria")
- Report judge and programmatic assertions in one combined table

Note: We still need to work on concept naming 

<img width=400
src="https://github.com/user-attachments/assets/fc719c93-467f-412b-8d47-68821bd8a5f5">

Release Notes:

- N/A

---------

Co-authored-by: Richard Feldman <oss@rtfeldman.com>
Co-authored-by: Max Brunsfeld <maxbrunsfeld@gmail.com>
Co-authored-by: Thomas Mickley-Doyle <tmickleydoyle@gmail.com>
This commit is contained in:
Agus Zubiaga
2025-04-22 23:58:58 -03:00
committed by GitHub
co-authored by Richard Feldman Max Brunsfeld Thomas Mickley-Doyle
parent 0d3fe474db
commit ce1a674eba
18 changed files with 1969 additions and 1229 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/eval/src/judge_diff_prompt.hbs
+9 -9
View File
@@ -1,5 +1,5 @@
You are an expert software developer. Your task is to evaluate a diff produced by an AI agent in response to a prompt.
Here is the prompt and the diff:
You are an expert software developer. Your task is to evaluate a diff produced by an AI agent
in response to a prompt. Here is the prompt and the diff:
<prompt>
{{{prompt}}}
@@ -9,17 +9,17 @@ Here is the prompt and the diff:
{{{repository_diff}}}
</diff>
Evaluate how many of the following criteria were satisfied by the diff:
Evaluate whether or not the diff passes the following assertion:
<criteria>
{{criteria}}
- There are no changes unrelated to the prompt
</criteria>
<assertion>
{{assertion}}
</assertion>
Analyze the diff hunk by hunk, and structure your answer in the following XML format:
```
<analysis>{YOUR ANALYSIS HERE}</analysis>
<total_criteria>{THE TOTAL NUMBER OF CRITERIA THAT WERE LISTED}</total_criteria>
<passing_criteria>{THE NUMBER OF CRITERIA THAT ARE MET BY THE DIFF}</passing_criteria>
<passed>{PASSED_ASSERTION}</passed>
```
Where `PASSED_ASSERTION` is either `true` or `false`.