oak-team/oak-gpui
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

zeta2: Compute diff-aware chrF metric (#43485)

Browse Source 
Zeta evals now include a character n-gram metric adapted for multi-edit diffs (“delta chrF”). It works as follows:

1. Reconstruct the original, golden (expected), and actual texts from unified diffs.
   - "original": the text before any edits
   - "golden": the text after applying the expected edits
   - "actual": the text after applying the actual edits

2. Compute n-gram count deltas between original→golden and original→actual.
   - n-grams are computed as in chrF (max n=6, whitespace ignored).

3. Compare these deltas to assess how well the actual edits match the expected edits.
   - As in standard chrF, classify n-grams as true positives, false positives, and false negatives, and report the F-beta score with beta=2.

Release Notes:

- N/A
This commit is contained in:
Oleksiy Syvokon
2025-11-27 11:10:35 +02:00
committed by GitHub
parent 91400e7489
commit 99d7b2fa1d
4 changed files with 465 additions and 125 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/language/src/buffer.rs
+2 -2
View File
@@ -758,8 +758,8 @@ impl EditPreview {
.to_point(&self.applied_edits_snapshot);
let start = Point::new(start.row.saturating_sub(3), 0);
let old_end = Point::new(old_end.row + 3, 0).min(self.old_snapshot.max_point());
let new_end = Point::new(new_end.row + 3, 0).min(self.applied_edits_snapshot.max_point());
let old_end = Point::new(old_end.row + 4, 0).min(self.old_snapshot.max_point());
let new_end = Point::new(new_end.row + 4, 0).min(self.applied_edits_snapshot.max_point());
Some(unified_diff(
&self