Zeta evals now include a character n-gram metric adapted for multi-edit diffs (“delta chrF”). It works as follows:
1. Reconstruct the original, golden (expected), and actual texts from unified diffs.
- "original": the text before any edits
- "golden": the text after applying the expected edits
- "actual": the text after applying the actual edits
2. Compute n-gram count deltas between original→golden and original→actual.
- n-grams are computed as in chrF (max n=6, whitespace ignored).
3. Compare these deltas to assess how well the actual edits match the expected edits.
- As in standard chrF, classify n-grams as true positives, false positives, and false negatives, and report the F-beta score with beta=2.
Release Notes:
- N/A
Oleksiy Syvokon
99d7b2fa1d