More resilient eval (#32257)

Bubbles up rate limit information so that we can retry after a certain
duration if needed higher up in the stack.

Also caps the number of concurrent evals running at once to also help.

Release Notes:

- N/A
This commit is contained in:
Ben Brandt
2025-06-09 18:07:22 +00:00
committed by GitHub
parent fa54fa80d0
commit e4bd115a63
22 changed files with 147 additions and 56 deletions
@@ -682,11 +682,12 @@ mod tests {
_: &AsyncApp,
) -> BoxFuture<
'static,
http_client::Result<
Result<
BoxStream<
'static,
http_client::Result<LanguageModelCompletionEvent, LanguageModelCompletionError>,
Result<LanguageModelCompletionEvent, LanguageModelCompletionError>,
>,
LanguageModelCompletionError,
>,
> {
unimplemented!()