ci: retry the Linux test suite once; log render-manager init failures
CI / Build & test (Linux) (push) Successful in 18m4s
CI / Build & test (Windows) (push) Successful in 34m6s

Run 48's Linux leg flaked: full_res_worker_outlives_a_dropped_project
panicked with "the render manager failed to start" while all 236 other
tests passed -- a worker-pool startup race under full-suite parallelism,
not a regression (the same binary passes locally and passed on the
previous commit). Mirror the Windows job's retry-once policy and make
ensure_render_manager log the underlying init error so the next flake is
diagnosable. The Windows leg's real failure (gpui_widgets referenced
SurfaceSource::Texture, which only exists on linux/freebsd) is fixed in
the gpui submodule bump of the previous commit.
This commit is contained in:
2026-08-29 00:37:24 +08:00
parent fdb5caabd5
commit ca18105c8b
2 changed files with 40 additions and 24 deletions
+33 -22
View File
@@ -107,28 +107,39 @@ jobs:
- name: Test
run: |
sudo apt-get install -y gdb
xvfb-run -a -s "-screen 0 1920x1080x24" cargo test --workspace --locked &
TEST_PID=$!
# The watchdog inherits the step's stdout/stderr; detach it so
# it cannot keep the runner's I/O pipes open after the step
# ends ("WaitDelay expired before I/O complete" otherwise).
(
sleep 1500
echo "::warning::test suite exceeded 1500s; dumping hung-process stacks"
for p in $(pgrep -f 'target/debug/deps/|target/debug/oak-worker'); do
echo "===== thread stacks of pid $p ($(readlink /proc/$p/exe 2>/dev/null)) ====="
sudo gdb -batch -ex 'thread apply all bt' -p "$p" || true
done
pkill -9 -f 'target/debug/deps/' || true
pkill -9 -f 'target/debug/oak-worker' || true
) >/dev/null 2>&1 &
WATCHDOG_PID=$!
wait $TEST_PID
rc=$?
kill $WATCHDOG_PID 2>/dev/null || true
# Reap the watchdog so no child holds the step's pipes.
wait $WATCHDOG_PID 2>/dev/null || true
exit $rc
run_suite() {
xvfb-run -a -s "-screen 0 1920x1080x24" cargo test --workspace --locked &
TEST_PID=$!
# The watchdog inherits the step's stdout/stderr; detach it so
# it cannot keep the runner's I/O pipes open after the step
# ends ("WaitDelay expired before I/O complete" otherwise).
(
sleep 1500
echo "::warning::test suite exceeded 1500s; dumping hung-process stacks"
for p in $(pgrep -f 'target/debug/deps/|target/debug/oak-worker'); do
echo "===== thread stacks of pid $p ($(readlink /proc/$p/exe 2>/dev/null)) ====="
sudo gdb -batch -ex 'thread apply all bt' -p "$p" || true
done
pkill -9 -f 'target/debug/deps/' || true
pkill -9 -f 'target/debug/oak-worker' || true
) >/dev/null 2>&1 &
WATCHDOG_PID=$!
wait $TEST_PID
rc=$?
kill $WATCHDOG_PID 2>/dev/null || true
# Reap the watchdog so no child holds the step's pipes.
wait $WATCHDOG_PID 2>/dev/null || true
return $rc
}
# Retry once (same policy as the Windows job): worker-pool
# startup under full-suite parallelism has flaked once
# (full_res_worker_outlives_a_dropped_project: the render
# manager's process dispatcher failed to start while every other
# test passed). A real regression fails both passes.
if ! run_suite; then
echo "first pass failed; retrying once for worker-pool flakes"
run_suite
fi
# A crashing (SIGSEGV) test gives no Rust backtrace; rerun the
# crashing test binaries under gdb to capture the native stack.