codec/render: silence hw-decode failures, vram-aware dynamic worker pool

- open_hw_accel marks the device unavailable when the decoder OPEN fails
  (cuvidCreateDecoder OOM at 4K) too, not just device-context creation:
  without it every subsequent decoder session retried CUDA and flooded
  the log per open.
- Decoder gains hardware_decoding(); the oak-render decode-session LRU
  evicts hardware sessions first (each pins a GPU surface pool — ~100 MB
  at 4K), so a full cache cannot exhaust video memory before the next
  open.
- Worker pool count now factors GPU vram: per-worker budget = 1 GiB
  (1080p peak) scaled by pixel ratio + 256 MiB idle floor, 10% reserve
  of free vram; applied when hardware decoding is on (nvidia-smi query,
  None otherwise falls back to the RAM/CPU policy).
- Dynamic pool resize: ProcessDispatcher::set_target_workers grows or
  retires workers; retiring ones stop claiming, drain their in-flight
  batch (future playback frames included), then exit naturally on the
  shutdown signal — no mid-work kill (30 s deadline only as a hung-
  decoder last resort). A retiring worker that dies re-queues its frames
  to surviving workers. Resizes are throttled to 2 s (a resolution burst
  merges; only the latest target applies) so 1080p<->4K flaps cannot
  thrash process spawns.
- RenderManager::set_workspace_size announces the sequence resolution;
  RealEngine calls it from refresh_sequence_info.
- Integration test: shrink 3->1 mid-wave (all frames complete, retired
  workers exit naturally) then regrow 1->3 and render a fresh wave.
This commit is contained in:
2026-08-30 21:59:06 +08:00
parent 5937e557a7
commit 9352fca4a9
9 changed files with 566 additions and 23 deletions
+37 -7
View File
@@ -793,6 +793,14 @@ static DECODERS: std::sync::OnceLock<
/// covers heavy multi-clip montages without reopen thrash; eviction only
/// drops the map entry — an in-flight render keeps its Arc alive and the
/// session dies with the last reference (Drop releases FFmpeg).
///
/// Eviction is hardware-first: hardware sessions pin GPU memory (each
/// NVDEC decoder holds a surface pool — ~100 MB at 4K), so a full cache
/// with live hardware sessions can exhaust the GPU's video memory and
/// make the NEXT decoder open fail with `cuvidCreateDecoder` OOM (the
/// "4K 切换后大量 CUDA_ERROR_OUT_OF_MEMORY 报错" log flood). Software
/// sessions (system RAM only) are evicted only when nothing else is
/// available.
const MAX_CACHED_DECODERS: usize = 16;
/// LRU tick source for [`DECODERS`].
@@ -809,6 +817,29 @@ fn decoders(
.unwrap_or_else(|e| e.into_inner())
}
/// Pick the victim to evict under the LRU cap: the least-recently-used
/// HARDWARE session when one exists (freeing GPU video memory first,
/// which is the scarce resource), otherwise the least-recently-used
/// session of any kind.
fn eviction_victim(
cache: &std::collections::HashMap<
(String, i32),
(Arc<dyn oak_codec::decoder::Decoder>, u64),
>,
) -> Option<(String, i32)> {
cache
.iter()
.filter(|(_, (decoder, _))| decoder.hardware_decoding())
.min_by_key(|(_, (_, t))| *t)
.map(|(k, _)| k.clone())
.or_else(|| {
cache
.iter()
.min_by_key(|(_, (_, t))| *t)
.map(|(k, _)| k.clone())
})
}
/// Open (or reuse) the decoder session for `(filename, stream_index)`.
fn open_decoder(filename: &str, stream_index: i32) -> Result<Arc<dyn oak_codec::decoder::Decoder>> {
let key = (filename.to_string(), stream_index);
@@ -826,14 +857,13 @@ fn open_decoder(filename: &str, stream_index: i32) -> Result<Arc<dyn oak_codec::
.open(&stream)
.map_err(|e| Error::Failed(format!("footage decode open: {e:?}")))?;
let mut cache = decoders();
// LRU eviction: the session is dropped here, but an in-flight render
// holds its own Arc — the FFmpeg context dies with the last reference.
// LRU eviction: hardware-first (free the GPU memory a full cache of
// hardware sessions pins — the scarcity that breaks the next open);
// the victim is dropped here, but an in-flight render holds its own
// Arc — the FFmpeg context (and its GPU surface pool) dies with the
// last reference.
while cache.len() >= MAX_CACHED_DECODERS {
let victim = cache
.iter()
.min_by_key(|(_, (_, t))| *t)
.map(|(k, _)| k.clone());
let Some(victim) = victim else {
let Some(victim) = eviction_victim(&cache) else {
break;
};
cache.remove(&victim);