codec/render: silence hw-decode failures, vram-aware dynamic worker pool
- open_hw_accel marks the device unavailable when the decoder OPEN fails (cuvidCreateDecoder OOM at 4K) too, not just device-context creation: without it every subsequent decoder session retried CUDA and flooded the log per open. - Decoder gains hardware_decoding(); the oak-render decode-session LRU evicts hardware sessions first (each pins a GPU surface pool — ~100 MB at 4K), so a full cache cannot exhaust video memory before the next open. - Worker pool count now factors GPU vram: per-worker budget = 1 GiB (1080p peak) scaled by pixel ratio + 256 MiB idle floor, 10% reserve of free vram; applied when hardware decoding is on (nvidia-smi query, None otherwise falls back to the RAM/CPU policy). - Dynamic pool resize: ProcessDispatcher::set_target_workers grows or retires workers; retiring ones stop claiming, drain their in-flight batch (future playback frames included), then exit naturally on the shutdown signal — no mid-work kill (30 s deadline only as a hung- decoder last resort). A retiring worker that dies re-queues its frames to surviving workers. Resizes are throttled to 2 s (a resolution burst merges; only the latest target applies) so 1080p<->4K flaps cannot thrash process spawns. - RenderManager::set_workspace_size announces the sequence resolution; RealEngine calls it from refresh_sequence_info. - Integration test: shrink 3->1 mid-wave (all frames complete, retired workers exit naturally) then regrow 1->3 and render a fresh wave.
This commit is contained in:
@@ -793,6 +793,14 @@ static DECODERS: std::sync::OnceLock<
|
||||
/// covers heavy multi-clip montages without reopen thrash; eviction only
|
||||
/// drops the map entry — an in-flight render keeps its Arc alive and the
|
||||
/// session dies with the last reference (Drop releases FFmpeg).
|
||||
///
|
||||
/// Eviction is hardware-first: hardware sessions pin GPU memory (each
|
||||
/// NVDEC decoder holds a surface pool — ~100 MB at 4K), so a full cache
|
||||
/// with live hardware sessions can exhaust the GPU's video memory and
|
||||
/// make the NEXT decoder open fail with `cuvidCreateDecoder` OOM (the
|
||||
/// "4K 切换后大量 CUDA_ERROR_OUT_OF_MEMORY 报错" log flood). Software
|
||||
/// sessions (system RAM only) are evicted only when nothing else is
|
||||
/// available.
|
||||
const MAX_CACHED_DECODERS: usize = 16;
|
||||
|
||||
/// LRU tick source for [`DECODERS`].
|
||||
@@ -809,6 +817,29 @@ fn decoders(
|
||||
.unwrap_or_else(|e| e.into_inner())
|
||||
}
|
||||
|
||||
/// Pick the victim to evict under the LRU cap: the least-recently-used
|
||||
/// HARDWARE session when one exists (freeing GPU video memory first,
|
||||
/// which is the scarce resource), otherwise the least-recently-used
|
||||
/// session of any kind.
|
||||
fn eviction_victim(
|
||||
cache: &std::collections::HashMap<
|
||||
(String, i32),
|
||||
(Arc<dyn oak_codec::decoder::Decoder>, u64),
|
||||
>,
|
||||
) -> Option<(String, i32)> {
|
||||
cache
|
||||
.iter()
|
||||
.filter(|(_, (decoder, _))| decoder.hardware_decoding())
|
||||
.min_by_key(|(_, (_, t))| *t)
|
||||
.map(|(k, _)| k.clone())
|
||||
.or_else(|| {
|
||||
cache
|
||||
.iter()
|
||||
.min_by_key(|(_, (_, t))| *t)
|
||||
.map(|(k, _)| k.clone())
|
||||
})
|
||||
}
|
||||
|
||||
/// Open (or reuse) the decoder session for `(filename, stream_index)`.
|
||||
fn open_decoder(filename: &str, stream_index: i32) -> Result<Arc<dyn oak_codec::decoder::Decoder>> {
|
||||
let key = (filename.to_string(), stream_index);
|
||||
@@ -826,14 +857,13 @@ fn open_decoder(filename: &str, stream_index: i32) -> Result<Arc<dyn oak_codec::
|
||||
.open(&stream)
|
||||
.map_err(|e| Error::Failed(format!("footage decode open: {e:?}")))?;
|
||||
let mut cache = decoders();
|
||||
// LRU eviction: the session is dropped here, but an in-flight render
|
||||
// holds its own Arc — the FFmpeg context dies with the last reference.
|
||||
// LRU eviction: hardware-first (free the GPU memory a full cache of
|
||||
// hardware sessions pins — the scarcity that breaks the next open);
|
||||
// the victim is dropped here, but an in-flight render holds its own
|
||||
// Arc — the FFmpeg context (and its GPU surface pool) dies with the
|
||||
// last reference.
|
||||
while cache.len() >= MAX_CACHED_DECODERS {
|
||||
let victim = cache
|
||||
.iter()
|
||||
.min_by_key(|(_, (_, t))| *t)
|
||||
.map(|(k, _)| k.clone());
|
||||
let Some(victim) = victim else {
|
||||
let Some(victim) = eviction_victim(&cache) else {
|
||||
break;
|
||||
};
|
||||
cache.remove(&victim);
|
||||
|
||||
Reference in New Issue
Block a user