feat(oakrender): render-process isolation S1 - dispatcher, scheduler, real worker
Per the M15 design (docs/zh/plans/riir/M15-render-process-isolation.md): - ipc.rs moved into oakrender with protocol v2: hello_caps, render_batch, batch_accepted, frame_failed; main-process-assigned slots; BGRA8 slot format. POSIX shm verified to 1GiB on macOS. - ProcessDispatcher: spawns oak-worker processes, handshake, stdio NDJSON control, shm segment lifecycle with generation-tagged keys, crash detection with bounded restart and frame redispatch, zero-copy ShmFrameRef delivery and copy counters. - PreviewScheduler: interleaved batch claiming (frame % W per worker, no work stealing), seek > playback-distance > background priority, credit-based flow control, crash recovery. - oak-worker renders for real: graph snapshot deserialization, montage decode+composite straight into the assigned shm slot, F32->BGRA8 final conversion in-worker, OFX plugin executor installed in-worker, crash hooks for isolation testing. Thread pool coexists for now (S2 removes it). Integration tests cover two-worker zero-copy rendering, crash isolation with redelivery, and real H.264 footage decode into slots.
This commit is contained in:
Generated
+6
@@ -4775,6 +4775,9 @@ name = "oak-worker"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"oakcore-rs",
|
||||
"oaknode",
|
||||
"oakplugin",
|
||||
"oakrender",
|
||||
"serde",
|
||||
"serde_json",
|
||||
@@ -4886,11 +4889,14 @@ dependencies = [
|
||||
name = "oakrender"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"oakcodec",
|
||||
"oakcommon",
|
||||
"oakcore-rs",
|
||||
"oaknode",
|
||||
"ocio-rs",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"thiserror 2.0.20",
|
||||
"wgpu 25.0.2",
|
||||
]
|
||||
|
||||
@@ -30,11 +30,24 @@ serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
|
||||
# POSIX shm_open/ftruncate/mmap/munmap/shm_unlink for the shared-memory
|
||||
# frame-slot transport (src/ipc.rs).
|
||||
# frame-slot transport (the oakrender::ipc shim above). libc also powers
|
||||
# the SIGSEGV crash hook (OAK_WORKER_CRASH_ON_TICKET) used by the crash-
|
||||
# isolation integration tests.
|
||||
libc = "0.2"
|
||||
|
||||
# M14 R2: oak-worker is a PURE module-crate consumer — the worker runtime
|
||||
# (src/worker.rs, src/ipc.rs) lives in this binary and calls the oak*
|
||||
# rlibs directly (oakrender for the render backend + color config). No
|
||||
# liboakengine dylib, no C ABI, no build.rs link step.
|
||||
# M15 S1: real per-process rendering — the worker loads the project graph
|
||||
# itself (oaknode::serializer) and runs nodes through the plugin runtime
|
||||
# (oakplugin::node_factory::install_render_executor wires the CPU render
|
||||
# executor). oakcodec/ocio-rs/ffmpeg arrive transitively through
|
||||
# oaknode/oakplugin — the same single-lib unification the engine uses.
|
||||
oakcore-rs = { path = "../oakcore" }
|
||||
oaknode = { path = "../oaknode" }
|
||||
oakplugin = { path = "../oakplugin" }
|
||||
|
||||
# M14 R2 / M15 S1: oak-worker is a PURE module-crate consumer — the
|
||||
# worker runtime (src/worker.rs) lives in this binary and calls the oak*
|
||||
# rlibs directly. oakrender supplies the render backend, the CPU
|
||||
# evaluation path and the shared ipc protocol (src/ipc.rs is a shim
|
||||
# re-exporting oakrender::ipc). No liboakengine dylib, no C ABI, no
|
||||
# build.rs link step.
|
||||
oakrender = { path = "../oakrender" }
|
||||
|
||||
+66
-39
@@ -19,10 +19,16 @@ dylib is needed at build or run time.
|
||||
`oakengine_worker_main()` and owns the whole runtime: render backend
|
||||
selection through the oakrender crate's direct Rust API (dynamic →
|
||||
OpenGL fallback), the startup handshake and the NDJSON control loop.
|
||||
`src/main.rs` only scans argv for `--backend` and forwards.
|
||||
- `src/ipc.rs` owns the shared-memory frame-slot transport (the real
|
||||
`SpscRingBuffer` + `FrameSlotPool` over POSIX `shm_open`/`mmap`) and the
|
||||
NDJSON control-plane message structs.
|
||||
Since M15 S1 it also renders for real: `load_graph` deserializes the
|
||||
snapshot through `oaknode::serializer`, and `render_frame` /
|
||||
`render_batch` render through `oakrender::eval` (generated frames,
|
||||
footage decode via oakcodec/ffmpeg, montage compositing) directly into
|
||||
the main-assigned shared-memory slots.
|
||||
- `src/ipc.rs` is a shim re-exporting `oakrender::ipc` (M15 S1): the
|
||||
NDJSON protocol and the shared-memory frame-slot transport moved to the
|
||||
oakrender crate so both ends of the pipe link one copy (the
|
||||
main-process dispatcher in `oakrender::procpool` creates the segments;
|
||||
this worker attaches to them).
|
||||
|
||||
The oakrender module crate (`../oakrender`) is a plain Rust dependency;
|
||||
it depends on `ocio-rs` with the `bundled` feature, whose first-time build
|
||||
@@ -39,49 +45,66 @@ CARGO_TARGET_DIR=/path/to/oak/crates/oakrender/target cargo build --release
|
||||
Same flow as the C++ main, in the same order:
|
||||
|
||||
1. **parse `--backend <name>`** (default `opengl`; `none` skips
|
||||
renderer creation and the process exits 1, like the C++ main).
|
||||
renderer creation and the process exits 1, like the C++ main;
|
||||
`cpu` is the M15 headless render mode — no renderer, but the session
|
||||
stays fully operational and renders through the CPU evaluation path).
|
||||
2. **initialize the render backend** (inside `src/worker.rs`): the
|
||||
oakrender `DisplayRenderer` direct Rust API, falling back to the direct
|
||||
OpenGL renderer exactly like the C++ `create_renderer()` fallback
|
||||
chain.
|
||||
chain. Then the runtime services load (color-manager default config,
|
||||
the oakplugin render executor).
|
||||
3. **write the startup handshake** (protocol version 1, empty shared-memory
|
||||
geometry — same as the C++ worker's startup handshake; the parent
|
||||
creates the segments and announces their geometry in its reply).
|
||||
4. **serve the NDJSON control loop** on stdin/stdout until a `shutdown`
|
||||
message or EOF: `handshake` attaches the announced shared-memory
|
||||
frame-slot pools through the real transport; `load_graph` /
|
||||
`render_frame` / `cancel` / `shutdown` are dispatched by the session.
|
||||
Responses are one compact JSON line per message.
|
||||
frame-slot pools through the real transport and answers `hello_caps`
|
||||
(protocol v2: supported slot formats + max slot size); `load_graph`
|
||||
deserializes the graph snapshot; `render_frame` renders one frame
|
||||
into an acquired slot; `render_batch` renders a batch of
|
||||
main-assigned-slot tickets (`batch_accepted` claim confirmation, then
|
||||
one `frame_ready`/`frame_failed` per ticket); `cancel` / `shutdown`
|
||||
are dispatched by the session. Responses are one compact JSON line
|
||||
per message.
|
||||
|
||||
## Implemented vs stubbed (nothing is faked)
|
||||
|
||||
**Real:** argument parsing, render backend initialization (real wgpu
|
||||
renderer, dynamic → OpenGL fallback), startup handshake, NDJSON framing,
|
||||
renderer, dynamic → OpenGL fallback), runtime initialization (color
|
||||
config + oakplugin render executor), startup handshake, NDJSON framing,
|
||||
message validation (protocol version, handshake geometry, `load_graph`
|
||||
file existence/size — the same messages the C++ worker emits), the
|
||||
**shared-memory frame-slot transport** (`src/ipc.rs` — POSIX
|
||||
`shm_open`/`mmap`/`munmap`/`shm_unlink`, the SPSC ring buffer and the
|
||||
frame-slot pool with the exact version-1 shared layout; a `handshake`
|
||||
genuinely attaches the output and input pools), unknown-type/
|
||||
malformed-message errors, shutdown/EOF termination.
|
||||
file existence/size), the **shared-memory frame-slot transport**
|
||||
(`oakrender::ipc` — POSIX `shm_open`/`mmap`/`munmap`/`shm_unlink`, the
|
||||
SPSC ring buffer and the frame-slot pool with the exact version-1 shared
|
||||
layout; a `handshake` genuinely attaches the output and input pools),
|
||||
**graph deserialization** (`oaknode::serializer::load`, plus the minimal
|
||||
`{"project_copy":N}` identity payload), **frame rendering into shm
|
||||
slots** (`render_frame` v1 + `render_batch` v2: generated frames,
|
||||
footage decode, montage compositing, end-of-pipe F32→BGRA8 conversion),
|
||||
unknown-type/malformed-message errors, shutdown/EOF termination.
|
||||
|
||||
**Stubbed (documented in `src/worker.rs`):**
|
||||
|
||||
| area | reason |
|
||||
|---|---|
|
||||
| node-graph deserialization (`load_graph` beyond the file checks) | the oaknode crate is a `todo!()` skeleton |
|
||||
| frame rendering (`render_frame`) | no graph/render-pipeline backing (the shm frame-slot transport is attached, but there is no graph to render) |
|
||||
|
||||
Stubbed requests answer with a clear `{"type":"error","message":…}` that
|
||||
names the missing piece (a `render_frame` error also carries the ticket,
|
||||
mirroring the C++ `error_message()` shape). A real `load_graph` on a
|
||||
non-existent/empty file produces the C++-identical error before reaching
|
||||
the stub.
|
||||
**Deferred to M15 S2/S3 (documented in `src/worker.rs`):** the loaded
|
||||
project's node-graph render path (plugin-node evaluation per graph
|
||||
snapshot update) — today tickets render from their wire spec
|
||||
(montage/footage/generate), which covers the preview pipeline.
|
||||
|
||||
**Deviation from the C++:** the startup handshake omits `gl_major`/
|
||||
`gl_minor` — the oakrender module exposes no GL context version (the C++
|
||||
worker reads them off its `QOpenGLContext`).
|
||||
|
||||
## Crash-isolation test hooks
|
||||
|
||||
The batch render path honors two environment variables used by the
|
||||
crash-isolation integration tests (`tests/procpool_integration.rs`):
|
||||
|
||||
- `OAK_WORKER_CRASH_ON_TICKET=<n>` — raise `SIGSEGV` while rendering
|
||||
ticket `n` (like a real plugin crash).
|
||||
- `OAK_WORKER_CRASH_MARKER=<path>` — when the marker file exists the
|
||||
crash is skipped; the hook writes the marker before dying, making the
|
||||
crash one-shot so the restarted worker renders the re-queued frame.
|
||||
|
||||
They are test-only; unset in production.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
@@ -90,20 +113,24 @@ src/
|
||||
forwards to worker::worker_main
|
||||
worker.rs the real worker runtime: backend selection (oakrender
|
||||
DisplayRenderer, dynamic -> OpenGL fallback), WorkerSession,
|
||||
handshake + NDJSON loop (M14 R2: the facade's port, owned by
|
||||
this binary since the facade C ABI was cut)
|
||||
ipc.rs control-plane message structs + NDJSON framing (serde),
|
||||
AND the real shared-memory frame-slot transport
|
||||
(SpscRingBuffer + FrameSlotPool over POSIX shm)
|
||||
tests/worker.rs binary-level tests (--backend none exit 1)
|
||||
handshake + NDJSON loop, real load_graph + render_frame +
|
||||
render_batch (M15 S1)
|
||||
ipc.rs shim re-exporting oakrender::ipc (M15 S1: both pipe ends
|
||||
link one copy of the protocol + shm transport)
|
||||
tests/worker.rs binary-level tests (--backend none exit 1;
|
||||
--backend cpu handshake + clean exit)
|
||||
tests/procpool_integration.rs M15 S1 end-to-end: real workers spawned by
|
||||
oakrender::procpool::ProcessDispatcher — batch
|
||||
renders into shm slots, crash isolation with
|
||||
restart + re-dispatch, zero-copy assertions
|
||||
```
|
||||
|
||||
The NDJSON control-loop behavior is exercised in-process in `src/worker.rs`
|
||||
against the local real shared memory (no GPU needed via `--backend none`
|
||||
sessions); a binary-level loop test would require a working GPU backend and
|
||||
is deliberately not part of the unit suite. Run the binary against a
|
||||
created segment to see the real attach path:
|
||||
against the local real shared memory (`--backend none` / `--backend cpu`
|
||||
sessions, no GPU needed); `tests/procpool_integration.rs` drives real
|
||||
worker processes end-to-end through the main-process dispatcher. Run the
|
||||
binary against a created segment to see the real attach path:
|
||||
|
||||
```sh
|
||||
target/release/oak-worker --backend opengl <<< '{"type":"shutdown"}'
|
||||
target/release/oak-worker --backend cpu <<< '{"type":"shutdown"}'
|
||||
```
|
||||
|
||||
+9
-1461
File diff suppressed because it is too large
Load Diff
+692
-50
@@ -23,43 +23,59 @@
|
||||
//! backend through the oakrender crate's direct Rust API
|
||||
//! ([`oakrender::backend::DisplayRenderer`]), falling back to the
|
||||
//! direct OpenGL renderer exactly like the C++ `create_renderer()`
|
||||
//! chain.
|
||||
//! chain. The headless `"cpu"` backend (M15 S1) skips the renderer
|
||||
//! entirely — the render path is CPU evaluation + decode, driven
|
||||
//! through `render_batch`.
|
||||
//! - **The session.** [`WorkerSession`] holds the renderer, the
|
||||
//! shared-memory frame-slot pools ([`crate::ipc::FrameSlotPool`]) and
|
||||
//! the shutdown flag, and answers one NDJSON control message at a time.
|
||||
//! loaded node graph, the shared-memory frame-slot pools
|
||||
//! ([`crate::ipc::FrameSlotPool`]) and the shutdown flag, and answers
|
||||
//! one NDJSON control message at a time.
|
||||
//! - **The main loop.** [`worker_main`] creates the session, loads the
|
||||
//! runtime config, writes the startup handshake, and serves the
|
||||
//! stdin/stdout NDJSON loop until a `shutdown` message or EOF.
|
||||
//! runtime config (including the oakplugin render executor), writes
|
||||
//! the startup handshake, and serves the stdin/stdout NDJSON loop
|
||||
//! until a `shutdown` message or EOF.
|
||||
//!
|
||||
//! The control-plane protocol is the same NDJSON the C++ worker speaks
|
||||
//! (`engine/render/ipc/ipcmessage.cpp`): one compact JSON object per line,
|
||||
//! `"type"`-dispatched ([`crate::ipc`]), with `handshake` carrying the
|
||||
//! shared-memory geometry the worker attaches to via the real
|
||||
//! [`crate::ipc`] transport. `load_graph`/`render_frame` reproduce the
|
||||
//! C++ validation and then answer with the documented "not yet available"
|
||||
//! errors (the oaknode graph crate is still a skeleton).
|
||||
//! Real rendering landed in M15 S1: `load_graph` deserializes the graph
|
||||
//! snapshot file (oaknode project XML, with the minimal
|
||||
//! `{"project_copy":N}` payload fallback); `render_frame` and
|
||||
//! `render_batch` render through [`oakrender::eval`] (generated frames,
|
||||
//! footage decode, montage compositing) directly into the main-assigned
|
||||
//! shm slots and publish `frame_ready` / `frame_failed` (protocol v2).
|
||||
|
||||
use std::io::{self, BufRead, Write};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use serde_json::Value;
|
||||
use serde_json::{json, Value};
|
||||
|
||||
use oakcore_rs::{PixelFormat, Rational};
|
||||
use oakrender::backend::{BackendKind, DisplayRenderer};
|
||||
use oakrender::eval;
|
||||
use oakrender::ticket::{MontageClip, VideoTicketParams};
|
||||
|
||||
use crate::ipc::{
|
||||
error_message, write_message, FrameSlotPool, HandshakeMsg, LoadGraphMsg, RenderFrameMsg,
|
||||
SharedMemoryRegion, ShmMode, TYPE_CANCEL, TYPE_HANDSHAKE, TYPE_LOAD_GRAPH, TYPE_RENDER_FRAME,
|
||||
TYPE_SHUTDOWN,
|
||||
error_message, write_message, BatchTicketSpec, FrameSlotPool, FrameSlotMeta, HandshakeMsg,
|
||||
LoadGraphMsg, RenderBatchMsg, RenderFrameMsg, SharedMemoryRegion, ShmMode, TYPE_CANCEL,
|
||||
TYPE_HANDSHAKE, TYPE_LOAD_GRAPH, TYPE_RENDER_BATCH, TYPE_RENDER_FRAME, TYPE_SHUTDOWN,
|
||||
SLOT_FORMAT_BGRA8,
|
||||
};
|
||||
use crate::{log_error, PROTOCOL_VERSION};
|
||||
|
||||
/// Why `load_graph` answers "not yet available" (after the real file checks).
|
||||
const GRAPH_STUB: &str = "load_graph: node-graph deserialization is not yet available in the \
|
||||
Rust worker (the oaknode crate is a todo!() skeleton; see worker/rust/README.md)";
|
||||
|
||||
/// Why `render_frame` answers "not yet available".
|
||||
const RENDER_STUB: &str = "render_frame: frame rendering is not yet available in the Rust \
|
||||
worker (no node-graph or render-pipeline backing; the shm frame-slot transport is \
|
||||
attached but there is no graph to render; see worker/rust/README.md)";
|
||||
/// A loaded graph snapshot (M15 S1): the snapshot file path plus what it
|
||||
/// deserialized into — a full oaknode project, or only the copied-project
|
||||
/// identity (the minimal `{"project_copy":N}` payload the
|
||||
/// [`oakrender::worker::GraphSnapshotStore`] writes before the app wires
|
||||
/// full graph uploads in S2).
|
||||
struct LoadedGraph {
|
||||
/// Snapshot file path (S2: graph_update diffing is path-based).
|
||||
#[allow(dead_code)]
|
||||
path: String,
|
||||
/// The deserialized project (S2: node-graph render path; today only
|
||||
/// montage/footage/generate tickets use the loaded context).
|
||||
#[allow(dead_code)]
|
||||
project: Option<Arc<Mutex<oaknode::project::Project>>>,
|
||||
project_copy: u64,
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Renderer (backend selection)
|
||||
@@ -71,6 +87,13 @@ pub fn is_no_backend(backend: &str) -> bool {
|
||||
backend.is_empty() || backend.eq_ignore_ascii_case("none")
|
||||
}
|
||||
|
||||
/// Whether `backend` is the M15 headless CPU render mode: like "none"
|
||||
/// (no GPU renderer) but the session stays fully operational — frames
|
||||
/// render through the CPU evaluation path ([`oakrender::eval`]).
|
||||
pub fn is_cpu_backend(backend: &str) -> bool {
|
||||
backend.eq_ignore_ascii_case("cpu")
|
||||
}
|
||||
|
||||
/// A live, initialized oakrender display renderer (destroyed on drop).
|
||||
pub struct Renderer {
|
||||
/// The oakrender crate's value-typed display renderer (single-lib
|
||||
@@ -148,6 +171,10 @@ pub struct WorkerSession {
|
||||
output_pool: Option<FrameSlotPool>,
|
||||
input_region: Option<SharedMemoryRegion>,
|
||||
input_pool: Option<FrameSlotPool>,
|
||||
graph: Option<LoadedGraph>,
|
||||
/// Reusable F32 staging buffer (BGRA8 slot conversion; the F32
|
||||
/// pipeline renders there before the end-of-pipe format convert).
|
||||
f32_scratch: Vec<u8>,
|
||||
}
|
||||
|
||||
impl WorkerSession {
|
||||
@@ -155,8 +182,10 @@ impl WorkerSession {
|
||||
/// `oakengine_worker_session_create()`: "none"/"" skips renderer
|
||||
/// creation, anything else initializes the render backend through the
|
||||
/// oakrender crate's direct Rust API (dynamic -> OpenGL fallback).
|
||||
/// The M15 `"cpu"` backend is the headless render mode: no renderer,
|
||||
/// CPU evaluation + decode via [`oakrender::eval`].
|
||||
pub fn create(backend: &str) -> Result<WorkerSession, String> {
|
||||
let renderer = if is_no_backend(backend) {
|
||||
let renderer = if is_no_backend(backend) || is_cpu_backend(backend) {
|
||||
None
|
||||
} else {
|
||||
Some(Renderer::create(backend)?)
|
||||
@@ -169,6 +198,8 @@ impl WorkerSession {
|
||||
output_pool: None,
|
||||
input_region: None,
|
||||
input_pool: None,
|
||||
graph: None,
|
||||
f32_scratch: Vec::new(),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -198,8 +229,14 @@ impl WorkerSession {
|
||||
"runtime: color-manager default config failed ({e}); continuing"
|
||||
));
|
||||
}
|
||||
// M15 S1: the plugin execution stack lives in the worker process
|
||||
// (OFX crashes take down this process, not the editor — design
|
||||
// §3.6). oakplugin installs its render driver into the oakrender
|
||||
// executor slot.
|
||||
log_error("runtime: installing oakplugin render executor");
|
||||
oakplugin::node_factory::install_render_executor();
|
||||
log_error(
|
||||
"runtime: config / node factory / frame manager / disk manager / project \
|
||||
"runtime: config / frame manager / disk manager / project \
|
||||
serializer have no Rust backing in the worker binary; skipped",
|
||||
);
|
||||
self.runtime_initialized = true;
|
||||
@@ -335,12 +372,23 @@ impl WorkerSession {
|
||||
self.input_pool = Some(input_pool);
|
||||
}
|
||||
|
||||
// Success: no response (worker.cpp leaves `response` untouched).
|
||||
None
|
||||
// Success: protocol v2 answers the geometry handshake with the
|
||||
// capability announcement (worker.cpp left the response empty;
|
||||
// main now waits for hello_caps to mark the worker alive).
|
||||
Some(json!({
|
||||
"type": crate::ipc::TYPE_HELLO_CAPS,
|
||||
"protocol_version": PROTOCOL_VERSION,
|
||||
"formats": [PixelFormat::F32 as i32, SLOT_FORMAT_BGRA8],
|
||||
"max_slot_bytes": hs.slot_data_bytes,
|
||||
}))
|
||||
}
|
||||
|
||||
/// `load_graph`: the file checks are real (mirror worker.cpp
|
||||
/// `load_graph()`); the deserialization is the documented stub.
|
||||
/// `load_graph` (M15 S1): the file checks mirror worker.cpp; the
|
||||
/// payload is deserialized for real — an oaknode project XML
|
||||
/// ([`oaknode::serializer::load`]) or the minimal
|
||||
/// `{"project_copy":N}` identity payload written by the snapshot
|
||||
/// store before full graph uploads land in S2. Success answers
|
||||
/// nothing (v1 semantics); failures answer an `error` message.
|
||||
fn handle_load_graph(&mut self, msg: &Value) -> Option<Value> {
|
||||
let load: LoadGraphMsg = match serde_json::from_value(msg.clone()) {
|
||||
Ok(l) => l,
|
||||
@@ -361,19 +409,466 @@ impl WorkerSession {
|
||||
load.path,
|
||||
md.len()
|
||||
));
|
||||
Some(error_message(GRAPH_STUB, None))
|
||||
let content = match std::fs::read_to_string(&load.path) {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
return Some(error_message(
|
||||
&format!("graph file unreadable: {e}"),
|
||||
None,
|
||||
))
|
||||
}
|
||||
};
|
||||
match oaknode::serializer::load(&content) {
|
||||
Ok(project) => {
|
||||
self.graph = Some(LoadedGraph {
|
||||
path: load.path.clone(),
|
||||
project: Some(project),
|
||||
project_copy: 0,
|
||||
});
|
||||
log_error("LoadGraph: oaknode project deserialized");
|
||||
None
|
||||
}
|
||||
Err(graph_err) => {
|
||||
// Fallback: the snapshot store's minimal payload
|
||||
// `{"project_copy":N}` (identity-only graph context).
|
||||
if let Ok(v) = serde_json::from_str::<Value>(&content) {
|
||||
if let Some(pc) = v.get("project_copy").and_then(Value::as_u64) {
|
||||
self.graph = Some(LoadedGraph {
|
||||
path: load.path.clone(),
|
||||
project: None,
|
||||
project_copy: pc,
|
||||
});
|
||||
log_error(&format!(
|
||||
"LoadGraph: identity-only snapshot (project_copy {pc})"
|
||||
));
|
||||
return None;
|
||||
}
|
||||
}
|
||||
Some(error_message(
|
||||
&format!("graph deserialization failed: {graph_err}"),
|
||||
None,
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `render_frame`: the graph/render pipeline has no Rust backing, so a
|
||||
/// render request is answered with a clear error carrying the ticket.
|
||||
/// `render_frame` (v1 single-frame path, M15 S1 real): generate the
|
||||
/// frame through [`oakrender::eval`], write it into an acquired shm
|
||||
/// slot and answer `frame_ready`. The v1 message carries no montage
|
||||
/// or footage fields, so this path renders the pipeline's generated
|
||||
/// frame; montage/footage tickets arrive via `render_batch`.
|
||||
fn handle_render_frame(&mut self, msg: &Value) -> Option<Value> {
|
||||
let render: RenderFrameMsg = match serde_json::from_value(msg.clone()) {
|
||||
Ok(r) => r,
|
||||
Err(_) => return Some(error_message("invalid render_frame message", None)),
|
||||
};
|
||||
Some(error_message(RENDER_STUB, Some(render.ticket)))
|
||||
let pool = match self.output_pool.as_ref() {
|
||||
Some(p) if p.is_valid() => p,
|
||||
_ => {
|
||||
return Some(error_message(
|
||||
"render_frame: no shared-memory pool attached",
|
||||
Some(render.ticket),
|
||||
))
|
||||
}
|
||||
};
|
||||
let (w, h) = if render.width > 0 && render.height > 0 {
|
||||
(render.width, render.height)
|
||||
} else {
|
||||
(
|
||||
oakrender::frame::VideoParamsPod::DEFAULT_WIDTH,
|
||||
oakrender::frame::VideoParamsPod::DEFAULT_HEIGHT,
|
||||
)
|
||||
};
|
||||
let format = if render.format < 0 {
|
||||
PixelFormat::F32
|
||||
} else {
|
||||
match render.format {
|
||||
f if f == PixelFormat::U8 as i32 => PixelFormat::U8,
|
||||
f if f == PixelFormat::U10 as i32 => PixelFormat::U10,
|
||||
f if f == PixelFormat::U16 as i32 => PixelFormat::U16,
|
||||
f if f == PixelFormat::F16 as i32 => PixelFormat::F16,
|
||||
f if f == PixelFormat::F32 as i32 => PixelFormat::F32,
|
||||
other => {
|
||||
return Some(error_message(
|
||||
&format!("render_frame: unsupported format {other}"),
|
||||
Some(render.ticket),
|
||||
))
|
||||
}
|
||||
}
|
||||
};
|
||||
let time = Rational::new(render.time_num, render.time_den);
|
||||
let frame = match eval::generate_frame(time, (w, h), format) {
|
||||
Ok(f) => f,
|
||||
Err(e) => {
|
||||
return Some(error_message(
|
||||
&format!("render_frame: generation failed: {e}"),
|
||||
Some(render.ticket),
|
||||
))
|
||||
}
|
||||
};
|
||||
|
||||
let slot = match self.acquire_slot(pool) {
|
||||
Some(s) => s,
|
||||
None => {
|
||||
return Some(error_message(
|
||||
"render_frame: no free shm slot",
|
||||
Some(render.ticket),
|
||||
))
|
||||
}
|
||||
};
|
||||
// Write meta + pixels into the slot, then publish.
|
||||
let data_size = frame.data.len();
|
||||
if data_size > pool.slot_data_bytes() {
|
||||
return Some(error_message(
|
||||
"render_frame: frame larger than the shm slot",
|
||||
Some(render.ticket),
|
||||
));
|
||||
}
|
||||
// SAFETY: `slot` was acquired; the slot block and meta are live
|
||||
// shared memory of the attached pool.
|
||||
unsafe {
|
||||
std::ptr::copy_nonoverlapping(frame.data.as_ptr(), pool.slot_data(slot), data_size);
|
||||
let meta = &mut *pool.meta(slot);
|
||||
*meta = FrameSlotMeta::default();
|
||||
meta.id = render.ticket;
|
||||
meta.time_num = render.time_num;
|
||||
meta.time_den = render.time_den;
|
||||
meta.width = w;
|
||||
meta.height = h;
|
||||
meta.format = format as i32;
|
||||
meta.channel_count = 4;
|
||||
meta.linesize = frame.linesize_bytes() as i32;
|
||||
meta.data_size = data_size as i32;
|
||||
if !pool.publish(slot) {
|
||||
return Some(error_message(
|
||||
"render_frame: ready ring full",
|
||||
Some(render.ticket),
|
||||
));
|
||||
}
|
||||
}
|
||||
Some(json!({
|
||||
"type": crate::ipc::TYPE_FRAME_READY,
|
||||
"ticket": render.ticket,
|
||||
"slot": slot,
|
||||
}))
|
||||
}
|
||||
|
||||
/// `render_batch` (protocol v2; M15 S1): claim confirmation followed
|
||||
/// by in-order rendering of every ticket into its main-assigned shm
|
||||
/// slot. Responses stream to `out`: one `batch_accepted`, then one
|
||||
/// `frame_ready` or `frame_failed` per ticket. Crashes the process
|
||||
/// deliberately when the crash-mode environment asks for it (the
|
||||
/// crash-isolation test hook).
|
||||
fn handle_render_batch_stream(
|
||||
&mut self,
|
||||
line: &str,
|
||||
out: &mut impl Write,
|
||||
) -> io::Result<()> {
|
||||
let batch: RenderBatchMsg = match serde_json::from_str(line) {
|
||||
Ok(b) => b,
|
||||
Err(_) => {
|
||||
return write_message(out, &error_message("invalid render_batch message", None))
|
||||
}
|
||||
};
|
||||
|
||||
// Explicit claim confirmation (design §3.3): these tickets are
|
||||
// owned by this worker now — no work stealing. The `type` tag is
|
||||
// built by hand because [`BatchAcceptedMsg`] only carries the
|
||||
// payload fields.
|
||||
let accepted = json!({
|
||||
"type": crate::ipc::TYPE_BATCH_ACCEPTED,
|
||||
"batch_id": batch.batch_id,
|
||||
"tickets": batch.tickets.iter().map(|t| t.ticket).collect::<Vec<_>>(),
|
||||
});
|
||||
write_message(out, &accepted)?;
|
||||
out.flush()?;
|
||||
|
||||
for spec in &batch.tickets {
|
||||
// Crash-isolation test hook: OAK_WORKER_CRASH_ON_TICKET=<n>
|
||||
// segfaults while rendering ticket n. A marker file (env
|
||||
// OAK_WORKER_CRASH_MARKER) makes the crash one-shot so the
|
||||
// restarted worker renders the frame for real.
|
||||
self.maybe_crash_for_testing(spec.ticket);
|
||||
|
||||
let response = match self.render_ticket_to_slot(spec) {
|
||||
Ok(slot) => json!({
|
||||
"type": crate::ipc::TYPE_FRAME_READY,
|
||||
"ticket": spec.ticket,
|
||||
"slot": slot,
|
||||
}),
|
||||
Err(e) => {
|
||||
log_error(&format!(
|
||||
"render_batch: ticket {} failed: {e}",
|
||||
spec.ticket
|
||||
));
|
||||
json!({
|
||||
"type": crate::ipc::TYPE_FRAME_FAILED,
|
||||
"ticket": spec.ticket,
|
||||
"error": e,
|
||||
})
|
||||
}
|
||||
};
|
||||
write_message(out, &response)?;
|
||||
out.flush()?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The crash-mode test hook (see [`Self::handle_render_batch_stream`]).
|
||||
fn maybe_crash_for_testing(&self, ticket: i64) {
|
||||
let Ok(want) = std::env::var("OAK_WORKER_CRASH_ON_TICKET") else {
|
||||
return;
|
||||
};
|
||||
let Ok(crash_on) = want.parse::<i64>() else {
|
||||
return;
|
||||
};
|
||||
if crash_on != ticket {
|
||||
return;
|
||||
}
|
||||
let marker = std::env::var("OAK_WORKER_CRASH_MARKER").ok();
|
||||
let should_crash = match &marker {
|
||||
Some(path) => !std::path::Path::new(path).exists(),
|
||||
None => true,
|
||||
};
|
||||
if !should_crash {
|
||||
return;
|
||||
}
|
||||
if let Some(path) = &marker {
|
||||
let _ = std::fs::write(path, b"crashed");
|
||||
}
|
||||
log_error(&format!("crash mode: dying on ticket {ticket}"));
|
||||
// Raise SIGSEGV like a real plugin crash; abort as the fallback.
|
||||
unsafe { libc::raise(libc::SIGSEGV) };
|
||||
std::process::abort();
|
||||
}
|
||||
|
||||
/// Acquire a free output slot, polling until one appears (the free
|
||||
/// ring is the only filler-side entry point; flow control). `None`
|
||||
/// on shutdown or after the 30 s safety deadline.
|
||||
fn acquire_slot(&self, pool: &FrameSlotPool) -> Option<u32> {
|
||||
let deadline = Instant::now() + Duration::from_secs(30);
|
||||
let mut slot = 0u32;
|
||||
loop {
|
||||
if self.shutdown_requested {
|
||||
return None;
|
||||
}
|
||||
// SAFETY: valid attached pool; the worker is the filler, so
|
||||
// popping the free ring is its SPSC role.
|
||||
if unsafe { pool.acquire(&mut slot) } {
|
||||
return Some(slot);
|
||||
}
|
||||
if Instant::now() > deadline {
|
||||
return None;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(1));
|
||||
}
|
||||
}
|
||||
|
||||
/// Render one batch ticket into its main-assigned slot (acquire,
|
||||
/// render, publish). Returns the published slot index.
|
||||
fn render_ticket_to_slot(&mut self, spec: &BatchTicketSpec) -> Result<u32, String> {
|
||||
// Clone the pool view (a cheap mapping-shared copy) so the render
|
||||
// path below can borrow `self` mutably (scratch buffer).
|
||||
let pool = match self.output_pool.clone() {
|
||||
Some(p) if p.is_valid() => p,
|
||||
_ => return Err("no shared-memory pool attached".to_string()),
|
||||
};
|
||||
|
||||
// Flow control: acquire through the free ring. Main seeds and
|
||||
// releases slots in assignment order, so the pop yields exactly
|
||||
// the assigned slot — anything else is a protocol violation.
|
||||
let acquired = self
|
||||
.acquire_slot(&pool)
|
||||
.ok_or_else(|| "no free shm slot (shutdown or timeout)".to_string())?;
|
||||
if acquired != spec.slot as u32 {
|
||||
return Err(format!(
|
||||
"slot assignment mismatch: acquired {acquired}, assigned {}",
|
||||
spec.slot
|
||||
));
|
||||
}
|
||||
let slot = acquired;
|
||||
|
||||
let result = self.render_spec_pixels(spec, &pool);
|
||||
match result {
|
||||
Ok(()) => {
|
||||
// SAFETY: `slot` was acquired above and rendered into.
|
||||
let published = unsafe { pool.publish(slot) };
|
||||
if !published {
|
||||
Err("ready ring full".to_string())
|
||||
} else {
|
||||
Ok(slot)
|
||||
}
|
||||
}
|
||||
// The slot was acquired but never published; main recycles it
|
||||
// when it sees frame_failed (the worker cannot push back to
|
||||
// the free ring — that is the drainer's SPSC role).
|
||||
Err(e) => Err(e),
|
||||
}
|
||||
}
|
||||
|
||||
/// Render `spec` into the slot's data block and fill the slot meta.
|
||||
fn render_spec_pixels(&mut self, spec: &BatchTicketSpec, pool: &FrameSlotPool) -> Result<(), String> {
|
||||
let w = spec.width;
|
||||
let h = spec.height;
|
||||
if w <= 0 || h <= 0 {
|
||||
return Err(format!("bad render size {w}x{h}"));
|
||||
}
|
||||
let time = Rational::new(spec.time_num, spec.time_den);
|
||||
let params = self.ticket_params(spec, time);
|
||||
|
||||
let bgra8 = spec.format == SLOT_FORMAT_BGRA8;
|
||||
let (dst_bpp, dst_linesize) = if bgra8 { (4, w * 4) } else { (16, w * 16) };
|
||||
let dst_need = (h as usize) * (dst_linesize as usize);
|
||||
if dst_need > pool.slot_data_bytes() {
|
||||
return Err(format!(
|
||||
"frame {}x{} needs {dst_need} bytes, slot holds {}",
|
||||
w,
|
||||
h,
|
||||
pool.slot_data_bytes()
|
||||
));
|
||||
}
|
||||
let _ = dst_bpp;
|
||||
|
||||
// SAFETY: `spec.slot` was acquired by render_ticket_to_slot; the
|
||||
// block is live shared memory of the attached pool.
|
||||
let dst = unsafe {
|
||||
std::slice::from_raw_parts_mut(pool.slot_data(spec.slot as u32), pool.slot_data_bytes())
|
||||
};
|
||||
|
||||
if !bgra8 {
|
||||
// F32 RGBA: render straight into the slot (no staging copy).
|
||||
render_f32_into(spec, ¶ms, time, (w, h), &mut dst[..dst_need])?;
|
||||
} else {
|
||||
// BGRA8: render the F32 pipeline frame into the session
|
||||
// scratch, then convert into the slot (the end-of-pipe format
|
||||
// convert is not an extra frame copy, design §3.1).
|
||||
let f32_need = (w as usize) * (h as usize) * 16;
|
||||
if self.f32_scratch.len() < f32_need {
|
||||
self.f32_scratch.resize(f32_need, 0);
|
||||
}
|
||||
render_f32_into(spec, ¶ms, time, (w, h), &mut self.f32_scratch[..f32_need])?;
|
||||
convert_f32_rgba_to_bgra8(&self.f32_scratch[..f32_need], &mut dst[..dst_need]);
|
||||
}
|
||||
|
||||
// Slot meta (fresh each publish).
|
||||
// SAFETY: slot in range of the attached pool.
|
||||
unsafe {
|
||||
let meta = &mut *pool.meta(spec.slot as u32);
|
||||
*meta = FrameSlotMeta::default();
|
||||
meta.id = spec.ticket;
|
||||
meta.time_num = spec.time_num;
|
||||
meta.time_den = spec.time_den;
|
||||
meta.width = w;
|
||||
meta.height = h;
|
||||
meta.format = spec.format;
|
||||
meta.channel_count = spec.channels.max(4);
|
||||
meta.linesize = dst_linesize;
|
||||
meta.data_size = dst_need as i32;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Map a wire ticket spec to the eval producer's ticket params.
|
||||
fn ticket_params(&self, spec: &BatchTicketSpec, time: Rational) -> VideoTicketParams {
|
||||
let footage = if spec.footage_file.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some((spec.footage_file.clone(), spec.footage_stream))
|
||||
};
|
||||
let montage: Vec<MontageClip> = spec
|
||||
.montage
|
||||
.iter()
|
||||
.map(|c| MontageClip {
|
||||
filename: c.filename.clone(),
|
||||
stream_index: c.stream_index,
|
||||
in_time: Rational::new(c.in_num, c.in_den),
|
||||
out_time: Rational::new(c.out_num, c.out_den),
|
||||
media_in: Rational::new(c.media_in_num, c.media_in_den),
|
||||
gain: c.gain,
|
||||
})
|
||||
.collect();
|
||||
VideoTicketParams {
|
||||
viewer: self
|
||||
.graph
|
||||
.as_ref()
|
||||
.map(|g| g.project_copy)
|
||||
.unwrap_or(0),
|
||||
time,
|
||||
force_size: Some((spec.width, spec.height)),
|
||||
force_format: Some(PixelFormat::F32),
|
||||
cache: None,
|
||||
cache_dir: None,
|
||||
cache_id: None,
|
||||
cache_timebase: None,
|
||||
footage,
|
||||
montage,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Render the F32 RGBA pipeline frame for `spec` into `dst`
|
||||
/// (`(w*h*16)` bytes): generated transparent black, footage decode,
|
||||
/// or montage composite — through [`oakrender::eval`].
|
||||
fn render_f32_into(
|
||||
spec: &BatchTicketSpec,
|
||||
params: &VideoTicketParams,
|
||||
time: Rational,
|
||||
size: (i32, i32),
|
||||
dst: &mut [u8],
|
||||
) -> Result<(), String> {
|
||||
let (w, h) = size;
|
||||
let stride = w * 16;
|
||||
if !params.montage.is_empty() {
|
||||
return eval::render_montage_frame_into(time, params, (w, h), dst, stride)
|
||||
.map_err(|e| format!("montage render: {e}"));
|
||||
}
|
||||
if !spec.footage_file.is_empty() {
|
||||
let decoded = eval::render_footage_frame(
|
||||
&spec.footage_file,
|
||||
spec.footage_stream,
|
||||
time,
|
||||
(w, h),
|
||||
PixelFormat::F32,
|
||||
)
|
||||
.map_err(|e| format!("footage decode: {e}"))?;
|
||||
let oakrender::texture::Texture::Cpu(frame) = &decoded else {
|
||||
return Err("decode produced a GPU texture".to_string());
|
||||
};
|
||||
let src_stride = frame.linesize_bytes() as usize;
|
||||
let row_bytes = (w as usize) * 16;
|
||||
if frame.data.len() < src_stride * (h as usize) || dst.len() < row_bytes * (h as usize) {
|
||||
return Err("decoded frame geometry mismatch".to_string());
|
||||
}
|
||||
for y in 0..h as usize {
|
||||
dst[y * row_bytes..(y + 1) * row_bytes]
|
||||
.copy_from_slice(&frame.data[y * src_stride..y * src_stride + row_bytes]);
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
// Generated frame: transparent black.
|
||||
dst[..(h as usize) * (stride as usize)].fill(0);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Convert F32 RGBA (`src`, 16 bytes/px) to 8-bit BGRA (`dst`, 4
|
||||
/// bytes/px) with clamping — the worker-side end-of-pipe convert for
|
||||
/// BGRA8 preview slots.
|
||||
fn convert_f32_rgba_to_bgra8(src: &[u8], dst: &mut [u8]) {
|
||||
let to_u8 = |v: f32| -> u8 { (v.clamp(0.0, 1.0) * 255.0 + 0.5) as u8 };
|
||||
let pixels = dst.len() / 4;
|
||||
for i in 0..pixels {
|
||||
let s = &src[i * 16..i * 16 + 16];
|
||||
let r = f32::from_le_bytes(s[0..4].try_into().unwrap());
|
||||
let g = f32::from_le_bytes(s[4..8].try_into().unwrap());
|
||||
let b = f32::from_le_bytes(s[8..12].try_into().unwrap());
|
||||
let a = f32::from_le_bytes(s[12..16].try_into().unwrap());
|
||||
let d = &mut dst[i * 4..i * 4 + 4];
|
||||
d[0] = to_u8(b);
|
||||
d[1] = to_u8(g);
|
||||
d[2] = to_u8(r);
|
||||
d[3] = to_u8(a);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -391,7 +886,9 @@ impl WorkerSession {
|
||||
pub fn worker_main(backend: &str) -> i32 {
|
||||
// 1. Session creation initializes the render backend through the
|
||||
// oakrender crate's direct Rust API
|
||||
// (oakengine_worker_session_create()).
|
||||
// (oakengine_worker_session_create()). The M15 "cpu" backend is
|
||||
// headless: no renderer, CPU evaluation + decode only.
|
||||
let cpu_mode = is_cpu_backend(backend);
|
||||
let mut session = match WorkerSession::create(backend) {
|
||||
Ok(s) => s,
|
||||
Err(msg) => {
|
||||
@@ -399,14 +896,14 @@ pub fn worker_main(backend: &str) -> i32 {
|
||||
return 1;
|
||||
}
|
||||
};
|
||||
if !session.has_renderer() {
|
||||
if !session.has_renderer() && !cpu_mode {
|
||||
// Mirrors oakengine_worker_main(): without a renderer the worker
|
||||
// cannot do anything, so it exits 1. ("--backend none" lands here.)
|
||||
log_error("no renderer initialized");
|
||||
return 1;
|
||||
}
|
||||
|
||||
// 2. Runtime services (config load etc.).
|
||||
// 2. Runtime services (config load, plugin executor install).
|
||||
if !session.initialize_runtime() {
|
||||
return 1;
|
||||
}
|
||||
@@ -443,6 +940,21 @@ pub fn worker_main(backend: &str) -> i32 {
|
||||
// Blank lines are skipped silently (read_message() semantics).
|
||||
continue;
|
||||
}
|
||||
// Protocol v2: render_batch streams its responses (one
|
||||
// batch_accepted + one frame_ready/frame_failed per ticket).
|
||||
if serde_json::from_str::<Value>(&line)
|
||||
.ok()
|
||||
.and_then(|v| v.get("type").and_then(Value::as_str).map(str::to_string))
|
||||
.as_deref()
|
||||
== Some(TYPE_RENDER_BATCH)
|
||||
{
|
||||
if let Err(e) = session.handle_render_batch_stream(&line, &mut out) {
|
||||
log_error(&format!("failed to serve render_batch: {e}"));
|
||||
exit_code = 1;
|
||||
break;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if let Some(response) = session.handle_line(&line) {
|
||||
if let Err(e) = write_message(&mut out, &response) {
|
||||
log_error(&format!("failed to write response: {e}"));
|
||||
@@ -607,8 +1119,20 @@ mod tests {
|
||||
fn handshake_attaches_real_output_pool() {
|
||||
let mut s = WorkerSession::create("none").unwrap();
|
||||
let (hs, out_region, _in) = parent_side(4, 4096, false);
|
||||
let resp = s.handle_line(&hs.to_string());
|
||||
assert!(resp.is_none(), "unexpected error: {resp:?}");
|
||||
let resp = s.handle_line(&hs.to_string()).expect("hello_caps response");
|
||||
// Protocol v2: a successful attach answers the geometry handshake
|
||||
// with the capability announcement.
|
||||
assert_eq!(resp["type"], crate::ipc::TYPE_HELLO_CAPS);
|
||||
assert_eq!(resp["protocol_version"], PROTOCOL_VERSION);
|
||||
let formats: Vec<i64> = resp["formats"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|v| v.as_i64().unwrap())
|
||||
.collect();
|
||||
assert!(formats.contains(&(PixelFormat::F32 as i64)));
|
||||
assert!(formats.contains(&(SLOT_FORMAT_BGRA8 as i64)));
|
||||
assert_eq!(resp["max_slot_bytes"], 4096);
|
||||
// The session now holds a real attached pool with the parent's
|
||||
// geometry.
|
||||
let out_pool = s.output_pool.as_ref().unwrap();
|
||||
@@ -648,8 +1172,8 @@ mod tests {
|
||||
fn handshake_attaches_input_pool_too() {
|
||||
let mut s = WorkerSession::create("none").unwrap();
|
||||
let (hs, _out, _in) = parent_side(2, 256, true);
|
||||
let resp = s.handle_line(&hs.to_string());
|
||||
assert!(resp.is_none(), "unexpected error: {resp:?}");
|
||||
let resp = s.handle_line(&hs.to_string()).expect("hello_caps response");
|
||||
assert_eq!(resp["type"], crate::ipc::TYPE_HELLO_CAPS);
|
||||
assert!(s.input_pool.is_some());
|
||||
let in_pool = s.input_pool.as_ref().unwrap();
|
||||
assert_eq!(in_pool.slot_count(), 2);
|
||||
@@ -722,7 +1246,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn load_graph_checks_are_real_then_stub() {
|
||||
fn load_graph_file_checks_then_real_deserialization() {
|
||||
let mut s = WorkerSession::create("none").unwrap();
|
||||
|
||||
let missing = "/definitely/not/a/real/graph.ove";
|
||||
@@ -747,32 +1271,150 @@ mod tests {
|
||||
);
|
||||
let _ = std::fs::remove_file(&empty);
|
||||
|
||||
// A real oaknode project round-trips through the serializer.
|
||||
let project = oaknode::project::Project::new();
|
||||
let xml = oaknode::serializer::save(&project.lock().unwrap_or_else(|e| e.into_inner()))
|
||||
.expect("serialize empty project");
|
||||
let real = std::env::temp_dir().join("oak_worker_main_test_graph.ove");
|
||||
std::fs::write(&real, b"<root/>").unwrap();
|
||||
std::fs::write(&real, &xml).unwrap();
|
||||
let resp = s.handle_line(
|
||||
&json!({ "type": "load_graph", "path": real.display().to_string() }).to_string(),
|
||||
);
|
||||
assert!(resp.is_none(), "unexpected error: {resp:?}");
|
||||
let graph = s.graph.as_ref().expect("graph loaded");
|
||||
assert!(graph.project.is_some(), "full project deserialized");
|
||||
let _ = std::fs::remove_file(&real);
|
||||
|
||||
// The minimal identity-only payload (`{"project_copy":N}`) loads as
|
||||
// a copied-project context.
|
||||
let ident = std::env::temp_dir().join("oak_worker_main_test_identity.ove");
|
||||
std::fs::write(&ident, r#"{"project_copy":7}"#).unwrap();
|
||||
let resp = s.handle_line(
|
||||
&json!({ "type": "load_graph", "path": ident.display().to_string() }).to_string(),
|
||||
);
|
||||
assert!(resp.is_none(), "unexpected error: {resp:?}");
|
||||
let graph = s.graph.as_ref().expect("graph loaded");
|
||||
assert!(graph.project.is_none());
|
||||
assert_eq!(graph.project_copy, 7);
|
||||
let _ = std::fs::remove_file(&ident);
|
||||
|
||||
// Garbage that is neither project XML nor identity JSON fails
|
||||
// explainably.
|
||||
let bad = std::env::temp_dir().join("oak_worker_main_test_bad.ove");
|
||||
std::fs::write(&bad, b"definitely not a graph").unwrap();
|
||||
let resp = s
|
||||
.handle_line(
|
||||
&json!({ "type": "load_graph", "path": real.display().to_string() }).to_string(),
|
||||
&json!({ "type": "load_graph", "path": bad.display().to_string() }).to_string(),
|
||||
)
|
||||
.unwrap();
|
||||
assert!(resp["message"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.contains("node-graph deserialization is not yet available"));
|
||||
let _ = std::fs::remove_file(&real);
|
||||
.starts_with("graph deserialization failed: "));
|
||||
let _ = std::fs::remove_file(&bad);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn render_frame_reports_stub_with_ticket() {
|
||||
fn render_frame_without_pool_reports_error_with_ticket() {
|
||||
let mut s = WorkerSession::create("none").unwrap();
|
||||
let resp = s
|
||||
.handle_line(r#"{"type":"render_frame","ticket":123,"node":"abc"}"#)
|
||||
.unwrap();
|
||||
assert_eq!(resp["type"], "error");
|
||||
assert_eq!(resp["ticket"], 123);
|
||||
assert!(resp["message"]
|
||||
.as_str()
|
||||
assert_eq!(
|
||||
resp["message"],
|
||||
"render_frame: no shared-memory pool attached"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn render_frame_v1_renders_generated_frame_into_slot() {
|
||||
let mut s = WorkerSession::create("none").unwrap();
|
||||
let (hs, out_region, _in) = parent_side(4, 4 * 4 * 16, false);
|
||||
let resp = s.handle_line(&hs.to_string()).expect("hello_caps response");
|
||||
assert_eq!(resp["type"], crate::ipc::TYPE_HELLO_CAPS);
|
||||
|
||||
// The v1 single-frame path renders the pipeline's generated frame
|
||||
// (transparent black F32; format -1 = pipeline default) into an
|
||||
// acquired slot and reports it.
|
||||
let resp = s
|
||||
.handle_line(
|
||||
r#"{"type":"render_frame","ticket":123,"time_num":1,"time_den":2,"width":4,"height":4,"format":-1}"#,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(resp["type"], "frame_ready", "unexpected: {resp}");
|
||||
assert_eq!(resp["ticket"], 123);
|
||||
let slot = resp["slot"].as_i64().unwrap() as u32;
|
||||
|
||||
// The parent (drainer) consumes the published slot and sees the
|
||||
// meta the worker wrote.
|
||||
let parent_pool = unsafe { FrameSlotPool::attach(out_region.data()) };
|
||||
let mut consumed = 0;
|
||||
assert!(unsafe { parent_pool.consume(&mut consumed) });
|
||||
assert_eq!(consumed, slot);
|
||||
let meta = unsafe { &*parent_pool.meta_const(consumed) };
|
||||
assert_eq!(meta.id, 123);
|
||||
assert_eq!(meta.width, 4);
|
||||
assert_eq!(meta.height, 4);
|
||||
assert_eq!(meta.format, PixelFormat::F32 as i32);
|
||||
assert_eq!(meta.data_size, 4 * 4 * 16);
|
||||
// Generated frame: transparent black.
|
||||
let data = unsafe { std::slice::from_raw_parts(parent_pool.slot_data_const(consumed), 4 * 4 * 16) };
|
||||
assert!(data.iter().all(|&b| b == 0));
|
||||
unsafe { parent_pool.release(consumed) };
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn render_batch_stream_renders_generated_frames_and_reports_failures() {
|
||||
let mut s = WorkerSession::create("none").unwrap();
|
||||
// Slots sized for 8x8 BGRA8.
|
||||
let (hs, out_region, _in) = parent_side(4, 8 * 8 * 4, false);
|
||||
let resp = s.handle_line(&hs.to_string()).expect("hello_caps response");
|
||||
assert_eq!(resp["type"], crate::ipc::TYPE_HELLO_CAPS);
|
||||
|
||||
let batch = json!({
|
||||
"type": "render_batch",
|
||||
"batch_id": 9,
|
||||
"tickets": [
|
||||
{ "ticket": 1, "slot": 0, "time_num": 0, "time_den": 1, "width": 8, "height": 8, "format": SLOT_FORMAT_BGRA8, "channels": 4 },
|
||||
{ "ticket": 2, "slot": 1, "time_num": 0, "time_den": 1, "width": 0, "height": 8, "format": SLOT_FORMAT_BGRA8, "channels": 4 },
|
||||
],
|
||||
});
|
||||
let mut out: Vec<u8> = Vec::new();
|
||||
s.handle_render_batch_stream(&batch.to_string(), &mut out)
|
||||
.unwrap();
|
||||
let lines: Vec<Value> = String::from_utf8(out)
|
||||
.unwrap()
|
||||
.contains("frame rendering is not yet available"));
|
||||
.lines()
|
||||
.map(|l| serde_json::from_str(l).unwrap())
|
||||
.collect();
|
||||
assert_eq!(lines.len(), 3, "accepted + one reply per ticket");
|
||||
assert_eq!(lines[0]["type"], "batch_accepted");
|
||||
assert_eq!(lines[0]["batch_id"], 9);
|
||||
assert_eq!(lines[1]["type"], "frame_ready");
|
||||
assert_eq!(lines[1]["ticket"], 1);
|
||||
assert_eq!(lines[1]["slot"], 0);
|
||||
assert_eq!(lines[2]["type"], "frame_failed");
|
||||
assert_eq!(lines[2]["ticket"], 2);
|
||||
|
||||
// The rendered slot holds opaque black BGRA8 (generated transparent
|
||||
// black F32 converted: alpha 0).
|
||||
let parent_pool = unsafe { FrameSlotPool::attach(out_region.data()) };
|
||||
let mut consumed = 0;
|
||||
assert!(unsafe { parent_pool.consume(&mut consumed) });
|
||||
assert_eq!(consumed, 0);
|
||||
let meta = unsafe { &*parent_pool.meta_const(consumed) };
|
||||
assert_eq!(meta.id, 1);
|
||||
assert_eq!(meta.format, SLOT_FORMAT_BGRA8);
|
||||
assert_eq!(meta.linesize, 8 * 4);
|
||||
assert_eq!(meta.data_size, 8 * 8 * 4);
|
||||
unsafe { parent_pool.release(consumed) };
|
||||
|
||||
// The failed ticket acquired slot 1 but never published it, so it
|
||||
// is still owned by the filler side (not in the ready ring) — the
|
||||
// ready ring is empty now.
|
||||
assert!(!unsafe { parent_pool.consume(&mut consumed) });
|
||||
}
|
||||
|
||||
// ---- oak-worker's in-process session tests (M14 R2: folded from the
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
// Oak Video Editor - Non-Linear Video Editor
|
||||
// Copyright (C) 2026 Oak Team
|
||||
//
|
||||
// This program is free software: you can redistribute it and/or modify
|
||||
// it under the terms of the GNU General Public License as published by
|
||||
// the Free Software Foundation, either version 3 of the License, or
|
||||
// (at your option) any later version.
|
||||
//
|
||||
// This program is distributed in the hope that it will be useful,
|
||||
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
// GNU General Public License for more details.
|
||||
//
|
||||
// You should have received a copy of the GNU General Public License
|
||||
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
//! M15 S1 end-to-end: real oak-worker processes driven by the
|
||||
//! main-process [`ProcessDispatcher`] — spawn, handshake, batched
|
||||
//! renders into shared-memory slots, crash isolation with restart and
|
||||
//! re-dispatch, and the zero-copy main-process guarantee.
|
||||
//!
|
||||
//! These tests spawn actual `oak-worker` child processes (located through
|
||||
//! `CARGO_BIN_EXE_oak-worker`), so they double as the binary-resolution
|
||||
//! regression gate for [`DispatcherConfig::worker_bin`].
|
||||
//!
|
||||
//! All tests in this file serialize on [`TEST_LOCK`]: they spawn worker
|
||||
//! processes that inherit the process environment, and the crash test
|
||||
//! sets crash-hook variables that must not leak into sibling runs.
|
||||
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use oakcore_rs::{PixelFormat, Rational};
|
||||
use oakrender::ipc::SLOT_FORMAT_BGRA8;
|
||||
use oakrender::procpool::{
|
||||
main_heap_frame_copies, reset_main_heap_frame_copies, DispatcherConfig, ProcessDispatcher,
|
||||
};
|
||||
use oakrender::ticket::{TicketPayload, TicketResult, VideoTicketParams};
|
||||
use oakrender::worker::{Job, JobDispatch};
|
||||
|
||||
/// Serialize every test in this file (shared process environment +
|
||||
/// real child processes).
|
||||
static TEST_LOCK: Mutex<()> = Mutex::new(());
|
||||
|
||||
fn lock_test() -> std::sync::MutexGuard<'static, ()> {
|
||||
TEST_LOCK.lock().unwrap_or_else(|e| e.into_inner())
|
||||
}
|
||||
|
||||
fn worker_bin() -> std::path::PathBuf {
|
||||
env!("CARGO_BIN_EXE_oak-worker").into()
|
||||
}
|
||||
|
||||
fn config(workers: usize, slots: u32) -> DispatcherConfig {
|
||||
DispatcherConfig {
|
||||
worker_bin: Some(worker_bin()),
|
||||
workers,
|
||||
slots_per_worker: slots,
|
||||
width: 64,
|
||||
height: 64,
|
||||
slot_format: SLOT_FORMAT_BGRA8,
|
||||
batch_size: 4,
|
||||
graph_snapshot: None,
|
||||
handshake_timeout_ms: 30_000,
|
||||
}
|
||||
}
|
||||
|
||||
fn params(time: Rational, footage: Option<(String, i32)>) -> Arc<VideoTicketParams> {
|
||||
Arc::new(VideoTicketParams {
|
||||
viewer: 1,
|
||||
time,
|
||||
force_size: Some((64, 64)),
|
||||
force_format: Some(PixelFormat::F32),
|
||||
cache: None,
|
||||
cache_dir: None,
|
||||
cache_id: None,
|
||||
cache_timebase: None,
|
||||
footage,
|
||||
montage: Vec::new(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Submit `count` generated-frame tickets; returns the shared results
|
||||
/// sink (completions land there from the dispatcher's poll pump).
|
||||
fn submit(
|
||||
dispatcher: &ProcessDispatcher,
|
||||
results: &Arc<Mutex<Vec<TicketResult>>>,
|
||||
count: usize,
|
||||
footage: Option<(String, i32)>,
|
||||
) {
|
||||
for i in 0..count {
|
||||
let results = results.clone();
|
||||
let footage = footage.clone();
|
||||
let job = Job {
|
||||
node_identity: 1,
|
||||
time: Rational::new(i as i64, 25),
|
||||
params: params(Rational::new(i as i64, 25), footage),
|
||||
// Never invoked on the process backend (workers render from
|
||||
// the wire spec); must still be a valid producer.
|
||||
produce: Arc::new(|_, _| {
|
||||
Err(oakrender::error::Error::Failed(
|
||||
"process backend does not use the in-process producer".into(),
|
||||
))
|
||||
}),
|
||||
done: Box::new(move |result| {
|
||||
results.lock().unwrap_or_else(|e| e.into_inner()).push(result);
|
||||
}),
|
||||
};
|
||||
assert!(dispatcher.post(job), "post accepted while alive");
|
||||
}
|
||||
}
|
||||
|
||||
/// Pump the dispatcher until `expect` completions arrive or the deadline
|
||||
/// passes.
|
||||
fn pump_until(dispatcher: &ProcessDispatcher, results: &Mutex<Vec<TicketResult>>, expect: usize) {
|
||||
let deadline = Instant::now() + Duration::from_secs(60);
|
||||
loop {
|
||||
dispatcher.poll();
|
||||
if results.lock().unwrap_or_else(|e| e.into_inner()).len() >= expect {
|
||||
return;
|
||||
}
|
||||
if Instant::now() > deadline {
|
||||
let have = results.lock().unwrap_or_else(|e| e.into_inner()).len();
|
||||
panic!("timeout: {have}/{expect} completions");
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(5));
|
||||
}
|
||||
}
|
||||
|
||||
/// Two real workers render two waves of generated frames into shm slots;
|
||||
/// the main process never copies frame bytes. Slots are released as
|
||||
/// frames arrive (the dispatcher's credit-based flow control then keeps
|
||||
/// the remaining tickets flowing).
|
||||
#[test]
|
||||
fn two_workers_render_two_waves_zero_copy() {
|
||||
let _guard = lock_test();
|
||||
let dispatcher = ProcessDispatcher::new(config(2, 4)).expect("dispatcher config");
|
||||
dispatcher.start().expect("workers start + handshake");
|
||||
assert_eq!(dispatcher.worker_count(), 2);
|
||||
assert!(dispatcher.is_alive(0));
|
||||
assert!(dispatcher.is_alive(1));
|
||||
|
||||
reset_main_heap_frame_copies();
|
||||
|
||||
// Wave 1: more tickets than slots in one worker, so both workers and
|
||||
// the slot-recycling path are exercised.
|
||||
let results = Arc::new(Mutex::new(Vec::new()));
|
||||
submit(&dispatcher, &results, 12, None);
|
||||
|
||||
let mut seen_worker = [false; 2];
|
||||
let mut completed = 0usize;
|
||||
let deadline = Instant::now() + Duration::from_secs(60);
|
||||
while completed < 12 {
|
||||
dispatcher.poll();
|
||||
let drained: Vec<TicketResult> =
|
||||
results.lock().unwrap_or_else(|e| e.into_inner()).drain(..).collect();
|
||||
for result in drained {
|
||||
let payload = result.expect("frame rendered");
|
||||
let TicketPayload::ShmFrame(frame) = payload else {
|
||||
panic!("process backend must deliver ShmFrame payloads");
|
||||
};
|
||||
assert!(frame.worker < 2);
|
||||
assert!(frame.slot < 4);
|
||||
assert_eq!(frame.meta.width, 64);
|
||||
assert_eq!(frame.meta.height, 64);
|
||||
assert_eq!(frame.meta.format, SLOT_FORMAT_BGRA8);
|
||||
assert_eq!(frame.meta.data_size, 64 * 64 * 4);
|
||||
assert_eq!(frame.meta.linesize, 64 * 4);
|
||||
// Generated frame (transparent black) converted to BGRA8: all
|
||||
// zero. Read through the mapping — never `slot_to_vec` (the
|
||||
// counted copy path).
|
||||
let pixels = frame.shm.slot_bytes(frame.slot);
|
||||
assert!(
|
||||
pixels[..frame.meta.data_size as usize]
|
||||
.iter()
|
||||
.all(|&b| b == 0),
|
||||
"generated frame is transparent black"
|
||||
);
|
||||
seen_worker[frame.worker as usize] = true;
|
||||
dispatcher.release_frame(&frame);
|
||||
completed += 1;
|
||||
}
|
||||
if Instant::now() > deadline {
|
||||
panic!("timeout: {completed}/12 completions");
|
||||
}
|
||||
if completed < 12 {
|
||||
std::thread::sleep(Duration::from_millis(5));
|
||||
}
|
||||
}
|
||||
// Zero copy: nothing bumped the main-process frame-copy counter.
|
||||
assert_eq!(main_heap_frame_copies(), 0);
|
||||
// Both workers participated (interleaved sharded claiming).
|
||||
assert!(seen_worker[0], "worker 0 rendered at least one frame");
|
||||
assert!(seen_worker[1], "worker 1 rendered at least one frame");
|
||||
|
||||
// Wave 2 through the same recycled slots.
|
||||
submit(&dispatcher, &results, 8, None);
|
||||
pump_until(&dispatcher, &results, 8);
|
||||
for result in results.lock().unwrap().drain(..) {
|
||||
let payload = result.expect("second wave rendered");
|
||||
let TicketPayload::ShmFrame(frame) = payload else {
|
||||
panic!("ShmFrame payload");
|
||||
};
|
||||
dispatcher.release_frame(&frame);
|
||||
}
|
||||
assert_eq!(main_heap_frame_copies(), 0);
|
||||
|
||||
dispatcher.shutdown();
|
||||
assert_eq!(main_heap_frame_copies(), 0);
|
||||
}
|
||||
|
||||
/// A worker crashing mid-render (SIGSEGV hook) must not take down the
|
||||
/// main process: the frame is re-queued, the worker restarted and the
|
||||
/// ticket still completes with a rendered frame.
|
||||
#[test]
|
||||
fn crash_isolation_restarts_worker_and_frame_still_renders() {
|
||||
let _guard = lock_test();
|
||||
|
||||
// One-shot crash hook: the worker dies with SIGSEGV while rendering
|
||||
// ticket 1; the marker file it leaves behind makes the restarted
|
||||
// worker render the re-queued frame for real.
|
||||
let marker = std::env::temp_dir().join(format!(
|
||||
"oak-procpool-crash-marker-{}",
|
||||
std::process::id()
|
||||
));
|
||||
let _ = std::fs::remove_file(&marker);
|
||||
std::env::set_var("OAK_WORKER_CRASH_ON_TICKET", "1");
|
||||
std::env::set_var("OAK_WORKER_CRASH_MARKER", &marker);
|
||||
struct EnvGuard;
|
||||
impl Drop for EnvGuard {
|
||||
fn drop(&mut self) {
|
||||
std::env::remove_var("OAK_WORKER_CRASH_ON_TICKET");
|
||||
std::env::remove_var("OAK_WORKER_CRASH_MARKER");
|
||||
}
|
||||
}
|
||||
let _env_guard = EnvGuard;
|
||||
|
||||
let dispatcher = ProcessDispatcher::new(config(1, 4)).expect("dispatcher config");
|
||||
dispatcher.start().expect("worker starts");
|
||||
assert_eq!(dispatcher.worker_count(), 1);
|
||||
|
||||
let results = Arc::new(Mutex::new(Vec::new()));
|
||||
// The dispatcher's first ticket id is 1 — exactly the crash ticket.
|
||||
submit(&dispatcher, &results, 4, None);
|
||||
pump_until(&dispatcher, &results, 4);
|
||||
|
||||
// The crash hit the worker (it restarted at least once)...
|
||||
assert!(
|
||||
dispatcher.restarts_of(0) >= 1,
|
||||
"crashed worker must be restarted (restarts={})",
|
||||
dispatcher.restarts_of(0)
|
||||
);
|
||||
// ...and every ticket still completed with a real frame.
|
||||
let mut crashed_ticket_seen = false;
|
||||
for result in results.lock().unwrap().drain(..) {
|
||||
let payload = result.expect("frame rendered despite the worker crash");
|
||||
let TicketPayload::ShmFrame(frame) = payload else {
|
||||
panic!("ShmFrame payload");
|
||||
};
|
||||
if frame.meta.id == 1 {
|
||||
crashed_ticket_seen = true;
|
||||
assert_eq!(frame.meta.width, 64);
|
||||
assert_eq!(frame.meta.data_size, 64 * 64 * 4);
|
||||
}
|
||||
dispatcher.release_frame(&frame);
|
||||
}
|
||||
assert!(crashed_ticket_seen, "ticket 1 delivered after the restart");
|
||||
assert!(marker.exists(), "the crash hook fired exactly once");
|
||||
let _ = std::fs::remove_file(&marker);
|
||||
|
||||
dispatcher.shutdown();
|
||||
}
|
||||
|
||||
/// Footage decode inside the worker process: a real H.264 frame from
|
||||
/// `tests/demo.mp4` is decoded, scaled and converted into a BGRA8 slot.
|
||||
#[test]
|
||||
fn worker_decodes_real_footage_into_slot() {
|
||||
let _guard = lock_test();
|
||||
let demo = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../tests/demo.mp4");
|
||||
assert!(demo.exists(), "repo fixture tests/demo.mp4 missing");
|
||||
|
||||
let dispatcher = ProcessDispatcher::new(config(1, 4)).expect("dispatcher config");
|
||||
dispatcher.start().expect("worker starts");
|
||||
|
||||
let results = Arc::new(Mutex::new(Vec::new()));
|
||||
submit(
|
||||
&dispatcher,
|
||||
&results,
|
||||
2,
|
||||
Some((demo.display().to_string(), 0)),
|
||||
);
|
||||
pump_until(&dispatcher, &results, 2);
|
||||
|
||||
for result in results.lock().unwrap().drain(..) {
|
||||
let payload = result.expect("footage frame rendered");
|
||||
let TicketPayload::ShmFrame(frame) = payload else {
|
||||
panic!("ShmFrame payload");
|
||||
};
|
||||
assert_eq!(frame.meta.width, 64);
|
||||
assert_eq!(frame.meta.height, 64);
|
||||
assert_eq!(frame.meta.format, SLOT_FORMAT_BGRA8);
|
||||
// Decoded video is opaque: every BGRA alpha byte is 255.
|
||||
let pixels = &frame.shm.slot_bytes(frame.slot)[..frame.meta.data_size as usize];
|
||||
let alpha_ok = pixels
|
||||
.chunks_exact(4)
|
||||
.filter(|px| px[3] == 255)
|
||||
.count();
|
||||
assert!(
|
||||
alpha_ok as f64 >= 0.99 * (64 * 64) as f64,
|
||||
"decoded frame must be opaque ({alpha_ok}/4096)"
|
||||
);
|
||||
dispatcher.release_frame(&frame);
|
||||
}
|
||||
|
||||
dispatcher.shutdown();
|
||||
}
|
||||
@@ -14,12 +14,13 @@
|
||||
// You should have received a copy of the GNU General Public License
|
||||
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
//! Binary-level tests for `oak-worker`: the process exit contract. The
|
||||
//! NDJSON control-loop behavior itself is exercised in-process in
|
||||
//! `src/session.rs` (a real loop test would require a working GPU backend,
|
||||
//! so it stays out of the unit suite).
|
||||
//! Binary-level tests for `oak-worker`: the process exit contract and the
|
||||
//! headless CPU mode's startup behavior. The NDJSON control-loop behavior
|
||||
//! itself is exercised in-process in `src/worker.rs`; end-to-end runs
|
||||
//! against the main-process dispatcher live in
|
||||
//! `tests/procpool_integration.rs`.
|
||||
|
||||
use std::process::Command;
|
||||
use std::process::{Command, Stdio};
|
||||
|
||||
fn bin() -> &'static str {
|
||||
env!("CARGO_BIN_EXE_oak-worker")
|
||||
@@ -40,3 +41,31 @@ fn backend_none_exits_one_like_the_cpp_main() {
|
||||
"stderr: {stderr}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn backend_cpu_is_headless_but_fully_operational() {
|
||||
// M15 S1: the "cpu" backend skips the renderer like "none" but the
|
||||
// session stays up — it writes the startup handshake and exits 0 on
|
||||
// EOF. (Also exercises the plugin-runtime install in the binary.)
|
||||
let mut child = Command::new(bin())
|
||||
.args(["--backend", "cpu"])
|
||||
.stdin(Stdio::piped())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::piped())
|
||||
.spawn()
|
||||
.expect("spawn oak-worker");
|
||||
// Close stdin right away: EOF ends the control loop.
|
||||
drop(child.stdin.take());
|
||||
let out = child.wait_with_output().expect("wait oak-worker");
|
||||
assert_eq!(out.status.code(), Some(0));
|
||||
let stdout = String::from_utf8_lossy(&out.stdout);
|
||||
let first = stdout.lines().next().unwrap_or("");
|
||||
assert!(
|
||||
first.contains("\"type\":\"handshake\""),
|
||||
"startup handshake on stdout, got: {stdout:?}"
|
||||
);
|
||||
assert!(
|
||||
first.contains("\"protocol_version\":1"),
|
||||
"protocol version 1, got: {first:?}"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -27,6 +27,14 @@ wgpu = "25"
|
||||
ocio-rs = { version = "0.2", features = ["bundled"] }
|
||||
# `std::error::Error` impls for the crate-internal error enum.
|
||||
thiserror = "2"
|
||||
# M15 S1: physical-memory query for the process-pool worker-count policy
|
||||
# (sysctlbyname on macOS / sysconf on Linux) in src/procpool.rs.
|
||||
libc = "0.2"
|
||||
# M15 S1: the render-worker NDJSON control protocol (src/ipc.rs moved here
|
||||
# from oak-worker so both ends of the pipe link one copy; src/procpool.rs
|
||||
# is the main-process side).
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
|
||||
[dev-dependencies]
|
||||
# oakcodec is also a plain dependency above; the dev-dependency re-entry
|
||||
|
||||
@@ -70,7 +70,16 @@ src/
|
||||
color.rs ColorProcessor over ocio-rs + default config + LUT library
|
||||
manager.rs RenderManager singleton + lifecycle + disk cache
|
||||
ticket.rs Ticket arena, params, exactly-once completion delivery
|
||||
worker.rs Worker pool + process pool (stub) + graph snapshot store
|
||||
worker.rs Worker pool + frozen pre-M15 ProcessPool facade stub +
|
||||
graph snapshot store
|
||||
scheduler.rs M15 PreviewScheduler: interleaved shard claiming,
|
||||
priority lanes (seek/playback/background), crash reclaim
|
||||
ipc.rs M15 render-worker IPC (moved from oak-worker): NDJSON
|
||||
control protocol (v1 + v2 messages) + the POSIX
|
||||
shared-memory frame-slot transport both pipe ends link
|
||||
procpool.rs M15 ProcessDispatcher: spawn/handshake oak-worker
|
||||
processes, main-assigned slot batches, crash detection +
|
||||
restart, zero-copy ShmFrameRef completions
|
||||
autocacher.rs PreviewAutoCacher
|
||||
eval.rs RenderHooks impl: the CPU evaluation seam
|
||||
backend.rs wgpu device/queue/texture management + DisplayRenderer
|
||||
@@ -86,7 +95,9 @@ tests/ contract + golden tests (common/ has shared helpers)
|
||||
1. `CHandle` only appears at the facade boundary: the crate's internal
|
||||
calls pass Rust types directly; `handle::make_owned`/`get`/`get_mut`
|
||||
are the facade entry points the oakengine stubs call.
|
||||
2. No `unsafe` outside `backend.rs` (GPU FFI) and `bridge/`.
|
||||
2. No `unsafe` outside `backend.rs` (GPU FFI), `bridge/`, and the M15
|
||||
process-isolation transport (`ipc.rs` / `procpool.rs`: POSIX shm +
|
||||
SPSC rings; every block carries its own SAFETY comment).
|
||||
3. F32 + ACEScg pipeline invariants are asserted in tests, not in
|
||||
comments (see tests/pipeline_test.rs).
|
||||
|
||||
@@ -106,9 +117,13 @@ tests/ contract + golden tests (common/ has shared helpers)
|
||||
in float. `oakrender_color_processor_create_transform` resolves the
|
||||
destination transform against the default config's reference role
|
||||
until the oakcommon color-transform bridge lands.
|
||||
- **Worker process isolation** — `ProcessPool` (oakengine_ipc worker
|
||||
binary) is a stub; `start/post` fail explainably; the crash-isolation
|
||||
tests are `#[ignore]`.
|
||||
- **Worker process isolation** — landed in M15 S1: `procpool.rs`
|
||||
(`ProcessDispatcher`) + `scheduler.rs` + `ipc.rs` drive real
|
||||
oak-worker processes (spawn, handshake, batched renders into
|
||||
main-assigned shm slots, crash restart, zero-copy completions); the
|
||||
end-to-end and crash-isolation tests live in
|
||||
`crates/oak-worker/tests/procpool_integration.rs`. The frozen pre-M15
|
||||
`worker::ProcessPool` facade stub remains for the C ABI.
|
||||
- **Audio rendering** — audio tickets complete with `Error::Failed`
|
||||
(the audio graph path is not implemented); `oakrender_ticket_get_samples`
|
||||
fails explainably.
|
||||
|
||||
@@ -314,7 +314,7 @@ mod tests {
|
||||
fn new_cacher() -> (PreviewAutoCacher, WorkerPool) {
|
||||
let mut pool = WorkerPool::new(2);
|
||||
pool.start();
|
||||
let arena = Arc::new(TicketArena::new(pool.clone(), frame_producer()));
|
||||
let arena = Arc::new(TicketArena::new(Arc::new(pool.clone()), frame_producer()));
|
||||
(PreviewAutoCacher::new(arena), pool)
|
||||
}
|
||||
|
||||
@@ -367,7 +367,7 @@ mod tests {
|
||||
f.allocate();
|
||||
Ok(crate::ticket::TicketPayload::Video(Texture::wrap_frame(f)))
|
||||
});
|
||||
let arena = Arc::new(TicketArena::new(pool.clone(), producer));
|
||||
let arena = Arc::new(TicketArena::new(Arc::new(pool.clone()), producer));
|
||||
(PreviewAutoCacher::new(arena), pool)
|
||||
}
|
||||
|
||||
@@ -465,7 +465,7 @@ mod tests {
|
||||
let mut c = {
|
||||
let mut pool = WorkerPool::new(1);
|
||||
pool.start();
|
||||
let arena = Arc::new(TicketArena::new(pool.clone(), frame_producer()));
|
||||
let arena = Arc::new(TicketArena::new(Arc::new(pool.clone()), frame_producer()));
|
||||
let c = PreviewAutoCacher::new(arena);
|
||||
pool.shutdown();
|
||||
c
|
||||
|
||||
@@ -683,28 +683,58 @@ fn render_montage_frame(
|
||||
) -> Result<Texture> {
|
||||
let mut acc = generate_frame(time, size, format)?;
|
||||
let stride = acc.linesize_bytes();
|
||||
let acc_data = &mut acc.data;
|
||||
render_montage_frame_into(time, params, size, acc_data, stride as i32)?;
|
||||
Ok(Texture::wrap_frame(acc))
|
||||
}
|
||||
|
||||
/// Composite the montage at `time` directly into `dst` (F32 RGBA rows of
|
||||
/// `dst_stride` bytes) — the M15 worker seam: the render worker passes a
|
||||
/// shared-memory slot slice as `dst`, so the composited frame lands in
|
||||
/// the slot with no staging copy. `dst` is zeroed first (transparent
|
||||
/// black base).
|
||||
pub fn render_montage_frame_into(
|
||||
time: Rational,
|
||||
params: &crate::ticket::VideoTicketParams,
|
||||
size: (i32, i32),
|
||||
dst: &mut [u8],
|
||||
dst_stride: i32,
|
||||
) -> Result<()> {
|
||||
let (w, h) = size;
|
||||
let mut acc32 = acc.data; // decode from bottom clip first
|
||||
let need = (h as usize).saturating_mul(dst_stride as usize);
|
||||
if w <= 0 || h <= 0 || dst.len() < need {
|
||||
return Err(Error::Invalid);
|
||||
}
|
||||
// Transparent-black base.
|
||||
dst[..need].fill(0);
|
||||
// Decode from the bottom clip first, composite topmost-last.
|
||||
for clip in ¶ms.montage {
|
||||
if time < clip.in_time || time >= clip.out_time {
|
||||
continue;
|
||||
}
|
||||
let media_time = clip.media_in + (time - clip.in_time);
|
||||
let decoded = render_footage_frame(&clip.filename, clip.stream_index, media_time, (w, h), format)?;
|
||||
let decoded = render_footage_frame(
|
||||
&clip.filename,
|
||||
clip.stream_index,
|
||||
media_time,
|
||||
(w, h),
|
||||
PixelFormat::F32,
|
||||
)?;
|
||||
let (src_data, src_stride) = match &decoded {
|
||||
Texture::Cpu(src) => (&src.data, src.linesize_bytes() as i32),
|
||||
_ => continue,
|
||||
};
|
||||
composite_over(&mut acc32, stride as i32, w, h, src_data, src_stride, clip.gain);
|
||||
composite_over(dst, dst_stride, w, h, src_data, src_stride, clip.gain);
|
||||
}
|
||||
acc.data = acc32;
|
||||
Ok(Texture::wrap_frame(acc))
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `src` over `dst` (premultiplied-ish alpha compositing; F32 RGBA).
|
||||
/// `gain` scales the source RGB (audio-style volume applied to video
|
||||
/// transparency is ignored here; gain scales color).
|
||||
fn composite_over(
|
||||
/// transparency is ignored here; gain scales color). Exposed for the M15
|
||||
/// render worker, which composites montage frames directly into
|
||||
/// shared-memory slots.
|
||||
pub fn composite_over(
|
||||
dst: &mut [u8],
|
||||
dst_stride: i32,
|
||||
w: i32,
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -32,6 +32,9 @@
|
||||
//! - `backend` — wgpu GPU context + display renderer
|
||||
//! - `copier` — render-side project-copy client (oaknode C ABI)
|
||||
//! - `cancelatom` — the cancellation primitive
|
||||
//! - `ipc` — render-worker NDJSON protocol + shm frame-slot transport
|
||||
//! - `scheduler` — preview frame scheduler (interleaved batch claims)
|
||||
//! - `procpool` — process-isolated render backend (M15)
|
||||
//! - `bridge` — direct-call C ABI bridges (oakcommon/oaknode/oakcodec)
|
||||
//! - `ffi` — the `include/render/*.h` export layer
|
||||
|
||||
@@ -49,7 +52,10 @@ pub mod error;
|
||||
pub mod eval;
|
||||
pub mod frame;
|
||||
pub mod handle;
|
||||
pub mod ipc;
|
||||
pub mod manager;
|
||||
pub mod procpool;
|
||||
pub mod scheduler;
|
||||
pub mod texture;
|
||||
pub mod ticket;
|
||||
pub mod worker;
|
||||
|
||||
@@ -31,8 +31,9 @@ use crate::autocacher::PreviewAutoCacher;
|
||||
use crate::backend::BackendKind;
|
||||
use crate::error::{Error, Result};
|
||||
use crate::eval;
|
||||
use crate::procpool::{DispatcherConfig, ProcessDispatcher};
|
||||
use crate::ticket::{TicketArena, TicketId};
|
||||
use crate::worker::WorkerPool;
|
||||
use crate::worker::{JobDispatch, WorkerPool};
|
||||
|
||||
static MANAGER: Mutex<Option<Arc<RenderManager>>> = Mutex::new(None);
|
||||
|
||||
@@ -40,11 +41,24 @@ fn lock<T>(m: &Mutex<T>) -> MutexGuard<'_, T> {
|
||||
m.lock().unwrap_or_else(|e| e.into_inner())
|
||||
}
|
||||
|
||||
/// The render backend the manager initializes (M15 S1: the thread pool
|
||||
/// and the process-isolated dispatcher coexist; S2 makes Processes the
|
||||
/// default and removes the pool).
|
||||
pub enum RenderBackendChoice {
|
||||
/// In-process thread pool (the default; C++ parity).
|
||||
Threads,
|
||||
/// Process-isolated oak-worker pool (crash isolation + shm frames).
|
||||
Processes(DispatcherConfig),
|
||||
}
|
||||
|
||||
/// The manager. Created by `oakrender_manager_init` (C ABI), accessed
|
||||
/// internally through [`RenderManager::global`].
|
||||
pub struct RenderManager {
|
||||
/// Worker pool.
|
||||
pub pool: WorkerPool,
|
||||
/// Video job dispatch (thread pool or process dispatcher, M15).
|
||||
pub dispatch: Arc<dyn JobDispatch>,
|
||||
/// Audio job dispatch — kept on main-process threads until S3
|
||||
/// (design §3.7: crash risk is dominated by video plugins).
|
||||
pub audio_dispatch: Arc<dyn JobDispatch>,
|
||||
/// Ticket arena.
|
||||
pub tickets: Arc<TicketArena>,
|
||||
/// Active GPU backend.
|
||||
@@ -59,23 +73,51 @@ pub struct RenderManager {
|
||||
}
|
||||
|
||||
impl RenderManager {
|
||||
/// Initialize the process-wide manager (idempotent; C++ instance()
|
||||
/// semantics — only the main GUI process does this).
|
||||
/// Initialize the process-wide manager with the default backend
|
||||
/// (in-process threads; idempotent; C++ instance() semantics — only
|
||||
/// the main GUI process does this).
|
||||
pub fn init() -> Result<()> {
|
||||
Self::init_with_backend(RenderBackendChoice::Threads)
|
||||
}
|
||||
|
||||
/// Initialize the process-wide manager with an explicit backend
|
||||
/// (M15 S1: `Threads` keeps the C++ parity path, `Processes` spawns
|
||||
/// the oak-worker pool).
|
||||
pub fn init_with_backend(choice: RenderBackendChoice) -> Result<()> {
|
||||
let mut guard = lock(&MANAGER);
|
||||
if guard.is_some() {
|
||||
return Err(Error::State);
|
||||
}
|
||||
let backend = BackendKind::from_user_config();
|
||||
let mut pool = WorkerPool::new(0);
|
||||
pool.start();
|
||||
let producer: crate::ticket::Producer = Arc::new(|time, params| {
|
||||
eval::render_produced_frame(time, params)
|
||||
.map(crate::ticket::TicketPayload::Video)
|
||||
});
|
||||
let tickets = Arc::new(TicketArena::new(pool.clone(), producer));
|
||||
let (dispatch, audio_dispatch): (Arc<dyn JobDispatch>, Arc<dyn JobDispatch>) =
|
||||
match choice {
|
||||
RenderBackendChoice::Threads => {
|
||||
let mut pool = WorkerPool::new(0);
|
||||
pool.start();
|
||||
let pool = Arc::new(pool);
|
||||
(pool.clone(), pool)
|
||||
}
|
||||
RenderBackendChoice::Processes(config) => {
|
||||
let dispatcher = ProcessDispatcher::new(config)?;
|
||||
dispatcher.start()?;
|
||||
// Audio stays on main-process threads (design §3.7).
|
||||
let mut audio = WorkerPool::new(2);
|
||||
audio.start();
|
||||
(dispatcher, Arc::new(audio))
|
||||
}
|
||||
};
|
||||
let tickets = Arc::new(TicketArena::new_with_audio(
|
||||
dispatch.clone(),
|
||||
audio_dispatch.clone(),
|
||||
producer,
|
||||
));
|
||||
*guard = Some(Arc::new(RenderManager {
|
||||
pool,
|
||||
dispatch,
|
||||
audio_dispatch,
|
||||
tickets,
|
||||
backend,
|
||||
requested_backend: backend,
|
||||
@@ -99,15 +141,16 @@ impl RenderManager {
|
||||
guard
|
||||
}
|
||||
|
||||
/// Shut down: cancel tickets, drain pool, release backend.
|
||||
/// Shut down: cancel tickets, drain both dispatch backends.
|
||||
pub fn shutdown() {
|
||||
let manager = lock(&MANAGER).take();
|
||||
if let Some(manager) = manager {
|
||||
manager.tickets.cancel_all();
|
||||
// Drop the manager (releases the pool clone) after the pool is
|
||||
// drained; the drain delivers queued completions.
|
||||
let mut pool = manager.pool.clone();
|
||||
pool.shutdown();
|
||||
// Drain after the cancels so queued completions fire. Both
|
||||
// dispatches are idempotent (the Threads backend shares one
|
||||
// Arc for video + audio).
|
||||
manager.dispatch.shutdown();
|
||||
manager.audio_dispatch.shutdown();
|
||||
drop(manager);
|
||||
}
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,568 @@
|
||||
// Oak Video Editor - Non-Linear Video Editor
|
||||
// Copyright (C) 2026 Oak Team
|
||||
//
|
||||
// This program is free software: you can redistribute it and/or modify
|
||||
// it under the terms of the GNU General Public License as published by
|
||||
// the Free Software Foundation, either version 3 of the License, or
|
||||
// (at your option) any later version.
|
||||
//
|
||||
// This program is distributed in the hope that it will be useful,
|
||||
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
// GNU General Public License for more details.
|
||||
//
|
||||
// You should have received a copy of the GNU General Public License
|
||||
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
//! The preview scheduler (M15 S1; design doc §3.2): frame-request
|
||||
//! ordering, interleaved batch claims, crash re-dispatch and flow
|
||||
//! control, as pure single-threaded logic.
|
||||
//!
|
||||
//! The scheduler knows nothing about processes or shared memory — it
|
||||
//! turns a stream of [`FrameRequest`]s into [`ClaimedBatch`]es for the
|
||||
//! [`crate::procpool::ProcessDispatcher`] to hand to workers:
|
||||
//!
|
||||
//! - **Interleaved batch claims.** With `W` workers, the pending frame
|
||||
//! stream is sharded round-robin: worker `i` claims frames whose
|
||||
//! frame number is `≡ i (mod W)`, in batches of `B ≈ 120 / W`
|
||||
//! (configurable). Adjacent frame numbers therefore land on
|
||||
//! different workers and finish at nearly the same time; every
|
||||
//! frame belongs to exactly one worker (no work stealing).
|
||||
//! - **Priorities.** Seek/current frame > playback window (nearer the
|
||||
//! playhead first) > background (export/thumbnails). Within one
|
||||
//! priority class batches keep ascending frame order.
|
||||
//! - **Crash recovery.** A crashed worker's claimed frames (its whole
|
||||
//! un-started batches plus the un-finished frames of started ones)
|
||||
//! are re-queued and may be claimed by ANY healthy worker — crash
|
||||
//! re-dispatch is failure recovery, not stealing.
|
||||
//! - **Flow control.** [`PreviewScheduler::claim_batch`] never claims
|
||||
//! more frames than the caller's `credit` (the worker's free shm
|
||||
//! slot count): slots are the credit.
|
||||
//! - **Cancellation.** Frame keys carry a parameter `version`;
|
||||
//! submitting a newer version of a key invalidates the older one,
|
||||
//! and [`PreviewScheduler::cancel_sequence`] drops a whole sequence.
|
||||
//!
|
||||
//! All methods are non-blocking; the dispatcher drives them from its
|
||||
//! poll loop (UI tick).
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// A frame request key: `(sequence, frame number, parameter version)`.
|
||||
/// The version covers graph/proxy/resolution/color-parameter changes —
|
||||
/// bumping it invalidates outstanding requests for the same frame.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
|
||||
pub struct FrameKey {
|
||||
/// Sequence identity.
|
||||
pub sequence: u64,
|
||||
/// Frame number within the sequence.
|
||||
pub frame: i64,
|
||||
/// Parameter version (graph version / proxy tier / resolution tier /
|
||||
/// color).
|
||||
pub version: u64,
|
||||
}
|
||||
|
||||
/// Frame request priority class (lower value = more urgent).
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
|
||||
pub enum FramePriority {
|
||||
/// Seek / current playhead frame (single-frame insert, top priority).
|
||||
Seek,
|
||||
/// Playback window frames (ordered by [`FrameRequest::distance`]).
|
||||
Playback,
|
||||
/// Background work (export, thumbnails, full-res stills).
|
||||
Background,
|
||||
}
|
||||
|
||||
/// One pending frame request. `payload` is opaque to the scheduler (the
|
||||
/// dispatcher uses it to find the ticket's params + completion).
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct FrameRequest<P> {
|
||||
/// The request key.
|
||||
pub key: FrameKey,
|
||||
/// Priority class.
|
||||
pub priority: FramePriority,
|
||||
/// Distance from the playhead in frames (orders the Playback class;
|
||||
/// unused by Seek/Background).
|
||||
pub distance: i64,
|
||||
/// Caller payload.
|
||||
pub payload: P,
|
||||
}
|
||||
|
||||
/// A batch of frames one worker claimed.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct ClaimedBatch<P> {
|
||||
/// Batch identity (unique per scheduler).
|
||||
pub batch_id: u64,
|
||||
/// The claiming worker index.
|
||||
pub worker: usize,
|
||||
/// The claimed frames in render order (priority class first, then
|
||||
/// ascending frame number).
|
||||
pub frames: Vec<FrameRequest<P>>,
|
||||
}
|
||||
|
||||
struct Claim<P> {
|
||||
worker: usize,
|
||||
batch_id: u64,
|
||||
request: FrameRequest<P>,
|
||||
}
|
||||
|
||||
struct PendingEntry<P> {
|
||||
request: FrameRequest<P>,
|
||||
/// True when any worker may claim this frame (crash re-dispatch);
|
||||
/// false when the interleaved shard rule applies.
|
||||
any_worker: bool,
|
||||
}
|
||||
|
||||
/// The scheduler state machine (single-threaded by contract).
|
||||
pub struct PreviewScheduler<P> {
|
||||
workers: usize,
|
||||
batch_size: usize,
|
||||
pending: Vec<PendingEntry<P>>,
|
||||
claimed: HashMap<FrameKey, Claim<P>>,
|
||||
next_batch_id: u64,
|
||||
/// Total frames re-queued by worker crashes (tests/metrics).
|
||||
crash_requeued: u64,
|
||||
}
|
||||
|
||||
impl<P: Clone> PreviewScheduler<P> {
|
||||
/// Scheduler for `workers` workers (at least 1). `batch_size` 0 picks
|
||||
/// the design default `max(1, 120 / workers)`.
|
||||
pub fn new(workers: usize, batch_size: usize) -> Self {
|
||||
let workers = workers.max(1);
|
||||
let batch_size = if batch_size == 0 {
|
||||
(120 / workers).max(1)
|
||||
} else {
|
||||
batch_size
|
||||
};
|
||||
Self {
|
||||
workers,
|
||||
batch_size,
|
||||
pending: Vec::new(),
|
||||
claimed: HashMap::new(),
|
||||
next_batch_id: 1,
|
||||
crash_requeued: 0,
|
||||
}
|
||||
}
|
||||
|
||||
/// The configured worker count.
|
||||
pub fn workers(&self) -> usize {
|
||||
self.workers
|
||||
}
|
||||
|
||||
/// The configured batch size.
|
||||
pub fn batch_size(&self) -> usize {
|
||||
self.batch_size
|
||||
}
|
||||
|
||||
/// Submit a frame request. An already-pending request with the same
|
||||
/// key is replaced; a key already claimed (in flight) is rejected —
|
||||
/// the dispatcher must cancel/re-version it first. Returns true when
|
||||
/// the request was accepted.
|
||||
pub fn submit(&mut self, request: FrameRequest<P>) -> bool {
|
||||
if self.claimed.contains_key(&request.key) {
|
||||
return false;
|
||||
}
|
||||
if let Some(entry) = self.pending.iter_mut().find(|e| e.request.key == request.key) {
|
||||
entry.request = request;
|
||||
return true;
|
||||
}
|
||||
self.pending.push(PendingEntry {
|
||||
request,
|
||||
any_worker: false,
|
||||
});
|
||||
true
|
||||
}
|
||||
|
||||
/// Claim the next batch for `worker`: the worker's interleaved shard
|
||||
/// (frame number `≡ worker (mod W)`, plus any crash-requeued frames),
|
||||
/// ordered by priority class / playhead distance / ascending frame,
|
||||
/// capped at `min(batch_size, credit)`. Returns `None` when nothing
|
||||
/// is claimable (`credit == 0`, unknown worker, empty shard).
|
||||
///
|
||||
/// Claimed frames never go to another worker while in flight (no
|
||||
/// stealing).
|
||||
pub fn claim_batch(&mut self, worker: usize, credit: usize) -> Option<ClaimedBatch<P>> {
|
||||
if worker >= self.workers || credit == 0 {
|
||||
return None;
|
||||
}
|
||||
let workers = self.workers;
|
||||
let mut indexes: Vec<usize> = self
|
||||
.pending
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter(|(_, e)| {
|
||||
e.any_worker || e.request.key.frame.rem_euclid(workers as i64) == worker as i64
|
||||
})
|
||||
.map(|(i, _)| i)
|
||||
.collect();
|
||||
if indexes.is_empty() {
|
||||
return None;
|
||||
}
|
||||
indexes.sort_by(|&a, &b| {
|
||||
let ra = &self.pending[a].request;
|
||||
let rb = &self.pending[b].request;
|
||||
ra.priority
|
||||
.cmp(&rb.priority)
|
||||
.then(ra.distance.cmp(&rb.distance))
|
||||
.then(ra.key.frame.cmp(&rb.key.frame))
|
||||
.then(ra.key.sequence.cmp(&rb.key.sequence))
|
||||
});
|
||||
indexes.truncate(self.batch_size.min(credit));
|
||||
|
||||
let batch_id = self.next_batch_id;
|
||||
self.next_batch_id += 1;
|
||||
|
||||
// Collect claimed frames (removal order does not matter; the batch
|
||||
// keeps the sorted order).
|
||||
let mut frames: Vec<FrameRequest<P>> = Vec::with_capacity(indexes.len());
|
||||
let mut marked: Vec<bool> = vec![false; self.pending.len()];
|
||||
for &i in &indexes {
|
||||
marked[i] = true;
|
||||
}
|
||||
let mut kept: Vec<PendingEntry<P>> = Vec::with_capacity(self.pending.len() - indexes.len());
|
||||
for (i, entry) in self.pending.drain(..).enumerate() {
|
||||
if marked[i] {
|
||||
self.claimed.insert(
|
||||
entry.request.key,
|
||||
Claim {
|
||||
worker,
|
||||
batch_id,
|
||||
request: entry.request.clone(),
|
||||
},
|
||||
);
|
||||
frames.push(entry.request);
|
||||
} else {
|
||||
kept.push(entry);
|
||||
}
|
||||
}
|
||||
self.pending = kept;
|
||||
frames.sort_by(|a, b| {
|
||||
a.priority
|
||||
.cmp(&b.priority)
|
||||
.then(a.distance.cmp(&b.distance))
|
||||
.then(a.key.frame.cmp(&b.key.frame))
|
||||
});
|
||||
Some(ClaimedBatch {
|
||||
batch_id,
|
||||
worker,
|
||||
frames,
|
||||
})
|
||||
}
|
||||
|
||||
/// Report a claimed frame as rendered (frame_ready). Returns the
|
||||
/// request when the key was in flight.
|
||||
pub fn frame_done(&mut self, key: &FrameKey) -> Option<FrameRequest<P>> {
|
||||
self.claimed.remove(key).map(|c| c.request)
|
||||
}
|
||||
|
||||
/// Report a claimed frame as permanently failed (frame_failed; the
|
||||
/// main process paints the fallback). The claim is dropped WITHOUT
|
||||
/// re-dispatch — a render error is not a crash. Returns the request
|
||||
/// when the key was in flight.
|
||||
pub fn frame_failed(&mut self, key: &FrameKey) -> Option<FrameRequest<P>> {
|
||||
self.claimed.remove(key).map(|c| c.request)
|
||||
}
|
||||
|
||||
/// Re-queue every frame claimed by `worker` (crash recovery): its
|
||||
/// un-started batches and the un-finished frames of started batches
|
||||
/// all come back as pending, claimable by ANY healthy worker.
|
||||
/// Returns the re-queued requests.
|
||||
pub fn worker_crashed(&mut self, worker: usize) -> Vec<FrameRequest<P>> {
|
||||
let mut reclaimed = Vec::new();
|
||||
self.claimed.retain(|_, claim| {
|
||||
if claim.worker == worker {
|
||||
reclaimed.push(claim.request.clone());
|
||||
false
|
||||
} else {
|
||||
true
|
||||
}
|
||||
});
|
||||
for request in reclaimed.iter().cloned() {
|
||||
self.pending.push(PendingEntry {
|
||||
request,
|
||||
any_worker: true,
|
||||
});
|
||||
}
|
||||
self.crash_requeued += reclaimed.len() as u64;
|
||||
reclaimed
|
||||
}
|
||||
|
||||
/// Cancel one key, pending OR claimed (single-frame cancellation;
|
||||
/// the dispatcher delivers the ticket's `Error::State` itself).
|
||||
/// Returns true when the key was known.
|
||||
pub fn cancel_key(&mut self, key: &FrameKey) -> bool {
|
||||
let before = self.pending.len();
|
||||
self.pending.retain(|e| &e.request.key != key);
|
||||
if self.pending.len() != before {
|
||||
return true;
|
||||
}
|
||||
self.claimed.remove(key).is_some()
|
||||
}
|
||||
|
||||
/// Cancel every pending AND claimed frame of `sequence` (frame-key
|
||||
/// invalidation; the `cancel` wire message covers the worker side).
|
||||
/// Returns the number of dropped requests.
|
||||
pub fn cancel_sequence(&mut self, sequence: u64) -> usize {
|
||||
let before_pending = self.pending.len();
|
||||
self.pending.retain(|e| e.request.key.sequence != sequence);
|
||||
let dropped_pending = before_pending - self.pending.len();
|
||||
let before_claimed = self.claimed.len();
|
||||
self.claimed.retain(|_, c| c.request.key.sequence != sequence);
|
||||
dropped_pending + (before_claimed - self.claimed.len())
|
||||
}
|
||||
|
||||
/// Pending (unclaimed) request count.
|
||||
pub fn pending_count(&self) -> usize {
|
||||
self.pending.len()
|
||||
}
|
||||
|
||||
/// Claimed (in-flight) request count.
|
||||
pub fn claimed_count(&self) -> usize {
|
||||
self.claimed.len()
|
||||
}
|
||||
|
||||
/// The worker currently holding `key` (None when not claimed).
|
||||
pub fn claimed_worker(&self, key: &FrameKey) -> Option<usize> {
|
||||
self.claimed.get(key).map(|c| c.worker)
|
||||
}
|
||||
|
||||
/// Total frames re-queued by crashes so far (tests/metrics).
|
||||
pub fn crash_requeued(&self) -> u64 {
|
||||
self.crash_requeued
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn req(seq: u64, frame: i64, prio: FramePriority) -> FrameRequest<u64> {
|
||||
FrameRequest {
|
||||
key: FrameKey {
|
||||
sequence: seq,
|
||||
frame,
|
||||
version: 0,
|
||||
},
|
||||
priority: prio,
|
||||
distance: frame,
|
||||
payload: frame as u64,
|
||||
}
|
||||
}
|
||||
|
||||
/// Claim until nothing is claimable by any worker; returns
|
||||
/// (worker, frame) pairs in claim order.
|
||||
fn claim_all(s: &mut PreviewScheduler<u64>) -> Vec<(usize, i64)> {
|
||||
let mut out = Vec::new();
|
||||
loop {
|
||||
let mut progress = false;
|
||||
for w in 0..s.workers() {
|
||||
while let Some(batch) = s.claim_batch(w, 1024) {
|
||||
for f in &batch.frames {
|
||||
out.push((w, f.key.frame));
|
||||
}
|
||||
progress = true;
|
||||
}
|
||||
}
|
||||
if !progress {
|
||||
break;
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_stealing_every_frame_claimed_exactly_once() {
|
||||
let mut s: PreviewScheduler<u64> = PreviewScheduler::new(3, 4);
|
||||
for f in 0..40 {
|
||||
assert!(s.submit(req(1, f, FramePriority::Playback)));
|
||||
}
|
||||
let claims = claim_all(&mut s);
|
||||
assert_eq!(claims.len(), 40, "every frame claimed");
|
||||
let mut frames: Vec<i64> = claims.iter().map(|(_, f)| *f).collect();
|
||||
frames.sort_unstable();
|
||||
frames.dedup();
|
||||
assert_eq!(frames.len(), 40, "no frame claimed twice (no stealing)");
|
||||
assert_eq!(s.pending_count(), 0);
|
||||
assert_eq!(s.claimed_count(), 40, "all claimed, none completed yet");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn interleave_adjacent_frames_on_different_workers() {
|
||||
let mut s: PreviewScheduler<u64> = PreviewScheduler::new(4, 2);
|
||||
for f in 0..16 {
|
||||
s.submit(req(1, f, FramePriority::Playback));
|
||||
}
|
||||
let claims = claim_all(&mut s);
|
||||
for (worker, frame) in &claims {
|
||||
assert_eq!(
|
||||
frame.rem_euclid(4),
|
||||
*worker as i64,
|
||||
"frame {frame} must be claimed by worker {}",
|
||||
frame.rem_euclid(4)
|
||||
);
|
||||
}
|
||||
// Adjacent frames are on different workers.
|
||||
let worker_of: HashMap<i64, usize> = claims
|
||||
.iter()
|
||||
.map(|(w, f)| (*f, *w))
|
||||
.collect();
|
||||
for f in 0..15 {
|
||||
assert_ne!(worker_of[&f], worker_of[&(f + 1)]);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn crash_requeues_to_any_healthy_worker() {
|
||||
let mut s: PreviewScheduler<u64> = PreviewScheduler::new(2, 8);
|
||||
for f in 0..8 {
|
||||
s.submit(req(1, f, FramePriority::Playback));
|
||||
}
|
||||
// Both workers claim their shards first.
|
||||
let batch0 = s.claim_batch(0, 8).unwrap();
|
||||
assert_eq!(batch0.frames.len(), 4); // frames 0,2,4,6
|
||||
let batch1 = s.claim_batch(1, 8).unwrap();
|
||||
assert_eq!(batch1.frames.len(), 4); // frames 1,3,5,7
|
||||
|
||||
// Worker 0 crashes: its frames come back...
|
||||
let reclaimed = s.worker_crashed(0);
|
||||
assert_eq!(reclaimed.len(), 4);
|
||||
assert_eq!(s.claimed_count(), 4, "worker 1 keeps its own batch");
|
||||
assert_eq!(s.pending_count(), 4);
|
||||
assert_eq!(s.crash_requeued(), 4);
|
||||
|
||||
// ...and worker 1 (NOT their shard) can claim them all.
|
||||
let batch = s.claim_batch(1, 8).unwrap();
|
||||
assert_eq!(batch.frames.len(), 4);
|
||||
let mut frames: Vec<i64> = batch.frames.iter().map(|f| f.key.frame).collect();
|
||||
frames.sort_unstable();
|
||||
assert_eq!(frames, vec![0, 2, 4, 6]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn priority_seek_beats_playback_beats_background() {
|
||||
let mut s: PreviewScheduler<u64> = PreviewScheduler::new(1, 100);
|
||||
// All frames on worker 0's shard (W=1): background first, then a
|
||||
// playback window, then a seek frame submitted last.
|
||||
for f in 0..5 {
|
||||
s.submit(req(1, f, FramePriority::Background));
|
||||
}
|
||||
for f in 10..15 {
|
||||
let mut r = req(1, f, FramePriority::Playback);
|
||||
r.distance = (f - 12).abs();
|
||||
s.submit(r);
|
||||
}
|
||||
s.submit(req(1, 100, FramePriority::Seek));
|
||||
|
||||
let batch = s.claim_batch(0, 100).unwrap();
|
||||
let order: Vec<(FramePriority, i64)> = batch
|
||||
.frames
|
||||
.iter()
|
||||
.map(|f| (f.priority, f.key.frame))
|
||||
.collect();
|
||||
// Seek first...
|
||||
assert_eq!(order[0], (FramePriority::Seek, 100));
|
||||
// ...then playback by playhead distance (12 nearest first)...
|
||||
assert_eq!(
|
||||
order[1..6]
|
||||
.iter()
|
||||
.map(|(_, f)| *f)
|
||||
.collect::<Vec<_>>(),
|
||||
vec![12, 11, 13, 10, 14]
|
||||
);
|
||||
// ...then background ascending.
|
||||
assert_eq!(
|
||||
order[6..]
|
||||
.iter()
|
||||
.map(|(_, f)| *f)
|
||||
.collect::<Vec<_>>(),
|
||||
vec![0, 1, 2, 3, 4]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flow_control_credit_limits_batch() {
|
||||
let mut s: PreviewScheduler<u64> = PreviewScheduler::new(1, 100);
|
||||
for f in 0..10 {
|
||||
s.submit(req(1, f, FramePriority::Playback));
|
||||
}
|
||||
// Zero credit claims nothing.
|
||||
assert!(s.claim_batch(0, 0).is_none());
|
||||
// Credit 3 claims exactly 3 (free slots are the credit).
|
||||
let batch = s.claim_batch(0, 3).unwrap();
|
||||
assert_eq!(batch.frames.len(), 3);
|
||||
assert_eq!(s.pending_count(), 7);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn batch_size_caps_the_claim() {
|
||||
let mut s: PreviewScheduler<u64> = PreviewScheduler::new(1, 4);
|
||||
for f in 0..10 {
|
||||
s.submit(req(1, f, FramePriority::Playback));
|
||||
}
|
||||
let batch = s.claim_batch(0, 100).unwrap();
|
||||
assert_eq!(batch.frames.len(), 4, "batch size B caps the claim");
|
||||
// Ascending frame order inside the batch.
|
||||
let frames: Vec<i64> = batch.frames.iter().map(|f| f.key.frame).collect();
|
||||
assert_eq!(frames, vec![0, 1, 2, 3]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn done_and_failed_drop_the_claim() {
|
||||
let mut s: PreviewScheduler<u64> = PreviewScheduler::new(1, 4);
|
||||
s.submit(req(1, 0, FramePriority::Playback));
|
||||
s.submit(req(1, 1, FramePriority::Playback));
|
||||
let batch = s.claim_batch(0, 4).unwrap();
|
||||
assert_eq!(batch.frames.len(), 2);
|
||||
let k0 = batch.frames[0].key;
|
||||
let k1 = batch.frames[1].key;
|
||||
assert!(s.frame_done(&k0).is_some());
|
||||
assert!(s.frame_failed(&k1).is_some());
|
||||
assert_eq!(s.claimed_count(), 0);
|
||||
assert_eq!(s.pending_count(), 0, "frame_failed is terminal (purple frame fallback)");
|
||||
// Unknown keys are no-ops.
|
||||
assert!(s.frame_done(&k0).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resubmit_of_claimed_key_is_rejected_until_done() {
|
||||
let mut s: PreviewScheduler<u64> = PreviewScheduler::new(1, 4);
|
||||
let r = req(1, 5, FramePriority::Playback);
|
||||
let key = r.key;
|
||||
s.submit(r);
|
||||
let _ = s.claim_batch(0, 4).unwrap();
|
||||
// In flight: rejected.
|
||||
assert!(!s.submit(req(1, 5, FramePriority::Seek)));
|
||||
s.frame_done(&key);
|
||||
// After completion the same key may be requested again (new
|
||||
// version in practice).
|
||||
assert!(s.submit(req(1, 5, FramePriority::Seek)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cancel_sequence_drops_pending_and_claimed() {
|
||||
let mut s: PreviewScheduler<u64> = PreviewScheduler::new(1, 4);
|
||||
for f in 0..6 {
|
||||
s.submit(req(7, f, FramePriority::Playback));
|
||||
}
|
||||
let _ = s.claim_batch(0, 4).unwrap(); // claims 4 of sequence 7
|
||||
s.submit(req(8, 0, FramePriority::Playback)); // other sequence
|
||||
assert_eq!(s.cancel_sequence(7), 6);
|
||||
assert_eq!(s.pending_count(), 1, "sequence 8 untouched");
|
||||
assert_eq!(s.claimed_count(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn default_batch_size_is_120_over_workers() {
|
||||
let s: PreviewScheduler<u64> = PreviewScheduler::new(4, 0);
|
||||
assert_eq!(s.batch_size(), 30);
|
||||
let s: PreviewScheduler<u64> = PreviewScheduler::new(0, 0);
|
||||
assert_eq!(s.workers(), 1);
|
||||
assert_eq!(s.batch_size(), 120);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_worker_claims_nothing() {
|
||||
let mut s: PreviewScheduler<u64> = PreviewScheduler::new(2, 4);
|
||||
s.submit(req(1, 0, FramePriority::Playback));
|
||||
assert!(s.claim_batch(2, 4).is_none());
|
||||
}
|
||||
}
|
||||
@@ -33,7 +33,7 @@ use oakcore_rs::{Rational, TimeRange};
|
||||
use crate::error::{Error, Result};
|
||||
use crate::eval;
|
||||
use crate::texture::Texture;
|
||||
use crate::worker::WorkerPool;
|
||||
use crate::worker::JobDispatch;
|
||||
|
||||
/// One clip of a sequence montage (M12 P0): the facade resolves the
|
||||
/// timeline into an ordered list of clips; the producer decodes each and
|
||||
@@ -128,6 +128,10 @@ pub enum TicketPayload {
|
||||
Video(Texture),
|
||||
/// Rendered interleaved audio.
|
||||
Audio(AudioSamples),
|
||||
/// A rendered frame living in a worker's shared-memory slot (M15
|
||||
/// process backend): zero copy — the consumer reads the pixels from
|
||||
/// the mapping and releases the slot through the dispatcher.
|
||||
ShmFrame(crate::procpool::ShmFrameRef),
|
||||
}
|
||||
|
||||
/// Completion payload: the rendered texture/samples or the failure
|
||||
@@ -239,18 +243,31 @@ fn lock<T>(m: &Mutex<T>) -> MutexGuard<'_, T> {
|
||||
/// The ticket arena (owned by the manager).
|
||||
pub struct TicketArena {
|
||||
next: AtomicU64,
|
||||
pool: WorkerPool,
|
||||
dispatch: Arc<dyn JobDispatch>,
|
||||
audio_dispatch: Arc<dyn JobDispatch>,
|
||||
slots: Mutex<HashMap<TicketId, Arc<TicketSlot>>>,
|
||||
shutting_down: AtomicBool,
|
||||
producer: Producer,
|
||||
}
|
||||
|
||||
impl TicketArena {
|
||||
/// Arena dispatching through `pool`; `producer` renders frames.
|
||||
pub fn new(pool: WorkerPool, producer: Producer) -> Self {
|
||||
/// Arena dispatching video and audio through the same backend.
|
||||
pub fn new(dispatch: Arc<dyn JobDispatch>, producer: Producer) -> Self {
|
||||
Self::new_with_audio(dispatch.clone(), dispatch, producer)
|
||||
}
|
||||
|
||||
/// Arena with separate video/audio backends (M15: video may run on
|
||||
/// the process dispatcher while audio stays on the main-process
|
||||
/// thread dispatch until S3 — design §3.7).
|
||||
pub fn new_with_audio(
|
||||
video: Arc<dyn JobDispatch>,
|
||||
audio: Arc<dyn JobDispatch>,
|
||||
producer: Producer,
|
||||
) -> Self {
|
||||
Self {
|
||||
next: AtomicU64::new(1),
|
||||
pool,
|
||||
dispatch: video,
|
||||
audio_dispatch: audio,
|
||||
slots: Mutex::new(HashMap::new()),
|
||||
shutting_down: AtomicBool::new(false),
|
||||
producer,
|
||||
@@ -318,8 +335,8 @@ impl TicketArena {
|
||||
produce: producer,
|
||||
done: Box::new(move |result| slot_done.finish(result)),
|
||||
};
|
||||
if !self.pool.post(job) {
|
||||
// Pool is gone (shutdown raced the submit): deliver now.
|
||||
if !self.dispatch.post(job) {
|
||||
// Backend is gone (shutdown raced the submit): deliver now.
|
||||
slot.finish(Err(Error::State));
|
||||
}
|
||||
id
|
||||
@@ -387,7 +404,7 @@ impl TicketArena {
|
||||
produce: producer,
|
||||
done: Box::new(move |result| slot_done.finish(result)),
|
||||
};
|
||||
if !self.pool.post(job) {
|
||||
if !self.audio_dispatch.post(job) {
|
||||
slot.finish(Err(Error::State));
|
||||
}
|
||||
id
|
||||
@@ -477,6 +494,7 @@ mod tests {
|
||||
|
||||
use crate::frame::VideoParamsPod;
|
||||
use crate::texture::Frame;
|
||||
use crate::worker::WorkerPool;
|
||||
|
||||
fn small_frame() -> Frame {
|
||||
let mut f = Frame::new();
|
||||
@@ -497,7 +515,7 @@ mod tests {
|
||||
let pool = WorkerPool::new(2);
|
||||
let mut pool = pool;
|
||||
pool.start();
|
||||
let arena = TicketArena::new(pool.clone(), ok_producer());
|
||||
let arena = TicketArena::new(Arc::new(pool.clone()), ok_producer());
|
||||
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let id = arena.submit_video(
|
||||
@@ -552,7 +570,7 @@ mod tests {
|
||||
}
|
||||
Ok(TicketPayload::Video(Texture::wrap_frame(small_frame())))
|
||||
});
|
||||
let arena = TicketArena::new(pool.clone(), producer);
|
||||
let arena = TicketArena::new(Arc::new(pool.clone()), producer);
|
||||
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let id = arena.submit_video(
|
||||
@@ -592,7 +610,7 @@ mod tests {
|
||||
let pool = WorkerPool::new(1);
|
||||
let mut pool = pool;
|
||||
pool.start();
|
||||
let arena = TicketArena::new(pool.clone(), ok_producer());
|
||||
let arena = TicketArena::new(Arc::new(pool.clone()), ok_producer());
|
||||
arena.cancel(TicketId(12345));
|
||||
assert!(!arena.is_finished(TicketId(12345)));
|
||||
pool.shutdown();
|
||||
@@ -603,7 +621,7 @@ mod tests {
|
||||
let pool = WorkerPool::new(1);
|
||||
let mut pool = pool;
|
||||
pool.start();
|
||||
let arena = TicketArena::new(pool.clone(), ok_producer());
|
||||
let arena = TicketArena::new(Arc::new(pool.clone()), ok_producer());
|
||||
assert_eq!(
|
||||
arena.wait(TicketId(999)).unwrap_err().code(),
|
||||
Error::NotFound.code()
|
||||
@@ -616,7 +634,7 @@ mod tests {
|
||||
let pool = WorkerPool::new(1);
|
||||
let mut pool = pool;
|
||||
pool.start();
|
||||
let arena = TicketArena::new(pool.clone(), ok_producer());
|
||||
let arena = TicketArena::new(Arc::new(pool.clone()), ok_producer());
|
||||
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let range = TimeRange::new(Rational::new(0, 1), Rational::new(10, 1));
|
||||
@@ -649,7 +667,7 @@ mod tests {
|
||||
let pool = WorkerPool::new(1);
|
||||
let mut pool = pool;
|
||||
pool.start();
|
||||
let arena = TicketArena::new(pool.clone(), ok_producer());
|
||||
let arena = TicketArena::new(Arc::new(pool.clone()), ok_producer());
|
||||
let a = arena.submit_video(
|
||||
VideoTicketParams {
|
||||
viewer: 1,
|
||||
|
||||
@@ -16,13 +16,15 @@
|
||||
|
||||
//! The worker layer (C++ RenderWorkerPool + RenderThread +
|
||||
//! workerprocess/workerjson): thread pool AND process-isolated pool
|
||||
//! behind one enum.
|
||||
//! behind one dispatch seam.
|
||||
//!
|
||||
//! This pass ships the in-process [`WorkerPool`] fully. The
|
||||
//! [`ProcessPool`] (crash isolation via oakengine_ipc worker processes)
|
||||
//! is a documented stub: the oakengine_ipc C ABI worker binary is not
|
||||
//! wired into the Rust world yet, so `start`/`post` fail with
|
||||
//! `Error::Failed` and the crash-isolation tests are `#[ignore]`d.
|
||||
//! process-isolated backend landed in M15 S1 as
|
||||
//! [`crate::procpool::ProcessDispatcher`] (spawn/handshake/crash-restart
|
||||
//! of oak-worker binaries over NDJSON + shared memory); both backends
|
||||
//! implement the [`JobDispatch`] seam the ticket arena posts through.
|
||||
//! [`ProcessPool`] below is the frozen pre-M15 facade stub kept for C
|
||||
//! ABI parity.
|
||||
|
||||
use std::collections::{HashMap, VecDeque};
|
||||
use std::panic::{catch_unwind, AssertUnwindSafe};
|
||||
@@ -52,6 +54,21 @@ fn lock<T>(m: &Mutex<T>) -> MutexGuard<'_, T> {
|
||||
m.lock().unwrap_or_else(|e| e.into_inner())
|
||||
}
|
||||
|
||||
/// The job-dispatch seam (M15 S1): the ticket arena posts [`Job`]s
|
||||
/// through this interface without knowing the backend. Implemented by
|
||||
/// the in-process [`WorkerPool`] (threads) and the process-isolated
|
||||
/// [`crate::procpool::ProcessDispatcher`] (oak-worker children); S2
|
||||
/// removes the thread pool and this seam becomes process-only.
|
||||
pub trait JobDispatch: Send + Sync {
|
||||
/// Enqueue a job; false when the backend is gone (the arena then
|
||||
/// delivers the completion itself with `Error::State`).
|
||||
fn post(&self, job: Job) -> bool;
|
||||
|
||||
/// Stop accepting work, deliver the queued completions (cancelled)
|
||||
/// and release the backend. Idempotent.
|
||||
fn shutdown(&self);
|
||||
}
|
||||
|
||||
/// Thread-pool backend (C++ RenderThread model). Cheap to clone (all
|
||||
/// state is behind an `Arc`); the manager and the ticket arena share one
|
||||
/// pool.
|
||||
@@ -132,6 +149,12 @@ impl WorkerPool {
|
||||
/// without running); running jobs are joined so no completion fires
|
||||
/// after shutdown returns.
|
||||
pub fn shutdown(&mut self) {
|
||||
self.shutdown_ref();
|
||||
}
|
||||
|
||||
/// [`Self::shutdown`] on a shared reference (the [`JobDispatch`]
|
||||
/// seam; all state is interior-mutable). Idempotent.
|
||||
pub fn shutdown_ref(&self) {
|
||||
// Set the flag and wake the workers while holding the queue lock.
|
||||
// Workers decide whether to block in `cv.wait` while holding that
|
||||
// lock, so a flag set outside it could land between a worker's
|
||||
@@ -158,6 +181,16 @@ impl WorkerPool {
|
||||
}
|
||||
}
|
||||
|
||||
impl JobDispatch for WorkerPool {
|
||||
fn post(&self, job: Job) -> bool {
|
||||
WorkerPool::post(self, job)
|
||||
}
|
||||
|
||||
fn shutdown(&self) {
|
||||
self.shutdown_ref();
|
||||
}
|
||||
}
|
||||
|
||||
fn worker_loop(inner: Arc<PoolInner>) {
|
||||
loop {
|
||||
let job = {
|
||||
|
||||
@@ -47,7 +47,10 @@ fn frame_producer() -> oakrender::ticket::Producer {
|
||||
fn cacher() -> (oakrender::autocacher::PreviewAutoCacher, WorkerPool) {
|
||||
let mut pool = WorkerPool::new(2);
|
||||
pool.start();
|
||||
let arena = Arc::new(TicketArena::new(pool.clone(), frame_producer()));
|
||||
let arena = Arc::new(TicketArena::new(
|
||||
Arc::new(pool.clone()),
|
||||
frame_producer(),
|
||||
));
|
||||
(oakrender::autocacher::PreviewAutoCacher::new(arena), pool)
|
||||
}
|
||||
|
||||
|
||||
@@ -83,7 +83,7 @@ impl Drop for GateRelease {
|
||||
fn ticket_completion_once_success() {
|
||||
let mut pool = WorkerPool::new(2);
|
||||
pool.start();
|
||||
let arena = TicketArena::new(pool.clone(), ok_producer());
|
||||
let arena = TicketArena::new(Arc::new(pool.clone()), ok_producer());
|
||||
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let id = arena.submit_video(
|
||||
@@ -120,7 +120,7 @@ fn ticket_completion_once_on_cancel() {
|
||||
}
|
||||
Ok(oakrender::ticket::TicketPayload::Video(Texture::wrap_frame(small_frame())))
|
||||
});
|
||||
let arena = TicketArena::new(pool.clone(), blocking);
|
||||
let arena = TicketArena::new(Arc::new(pool.clone()), blocking);
|
||||
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let id = arena.submit_video(
|
||||
@@ -159,7 +159,7 @@ fn shutdown_drains_completions() {
|
||||
Ok(oakrender::ticket::TicketPayload::Video(Texture::wrap_frame(small_frame())))
|
||||
})
|
||||
};
|
||||
let arena = TicketArena::new(pool.clone(), blocking);
|
||||
let arena = TicketArena::new(Arc::new(pool.clone()), blocking);
|
||||
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let mut ids = Vec::new();
|
||||
@@ -197,7 +197,7 @@ fn shutdown_drains_completions() {
|
||||
fn pool_saturation() {
|
||||
let mut pool = WorkerPool::new(4);
|
||||
pool.start();
|
||||
let arena = TicketArena::new(pool.clone(), ok_producer());
|
||||
let arena = TicketArena::new(Arc::new(pool.clone()), ok_producer());
|
||||
let (tx, rx) = mpsc::channel();
|
||||
let mut ids = Vec::new();
|
||||
for i in 0..64u64 {
|
||||
@@ -228,7 +228,7 @@ fn pool_saturation() {
|
||||
fn ticket_id_monotonic() {
|
||||
let mut pool = WorkerPool::new(1);
|
||||
pool.start();
|
||||
let arena = TicketArena::new(pool.clone(), ok_producer());
|
||||
let arena = TicketArena::new(Arc::new(pool.clone()), ok_producer());
|
||||
let a = arena.submit_video(params(Rational::new(0, 1)), Box::new(|_| {}));
|
||||
let b = arena.submit_video(params(Rational::new(1, 1)), Box::new(|_| {}));
|
||||
let c = arena.submit_video(params(Rational::new(2, 1)), Box::new(|_| {}));
|
||||
|
||||
@@ -660,7 +660,7 @@ impl RenderTask {
|
||||
oakrender::eval::render_produced_frame(time, params)
|
||||
.map(TicketPayload::Video)
|
||||
});
|
||||
let arena = Arc::new(TicketArena::new(pool.clone(), producer));
|
||||
let arena = Arc::new(TicketArena::new(Arc::new(pool.clone()), producer));
|
||||
(arena, Some(pool))
|
||||
}
|
||||
};
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
# M15:渲染进程隔离(oak-worker 真实化)设计
|
||||
|
||||
> 状态:已批准(用户 2026-08-18 提出,作为独立追加任务,不阻塞 M12 其余阶段)。
|
||||
> 前置调研:见会话调研报告(oak-worker/ipc.rs 传输层已完整、TicketArena 投递口收敛、上屏链路 6 处拷贝点)。
|
||||
|
||||
## 1. 目标(用户原文要求)
|
||||
|
||||
1. oak-worker 做成真实渲染进程;**删除主进程内部渲染线程池**(oakrender `WorkerPool`),统一走"渲染进程 + 主进程"隔离模型;OFX 插件崩溃不得连累主进程。
|
||||
2. **全链路零拷贝**:主进程与 worker 之间 stdio 传控制指令、共享内存传帧;所有渲染操作在共享内存池内完成;除 GPU 上传外不做任何拷贝;渲染完直接从共享内存池零拷贝上屏。
|
||||
3. **零锁**:帧槽交接用无锁 SPSC 环(已有);调度器单线程;worker 单渲染线程。
|
||||
4. **批量认领、无工作窃取**:worker 认领一批帧(约 120 帧量级)后,这些帧不再分配给别的 worker。
|
||||
5. **调度均匀性**:相邻的帧要几乎同时渲染完——相邻帧分配给不同 worker。
|
||||
|
||||
## 2. 架构总览
|
||||
|
||||
```
|
||||
主进程 (GUI / CLI / Export)
|
||||
├── PreviewScheduler(新,oakrender/src/scheduler.rs)
|
||||
│ 帧需求队列(播放窗口/seek/导出/缩略图)→ 分批 → 交织派发给 worker
|
||||
├── ProcessDispatcher(新,oakrender/src/procpool.rs)
|
||||
│ N 个 WorkerHandle:spawn oak-worker、handshake、NDJSON 控制、崩溃检测重启
|
||||
├── TicketArena(现有)—— 投递口从 WorkerPool.post 换成 ProcessDispatcher
|
||||
└── ShmPool(主进程侧,ipc.rs SharedMemoryRegion 复用)
|
||||
段 0..N:每 worker 一段;段内 FrameSlotPool(free/ready SPSC 环,已有)
|
||||
│ stdio NDJSON(控制面)
|
||||
▼
|
||||
oak-worker × N(渲染进程)
|
||||
WorkerSession(已有骨架)+ 真实渲染栈(图反序列化/解码/合成/插件)
|
||||
渲染结果直接写入 shm 槽 → frame_ready(ticket, slot)
|
||||
```
|
||||
|
||||
## 3. 关键设计决策
|
||||
|
||||
### 3.1 共享内存布局
|
||||
|
||||
- **每 worker 一个 shm 段**(主进程 create、worker attach,复用 `SharedMemoryRegion` 双模式与 `FrameSlotPool`,key 规范沿用 `olive-rw-<pid>-<index>`)。
|
||||
- **槽由主进程统一编址**:render 指令携带目标 slot id,worker 无权自行选槽 → 主进程可以把"预览缓存"直接建在槽上:预渲染帧的槽即缓存,上屏读槽即零拷贝;槽释放 = 缓存淘汰。当前帧(播放头)上屏路径:**shm 槽切片 → `queue.write_texture`**(GPU 上传是用户许可的唯一拷贝)。
|
||||
- **槽格式**:viewer 预览票请求 **BGRA8**(新 force_format;worker 在渲染管线末端 F32→BGRA8 转换后写入槽——格式转换不是拷贝);导出/全分辨率/scopes 票请求 **F32 RGBA**。槽大小 = 该段服务过的最大帧(64 对齐),段按需 ftruncate 扩容或重建。
|
||||
- **槽数**:每 worker 8 槽起步(决定单 worker 在飞帧数;内存 = N × 8 × 8.3MB(BGRA8 1080p))。
|
||||
- ⚠️ **Spike 必验**:macOS POSIX `shm_open` 单段大小上限(目标 ≥ 512MB)。不达标则回退"临时文件 + `mmap(MAP_SHARED)`"(接口封装在 `SharedMemoryRegion` 内,加 backend 枚举,协议不变)。Linux 用 POSIX shm 即可。
|
||||
|
||||
### 3.2 调度器(本任务核心,无 C++ 参照)
|
||||
|
||||
- **帧需求模型**:键 = (sequence, 帧号, 参数版本[图版本/代理档/分辨率档/色彩])。来源:
|
||||
1. 播放前向窗口(默认前向 120 帧 + 后向少量,随播放头滑动);
|
||||
2. seek/当前帧(最高优先,插单帧);
|
||||
3. 导出(连续全量,经 oaktask);
|
||||
4. 缩略图/静止全分辨率帧。
|
||||
- **交织批量认领**:W 个 worker。待渲帧流按 **round-robin 分片**:worker i 认领帧号 ≡ i (mod W) 的子序列。每次握手以"批"为单位:worker 认领自己等差序列中的下 B 帧(B ≈ 120/W,可配置)。性质:
|
||||
- 每帧恰好属于一个 worker(**无窃取**,满足要求 4);
|
||||
- 相邻帧号落在不同 worker 上并行渲染 → 相邻帧完成时间近似相同(满足要求 5);
|
||||
- worker 批内按帧号升序渲染,完成即发布(槽就绪顺序对主进程透明,主进程按帧号索引消费)。
|
||||
- **故障恢复(非窃取)**:worker 崩溃(管道 EOF/退出码非零)→ 其**未开始**的批整体回收重派;**已开始**的批中未 frame_ready 的帧标记失败并重派给健康 worker(崩溃帧重派属故障恢复,不违反无窃取)。慢 worker 不干预(防卡死优先于 fairness)。
|
||||
- **流量控制**:主进程只在目标 worker 有 free 槽时派批(槽即信用);worker 批内渲到无 free 槽时等待(SPSC 环 poll + 超时让出)。
|
||||
- **优先级**:seek/当前帧 > 播放窗口(距播放头近者优先)> 导出后台。取消 = cancel 消息 + 帧键失效(版本号 bump)。
|
||||
|
||||
### 3.3 协议(NDJSON v2,向后兼容 v1 消息名)
|
||||
|
||||
已有:handshake / load_graph / render_frame / frame_ready / cancel。
|
||||
新增:
|
||||
- `hello_caps`(worker→main):支持格式、最大帧尺寸(协商槽大小)。
|
||||
- `render_batch { tickets: [{ticket_id, time, slot_id, params…}] }`:一批帧 + 指定槽。
|
||||
- `batch_accepted { batch_id, tickets[] }`:认领确认(认领语义显式化)。
|
||||
- `frame_failed { ticket_id, error }`:渲染失败(主进程兜底紫帧)。
|
||||
- `shutdown` / 崩溃检测:主进程 waitpid + EOF。
|
||||
|
||||
### 3.4 拆除线程池的影响面(调研结论)
|
||||
|
||||
唯一投递口 `TicketArena.pool.post`(ticket.rs:321,390)。消费者 4 个(app renderops、oak-cli、oaktask 导出、oakengine facade)API 不变。`WorkerBackend::{Threads,Processes}` 枚举已预留。oaktask 导出自建私有池改为自建私有 ProcessDispatcher(max_inflight 语义由调度器接管)。**WorkerPool 及其线程在 S2 彻底删除**(用户明确要求;单测需要的同步执行语义由 dispatcher 的 `run_inline` 测试模式提供,不保留生产线程池)。
|
||||
|
||||
### 3.5 上屏零拷贝改造(S2,src/ 侧)
|
||||
|
||||
现状拷贝点(调研报告):ticket result `frame.data.clone()` → linesize 重排 → F32→BGRA8 → 缓存 → atlas write_texture。
|
||||
改后:ticket 完成返回 `ShmFrameRef{worker, slot, meta}`;viewer 预览帧(BGRA8 槽)直接切片喂 `write_texture`;scopes 分析改为读 BGRA8(精度足够)或另请 F32 票;长期缓存(静止全分辨率单帧槽、缩略图)从 shm 拷出一次(必要拷贝,槽需回收)。断言手段:`renderops` 加拷贝字节计数器,测试断言播放路径帧字节拷贝 = 0(GPU upload 除外)。
|
||||
|
||||
### 3.6 OFX 崩溃隔离
|
||||
|
||||
插件执行器(oakplugin `install_render_executor`)装在 **worker 进程**;主进程不再链接执行栈(app 只经 ticket API)。测试插件加"崩溃模式"(环境变量触发 raise(SIGSEGV))→ 验收:主进程存活、受影响帧重派、worker 自动重启、渲染结果仍正确。
|
||||
|
||||
### 3.7 音频
|
||||
|
||||
音频票同协议走 shm(AudioSamples 入槽)。S1/S2 可先保持主进程音频路径(崩溃风险主要来自视频插件),S3 迁移。
|
||||
|
||||
## 4. 分期
|
||||
|
||||
| 期 | 范围 | 验收 |
|
||||
|---|---|---|
|
||||
| S1(crates only) | shm spike(macOS 段上限);协议 v2;ProcessDispatcher + WorkerHandle + 崩溃重启;worker 侧真实渲染(图快照反序列化 + montage/解码/合成 + 插件执行器);PreviewScheduler(交织批量认领);与线程池**并存**(配置切换)。单测 + 集成测试(崩溃隔离/无窃取/均匀性/零拷贝计数)。 | `cargo test -p oakrender -p oak-worker` 全绿;集成测试演示 4 worker 渲 480 帧无重分配、相邻帧完成时间差有界 |
|
||||
| S2 | RenderManager 默认 Processes;**删除 WorkerPool**;oaktask/oak-cli 接入;src/ 上屏零拷贝(real.rs/renderops.rs/frames.rs);app 播放前向窗口接入调度器 | `cargo test` 全绿;`cargo run` 播放流畅;拷贝计数=0;kill -SEGV worker 后播放继续 |
|
||||
| S3 | 音频迁移;压测调优(B、槽数、worker 数自适应);README/docs 收尾 | 性能报告;文档 |
|
||||
|
||||
## 5. 风险
|
||||
|
||||
1. **macOS POSIX shm 上限** → 3.1 回退方案。
|
||||
2. worker 内 wgpu Device:montage/合成现状为 CPU 路径,S1/S2 保持 CPU;GPU 合成后续议。
|
||||
3. 图快照一致性:GraphSnapshotStore 引用计数已有;图变更(编辑)→ 版本 bump + 重新 load_graph。
|
||||
4. 调度器位于主进程 UI tick 驱动,需避免 tick 内阻塞(全部非阻塞 poll)。
|
||||
5. oakengine facade(冻结 C ABI)引用 WorkerPool 的部分同步改为 dispatcher(非默认构建成员,但仍需编译通过)。
|
||||
Reference in New Issue
Block a user