oak-common is gone; its modules (configstore, xmlutils, ocioutils,
oiioutils, colormath, colortransform, videoparams, ffmpegutils, ...)
now live in oak-core alongside the value types. The render value/GPU
types moved too: backend (wgpu context + DisplayRenderer), color
(ColorProcessor over ocio-rs), texture, frame, and the commonutil
config helpers.
Fix-ups to make the merged tree build and pass tests:
- oak-core Cargo.toml: wgpu back to 25 (the moved backend code is
written against that API generation); add the toml/quick-xml/image
deps oak-common carried.
- lib.rs: drop the duplicate 'pub mod error;'.
- error.rs: unified OAKCORE_* codes; restore Error::new() and
From<OcioError> from oak-common's error type.
- backend.rs/color.rs: oak_core::/oak_render:: self-references
rewritten to crate::; the shaderfx-dependent GPU effect test moved
to oak-render's shaderfx tests (shaderfx depends on oak-node and
cannot live in oak-core).
- oak-render's error module re-exports oak_core::error::{Error,
Result}; the OAKRENDER_* codes stay as the public-code contract.
- oak-node jobs.rs: ColorProcessor imported from oak_core::color.
- Integration tests repointed at oak_core::{texture, frame, backend,
color, colormath}.
- the display-ICC regression test treats an empty OAK_DISPLAY_ICC as
unset, matching displayicc::env_override_icc.
The OCIO grading nodes previously pushed null texture handles; they now
push real ShaderJobPayloads whose GLSL is the OCIO-generated dynamic
grading-primary GPU shader — the exact code the C++ path applies, so no
approximation:
- color: grading_primary_function_shader(style) builds a dynamic
GradingPrimaryTransform (LIN/LOG) on the default config, extracts the
GLSL via GpuShaderDesc (function 'ove_grading_primary', resource
prefix ocio_, no LUT textures) and caches it per style + config id.
- eval: OCIO_GRADING_STUBS maps the two node type ids to the grading
style; process_shader_job resolves the stub and splices it into the
node's %1 marker (same wiring as the chromakey OCIO stub); the
pipeline cache key folds the stub text so a config change recompiles.
- nodes: value() pushes a ShaderJobPayload with the C++ value()
rewrite applied to the row — vec4 (RGBM x=master) grading inputs to
the vec3 GPU uniform form (lin: contrast RGB=c*m, offset RGB=c+m,
exposure RGB=2^(c+m); log: lift RGB=c+m, gain c*m, gamma c*m), plus
pivot/saturation floats, the log pivotBlack/pivotWhite normalization
range (0/1), clamp sentinels (NoClampBlack -1 / NoClampWhite 2),
white>black enforcement per frame, and localBypass=false. Generated
uniform names bind by name (the log node's OCIO_NAMESPACE_ id text
normalizes to the ocio_ resource prefix).
- Tests: grading stub generation (analytic GLSL, cache) in color,
end-to-end GPU exposure doubling for lin (+1 stop on 0.2 gray -> 0.4)
and lift for log, node payload rewrite assertions, and the
all-shaders sweep now retries grading stubs. oak-render 181,
oak-node 441, oak-app 272 lib tests pass.
Polygon and mask previously pushed null texture handles ('fake'
implementations). They now generate real ShaderJobPayloads and render
through the existing GPU shader pipeline:
- shaderfx: std140 uniform array support (Vec4Array(N)) — the parser
accepts 'uniform <type> <name>[N];' declarations, translate()
re-emits them as vec4[N] block members with per-element std140
offsets, and pack_uniforms writes array items from the new
NodeValue::Vec4Array value, padding short arrays to the declared N.
- polygon: value() collects the inherited points array (row element
keys 'points_in[i]', else the node's own per-element values — an
unconnected array resolves to the default pentagon via
GetValueAtTime parity) and pushes a ShaderJobPayload; the 'rgb'
fragment shader rasterizes the closed point loop with an odd-even
fill in screen space (center-translated, y-flipped to match the C++
point convention) and outputs color_in inside / transparent outside.
The CPU QPainterPath generate_frame stays a documented no-op.
- mask: value() pushes a single ShaderJobPayload whose new 'mask'
fragment shader folds the whole C++ chain into one GPU pass — base
texture multiplied by the polygon matte, optional invert, and the
optional feather gaussian softens the matte during sampling (the
separable blur.frag h/v iterations as a one-pass product,
density-normalized, radius capped at 16 px).
- Tests: translate/pack array coverage in shaderfx, GPU end-to-end
rasterization of the pentagon (center white, corner transparent),
mask multiply/invert/feather on real frames, and updated oak-node
payload assertions. oak-render 178, oak-node 440, oak-app 272 lib
tests pass.
Tests run in parallel inside one process; each GraphSnapshotStore::new()
used the same 'oakrender-snapshots-<pid>' root, so one test's cleanup()
deleted another test's live snapshot — the
acquire_rewrite_forces_file_rewrite_on_same_key flake on the 16-core
Windows runner (file written, then exists() == false). Append a per-store
sequence number to the directory.
- multicamnode: type_id() assertions called the method on dyn
NodeBehavior, which method resolution routed to std::any::Any's
TypeId::of — qualify via NodeBehavior::type_id so the trait method
(the &str node type id) is compared
- procpool: the per-worker GPU budget grew a security headroom (×2)
for the CUDA-OOM flood; the two budget tests now assert against the
real formula (2 GiB + 256 MiB at 1080p24, 2 workers at 4K/24 GiB)
- cli info fixture: the fixture runs 29.97 fps; the assertion expected
30/1 (stale from the older fixture)
The extra oak_task export fields (bitrates + color override code
points) added in the export-settings protocol are zero/defaults in the
CLI's sync exporter — audio/video bit rates codec-default, no explicit
color override (the project's delivery colorspace applies).
- 导出工程文件: the in-app dialog only picks the format (OTIO / OVE /
FCPXML); OK asks the system save dialog with the suggested file name
carrying the format's extension, then exports
- 打开项目: the system open dialog restricts the choosable files to
.ove/.ovexml/.otio/.fcpxml (PathPromptOptions.allowed_extensions
plumbed through gpui; Linux portal glob filter)
The three values are SpinBoxes (typed input + stepping): width & height
(0 = use the sequence size), frame rate in fps (0 = sequence rate),
bit rate in bit/s (0 = the codec default). The engine already accepted
arbitrary values, so the dialog now exposes them instead of the closed
preset lists. i18n across the 8 packs.
- ExportDialogContent: container format (existing), then the video and
audio codec combos rebuilt per container from the compatibility tables
(an unsupported codec is never offered — switching container
re-selects the first compatible entry), a color-space combo (SDR
Rec.709 8-bit / HDR BT.2020 10-bit), an export-range combo (whole
timeline / in-out) and the output path
- ComboBox::set_options added for the container-driven list rebuild
- settings() gathers the ExportSettings the engine consumes
- begin_export routes through start_export_with
- i18n keys across the 8 packs (zh-CN uses distinct values so the
en==zh equality test passes)
- ExportSettings: container format, video/audio codecs, size/fps/
bitrate, bit depth (8/10 → U8/U10), SDR 709 + HDR 2020 code points,
export range
- AppEngine::start_export_with(settings, path); real builds the
task params via encoding_params_with_settings (codec tables gate
each container — a non-supporting combo errors, never silently
defaults; the one menu only offers compatible codecs)
- oak-task EncodingParams: video_bit_rate / audio_bit_rate +
color_override_enabled + primaries/trc/space (override wins over
the project's delivery colorspace)
- mock start_export_with (no-op fake progress)
The dialog UI rebuild (fields + container→codec→color enability
linking) follows in the next unit.
- save_project_as(name): library_duplicate the open project's row,
rename the copy, open it as the current project
- 文件 → 另存为 opens a name dialog (prefilled '<name> Copy'); OK
duplicates + switches (the save-as semantics: a new, differently-
named project in the library, editing continues on the copy)
- 文件 → 新建项目 opens an in-app dialog: project name + the first
sequence's format (preset / custom / interlaced, reusing
SequenceFormatFields); OK creates the library project then seeds the
first sequence from the chosen format
- 打开项目 prompt advertises the supported project formats
(.ove / .otio / .fcpxml)
- manager.new.project_name across the 8 i18n packs
- action-search first-item test follows the new flow (New Project now
opens its dialog)
文件 → 导出工程文件… now opens an in-app dialog (format dropdown:
OpenTimelineIO / Oak / FCPXML + output path) instead of the bare
platform save prompt; OK routes the chosen extension through the
engine's export_project_path. Adds project.export.format i18n across
the 8 packs.
The 窗口 → 多机位向导 entry is removed from the menu bar and the
action-search list (menu_id 0, the hidden convention); the engine
pathway (wizard dialog, sync, multicam creation) stays intact behind
the action dispatch for a future release. The panel's multi-cam
feature remains.
- MulticamPanel: playback angle refresh runs one cycle per 3 ticks
instead of re-requesting every source every tick — each angle decode
is a keyframe-scanning FFmpeg seek that stole worker capacity from
the main viewer
- PreRender frames default 120 -> 12: a window larger than what the
pool can render in real time queues far ahead of the playhead, so
the painted frame lags seconds behind (playback frozen); a smaller
window keeps the backlog bounded
- render_graph_frame: OAK_PERF clip-level timing
The per-process 'hardware session == 1, evict before every new open'
guard was turning every frame's open into a decoder re-open: after
inserting the fresh session, the NEXT frame's pre-open eviction dropped
it again, so no request ever hit the cache ([open] (request) on every
single frame, zero CACHED hits). Every decode then cost a full FFmpeg
session open (~0.5 s) + a keyframe-scanning seek — playback could never
keep up (the 'main viewer barely moves' report; [perf] showed 1.6-2.2 s
per frame).
Hardware VRAM is bounded by the LRU cap (hardware-first eviction at
MAX_CACHED_DECODERS) and the gpu-vram worker-count policy; the eager
pre-open eviction was the regression.
Re-applies the proxy selection (with the -1 tolerance) after the
revert: the node-graph viewer decode must stand in a ready proxy for
the 4K original, and a proxy generated before the stream-index
recording (index -1) still matches the first video stream rather than
silently falling back to the 4K source (the 'main viewer got stuck'
report).
- multicam_switch_to: a NO-SPLIT switch only changes current source —
push snapshot + invalidate preview windows, skip the full
apply_edit rebuild (the 'switch once then everything grinds' hit)
- MulticamPanel: playback angle refresh runs one cycle per 3 ticks
instead of re-requesting every source every tick (each angle decode
is a keyframe-scanning FFmpeg seek)
- footage value(): a ready proxy stands in for the original media in
the node-graph decode request (proxy enabled but 4K decoded report)
- proxy_resolution made public (engine's proxy-first preview size)
gpu_vram_bytes() now chains per vendor/platform:
- NVIDIA everywhere: nvidia-smi (shipped by the NVIDIA driver on every
OS) — the NVDEC path's primary device.
- AMD/Intel on Linux: the DRM mem_info_vram_total/used sysfs attributes
(amdgpu, i915, Xe). Free = total - used; the first non-zero card wins
(an iGPU without dedicated vram reports 0 and is skipped). The walk is
now testable via an injectable sysfs root; a card missing the attrs is
skipped, never aborts the walk.
- Apple Silicon: unified memory — no separate vram exists; the RAM/4
budget IS the correct bound for decode surfaces and render targets, so
no query (an explicit vram budget would double-count the same pool).
- Windows AMD/Intel: no portable CLI; DXGI QueryVideoMemoryInfo is the
real API but wgpu 25 does not expose it. Falling back to the RAM
policy is safe-side (under-sized pool loses throughput, never OOMs the
device).
Tests: the sysfs walk (fixture with a missing-attr card, a 0-total
iGPU and a discrete winner) and the per-worker budget scaling (1080p
baseline, 4K ~4x, 60 fps over-provision).
- open_hw_accel marks the device unavailable when the decoder OPEN fails
(cuvidCreateDecoder OOM at 4K) too, not just device-context creation:
without it every subsequent decoder session retried CUDA and flooded
the log per open.
- Decoder gains hardware_decoding(); the oak-render decode-session LRU
evicts hardware sessions first (each pins a GPU surface pool — ~100 MB
at 4K), so a full cache cannot exhaust video memory before the next
open.
- Worker pool count now factors GPU vram: per-worker budget = 1 GiB
(1080p peak) scaled by pixel ratio + 256 MiB idle floor, 10% reserve
of free vram; applied when hardware decoding is on (nvidia-smi query,
None otherwise falls back to the RAM/CPU policy).
- Dynamic pool resize: ProcessDispatcher::set_target_workers grows or
retires workers; retiring ones stop claiming, drain their in-flight
batch (future playback frames included), then exit naturally on the
shutdown signal — no mid-work kill (30 s deadline only as a hung-
decoder last resort). A retiring worker that dies re-queues its frames
to surviving workers. Resizes are throttled to 2 s (a resolution burst
merges; only the latest target applies) so 1080p<->4K flaps cannot
thrash process spawns.
- RenderManager::set_workspace_size announces the sequence resolution;
RealEngine calls it from refresh_sequence_info.
- Integration test: shrink 3->1 mid-wave (all frames complete, retired
workers exit naturally) then regrow 1->3 and render a fresh wave.
- Project explorer header gains a 新建序列 button (opens the existing
new-sequence dialog, seeded like the menu action).
- Sequence presets gain 4K UHD (3840x2160@25) and 4K DCI (4096x2160@24);
the sequence-properties dialog re-selects them on reopen. Format fields
are also seedable from a probed footage format (the drop flow).
- Chroma Key (and Color Difference Key) value() now box a ShaderJobPayload
like Despill: the old OCIO-processor gate pushed nothing (the processor
is never populated without the render bridge), so the traverser handed
the clip NodeValue::None and the rendered frame lost the clip. The
renderer resolves the OCIO stub at compile time from OCIO_SHADER_STUBS.
End-to-end graph test: green key on a green frame keys out, red key
keeps it.
- The OFX color picker's SV palette / hue bar / preview / swatch canvases
get size_full(): the bare canvases collapsed to zero height in the
block layout, so the palette painted nothing (the reported 色板没显示).
Regression test clicks the palette center and expects mid s/v.
On multi-GPU boxes VAAPI's default render node can point at a device
with no VA driver (NVIDIA without libva-nvidia-driver), failing before
the working AMD/Intel node is ever tried; trying CUDA (NVDEC) first is
both the discrete-GPU path and sidesteps that misdirection. VAAPI stays
as the fallback for AMD/Intel-only machines.
Adds a hwcheck example printing which hw device a stream opens with and
the raw av_hwdevice_ctx_create result codes per device type.
Per-chunk submit->render wait and render time (audio_thread), the push
lead against the playhead, the negotiated stream config, and the true
device-side callback rate (outputdevice) -- the ground truth used to
localize the playback starvation to the sequence-rate/clock mismatch.
Two playback-audio defects found while chasing periodic pops, clipped
words and repeated phrases:
1. The playback clocks are created at engine init with the fallback
rate (hd_1080p25) and never updated, while audio chunks are sized by
the SEQUENCE time base (30/1 from the broken default). The playhead
advanced at 25 fps but each chunk carried only 1600 samples, so
production ran at 40000 samples/s against the device's 48000 -- ~17%
of playback time was zero-filled silence (clipped words). tick() now
syncs both clocks to the sequence's actual rate.
2. The underrun self-heal cleared the whole device queue (~190 ms of
unheard content skipped), reset the prefetch (whose first
re-rendered chunk was then always dropped as late) and re-anchored
the master clock, re-rendering already-played audio whenever the
playhead lagged (audible as a pop plus a repeated phrase, every
~1.1 s). It now trims just the stale lead from the queue
(PreviewAudioDevice::drop_front_frames): the zero-fill already
advanced the output clock past that content, so the rest of the
queue stays audible and the anchor stays valid.
Also adds the OAK_DEBUG_AUDIO event log used to diagnose all of this
(prefetch resets, late drops, underruns, heals, reanchor jumps,
per-tick queue state).
set_default_parameters looked up DefaultSequenceFrameRateNum/Den -- keys
that exist nowhere in the config -- so every sequence silently fell back
to 30/1. The canonical setting is the "num/den" string
DefaultSequenceFrameRate (a frame duration; rate = den/num, matching the
footage import path in oak-task). Sequences now default to the
configured 1001/30000.
threads=auto spawns one decode thread per logical core per decoder (16
observed on a 24-core box); with cores-2 render workers decoding
concurrently the machine is oversubscribed several times over and the
playback-audio render thread gets starved (late chunks are dropped).
4 threads decode 1080p h264 far past real-time; preview throughput comes
from the worker pool, not per-decoder threading. Applies to both the
software open path and the hwaccel open path.
retrieve_audio_to decodes whole codec frames but copies only the part
inside the requested chunk; the tail of the frame crossing the chunk end
(up to 1023 samples for AAC) was consumed by the decoder and lost, so
the next chunk started with a hole. On the playback grid (1920 samples
at 25fps/48kHz) the hole cycles 128..896 samples and hits 7 of 8 chunk
starts -- the heavy stutter/noise heard during playback.
Keep the overflow (decode and resampler-flush tails) in a per-session
carry buffer and serve it at the start of the next contiguous chunk;
clear it on seek, format change and chunk failure. Also make seek()
actually drop the cached resampler as its comment claimed.
Verified sample-exact: chunked decode of a 440Hz tone now matches a
one-shot decode bit for bit, and chunked renders of real media line up
with the ffmpeg CLI reference at correlation 1.0 / drift 0.
Adds a regression test (playback_sized_chunks_match_oneshot_sample_exact)
with a new tone fixture -- demo.mp4's audio is -90dB digital silence and
cannot expose the holes -- plus a render_audio_wav example used to
diagnose chunk-boundary artifacts offline.
Overlapping clips sum linearly in mix_audio_montage and can exceed full
scale (two hot clips reach +/-2; gain > 1 would too); the cpal sink
forwarded samples unclamped, so overlaps clipped at the DAC. Clamp the
accumulator after the montage mix (both the heap and shm-slot paths
share mix_audio_montage) and document it on render_audio_samples.
Run 48's Linux leg flaked: full_res_worker_outlives_a_dropped_project
panicked with "the render manager failed to start" while all 236 other
tests passed -- a worker-pool startup race under full-suite parallelism,
not a regression (the same binary passes locally and passed on the
previous commit). Mirror the Windows job's retry-once policy and make
ensure_render_manager log the underlying init error so the next flake is
diagnosable. The Windows leg's real failure (gpui_widgets referenced
SurfaceSource::Texture, which only exists on linux/freebsd) is fixed in
the gpui submodule bump of the previous commit.
Preview now follows the project output colorspace end to end: the
display chain derives its content space from the project's OutputColorSpec
instead of a hardcoded sRGB name, self-managed ICC transforms go through
an XYZ D65 interchange stage (OCIO cie_xyz_d65_interchange) for non-sRGB
targets, and the platform layer declares the content colorspace (gpui
submodule bump). macOS defaults to OS-managed (fixes wide-gamut UI
oversaturation); Windows ACM warns once on non-sRGB targets.
Multi-monitor: the display ICC is looked up per the window's current
screen (macOS display id, Windows per-monitor DC, X11 RandR output
profile) with a throttled poll that invalidates frame caches on moves.
Pipeline precision: 10-bit+ sources fall back to YUV444P16LE + a Rust
matrix conversion when swscale lacks F32 output (no more 8-bit
truncation); BT.709/2020 SDR decodes with BT.1886 gamma 2.4 instead of
the sRGB EOTF; working-space compositing no longer clamps RGB to [0,1]
(alpha still clamped); the output node clamps to the target gamut;
frames without colorimetry metadata convert with BT.709 defaults
(warned once) instead of passing through; scopes read the
output-colorspace signal on both F32 paths.
Also: only emit rerun-if-changed for .env when it exists (a missing file
made every build fully dirty).
- the popup now offers a large SV palette with a hue bar plus RGB/HSV
mode tabs (shared slider model swap, hsv<->rgb pure functions)
- while the eyedropper is armed, clicks outside the popup pick a colour
instead of dismissing it, so the picked draft can still be confirmed
- regression test pinning the OFX plugin colour set/read/undo chain
- i18n: ofx.color.mode_rgb/mode_hsv in all eight packs
It is a plain deferred div rather than an anchored element, so it gets
no automatic BlockMouse hitbox; clicks on its padding fell through to
the controls below.
- oak-worker gains an LRU frame cache (default 64 MiB) keyed by the
render-deterministic spec subset; repeat frames (paused frames,
scrubs over rendered ranges, re-renders after an effect change) are
memcpy-cheap, which is what made adding an OFX plugin -- and the
in-flight batches after removing one -- stall the UI
- audio dispatch on the Processes backend now mixes inline on the UI
tick instead of queueing behind video batches in the worker pool, so
video stalls no longer starve the ~100ms cpal output buffer into
silence
The display path no longer passes through BGRA8: render workers are
asked for F32, rows are repacked, colour management runs on the F32
data, and the frame uploads to an Rgba16Float wgpu texture that the
viewer presents directly on the Rgb10a2Unorm swapchain. The BGRA8
image remains only for scopes, the eyedropper, thumbnails and the
no-GPU fallback.
Preferences gains a display-bit-depth combo stored in the config store;
at startup the app forwards it to gpui_wgpu via OAK_DISPLAY_BIT_DEPTH,
which prefers Rgb10a2Unorm for 10-bit presentation. Takes effect after
restart (noted in the dialog); i18n in all eight packs.
The 18 multicam switch actions were bound globally to keys 1-9, so
gpui dispatched every digit keystroke as an action and text fields
never received it -- every numeric input in the app (slider value
boxes, spin boxes, the colour picker's hex field) could only type 0,
and the colour swatch never updated because hex parsing failed.
The bindings now carry the MulticamPanel key context: digits switch
angles while the panel is focused and type normally everywhere else.
- paused full-res renders now scale by the playback divider instead of
always rendering at native sequence size
- when a clip's proxy is active, sequence/footage renders clamp to the
proxy resolution (never upscale a 720p proxy to 1080p and back),
which was making proxied playback slower than no proxy
Replaces the project-thumbnail picker: the color popup's eyedropper
button arms the program viewer, the next click on the picture samples
that pixel (contain-fit mapped, BGRA from the current CPU frame) into
the picker's draft; Esc or the button cancels. Coordinated through a
global eyedropper mailbox on the engine.
- the eight toolbar tools now drive real editing: razor splits at the
click point, ripple/slip/roll/slide emit new undoable composite
commands (ripple_trim_clip, roll_edit, slide_clip, slip_clip),
zoom clicks zoom anchored at the cursor, track-select selects the
track right of the click; panel tool state and the Tools menu stay
in sync both ways
- the zoom-in/out toolbar icons now actually zoom (anchored at the
playhead) and the snap magnet icon toggles snapping like its checkbox
Every item in the program/source viewer menu now works and reflects
real state: zoom levels, full-screen toggle, safe margins (off/on/
custom), stop-on-last (honoured by both playback clocks), waveform
mode, show-fps overlay, and save-frame (writes a PNG of the current
frame). The viewer widget's in/out/clear buttons are wired to the
shared program workarea.
- File > New > Sequence opens a real dialog (name, PAL/NTSC/HD presets,
width/height/frame rate, progressive/interlaced); VideoParams gains an
interlaced flag
- File > New > Folder creates a folder in the project root
- sequences are mounted under the root folder so they appear in the
project explorer (including the auto-created Sequence 1 and orphans
from older projects); right-click > Sequence Properties edits the
parameters afterwards
- dropping footage onto an empty, sequence-less timeline auto-creates a
sequence from the footage's first video stream
- the swatch now follows the draft color while the popup is open
(cancel still reverts) and set_effect_param failures are logged
instead of silently reverting the swatch
- the popup lists project footage thumbnails; clicking a pixel picks
its color into the draft (contain-fit coordinate mapping, no new
dependencies)
- i18n: ofx.color.pick_project in all eight packs
- clicking anywhere on the track moves the thumb to that position
(double-click still opens the inline editor)
- a compact numeric field next to every slider shows the value and
accepts typed input, clamped to the slider's min/max range
mov/mp4 muxers rewrite the stream time base at write_header (mov timescale
>= 10000) and FFmpeg 9 no longer rescales packets for us, so mpeg2video
clips were muxed with pts in encoder ticks -- a 10-frame/10fps clip became
1ms long and every seek past t=0 decoded to the EOF frame.
- read back the real stream time base after write_header and rescale
packets (including the flush path) before write_interleaved
- warn once when retrieve_frame's EOF fallback returns a frame far from
the requested timestamp
- testmedia round-trip now asserts container duration and per-frame
decode instead of only t=0
Mike-Solar