- oak-worker gains an LRU frame cache (default 64 MiB) keyed by the
render-deterministic spec subset; repeat frames (paused frames,
scrubs over rendered ranges, re-renders after an effect change) are
memcpy-cheap, which is what made adding an OFX plugin -- and the
in-flight batches after removing one -- stall the UI
- audio dispatch on the Processes backend now mixes inline on the UI
tick instead of queueing behind video batches in the worker pool, so
video stalls no longer starve the ~100ms cpal output buffer into
silence
The display path no longer passes through BGRA8: render workers are
asked for F32, rows are repacked, colour management runs on the F32
data, and the frame uploads to an Rgba16Float wgpu texture that the
viewer presents directly on the Rgb10a2Unorm swapchain. The BGRA8
image remains only for scopes, the eyedropper, thumbnails and the
no-GPU fallback.
Preferences gains a display-bit-depth combo stored in the config store;
at startup the app forwards it to gpui_wgpu via OAK_DISPLAY_BIT_DEPTH,
which prefers Rgb10a2Unorm for 10-bit presentation. Takes effect after
restart (noted in the dialog); i18n in all eight packs.
The 18 multicam switch actions were bound globally to keys 1-9, so
gpui dispatched every digit keystroke as an action and text fields
never received it -- every numeric input in the app (slider value
boxes, spin boxes, the colour picker's hex field) could only type 0,
and the colour swatch never updated because hex parsing failed.
The bindings now carry the MulticamPanel key context: digits switch
angles while the panel is focused and type normally everywhere else.
- paused full-res renders now scale by the playback divider instead of
always rendering at native sequence size
- when a clip's proxy is active, sequence/footage renders clamp to the
proxy resolution (never upscale a 720p proxy to 1080p and back),
which was making proxied playback slower than no proxy
Replaces the project-thumbnail picker: the color popup's eyedropper
button arms the program viewer, the next click on the picture samples
that pixel (contain-fit mapped, BGRA from the current CPU frame) into
the picker's draft; Esc or the button cancels. Coordinated through a
global eyedropper mailbox on the engine.
- the eight toolbar tools now drive real editing: razor splits at the
click point, ripple/slip/roll/slide emit new undoable composite
commands (ripple_trim_clip, roll_edit, slide_clip, slip_clip),
zoom clicks zoom anchored at the cursor, track-select selects the
track right of the click; panel tool state and the Tools menu stay
in sync both ways
- the zoom-in/out toolbar icons now actually zoom (anchored at the
playhead) and the snap magnet icon toggles snapping like its checkbox
Every item in the program/source viewer menu now works and reflects
real state: zoom levels, full-screen toggle, safe margins (off/on/
custom), stop-on-last (honoured by both playback clocks), waveform
mode, show-fps overlay, and save-frame (writes a PNG of the current
frame). The viewer widget's in/out/clear buttons are wired to the
shared program workarea.
- File > New > Sequence opens a real dialog (name, PAL/NTSC/HD presets,
width/height/frame rate, progressive/interlaced); VideoParams gains an
interlaced flag
- File > New > Folder creates a folder in the project root
- sequences are mounted under the root folder so they appear in the
project explorer (including the auto-created Sequence 1 and orphans
from older projects); right-click > Sequence Properties edits the
parameters afterwards
- dropping footage onto an empty, sequence-less timeline auto-creates a
sequence from the footage's first video stream
- the swatch now follows the draft color while the popup is open
(cancel still reverts) and set_effect_param failures are logged
instead of silently reverting the swatch
- the popup lists project footage thumbnails; clicking a pixel picks
its color into the draft (contain-fit coordinate mapping, no new
dependencies)
- i18n: ofx.color.pick_project in all eight packs
- clicking anywhere on the track moves the thumb to that position
(double-click still opens the inline editor)
- a compact numeric field next to every slider shows the value and
accepts typed input, clamped to the slider's min/max range
mov/mp4 muxers rewrite the stream time base at write_header (mov timescale
>= 10000) and FFmpeg 9 no longer rescales packets for us, so mpeg2video
clips were muxed with pts in encoder ticks -- a 10-frame/10fps clip became
1ms long and every seek past t=0 decoded to the EOF frame.
- read back the real stream time base after write_header and rescale
packets (including the flush path) before write_interleaved
- warn once when retrieve_frame's EOF fallback returns a frame far from
the requested timestamp
- testmedia round-trip now asserts container duration and per-frame
decode instead of only t=0
Match the timeline UI (V_max drawn topmost): composite tracks from
V1 up to V_max so the highest-numbered track is composited last, in
both the montage path and direct graph evaluation.
- take_node clears the partner's dangling link references
- are_linked checks both directions; link() repairs asymmetric pairs
- paste writes links through the graph API instead of BlockCore.links
- drop link undo removes only its own pair instead of restoring a snapshot
- move_clip_with_links skips off-track partners instead of failing
Proxy generation no longer pegs the machine:
- the transcode probes for a hardware H.264 encoder once per process
and uses it when present (macOS h264_videotoolbox; Windows/Linux
h264_nvenc -> h264_qsv -> h264_amf; libx264 remains the universal
fallback and the untouched C++ parity path), with decode-side
-hwaccel auto (HEVC 4:2:2 10-bit decodes in hardware on Apple
Silicon / RTX 50+ / Intel GPUs and falls back to software cleanly);
quality/preset map from the proxy CRF/preset per encoder
- the concurrency limit is configurable (ProxyMaxConcurrent, default
1) in the Proxy Settings dialog; auto-generated jobs queue and the
next starts when a slot frees. The invariant
concurrency x per-job threads <= logical cores / 2 holds by
construction (thread budget = half the cores / concurrency, clamped
to [1,8], covered by a unit test), and on Unix ffmpeg runs nice -n 10
so the background task never starves the foreground
- deleting a footage's proxy also removes it from the auto-generation
queue
playback_display_tracks_the_playhead asserted the displayed frame lag
the playhead by less than 4 frames at playhead 120 — a render-
throughput assumption that fails under machine load (parallel builds)
rather than for a broken pipeline, which made the test flaky. It now
asserts the machine-speed-independent property: the display advances
while the playhead advances (peak displayed grows between the two
checkpoints), which still catches a permanently frozen picture.
rebuild_timeline synchronously extracted every audio clip's waveform,
and the extractor decodes the clip's ENTIRE audio inline — with two
40-minute files on the timeline, opening the project froze the UI for
seconds ("the project takes a while to open"). Extraction now runs on
a background thread; the cache bumps a version counter on insert and
the engine tick repaints the timeline when it changes.
With the global proxy switch on, footage that needs a proxy now gets
one generated in the background automatically — on import, on project
open, and when proxy use is enabled for a footage — instead of only
through the manual Generate menu action. Footage explicitly opted out
(a recorded proxy path with use disabled) and footage whose last
generation failed are not restarted; the generation path enables proxy
use so preview switches to the proxy as soon as it is ready.
The bin shows the lifecycle as a corner badge on each footage entry
(green P = ready, amber P = generating, red ! = failed; the widget
half lives in the gpui fork, submodule bumped here).
Includes a real-media test driving ProxyTask end-to-end on 4K H.265
4:2:2 10-bit (the media class the 4K playback lag was reported on):
180 s transcodes to a 720p proxy in ~25 s.
The pipeline is ACEScg + F32 end to end by design; a plugin that does
not support F32 must not fail the render — its inputs convert down to
the negotiated depth and its output converts back to F32 (the previous
"Phase 2 F32 only" error path purple-framed those plugins).
- render_driver maps getClipPreferences' output bit depth to
Byte/Short/Half/Float, sets it on the input/output clips, allocates
the output image at the negotiated depth and converts the result
back to F32 for frame assembly (both the CPU path and the
GL-failure CPU fallback)
- the GL path keeps F32 clip params: GL textures are created in the
pipeline format and kOfxOpenGLPropPixelDepth is negotiated
separately, so clip props must match what the plugin actually sees
- fetch_image converts the decoded input down to the clip's
negotiated depth (default F32)
- new Image::convert_depth ([0,1]-normalized conversion between all
depth pairs) plus round-trip and f16 edge tests; f16 helpers moved
into image.rs and shared with clip.rs
Root causes found for the 4K stalls and the second-footage memory
blowup (audit + code review):
- ticket bookkeeping leaked unbounded: the procpool ticket table and
the arena slot map only ever grew (50-100 tickets/sec during
playback, each pinning montage params and shm region views).
Completed/cancelled/superseded/crashed entries are now removed, and
the arena reaps fire-and-forget tickets once finished; the sync poll
path reaps via a terminal result() read. InFlight duplicate submits
now answer State immediately instead of sitting in the map forever.
- decode ran a full-resolution swscale to F32 RGBA (~132 MB at 4K)
plus a second full-res copy before downscaling to the 480px proxy:
RetrieveVideoParams.target_size lets swscale convert AND resize in
one pass (bilinear, matching the old Rust resampler), so a 4K
preview frame costs ~1 MB instead of ~260 MB of churn. This applies
to proxy AND full-res requests alike.
- per-process decoder cache was unbounded (each session pins an FFmpeg
context + 2 native decoded frames): LRU-capped at 16, eviction drops
the map entry (in-flight renders keep their Arc; Drop releases
FFmpeg).
- playback window completions were not generation-gated: a stale
render from before an edit landed in the rebuilt window (wrong frame
displayed, fresh request blocked). Stale completions now return
their shm slot credit instead.
- async audio prefetch used the polling ticket submit without ever
polling: switched to the fire-and-forget submit so entries reap.
Three independent host-side gaps kept real plugins from instantiating:
- clips now define OfxImageClipPropFieldOrder (default OfxFieldNone):
ofxs Clip::getFieldOrder() is a strong read with no default, so
field-aware plugins (Mirror) threw PropertyUnknownToHost ->
MissingHostFeature from createInstance
- OfxParamPropParametricUIColour is no longer predefined as an empty
array: the OFX implicit-create semantics let the plugin's first
propSetDouble create the property and grow it index by index, while
the empty predefined array rejected every write at index >= 1 with
BadIndex (ColorLookup, HueCorrect describeInContext)
- Host::create_instance_preferred: context selection with fallback
(filter -> general -> tracker -> paint -> rest). TrackerPM advertises
the filter context but its createInstance fetches a Mask clip that
its describe only defines for tracker/general/paint; Natron simply
instantiates it in the tracker context, and now so do we. Both the
node-registration scan and the shared instance factory use it.
Still unsupported, by design: Premult/Unpremult (this openfx-misc
build hard-requires the Nuke multi-plane suite + dynamic choices) and
the stereo view plugins (Switch/anaglyph/joinViews/etc. need Natron's
isNatron multi-clip folding or view rendering).
Verified: ColorLookup, HueCorrect, Mirror, TrackerPM now instantiate;
full oak-plugin suite green.
The OFX abort contract is "0 = keep rendering, anything else = stop"
and the ofxs support library literally checks abort(...) != 0. Our
host answered kOfxStatReplyNo (13) for "not cancelled", so every
openfx-misc processor aborted its pixel loop before writing a single
row: the render action returned OK with the output buffer untouched —
a solid black frame everywhere an effect was applied (the previous
smoke test only asserted "not the purple failure frame", which let
this slip through as a fake pass).
- image-effect abort: return 0 when not cancelled, 1 when cancelled
(instance cancel flag or progress cancellation), BadHandle for
descriptor handles as before
- strengthen the real-plugin smoke tests to also reject an all-zero
output frame, so a silent no-write can no longer pass
- more trace-gated [ofx] diagnostics (value dumps for propGetN /
propGetPointer / clipGetImage fetches, multiThread entry, render
output buffer address) — the tooling that pinned this down
Verified with lldb watchpoints: the AddOFX/ChromaKeyerOFX processors
now write the real pixels through the executor path.
gpui delivers DragMoveEvent to every drop target with a matching
payload type under the cursor, and every slider shared the SliderDrag
payload type — dragging one slider (e.g. an OFX panel parameter) also
moved every other slider the cursor passed over (timeline zoom, track
height). Tag the payload with the starting control and ignore drag
moves that are not ours.
Also finish the gesture on mouse-up (inside or outside the track)
instead of on_drop: gpui only delivers drop events to the hovered
target, so releasing outside the track used to leave the drag
unfinished and the final value/undo edit unemitted.
The Cmd+K split ran the plain per-block split for every selected clip,
so an originally linked audio/video pair came out with its front halves
linked but the two rear halves unlinked — dragging one rear clip left
its mate behind and Link/Relink could not repair it.
Split the whole target set as ONE BlockSplitPreservingLinksCommand (the
same command the multicam path uses): any pair of originally linked
blocks split at the same time gets its new halves linked too. Adds a
gpui test asserting front and rear halves of a dropped A/V clip stay
linked after split_at_playhead.
Real plugins (CImg ChromaKeyerOFX, AddOFX) failed the render action with
kOfxStatFailed / MissingHostFeature and painted the magenta failure frame:
- images lacked the mandatory ImageBase properties (OfxPropType,
PixelAspectRatio, PreMultiplication, Field, RenderScale); the ofxs
ImageBase constructor throws on the missing/invalid strong reads
- PreMultiplication used the made-up string "OfxImagePreMultiplied";
kOfxImagePreMultiplied is actually "OfxImageAlphaPremultiplied", the
only value mapStrToPreMultiplicationEnum accepts (lldb __cxa_throw
backtrace pinpointed this)
- RenderWindow is Int x4 per ofxsPropertyValidation, not Double x4
- field strings use the real constant "OfxFieldNone"
- clips define OfxImageClipPropConnected (isConnected is a strong read;
optional mask clips blew up without it)
- choice params predefine empty ChoiceEnum / ChoiceLabelOption arrays
- isIdentity failure is no longer fatal (the C++ plugin renderer never
calls it; plugins that error on it simply render normally)
- property suite coerces Int <-> Double on reads (the CImg framework
reads the render window with propGetIntN against a Double store)
- in-args carry NatronOfxPropNativeOverlays=0 for the Natron framework
- plugin jobs pass a GL-kind marker so GL-only plugins take the real
gl_bridge offscreen path instead of the CPU MissingHostFeature path
- trace-gated [ofx] diagnostics for property misses and suite calls
Verified with new smoke tests that render the real AddOFX and
ChromaKeyerOFX plugins through the executor and assert the output is
not the purple failure frame.
Adding an effect to a clip did nothing: the sequence render is
flattened into a montage (decode + composite), and MontageClip carried
no effect data at all.
- MontageClip gains an ordered effect stack (type id / enabled /
effect input / parameter values); protocol v2 carries it as an
additive wire field (older peers default to an empty stack).
- renderops::video_montage fills the stack from the effect chain
(the footage source node — the chain end without an effect input —
is dropped; the montage decodes the footage itself). Export
(oak-task) and the multicam single-track montage fill it too.
- The worker applies the stack between decode and composite: built-in
Opacity gets a CPU evaluator (C++ opacity.frag parity — whole vec4,
alpha included, unity pass-through); everything else dispatches as an
OFX plugin job through a new instance-factory slot (oak-plugin
lazily creates + caches one instance per identifier per render
process) with the montage's parameters injected. Disabled effects
bypass (the C++ traverser pushes the effect input through). Unknown
types warn once per type id and pass through — no silent no-ops.
Not covered (explicitly): Transform/Crop and the other ~30 built-in
effects have no CPU evaluator in oak-render (they pass through with a
warning), keyframed parameter animation, audio effect chains, and the
CLI's simplified montage.
Acceptance: a real 50% Opacity on real media quarters the rendered
pixels both in-process (renderops test) and through a real worker
process over IPC + shared memory (procpool_integration test);
disabling restores the plain render byte-for-byte.
Link toggle rule refinement (on top of 1ad2d71c8): the toggle now
unlinks only when the selection is FULLY linked internally — the C++
crude "any member has ANY link" check made split halves (which inherit
the original clip's A/V links via BlockSplitPreservingLinksCommand)
impossible to link to each other. Test: link_unlink_toggles_the_
selection_links covers toggle/undo/split-link.
Node editor: only stream inputs (texture/samples) become ports — OFX
plugins declare every parameter as an input, which buried the card
under internal params (the garbled NatronOfxParamProcess* wall). Secret
inputs stay hidden; port labels use the input's display name (the OFX
param label); edges resolve ports by input id, not by label.
编辑 > 链接/重新链接 (Cmd+L) was an unhandled stub
(PanelCommandHandler::toggle_links returned false), so linking clips
never created graph links and linked drags never happened.
- graphops::set_clips_linked: one undoable entry toggling every pair of
the selection; the undo restores the exact prior internal topology.
- RealEngine::toggle_clip_links: fully-linked selections unlink,
otherwise the selection links together (the C++ crude "any member has
ANY link" rule would have made split halves — which inherit the
original clip's A/V links — impossible to link to each other).
- TimelinePanel overrides toggle_links, routing the timeline selection.
- MockEngine stores demo link pairs.
- Test: link_unlink_toggles_the_selection_links — dropped A/V pair
toggles off/on with undo, and split halves link manually.
- docs/zh/plans: finished plans move to completed/ (the RIIR series, the
event-bridge and dependency plans, the v04 manual test plan).
- New design docs: the external (functional) plugin system
(process-isolated, JSON-RPC/shm) and its protocol.
- ai-agent-design refreshed; README pointers follow the moves.
Touchpad fixes from user testing:
- SpinBox: the wheel only steps a FOCUSED field — hover-wheel is inert,
so two-finger scrolling across a dialog no longer drifts values.
- SpinBox: direct numeric entry — double-click opens the editor (the
app's text input; commit on blur, Escape cancels), typing a digit on
the focused field starts editing with that character. Previously the
field accepted no text at all.
- ComboBox: a click anywhere outside closes the open popup
(on_mouse_down_out).
- gpui bump: the timeline header-occlusion fix.
- Help > About Oak… opens a real dialog (was a placeholder print): name
+ version, the GPL-3.0 line, the Olive fork notice and the thanks to
Enzo GD (Community Promoter).
- Modal (gpui_widgets, submodule bump): a title-row close button on
every dialog and opt-in backdrop dismissal (dismiss_on_mask) — Help >
Search Actions and About use it, so they close on an outside click
(previously a buttonless modal could only be closed with Escape).
- Preferences general tab grows two live settings: the cache-ahead
pre-render window (PlaybackPreRenderFrames, consumed by the preview
scheduler every playback tick) and the library storage backend
(Storage/Backend sqlite/pg + Storage/PgUrl connection string, read by
oak-storage's write-through when a project binds; the pg URL field
shows only for PostgreSQL and commits on dialog close). The C++
auto-recovery toggle is skipped on purpose: the Rust snapshot loop
has no disable semantic, so the switch would be dead.
i18n keys for all eight packs; new tests cover the about modal and the
preferences writes.
The popup was a plain absolute child: dialog rows after the combo drew
OVER it, making the list look transparent (its text overlapped the
content behind). deferred() paints it after all ancestors.
The menu item was a placeholder print; it now opens a real dialog (the
C++ ProjectPropertiesDialog):
- Per-project OCIO config override with a 浏览… picker: validated on OK
(an invalid config keeps the dialog open with the error shown, like
the C++ accept()), persisted in the project settings, applied to the
display color pipeline on accept and on project open, and reverted to
the app default when the project closes. oak-render gains
set_up_default_config_from for the explicit-path load.
- Disk-cache location (default / alongside the project / custom path):
persisted through the OVE serializer (cachesetting/customcachepath
round-trip the settings map, clamped on load) and honored by the
thumbnail writer — the first live consumer of Project::cache_path.
- PathField gains an enabled state (the custom path field follows the
combo selection).
The C++ color tab's Default Input Color Space and Reference Space
combos are intentionally absent: the Rust render pipeline has no
consumer for them today (decode performs no input transfer conversion),
so showing them would be dead settings.
Tests: dialog opens, OK applies the cache location, an invalid OCIO
config keeps the dialog open with the error row. i18n keys for all
eight packs.
convert_bgra8 applied packed u8 pixels to the default (F32-finalized)
OCIO CPU processor, which rejects them with a bit-depth mismatch; the
callers swallow the error, so the display-ICC transform was silently
inert on the shm preview path. ocio-rs exposes no Uint8-finalized CPU
processor, so the conversion now detours through F32. Verified against
the machine's actual display profile with the new
display_icc_bgra8_never_outputs_black test (OAK_DISPLAY_ICC).
Also:
- procpool_integration: audio tickets are Seek priority and claimable
by any worker now, so the shard-spread assertion goes (rendering on a
live worker is what matters).
- OAK_DEBUG_VIEWER=1: the program viewer logs frame pushes and dumps
the displayed frame to /tmp/oak_viewer_frame.ppm (the black-screen
investigation tooling).
RealEngine::set_track_height held the project graph lock while calling
graphops::set_track_height, which locks the same mutex — an instant
same-thread deadlock with any sequence open (sampled live: main thread
in nudge_track_height -> set_track_height -> Mutex::lock, no other
lock-holder thread). Collect the track ids under the lock, drop it,
then apply the heights.
Also:
- Regression test set_track_height_does_not_self_deadlock.
- save_load_roundtrips_a_timeline_clip: real footage through the OVE
serializer (also the fixture generator for viewer debugging).
Three compounding bugs froze the UI when dragging the playhead after
playback:
1. Self-deadlock on preview_windows: supply_preview_window /
cancel_preview_windows / cancel_preview_window called
cancel_preview_sequence / cancel_preview_frame while HOLDING the
preview_windows mutex; those calls fire completions synchronously and
the completion locks preview_windows again. Caught by sampling the
hung process: UI thread in cancel_preview_sequence -> TicketSlot::
finish -> completion -> Mutex::lock. Cancels/releases are now
collected under the lock and fired after it is dropped.
2. Seek starvation by shard pinning: a Seek request's scheduler frame
is its ticket id, pinning it to worker (id mod W). The playback
window fills every worker's slots (window slots are only released by
UI-thread consumption), so the seek's pinned worker could have zero
free slots while the UI thread blocked on the seek — permanent
starvation. Seeks (interactive frame / real-time audio) are now
claimable by ANY worker; the no-stealing shard rule stays for
Playback frames (adjacent frames finish together).
3. No per-worker reserve: the global preview_window_capacity reserve is
pool-wide accounting, but exhaustion happens per worker. Playback /
Background claims now leave one credit unused per worker; Seek
claims may use the last slot (they complete on the worker without
UI involvement).
Also: RealEngine::drop cancels the preview windows — ShmFrameRef has no
self-release, so every dropped engine leaked its window's slots from
the shared pool, starving later windows (surfaced as the full-suite
playback_window_supplies_playhead_frames failure once the new probe
test shifted the test schedule). new_sequence_has_default_two_video_
two_audio_tracks now takes the engine test lock (it asserts on the
global undo stack; running lock-free raced parallel undo histories).
New regression probe interactive_seek_renders_without_hanging: play 30
ticks (window fills and holds shm slots), pause, seek, synchronously
render — must not hang. Scheduler tests updated for the reserve and
seek-any-worker contract. OAK_DEBUG_DISPATCH=1 enables the dispatcher
starvation/pool diagnostics used to track this down.
Mike-Solar