diff --git a/README.md b/README.md index 144627ec1..38c63bb84 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,11 @@ The binary can be downloaded here: See [`docs/build.md`](docs/build.md) for build instructions on Windows (MSYS2), Linux (Debian/Ubuntu, Fedora, Arch Linux), and macOS. +## Documentation + +- [Project Storage Architecture](docs/project-storage.md) ([中文](docs/zh/project-storage.md)) — database write-through persistence, node-granular journal, persistent undo +- [Build guide](docs/build.md) · [工程文件格式](docs/zh/project-file-reference.md) + ## Roadmap | Version | Theme | Core Deliverables | Boundary Notes | diff --git a/docs/README.md b/docs/README.md index eb6637999..c102b60e2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -30,4 +30,5 @@ Oak Video Editor is a renamed fork of Olive, focused on delivering a polished, c - Build from source on Windows, macOS, or Linux using the Build guide. - Learn how project data is stored in the Project File Reference. +- Understand the database write-through persistence in the [Project Storage Architecture](project-storage.md). - Keep releases solid by following the Test Plan. diff --git a/docs/project-storage.md b/docs/project-storage.md new file mode 100644 index 000000000..6eaeeca6a --- /dev/null +++ b/docs/project-storage.md @@ -0,0 +1,103 @@ +# Project Storage Architecture + +[中文](zh/project-storage.md) + +Oak persists projects in a **database** (SQLite by default, PostgreSQL +supported), with **write-through persistence**: every edit is committed +to the database as it happens — there is no Save button and nothing to +lose. The `.ove` XML file, OTIO and FCPXML are import/export formats, +not the working store. + +## Design principles + +- **Aggregate-granular persistence, not full relational mapping.** The + database stores four things: project metadata, key/value settings, + periodic full snapshots, and a per-node command journal. Domain + semantics (timeline structure, node connections, keyframes, effect + chains) stay inside each node's XML payload. A project is exactly + "the node graph plus settings" — nothing else — so node granularity + is a closed, complete model. +- **One serialization truth.** The node XML in the database is the same + document the `.ove` serializer produces (`oaknode::serializer`). + New features (e.g. adjustment layers) extend the XML schema only — + the database schema never changes. +- **The journal is produced by diffing, not by instrumenting commands.** + After every undoable command the in-memory project is re-serialized + and compared node-by-node with the previous state; changed/added/ + removed nodes each become one journal row. Existing and future + command types are covered automatically. +- **The journal is the persistent undo history.** Replaying applies + each node's newest image in command order; rewinding applies the + previous images in reverse. Undo survives restarts. + +## Schema (SQLite / PostgreSQL, sea-orm) + +```sql +projects(id PK, uuid UNIQUE, name, schema_ver, created_at, modified_at, command_seq) +settings(project_id FK, key, value, PK(project_id, key)) +snapshots(project_id FK, command_seq, payload, written_at, PK(project_id, command_seq)) +journal(project_id FK, seq, node_identity, kind, old_xml, new_xml, at, + PK(project_id, seq, node_identity)) +``` + +- `snapshots.payload` is the full project XML, written every + `Storage/SnapshotIntervalSec` (default 600) while dirty, pruned to + the newest 3. It only accelerates loading — the journal alone can + rebuild the project from scratch. +- `journal` rows are whole-node before/after images: `old_xml` is NULL + for created nodes, `new_xml` is NULL for removed nodes. Loading takes + the newest snapshot and replaces each node with its newest image in + `seq` order; rewinding to command `N` applies `old_xml` backwards. + `node_identity = 0` is the settings pseudo-node. +- Every command is one synchronous transaction (journal rows + + `command_seq + 1`), so a `kill -9` loses zero commands. +- `Storage/JournalRetentionDays` (default 0 = keep forever) bounds the + undo window; rows are kilobytes each. + +## Timeline representation + +The timeline is a chain of node references inside node XML: + +``` +sequence ────▶ tracklist ────▶ track ────▶ clip ────▶ footage +``` + +The clip row carries its timeline range (``), media +offset (``) and footage reference; effect chains are +connection records inside the effect nodes' XML. Loading re-links +these identities in two passes (see `oaknode::serializer`), so no +join tables are needed. Timeline edits map to a handful of node rows: +moving a clip touches its track and the clip; splitting adds one node +and updates two; ripple edits touch the affected tracks and delete the +removed clips. + +## Write-through flow + +1. A facade undo push (`oakengine_undo_push`, group end, undo/redo/ + jump) succeeds. +2. The project is re-serialized in memory (microseconds to low + milliseconds) and diffed against the last state. +3. One transaction writes the changed node rows, bumps `command_seq` + and `modified_at`. +4. A background thread writes snapshots latest-wins; on exit the queue + is flushed. + +## Import / export + +- Import: `.ove` / `.otio` / `.fcpxml` are parsed by their existing + oakstorage backends and inserted as a new project row with + `kind = 'import'` journal entries. +- Export: the in-memory serialization is written through the ove-xml + or otio backend; nothing is read from or written to the database + beyond the current state. + +## Multi-writer and platforms + +v1 assumes a single writer per database (SQLite `busy_timeout`, PG row +locks). Multi-writer collaboration is future work (M14). The default +database is a single user-level SQLite file; PostgreSQL is selected +with an `oakdb+pg://` URI. + +See also: [M10 oakstorage manual](plans/riir/M10-oakstorage.md), +[M13 write-through plan](plans/riir/M13-storage-live.md), +[project file reference](project-file-reference.md). diff --git a/docs/zh/README.md b/docs/zh/README.md index 8ba89f6a8..f1473fca1 100644 --- a/docs/zh/README.md +++ b/docs/zh/README.md @@ -30,6 +30,7 @@ Oak 视频编辑器是 Olive 的重命名分支,目标是打造更完善、更 - 按《构建指南》在 Windows/macOS/Linux 上从源码构建。 - 在《工程文件参考》了解项目数据结构。 +- 在《[工程存储架构](project-storage.md)》了解数据库写穿持久化与持久撤销历史。 - 按《测试计划》确保发布质量。 ## 下载 diff --git a/docs/zh/project-storage.md b/docs/zh/project-storage.md new file mode 100644 index 000000000..32f3e2011 --- /dev/null +++ b/docs/zh/project-storage.md @@ -0,0 +1,85 @@ +# 工程存储架构 + +[English](../project-storage.md) + +Oak 的工程持久化在**数据库**(默认 SQLite,支持 PostgreSQL),采用 +**写穿持久化**:每一次编辑都即时落库——没有保存按钮,也没有可丢失 +的数据。.ove XML 文件、OTIO、FCPXML 是导入/导出格式,不是日常存储。 + +## 设计原则 + +- **按聚合粒度持久化,不做全关系型映射。** 数据库只存四样东西: + 工程元信息、KV 设置、周期全量快照、节点粒度的命令日志。领域语义 + (时间线结构、节点连接、关键帧、效果链)全部留在每个节点的 XML + payload 里。工程 = 节点图 + settings,没有第三种东西——节点粒度 + 因此是封闭全集。 +- **单一序列化事实。** 库里的节点 XML 与 .ove 序列化器 + (`oaknode::serializer`)产出的是同一份文档。新功能(比如调整图层) + 只需要扩展 XML schema,数据库 schema 永远不变。 +- **journal 由 diff 产生,不靠命令申报。** 每条 undoable 命令成功后, + 在内存里重新序列化工程并与上一状态逐节点比对;变化/新增/删除的 + 节点各落一行。现有和未来的命令类型自动覆盖。 +- **journal 就是持久化撤销历史。** 回放按命令序应用各节点最新像; + 回退按逆序应用旧像。撤销跨会话存活。 + +## Schema(SQLite / PostgreSQL,sea-orm) + +```sql +projects(id PK, uuid UNIQUE, name, schema_ver, created_at, modified_at, command_seq) +settings(project_id FK, key, value, PK(project_id, key)) +snapshots(project_id FK, command_seq, payload, written_at, PK(project_id, command_seq)) +journal(project_id FK, seq, node_identity, kind, old_xml, new_xml, at, + PK(project_id, seq, node_identity)) +``` + +- `snapshots.payload` 是全工程 XML,脏状态下每 + `Storage/SnapshotIntervalSec`(默认 600 秒)写一份,只留最近 3 份。 + 它只是加载加速器——journal 自己就能从零重建工程。 +- `journal` 行是整节点前后像:新增节点 `old_xml` 为 NULL,删除节点 + `new_xml` 为 NULL。加载 = 取最新快照,其后按 `seq` 顺序把每个节点 + 替换为最新像;回退到命令 N = 逆序回写 `old_xml`。 + `node_identity = 0` 是 settings 伪节点。 +- 每条命令一个同步事务(journal 行 + `command_seq + 1`), + `kill -9` 丢 0 条命令。 +- `Storage/JournalRetentionDays`(默认 0 = 全保留)限定撤销窗口; + 每行仅 KB 级。 + +## 时间线的表示 + +时间线是节点 XML 内部的引用链: + +``` +sequence ────▶ tracklist ────▶ track ────▶ clip ────▶ footage +``` + +clip 行自带时间线区间(``)、媒体偏移(``) +和素材引用;效果链是效果节点 XML 里的连接记录。加载时两阶段重连 +这些 identity(见 `oaknode::serializer`),所以不需要任何连接表。 +时间线编辑映射为少数节点行:移动 clip 触及它的 track 和 clip 本身; +分割新增一个节点、更新两个;ripple 编辑触及受影响的 track 并删除 +被移除的 clip。 + +## 写穿流程 + +1. facade 的 undo 推送(`oakengine_undo_push`、group_end、 + undo/redo/jump)成功; +2. 工程在内存里重新序列化(微秒到低毫秒级)并与上一状态 diff; +3. 一个事务写入变化节点行、`command_seq + 1`、`modified_at`; +4. 后台线程 latest-wins 写快照;退出前排空队列。 + +## 导入 / 导出 + +- 导入:.ove / .otio / .fcpxml 由既有 oakstorage 后端解析后,以 + `kind = 'import'` 的 journal 行写为新工程行。 +- 导出:内存序列化经 ove-xml 或 otio 后端写出;除当前状态外不读写 + 数据库。 + +## 多写者与平台 + +v1 假设单写者(SQLite `busy_timeout`,PG 行锁);多写者协作是后续 +工作(M14)。默认数据库是用户级单一 SQLite 文件;PostgreSQL 用 +`oakdb+pg://` 连接串选择。 + +另见:[M10 oakstorage 手册](plans/riir/M10-oakstorage.md)、 +[M13 写穿计划](plans/riir/M13-storage-live.md)、 +[工程文件格式参考](project-file-reference.md)。