Skip to content

feat(handler) : add format handler for BTRFS stream #1451

Merged
qkaiser merged 2 commits intoonekey-sec:mainfrom
Corv0O:1372-btrfs-stream
Apr 15, 2026
Merged

feat(handler) : add format handler for BTRFS stream #1451
qkaiser merged 2 commits intoonekey-sec:mainfrom
Corv0O:1372-btrfs-stream

Conversation

@Corv0O
Copy link
Copy Markdown
Contributor

@Corv0O Corv0O commented Mar 25, 2026

#1372

A btrfs stream file is a binary format produced by btrfs send to transfer filesystem snapshots. It encodes filesystem operations as a sequence of TLV-based commands that btrfs receive replays to reconstruct the original snapshot.

Format

Stream header :

Magic : 62 74 72 66 73 2d 73 74 72 65 61 6d 00 (btrfs-stream\0)
Version : 1 or 2

After the stream header, the format consists of a sequence of commands replayed by btrfs receive. Each command has a fixed 10-byte header:

Command header :

Length of data : 4 bytes
Type of the command : 2 bytes
Crc32c : 4 bytes (use Castagnoli reflected polynomial, seed = 0x0, no final XOR)

Command payload (TLV attributes) :

Type : 2 bytes (PATH, UUID, ...)
Length : 2 bytes
Value : [length] bytes

The format has two modes:

  • Full sends: complete snapshot serialization with all the commands to recreate the file systems
  • Incremental sends: diff between two snapshots

This handler only handle Full send file

Sample :

setup

truncate -s 2G /tmp/btrfs.img && mkfs.btrfs /tmp/btrfs.img
sudo mount -o loop /tmp/btrfs.img /mnt/source
sudo btrfs subvolume create /mnt/source/vol
echo "hello" > /mnt/source/vol/hello.txt

full send

sudo btrfs subvolume snapshot -r /mnt/source/vol /mnt/source/snap
sudo btrfs send /mnt/source/snap -f sample.bin

you can verify the content of the btfrs stream with this command
btrfs receive --dump -f sample.bin

[References]

https://btrfs.readthedocs.io/en/latest/dev/dev-send-stream.html#btrfs-send-c-encoded-write-25
https://formats.kaitai.io/btrfs_stream/
https://archive.kernel.org/oldwiki/btrfs.wiki.kernel.org/index.php/Design_notes_on_Send/Receive.html

@qkaiser qkaiser self-requested a review March 25, 2026 09:47
@qkaiser qkaiser linked an issue Mar 25, 2026 that may be closed by this pull request
@qkaiser qkaiser added format:filesystem python Pull requests that update Python code labels Mar 25, 2026
@Corv0O Corv0O changed the title git feat(handler) : add format handler for BTRFS stream feat(handler) : add format handler for BTRFS stream Mar 25, 2026
@Corv0O Corv0O force-pushed the 1372-btrfs-stream branch from 49d95e3 to 3f84ef2 Compare March 25, 2026 10:17
@qkaiser qkaiser self-assigned this Mar 25, 2026
Comment thread python/unblob/file_utils.py
Comment thread python/unblob/file_utils.py
Comment thread python/unblob/file_utils.py
Comment thread python/unblob/file_utils.py
Comment thread python/unblob/file_utils.py
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
@Corv0O Corv0O force-pushed the 1372-btrfs-stream branch 2 times, most recently from fc491ff to 234f4d4 Compare March 26, 2026 13:21
Copy link
Copy Markdown
Contributor

@qkaiser qkaiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't forget unit tests. Add integration samples that exercise:

  • different types of compression
  • with prefix and suffix data

Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
@qkaiser
Copy link
Copy Markdown
Contributor

qkaiser commented Mar 27, 2026

@Corv0O don't forget to rebase on upstream main for CI checks to pass

@Corv0O Corv0O force-pushed the 1372-btrfs-stream branch from 234f4d4 to b1963e0 Compare March 30, 2026 11:48
Comment thread pyproject.toml Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py
@qkaiser qkaiser force-pushed the 1372-btrfs-stream branch from b1963e0 to dc37e20 Compare April 1, 2026 13:10
@Corv0O Corv0O force-pushed the 1372-btrfs-stream branch from dc37e20 to 1df9e4d Compare April 7, 2026 12:27
Copy link
Copy Markdown
Contributor

@qkaiser qkaiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few changes and we're good to go.

Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Outdated
@Corv0O Corv0O force-pushed the 1372-btrfs-stream branch from 1df9e4d to 2a48143 Compare April 14, 2026 14:57
@qkaiser qkaiser force-pushed the 1372-btrfs-stream branch from 2a48143 to 5bc4e1d Compare April 14, 2026 15:06
Comment thread python/unblob/handlers/filesystem/btrfs_stream.py Fixed
@Corv0O Corv0O force-pushed the 1372-btrfs-stream branch from 5bc4e1d to c3612c9 Compare April 15, 2026 10:28
Corv0O added 2 commits April 15, 2026 13:22
These functions are implemented by BTRFS stream and we want to make sure
they are properly implemented and secured against path traversal
payloads.
Instead of relying on an external CRC32C package, I implemented the algorithm directly in Python.
This manual implementation was necessary because existing libraries lacked the flexibility to support the specific requirement used by BTRFS stream.
The checksum uses the Castagnoli reflected polynomial with a seed of 0x0 and no final XOR operation.
crc32c and crcmod wasn't able to fullfill those requirements and calculate the right checksum.
@Corv0O Corv0O force-pushed the 1372-btrfs-stream branch from c3612c9 to 93056e7 Compare April 15, 2026 11:24
@qkaiser qkaiser added this pull request to the merge queue Apr 15, 2026
Merged via the queue into onekey-sec:main with commit 83e75fe Apr 15, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

format:filesystem python Pull requests that update Python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for BTRFS stream

3 participants