Support for xattrs, in TAR archives?

[vmlemon]

Thanks so much, for making ratarmount - it's solved a long-running need, in a really elegant manner. That said, are there any plans for supporting extended attributes, in TAR archives, since it'd make it easier, to work with files from Haiku, and MacOS, on Linux?

At least in the typical LIBARCHIVE/SCHILLY/BSDTAR-style format:

98 LIBARCHIVE.xattr.user.haiku.BEOS:TYPE#MIMS=YXBwbGljYXRpb24veC12bmQuaGFpa3Utc21iLXdvcmtncm91cAA
81 SCHILY.xattr.user.haiku.BEOS:TYPE#MIMS=application/x-vnd.haiku-smb-workgroup


Although IBM i/AIX, and Solaris seem to have PAX type-E records, of their own formats, and MPE/IX (example file, at https://www.openeloquence.com/support/B07/mpeclient2/tar.html) has records that look like, in "strings":

18 MPE.FILECODE=0
21 MPE.FILELIMIT=256
20 MPE.NUMEXTENTS=1
23 MPE.NUMUSERLABELS=0
22 MPE.BLOCKFACTOR=16
22 MPE.RECORDFORMAT=0
14 MPE.CCTL=0
15 MPE.ASCII=1
21 MPE.RECORDSIZE=80


I can provide example files, if it helps?

[mxmlnkn]

I'm glad ratarmount solved your need!

This seems like something that ratarmount should support in the future. Example files would be very welcome for testing!

This also seems to be relevant for the OCI format and the issue has interesting links for further reading.

[vmlemon]

No worries. borgbackup/borg#8479 has some IBM i, and Solaris ones, at least.

[vmlemon]

zos.tar.gz has an example, of a PAX archive, with extended attributes, from IBM z/OS.

[mxmlnkn]

I am currently looking into this whole topic, but as a short interjection: Both the MPE and the ZOS file are examples of vendor-specific extended pax header keywords. These are not filesystem extended attributes, such as those stored in LIBARCHIVE.xattr and SCHILY.xattr. Supporting LIBARCHIVE.xattr and SCHILY.xattr seems fine and useful to me, but I don't think it make much sense to support the MPE and ZOS pax keywords. It probably might even be considered wrong to expose them as extended file attributes, except maybe in a subnamespace in user.pax.ZOS and user.pax.MPE.

[mxmlnkn]

As I have never heard of extended attributes, I need to get an overview at first. These are just notes for myself.

File System / Operating System

Extended attributes seem to be foremost a simple key-value filesystem feature associated with files supported by virtually every filesystem including the ones I usually use such as FAT, EXT4, NTFS, and even NFS since 2020.
They are "extended" because they are in addition to predefined metadata such as the file time and permissions.

On the operating system side, access to extended attributes is e.g. exposed via libc API functions such as listxattr, getxattr, setxattr, and removexattr.
The API functions assume a null-terminated key string and a value with arbitrary data including possibly null bytes.
Normally, key strings contain dot-separated namespaces (namespace.attribute) such as user., trusted., system., security..
The namespace is enforced on Linux. During tests with setfattr, setting the key mytag and system.mytags fails with "Operation not supported" while setting security.mytags fails with Operation not permitted and works as root. Setting user.mytags works even as non-root.
Each filesystem and API also has different limitations on extended attributes.
In general, it cannot be assumed that much more than a few hundred bytes can be stored in extended attributes.

On the userspace side, there are tools such as getfattr and setfattr from the attr package.
By default, getfattr only lists user. attributes, but this can be overwritten with -m ''.
Tools such as cp with --archive and GNU tar with --xattrs also offer support for extended attributes.

Usage

My personal impression, as displayed by my lack of knowledge about this in the first place, is that extended attributes are rather rarely utilized.
A recursive search for user.-namespaced keys only yields roughly 100 matches on my notebook:

• Roughly half are timestamps for /var/log/journal/*/*/*.journal files, e.g., user.crtime_usec=0sqJvUUToxBgA=
• The other half are invocation IDs for /sys/fs/cgroup/**.{mount,socket,service}, e.g., user.invocation_id="cc7887111babcdef01cb098a88aef28c", for systemd control groups, which can also be shown with systemd-cgls --xattr=true. The special sysfs exposes the invocation IDs as extended user-namespaced attributes for some reason.
• Lastly, there is this single "file" /proc/2495/fd/7, which has user.invocation_id (as discussed above) and user.delegate="1" extended attributes. Process 2495 is /usr/lib/systemd/systemd --user --deserialize=8.

When looking for all extended attributes, not only user-namespaced keys, with getfattr -R -m '' --dump /, there are some additional usages:

• system.sockprotoname="UNIX-STREAM", system.posix_acl_access on some file descriptors in procfs
• system.posix_acl_default on a few /var/log/journal files.
• security.capability on /usr/bin/ping*
• system.nfs4_acl on NFS-mounted files.

Some usages from another system:

• security.selinux to store SELinux security context labels.
• user.Librepo.checksum.sha256 for /var/cache/dnf/ files as used by DNF via lr_checksum_fd_compare in librepo. (This is one usage I can get behind and probably something one could do with user scripts to add checksumming to filesystems that do not support it out of the box)

Not even the "notes" feature of the file browser I use, caja, is implemented with extended attributes as might have made sense.
Instead it uses the GNOME Virtual file system metadata daemon gvfsd-metadata, which stores its data in $XDG_DATA_HOME/gvfs-metadata, and which can be manipulated with gvfs-set-attribute and viewed with gvfs-info, or rather more modernly with gio set and gio info.

Many other ideal userspace usages such as author information or tags are more commonly stored in media files themselves, e.g., in PDF metadata, MP3 ID3 tags, EXIF metadata for images, and many more, probably because this makes it possible to exchange files with metadata via the internet, for which even a copy command line tool with extended attributes support does not help.
It seems that wget and curl support to store the origin/referrer URL in extended attributes and dolphin and baloo can store a rating.

File attributes seem to have more uses on MacOS, e.g., for filtering files for backups, and borg seems to be working on support.

TAR Support

Some nice overviews can be found here: [1] [2] [3].

The Unix Standard TAR (ustar) as defined in the POSIX standard (IEEE 1003.1) has no support for extended attributes.
The widely supported pax interchange format prepends a pax header block and pax extended header to each TAR header block, which is designed to be compatible to ustar, which would interpret them as a file record with a special name.

The pax extended header contains key-value pairs in the form <length> <keyword>=<value>\n, where the key must not contain = and key and value are mostly encoded using UTF-8.

Keywords consisting entirely of lowercase letters, digits, and periods are reserved for future standardization.

The extended header records shall be encoded according to the ISO/IEC 10646-1:2000 standard (UTF-8).

Unlike the ustar header block fields, NULs shall not delimit s; all characters within the field shall be considered data for the field. None

An empty value will delete a previously set key.
Vendor-specific keys should have the form VENDOR.keyword.
Use cases are nanosecond timestamps and arbitrarily long paths and names.
Unfortunately, GNU tar has by default a separate file format using ././@LongLink named special files to implement long names, probably because it precedes the POSIX pax standardization.
This can be changed with tar --format=pax.

I find this language regarding encoding a bit confusing because it tries to enable binary data of any kind including NUL bytes but still specifies it to be encoded as UTF-8.
It seems that even such binary data should be converted to UTF-8, possibly leading to some bloat for byte values >127, which will require two UTF-8 bytes, but so be it.
Unfortunately, GNU tar did not implement this correctly in the past.
In the end, this should not matter for ratarmount, but for CPython's tarfile, which is used.
The comments look pretty helpful and care seems to have been taken to be compatible.
As mentioned there, POSIX 2008 seems to have added binary encoding support for extended pax header records.
CPython's tarfile itself does not (need to) have specific xattr support but makes all parsed pax headers available via an interface.
This also includes pax headers such as precise timestamps that have already been parsed into ofer TarInfo members and therefore seem somewhat unnecessary.
Although, timestamps in particular seem to have the problem of being parsed to double and therefore losing the last two or so digits of nanosecond precision, therefore reparsing the pax headers for those might actually be useful!

Note that pax adds another key-value store in addition to the filesystem's extended attributes.
It is important to differentiate between the two.
The posted MPE eloqtar.tar and zos.tar.gz archives contain extended pax header keywords in the form of VENDOR.KEYWORD.
These are not filesystem extended attributes, as far as I understand.
I don't see the value to support these in ratarmount.

And MPE seems to be discontinued.
Trying to search for these keys basically only brings up this blog post from 12 years ago, which basically asserts these to be "not externally documented", and an issue on libarchive.
There also is this second-hand excerpt from the MPEiX tar man page, which I was not able to find in its original.
Some filecodes can be found here but I honestly wouldn't know what to do with those.

Unfortunately, there are no standardized keys for extended file attributes.
But, the SCHILY.xattr and LIBARCHIVE.xattr keys seem to be quasi-standard and for example supported by tar2sqfs

star

Schily Tar, formerly maintained by Jörg Schilling and now somewhat inactive here introduced the SCHILY.xattr pax keyword.
According to comments on the Apache Commons compress TAR package, there are encoding issues, i.e., extended file attribute keys and values are not checked for validity with pax encoding specification, while libarchive does proper escaping and encoding for both.
For that reason, LIBARCHIVE.xattr seems to be preferable if it is available.

libarchive / bsdtar

Contrary to what is stated here at the bottom, but correctly stated here, bsdtar 3.7.2 adds, both, LIBARCHIVE.xattr and SCHILY.xattr to the TAR, probably for compatibility.

echo bar > foo
setfattr --name user.tags --value mytag foo
bsdtar --create --file result.tar bar
dd bs=512 if=file-with-attribute.bsd.tar count=1 skip=1 2>/dev/null | tr -d '\000'
# 28 mtime=1748164058.1061919
# 38 LIBARCHIVE.xattr.user.tags=bXl0YWc
# 32 SCHILY.xattr.user.tags=mytag
python3 -c 'import tarfile; print(next(iter(tarfile.open("file-with-attribute.bsd.tar"))).pax_headers)'
# {'mtime': '1748164058.1061919', 'LIBARCHIVE.xattr.user.tags': 'bXl0YWc', 'SCHILY.xattr.user.tags': 'mytag'}

As mentioned for the shortcomings of star, libarchive is encoded as such:

Libarchive stores POSIX.1e-style extended attributes using keys
of this form. The key value is URL-encoded: All non-ASCII
characters and the two special characters "=" and "%" are en-
coded as "%" followed by two uppercase hexadecimal digits. The
value of this key is the extended attribute value encoded in
base 64. XXX Detail the base-64 format here XXX

As already the case in the example above, the base64 seems to strip the trailing padding characters.
Checks against that need to be silenced when decoding the value.

GNU Tar

Similarly to bsdtar, --xattrs flags need to be specified, and even --selinux or better --xattrs-include=* to archive non-user. keys.

Implementation

On the libfuse side, only listxattr and getxattr need to be implemented.
Fusepy also supports these two callbacks.
On the ratarmount side, I only need to collect the *.xattr pax keys and store them somewhere.
As there can be multiple key-values to the same file, it probably makes sense to store these in a separate SQLite table, to avoid having to add a new column that stores something like JSON or another hard-to-parse string.
Each row needs to somehow be linked to the corresponding file.
Unfortunately, the offset and offsetheader columns, which sometimes are usable as unique keys may have duplicates because of the implicitly created parent folders in _tryAddParentFolders.
We wouldn't want to file attributes to also apply to the parent folders.
But I guess it would be possible to test for "are attributes requested for a folder and does a file for it exist at the same offset, then do not return any attributes?".

The table for file attributes also brings up another issue: the keys will only be out of a very limited set of strings and may be repeated for each file.
I would have thought that SQLite can deduplicate such use cases, e.g., by assigning each string a unique ID, as is basically done for compile-time const char* in C/C++.
But a quick test shows that this is not the case:

import sqlite3
import os

for use_string in [True, False]:
    db_path = ("string" if use_string else "int") + "_db.sqlite"

    connection = sqlite3.connect(db_path)
    c = connection.cursor()
    c.execute(f"CREATE TABLE data (value {'VARCHAR(65535)' if use_string else 'INTEGER'})")
    c.executemany(
        "INSERT INTO data (value) VALUES (?)",
        [("some_string " * 10 if use_string else i,) for i in range(100_000)],
    )
    connection.commit()
    connection.close()

    print(f"{db_path} size: {os.path.getsize(db_path) // 1024} KiB")
    os.remove(db_path)

# string_db.sqlite size: 12944 KiB
# int_db.sqlite size   : 1040 KiB

Dictionary deduplication could be easily done with another table, but it would also be very nice to implement this for the path/name columns in the existing files table, especially because (folder) paths will almost always be repeated.
To implement it for that existing table, it would have to work transparently to be downwards compatible, i.e., do not require explicitly querying the dictionary table.
This seems to be possible with foreign keys and views, but I'll need to try it out and benchmark the overhead of this.

[mxmlnkn]

This seems to be possible with foreign keys and views, but I'll need to try it out and benchmark the overhead of this.

I got a bit distracted benchmarking these even though there are backwards compatibility issues.

I have pushed support for extended attributes to the xattrs branch. You can try it out with:

python3 -m pip install --user --force-reinstall \
    'git+https://github.com/mxmlnkn/ratarmount.git@xattrs#egginfo=ratarmountcore&subdirectory=core' \
    'git+https://github.com/mxmlnkn/ratarmount.git@xattrs#egginfo=ratarmount'

Please send feedback if you experience any problems.

For the libarchive backend, I still need support upstream, but as extended attributes seem to mostly only be available for TAR and because libarchive is only used as a fallback for TAR by ratarmount, the missing python-libarchive-c bindings extended attributes support shouldn't be a problem.

[vmlemon]

Thanks!

Looks like I can see the extended attributes, on the BSD test archive, that you checked in, but not on the GNU one, under MacOS, from a quick test:

(RTM) tyson@Tysons-MacBook-Air ~ % mkdir Test1
(RTM) tyson@Tysons-MacBook-Air ~ % ratarmount Downloads/file-with-attribute.gnu.tar.bz2 Test1 
Creating offset dictionary for /Users/tyson/Downloads/file-with-attribute.gnu.tar.bz2 ...
Creating offset dictionary for /Users/tyson/Downloads/file-with-attribute.gnu.tar.bz2 took 0.00s
[Info] The index does not yet contain bz2 block offset data. Will write it out.
Writing out TAR index to /Users/tyson/Downloads/file-with-attribute.gnu.tar.bz2.index.sqlite took 0s and is sized 32768 B
Created mount point at: /Users/tyson/Test1
(RTM) tyson@Tysons-MacBook-Air ~ % xattr -l Test1/*
xattr: [Errno 13] Permission denied: 'Test1/foo'
(RTM) tyson@Tysons-MacBook-Air ~ % xattr -l Test1/ 
com.apple.FinderInfo: 
(RTM) tyson@Tysons-MacBook-Air ~ % xattr -l Test1/*
xattr: [Errno 13] Permission denied: 'Test1/foo'
(RTM) tyson@Tysons-MacBook-Air ~ % ls Test1 
foo
(RTM) tyson@Tysons-MacBook-Air ~ % ls -la Test1
total 512
drwxrwxrwx@   1 tyson  staff     0 May 29 01:58 .
drwxr-xr-x+ 120 tyson  staff  3840 May 29 01:57 ..
-rwx------    1 1000   1000      4 May 25 10:07 foo
(RTM) tyson@Tysons-MacBook-Air ~ % umount Test1 
(RTM) tyson@Tysons-MacBook-Air ~ % sudo ratarmount Downloads/file-with-attribute.gnu.tar.bz2 Test1
Password:
Successfully loaded offset dictionary from /Users/tyson/Downloads/file-with-attribute.gnu.tar.bz2.index.sqlite
Created mount point at: /Users/tyson/Test1
(RTM) tyson@Tysons-MacBook-Air ~ % ls Test1
ls: Test1: No such file or directory
(RTM) tyson@Tysons-MacBook-Air ~ % sudo ls Test1
foo
(RTM) tyson@Tysons-MacBook-Air ~ % sudo xattr -l Test1
com.apple.FinderInfo: 
(RTM) tyson@Tysons-MacBook-Air ~ % sudo xattr -l Test1/*
zsh: no matches found: Test1/*
(RTM) tyson@Tysons-MacBook-Air ~ % sudo xattr -l Test1/foo
(RTM) tyson@Tysons-MacBook-Air ~ % sudo ls -la Test1/foo
-rwx------  1 1000  1000  4 May 25 10:07 Test1/foo
(RTM) tyson@Tysons-MacBook-Air ~ % umount Test1         
umount: Test1: not currently mounted
(RTM) tyson@Tysons-MacBook-Air ~ % sudo umount Test1
(RTM) tyson@Tysons-MacBook-Air ~ % sudo ratarmount Downloads/file-with-attribute.bsd.tar.bz2 Test1 
Creating offset dictionary for /Users/tyson/Downloads/file-with-attribute.bsd.tar.bz2 ...
Creating offset dictionary for /Users/tyson/Downloads/file-with-attribute.bsd.tar.bz2 took 0.00s
[Info] The index does not yet contain bz2 block offset data. Will write it out.
Writing out TAR index to /Users/tyson/Downloads/file-with-attribute.bsd.tar.bz2.index.sqlite took 0s and is sized 49152 B
Created mount point at: /Users/tyson/Test1
(RTM) tyson@Tysons-MacBook-Air ~ % sudo ls -la Test1/foo                                     
-rwx------@ 1 1000  1000  4 May 25 10:07 Test1/foo
(RTM) tyson@Tysons-MacBook-Air ~ % sudo xattr -l Test1/foo
user.tags: mytag

Looks like creating a directory, with Finder Labels, and Comments, TAR-ing it up, and mounting it, also works:

(RTM) tyson@Tysons-MacBook-Air ~ % tar cvf BlueBerries.tar BlueBerries
a BlueBerries
(RTM) tyson@Tysons-MacBook-Air ~ % strings BlueBerries.tar| less
(RTM) tyson@Tysons-MacBook-Air ~ % xattr -l BlueBerries     
com.apple.FinderInfo: 
com.apple.metadata:_kMDItemUserTags: bplist00?VBlue
4

com.apple.metadata:kMDItemFinderComment: bplist00_This is a Finder Comment

(RTM) tyson@Tysons-MacBook-Air ~ % sudo ratarmount BlueBerries.tar Test1
Creating offset dictionary for /Users/tyson/BlueBerries.tar ...
Creating offset dictionary for /Users/tyson/BlueBerries.tar took 0.00s
Created mount point at: /Users/tyson/Test1
(RTM) tyson@Tysons-MacBook-Air ~ % xattr -l Test1
xattr: [Errno 2] No such file or directory: 'Test1'
(RTM) tyson@Tysons-MacBook-Air ~ % sudo xattr -l Test1
com.apple.FinderInfo: 
(RTM) tyson@Tysons-MacBook-Air ~ % sudo xattr -l Test1/*
zsh: no matches found: Test1/*
(RTM) tyson@Tysons-MacBook-Air ~ % sudo xattr -l Test1/BlueBerries
com.apple.metadata:kMDItemFinderComment: bplist00_This is a Finder Comment
com.apple.metadata:_kMDItemUserTags: bplist00?VBlue
4

com.apple.FinderInfo: 

Attaching that archive, for testing purposes:
BlueBerries.tar.gz

Looks like the TAR, shipped in MacOS 15.5 is:

bsdtar 3.5.3 - libarchive 3.7.4 zlib/1.2.12 liblzma/5.4.3 bz2lib/1.0.8 

Seems that it writes a combination, of SCHILY, and LIBARCHIVE stanzas:

153 LIBARCHIVE.xattr.com.apple.metadata:kMDItemFinderComment=YnBsaXN0MDBfEBhUaGlzIGlzIGEgRmluZGVyIENvbW1lbnQIAAAAAAAAAQEAAAAAAAAAAQAAAAAAAAAAAAAAAAAAACM
126 SCHILY.xattr.com.apple.metadata:kMDItemFinderComment=bplist00_
This is a Finder Comment
126 LIBARCHIVE.xattr.com.apple.metadata:_kMDItemUserTags=YnBsaXN0MDChAVZCbHVlCjQICgAAAAAAAAEBAAAAAAAAAAIAAAAAAAAAAAAAAAAAAAAR
105 SCHILY.xattr.com.apple.metadata:_kMDItemUserTags=bplist00
VBlue
85 LIBARCHIVE.xattr.com.apple.FinderInfo=AAAAAAAAAAAACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
70 SCHILY.xattr.com.apple.FinderInfo=

Haven't gotten around, to testing it, with other TAR implementations, so far, though...

[mxmlnkn]

Note that you don't need to create the mount point folder beforehand, ratarmount does that for you and also cleans it up after unmount if it was created by ratarmount. I always found it cumbersome that other mount tools require an existing folder.

I found one bug while testing manually again, but it only occurs on ls -la and everything else, including getfattr --match=.* --dump -R mounted works for me.

Uncaught exception from FUSE operation getxattr, returning errno.EINVAL.
Traceback (most recent call last):
  File "/projects/mfusepy/mfusepy.py", line 1249, in _wrapper
    return func(*args, **kwargs) or 0
           ^^^^^^^^^^^^^^^^^^^^^
  File "/projects/mfusepy/mfusepy.py", line 1448, in getxattr
    ret = self.operations('getxattr', path.decode(self.encoding), name.decode(self.encoding), *args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/mfusepy/mfusepy.py", line 1731, in __call__
    return getattr(self, op)(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/ratarmount/worktrees/1/ratarmount.py", line 959, in getxattr
    value = self.mountSource.getxattr(self._getFileInfo(path), name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/ratarmount/worktrees/1/core/ratarmountcore/FileVersionLayer.py", line 242, in getxattr
    return self.mountSource.getxattr(fileInfo, key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/ratarmount/worktrees/1/core/ratarmountcore/SQLiteIndexMountSource.py", line 56, in getxattr
    return self.index.getxattr(fileInfo, key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/projects/ratarmount/worktrees/1/core/ratarmountcore/SQLiteIndex.py", line 975, in getxattr
    return row[0]
           ~~~^^^
TypeError: 'NoneType' object is not subscriptable

I pushed a fix for that but I doubt that it improves anything for you.

You can run ratarmount with -f in foreground mode to see errors after the mount point has been created. And with -d 3 you can turn up debug output. As I can't reproduce a problem with the *.gnu.tar, could you please try if you see anything helpful with this.

If you want to go a step further to help and there is no helpful output with -f -d 3, then you could try adding debug output, e.g., for tarInfo.pax_headers in these lines:

ratarmount/core/ratarmountcore/SQLiteIndexedTar.py

Lines 207 to 221 in f7201ea

	prefix = 'SCHILY.xattr.'
	xattrs = {
	urllib.parse.unquote(key[len(prefix) :]): value.encode('utf-8', errors='surrogateescape')
	for key, value in tarInfo.pax_headers.items()
	if key.startswith(prefix)
	}
	prefix = 'LIBARCHIVE.xattr.'
	xattrs.update(
	{
	urllib.parse.unquote(key[len(prefix) :]): decodeUnpaddedBase64(value)
	for key, value in tarInfo.pax_headers.items()
	if key.startswith(prefix)
	}
	)
	xattrRows = [(offsetHeader, key, value) for key, value in xattrs.items()]

I already see one copy-paste error in these lines. The urllib.parse.unquote is wrong for the GNU tar, but this also shouldn't result in your problem. I would only expect problems if there were % symbols in the key. I have pushed a fix for that.

It is also unexpected that you get permission denied errors, probably because of the mismatching user ID, but even after changing the TAR to a wrong user ID on my system, I do not get permission denied errors. I think that your FUSE implementation (osxfuse) has permission checking on by default while libfuse does not?

You may be able to work around that without sudo with ratarmount -o allow_other or with -o uid=$UID or -o uid=$UID,gid=$( id -g $USER ).

[vmlemon]

FWIW, https://www.ibm.com/docs/en/zos/2.5.0?topic=descriptions-pax-interchange-portable-archives, and https://www.ibm.com/docs/en/zos/2.4.0?topic=descriptions-extattr-set-reset-display-extended-attributes-files has some documentation, on the z/OS attributes, and ACLs.

I don't currently have access to a z/OS machine, and last time that I checked, only "pax" would preserve them, with some specific arguments, although I probably have some larger archives, containing them.

Solaris, and AIX/IBM i also had their own mechanisms, for defining xattrs in TAR/pax archives, with "E"-type records, if I remember correctly, and there's some other serialisations, for RISC OS, although it's up to you, as to if it's worth supporting those. (https://www.ibm.com/docs/en/i/7.5.0?topic=concepts-extended-attributes mentions the most common ones, for IBM i, and AIX uses the same serialisation, from what I can tell, for compatibility).

The current solution, at least covers the common modern MacOS/Linux/Haiku/FreeBSD/NetBSD bases, nicely, though. :)

[vmlemon]

Just had a play with BSDTAR, on Haiku, and it looks like I have an archive, with a "Person" file (which is made entirely, of xattrs), with attributes, that don't appear, even though the structure looks otherwise well-formed:

Haiku-Person.tar.gz

root@Tysons-MacBook-Air ~ # ratarmount Haiku-Person.tar HP1
Creating offset dictionary for /Users/tyson/Haiku-Person.tar ...
Creating offset dictionary for /Users/tyson/Haiku-Person.tar took 0.00s
Created mount point at: /Users/tyson/HP1
root@Tysons-MacBook-Air ~ # xattr -l HP1 
com.apple.FinderInfo: 
root@Tysons-MacBook-Air ~ # xattr -l HP1/*
root@Tysons-MacBook-Air ~ # ls -la HP1 
total 0
drwxrwxrwx@   1 root   wheel     0 May 29 21:16 .
drwxr-xr-x+ 130 tyson  staff  4160 May 29 21:16 ..
-rw-r--r--    1 root   wheel     0 Jul  2  1908 Name

strings says this, about the xattrs:

63 LIBARCHIVE.xattr.user.haiku.META:nickname#CSTR=Tmlja25hbWUA
56 SCHILY.xattr.user.haiku.META:nickname#CSTR=Nickname
56 LIBARCHIVE.xattr.user.haiku.META:group#CSTR=R3JvdXAA
50 SCHILY.xattr.user.haiku.META:group#CSTR=Group
52 LIBARCHIVE.xattr.user.haiku.META:url#CSTR=VVJMAA
46 SCHILY.xattr.user.haiku.META:url#CSTR=URL
52 LIBARCHIVE.xattr.user.haiku.META:fax#CSTR=RmF4AA
46 SCHILY.xattr.user.haiku.META:fax#CSTR=Fax
64 LIBARCHIVE.xattr.user.haiku.META:wphone#CSTR=V29yayBwaG9uZQA
56 SCHILY.xattr.user.haiku.META:wphone#CSTR=Work phone
67 LIBARCHIVE.xattr.user.haiku.META:mphone#CSTR=TW9iaWxlIHBob25lAA
58 SCHILY.xattr.user.haiku.META:mphone#CSTR=Mobile phone
64 LIBARCHIVE.xattr.user.haiku.META:hphone#CSTR=SG9tZSBwaG9uZQA
56 SCHILY.xattr.user.haiku.META:hphone#CSTR=Home phone
58 LIBARCHIVE.xattr.user.haiku.META:email#CSTR=RS1tYWlsAA
51 SCHILY.xattr.user.haiku.META:email#CSTR=E-mail
61 LIBARCHIVE.xattr.user.haiku.META:country#CSTR=Q291bnRyeQA
54 SCHILY.xattr.user.haiku.META:country#CSTR=Country
52 LIBARCHIVE.xattr.user.haiku.META:zip#CSTR=WmlwAA
46 SCHILY.xattr.user.haiku.META:zip#CSTR=Zip
56 LIBARCHIVE.xattr.user.haiku.META:state#CSTR=U3RhdGUA
50 SCHILY.xattr.user.haiku.META:state#CSTR=State
54 LIBARCHIVE.xattr.user.haiku.META:city#CSTR=Q2l0eQA
48 SCHILY.xattr.user.haiku.META:city#CSTR=City
61 LIBARCHIVE.xattr.user.haiku.META:address#CSTR=QWRkcmVzcwA
54 SCHILY.xattr.user.haiku.META:address#CSTR=Address
61 LIBARCHIVE.xattr.user.haiku.META:company#CSTR=Q29tcGFueQA
54 SCHILY.xattr.user.haiku.META:company#CSTR=Company
54 LIBARCHIVE.xattr.user.haiku.META:name#CSTR=TmFtZQA
48 SCHILY.xattr.user.haiku.META:name#CSTR=Name
75 LIBARCHIVE.xattr.user.haiku.BEOS:TYPE#MIMS=YXBwbGljYXRpb24veC1wZXJzb24A
64 SCHILY.xattr.user.haiku.BEOS:TYPE#MIMS=application/x-person

[mxmlnkn]

Weird, works for me:

> python3 ratarmount.py -c -f -d 3 Haiku-Person.tar mounted
> getfattr --match=.* --dump -R mounted
# file: mounted/Name
user.haiku.BEOS:TYPE#MIMS="application/x-person"
user.haiku.META:address#CSTR="Address"
user.haiku.META:city#CSTR="City"
user.haiku.META:company#CSTR="Company"
user.haiku.META:country#CSTR="Country"
user.haiku.META:email#CSTR="E-mail"
user.haiku.META:fax#CSTR="Fax"
user.haiku.META:group#CSTR="Group"
user.haiku.META:hphone#CSTR="Home phone"
user.haiku.META:mphone#CSTR="Mobile phone"
user.haiku.META:name#CSTR="Name"
user.haiku.META:nickname#CSTR="Nickname"
user.haiku.META:state#CSTR="State"
user.haiku.META:url#CSTR="URL"
user.haiku.META:wphone#CSTR="Work phone"
user.haiku.META:zip#CSTR="Zip"

Could you please try with -f -d 3?

And with -c. This forces recreating the index, else it may happen that it loads an old index that does not yet contain the attributes.

[vmlemon]

Probably not very useful, but I get this, and nothing else, even if I run xattr -lxr H1/Name, or xattr -lxr H1, to try recursively listing the mount point, and its contents, in hex format:

root@Tysons-MacBook-Air ~ # ratarmount -f -d 3 Haiku-Person.tar.gz H1 
[Info] Try to open with rarfile
[Info] Try to open with tarfile
[Info] Detected compression gz for file object: <_io.BufferedReader name='/Users/tyson/Haiku-Person.tar.gz'>
[Info] Do not reopen with rapidgzip backend because:
[Info]  - is too small to qualify for parallel decompression.
[Info] Undid gz file compression by using: IndexedGzipFile
[Info] Because of the given positive index minimum file count (1000) and because no explicit index file path is given, try to open an SQLite database in memory first.
[Info] Create new index in memory because memory is to be preferred, e.g., for small archives.
Creating offset dictionary for /Users/tyson/Haiku-Person.tar.gz ...
Resorting files by path ...
Creating offset dictionary for /Users/tyson/Haiku-Person.tar.gz took 0.00s
[Info] Will try to reopen the database on disk even though the file count threshold (1000) might not be exceeded (1) because the archive is compressed.
[Info] Try to reopen SQLite database on disk at: :memory:
other index paths: ['/Users/tyson/Haiku-Person.tar.gz.index.sqlite', '/Users/tyson/.cache/ratarmount/_Users_tyson_Haiku-Person.tar.gz.index.sqlite', '/Users/tyson/.ratarmount/_Users_tyson_Haiku-Person.tar.gz.index.sqlite']
[Info] Back up database from :memory: -> /Users/tyson/Haiku-Person.tar.gz.index.sqlite
[Info] The index does not yet contain gzip block offset data. Will write it out.
Writing out TAR index to /Users/tyson/Haiku-Person.tar.gz.index.sqlite took 0s and is sized 32768 B
[Info] Opened archive with tarfile backend.
Created mount point at: /Users/tyson/H1

The "Name" file appears without any extended attributes.

Interesting, to see that you can get them to appear, under Linux, so I wonder if MacOS quietly chokes on the hash characters, in the attribute names? (Haiku/BFS uses them, internally, to denote attribute types, as its attributes are technically not just HashMap/key-value pairs, but exposes them, in a flattened, Linux-style method, via a compatible API, which BSDTAR consumes, when archiving files, on this platform. However, GNU TAR drops them on the floor, since the extended attributes API isn't plumbed in, on Haiku).

[vmlemon]

Same deal, with -c:

root@Tysons-MacBook-Air ~ # ratarmount -f -d 3 -c Haiku-Person.tar.gz H1
[Info] Try to open with rarfile
[Info] Try to open with tarfile
[Info] Detected compression gz for file object: <_io.BufferedReader name='/Users/tyson/Haiku-Person.tar.gz'>
[Info] Do not reopen with rapidgzip backend because:
[Info]  - is too small to qualify for parallel decompression.
[Info] Undid gz file compression by using: IndexedGzipFile
[Info] Because of the given positive index minimum file count (1000) and because no explicit index file path is given, try to open an SQLite database in memory first.
[Info] Create new index in memory because memory is to be preferred, e.g., for small archives.
Creating offset dictionary for /Users/tyson/Haiku-Person.tar.gz ...
Resorting files by path ...
Creating offset dictionary for /Users/tyson/Haiku-Person.tar.gz took 0.00s
[Info] Will try to reopen the database on disk even though the file count threshold (1000) might not be exceeded (1) because the archive is compressed.
[Info] Try to reopen SQLite database on disk at: :memory:
other index paths: ['/Users/tyson/Haiku-Person.tar.gz.index.sqlite', '/Users/tyson/.cache/ratarmount/_Users_tyson_Haiku-Person.tar.gz.index.sqlite', '/Users/tyson/.ratarmount/_Users_tyson_Haiku-Person.tar.gz.index.sqlite']
[Info] Back up database from :memory: -> /Users/tyson/Haiku-Person.tar.gz.index.sqlite
[Info] The index does not yet contain gzip block offset data. Will write it out.
Writing out TAR index to /Users/tyson/Haiku-Person.tar.gz.index.sqlite took 0s and is sized 32768 B
[Info] Opened archive with tarfile backend.
Created mount point at: /Users/tyson/H1

root@Tysons-MacBook-Air ~ # xattr -lxr H1/Name
root@Tysons-MacBook-Air ~ # xattr -lxr H1     
H1: com.apple.FinderInfo:
00000000  00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000020
root@Tysons-MacBook-Air ~ # ls -la H1    
total 0
drwxrwxrwx@   1 root   wheel     0 May 29 23:12 .
drwxr-xr-x+ 116 tyson  staff  3712 May 29 23:11 ..
-rw-r--r--    1 root   wheel     0 Jul  2  1908 Name

[vmlemon]

And, that's bizarre - it doesn't seem that colons, or hashes make a difference, if I use them in the names, with the xattr utility, on a test file:

root@Tysons-MacBook-Air ~ # xattr -w "Test:1" "Value" AttribTest 
root@Tysons-MacBook-Air ~ # xattr -w "Test#1" "Value" AttribTest
root@Tysons-MacBook-Air ~ # xattr -l AttribTest 
Test#1: Value
Test:1: Value