Shrink imagesize and improve read performance by changing mksquashfs syntax
Closed, ResolvedPublicFEATURE REQUEST
Actions

Assigned To

Authored By

	Apachez
	Aug 26 2023, 2:06 AM

Description

Suggestion to modify the syntax which the filesystem.squashfs is being created with.

Using VyOS 1.4-rolling-202308250021 in below example.

Original filesystem.squashfs is: 376 418 304 bytes

Extracted size: 1 361 427 831 bytes.

Compressed size using below syntax: 357 666 816 bytes

mksquashfs squashfs-root/ filesystem.squashfs -no-recovery -always-use-fragments -b 256k -no-duplicates -comp xz -Xbcj x86

The above saves 18 751 488 bytes (~5%) and increase read performance (estimated by 10-15% according to https://jonathancarter.org/2015/04/06/squashfs-performance-testing/ - also verified with other systems utilizing squashfs).

The penalty is estimated 10-15% longer time to create (compress) the filesystem.squashfs file .

For ARM-based systems the following syntax should be used (alter Xbcj):

mksquashfs squashfs-root/ filesystem.squashfs -no-recovery -always-use-fragments -b 256k -no-duplicates -comp xz -Xbcj arm

Original filesystem.squashfs info:

# unsquashfs -s ./filesystem.squashfs 
Found a valid SQUASHFS 4:0 superblock on ./filesystem.squashfs.
Creation or last append time Fri Aug 25 02:22:28 2023
Filesystem size 376417488 bytes (367595.20 Kbytes / 358.98 Mbytes)
Compression xz
Block size 131072
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Uids/Gids (Id table) are compressed
Fragments are compressed
Always-use-fragments option is not specified
Xattrs are compressed
Duplicates are removed
Number of fragments 2621
Number of inodes 69488
Number of ids 63
Number of xattr ids 8

Modified filesystem.squashfs info:

# unsquashfs -s ./filesystem.squashfs 
Found a valid SQUASHFS 4:0 superblock on ./filesystem.squashfs.
Creation or last append time Sat Aug 26 03:45:28 2023
Filesystem size 357666467 bytes (349283.66 Kbytes / 341.10 Mbytes)
Compression xz
	Dictionary size 262144
	Filters selected: x86
Block size 262144
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Uids/Gids (Id table) are compressed
Fragments are compressed
Always-use-fragments option is specified
Xattrs are compressed
Duplicates are not removed
Number of fragments 2021
Number of inodes 69488
Number of ids 63
Number of xattr ids 8

Details

Difficulty level: Unknown (require assessment)
Version: -
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Unspecified (possibly destroys the router)
Issue type: Improvement (missing useful functionality)

Event Timeline

Apachez created this task.Aug 26 2023, 2:06 AM

pasik added a subscriber: pasik.Aug 26 2023, 7:18 AM

Some tests on filesystem.squashfs from VyOS 1.4-rolling-202308280021.

The below results are non-scientific (as in they have not been runned multiple times to find an average) but could get a hint of the differences.

mksquashfs from squashfs-tools 4.5.1 have been used on an Intel i5-4250U CPU with 16GB of RAM.

The original filesystem.squashfs is created with -comp xz and default blocksize which is equal to -b 128k:

Filesize on disk: 375 422 976 bytes.

# unsquashfs -s ./filesystem.squashfs 
Found a valid SQUASHFS 4:0 superblock on ./filesystem.squashfs.
Creation or last append time Mon Aug 28 02:23:31 2023
Filesystem size 375420131 bytes (366621.22 Kbytes / 358.03 Mbytes)
Compression xz
Block size 131072
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Uids/Gids (Id table) are compressed
Fragments are compressed
Always-use-fragments option is not specified
Xattrs are compressed
Duplicates are removed
Number of fragments 2618
Number of inodes 69516
Number of ids 62
Number of xattr ids 8

First test is to create a squashfs-file using -comp xz but with -b 256k as blocksize and -Xbcj x86 as optimization:

time mksquashfs squashfs-root/ filesystem256k_xz.squashfs -no-recovery -always-use-fragments -b 256k -no-duplicates -comp xz -Xbcj x86

real	6m21,334s
user	24m11,407s
sys	0m3,393s

The result:

Filesize on disk: 356 036 608 bytes.

# unsquashfs -s ./filesystem256k_xz.squashfs 
Found a valid SQUASHFS 4:0 superblock on ./filesystem256k_xz.squashfs.
Creation or last append time Mon Aug 28 03:36:44 2023
Filesystem size 356036423 bytes (347691.82 Kbytes / 339.54 Mbytes)
Compression xz
	Dictionary size 262144
	Filters selected: x86
Block size 262144
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Uids/Gids (Id table) are compressed
Fragments are compressed
Always-use-fragments option is specified
Xattrs are compressed
Duplicates are not removed
Number of fragments 2018
Number of inodes 69516
Number of ids 62
Number of xattr ids 8

Second test is to use -comp zstd (that is zstandard as compression instead of xz) with maximum compression level through -Xcompression-level 22:

time mksquashfs squashfs-root/ filesystem256k_zstd.squashfs -no-recovery -always-use-fragments -b 256k -no-duplicates -comp zstd -Xcompression-level 22

real	5m42,547s
user	21m37,545s
sys	0m3,419s

The result:

Filesize on disk: 395 042 816 bytes.

# unsquashfs -s ./filesystem256k_zstd.squashfs 
Found a valid SQUASHFS 4:0 superblock on ./filesystem256k_zstd.squashfs.
Creation or last append time Mon Aug 28 03:44:01 2023
Filesystem size 395039343 bytes (385780.61 Kbytes / 376.74 Mbytes)
Compression zstd
	compression-level 22
Block size 262144
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Uids/Gids (Id table) are compressed
Fragments are compressed
Always-use-fragments option is specified
Xattrs are compressed
Duplicates are not removed
Number of fragments 2018
Number of inodes 69516
Number of ids 62
Number of xattr ids 8

Summary:

Using optimized xz compression will result in a 5.16% smaller image (19 386 368 bytes decrease) compared to default/current.

While using optimized zstandard compression will result in a 5.22% larger image (19 619 840 bytes increase) compared to defult/current.

The difference in compression time between using xz vs zstd in this case is less than 30 seconds on a modern hardware (in favour to zstd).

Even if benchmarks shows that squashfs-file created using zstd have a greater read performance compared to xz the use for this in VyOS case is probably limited.

That is VyOS will probably benefit more from a 5% smaller image (xz) than a 5% larger image (zstd) even if the later can be slightly faster when it comes to reads.

I have not (yet) tested how the different squashfs-files (orignal, optimized xz, optimized zstd) affects read performance of VyOS.

Ref:

https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/benchmark.txt

https://jonathancarter.org/2015/04/06/squashfs-performance-testing/

syncer triaged this task as Normal priority.Aug 28 2023, 3:42 AM

PR created: https://github.com/vyos/vyos-build/pull/389

PR389 build failed:

https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6055144564/job/16433670804

I: Setting up additional APT entries
Traceback (most recent call last):
  File "/vyos/./build-vyos-image", line 455, in <module>
    lb_config_command = lb_config_tmpl.render(build_config)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.11/dist-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 2, in top-level template code
jinja2.exceptions.UndefinedError: 'chroot' is undefined
Error: Process completed with exit code 1.

Assuming this is due to that the variable is using "-" where it should be using "_".

Will create another PR to fix that.

PR created: https://github.com/vyos/vyos-build/pull/390

Apachez claimed this task.Sep 2 2023, 11:18 AM

Was missing quotes around the variable within lb_config_tmpl like so:

--chroot-squashfs-compression-type "{{squashfs_compression_type}}" \

Fixed by:

https://vyos.dev/R3:9403274bf29110b3d1aa5be99a029a97103e02f1

Regarding filesystem.squashfs the changes through changed mksquashfs syntax are:

filesystem.squashfs (from vyos-1.4-rolling-202308310021-amd64.iso)
374605951 bytes (357.3MB)

# unsquashfs -s ./filesystem.squashfs
Found a valid SQUASHFS 4:0 superblock on ./filesystem.squashfs.
Creation or last append time Thu Aug 31 02:22:47 2023
Filesystem size 374605951 bytes (365826.12 Kbytes / 357.25 Mbytes)
Compression xz
Block size 131072
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Uids/Gids (Id table) are compressed
Fragments are compressed
Always-use-fragments option is not specified
Xattrs are compressed
Duplicates are removed
Number of fragments 2618
Number of inodes 69451
Number of ids 47
Number of xattr ids 8

filesystem.squashfs (from vyos-1.4-rolling-202309030023-amd64.iso)
356459695 bytes (339.9MB)

# unsquashfs -s ./filesystem.squashfs
Found a valid SQUASHFS 4:0 superblock on ./filesystem.squashfs.
Creation or last append time Sun Sep  3 03:40:01 2023
Filesystem size 356459695 bytes (348105.17 Kbytes / 339.95 Mbytes)
Compression xz
	Dictionary size 262144
	Filters selected: x86
Block size 262144
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Uids/Gids (Id table) are compressed
Fragments are compressed
Always-use-fragments option is specified
Xattrs are compressed
Duplicates are not removed
Number of fragments 2020
Number of inodes 69526
Number of ids 47
Number of xattr ids 8

Or a saving of 374605951-356459695 = 18146256 bytes (17.3MB) or 4.84%.

Looking at the total iso size (for download) the changes are:

vyos-1.4-rolling-202308310021-amd64.iso
430964736 bytes (411MB)

vyos-1.4-rolling-202309030023-amd64.iso
413138944 bytes (394MB)

Or a saving of 430964736-413138944 = 17825792 bytes (17MB) or 4.14%.

Note: The difference of 18146256-17825792 = 320464 bytes between how much the filesystem.squashfs shrunk vs how much the iso shrunk is most likely due to different kernels (6.1.49 vs 6.1.50) being used (which takes space within the iso):

# ls -l | awk '{print $5, $6, $7, $9}'
29210016 aug 31 initrd.img
29210016 aug 31 initrd.img-6.1.49-amd64-vyos
6278432 aug 28 vmlinuz
6278432 aug 28 vmlinuz-6.1.49-amd64-vyos

# ls -l | awk '{print $5, $6, $7, $9}'
30432087 sep 3 initrd.img
30432087 sep 3 initrd.img-6.1.50-amd64-vyos
6278816 aug 31 vmlinuz
6278816 aug 31 vmlinuz-6.1.50-amd64-vyos

Reference to https://jonathancarter.org/2015/04/06/squashfs-performance-testing/ using 1M blocksize will give approx the same readspeed as with default 128k blocksize but result in an even smaller file.

However the creation time will further increase slightly (approx 30% compared to 128k blocksize or approx 13% compared to 256k blocksize - tests below show 16.5% compared to 128k blocksize and 11.6% compared to 256k blocksize, so your mileage may vary).

For the below I have reverted the use of "-no-duplicates" (back to default value which is to scan for duplicates and only store one physical copy of a duplicated file and link the rest (duplicates) within the squashfs).

Using filesystem.squashfs from vyos-1.4-rolling-202309030023-amd64.iso:

Original file (which was created with "-no-duplicates"):

#unsquashfs -s ./filesystem.squashfs
Found a valid SQUASHFS 4:0 superblock on ./filesystem.squashfs.
Creation or last append time Sun Sep  3 03:40:01 2023
Filesystem size 356459695 bytes (348105.17 Kbytes / 339.95 Mbytes)
Compression xz
	Dictionary size 262144
	Filters selected: x86
Block size 262144
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Uids/Gids (Id table) are compressed
Fragments are compressed
Always-use-fragments option is specified
Xattrs are compressed
Duplicates are not removed
Number of fragments 2020
Number of inodes 69526
Number of ids 47
Number of xattr ids 8

Note that the below times are only to give a rough estimate, using Intel i5-4250U CPU with 2 cores and 2 threads per core so 4 HT (and various readcaches can be in play etc).

#time mksquashfs squashfs-root/ filesystem128k.squashfs -comp xz -Xbcj x86 -b 128k -always-use-fragments -no-recovery
real	6m2,767s
user	23m13,798s
sys	0m16,956s

#unsquashfs -s ./filesystem128k.squashfs
Found a valid SQUASHFS 4:0 superblock on ./filesystem128k.squashfs.
Creation or last append time Sun Sep  3 05:24:24 2023
Filesystem size 364609085 bytes (356063.56 Kbytes / 347.72 Mbytes)
Compression xz
	Dictionary size 131072
	Filters selected: x86
Block size 131072
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Uids/Gids (Id table) are compressed
Fragments are compressed
Always-use-fragments option is specified
Xattrs are compressed
Duplicates are removed
Number of fragments 3230
Number of inodes 69526
Number of ids 47
Number of xattr ids 8

#time mksquashfs squashfs-root/ filesystem256k.squashfs -comp xz -Xbcj x86 -b 256k -always-use-fragments -no-recovery
real	6m19,335s
user	24m8,593s
sys	0m4,017s

#unsquashfs -s ./filesystem256k.squashfs
Found a valid SQUASHFS 4:0 superblock on ./filesystem256k.squashfs.
Creation or last append time Sun Sep  3 05:31:43 2023
Filesystem size 352339908 bytes (344081.94 Kbytes / 336.02 Mbytes)
Compression xz
	Dictionary size 262144
	Filters selected: x86
Block size 262144
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Uids/Gids (Id table) are compressed
Fragments are compressed
Always-use-fragments option is specified
Xattrs are compressed
Duplicates are removed
Number of fragments 1963
Number of inodes 69526
Number of ids 47
Number of xattr ids 8

#time mksquashfs squashfs-root/ filesystem1M.squashfs -comp xz -Xbcj x86 -b 1M -always-use-fragments -no-recovery
real	7m2,796s
user	25m48,960s
sys	0m4,133s

#unsquashfs -s ./filesystem1M.squashfs
Found a valid SQUASHFS 4:0 superblock on ./filesystem1M.squashfs.
Creation or last append time Sun Sep  3 05:39:50 2023
Filesystem size 332707512 bytes (324909.68 Kbytes / 317.29 Mbytes)
Compression xz
	Dictionary size 1048576
	Filters selected: x86
Block size 1048576
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Uids/Gids (Id table) are compressed
Fragments are compressed
Always-use-fragments option is specified
Xattrs are compressed
Duplicates are removed
Number of fragments 766
Number of inodes 69526
Number of ids 47
Number of xattr ids 8

Summary:

Original file (filesystem.squashfs):

356462592 bytes (339.9 MB)

Using 128k blocksize as reference for below comparisons:

Size: 364609536 bytes (347.7 MB)
Create time: 6 min 3 sec.

256k blocksize:

Size: 352342016 bytes (336 MB)
Create time: 6 min 19 sec.
vs 128k: 16 sec increase or approx 4.4% longer time compared to 128k.
vs 128k: 12267520 bytes (11.7 MB) or approx 3.4% size reducation compared to 128k.

1M blocksize:

Size: 332709888 bytes (317.2 MB)
Create time: 7 min 3 sec.
vs 128k: 60 sec increase or approx 16.5% longer time compared to 128k.
vs 128k: 31899648 bytes (30.4 MB) or approx 8.7% size reducation compared to 128k.
vs 256k: 44 sec increase or approx 11.6% longer time compared to 256k.
vs 256k: 19632128 bytes (18.7 MB) or approx 5.6% size reducation compared to 256k.

Some further testing:

Boot time using VyOS 1.4-rolling-202308310021 (128k blocksize):

[24] vyos-router []: Waiting for NICs to settle down.
[30] vyos-router []: Mounting VyOS Config... done.
[55] vyos-router []: Starting VyOS router: migrate configure.
[56] vyos-router []: Configuration success

Boot time using VyOS 1.4-rolling-202309030023 (128k blocksize and removed -no-duplicates):

[24] vyos-router []: Waiting for NICs to settle down.
[30] vyos-router []: Mounting VyOS Config... done.
[53] vyos-router []: Starting VyOS router: migrate configure.
[54] vyos-router []: Configuration success

Boot time using VyOS 1.4-rolling-202309030023 (256k blocksize and removed -no-duplicates):

[24] vyos-router []: Waiting for NICs to settle down.
[31] vyos-router []: Mounting VyOS Config... done.
[55] vyos-router []: Starting VyOS router: migrate configure.
[56] vyos-router []: Configuration success

Boot time using VyOS 1.4-rolling-202309030023 (1M blocksize and removed -no-duplicates):

[30] vyos-router []: Waiting for NICs to settle down.
[37] vyos-router []: Mounting VyOS Config... done.
[65] vyos-router []: Starting VyOS router: migrate configure.
[65] vyos-router []: Configuration success

Summary:

Not much change when it comes to boot times between 230831 and 230903 when it comes to 128k or 256k blocksize.

However using 1M blocksize will increase the boot times by approx 16%.

If this would just affect the boot time by approx 10 seconds I would say 1M blocksize is a winner.

However Im afraid that commit times when VyOS is up and running can be negatively affected aswell.

I will let the maintainers decide if 1M blocksize should be tested in 1.4-rolling (to get real life measurements specially from larger deployments) or not.

In the meantime I will create a commit to remove the "-no-duplicates" to further slice of a few megabytes from the resulting filesystem.squashfs (and iso file).

PR created: https://github.com/vyos/vyos-build/pull/392

PR392 was merged in VyOS 1.4-rolling-202309070021.

The result is that another approx 4MB were shaved off the iso size:

root@vyos:/home/vyos# unsquashfs -s /usr/lib/live/mount/persistence/boot/1.4-rolling-202309070021/1.4-rolling-202309070021.squashfs
Found a valid SQUASHFS 4:0 superblock on /usr/lib/live/mount/persistence/boot/1.4-rolling-202309070021/1.4-rolling-202309070021.squashfs.
Creation or last append time Thu Sep  7 03:32:59 2023
Filesystem size 352553024 bytes (344290.06 Kbytes / 336.22 Mbytes)
Compression xz
	Dictionary size 262144
	Filters selected: x86
Block size 262144
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Uids/Gids (Id table) are compressed
Fragments are compressed
Always-use-fragments option is specified
Xattrs are compressed
Duplicates are removed
Number of fragments 1965
Number of inodes 69603
Number of ids 47
Number of xattr ids 8

I havent noticed any misbehaviour from VyOS 1.4-rolling-202309070021 which gives that I will put this task to resolved (it can be reopened if some anomaly due to squashfs is detected later on).

Summary (filesize on disk filesystem.squashfs):

VyOS 1.4-rolling-202308280021: 375 422 976 bytes.
vyos-1.4-rolling-202308280021-amd64.iso: 413MB

VyOS 1.4-rolling-202308310021: 374 605 951 bytes.
vyos-1.4-rolling-202308310021-amd64.iso: 411MB

VyOS 1.4-rolling-202309030023: 356 459 695 bytes.
vyos-1.4-rolling-202309030023-amd64.iso: 394MB

VyOS 1.4-rolling-202309070021: 352 553 024 bytes.
vyos-1.4-rolling-202309070021-amd64.iso: 391MB

Saved 375422976-352553024 = 22 869 952 bytes (21.8MB) or approx 6.1% size reduction.

Note that between the various releases above other changes have been made to VyOS such as different Linux kernel etc.

However most of them (such as the Linux kernel) have increased in size between the releases (which is why the iso itself didnt shrink as much as above numbers for the filesystem.squashfs).

Ref: https://github.com/vyos/vyos-rolling-nightly-builds/releases

Apachez closed this task as Resolved.Sep 7 2023, 5:15 AM

Shrink imagesize and improve read performance by changing mksquashfs syntaxClosed, ResolvedPublicFEATURE REQUESTActions

Description

Details

Event Timeline

Shrink imagesize and improve read performance by changing mksquashfs syntax
Closed, ResolvedPublicFEATURE REQUEST
Actions