Page MenuHomeVyOS Platform

Cleanup of unused directories (and files) in order to shrink image-size
Closed, ResolvedPublicFEATURE REQUEST

Description

Follow up on https://vyos.dev/T5468

Suggestion to also remove the following unused directories from the image (some such as /var/cache/* will be recreated when needed during runtime):

/ usr/local/man
/ usr/local/share/man
/ usr/games
/ usr/local/games
/ var/cache/*

And the following backup-copies:

/ etc/sudoers.bak
/ etc/xml/polkitd.xml.old
/ etc/xml/catalog.old
/ etc/xml/xml-core.xml.old
/ root/.gnupg/pubring.kbx~
/ var/lib/dpkg/status-old
/ var/lib/dpkg/diversions-old
/ var/lib/sgml-base/supercatalog.old

And the following log-files (will be recreated when needed):

/ var/log/apt/history.log
/ var/log/apt/term.log
/ var/log/stunnel4/stunnel.log
/ var/log/nginx/access.log
/ var/log/nginx/error.log
/ var/log/squidguard/squidGuard.log
/ var/log/bootstrap.log
/ var/log/dpkg.log
/ var/log/alternatives.log

The above actions will shrink the image by approx 11 MB.

Note: Had to add a space between first / and the rest of the path to bypass the Cloudflare WAF who otherwise would block this post.

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Improvement (missing useful functionality)

Event Timeline

sarthurdev changed the task status from Open to Needs testing.Aug 27 2023, 8:07 AM
sarthurdev moved this task from Need Triage to In Progress on the VyOS 1.4 Sagitta board.

A note from https://forum.vyos.io/t/clear-logs-on-vyos/6878/10?u=viacheslav that there might be issues if removing directories from within / var/log/* doesnt occur to PR381 since that PR was specific about which files and directories to remove when it comes to / var/log. That is only files NOT directories were removed from / var/log.

Also verified by trying to start nginx which started without issues using VyOS 1.4-rolling-202308280021:

vyos@vyos:~$ sudo systemctl start nginx
vyos@vyos:~$ ps auxwww | grep -i nginx
root        3058  0.0  0.0  10396   988 ?        Ss   03:15   0:00 nginx: master process / usr/sbin/nginx -g daemon on; master_process on;
www-data    3059  0.0  0.0  10740  2680 ?        S    03:15   0:00 nginx: worker process
www-data    3060  0.0  0.0  10740  2680 ?        S    03:15   0:00 nginx: worker process

Verified that most files have been deleted according to https://vyos.dev/T5511

However when copying the filesystem.squashfs from VyOS 1.4-rolling-202308280021 and extracting it using unsquashfs I can see that the content of /var/cache isnt fully removed.

Apt and localpurge manages to find its way back meaning about 4 MB of unnecessary files remains:

root@box:/ opt/vyos/squashfs-root/var/cache# ls -laR
.:
total 16
drwxr-xr-x  4 root root 4096 aug 28 02:23 .
drwxr-xr-x 13 root root 4096 aug 28 02:23 ..
drwxr-xr-x  3 root root 4096 aug 28 02:23 apt
drw-r--r--  2 root root 4096 aug 28 02:23 localepurge

./ apt:
total 3712
drwxr-xr-x 3 root root    4096 aug 28 02:23 .
drwxr-xr-x 4 root root    4096 aug 28 02:23 ..
drwxr-xr-x 3 root root    4096 aug 28 02:23 archives
-rw-r--r-- 1 root root 2189971 aug 28 02:23 pkgcache.bin
-rw-r--r-- 1 root root 1594438 aug 28 02:23 srcpkgcache.bin

./ apt/archives:
total 12
drwxr-xr-x 3 root root 4096 aug 28 02:23 .
drwxr-xr-x 3 root root 4096 aug 28 02:23 ..
-rw-r----- 1 root root    0 aug 28 02:23 lock
drwxr-xr-x 2 root root 4096 aug 28 02:23 partial

./ apt/archives/partial:
total 8
drwxr-xr-x 2 root root 4096 aug 28 02:23 .
drwxr-xr-x 3 root root 4096 aug 28 02:23 ..

./ localepurge:
total 12
drw-r--r-- 2 root root 4096 aug 28 02:23 .
drwxr-xr-x 4 root root 4096 aug 28 02:23 ..
-rw-r--r-- 1 root root  464 aug 28 02:23 localelist
root@box:/ opt/vyos/squashfs-root/var/cache# du -sm .
4	.

I assume this is because / data/live-build-config/hooks/live/80-delete-docs.chroot is not running late enough that is apt is runned after the hooks have been processed.

The workflow log also suggests this:

https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/5994053207/job/16255182905#step:8:6210

Where apt is runned later in the process such as:

https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/5994053207/job/16255182905#step:8:6236

https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/5994053207/job/16255182905#step:8:6316

etc...

I think it would be prefered to insert a cleanup process just before "I: Configuring live-build":

https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/5994053207/job/16255182905#step:8:7638

Do there exist some other way to perform a cleanup of the chroot just before the mksquashfs process is runned?

Found out that mksquashfs supports -ef EXCLUDE_FILE as a file that (line by line) defines which files and directories to be excluded during creation of filesystem.squashfs. Adding -wildcard will make it possible to use wildcards within the EXCLUDE_FILE.

It also turns out that binary_rootfs who actually creates the filesystem.squashfs (using mksquashfs) in live-build will automatically use such EXCLUDE_FILE (with -wildcard) if its accessible in config/rootfs/excludes.

The above means that all thats should be needed is to create such EXCLUDE_FILE in:

https://github.com/vyos/vyos-build/tree/current/data/live-build-config/rootfs/excludes

and populate it with files and directories (including wildcards) to be excluded from the creation of filesystem.squashfs.

I will do a first attempt by excluding:

/var/cache/apt/*

Bonus: It seems that newer live-build should automatically clean /var/cache/apt/* through use of chroot_archives.

Ref:

https://manpages.debian.org/bookworm/squashfs-tools/mksquashfs.1.en.html

The excludes-file in PR406 had incorrectly a '/' as first character (for the directory to be excluded from the squashfs-file).

Turns out that if you want to exclude '/var/cache/*' that should in the excludes-file be written as 'var/cache/*'.

Also added comments and some documentations and references in the excludes-file.

New PR created: https://github.com/vyos/vyos-build/pull/407

PR created: https://github.com/vyos/vyos-build/pull/414

Hopefully with final changes to the rootfs/excludes-file :-)

Verified to be working as expected.

When future files or directories needs to be removed from the build (just before compile of the filesystem.squashfs) then using rootfs/excludes-file is the favoured method.

Only if the exclude cannot be performed through wilrdcard then a hooks-file should be used instead.