Page MenuHomeVyOS Platform

Use http instead of https for rolling apt repo access
Closed, WontfixPublicFEATURE REQUEST

Description

The rolling apt repo with pre-built packages is not confidential and the packages are signed, so the extra protection by https is not necessary and consumes extra CPU power.
When more people start building their own rolling images (because LTS can't be built anymore), it might cause an overload and repeat of the recent situation with LTS.
Simple change (see below for a patch) to do what Debian does (use plain http instead of https) and save the planet (OK, really just some electricity and download time).
Using plain http also helps cache web traffic with caching web proxies, speeding up repeated downloads and reducing usage for people with slow/metered Internet connections.

Note, the change is not ready to apply yet - I have tried to build rolling with it and there seems to be an issue with the server.
Sometimes error 521 is returned for http while https works. Google search finds it's some Cloudflare-specific error when the origin server is down.
But http works sometimes, while https works always - not sure what exactly triggers the http 521 error, for now please just look into it if possible.

diff --git a/data/defaults.toml b/data/defaults.toml
index 67154cd..83b1c4c 100644
--- a/data/defaults.toml
+++ b/data/defaults.toml
@@ -9,7 +9,7 @@ debian_security_mirror = "http://deb.debian.org/debian-security"

 debian_archive_areas = "main contrib non-free non-free-firmware"

-vyos_mirror = "https://rolling-packages.vyos.net/current"
+vyos_mirror = "http://rolling-packages.vyos.net/current"

 vyos_branch = "current"
 release_train = "current"

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Perfectly compatible
Issue type
Infrastructure issue or change

Event Timeline

syncer claimed this task.
syncer removed a project: VyOS 1.5 Circinus.
syncer subscribed.

The current host is scheduled for decommissioning.
Future repositories will use object storage that doesn't support plain HTTP

I see. Thank you for explaining this.
Will this new storage continue to be public - or only for those with paid subscriptions?
If it will be public - will it be possible to mirror, to serve plain http to those who need it?

Public - yes.
Mirroring is not sure, as object storage does not support rsync, but maybe other tools can do it.

If you can use the APT then you can create mirror as well - the same way APT talks to the repository. There is no need for additional protocols like rsync.

The utility apt-mirror takes input similar as the sources.list and will create/update mirror for you over whatever protocol you can access the APT repository with (http, https, ftp, ...).

Obtain the utility https://packages.debian.org/bookworm/apt-mirror

Create vyos.list

############# config ##################
#
set base_path    /location-of-your-mirror
#
# set mirror_path  $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch  <running host architecture>
# set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads     20
set _tilde 0
#
############# end config ##############

deb-amd64 https://rolling-packages.vyos.net/current current main
deb-src https://rolling-packages.vyos.net/current current main

clean https://rolling-packages.vyos.net/current

You may include other architectures as well, just add another definition with deb-armhf/deb-arm64. This example is mirroring only amd64 + sources.

Then run:

apt-mirror vyos.list

And in no time you have mirror in /location-of-your-mirror.
If you do another run to update then you want to also do cleanup of superseded packages.
The apt-mirror will autogenerate cleanup script after you do run in the location of the mirror.

Thus run /location-of-your-mirror/var/clean.sh and this will delete obsolete versions of duplicated packages if there are any - otherwise it does nothing.

If you want to use your new mirror then simply point your webserver to /location-of-your-mirror/mirror/rolling-packages.vyos.net and you get the same thing as https://rolling-packages.vyos.net.

It's not getting easier than this :).

@blueish want to do a docs article for this?

@blueish - thanks! Yes, apt-mirror works now - but will it continue to work with the new storage too?
BTW, good to see "deb-src" - but only a few source packages are in there. I think it would be great to have corresponding source for all these *.deb packages in the Debian source package format, then anyone who wants to contribute will be able to use dpkg-buildpackage to rebuild them.

@syncer I could try. I'm just not sure what the motivation/use-case would be? I can think only of very specific cases where someone would want to do mirror thus I'm not sure if it justifies the existence in the docs. Someone may want to have local mirror if they do a lot of reruns of the image build process but that's seems like very much edge-case.

@marekm Yes, apt-mirror doesn't depend on anything special. It relies on the same files that apt update/install uses - thus it's pretty much independent on the server. The process is also fairly efficient thanks to the index files that tell what changed so only the first pull makes many requests, thus I don't think you would hit any rate limits either and if you do - you can always slow the synchronization down. Basically if apt can, then apt-mirror can too. The source packages are nice but you can dpkg-buildpackage even without them (well most of the time) you just call git clone instead apt source. It would be great to have source packages as well but perhaps redundant.

You are right @blueish, better to spend time on something more meaningful