Page MenuHomeVyOS Platform

More easily produceable source builds
Closed, WontfixPublic

Assigned To
Authored By
runar
Apr 10 2024, 8:13 PM
Tags
None
Referenced Files
None
Tokens
"Love" token, awarded by fetzerms."Like" token, awarded by HappyShr00m."Like" token, awarded by marekm.

Description

Intro

In the current implementation of build scripts everything is tightly integrated into the current official Jenkins CI pipelines.
This makes creating images from source builds quite hard as if a user wants to fetch and build a package from source it needs to fetch build instructions from the Jenkinsfile's before commencing the build.
On a one packet scale this is often a manageable task, but on a "whole VyOS from source" scale this does not scale to a easily manageable solution.
This should be made much more efficient, so a user potentially could create a totally local image without even needing internet on the build device when the build commences (as long as the user have downloaded all sources locally, and yea it's not a realistic scenario as also debian-mirror and other 3rd party repositories needs to be made local).

Current build process

The current build process is like this: First fetch vyos-build repository of the version requested, then after the command to build have started all needed packages are fetched from a Debian repositories, VyOS upstream repository and other third party repositories. After this all local packages in vyos-build/packages/ are installed into the image at build-time offering the possibility to add custom packages to the resulting image generated.

This means that a official VyOS package repository will need to be available at build time to commence the build. this also means that to install any packages other than the latest package is not possible as the package repository only installs the latest version for each package.

The changes described here will not change the need of a official Debian mirror, or third party app mirrors, but only change the need for the internal VyOS packages installed on top of the Debian installed.

New proposal for source builds

These changes will be devided into multiple sub-parts thats tigthly integrated to create the final image:

1) Local package index in vyos-build

To allow a local build like this it is needed to create a index of what "custom" packages are needed to create a image of VyOS.
this could be created using eg. a yaml/toml that lists all packages needed and possibly a reference to what branch/git-tag/commit-reference is needed for a reproduce-able build. this file could also have custom build instructions and info on what files/deb's are needed to be fetched from the resulting build.(discussable)
example:

vyos-1x:
    type: git
    path: https://github.com/vyos/vyos-1x
    branch: current
    script: build.sh

vyatta-cfg:
    type: git
    path: https://github.com/vyos/libvyosconfig
    branch: current
    script: build.sh
......

This file should be extended with all repositoies needed to get a fully working VyOS image and if needed correct references to what version of the packages is being built.

An alternative is to make these build-indexes available for the different versions of VyOS, eg. a new index could be created at every LTS/sub-LTS release with references to each specific git-tag used on each package, but this is not a requirement. these indexes will also be dependent on the version of vyos-build beeing built, so this could be a hit-n-miss to get working if eg. the vyos-build scripts are update and a build is executed with an old index with eg. an old kernel.. maybe do you also need a specific version of the vyos-build repo to do a full build of an old image.

2) Split CI buildscripts into build and "orchestrational" changes (in each repository)

All Jenkins build scripts needs to be splitted, so the part of actually building the package is done in an external script,
for most cases this will be just to move the dpkg-build* commands into an external script that is executable outside jenkins but there are more advanced cases, eg the kernel build pipeline that needs more work done into these scripts.
This could be done in a Makefile, a approriate shell-script, or as a "blueprint" executed by the build-script noted in 3)

example of buildscript "handcrafted" from the vyos-build jenkins pipeline scripts:

#!/bin/sh
set -e
if [ -f debian/control ]; then
    sudo mk-build-deps --install --tool "apt-get --yes --no-install-recommends"
    sudo dpkg -i *build-deps*.deb
fi
dpkg-buildpackage -uc -us -tc -b
3) Source build script

To execute the source build pipeline, an "orchestrator" script is needed. This script will be in charge of:

  • fetching the source packages from git using the index in 1) and put them into the correct places.
  • execute the build scripts from 2)
  • fetching the interesting parts of the build, eg. .deb files used etc. and put them into a directory usable by the image builder at the later stage.
  • eventually cleanup after an old build

At this point you should end up with an index of .deb files that could be made into a local-apt-repository usable by the installer.
it will not be wise to install everything built like being done with the /packages/ directory, but instead a local apt repository should be created that is usable by apt directly while the normal VyOS build-scripts are executing.

Wrapup

The hardest parts here i think will be creating the orchestrator script, but as this is just fetching/copying files and executing commands it should not be that hard to execute. Testing this could also be done by removing the original official VyOS apt repository from the build and replacing it with the local repository created in the build.

This was my small plan of how this could be executed, but i would say that it for now have many loose ends and its much here that potentially could need discussions.
I'm all inn for any comments on this.

Details

Difficulty level
Unknown (require assessment)
Version
1.5
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Perfectly compatible
Issue type
Internal change (not visible to end users)

Event Timeline

runar created this object in space S1 VyOS Public.

It would be useful for 1.4 as well, to be able to rebuild the LTS with local patches to some packages but otherwise unmodified, for example to make it easier to test bug fixes in a single package (such as accel-ppp https://vyos.dev/T4600 ) without much risk to introduce new bugs.

syncer claimed this task.
syncer removed a project: VyOS 1.5 Circinus.
syncer subscribed.

this was superseded with new build system

The new build system is welcome, thank you. Restricted access to LTS sources, not so much...

@marekm, stop spamming in every message about LTS.
You know all the ways to get LTS images, including entirely free ones.
Thanks!

@Viacheslav please, one of the free ways that I know and actually used was to build from source - and it's not updated anymore (moved to private repo). BTW, it''s still mentioned on the VyOS website, which is misleading. So this is very much related to this report, I'd like to build my own sagitta images using the new improved build system (hooray no more Jenkins!) but can't, it's not spamming just raising valid concerns. I don't care about pre-built LTS images anymore, just open the sources again so it is possible to build own images as before. Thank you.

It's not just me, there is significant interest in the community to do builds from source, and even in your own words @syncer https://vyos.dev/T6264#185011 it appears it was OK just a few months ago.
People took the challenge and have done some good work to make builds reasonably easy, complete with de-branding to address the tradermark concerns, and a few fixes in the build process.
As with any open source project, it's kind of natural to want to build from source. So I really don't understand all this hostility. I try to stay calm and not insult anyone, please try to do the same.
BTW, I try to contribute to some related projects (accel-ppp) and even though I'm strictly technical there (testing, reporting bugs, sending patches), issues are converted to discussions but that's all, just silence.
Let's stop fighting against each other and start some normal discussion (OK maybe here is not the best place, so I'll stop now). Peace.

Source is at https://github.com/vyos
Project around for 11 years but instead of participating, you decided you were smarter than everybody else and would abuse subscriptions for home users to run your business.
When we cut of people like you, instead of joining our efforts, you again decided you were smarter, you wanted per GPL, and now it will be per GPL
Not even to mention your shit-talking about Team and me. Go elsewhere, you are not welcome here

Sorry @syncer - I beg to differ.
It should be clear from the context that I mean the sagitta branch and at https://github.com/vyos it is no longer being updated.
I did participate, it's easy to find some evidence so I won't repeat it here, and all was well until a few months ago.
Only discovered and started using VyOS 1.2.x in 2020 (easy to check when I registered), not for 11 years. Earlier there was Vyatta, later EdgeOS on ERPro-8 for BGP, and MikrotTik as PPPoE server.
The only reason for moving to VyOS was non-commercial - wanted to enable IPv6, no customer asked or paid extra for it, it's just that I feel it was the right thing to do, but both EdgeOS and RouterOS had poor IPv6 support.
My business is a small local ISP for a few hundreds of people in a rural area, yes really that small and it's just me doing everything (no employees), not what you usually think when you see "ISP" (big companies with lots of money).
All IPv4 address space I use is 1024 addresses, easy to check in BGP (AS56523), and no CGNAT so each customer gets a single static IPv4, so you get the idea about the size of my business. My service costs about $15/month and I also provide some free services (for example to volunteer firefighters - Ochotnicza Straz Pozarna), also have two local schools in range but they were not interested (already have EU subsidized Orange fiber).
I never said I was smarter, and I don't understand what you mean by shit-talking. It's the first time ever in my life when I'm being accused of such things. Looks like some horrible misunderstanding.
If you mean something specific, please point me to it, and if it was indeed the case I will publicly apologize. Really I have no idea what you mean here.
But simply having some different opinions is not shit-talking. Actually I highly respect the Team for all the good work, it's just the last few months when things went horribly wrong in relations with some (not all) of you.
One more thing, I've been using Linux and other open source projects since 1994 (yes 30 years), and VyOS was the only ever open source project I supported with donations, $25/month from March 2022 to June 2024 when you stopped accepting them, and I did that voluntarily (didn't have to do it to get access to images, as I already had it granted as open source contributor).
So I hope you can see why I don't want to go away until we can talk like normal people and clear all the horrible misunderstandings.