Page MenuHomeVyOS Platform

Cloud-init images refuse to work with network-based datasource such as Ec2 or OpenStack (but do work with OpenStack's config drive)
Closed, InvalidPublicBUG

Description

The VyOS build system has targets for cloud environments, but ships with a broken cloud-init. In EC2, instance never come up as accessible to the user, showing eth0 as down in the cloud-init boot messages on ttyS0 and an inability to make network connection with the 169.254.169.254 address.
I have brought this issue up in slack and been told to use separate AMI build repos to achieve this goal. The problem isnt the image being built, its how cloud-init interacts with the network stack of VyOS. An AWS instance does briefly get an IP - if i start a ping run against its IP while its initializing i get a single echo reply out of 100s of pings sent and then it goes dark again.
Clouds are a pretty major target for the OS and likely a major revenue source. Could VyOS build system please address the issue? Its likely going to require changes to cloud-init to deal with how VyOS configures its networks, but a config.boot.default explicitly specifying dhcp for eth0 doesnt help - cloud-init appears to override the address assignment from the vyatta tier at the linux tier and then everything breaks. Obviously once that happens you're screwed in AWS because those "people" dont provide a proper console like every other sane cloud on the planet.

Details

Difficulty level
Unknown (require assessment)
Version
1.3
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Infrastructure issue or change

Event Timeline

zsdc added a subscriber: zsdc.

The main reason for such issues is missing a good one instructions on how to build a proper one image.

A step-by-step how to prepare the VyOS AMI for AWS properly on the Linux host.

Pre-requirement: You will need an AWS account credentials for the awscli (AWS Access Key ID and AWS Secret Access Key). See details here: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html

  1. Clone vyos-build and build-amirepositories:
git clone https://github.com/vyos/vyos-build.git /tmp/vyos-build/build/
git clone https://github.com/vyos/build-ami.git /tmp/vyos-build/ami/
  1. Pull the vyos-build Docker image.
docker pull vyos/vyos-build:current
  1. Run the Docker image:
docker run --rm --privileged -it -v /tmp/vyos-build/:/vyos-build/ -w /vyos-build/build/ vyos/vyos-build:current bash
  1. Configure the build:
./configure
  1. Create VyOS build for AWS:
sudo make AWS
  1. Upload the VyOS ISO image from the build directory (/tmp/vyos-build/build/build/live-image-amd64.hybrid.iso file on the host) to any publicly available HTTP server.
  2. Install Ansible and awscli with requirements (we still inside a Docker container):
sudo apt update
sudo apt install ansible python-boto3 python-botocore python-boto awscli
  1. Configure the awscli:
aws configure
  1. Run the vyos-build-ami script to create your AMI:
/vyos-build/ami/vyos-build-ami <ISO URL>

And you will get the new AMI ready to use on AWS EC2.

erkin set Issue type to Infrastructure issue or change.Aug 29 2021, 1:04 PM
erkin removed a subscriber: Active contributors.

I've managed to get this working in our own builds by restoring the openstack target and making some changes there - runs fine in AWS, even with a grsec kernel and hardened userspace (Xen is often the worst visor for ring0 memory defenses).