Page MenuHomeVyOS Platform
Create Task
Maniphest T7299

pki: race condition for acme requested certificates - CA auto import only on the second run
Closed, ResolvedPublicFEATURE REQUEST
Actions

Description

Summary

When using the VyOS internal PKI subsystem to request a certificate using ACME, the issuer CA is not automatically imported in the PKI subsystem on the first run due to a race condition.

Use case

It's always a good idea to provide the full certificate chain when running a daemon that uses SSL certificates

Additional information

This can be reproduced by:

Check if there are no ACME related certificates on the system.

vyos@vyos# ls /config/auth/letsencrypt/live
ls: cannot access '/config/auth/letsencrypt/live': No such file or directory

Request an ACME certificate from the LetsEncrypt staging API

set pki certificate LR5.wue4 acme domain-name 'LR5.wue4.vyos.net'
set pki certificate LR5.wue4 acme email 'LR5@vyos.net'
set pki certificate LR5.wue4 acme url 'https://acme-staging-v02.api.letsencrypt.org/directory'

Check installed PKI certificates:

cpo@LR5.wue4# run show pki
Certificate Authorities:
Name    Subject    Issuer CN    Issued    Expiry    Private Key    Parent
------  ---------  -----------  --------  --------  -------------  --------

Certificates:
Name      Type    Subject CN             Issuer CN                            Issued               Expiry               Revoked    Private Key    CA Present
--------  ------  ---------------------  -----------------------------------  -------------------  -------------------  ---------  -------------  ------------
LR5.wue4  Server  CN=lr5.wue4.vyos.net   CN=(STAGING) Wannabe Watercress R11  2025-03-30 11:45:10  2025-06-28 11:45:09  No         Yes            No

Certificate Revocation Lists:
CA Name    Updated    Revokes
---------  ---------  ---------
[edit]

It misses the auto imported CA chain. Currently only a reboot will auto import the issuing CA into pki ca certificate tree.

cpo@LR5.wue4:~$ show pki
Certificate Authorities:
Name                Subject                                                             Issuer CN                     Issued               Expiry               Private Key    Parent
------------------  ------------------------------------------------------------------  ----------------------------  -------------------  -------------------  -------------  --------
AUTOCHAIN_LR5.wue4  CN=(STAGING) Counterfeit Cashew R10,O=(STAGING) Let's Encrypt,C=US  CN=(STAGING) Pretend Pear X1  2024-03-13 00:00:00  2027-03-12 23:59:59  No             N/A

Certificates:
Name      Type    Subject CN             Issuer CN                            Issued               Expiry               Revoked    Private Key    CA Present
--------  ------  ---------------------  -----------------------------------  -------------------  -------------------  ---------  -------------  ------------------------
LR5.wue4  Server  CN=lr5.wue4.vyos.net  CN=(STAGING) Counterfeit Cashew R10  2025-03-30 11:48:57  2025-06-28 11:48:56  No         Yes            Yes (AUTOCHAIN_LR5.wue4)

Certificate Revocation Lists:
CA Name    Updated    Revokes
---------  ---------  ---------

Details

Version
2025.03.30-0020-rolling
Is it a breaking change?
Perfectly compatible
Issue type
Feature (new functionality)

Related Objects
Search...

Event Timeline

c-po created this task.Sun, Mar 30, 12:49 PM
c-po changed the task status from Open to In progress.
c-po claimed this task.
c-po triaged this task as Normal priority.
c-po changed Version from - to 2025.03.30-0020-rolling.
c-po updated the task description. (Show Details)
c-po added a comment.Sun, Mar 30, 1:09 PM

https://github.com/vyos/vyos-1x/pull/4424

c-po moved this task from Need Triage to Backport Candidates on the VyOS Rolling board.Sun, Mar 30, 1:22 PM
pasik subscribed.Sun, Mar 30, 3:28 PM
c-po mentioned this in rVYOSONEX1f82952b36c7: pki: T7299: race condition for acme requested certificates / CA chain.Mon, Mar 31, 3:59 PM
Restricted Repository Identity mentioned this in rVYOSONEXa6ff1933a99a: Merge pull request #4424 from c-po/acme-race-T7299.Mon, Mar 31, 3:59 PM
zsdc subscribed.Mon, Mar 31, 7:51 PM

The current fix is necessary, but it's not complete.

The rest of the issue lies in this part of the code: config_dict_mangle_acme()

The problem is that if we use get_config_dict(..., with_pki=True), the config_dict_mangle_acme() function receives only the ACME certificate name and returns a single certificate–private key pair. However, it completely ignores the Let's Encrypt CAs. Therefore, later find_chain() has not enough material to extract a full chain.

From my understanding of the overall structure, the expected behavior would be for config_dict_mangle_acme() to additionally check for the presence of a chain.pem file and include all certificates from it as well. A little modification of get_config_dict is required as well.

c-po added a comment.EditedTue, Apr 1, 12:25 PM

Hi @zsdc,

your assumption is correct, but:

config_dict_mangle_acme() is indeed used to to "blend in" the BASE64 PEM encoded certificate data into the config dict object. This mimics the behavior if someone will load manual certificate data into set pki certificate NAME certificate or set pki certificate NAME private key as the ACME retrieved certificate data is stored into exactly these key inside the config dictionary - we do not do anything special here, so all existing code can be re-used for loading the full certificate chain.

Example CLI:

set pki certificate LR5.wue4 acme domain-name 'LR5.wue4.mybll.net'
set pki certificate LR5.wue4 acme email 'foo@bar.com'
cpo@LR5.wue4# run show pki
Certificate Authorities:
Name                Subject                      Issuer CN        Issued               Expiry               Private Key    Parent
------------------  ---------------------------  ---------------  -------------------  -------------------  -------------  --------
AUTOCHAIN_LR5.wue4  CN=R11,O=Let's Encrypt,C=US  CN=ISRG Root X1  2024-03-13 00:00:00  2027-03-12 23:59:59  No             N/A

Certificates:
Name      Type    Subject CN             Issuer CN    Issued               Expiry               Revoked    Private Key    CA Present
--------  ------  ---------------------  -----------  -------------------  -------------------  ---------  -------------  ------------------------
LR5.wue4  Server  CN=lr5.wue4.mybll.net  CN=R11       2025-04-01 11:20:34  2025-06-30 11:20:33  No         Yes            Yes (AUTOCHAIN_LR5.wue4)

Now startup OpenConnect:

set vpn openconnect authentication local-users username random password 'random12345'
set vpn openconnect authentication mode local 'password'
set vpn openconnect network-settings client-ip-settings subnet '10.0.0.0/29'
set vpn openconnect network-settings name-server '1.1.1.1'
set vpn openconnect ssl certificate 'LR5.wue4'

And you will see the full certificate chain added to ocserv

cpo@LR5.wue4# cat /run/ocserv/cert.pem
-----BEGIN CERTIFICATE-----
MIIFJjCCBA6gAwI...
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIFBjCCAu6...
-----END CERTIFICATE-----

I think what you are experiencing is that when you run this in a single commit:

set pki certificate LR5.wue4 acme domain-name 'LR5.wue4.mybll.net'
set pki certificate LR5.wue4 acme email 'foo@bar.com'

set vpn openconnect authentication local-users username random password 'random12345'
set vpn openconnect authentication mode local 'password'
set vpn openconnect network-settings client-ip-settings subnet '10.0.0.0/29'
set vpn openconnect network-settings name-server '1.1.1.1'
set vpn openconnect ssl certificate 'LR5.wue4'

The full chain is missing from /run/ocserv/cert.pem most likely because of https://vyos.dev/T7307

c-po mentioned this in T7307: Data added via vyos.utils.configfs.add_cli_node() not available when script is called via call_depends().Tue, Apr 1, 12:26 PM
c-po moved this task from Backport Candidates to Completed on the VyOS Rolling board.Tue, Apr 1, 2:09 PM
c-po moved this task from Open to Finished on the VyOS 1.5 Circinus board.
dmbaturin edited projects, added VyOS 1.4 Sagitta (1.4.2); removed VyOS 1.4 Sagitta (1.4.3).Tue, Apr 1, 3:01 PM
dmbaturin changed Is it a breaking change? from Unspecified (possibly destroys the router) to Perfectly compatible.
c-po closed this task as Resolved.Tue, Apr 1, 3:16 PM
c-po moved this task from Backlog to Finished on the VyOS 1.4 Sagitta (1.4.2) board.
dmbaturin mentioned this in 1.4.2.Wed, Apr 2, 3:24 PM