certctl: Add support for generating a certificate bundle (CAfile)

[michaelo]

Summary

There are situations where a piece of software or its environment cannot be
modified to use a certificate directory, but requires a certificate bundle to
be present. Add options to generate/remove such a bundle from the canonical
truststore as a stop-gap solution until a port is fixed or accepted not
to be capable of.

Co-authored-by: Michael Osipov <[e-mail]>
PR: 284749
Tested by: michaelo
MFC after: 2 weeks

[michaelo]

Please add additional reviews as you think fit.

Note: I know that there is a draft about the reimplementation of certctl in C from @des , but it has been paused for quite some time.

[franco_opnsense.org]

usr.sbin/certctl/certctl.sh

410

I understand the intention here, but:

Why not write ${LOCALBASE}/openssl/certs by default as that is what certctl was built for.

Write bundle to isolated location ${LOCALBASE}/etc/ssl/cert.pem and point problematic applications to it (either by source code or configuration).

This avoids any duplication between directory and bundle and you can create a bundle by default (even if it breaks ca_root_nss default bundle contents in that particular place, but all its certificates should be included)

[michaelo]

usr.sbin/certctl/certctl.sh

410

Well, I am not the author of the patch, see PR, but from PoV:

• certctl(8) should only effectively write into the base system
• The OpenSSL ports should have an option to use the store from base by patching: https://github.com/openssl/openssl/blob/1eb5ffcdc8a270b6d49b6b6f5097ebe61f66f648/include/internal/common.h#L85-L87

I'd be even willing to write a patch, it shouldn't be that hard.

WDYT?

[michaelo]

@brnrd Can you comment on the OpenSSL from ports ideas?

[franco_opnsense.org]

usr.sbin/certctl/certctl.sh

410

certctl(8) should only effectively write into the base system

Not a great argument, it makes all of this moot. If you want a "base" bundle we will never replace ca_root_nss as is. And as I said tainting /etc/ssl/cert.pem with the same certificates is not going to help anyone. One could aim for /etc/ssl/bundle.pem but then you get into trouble for adding a new problem with a new non-standard path. It will be another perpetual workaround.

The OpenSSL ports should have an option to use the store from base by patching: https://github.com/openssl/openssl/blob/1eb5ffcdc8a270b6d49b6b6f5097ebe61f66f648/include/internal/common.h#L85-L87

Not sure what you are suggesting. /etc/ssl/certs works fine, /etc/ssl/cert.pem currently occupied by ca_root_nss ETCSYMLINK option. Ideally, /etc/ssl/cert.pem should be a secondary user location that certctl only handles as read-only perhaps when creating the bundle.

Unfortunately, we went through a certification process with the FreeBSD base and ports in OPNsense there are a number of loose ends that need straightening out and I don't have the commit bit to help with that. As said before it would be good to have a long term plan here and it has been left unattended for long enough. :)

[heliosyne_gmail.com]

I'm the originator of patch. I'll try to answer specific questions, but all the discussion of which paths are correct is somewhat moot. Some of it was already hashed out in the PR, but in short, hier(7) and the general base vs local engineering fence means /etc/ssl/certs and /etc/ssl/cert.pem are probably the best choices for canonical locations managed by a base program.

About the questions of ensuring third-party compliance:

Even if we fixed every port (and we should), that's still a subset of the software certctl needs to support. Binary distributions, composer, rust crates, pip, etc. all complicate changing away from the three CAfiles precedent that's held for half a decade or more at this point. What is quite easy to do, however, is make certctl compatible with the status quo. And that's exactly what this patch does.

This avoids any duplication between directory and bundle

This was brought up in the PR before, and I'll ask again here: have you observed operational problem(s) with CApath and CAfile intersecting and in concurrent use?

[michaelo]

usr.sbin/certctl/certctl.sh

410

You can already do that by manipulating the environment variables with certctl(8). It is fully flexible, IMHO.

410

See my comment above and I think that this change will make ca_root_nss obsolete. At least I cannot think of a use case why ca_root_nss has to stay after that.

[michaelo]

In D49130#1120635, @heliosyne_gmail.com wrote:

I'm the originator of patch. I'll try to answer specific questions, but all the discussion of which paths are correct is somewhat moot. Some of it was already hashed out in the PR, but in short, hier(7) and the general base vs local engineering fence means /etc/ssl/certs and /etc/ssl/cert.pem are probably the best choices for canonical locations managed by a base program.

About the questions of ensuring third-party compliance:

Even if we fixed every port (and we should), that's still a subset of the software certctl needs to support. Binary distributions, composer, rust crates, pip, etc. all complicate changing away from the three CAfiles precedent that's held for half a decade or more at this point. What is quite easy to do, however, is make certctl compatible with the status quo. And that's exactly what this patch does.

As for these:

• composer: I have patched it upstream, it just works for months now
• rust (cargo): I have a patch here and waiting for a upstream and downstream: https://reviews.freebsd.org/D49120
• pip: it works, what does not work with base?

But I agree, there are situations (which I don't have) which require a CAfile. Period. I am with you.

This avoids any duplication between directory and bundle

This was brought up in the PR before, and I'll ask again here: have you observed operational problem(s) with CApath and CAfile intersecting and in concurrent use?

I have laid out how OpenSSL traverses both stores, I don't see a problem -- especially that CAfile is created on request only.

[franco_opnsense.org]

First of all thanks for your work. I'm in the same boat as you wanting to see this progress.

In D49130#1120635, @heliosyne_gmail.com wrote:

I'm the originator of patch. I'll try to answer specific questions, but all the discussion of which paths are correct is somewhat moot. Some of it was already hashed out in the PR, but in short, hier(7) and the general base vs local engineering fence means /etc/ssl/certs and /etc/ssl/cert.pem are probably the best choices for canonical locations managed by a base program.

I understand the need for alignment. The alignment is insofar problematic as it immediately jumps out of the alignment when trying to fix ports with a base tool on the surface of it. I'd rather see a structural fix then with:

1. hooks for pre and post rehash (with the pre hooks being able to abort a rehash even). I asked for comments on those a year ago and had no response. Hooks would guarantee that a port could be shipped that holds the functionality without tainting the base tool with a feature that pertains to ports specifically. It adds unnecessary load to the standard library load since it will start opening the file by default.

2. add the hooks to security/openssl even to pull the /etc/ssl/certs to the configured /usr/local/openssl/certs directory -- no ca file necessary at all. There are at least 3 different problem scopes at play here. Trying to solve all in one is not the best way forward in my opinion.

3. document the caveats with dir and file interaction, different locations and so on now rather than later. It's not explained why this matters and how this interacts and what the best practice is when having to specify a bundle file in a configuration file of a port for example.

These are the problems worth solving that I see:

1. Problem: /etc/ssl/cert.pem shouldn't be written since it duplicates /etc/ssl/certs. certctl was written to avoid this as ca_root_nss was removed from post ports to accommodate this. It prevents users from using /etc/ssl/cert.pem as a manual location if needed, too.
2. Problem: /usr/local/openssl/certs isn't written (not for base/ports separation clarity, just because it's not a default way of doing it...) so fixing security/openssl requires setting up that directory. Ideally from the port itself.
3. Problem: /usr/local/etc/ssl/cert.pem historically fed by ca_root_nss should be created for general consumption (that's great, let's document it, but then where if it's a port scope not base scope?)

Personally, the MFC is also daring for a structural change like this. It leaves little time to improve this based on feedback once it's in.

Here's my previous mail BTW https://lists.freebsd.org/archives/freebsd-ports/2023-October/004739.html touching all these subjects and no responses so it is what it is. Adding general purpose pre and post rehash hooks would be much more effective long term before solving the bundle situation with it in a hardcoded way now.

[michaelo]

In D49130#1120674, @franco_opnsense.org wrote:

First of all thanks for your work. I'm in the same boat as you wanting to see this progress.

In D49130#1120635, @heliosyne_gmail.com wrote:

I'm the originator of patch. I'll try to answer specific questions, but all the discussion of which paths are correct is somewhat moot. Some of it was already hashed out in the PR, but in short, hier(7) and the general base vs local engineering fence means /etc/ssl/certs and /etc/ssl/cert.pem are probably the best choices for canonical locations managed by a base program.

I understand the need for alignment. The alignment is insofar problematic as it immediately jumps out of the alignment when trying to fix ports with a base tool on the surface of it. I'd rather see a structural fix then with:

1. hooks for pre and post rehash (with the pre hooks being able to abort a rehash even). I asked for comments on those a year ago and had no response. Hooks would guarantee that a port could be shipped that holds the functionality without tainting the base tool with a feature that pertains to ports specifically. It adds unnecessary load to the standard library load since it will start opening the file by default.

Is this really necessary? If a port is broken, it needs to be fixed. Fiddling with the truststore should be beyond the port's capabilities...

2. add the hooks to security/openssl even to pull the /etc/ssl/certs to the configured /usr/local/openssl/certs directory -- no ca file necessary at all. There are at least 3 different problem scopes at play here. Trying to solve all in one is not the best way forward in my opinion.

I have provided a possible solution to that w/o copying or linking anything, read apply local patch to OpenSSL/LibreSSL/YouNameIt.

3. document the caveats with dir and file interaction, different locations and so on now rather than later. It's not explained why this matters and how this interacts and what the best practice is when having to specify a bundle file in a configuration file of a port for example.

Sure, refer to the appropriate OpenSSL manpage, but consider again that the CAfile is not built by default which means that most people will never care.

These are the problems worth solving that I see:

1. Problem: /etc/ssl/cert.pem shouldn't be written since it duplicates /etc/ssl/certs. certctl was written to avoid this as ca_root_nss was removed from post ports to accommodate this. It prevents users from using /etc/ssl/cert.pem as a manual location if needed, too.

That's the point of duplication since not everyone can use CApath. Everything else does not make sense and look at truss, if both are present OpenSSL will prefer the CAfile and then fallback to the CApath.

2. Problem: /usr/local/openssl/certs isn't written (not for base/ports separation clarity, just because it's not a default way of doing it...) so fixing security/openssl requires setting up that directory. Ideally from the port itself.

See for a solution above.

3. Problem: /usr/local/etc/ssl/cert.pem historically fed by ca_root_nss should be created for general consumption (that's great, let's document it, but then where if it's a port scope not base scope?)

ca_root_nss can be finally removed after this.

Personally, the MFC is also daring for a structural change like this. It leaves little time to improve this based on feedback once it's in.

I don't see a problem because it is off by default. We just need to add a note to UPDATING and maybe even to release notes. @heliosyne_gmail.com, can you provide a few nice lines for UPDATING?

Here's my previous mail BTW https://lists.freebsd.org/archives/freebsd-ports/2023-October/004739.html touching all these subjects and no responses so it is what it is. Adding general purpose pre and post rehash hooks would be much more effective long term before solving the bundle situation with it in a hardcoded way now.

I did, most others simply don't care or consider it a too complex problem.

[michaelo]

Found one nit.

usr.sbin/certctl/certctl.sh

294

I think this should not intervene with ca_root_nss for the time being and should be changed to -f. If ca_root_nss has been installed certctl(8) shouldn't touch it. WDYT?

[franco_opnsense.org]

Let me throw in the towel here. I've left two review comments.

FWIW, allowing security/openssl and others being able to run hooks would be highly beneficial to targeted automatic solutions with clear separation of concerns.

usr.sbin/certctl/certctl.sh

295

Consider doing an atomic rm/m so the CA file doesn't disappear for a brief moment when it's merely rewritten.

296

Side effect: when ca_root_nss with ETCSYMLINK=on (which is still the default) is installed this will start rewriting the files.

[michaelo]

usr.sbin/certctl/certctl.sh

296

Which? -f should stop that, hence my proposal.

[franco_opnsense.org]

usr.sbin/certctl/certctl.sh

296

That guess is currently correct for ca_root_nss.

Now the user put /etc/ssl/cert.pem with his self-signed nextcloud CA cert because that works for him and we're doing what?

[michaelo]

usr.sbin/certctl/certctl.sh

296

All custom leaf, intermediate or root certificates belong to /usr/local/share/certs as described by the manpage. It has been working perfectly for me for years with 30 intermediate and root CAs here. See TRUSTPATH.

[franco_opnsense.org]

usr.sbin/certctl/certctl.sh

296

my point is if we rely on users to read man pages we might as well consider users can solve this with 2 lines of script and a cron job while the code here remains opt-in guesstimating what it should do which caters to a rather small user base

[michaelo]

usr.sbin/certctl/certctl.sh

296

Well, we brag with decent manpages, can't we expect people to read them? RTFM? I do agree with the cronjob, but I think that this is a good tradeoff regarding automation and integration with everyone calling certctl(8) where you don't have direct control.

[heliosyne_gmail.com]

@franco_opnsense.org

I think this should not intervene with ca_root_nss for the time being and should be changed to -f

No. The whole point of this patch is to intervene on behalf of ca_root_nss. Per the original PR, this patch is expected to work in conjunction with modifications to ca_root_nss. Paraphrasing what I wrote in the PR: with this in base, ca_root_nss would exec certctl -b rehash and unexec certctl -B rehash instead of installing LOCALBASE/share/certs/ca-root-nss.crt and its symlinks. IOW, certctl has to step where ca_root_nss currently does or it can't do its job, and the patched certctl has to exist before ca_root_nss can be modified.

Side effect: when ca_root_nss with ETCSYMLINK=on (which is still the default) is installed this will start rewriting the files.

Ohhh let's do a hypothetical! I love those. Let's say we've some overly adventurous, boldly uninformed sysadmin who runs certctl -b rehash because they "saw it in a Google result", then later installs something that installs ca_root_nss. Here's what would happen:

1. ca_root_nss steps on CERTDESTFILE and CERTDESTFILELINKS (assuming pkg doesn't complain fatally);
2. base keeps working, because it uses CERTDESTDIR;
3. if they didn't have local certificates for something that only uses a CAfile, nothing breaks at all, they just keep on keepin on until something else runs certctl rehash (like freebsd-update);
4. when that rehash runs, it sees CERTDESTFILE existing in any form, and deletes it after also deleting the other two symlinks ca_root_nss made;
5. rehash then scans TRUSTPATH, finds LOCALBASE/share/certs/ca-root-nss.crt, splits it, and installs it into CERTDESTDIR as if it was just more local certs;
6. CERTDESTFILE and CERTDESTFILELINKS then get recreated and now magically contain ca-root-nss's data, other local certificates, the base certificate set, everything. All in one cute little bundle of cryptographic joy.

That is, it works. Magically, automatically, it works. Nothing gets harmed except some symlinks (those poor, poor symlinks...), and it will actually self-heal with just certctl rehash. AMAZING

This is why that test is -e "$CERTDESTFILE" and not -f "$CERTDESTFILE"

Consider doing an atomic rm/m so the CA file doesn't disappear for a brief moment when it's merely rewritten.

certctl isn't atomic to begin with. cmd_rehash wipes out the entire CERTDESTDIR and then recreates it live one hash link at a time. Making this atomic would require a larger rewrite beyond the scope of this patch. Even if it was made atomic, things would need to reread the CAfile anyway, so even an atomic write to CERTDESTFILE would still require third-party program good behaviour to actually be an atomic update.

p.s., if there is official interest in bringing certctl into the nuclear age, I can rewrite it. again.

[heliosyne_gmail.com]

@michaelo

As for these:

• composer: I have patched it upstream, it just works for months now
• rust (cargo): I have a patch here and waiting for a upstream and downstream: https://reviews.freebsd.org/D49120
• pip: it works, what does not work with base?

An app that rolls its own package management. They decided to release via GitHub, and their updater wraps pip. Pip itself works perfectly. But that's a bit point-adjacent. I was making a counterpoint that "fixing ports" wouldn't fix enough to make a patch like this unnecessary.

I have laid out how OpenSSL traverses both stores, I don't see a problem -- especially that CAfile is created on request only.

I don't either, but @franco_opnsense.org has brought it up repeately and I want to give them the opportunity to share any data they have demonstrating a problem.

[brnrd]

Whatever we do, this should be generated from TRUSTPATH not from DESTDIR as is the source for all trusted files.

usr.sbin/certctl/certctl.sh

195

Whatever we do, this should be generated from TRUSTPATH not from DESTDIR as is the source for all trusted files.

[michaelo]

usr.sbin/certctl/certctl.sh

195

This would completely duplicate the work which has been done to compile from TRUSTPATH, obeyed bad ones, expired, blocked, etc, to CERTDESTDIR.

[jrm]

Thanks for this. I still had ca_root_nss installed because local_unbound configured with TLS required a certificate bundle. Unbound with TLS is now working without ca_root_nss.

[michaelo]

In D49130#1124932, @jrm wrote:

Thanks for this. I still had ca_root_nss installed because local_unbound configured with TLS required a certificate bundle. Unbound with TLS is now working without ca_root_nss.

Yes, this is one of the reasons, though I think that unbound can be patched unless it runs in a chroot env. I do remember @des talking about it.

[michaelo]

In D49130#1124932, @jrm wrote:

Thanks for this. I still had ca_root_nss installed because local_unbound configured with TLS required a certificate bundle. Unbound with TLS is now working without ca_root_nss.

Regarding unbound, if you set tls-system-cert to true you will have the default truststore enabled: https://github.com/NLnetLabs/unbound/blob/5c84bb573f9728c10bcb3592dbd12be403d362de/util/net_help.c#L1593-L1601

Only chroot requires a bundle which sitll can use tls-system-cert.

In my opinion the unbound in base should have this option set to true by default to reduce the user headache.

[jrm]

In D49130#1124947, @michaelo wrote:

In D49130#1124932, @jrm wrote:

Thanks for this. I still had ca_root_nss installed because local_unbound configured with TLS required a certificate bundle. Unbound with TLS is now working without ca_root_nss.

Regarding unbound, if you set tls-system-cert to true you will have the default truststore enabled: https://github.com/NLnetLabs/unbound/blob/5c84bb573f9728c10bcb3592dbd12be403d362de/util/net_help.c#L1593-L1601

Only chroot requires a bundle which sitll can use tls-system-cert.

In my opinion the unbound in base should have this option set to true by default to reduce the user headache.

In D49130#1124947, @michaelo wrote:

In D49130#1124932, @jrm wrote:

Thanks for this. I still had ca_root_nss installed because local_unbound configured with TLS required a certificate bundle. Unbound with TLS is now working without ca_root_nss.

Regarding unbound, if you set tls-system-cert to true you will have the default truststore enabled: https://github.com/NLnetLabs/unbound/blob/5c84bb573f9728c10bcb3592dbd12be403d362de/util/net_help.c#L1593-L1601

Only chroot requires a bundle which sitll can use tls-system-cert.

In my opinion the unbound in base should have this option set to true by default to reduce the user headache.

Please confirm, but I believe if you put something like this in /etc/rc.conf:

local_unbound_enable="YES"
local_unbound_forwarders="x.x.x.x@853#xxx"
local_unbound_tls="YES"

Then run # service local_unbound setup, you will get that option in /var/unbound/unbound.conf and it will use a chroot, so you do need the bundle.

[michaelo]

In D49130#1124975, @jrm wrote:

In D49130#1124947, @michaelo wrote:

In D49130#1124932, @jrm wrote:

Thanks for this. I still had ca_root_nss installed because local_unbound configured with TLS required a certificate bundle. Unbound with TLS is now working without ca_root_nss.

Regarding unbound, if you set tls-system-cert to true you will have the default truststore enabled: https://github.com/NLnetLabs/unbound/blob/5c84bb573f9728c10bcb3592dbd12be403d362de/util/net_help.c#L1593-L1601

Only chroot requires a bundle which sitll can use tls-system-cert.

In my opinion the unbound in base should have this option set to true by default to reduce the user headache.

In D49130#1124947, @michaelo wrote:

In D49130#1124932, @jrm wrote:

Thanks for this. I still had ca_root_nss installed because local_unbound configured with TLS required a certificate bundle. Unbound with TLS is now working without ca_root_nss.

Regarding unbound, if you set tls-system-cert to true you will have the default truststore enabled: https://github.com/NLnetLabs/unbound/blob/5c84bb573f9728c10bcb3592dbd12be403d362de/util/net_help.c#L1593-L1601

Only chroot requires a bundle which sitll can use tls-system-cert.

In my opinion the unbound in base should have this option set to true by default to reduce the user headache.

Please confirm, but I believe if you put something like this in /etc/rc.conf:

local_unbound_enable="YES"
local_unbound_forwarders="x.x.x.x@853#xxx"
local_unbound_tls="YES"

Then run # service local_unbound setup, you will get that option in /var/unbound/unbound.conf and it will use a chroot, so you do need the bundle.

Yes, this looks good. Just checked again unbound's source code and usr.sbin/unbound/setup/local-unbound-setup.sh, it will properly do the following in gen_unbound_conf():

261     if [ "${use_tls}" = "yes" ] ; then
262         echo "        tls-system-cert: yes"

with

SSL_CTX_set_default_verify_paths(ctx)

. According to unbound docs all config is read, and I assume that OpenSSL is initialized in that phase, and only *after* that chroot() is invoked with /var/unbound. So this looks good to me. Consider that when you issue a reload it won't re-read the bundle because unbound is in the chroot and has no access to an updated bundle.
Please give it a try and intercept system calls with truss, it will show you the fopen/openat(..., "/etc/ssl/cert.pem") upfront.

I don't use local unbound, but unbound from ports on two redundant hosts to speed up DNS requests in my broadcast domain. TLS isn't available here.

[michaelo]

@jrm, did you get a chance to test that with local unbound?