Remove IPSEC from GENERIC due to performance issues

[gallatin]

Summary

Having IPSEC compiled into the kernel imposes a non-trivial performance penalty on multi-threaded workloads due to IPSEC refcounting.

In my benchmarks of multi-threaded UDP transmit (connected sockets), I've seen a roughly 20% performance penalty when the IPSEC option is included in the kernel (16.8Mpps vs 13.8Mpps with 32 senders on a 14 core / 28 HTT Xeon 2697v3)). This is largely due to key_addref() incrementing and decrementing an atomic reference count on the default policy. This cause all CPUs to stall on the same cacheline, as it bounces between different CPUs.

Given that relatively few users use ipsec, and that it can be loaded as a module, it seems reasonable to ask those users to load the ipsec module so as to avoid imposing this penalty on the GENERIC kernel. Its my hope that this will make FreeBSD look better in "out of the box" benchmark comparisons with other operating systems.

[gallatin]

To make this change less painful for users, it seems like we should enable ifconfig to auto-load ipsec when ifconfig is invoked with ipsecX. ifconfig will currently try to load if_ipsec, and this does not work because the kernel module is ipsec.ko. So it seems like there are 3 obvious workarounds:

1. teach ifconfig that ipsec is "special", and avoid prepending "if_" to its name when trying to load the kld.
2. rename ipsec to if_ipsec. That would break users configs, and I don't like it.
3. link if_ipsec.ko to ipsec.ko. I have heard the existing linking mechanism for em/igb causes issues.

Anything I'm not considering? I tend to think (1) is the simplest option.

[cem]

No objection to moving IPSEC out of GENERIC (but continuing to build it as a module). Check with gnn@, as he originally added it.

I agree that teaching ifconfig about ipsec (option 1) seems the most straightforward way to provide compatibility through that interface.

[ae]

I think there are too few users of if_ipsec, to make assumption that all users who use IPsec will use ifconfig(8). AFAIR, it is not the problem, you can just add symlink if_ipsec.ko -> ipsec.ko. But you also need some tweaks that will load ipsec,ko when ipsec_enable is "YES".

[gallatin]

In D20163#434345, @cem wrote:

No objection to moving IPSEC out of GENERIC (but continuing to build it as a module). Check with gnn@, as he originally added it.

I agree that teaching ifconfig about ipsec (option 1) seems the most straightforward way to provide compatibility through that interface.

Thanks. I've added @gnn to the review.

[delphij]

No objection from me.

[jhb]

FWIW, my limited testing of IPsec doesn't use if_ipsec, but instead I used setkey. I think having the rc.d scripts for 'ipsec_enable' autoloading ipsec.ko is reasonable. Maybe if there are ports scripts for IKE daemons (raccoon, strongswan?) those rc.d scripts should also try to load modules they need.

[ae]

In D20163#434567, @jhb wrote:

FWIW, my limited testing of IPsec doesn't use if_ipsec, but instead I used setkey. I think having the rc.d scripts for 'ipsec_enable' autoloading ipsec.ko is reasonable.

I added this today in rS347178.

Maybe if there are ports scripts for IKE daemons (raccoon, strongswan?) those rc.d scripts should also try to load modules they need.

I think this should be done by port maintainers.

[gallatin]

In D20163#434567, @jhb wrote:

FWIW, my limited testing of IPsec doesn't use if_ipsec, but instead I used setkey. I think having the rc.d scripts for 'ipsec_enable' autoloading ipsec.ko is reasonable. Maybe if there are ports scripts for IKE daemons (raccoon, strongswan?) those rc.d scripts should also try to load modules they need.

Now that @ae has added this feature to autoload ipsec.ko, are you OK with this change?

[jhb]

In D20163#435313, @gallatin wrote:

In D20163#434567, @jhb wrote:

FWIW, my limited testing of IPsec doesn't use if_ipsec, but instead I used setkey. I think having the rc.d scripts for 'ipsec_enable' autoloading ipsec.ko is reasonable. Maybe if there are ports scripts for IKE daemons (raccoon, strongswan?) those rc.d scripts should also try to load modules they need.

Now that @ae has added this feature to autoload ipsec.ko, are you OK with this change?

Oh, yes, sorry I wasn't objecting to the change at all. I was intending to refer to ae@'s commit that I had already seen when I made my original comment as being sufficient.