Support hardware rate limiting (pacing) with TLS offload.

[jhb]

Summary

• Add a new send tag type for a send that that supports both packet pacing and TLS offload (mostly similar to D22669 but adds a separate structure when allocating the new tag type).

• When allocating a send tag for TLS offload, check to see if the connection already has a rate limiting tag. If so, allocate a tag that supports both rate limiting and TLS offload rather than a plain TLS offload tag.

• When re-allocating a TLS offload tag due to a route change, pass the old tag to ktls_alloc_snd_tag() so that its pacing rate can be copied to the new tag if the old tag supported rate limiting.

• When setting an initial rate on an existing ifnet KTLS connection, set the rate on the inp and then reset the TLS send tag (via ktls_output_eagain) to reallocate a TLS + ratelimit send tag. This allocates the TLS send tag asynchronously from a task queue, so the TLS rate limit tag alloc is always sleepable.

• When modifying a rate on a connection using KTLS, look for a TLS send tag. If the send tag is only a plain TLS send tag, assume we failed to allocate a TLS ratelimit tag (either during the TCP_TXTLS_ENABLE socket option, or during the send tag reset triggered by ktls_output_eagain) and ignore the new rate. If the send tag is a ratelimit TLS send tag, change the rate on the TLS tag and leave the inp tag alone.

• Lock the inp lock when setting sb_tls_info for a socket send buffer so that the routines in tcp_ratelimit can safely deref the pointer without needing to grab the socket buffer lock.

• Add a IFCAP_TXTLS_RTLMT capability flag and associated administrative controls in ifconfig(8). TLS rate limit tags are only allocated if this capability is enabled. Note that TLS offload (whether unlimited or rate limited) always requires IFCAP_TXTLS[46].

Test Plan

• Drew tested this at Netflix

[jhb]

One note is that we do not free the inp send tag once we have enabled TLS offload (KTLS) on a rate limited socket. The tag will hang around and not be freed until the inp is torn down. Part of the reason for this is that you'd have to wait until there were no more TLS imbues in the socket buffer before knowing it is safe to free the tag reference from the inp as otherwise it needs to stay in place so that any queued transmit data before TLS was enabled needs to use that tag.

Note also that there is no way in this setup to handle "unlimited" packets such as retransmits. For plain TCP these are handled by not tagging them with the rate limit tag, but we have to have the tag always for TLS offload to work correctly. Drivers could perhaps ignore the pacing for retransmits (drivers can detect those and often need to for TLS offload) if the implementation supports individual TLS request without a pacing rate. (I think the way TLS offload works on T6, the pacing rate is assigned to the connection, so you couldn't do that, for example.)

sys/netinet/tcp_ratelimit.c

1415

I mostly think it's ok to punt on this and would be ok removing this comment if that is the consensus of others.

[• hselasky]

Looks good from mlx5_en_main.c's point of view.
@gallatin: Did you test this with mlx5en?

[gallatin]

In D26691#595529, @hselasky wrote:

Looks good from mlx5_en_main.c's point of view.
@gallatin: Did you test this with mlx5en?

Yes

[jhb]

sys/kern/uipc_ktls.c

1315

This introduces a use-after-free (and it's a feature that we release the old tag before allocating the new one, T6 requires it for lagg failover to work). I will have to adjust this.

[jhb]

• Rebase on D26875.

[jhb]

sys/kern/uipc_ktls.c

823

Note that we don't support the socket pacing rate set via SO_MAX_PACING_RATE. It would not be hard to support that in the future if desired, though we would need to define how the two pacing models should interact (e.g. should TCP pacing always override the socket option or vice versa). It's not clear to me how those interact today in the non-TLS case.

[jhb]

sys/kern/uipc_ktls.c

885

This looks to be a bug on my part.

[• hselasky]

head/sys/net/if.h

253 ↗

(On Diff #78880)

I think you need to spell this like 0x80000000U .