tcp: allow connections to IPv6 anycast address

[ivy]

Summary

currently, we reject incoming TCP connections to an IPv6 anycast address
based on IETF I-D "draft-itojun-ipv6-tcp-to-anycast-01"[0]. the
rationale is that since RFC2373 prohibits sending IPv6 packets with an
anycast address as the source address, it would be impossible to
establish a TCP connection to such an address since the destination host
could not send any replies.

however, this restriction was lifted in RFC4291 and it is no longer
forbidden to send packets from an anycast address; therefore, it's both
possible and permitted to establish a TCP connection using an anycast
address as src or dst address (or both).

based on the above, delete this restriction and allow people to do this.

while there are certain operational reasons to avoid TCP anycast (such
as the risk of the route changing while the connection is open), these
also apply to IPv4 anycast and are specific to the local environment;
for example, it's perfectly valid to have an anycast address which is
only ever assigned to one node.

[0] https://www.ietf.org/archive/id/draft-itojun-ipv6-tcp-to-anycast-01.txt

[tuexen]

I do understand that anycast addresses can be used as source and destination addresses in combination with UDP, but I share the view of stated in section 3.1 of RFC 7094:

Most stateful transport protocols (e.g., TCP), without modification,
do not understand the properties of anycast; hence, they will fail
probabilistically, but possibly catastrophically, when using anycast
addresses in the presence of "normal" routing dynamics.

In section 4.2 a possible way of handling anycast addresses in TCP is described, which is not implemented as far as I know. For SCTP we can implement something like https://datatracker.ietf.org/doc/html/draft-tuexen-tsvwg-sctp-init-fwd-02.

So I think we should not allow binding a TCP endpoint to an anycast address... Doing it for UDP is fine.

[ivy]

i agree there are reasons not to do this, but by forbidding it we're essentially saying there's never any reason to do this, which i don't think is true. for example, i have some anycast addresses in my network which are only on one node, and these work fine, although of course they are not currently configured as IFF_ANYCAST addresses due to this restriction.

my feeling here is that we shouldn't nanny the user; anyone configuring anycast addresses should be aware of the operational implications of doing this and make their own decision about it. currently, they're forced to use non-IFF_ANYCAST addresses to do this, which increases of risk of error from e.g. accidentally using an anycast address as the source of an outgoing connection due to source address selection algorithm. that seems definitely worse than using an anycast address.

the solution in RFC 7094 §4.2 is quite clever, but does anyone currently implement either the client or server side of this? it seems like it would have implications for the security of TCP connections.

[adrian]

In D50019#1140715, @ivy wrote:

i agree there are reasons not to do this, but by forbidding it we're essentially saying there's never any reason to do this, which i don't think is true. for example, i have some anycast addresses in my network which are only on one node, and these work fine, although of course they are not currently configured as IFF_ANYCAST addresses due to this restriction.

my feeling here is that we shouldn't nanny the user; anyone configuring anycast addresses should be aware of the operational implications of doing this and make their own decision about it. currently, they're forced to use non-IFF_ANYCAST addresses to do this, which increases of risk of error from e.g. accidentally using an anycast address as the source of an outgoing connection due to source address selection algorithm. that seems definitely worse than using an anycast address.

+1, I think it's fine to let the user shoot their own foot so to speak. If they're configuring ANYCAST then they should know what that entails.

the solution in RFC 7094 §4.2 is quite clever, but does anyone currently implement either the client or server side of this? it seems like it would have implications for the security of TCP connections.

Maybe add a comment to replace the section you deleted covering that ANYCAST is fine now, see RFC xxxx ?

Oh that's a neat hack, but yeah, at that point you should be using anycast for service location, and then protocol level redirects (eg HTTP 3xx) to clean things up.

[ivy]

i discussed this a bit elsewhere and someone pointed out that if you want to use anycast for DNS then you have to enable TCP or your DNS server won't work properly. given how many people have successfully deployed anycast DNS, it seems like there's a strong operational argument to permit this.

In D50019#1140722, @adrian wrote:

Maybe add a comment to replace the section you deleted covering that ANYCAST is fine now, see RFC xxxx ?

i considered this but it seemed odd to have a comment with no associated code saying "don't do anything here since we don't disallow this". in exchange i tried to make the commit message as informative as possible for anyone who is curious :-)

[tuexen]

In D50019#1140724, @ivy wrote:

i discussed this a bit elsewhere and someone pointed out that if you want to use anycast for DNS then you have to enable TCP or your DNS server won't work properly. given how many people have successfully deployed anycast DNS, it seems like there's a strong operational argument to permit this.

I am aware that you use DNS/TCP in addition to DNS/UDP and I am aware that for DNS/UDP anycast is used. But are you saying that people deploy TCP in combination with anycast at scale?

[tuexen]

my feeling here is that we shouldn't nanny the user; anyone configuring anycast addresses should be aware of the operational implications of doing this and make their own decision about it. currently, they're forced to use non-IFF_ANYCAST addresses to do this, which increases of risk of error from e.g. accidentally using an anycast address as the source of an outgoing connection due to source address selection algorithm. that seems definitely worse than using an anycast address.

OK. But if TCP connections are failing, we can't do much about it.

the solution in RFC 7094 §4.2 is quite clever, but does anyone currently implement either the client or server side of this? it seems like it would have implications for the security of TCP connections.

In my view this breaks a couple of things. Therefore, I doubt that it will be implemented.
One could use a variant of MPTCP (which is not implemented in FreeBSD right now) for the TCP use case of the INIT forwarding for SCTP (which is easy to implement).

[ivy]

In D50019#1140725, @tuexen wrote:

are you saying that people deploy TCP in combination with anycast at scale?

yes, for example see: https://www.ripe.net/publications/docs/ripe-393/

The impact of this problem [i.e., stateful connections over anycast] is not clear: for example, in a study of J-root [5] the authors state that this is a serious problem and recommend that stateful services not be run on anycast at all. Other work has since concluded that the impact of node switches is not significant enough to be a concern [6, 12]. Our own results for K-root are presented in Section 4.3.

so, there are mixed results, but some people are using it successfully at scale.

[tuexen]

In D50019#1140737, @ivy wrote:

In D50019#1140725, @tuexen wrote:

are you saying that people deploy TCP in combination with anycast at scale?

yes, for example see: https://www.ripe.net/publications/docs/ripe-393/

The impact of this problem [i.e., stateful connections over anycast] is not clear: for example, in a study of J-root [5] the authors state that this is a serious problem and recommend that stateful services not be run on anycast at all. Other work has since concluded that the impact of node switches is not significant enough to be a concern [6, 12]. Our own results for K-root are presented in Section 4.3.

so, there are mixed results, but some people are using it successfully at scale.

I see. They are using short lived TCP connections for that time scale the anycast routing seems to be stable.

Thank you very much for sharing this document!

[rrs]

How about a compromise here... I do think Michael has a valid point...

Add in a sysctl that defaults to off. And add back the code you took out with a

if ((sysctl_var == true) && (ia6 && ,,,,) {+
}

This way you have to set the sysctl to true (blatantly shooting yourself in the foot).

R

[tuexen]

In D50019#1141053, @rrs wrote:

How about a compromise here... I do think Michael has a valid point...

Add in a sysctl that defaults to off. And add back the code you took out with a

if ((sysctl_var == true) && (ia6 && ,,,,) {+
}

This way you have to set the sysctl to true (blatantly shooting yourself in the foot).

R

I agree with Randall. I think in other cases the default values are safe and you have to do something (change a sysctl variable, for example) to shoot yourself in the foot...

[ivy]

here's the thing though, you can already accept TCP connections to an anycast address by simply not marking the address as IFF_ANYCAST, and this is what everyone does today. if you force users to set a sysctl to do this, they will just not bother marking addresses as IFF_ANYCAST and will be more likely to run into problems from e.g. accidentally originating outgoing connections from an anycast address.

or in other words, the choice here isn't "should we let users accept TCP connections to anycast addresses?" because it's impossible to prevent that, it's "should we allow users to correctly mark their existing anycast addresses as anycast addresses?". i don't see any advantage to forcing users to set a sysctl in order to configure networking more correctly.

more sysctls is more confusing and creates more pain for users, removing special cases and unnecessary code is more beautiful. :-)

however, i would be willing to add some text to ifconfig.8 to mention the potential issues with TCP anycast and refer them to some appropriate documentation.

[rrs]

In D50019#1141170, @ivy wrote:

here's the thing though, you can already accept TCP connections to an anycast address by simply not marking the address as IFF_ANYCAST, and this is what everyone does today. if you force users to set a sysctl to do this, they will just not bother marking addresses as IFF_ANYCAST and will be more likely to run into problems from e.g. accidentally originating outgoing connections from an anycast address.

or in other words, the choice here isn't "should we let users accept TCP connections to anycast addresses?" because it's impossible to prevent that, it's "should we allow users to correctly mark their existing anycast addresses as anycast addresses?". i don't see any advantage to forcing users to set a sysctl in order to configure networking more correctly.

more sysctls is more confusing and creates more pain for users, removing special cases and unnecessary code is more beautiful. :-)

however, i would be willing to add some text to ifconfig.8 to mention the potential issues with TCP anycast and refer them to some appropriate documentation.

And who exactly is "everyone"?

I know for a fact Netflix (one of the main users of FreeBSD) does not.

I strongly suggest you add the sysctl and suggested code. In general I think it best to make it so naive users will not "shoot themselves in the foot". And let only power users do that.

[ivy]

In D50019#1141201, @rrs wrote:

here's the thing though, you can already accept TCP connections to an anycast address by simply not marking the address as IFF_ANYCAST, and this is what everyone does today.

And who exactly is "everyone"?

I know for a fact Netflix (one of the main users of FreeBSD) does not.

by 'everyone' i mean 'everyone using TCP on an anycast address' - sorry if this was not clear.

I strongly suggest you add the sysctl and suggested code. In general I think it best to make it so naive users will not "shoot themselves in the foot". And let only power users do that.

i do not think simply wrapping the current code in a sysctl is the right approach, but let me explain why and suggest an alternative.

firstly, regardless of anything else, i think the current code should be removed as it's based on an 20-year-old expired I-D, and the only reason it's done this way is that at the time you couldn't send a RST packet from an anycast address. but now you can do that, so if we're going to forbid TCP connections to anycast addresses, i think we should do this with a RST, not an ICMP error. does that seem reasonable?

secondly, this problem isn't specific to TCP, there are also UDP protocols that don't always work well over anycast:

• QUIC (the protocol supports anycast, but server implementation may not or admin may not configure it)
• NTP (client may keep state with server)
• DNS with EDNS cookies (depending on how the server is configured)
• probably more i haven't thought of; any UDP service that keeps state and is not explicitly anycast-aware could be affected.

of course, it's possible to implement these services in such a way that they work with anycast, but the idea here is to protect users who don't understand that this might be required or don't even realise it's a problem.

so my suggestion, which solves both of these issues, is to remove the current code in tcp and instead place bind() behind a sysctl, i.e., you cannot bind a socket to an IFF_ANYCAST address unless the sysctl is enabled, regardless of protocol - this would apply to TCP, UDP, SCTP, raw sockets, whatever.

does this seem reasonable? i believe it's simpler for users because behaviour of IFF_ANYCAST is the same regardless of protocol, and it also more thoroughly addresses the current objection.