tcp: allow connections to IPv6 anycast address

[ivy]

Summary

currently, we reject incoming TCP connections to an IPv6 anycast address
based on IETF I-D "draft-itojun-ipv6-tcp-to-anycast-01"[0]. the
rationale is that since RFC2373 prohibits sending IPv6 packets with an
anycast address as the source address, it would be impossible to
establish a TCP connection to such an address since the destination host
could not send any replies.

however, this restriction was lifted in RFC4291 and it is no longer
forbidden to send packets from an anycast address; therefore, it's both
possible and permitted to establish a TCP connection using an anycast
address as src or dst address (or both).

based on the above, delete this restriction and allow people to do this.

while there are certain operational reasons to avoid TCP anycast (such
as the risk of the route changing while the connection is open), these
also apply to IPv4 anycast and are specific to the local environment;
for example, it's perfectly valid to have an anycast address which is
only ever assigned to one node.

[0] https://www.ietf.org/archive/id/draft-itojun-ipv6-tcp-to-anycast-01.txt

[tuexen]

I do understand that anycast addresses can be used as source and destination addresses in combination with UDP, but I share the view of stated in section 3.1 of RFC 7094:

Most stateful transport protocols (e.g., TCP), without modification,
do not understand the properties of anycast; hence, they will fail
probabilistically, but possibly catastrophically, when using anycast
addresses in the presence of "normal" routing dynamics.

In section 4.2 a possible way of handling anycast addresses in TCP is described, which is not implemented as far as I know. For SCTP we can implement something like https://datatracker.ietf.org/doc/html/draft-tuexen-tsvwg-sctp-init-fwd-02.

So I think we should not allow binding a TCP endpoint to an anycast address... Doing it for UDP is fine.

[ivy]

i agree there are reasons not to do this, but by forbidding it we're essentially saying there's never any reason to do this, which i don't think is true. for example, i have some anycast addresses in my network which are only on one node, and these work fine, although of course they are not currently configured as IFF_ANYCAST addresses due to this restriction.

my feeling here is that we shouldn't nanny the user; anyone configuring anycast addresses should be aware of the operational implications of doing this and make their own decision about it. currently, they're forced to use non-IFF_ANYCAST addresses to do this, which increases of risk of error from e.g. accidentally using an anycast address as the source of an outgoing connection due to source address selection algorithm. that seems definitely worse than using an anycast address.

the solution in RFC 7094 §4.2 is quite clever, but does anyone currently implement either the client or server side of this? it seems like it would have implications for the security of TCP connections.

[adrian]

In D50019#1140715, @ivy wrote:

i agree there are reasons not to do this, but by forbidding it we're essentially saying there's never any reason to do this, which i don't think is true. for example, i have some anycast addresses in my network which are only on one node, and these work fine, although of course they are not currently configured as IFF_ANYCAST addresses due to this restriction.

my feeling here is that we shouldn't nanny the user; anyone configuring anycast addresses should be aware of the operational implications of doing this and make their own decision about it. currently, they're forced to use non-IFF_ANYCAST addresses to do this, which increases of risk of error from e.g. accidentally using an anycast address as the source of an outgoing connection due to source address selection algorithm. that seems definitely worse than using an anycast address.

+1, I think it's fine to let the user shoot their own foot so to speak. If they're configuring ANYCAST then they should know what that entails.

the solution in RFC 7094 §4.2 is quite clever, but does anyone currently implement either the client or server side of this? it seems like it would have implications for the security of TCP connections.

Maybe add a comment to replace the section you deleted covering that ANYCAST is fine now, see RFC xxxx ?

Oh that's a neat hack, but yeah, at that point you should be using anycast for service location, and then protocol level redirects (eg HTTP 3xx) to clean things up.

[ivy]

i discussed this a bit elsewhere and someone pointed out that if you want to use anycast for DNS then you have to enable TCP or your DNS server won't work properly. given how many people have successfully deployed anycast DNS, it seems like there's a strong operational argument to permit this.

In D50019#1140722, @adrian wrote:

Maybe add a comment to replace the section you deleted covering that ANYCAST is fine now, see RFC xxxx ?

i considered this but it seemed odd to have a comment with no associated code saying "don't do anything here since we don't disallow this". in exchange i tried to make the commit message as informative as possible for anyone who is curious :-)

[tuexen]

In D50019#1140724, @ivy wrote:

i discussed this a bit elsewhere and someone pointed out that if you want to use anycast for DNS then you have to enable TCP or your DNS server won't work properly. given how many people have successfully deployed anycast DNS, it seems like there's a strong operational argument to permit this.

I am aware that you use DNS/TCP in addition to DNS/UDP and I am aware that for DNS/UDP anycast is used. But are you saying that people deploy TCP in combination with anycast at scale?

[tuexen]

my feeling here is that we shouldn't nanny the user; anyone configuring anycast addresses should be aware of the operational implications of doing this and make their own decision about it. currently, they're forced to use non-IFF_ANYCAST addresses to do this, which increases of risk of error from e.g. accidentally using an anycast address as the source of an outgoing connection due to source address selection algorithm. that seems definitely worse than using an anycast address.

OK. But if TCP connections are failing, we can't do much about it.

the solution in RFC 7094 §4.2 is quite clever, but does anyone currently implement either the client or server side of this? it seems like it would have implications for the security of TCP connections.

In my view this breaks a couple of things. Therefore, I doubt that it will be implemented.
One could use a variant of MPTCP (which is not implemented in FreeBSD right now) for the TCP use case of the INIT forwarding for SCTP (which is easy to implement).

[ivy]

In D50019#1140725, @tuexen wrote:

are you saying that people deploy TCP in combination with anycast at scale?

yes, for example see: https://www.ripe.net/publications/docs/ripe-393/

The impact of this problem [i.e., stateful connections over anycast] is not clear: for example, in a study of J-root [5] the authors state that this is a serious problem and recommend that stateful services not be run on anycast at all. Other work has since concluded that the impact of node switches is not significant enough to be a concern [6, 12]. Our own results for K-root are presented in Section 4.3.

so, there are mixed results, but some people are using it successfully at scale.

[tuexen]

In D50019#1140737, @ivy wrote:

In D50019#1140725, @tuexen wrote:

are you saying that people deploy TCP in combination with anycast at scale?

yes, for example see: https://www.ripe.net/publications/docs/ripe-393/

The impact of this problem [i.e., stateful connections over anycast] is not clear: for example, in a study of J-root [5] the authors state that this is a serious problem and recommend that stateful services not be run on anycast at all. Other work has since concluded that the impact of node switches is not significant enough to be a concern [6, 12]. Our own results for K-root are presented in Section 4.3.

so, there are mixed results, but some people are using it successfully at scale.

I see. They are using short lived TCP connections for that time scale the anycast routing seems to be stable.

Thank you very much for sharing this document!