Page MenuHomeFreeBSD
Differential D29825

tcp: keep SACK scoreboard sorted when doing rescue retransmission
ClosedPublic
Actions

Authored by rscheff on Apr 18 2021, 6:48 PM.
Tags
None
Referenced Files
F102843502: D29825.diff
Sun, Nov 17, 9:45 PM
Unknown Object (File)
Mon, Nov 11, 4:26 PM
Unknown Object (File)
Wed, Nov 6, 10:03 AM
Unknown Object (File)
Sat, Nov 2, 7:07 PM
Unknown Object (File)
Sep 27 2024, 3:41 AM
Unknown Object (File)
Sep 19 2024, 10:52 PM
Unknown Object (File)
Sep 19 2024, 6:01 AM
Unknown Object (File)
Sep 18 2024, 5:00 PM
View All 60 Files
Subscribers
imp
melifaro

Details

Reviewers
pho
tuexen
kbowling
jtl
thj
rrs
Group Reviewers
transport
Commits
rGd76eb0da7283: tcp: keep SACK scoreboard sorted when doing rescue retransmission
rGb87cf2bc841b: tcp: keep SACK scoreboard sorted when doing rescue retransmission
Summary

Once a rescue retransmission was prepared and sent, subsequent
SACK ranges may need to be inserted just prior of the hole
covering this rescue retransmission, rather than at the tail.
Failing to maintain a strictly ordered scoreboard will result
in spurious retransmissions, or a KASSERT when invariants are
active.

MFC after: 3 days
Sponsored by: NetApp, Inc.

Test Plan

Run /tools/test/stress2/misc/jumbo.sh. The high load
can result in CPU affinity shifts, looking like more
or less severe reordering and partial ACKs while
SACK loss recovery is still ongoing. Without this fix,
"panic: tcp_output: sack block to the left of una"
may be observed.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

rscheff created this revision.Apr 18 2021, 6:48 PM
Herald added a reviewer: transport. · View Herald TranscriptApr 18 2021, 6:48 PM
Herald added subscribers: melifaro, imp. · View Herald Transcript
Harbormaster completed remote builds in B38673: Diff 87668.Apr 18 2021, 6:48 PM
rscheff requested review of this revision.Apr 18 2021, 6:48 PM
tuexen accepted this revision.Apr 18 2021, 7:02 PM
This revision is now accepted and ready to land.Apr 18 2021, 7:02 PM
kbowling accepted this revision.Apr 18 2021, 7:40 PM
pho added a comment.Apr 18 2021, 8:47 PM
20210418 22:25:18 all (1/1): jumbo.sh
stress2: pts leak: 1/2
20210418 22:26:21 all (1/1): jumbo.sh
20210418 22:27:20 all (1/1): jumbo.sh
20210418 22:28:20 all (1/1): jumbo.sh
20210418 22:29:20 all (1/1): jumbo.sh
20210418 22:30:20 all.sh done, elapsed 0 day(s), 00:05.03

20210418 22:33:17 all (1/1): tcp4.sh
witness_lock_list_get: witness exhausted
Expensive timeout(9) function: 0xffffffff80e04fe0(0xfffffe01441b34d8) 0.011586722 s
Expensive timeout(9) function: 0xffffffff80e04fe0(0xfffffe016594a0c0) 0.041485479 s
Expensive timeout(9) function: 0xffffffff80f5b210(0) 0.058046346 s
Expensive timeout(9) function: 0xffffffff80e04fe0(0xfffffe016709a478) 0.136817083 s
Apr 18 22:36:54 mercat1 su[80814]: pho to root on /dev/pts/1
panic: tcp_output: sack block to the left of una : -3778580
cpuid = 2
time = 1618778260
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00e49b0530
vpanic() at vpanic+0x181/frame 0xfffffe00e49b0580
panic() at panic+0x43/frame 0xfffffe00e49b05e0
tcp_output() at tcp_output+0x2b8b/frame 0xfffffe00e49b07c0
tcp_do_segment() at tcp_do_segment+0x3246/frame 0xfffffe00e49b08b0
tcp_input_with_port() at tcp_input_with_port+0xc13/frame 0xfffffe00e49b0a00
tcp_input() at tcp_input+0xb/frame 0xfffffe00e49b0a10
ip_input() at ip_input+0x194/frame 0xfffffe00e49b0aa0
swi_net() at swi_net+0x1a1/frame 0xfffffe00e49b0b20
ithread_loop() at ithread_loop+0x279/frame 0xfffffe00e49b0bb0
fork_exit() at fork_exit+0x80/frame 0xfffffe00e49b0bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49b0bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 12 tid 100100 ]
Stopped at      kdb_enter+0x37: movq    $0,0x128410e(%rip)
db> x/s version
version:        FreeBSD 14.0-CURRENT #1 main-n246155-b6a572d03f6-dirty: Sun Apr 18 22:20:42 CEST 2021\012    pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012
db>
tuexen added a comment.Apr 18 2021, 8:53 PM
In D29825#669253, @pho wrote:
20210418 22:25:18 all (1/1): jumbo.sh
stress2: pts leak: 1/2
20210418 22:26:21 all (1/1): jumbo.sh
20210418 22:27:20 all (1/1): jumbo.sh
20210418 22:28:20 all (1/1): jumbo.sh
20210418 22:29:20 all (1/1): jumbo.sh
20210418 22:30:20 all.sh done, elapsed 0 day(s), 00:05.03

20210418 22:33:17 all (1/1): tcp4.sh
witness_lock_list_get: witness exhausted
Expensive timeout(9) function: 0xffffffff80e04fe0(0xfffffe01441b34d8) 0.011586722 s
Expensive timeout(9) function: 0xffffffff80e04fe0(0xfffffe016594a0c0) 0.041485479 s
Expensive timeout(9) function: 0xffffffff80f5b210(0) 0.058046346 s
Expensive timeout(9) function: 0xffffffff80e04fe0(0xfffffe016709a478) 0.136817083 s
Apr 18 22:36:54 mercat1 su[80814]: pho to root on /dev/pts/1
panic: tcp_output: sack block to the left of una : -3778580
cpuid = 2
time = 1618778260
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00e49b0530
vpanic() at vpanic+0x181/frame 0xfffffe00e49b0580
panic() at panic+0x43/frame 0xfffffe00e49b05e0
tcp_output() at tcp_output+0x2b8b/frame 0xfffffe00e49b07c0
tcp_do_segment() at tcp_do_segment+0x3246/frame 0xfffffe00e49b08b0
tcp_input_with_port() at tcp_input_with_port+0xc13/frame 0xfffffe00e49b0a00
tcp_input() at tcp_input+0xb/frame 0xfffffe00e49b0a10
ip_input() at ip_input+0x194/frame 0xfffffe00e49b0aa0
swi_net() at swi_net+0x1a1/frame 0xfffffe00e49b0b20
ithread_loop() at ithread_loop+0x279/frame 0xfffffe00e49b0bb0
fork_exit() at fork_exit+0x80/frame 0xfffffe00e49b0bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49b0bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 12 tid 100100 ]
Stopped at      kdb_enter+0x37: movq    $0,0x128410e(%rip)
db> x/s version
version:        FreeBSD 14.0-CURRENT #1 main-n246155-b6a572d03f6-dirty: Sun Apr 18 22:20:42 CEST 2021\012    pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012
db>

Is this with or without the patch in this review?

Closed by commit rGb87cf2bc841b: tcp: keep SACK scoreboard sorted when doing rescue retransmission (authored by rscheff). · Explain WhyApr 18 2021, 9:13 PM
This revision was automatically updated to reflect the committed changes.
rscheff added a commit: rGb87cf2bc841b: tcp: keep SACK scoreboard sorted when doing rescue retransmission.
pho added a comment.Apr 18 2021, 9:58 PM

db> x/s version
version: FreeBSD 14.0-CURRENT #1 main-n246155-b6a572d03f6-dirty: Sun Apr 18 22:20:42 CEST 2021\012 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012
db>




Is this with or without the patch in this review?



With.
pho added a comment.Apr 19 2021, 5:22 AM
With main-n246162-b87cf2bc841 I now get:



0210418 23:28:21 all (1/1): tcp4.sh
witness_lock_list_get: witness exhausted
Expensive timeout(9) function: 0xffffffff80e05ea0(0xfffffe015b24ec48) 0.013259969 s
Expensive timeout(9) function: 0xffffffff80e06f00(0xfffffe015ea62060) 0.033885131 s
panic: tcp_output: sack block to the left of una : -5546740
cpuid = 10
time = 1618781362
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00e49b0530
vpanic() at vpanic+0x181/frame 0xfffffe00e49b0580
panic() at panic+0x43/frame 0xfffffe00e49b05e0
tcp_output() at tcp_output+0x2b8b/frame 0xfffffe00e49b07c0
tcp_do_segment() at tcp_do_segment+0x3246/frame 0xfffffe00e49b08b0
tcp_input_with_port() at tcp_input_with_port+0xc13/frame 0xfffffe00e49b0a00
tcp_input() at tcp_input+0xb/frame 0xfffffe00e49b0a10
ip_input() at ip_input+0x194/frame 0xfffffe00e49b0aa0
swi_net() at swi_net+0x1a1/frame 0xfffffe00e49b0b20
ithread_loop() at ithread_loop+0x279/frame 0xfffffe00e49b0bb0
fork_exit() at fork_exit+0x80/frame 0xfffffe00e49b0bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e49b0bf0



https://people.freebsd.org/~pho/stress/log/log0095.txt
rscheff added a commit: rGd76eb0da7283: tcp: keep SACK scoreboard sorted when doing rescue retransmission.Apr 23 2021, 7:30 AM
Revision Contents
Changeset List
PathSize
sys/
netinet/
12 lines
Diff 87673
View Options
sys/netinet/tcp_sack.c
Loading...