In the system where ifnets are often destroyed it is possible to
catch the race, when ND6 did already freed its resources, but ND6
callout timer fires nd6_llinfo_timer and it tries to get access
to ND6 data in the ifnet's if_afdata. This leads NULL pointer
dereference. Avoid this by freeing LLE table before ND6 resources.
Details
Details
Diff Detail
Diff Detail
- Repository
- rS FreeBSD src repository - subversion
- Lint
Lint Passed - Unit
No Test Coverage - Build Status
Buildable 42533 Build 39421: arc lint + arc unit
Event Timeline
Comment Actions
This leads to a lot of questions:
(a) can we check for the NULL pointer and gracefully handle it? Or at least add a KASSERT to document it?
(b) I guess I should go and look where the nd6_llinfo_timer() comes from,,, or let me ask: what drains the callout?
Changing the order to me sounds like hiding a different problem (which likely is a missing callout_drain)?
Comment Actions
Yes, this diff initially was part of D32811, @melifaro asked me to make it separate.
The panic that I'm trying to fix was in the line ndi = ND_IFINFO(ifp);:
Fatal trap 12: page fault while in kernel mode
cpuid = 6; apic id = 0c
fault virtual address = 0x10
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80d2b21c
stack pointer = 0x28:0xfffffe00d7e399c0
frame pointer = 0x28:0xfffffe00d7e39a50
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (swi4: clock (0))
trap number = 12
panic: page fault
cpuid = 6
time = 1633476489
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00d7e39680
vpanic() at vpanic+0x182/frame 0xfffffe00d7e396d0
panic() at panic+0x43/frame 0xfffffe00d7e39730
trap_fatal() at trap_fatal+0x387/frame 0xfffffe00d7e39790
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00d7e397e0
trap() at trap+0x271/frame 0xfffffe00d7e398f0
calltrap() at calltrap+0x8/frame 0xfffffe00d7e398f0
--- trap 0xc, rip = 0xffffffff80d2b21c, rsp = 0xfffffe00d7e399c0, rbp = 0xfffffe00d7e39a50 ---
nd6_llinfo_timer() at nd6_llinfo_timer+0x9c/frame 0xfffffe00d7e39a50
softclock_call_cc() at softclock_call_cc+0x139/frame 0xfffffe00d7e39b00
softclock() at softclock+0x79/frame 0xfffffe00d7e39b20
ithread_loop() at ithread_loop+0x24d/frame 0xfffffe00d7e39bb0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00d7e39bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d7e39bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 8d12h20m43s
Dumping 13931 out of 131002 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55 /usr/src/sys/amd64/include/pcpu_aux.h: No such file or directory.
(kgdb) bt
#0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:394
#2 0xffffffff80b377eb in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:481
#3 0xffffffff80b37c6a in vpanic (fmt=<optimized out>, ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:913
#4 0xffffffff80b37a83 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:839
#5 0xffffffff80efab67 in trap_fatal (frame=0xfffffe00d7e39900, eva=16) at /usr/src/sys/amd64/amd64/trap.c:915
#6 0xffffffff80efabbf in trap_pfault (frame=0xfffffe00d7e39900, usermode=<optimized out>, signo=<optimized out>, ucode=<optimized out>)
at /usr/src/sys/amd64/amd64/trap.c:732
#7 0xffffffff80efa221 in trap (frame=0xfffffe00d7e39900) at /usr/src/sys/amd64/amd64/trap.c:398
#8 <signal handler called>
#9 0xffffffff80d2b21c in nd6_llinfo_timer (arg=0xfffff80ad277a800) at /usr/src/sys/netinet6/nd6.c:775
#10 0xffffffff80b53a99 in softclock_call_cc (c=<optimized out>, cc=0xffffffff81878d00 <cc_cpu>, direct=0) at /usr/src/sys/kern/kern_timeout.c:703
#11 0xffffffff80b53ef9 in softclock (arg=0xffffffff81878d00 <cc_cpu>) at /usr/src/sys/kern/kern_timeout.c:823
#12 0xffffffff80af8f8d in intr_event_execute_handlers (p=<optimized out>, ie=0xfffff80101cee600) at /usr/src/sys/kern/kern_intr.c:1168
#13 ithread_execute_handlers (p=<optimized out>, ie=0xfffff80101cee600) at /usr/src/sys/kern/kern_intr.c:1181
#14 ithread_loop (arg=0xfffff80101ce6160) at /usr/src/sys/kern/kern_intr.c:1269
#15 0xffffffff80af5a3e in fork_exit (callout=0xffffffff80af8d40 <ithread_loop>, arg=0xfffff80101ce6160, frame=0xfffffe00d7e39c00)
at /usr/src/sys/kern/kern_fork.c:1052
#16 <signal handler called>
(kgdb) f 9
#9 0xffffffff80d2b21c in nd6_llinfo_timer (arg=0xfffff80ad277a800) at /usr/src/sys/netinet6/nd6.c:775
775 /usr/src/sys/netinet6/nd6.c: No such file or directory.
(kgdb) i lo
et = {et_link = {tqe_next = 0x0, tqe_prev = 0xfffffe00d7e26bd8}, et_td = 0xfffffe00d3e8d300, et_section = {bucket = 1}}
src = {__u6_addr = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}
ln = 0xfffff80ad277a800
ifp = 0xfffff811058dd800
saved_vnet = <optimized out>
send_ns = <optimized out>
ndi = <optimized out>
pdst = <optimized out>
dst = <optimized out>
do_switch = <optimized out>
psrc = <optimized out>
delay = <optimized out>
(kgdb) p *ifp
$1 = {if_link = {cstqe_next = 0xfffff811058fc000}, if_clones = {le_next = 0xfffff801520dc000, le_prev = 0xfffff811058fc008}, if_groups = {cstqh_first = 0x0,
cstqh_last = 0xfffff811058dd818}, if_alloctype = 6 '\006', if_numa_domain = 255 '\377', if_softc = 0xfffff81a34ef5300, if_llsoftc = 0x0, if_l2com = 0x0,
if_dname = 0xffffffff81395998 <vlanname> "vlan", if_dunit = 419, if_index = 16, if_index_reserved = 0, if_xname = "vlan419\000\000\000\000\000\000\000\000",
if_description = 0x0, if_flags = 2267394, if_drv_flags = 0, if_capabilities = 6817539, if_capenable = 524288, if_linkmib = 0x0, if_linkmiblen = 0, if_refcount = 0,
if_type = 135 '\207', if_addrlen = 6 '\006', if_hdrlen = 4 '\004', if_link_state = 0 '\000', if_mtu = 1500, if_metric = 0, if_baudrate = 10000000000,
if_hwassist = 0, if_epoch = 51486, if_lastchange = {tv_sec = 1633476486, tv_usec = 61710}, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0,
ifq_maxlen = 2048, ifq_mtx = {lock_object = {lo_name = 0xfffff811058dd858 "vlan419", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0},
ifq_drv_head = 0x0, ifq_drv_tail = 0x0, ifq_drv_len = 0, ifq_drv_maxlen = 0, altq_type = 0, altq_flags = 0, altq_disc = 0x0, altq_ifp = 0xfffff811058dd800,
altq_enqueue = 0x0, altq_dequeue = 0x0, altq_request = 0x0, altq_clfier = 0x0, altq_classify = 0x0, altq_tbr = 0x0, altq_cdnr = 0x0}, if_linktask = {ta_link = {
stqe_next = 0x0}, ta_pending = 0, ta_priority = 0 '\000', ta_flags = 0 '\000', ta_func = 0xffffffff80c44d00 <do_link_state_change>,
ta_context = 0xfffff811058dd800}, if_addmultitask = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0 '\000', ta_flags = 0 '\000',
ta_func = 0xffffffff80c44f80 <if_siocaddmulti>, ta_context = 0xfffff811058dd800}, if_addr_lock = {lock_object = {lo_name = 0xffffffff80f62755 "if_addr_lock",
lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, if_addrhead = {cstqh_first = 0x0, cstqh_last = 0xfffff811058dd9c8}, if_multiaddrs = {
cstqh_first = 0x0, cstqh_last = 0xfffff811058dd9d8}, if_amcount = 0, if_addr = 0xfffff818b493ab00, if_hw_addr = 0xfffff81d58d89100,
if_broadcastaddr = 0xffffffff81394ab0 <etherbroadcastaddr> "\377\377\377\377\377\377", if_afdata_lock = {lock_object = {lo_name = 0xffffffff80fd2876 "if_afdata",
lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, if_afdata = {0x0 <repeats 43 times>}, if_afdata_initialized = 0, if_fib = 0,
if_vnet = 0xfffff801012da8c0, if_home_vnet = 0xfffff801012da8c0, if_vlantrunk = 0x0, if_bpf = 0xffffffff818c5128 <dead_bpf_if>, if_pcount = 0, if_bridge = 0x0,
if_lagg = 0x0, if_pf_kif = 0x0, if_carp = 0x0, if_label = 0x0, if_netmap = 0x0, if_output = 0xffffffff80c4ecb0 <ifdead_output>,
if_input = 0xffffffff80c4ecd0 <ifdead_input>, if_bridge_input = 0x0, if_bridge_output = 0x0, if_bridge_linkstate = 0x0,
if_start = 0xffffffff80c4ece0 <ifdead_start>, if_ioctl = 0xffffffff80c4ecf0 <ifdead_ioctl>, if_init = 0xffffffff80c5a680 <vlan_init>,
if_resolvemulti = 0xffffffff80c4ed00 <ifdead_resolvemulti>, if_qflush = 0xffffffff80c4ed20 <ifdead_qflush>, if_transmit = 0xffffffff80c4ed30 <ifdead_transmit>,
if_reassign = 0xffffffff80c50350 <ether_reassign>, if_get_counter = 0xffffffff80c4ed50 <ifdead_get_counter>,
if_requestencap = 0xffffffff80c50280 <ether_requestencap>, if_counters = {0xfffffe03abc91140, 0xfffffe03abc91138, 0xfffffe03abc91130, 0xfffffe03abc91128,
0xfffffe03abc91120, 0xfffffe03abc91118, 0xfffffe03abc91110, 0xfffffe03abc91108, 0xfffffe03abc91100, 0xfffffe03abc910f8, 0xfffffe03abc910f0, 0xfffffe03abc910e8},
if_hw_tsomax = 65518, if_hw_tsomaxsegcount = 23, if_hw_tsomaxsegsize = 65536, if_snd_tag_alloc = 0xffffffff80c4ed60 <ifdead_snd_tag_alloc>,
if_snd_tag_modify = 0xffffffff80c4ed70 <ifdead_snd_tag_modify>, if_snd_tag_query = 0xffffffff80c4ed80 <ifdead_snd_tag_query>,
if_snd_tag_free = 0xffffffff80c4ed90 <ifdead_snd_tag_free>, if_ratelimit_query = 0xffffffff80c4eda0 <ifdead_ratelimit_query>, if_ratelimit_setup = 0x0,
if_pcp = 0 '\000', if_debugnet_methods = 0x0, if_epoch_ctx = {data = {0xffffffff80c45290 <if_destroy>, 0xfffff801d5acfdc0}}, if_ispare = {0, 0, 0, 0}}
(kgdb) p ifp->if_afdata
$2 = {0x0 <repeats 43 times>}
(kgdb) p ifp->if_afdata[28]
$3 = (void *) 0x0
(kgdb) p ifp->if_flags
$4 = 2267394
(kgdb) p/x ifp->if_flags
$5 = 0x229902
(kgdb)(b) I guess I should go and look where the nd6_llinfo_timer() comes from,,, or let me ask: what drains the callout?
Changing the order to me sounds like hiding a different problem (which likely is a missing callout_drain)?
lltable_free() will stop all LLE related callouts. And in time when we run nd6_ifdetach() LLE timer can not fire.