Differential D29832
ntpd: Change default stack limit to 4096 pages. Authored by dgr_semihalf.com on Apr 19 2021, 10:45 AM. Tags None Referenced Files
Subscribers
Details Default stack limit of 50 pages almost always causes ntpd to segfault Stack limit can be configured to previous value by adding
Diff Detail
Event TimelineComment Actions Does ntpd still mlocall(2) itself? Issue there, I believe, is that all this memory, including gap, is wired. Why not disable stack gap instead, with elfctl -e +noaslrstkgap ntpd (I think this is the right syntax)? Comment Actions As far as I know ntpd does not mlockall(2) istelf anymore. However, it still calls setrlimit to set maximum stack, which results in the process dying with default ASLR stack gap. This is clearly visible when using ktrace on nptd - setrlimit is the last thing that ntpd does before receiving SIGSEGV. Using elfctl to disable stack gap will work, I sent the patch proposing it earlier in D29553 (along with D29550, D29551 and D29552, which allow it to actually work), but @imp suggested to tweak the stack size. Comment Actions IMO using elfctl is the right approach. Do D29533 is fine with me (I cannot accept it because it is abandoned). For build system changes and perhaps for elfctl endianess change, you should ask maintainers. Comment Actions I re-opened D29553. I'm leaving this up as I don't know which will be the accepted method for solving the problem. Comment Actions We already disable stack gap at line 447 of ntpd.c. Upstream inserted this patch after we applied it locally.
Comment Actions Currently, ntpd does not start when ASLR is enabled, which means that calling procctl does not help. However, either disabling stackgap with elfctl or significantly increasing the stack limit help with this issue. If I understand correctly, when using procctl the stack gap still exists but we can write to it. Because of this gap our stack is bigger than the limit being set, so we're getting a segfault right after calling setrlimit. Elfctl, by comparison, stops stack gap from being created. However, I'm not 100% sure that this is what happens, as I'm not an expert when it comes to FreeBSD ASLR implementation. The problem can be easily reproduced by installing recent -CURRENT snapshot with enable_aslr selected in installer. Starting ntpd should then result in a segfault. The fact that this happens after setrlimit is clearly visible when using ktrace, so I'm attaching ktrace.out. I used the snapshot from 15th April. There is also a recent PR253208, which mentions this problem.
Comment Actions This is incorrect. I use ASLR and ntpd and have done so for a couple of years on -CURRENT.
There is no segfault after calling procctl(). The stack gap exists when ntpd starts. It calls procctl(), then forks itself disabling the stack gap and avoiding the segfault.
I have taken that bug and will look at it. I am not sure why that user is having a problem while I am not. The only thing that comes to mind might be a possible -n option. But, again, I'm running -CURRENT as of yesterday here and I have no such problem. PR/253208 will require more investigation. Comment Actions Ok, I can finally reproduce the problem with kern.elf64.aslr.pie_enable=1. PIE introduces a new set of problems that were not addressed by procctl() at the time. Whatever we do here will need to be pushed back to nwtime.org. Comment Actions Adjusting the stack to 200, like OpenBSD does, resolves this issue when PIE is enabled. case "$host" in
*-*-openbsd*)
ans=200
;;
*) ans=50
;;
esacWe can either set stack to 200 in config.h (like upstream does for OpenBSD) or test the sysctl and if it's enabled adjust the stack to 200 and adjust it at around line 1060 of ntpd.c. I've reached out to the folks at nwtime.org for their input. The simplest is to adjust the stack to 200. Whatever we do here will need to be applied to the port for those who want to enable drivers that are not enabled in base. Comment Actions Juxtapositioned to this is that enabling PIE causes firefox to segfault but that's for another review. Anyhow, I'm ok with this if we make the above two changes.
Comment Actions Having done a couple of reboots since enabling PIE and increasing stack to 200, now after this last reboot about an hour ago it now fails regardless of how much stack I give it (even obscene amounts). Increasing stack to 200, 1024, 20000, 200000, all of which I've tried, will not work after this latest reboot. BTW, the following does not work during this boot since uptime: kern.elf64.aslr.stack_gap: 3 Notice that ASLR is disabled but PIE is enabled. But ASLR enabled and PIE disabled works: kern.elf64.aslr.stack_gap: 3 The only solution is D29553. Comment Actions The work around works on my sandbox machine today but does not on the laptop when PIE is enabled on both. Without PIE and with ASLR there is no need for the workaround. Comment Actions So, I've looked at this more and setting the stack size to 4096 pages should technically also resolve this problem. I tested this restarting the daemon multiple times and in my case it worked. However, the problem will occur randomly due to random nature of stack gap randomization. By default, the maximum size of stack gap is 15.36MiB on amd64, I'm not sure about other platforms. The default value of kern.elf64.aslr.stack_gap is 3, so in case of amd64, where the default stack limit is equal to 512MiB, that gives us the 15.36MiB number. I used 4096 as this means that we are setting the stack limit to 16MiB, which should leave enough stack for ntpd to work - I think that 15.56MiB is needed (15.36 of stack gap and 200K taken from the fact that we were limiting this to 200K previously), but I rounded this up to 16MiB. Setting rlimit through ntp.conf to a higher value will not work, as from what I've seen, ntpd first sets the stack limit to default value then reads the configuration file and then calls setrlimit again to set the value to the one specified in config. Of course, setting the ELF flag to disable ASLR stack gap as in D29553 will also work. I do in fact prefer the elfctl approach, however I will leave this revision up for a while.
Comment Actions Technically, adjusting the stack should work. It worked on my laptop after two reboots and after each of two more it didn't. But consistently works on my sandbox so far. Increasing the stack is an inconsistent solution.
Increasing the stack does work in ntp.conf because it forks itself after that. This is how procctl() works.
I think this is the stopgap solution until our upstream has had a chance to look at it. I made them aware of it. My closer contact there tells me their senior developer (perly) is aware of it now. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||