Enable ASLR by default for 64-bit executables.

[mw]

Summary

Address Space Layout Randomization (ASLR) is an exploit mitigation technique implemented in the majority of modern operating systems. It involves randomly positioning the base address of an executable and the position of libraries, heap, and stack, in a process’s address space. Although over the years ASLR proved to not guarantee full OS security on its own, this mechanism can make exploitation more difficult.

Tests on the tier 1 64-bit architectures demonstrated that the ASLR is stable and does not result in noticeable performance degradation, therefore it should be safe to enable this mechanism by default. Moreover its effectiveness is increased for PIE (Position Independent Executable) binaries. Thanks to commit 9a227a2fd642 (“Enable PIE by default on 64-bit architectures”), building from src is not necessary to have PIE binaries. It is enough to control usage of ASLR in the OS solely by setting the appropriate sysctls.

This patch toggles the kernel settings to use address map randomization for PIE & non-PIE 64-bit binaries. It also disables SBRK, in order to allow utilization of the bss grow region for mappings. The latter has no effect if ASLR is disabled, so apply it to all architectures.

As for the drawbacks, a consequence of using the ASLR is more significant VM fragmentation, hence the issues may be encountered in the systems with a limited address space in high memory consumption cases, such as buildworld. As a result, although the tests on 32-bit architectures with ASLR enabled were mostly on par with what was observed on 64-bit ones, the defaults for the
former are not changed at this time. Also, for the sake of safety keep the feature disabled for 32-bit executables on 64-bit machines, too.

The committed change affects the overall OS operation, so the following should be taken into consideration:

• Address space fragmentation.
• A changed ABI due to modified layout of address space.
• More complicated debugging due to:
  • Non-reproducible address space layout between runs.
  • Some debuggers automatically disable ASLR for spawned processes, making target’s environment different between debug and non-debug runs.

In order to confirm/rule-out the dependency of any encountered issue on ASLR it is strongly advised to re-run the test with the feature disabled - it can be done by setting the following sysctls in the /etc/sysctl.conf file:
kern.elf64.aslr.enable=0
kern.elf64.aslr.pie_enable=0

Test Plan

Validation summary of the ALSR/PIE feature enablement in FreeBSD-HEAD:
https://drive.google.com/file/d/1Ujdiv2aN9kiD0dnjvH7VlqpwOPdA-xDK

It comprises a verification whether the change introduces any functional/performance regression in the OS. It also presents a comparison of the results between all used setups:

• amd64 desktop
• amd64 server
• arm64 server
• armv7 board

Following test cases were executed:

• buildworld
• kyua
• WRK
• openssl speed

[val_packett.cool]

sys/kern/imgact_elf.c

195

Since https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239873 is not yet resolved, maybe disable stackgap out of the box for now?

[mw]

sys/kern/imgact_elf.c

195

Thank you @kib for immediate patch (https://svnweb.freebsd.org/changeset/base/368772)

[mw]

Hi! Any thoughts about the patch? Do you have objections to get it merged?

[mw]

Hi! Any thoughts? If no objections, I plan to merge the patch after February 5th.

[emaste]

Related PRs/reviews to check out:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252629
D28139
D28140

[mw]

In D27666#628869, @emaste wrote:

Related PRs/reviews to check out:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252629
D28139
D28140

Thanks @emaste. Is there any part requiring additional work / contribution? If yes, please reach me out, so we can sync and help with whatever is needed.

[emaste]

In D27666#629463, @mw wrote:

Thanks @emaste. Is there any part requiring additional work / contribution? If yes, please reach me out, so we can sync and help with whatever is needed.

Mainly I think we need to make a concerted effort towards identifying ports that require ELF feature tags to opt out of various features, and some common mechanism (either infrastructure or template approach) for setting those bits. I submitted PR252629

As above PR239873 reports issues with firefox and thunderbird with ASLR stack gap. I found libreoffice is incompatible with W^X and submitted PR252689 for that.

[emaste]

share/mk/bsd.opts.mk

64 ↗

(On Diff #80880)

IMO we should tackle this change first, in isolation, and commit this part ASAP.

There is a somewhat of an open question if we should change this for i386 (since i386 is register-starved and PIE is relatively more costly there compared to amd64, and because the primary motivation for building PIE is to enable ASLR, which is not particularly interesting on 32-bit archs).

[mw]

share/mk/bsd.opts.mk

64 ↗

(On Diff #80880)

IMO we should tackle this change first, in isolation, and commit this part ASAP.

I will prepare a revision for that then. I think it would be useful to have it in HEAD an later rely only on the sysctl knob settings.

There is a somewhat of an open question if we should change this for i386 (since i386 is register-starved and PIE is relatively more costly there compared to amd64, and because the primary motivation for building PIE is to enable ASLR, which is not particularly interesting on 32-bit archs).

Apart from the memory starvation during buildworld, 32-bit seem to work but I agree it may be better to omit them (at least for now).

About the change itself I'm thinking about adding below in share/mk/src.opts.mk:

.if ${__T} == "aarch64" || ${__T} == "amd64" || ${__T:Mmips64*} || ${__T} == "powerpc64"
__DEFAULT_YES_OPTIONS+=PIE
.else
__DEFAULT_NO_OPTIONS+=PIE
.endif

What do you think?

[emaste]

share/mk/bsd.opts.mk

64 ↗

(On Diff #80880)

I think that would be fine with RISC-V added.

[mw]

In D27666#629538, @emaste wrote:

In D27666#629463, @mw wrote:

Thanks @emaste. Is there any part requiring additional work / contribution? If yes, please reach me out, so we can sync and help with whatever is needed.

Mainly I think we need to make a concerted effort towards identifying ports that require ELF feature tags to opt out of various features, and some common mechanism (either infrastructure or template approach) for setting those bits. I submitted PR252629

As above PR239873 reports issues with firefox and thunderbird with ASLR stack gap. I found libreoffice is incompatible with W^X and submitted PR252689 for that.

WRT ports do you see a better option than enable PIE/ASLR by default, let the community / port maintainers identify problems, create PR's and opt-out this option until fixed?

[kbowling]

In D27666#629768, @mw wrote:

In D27666#629538, @emaste wrote:

In D27666#629463, @mw wrote:

Thanks @emaste. Is there any part requiring additional work / contribution? If yes, please reach me out, so we can sync and help with whatever is needed.

Mainly I think we need to make a concerted effort towards identifying ports that require ELF feature tags to opt out of various features, and some common mechanism (either infrastructure or template approach) for setting those bits. I submitted PR252629

As above PR239873 reports issues with firefox and thunderbird with ASLR stack gap. I found libreoffice is incompatible with W^X and submitted PR252689 for that.

WRT ports do you see a better option than enable PIE/ASLR by default, let the community / port maintainers identify problems, create PR's and opt-out this option until fixed?

portmgr@ can do an exp-run make a bugzilla with the patch and request similar to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214864

[mw]

Update revision after splitting PIE enablement to a separate patch (https://reviews.freebsd.org/D28328)

[dgr_semihalf.com]

I created D29550, D29551, D29552 and D29553, which allow us to disable stack gap for ntpd during build.

[mw]

Hi,

I'm refreshing the discussion. The current status is following:

1. PIE enabled by default in 64-builds.
2. Ports' exp-run issues are fixed (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214864)
3. Fixes for the outstanding bugs ntpd (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253208) and Firefox/Thunderbird (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239873) landed on Friday. Hopefully this will cover all cases that might have remained unknown until now.

Since IMO its a good timing for such change (still a decent amount of time ahead before 14.0 release), how about merging this patch and toggle ASLR to become enabled by default/opt-out for 64-bit platforms? Any thougts, objections?

Thanks,
Marcin

[lattera-gmail.com]

In D27666#734443, @mw wrote:

Hi,

I'm refreshing the discussion. The current status is following:

3. Fixes for the outstanding bugs ntpd (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253208) and Firefox/Thunderbird (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239873) landed on Friday. Hopefully this will cover all cases that might have remained unknown until now.

It seems like a problem with the underlying AS{L}R implementation. HardenedBSD has not needed to make any changes to any application since it completed its PaX-inspired ASLR implementation in 2015. If applications experience issues with FreeBSD's AS{L}R implementation, it'd be safe to assume a problem with the AS{L}R implementation, not the application.

[mw]

In D27666#734447, @lattera-gmail.com wrote:

In D27666#734443, @mw wrote:

Hi,

I'm refreshing the discussion. The current status is following:

3. Fixes for the outstanding bugs ntpd (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253208) and Firefox/Thunderbird (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239873) landed on Friday. Hopefully this will cover all cases that might have remained unknown until now.

It seems like a problem with the underlying AS{L}R implementation. HardenedBSD has not needed to make any changes to any application since it completed its PaX-inspired ASLR implementation in 2015. If applications experience issues with FreeBSD's AS{L}R implementation, it'd be safe to assume a problem with the AS{L}R implementation, not the application.

Actually none of the application required fixing. The mentioned patches improved propagation of the stack gap information from the kernel to user space.

[emaste]

sys/kern/imgact_elf.c

170–174

It seems a little strange to check __ELF_WORD_SIZE and then use the __elfN macro anyway, although I think it's going to be awkward either way.

185–190

does this have any effect if enable_aslr (and pie equiv) is disabled anyhow?

[imp]

sys/kern/imgact_elf.c

170–174

Just keep the old line and replace 0 with the #if clause.

176–179

Same

[mw]

sys/kern/imgact_elf.c

176–179

I changed the default, as this variable is ignored in case ASLR is disabled.

185–190

Good point. Both auxiliary sysctls (pie + sbrk) are respected only in case the ASLR is enabled, so in order to simplify the code I toggled the default values unconditionally.

[mw]

Hi! Any comments/remarks to the updated version?

[dgr_semihalf.com]

sys/kern/imgact_elf.c

185–190

Actually, while sbrk and stack gap sysctls are respected only when ASLR is enabled, both pie_enable and enable control if ASLR is actually enabled. If aslr.pie_enabled is set to 1, then ASLR is enabled for PIE binaries (ET_DYN) and if aslr.enable is set to 1, then ASLR is enabled for non-PIE binaries (ET_EXEC). This can be seen in: https://cgit.freebsd.org/src/tree/sys/kern/imgact_elf.c#n1188 and https://cgit.freebsd.org/src/tree/sys/kern/imgact_elf.c#n1230.

[mw]

sys/kern/imgact_elf.c

185–190

You are right, thanks. I will submit the updated setting of __elfN(pie_aslr_enabled)

[mw]

Limit setting of __elfN(pie_aslr_enabled) for only 64-bit PIE binaries.

[kib]

I think it is better to provide short and concise list of potential issues with ASLR, like this:

• changed ABI due to modified layout of address space
• address space fragmentation
• non-reproducable address space layout between runs
• harder debugging
• some debuggers automatically disable ASLR for spawned targets, making target' environment different between debug and non-debug runs

What else?

[kib]

I suggest also dropping the

In case any change in the OS behavior is observed, that can be possibly
caused by this patch, it is recommended to use [e-mail]
mailing list for reporting and discussing the encountered issue. Also,

wording.

[mw]

In D27666#744903, @kib wrote:

I suggest also dropping the

In case any change in the OS behavior is observed, that can be possibly
caused by this patch, it is recommended to use [e-mail]
mailing list for reporting and discussing the encountered issue. Also,

wording.

Would you like to leave only:

In order to confirm/rule-out the dependency of any encountered issue on ASLR
it is strongly advised to re-run the test with the feature disabled - it can be done
by setting the following sysctls in the /etc/sysctl.conf file:
kern.elf64.aslr.enable=0
kern.elf64.aslr.pie_enable=0

Or drop this section entirely?

[kib]

Yes, leave it. I think it is verbose but explicit so that more people can notice that if pointed to.

[mw]

In D27666#744905, @kib wrote:

Yes, leave it. I think it is verbose but explicit so that more people can notice that if pointed to.

Done.

[emaste]

As a result, although the tests on 32-bit architectures with ASLR enabled were
mostly on par with what was observed on 64-bit ones, the defaults for the
former are not changed. Also, for the sake of safety keep the feature disabled for 32-bit
executables on 64-bit machines, too.

I might write are not changed at this time to suggest this is not necessarily a final decision (in fact, it's not really a decision at all, 32-bit is probably not relevant enough to spend much effort on).

[mw]

In D27666#744939, @emaste wrote:

As a result, although the tests on 32-bit architectures with ASLR enabled were
mostly on par with what was observed on 64-bit ones, the defaults for the
former are not changed. Also, for the sake of safety keep the feature disabled for 32-bit
executables on 64-bit machines, too.

I might write are not changed at this time to suggest this is not necessarily a final decision (in fact, it's not really a decision at all, 32-bit is probably not relevant enough to spend much effort on).

Done.

[kib]

In D27666#744939, @emaste wrote:

I might write are not changed at this time to suggest this is not necessarily a final decision (in fact, it's not really a decision at all, 32-bit is probably not relevant enough to spend much effort on).

For 32bit, it might make sense to enable ASLR for 32bit binaries on 64bit host, still. That said, i386 kernel provides almost full 4G for UVA, so it might make sense to enable there as well, but lets limit the change to 64bit kernels, indeed.

[mw]

In D27666#744977, @kib wrote:

In D27666#744939, @emaste wrote:

I might write are not changed at this time to suggest this is not necessarily a final decision (in fact, it's not really a decision at all, 32-bit is probably not relevant enough to spend much effort on).

For 32bit, it might make sense to enable ASLR for 32bit binaries on 64bit host, still. That said, i386 kernel provides almost full 4G for UVA, so it might make sense to enable there as well, but lets limit the change to 64bit kernels, indeed.

Altough I agree it should be safe to enable ASLR 32-bit, for now I'd stay with 64-bit only - please remember that after a discussion it was decided to compile only 64-bit executables "WITH_PIE".

[kib]

In D27666#745003, @mw wrote:

Altough I agree it should be safe to enable ASLR 32-bit, for now I'd stay with 64-bit only - please remember that after a discussion it was decided to compile only 64-bit executables "WITH_PIE".

WITH_PIE is for different reasons, basically 1. matching the decision to not enable ASLR on 32bit kernels 2, desire to somewhat match 32bit build results on 64 bit host. There is only ldd32 32bit binary anyway.

[emaste]

I did a minor edit on the proposed commit message to clarify some things (sorry I do not have it as a diff)

Address Space Layout Randomization (ASLR) is an exploit mitigation technique implemented in the majority of modern operating systems. It involves randomly positioning the base address of an executable and the position of libraries, heap, and stack, in a process’s address space. Although over the years ASLR proved to not guarantee full OS security on its own, this mechanism can make exploitation more difficult.

Tests on the tier 1 64-bit architectures demonstrated that the ASLR is stable and does not result in noticeable performance degradation, therefore it should be safe to
enable this mechanism by default. Moreover its effectiveness is increased for PIE (Position Independentent Executable) binaries. Thanks to commit 9a227a2fd642 (“Enable PIE by default on 64-bit architectures”), building from src is not necessary to have PIE binaries. It is enough to control usage of ASLR in the OS solely by setting the appropriate sysctls.

This patch toggles the kernel settings to use address map randomization for PIE & non-PIE 64-bit binaries. It also disables SBRK, in order to allow utilization of the bss grow region for mappings. The latter has no effect if ASLR is disabled, so apply it to all architectures.

As for the drawbacks, a consequence of using the ASLR is more significant VM fragmentation, hence the issues may be encountered in the systems with a limited address space in high memory consumption cases, such as buildworld. As a result, although the tests on 32-bit architectures with ASLR enabled were mostly on par with what was observed on 64-bit ones, the defaults for the
former are not changed at this time. Also, for the sake of safety keep the feature disabled for 32-bit executables on 64-bit machines, too.

The committed change affects the overall OS operation, so the following should be taken into consideration:

• Address space fragmentation.
• A changed ABI due to modified layout of address space.
• More complicated debugging due to:
  • Non-reproducible address space layout between runs.
  • Some debuggers automatically disable ASLR for spawned processes, making target’s environment different between debug and non-debug runs.

In order to confirm/rule-out the dependency of any encountered issue on ASLR it is strongly advised to re-run the test with the feature disabled - it can be done by setting the following sysctls in the /etc/sysctl.conf file:

kern.elf64.aslr.enable=0
kern.elf64.aslr.pie_enable=0