Use the HSM SBI extension to start APs

[mhorne]

Summary

The addition of the HSM SBI extension to OpenSBI introduces a new
breaking change: secondary harts will remain parked in the firmware,
until they are brought up explicitly via sbi_hsm_hart_start(). Add
the call to do this, sending the secondary harts to mpentry.

If the HSM extension is not present, secondary harts are assumed to be
released by the firmware, as is the case for OpenSBI =< v0.6 and BBL.

On non-INVARIANTS kernels secondary harts may fail to start, but the
system will continue to boot and print a warning to the console. Harts
which are started but fail to come up during release_aps() will continue
to cause a panic.

[mhorne]

Update smp_after_idle_runnable to use mp_maxid. This way we can still free all
bootstacks even if the CPUs that came online have a sparse layout.

[markj]

sys/riscv/riscv/mp_machdep.c

301

Shouldn't it be <=? mp_maxid is a valid CPU ID.

[mhorne]

Use <= not <.

[markj]

sys/riscv/riscv/mp_machdep.c

458

What are the possible error conditions here?

IMO it is a bit weird to only allow bootup without INVARIANTS - I would use a tunable to control it instead. If we're not going to allow execution to continue with debug assertions on, it means that this code path is not properly tested, so the kernel is likely to eventually blow up if an error occurs in a !INVARIANTS kernel. I think arm64 does something similar though, so I don't object.

[mhorne]

sys/riscv/riscv/mp_machdep.c

458

I did get the idea from arm64, but you're right that it is a little weird. Maybe we should just go with one or the other; either panic every time or allow the system to proceed. It seems to me that it would be useful to establish common semantics for this across architectures, although that might end up being more work than benefit.

sbi_hsm_hart_start() can return the following errors:

SBI_ERR_INVALID_ADDRESS - start_addr is not valid
SBI_ERR_INVALID_PARAM - hartid is not a valid hartid/can't be started in supervisor mode
SBI_ERR_ALREADY_AVAILABLE - The given hartid is already started.
SBI_ERR_FAILED - other "unknown" error