kyua: Add FreeBSD Jail execution environment support

[igoro]

Summary

A new Kyua concept is added -- "execution environment". A test can be
configured to be run within a specific environment. The test case
lifecycle is extended respectively:
- execenv init (creates a jail or does nothing for default
  execenv="host")
- test exec
- cleanup exec (optional)
- execenv cleanup (removes a jail or does nothing for default
  execenv="host")

The following new functionality is provided, from bottom to top:

1 ATF based tests

- The new "execenv" metadata property can be set to explicitly ask for
  an execution environment: "host" or "jail". If it's not defined, as
  all existing tests do, then it implicitly means "host".

- The new "execenv.jail.params" metadata property can be optionally
  defined to ask Kyua to use specific jail(8) parameters during creation
  of a temporary jail. An example is "vnet allow.raw_sockets".

  Kyua implicitly adds "children.max" to "execenv_jail_params"
  parameters with the maximum possible value. A test case can override
  it.

2 Kyuafile

- The same new metadata properties can be defined on Kyuafile level:
  "execenv" and "execenv_jail_params".

- Note that historically ATF uses dotted style of metadata naming, while
  Kyua uses underscore style. Hence "execenv.jail.params" vs.
  "execenv_jail_params".

3 kyua.conf, kyua CLI

- The new "execenvs" engine configuration variable can be set to a list
  of execution environments to run only tests designed for. Tests of not
  listed environments are skipped.

- By default, this variable lists all execution environments supported
  by a Kyua binary, e.g. execenvs="host jail".

- This variable can be changed via "kyua.conf" or via kyua CLI's "-v"
  parameter. For example, "kyua -v execenvs=host test" will run only
  host-based tests and skip jail-based ones.

- Current value of this variable can be examined with "kyua config".

Signed-off-by: Igor Ostapenko <[e-mail]>

[igoro]

This is a proposal to apply the jail Kyua patch directly to freebsd-src. The idea is to start battle testing and polishing sooner. While respective integration and synchronization with the github/freebsd/kyua project could go ahead with its own pace.

I've intentionally omitted Kyua build files update (autotools based) due to freebsd-src is not aligned with the latest build changes on the vendor side. It's expected to ease future vendor imports by minimizing potential merge conflicts.

The history of the work, for reference:

• The very first iterations: https://reviews.freebsd.org/D42350
• The latest iteration on the vendor side: https://github.com/freebsd/kyua/pull/224

[markj]

Having done some testing with this patch applied, I would like to go ahead with this and start converting tests to use the new feature. This lets us run many tests in parallel that otherwise do not support that, which means that we can get test results much more quickly than before. That's particularly helpful when debugging features like KASAN are enabled, since they inflate test runtimes quite a lot.

If there are no objections, I'll apply this patch to directly FreeBSD main in about a week and we can start annotating tests. Depending on how that goes, we can improve this feature in FreeBSD and reconcile with upstream kyua later once there's more confidence in the feature.

[imp]

I support moving forward with this plan. The risk seems to be in the new jail stuff, which should make some flakey tests today more reliable, so what's not to love about that?

[lwhsu]

I also support this plan, testing directly in FreeBSD CURRENT then going upstream sounds a good idea to give us more confidence. The only suggestion to me is that don't get this MFC'd before we merged in the upstream.

[kp]

Agreed. Let's get this in. We've wanted this for ages, Igor has done the work. Mark and I have both tested it and found it to work, and found it to achieve the results we'd hoped for from this feature as well.

[markj]

@igor.ostapenko_pm.me could you please send me a git-formatted patch that I can apply? I ran this through a build tinderbox and the regression test suite, and it's ready to be committed.

[ngie]

This modifies config file support slightly. Please bump the schema version and add appropriate tests for the new config schema vs the old config schema.

[markj]

In D45865#1049004, @ngie wrote:

This modifies config file support slightly. Please bump the schema version and add appropriate tests for the new config schema vs the old config schema.

This is a very vague request. I guess you are referring to syntax_version in engine/config.cpp. If so, then a bump is not necessary. Old versions of kyua simply reject unknown configuration variables. The addition of parallelism similarly did not require a version bump.