kyuafile.5: ATF metadata mapping reference

[igoro]

Summary

MFC after: 1 month

[igoro]

This is a follow-up to https://reviews.freebsd.org/D47671, where we discussed documenting of the mapping.

Also, it was discovered that required_disk_space is not mapped yet -- it will be a separate patch.

[ngie]

It honestly seems like this documentation should live in another place, e.g., a "test driver" [1] (kyua-atf-driver(4)?) manpage. Putting all of this information in a single manpage seems like it would really clutter up what's being explained here, and the more "test drivers" get bolted on to kyua, the more complex the documentation will become.

1. I'm not sold on the name "test driver", but it's the best I can offer for the concept right now.

contrib/kyua/doc/kyuafile.5.in

158

(picking a line)
Is this how the manpage maps values from ATF <-> kyua? This doesn't look like the right way to do it; each of these sections should probably reference atf-test-case(4), or something similar..

159

(picking a line)
Should this be .It Va (how does that look when rendered)?

[igoro]

In D48190#1099037, @ngie wrote:

It honestly seems like this documentation should live in another place, e.g., a "test driver" [1] (kyua-atf-driver(4)?) manpage. Putting all of this information in a single manpage seems like it would really clutter up what's being explained here, and the more "test drivers" get bolted on to kyua, the more complex the documentation will become.

1. I'm not sold on the name "test driver", but it's the best I can offer for the concept right now.

Agree, and it seems that we could go with a much more useful reference "just in place" for now and re-work it when it's time to cover a couple or more drivers. For now ATF & Kyua are very coupled from the perspective of our test suite, and it seems that lack of such quick reference does not allow test authors to squeeze maximum from both frameworks -- kyuafile.5 already states that ATF test case could override Kyua metadata, but it may be left non-obvious for a reader how exactly ATF naming is mapped. It could be guessed by comparing with atf-test-case.4 but the things like require.machine vs allowed_platforms can be tricky and need experimenting to make sure. Another issue is that Kyua could read Kyua-specific metadata from ATF side even if ATF knows nothing about it and it's not documented anyhow respectively, e.g. execenv and execenv_jail_params.

What do you think?

[ngie]

In D48190#1099473, @igoro wrote:

In D48190#1099037, @ngie wrote:

It honestly seems like this documentation should live in another place, e.g., a "test driver" [1] (kyua-atf-driver(4)?) manpage. Putting all of this information in a single manpage seems like it would really clutter up what's being explained here, and the more "test drivers" get bolted on to kyua, the more complex the documentation will become.

1. I'm not sold on the name "test driver", but it's the best I can offer for the concept right now.

Agree, and it seems that we could go with a much more useful reference "just in place" for now and re-work it when it's time to cover a couple or more drivers. For now ATF & Kyua are very coupled from the perspective of our test suite, and it seems that lack of such quick reference does not allow test authors to squeeze maximum from both frameworks -- kyuafile.5 already states that ATF test case could override Kyua metadata, but it may be left non-obvious for a reader how exactly ATF naming is mapped. It could be guessed by comparing with atf-test-case.4 but the things like require.machine vs allowed_platforms can be tricky and need experimenting to make sure. Another issue is that Kyua could read Kyua-specific metadata from ATF side even if ATF knows nothing about it and it's not documented anyhow respectively, e.g. execenv and execenv_jail_params.

What do you think?

I think submitting this upstream (freebsd/kyua) and filing an issue capturing the feedback/concerns would probably be ok.

[igoro]

In D48190#1100132, @ngie wrote:

I think submitting this upstream (freebsd/kyua) and filing an issue capturing the feedback/concerns would probably be ok.

Deal. Thanks.

contrib/kyua/doc/kyuafile.5.in

159

The man lists Kyua metadata properties and the corresponding ATF names are provided per metadata. It's rendered the same as Kyua metadata name.