Testing: add framework for the kernel unit tests

[melifaro]

Summary

Kernel testing can be cumbersome. Sometimes it is tedious to verify correctness of something once; it is much harder to do it continuously.
This changes intends to reduce the bar to the kernel unit-testing approach by introducing a new kernel-testing framework ("ktest") based on Netlink, loadable test modules and python test suite integration.

This framework provides the following kernel-testing features:

• Integration to the FreeBSD test suite
• Automatic test discovery
• Automatic test module loading
• Minimal boiler-plate code in both kernel and userland
• Passing any metadata to the test
• Convenient environment pre-setup using python testing framework
• Streaming messages from the kernel to the userland
• Running tests in the dedicated taskqueues
• Skipping or parametrizing tests

Test Plan

21:28 [0] m@devel3 s kyua list -k /usr/tests/examples/Kyuafile test_ktest_example.py
test_ktest_example.py:__test_cases_list__
21:28 [0] m@devel3 s kldload ktest_example
21:29 [0] m@devel3 s kyua list -k /usr/tests/examples/Kyuafile test_ktest_example.py
test_ktest_example.py:TestExample::test_failed
test_ktest_example.py:TestExample::test_failed2
test_ktest_example.py:TestExample::test_something
test_ktest_example.py:TestExample::test_something_else
test_ktest_example.py:TestExample::test_with_params[1_2_Sum]
test_ktest_example.py:TestExample::test_with_params[3_4_Sum]
21:29 [0] m@devel3 s kyua debug -k /usr/tests/examples/Kyuafile 'test_ktest_example.py:TestExample::test_something'
============================= test session starts ==============================
platform freebsd14 -- Python 3.9.16, pytest-7.2.1, pluggy-1.0.0 -- /usr/local/bin/python3.9
rootdir: /usr/tests/examples
collecting ... collected 1 item

../../../../usr/tests/examples/test_ktest_example.py::TestExample::test_something
0.000S ktest.c:run_test:229 start running test_something
0.000L ktest_example.c:test_something:36 I'm here, [test_something]
0.101L ktest_example.c:test_something:40 done
0.101E ktest.c:run_test:231 end running test_something
PASSED

============================== 1 passed in 0.11s ===============================
test_ktest_example.py:TestExample::test_something  ->  passed
21:29 [0] m@devel3 s kyua debug -k /usr/tests/examples/Kyuafile 'test_ktest_example.py:TestExample::test_with_params[3_4_Sum]'
============================= test session starts ==============================
platform freebsd14 -- Python 3.9.16, pytest-7.2.1, pluggy-1.0.0 -- /usr/local/bin/python3.9
rootdir: /usr/tests/examples
collecting ... collected 1 item

../../../../usr/tests/examples/test_ktest_example.py::TestExample::test_with_params[3_4_Sum]
0.000S ktest.c:run_test:229 start running test_with_params
0.000L ktest_example.c:test_with_params:100 Get 'test string'
0.000L ktest_example.c:test_with_params:102 3 + 4 = 7
0.000E ktest.c:run_test:231 end running test_with_params
PASSED

============================== 1 passed in 0.01s ===============================
test_ktest_example.py:TestExample::test_with_params[3_4_Sum]  ->  passed

[melifaro]

Add module autoload.

[melifaro]

Do not print message types other than failure.

[ngie]

sys/modules/ktest/ktest_example/Makefile

13

Spurious newline

sys/tests/ktest.c

46

Does sys/standarg.h exist?
(Maybe I’ve been playing around in Linux too long)

64

const char* (x2)?

189

If strcmp != 0, continue, and dedent the remaining blocks?
I have gotten into the habit of logic more like this because it seriously cuts down on the indentation which benefits python a lot.

sys/tests/ktest.h

6

Correct me if I’m wrong, but isn’t the shortened SPDX license header permitted in lieu of including the entire text?

47

Should this be size_t instead?

91

The nitems and __arraycount macros are equivalent (nitems is available in sys/params.h, IIRC).

tests/atf_python/atf_pytest.py

9

Relative imports are better, assuming this is a (namespace )? package. It makes it easier to lint items like this without installing everything beforehand.

tests/atf_python/ktest.py

117

What does tp mean? Can this variable please be unshortened for readers?

125

This honestly looks like it’s reinventing a lot of the logging module — could you please setup a logger for the package, then tune the Formatter to meet your desired output?

172

This looks pretty repetitive — also, log_msg.get_nla is called multiple times. Maybe using an intermediate variable that returns all of these values and cherry-picks them appropriately, e.g., dict/keyword formatting, would abbreviate things, or use the shortened name as a string literal and do the translation in the get_nla method.

tests/atf_python/sys/netlink/netlink_generic.py

2

Every time I see ctypes I cringe… it’s the worst of C and python IMHO: it’s unmanaged python that can crash the interpreter with bad memory accesses, can result in resource leaks, etc. CFFI expressed C extensions make a whole lot more sense because it does some C compiler level optimizations/checks and hides away a lot of the complexity of C/cpython resource management and boilerplate.

165

Why 4? It seems like this should be a constant.
(Similarly for the next few instances…)

[markj]

How does this relate to "testfrwk" (under sys/tests, I don't think it's been extended much since the original revision) and @rstone's sysunit framework?

[rstone]

In D39385#896786, @markj wrote:

How does this relate to "testfrwk" (under sys/tests, I don't think it's been extended much since the original revision) and @rstone's sysunit framework?

@melifaro has been bugging me periodically about sysunit but unfortunately I just don't have time to devote to it anymore (and Alexander, sorry for not replying to your email a couple of weeks back).

[melifaro]

Address first part of the comments from ngie.

[melifaro]

sys/tests/ktest.c

46

No. Other consumers in the kernel uses machine/ version as well.

64

It's possible, but I'm not sure I see any notable benefit from making them const. Nothing bad will happen if these strings are modified.

tests/atf_python/ktest.py

172

Sorry, I'm not sure I understood this right.
I agree GenlKtestMsgAttrType.KTEST_MSG_ATTR_FILE looks repetitive. I'll try to shorten the first part.
My intent is to keep the attribute (enum) names the same as in the C headers. It'll help the person to easily relate one to another (or even import those directly if CFFI finally manages to understand ifdefs).

I didn't understand the intermediate variable part - could you please elaborate?
Effectively we need to map the received TLVs to the (some) target structure, which is what's happening here.

tests/atf_python/sys/netlink/netlink_generic.py

2

I also wish I didn't have to do it, but there's no struct timespec in python: https://github.com/python/cpython/issues/67273 . I'm not opposed to CFFI, but my bar to switching lies somewhat higher.

[melifaro]

In D39385#896786, @markj wrote:

How does this relate to "testfrwk" (under sys/tests, I don't think it's been extended much since the original revision) and @rstone's sysunit framework?

It's a new development. As far as I understand, "testfrwk" is more targeted for running large-scale performance tests. There is no attribution of the tests output to the particular test and there is no test listing.
I'd love to see sysunit in FreeBSD (and it's superior to the approach in this diff, as running tests in userland adds much more possibilities) but it doesn't look we have enough resources.

This change is an example of something reasonably cheap (development-wise) and may be good enough, as it offers quite a low bar for writing unit (and some intergration) tests. At least I'm going to add some UTs for routing/netlink that I long wanted to have.

[melifaro]

In D39385#896810, @rstone wrote:

In D39385#896786, @markj wrote:

How does this relate to "testfrwk" (under sys/tests, I don't think it's been extended much since the original revision) and @rstone's sysunit framework?

@melifaro has been bugging me periodically about sysunit but unfortunately I just don't have time to devote to it anymore (and Alexander, sorry for not replying to your email a couple of weeks back).

No worries! ENOTIME is not a rare error :-)
Also - is the version on the github the latest&greatest?

[rstone]

In D39385#896855, @melifaro wrote:

Also - is the version on the github the latest&greatest?

It is the latest.

[melifaro]

Convert kernel-originated messages to the standard logger messages.

[melifaro]

fix formtting.

[melifaro]

remove now-unused code.

[melifaro]

sys/tests/ktest.h

6

I agree that the shortened version may look nicer, but as far as I understand, the preferred license format for the files in base is the full one, like /usr/share/examples/etc/bsd-style-copyright

tests/atf_python/atf_pytest.py

9

I don't have a hard opinion on this. Currently, it's a base-only "package" being installed by the FreeBSD makefile. In its current form, the directory structure there somewhat replicates the base system layout, resulting in directory depth being 3-4 items. The imports from the depth-most files may look a bit ugly.
Also, the other parts of the module use absolute imports. I'd suggest discussing it as a whole, to make the module look consistent. If you feel that the relative imports are the way to go, I'm happy to review/accept such a diff.

tests/atf_python/ktest.py

117

This part got removed.

125

Thank you, that's a good one!
I was focused on the C part at the time of writing and it didn't occurred to me :-)
Hope this version is better.

[melifaro]

Fix string formatting.

[melifaro]

I'd love to move forward with this revision. @ngie: do you think you're okay with the updated diff?

[melifaro]

I intend to commit this change on Thursday, Apr 13, unless I get any objections.