Hardware Trace (HWT) framework

[br]

Summary

Hardware Trace (HWT) v0.1

HWT framework manages hardware technology that collects information about software execution and records it as "events" in highly compressed format into DRAM. The events cover information about control flow changes of a program, whether branches taken or not, exceptions taken, timing information, cycles elapsed and more. This allowing us to restore entire program flow of a given application.

Initially targeting ARM Coresight technology included into ARM64 processors.

Part2: ARM Coresight Support
Part3: hwt(8) instrumentation tool
Part4: ARM Statistical Profiling Extension (SPE) support
Part5: Intel Processor Trace (PT) support

Test Plan

Tested on ARM Morello

[br]

sys/dev/hwt/hwt.c

656

Implemented, thanks!

sys/dev/hwt/hwt.h

51 ↗

(On Diff #123108)

I have re-implemented this part, so we now have just one context, but multiple threads within context. Each thread has its own trace buffers and character device /dev/hwt_%pid_%tid so that user can map trace buffers.

53 ↗

(On Diff #123108)

It is passed from userspace to kernel, so why not

[br]

ip range filtering (draft) added

[br]

suspend specific thread on mmap, not entire proc

[br]

regenerate

[br]

Initial work on cpu/system mode of operation

[br]

Kernel-mode tracing support added.

[andrew]

sys/kern/imgact_elf.c

1598

How are the HWT_RECORD_EXECUTABLE and HWT_RECORD_INTERP records expected to be used? They use CheriBSD specific variables.

1603–1604

Should the addr and size be related? The address is the entry point and the size appears to be the size of the section, however the entry point may not be at the start of the section.

[br]

Remove cheribsd-specific "size" member from hwt_record as not needed

[br]

sys/kern/imgact_elf.c

1598

these are fed into pmcstat_image_link(). Size arg is not needed so I removing it. Should work on both FreeBSD/cheribsd with minor (non-functional) changes

1603–1604

thanks "size" was used for debugging only.
pmcstat_image_link() does not require size argument, so I've removed it

[br]

Allow tracing multiple cpus in CPU(system) mode

[br]

Code cleanup. Implement some TODOs

[br]

hwt kernel module added

[br]

Regenerate

[br]

regenerate

[br]

Regenerate

[br]

Fix context ID allocation, do not allocate twice and check the status of allocation

[br]

Add description for CPU mode of operation into hwt.c

[br]

Allow attaching to an existed proc

[br]

Regenerate

[br]

change backend lock, ident set lock, ownerhash lock to standard mtx

[andrew]

sys/sys/hwt.h

60

The cpuset_t could cause ABI issues when we bump MAXCPU. vmm.ko on amd64 was recently changed to not embed it in the struct in rGe17eca327633efc511450310afb5e4a662724e3d

[br]

Replace most of spin locks to standard mtx. Use refcounting in some places

[br]

Fix typo.

[br]

User provides cpusetsize together with *cpuset_t.

[br]

sys/sys/hwt.h

60

Thanks I have updated; cpusetsize is now supplied by a user; cpuset_t is now a pointer

[andrew]

Can you rebase the stack on FreeBSD so we can test on non-Morello hardware?

[br]

rebase for freebsd

[br]

Pass ctx to backend_deinit().

[zachary.leaf_arm.com]

sys/dev/hwt/hwt_context.c

71

In what situations do we have more than one hwt_context? This allocates a unique ident from 0 to 1<<8 but as far as I can tell in both hwt_ioctl_alloc_mode_{thread,cpu} we only allocate a single context aka ident 0.

[br]

sys/dev/hwt/hwt_context.c

71

I guess this may happen if two users simultaneously using hwt. This is not possible in coresight, but possible in Intel PT and probably SPE ?

[zachary.leaf_arm.com]

sys/dev/hwt/hwt_context.c

71

Ah yep - okay makes sense. It should be possible to run Coresight and SPE simultaneously at least.

On SPE everything is per core, so you *should* be able to trace on all cores separately. Two users tracing the same core would not be possible. Not sure how that would work with THREAD_MODE, as would depend on where the task is scheduled - I guess no concurrent SPE users will be possible there. I've only implemented CPU_MODE so far.

[br]

Regenerate

[br]

Regenerate

[andrew]

How do you handle different conflicting modes? e.g. a user is tracing a pid & another tries to enable thread tracing. The hardware may not be able to perform both operations.

[br]

Regenerate

[br]

Prevent tracing the same proc by multiple users.

[br]

In D40466#1056428, @andrew wrote:

How do you handle different conflicting modes? e.g. a user is tracing a pid & another tries to enable thread tracing. The hardware may not be able to perform both operations.

I have added a session counter to coresight to ensure it is used for 1 simultaneous sessions at max (regardless of type). @bnovkov is currently working on similar arrangements for Intel PT depending on hardware capabilities.
Generally we could allow thread tracing for multiple users simultaneously in Intel PT.
Coresight is more restricted disallowing us to continue a trace session from specific place in buffer, so we completely disallow multiple users to trace simultaneously.

[markj]

I don't see any documentation for this, is there a separate review for a man page somewhere?

sys/kern/kern_hwt.c

43

We definitely don't need all of these includes.

sys/kern/vfs_vnops.c

2938

We should only do this work if the hook is actually enabled.

[markj]

sys/dev/hwt/hwt_hook.h

29

These tags shouldn't appear in new code.

[markj]

sys/vm/vm_mmap.c

1067

How is this used? You are not passing the size of the mapping anywhere, which seems wrong.

[br]

• Remove $FreeBSD$ tags
• Remove unused include headers
• Check if HWT hooks are installed before constructing data structures

sys/vm/vm_mmap.c

1067

The address of mapping passed to pmcstat. It loads elf file based on mapping address (fullpath). So it knows the size of mapping. See pmcstat_image_link().

[br]

Resubmit updated patch