Paths

Table of Contentst

Hardware Trace (HWT) framework
Needs ReviewPublic
Actions

Authored by br on Jun 7 2023, 3:02 PM.

Details

Reviewers

andrew
bnovkov
zachary.leaf_arm.com

Summary

Hardware Trace (HWT) v0.1

HWT framework manages hardware technology that collects information about software execution and records it as "events" in highly compressed format into DRAM. The events cover information about control flow changes of a program, whether branches taken or not, exceptions taken, timing information, cycles elapsed and more. This allowing us to restore entire program flow of a given application.

Initially targeting ARM Coresight technology included into ARM64 processors.

Part2: ARM Coresight Support
Part3: hwt(8) instrumentation tool
Part4: ARM Statistical Profiling Extension (SPE) support
Part5: Intel Processor Trace (PT) support

Test Plan

Tested on ARM Morello

Diff Detail

Lint

Lint Skipped

Unit

Tests Skipped

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Remove mtx_threads and mtx_records. Use mtx of entire context

Add hwt_hook variable and HWT_CALL_HOOK macro

Add short overview and context lifecycle.

Move hwt files to conf/files.

Move changes related to coresight to another review

In D40466#921159, @andrew wrote:

This looks like a kernel with HWT_HOOKS but not HWT wouldn't link as, for example, hwt_switch_in will be missing. HWPMC handles this by adding a function pointer and protects it with an epoch.

I added this functionality. I am not sure if epoch is needed. PMC have several macroses for hook invocation, only some of them require epoch.

We should handle multiple trace backends, e.g. CoreSight and SPE. One way to test multiple backends is to have an exception trace that logs exception info when taking one from userspace.

Yes, I added this. Looking forward to help with SPE.

sys/dev/hwt/hwt.c
349	I added option to user API so user have to specify buffer size for each thread on context allocation.
355–356	For mapping trace buffers to userspace. One device per each thread of target proc.
656	Implemented, thanks!
sys/dev/hwt/hwt.h
51 ↗	(On Diff #123108)	I have re-implemented this part, so we now have just one context, but multiple threads within context. Each thread has its own trace buffers and character device `/dev/hwt_%pid_%tid` so that user can map trace buffers.
53 ↗	(On Diff #123108)	It is passed from userspace to kernel, so why not

ip range filtering (draft) added

suspend specific thread on mmap, not entire proc

regenerate

Initial work on cpu/system mode of operation

Kernel-mode tracing support added.

andrew added inline comments.Jul 24 2023, 11:03 AM

sys/kern/imgact_elf.c
1611	How are the `HWT_RECORD_EXECUTABLE` and `HWT_RECORD_INTERP` records expected to be used? They use CheriBSD specific variables.
1616–1617	Should the addr and size be related? The address is the entry point and the size appears to be the size of the section, however the entry point may not be at the start of the section.

Remove cheribsd-specific "size" member from hwt_record as not needed

br added inline comments.Jul 24 2023, 1:12 PM

sys/kern/imgact_elf.c
1611	these are fed into pmcstat_image_link(). Size arg is not needed so I removing it. Should work on both FreeBSD/cheribsd with minor (non-functional) changes
1616–1617	thanks "size" was used for debugging only. pmcstat_image_link() does not require size argument, so I've removed it

Allow tracing multiple cpus in CPU(system) mode

Code cleanup. Implement some TODOs

hwt kernel module added

Regenerate

regenerate

Herald added a reviewer: shurd. · View Herald TranscriptAug 3 2023, 6:12 PM

Herald added a subscriber: glebius. · View Herald Transcript

Regenerate

br removed a reviewer: shurd.Aug 3 2023, 6:35 PM

br removed subscribers: glebius, lwhsu, andrew.

Fix context ID allocation, do not allocate twice and check the status of allocation

Add description for CPU mode of operation into hwt.c

br edited the summary of this revision. (Show Details)Aug 8 2023, 1:22 PM

br edited the test plan for this revision. (Show Details)

cy added a subscriber: cy.Aug 9 2023, 4:48 PM

cy removed a subscriber: cy.

Allow attaching to an existed proc

Regenerate

change backend lock, ident set lock, ownerhash lock to standard mtx

andrew added inline comments.Sep 6 2023, 10:18 AM

sys/sys/hwt.h
60	The `cpuset_t` could cause ABI issues when we bump `MAXCPU`. `vmm.ko` on amd64 was recently changed to not embed it in the struct in rGe17eca327633efc511450310afb5e4a662724e3d

Replace most of spin locks to standard mtx. Use refcounting in some places

Fix typo.

User provides cpusetsize together with *cpuset_t.

br added inline comments.Sep 6 2023, 3:42 PM

sys/sys/hwt.h
60	Thanks I have updated; cpusetsize is now supplied by a user; cpuset_t is now a pointer

Can you rebase the stack on FreeBSD so we can test on non-Morello hardware?

rebase for freebsd

Pass ctx to backend_deinit().

mhorne mentioned this in D35709: fix pmcstat .Oct 10 2023, 5:24 PM

zachary.leaf_arm.com added inline comments.Oct 12 2023, 9:01 AM

sys/dev/hwt/hwt_context.c
71	In what situations do we have more than one `hwt_context`? This allocates a unique ident from 0 to 1<<8 but as far as I can tell in both `hwt_ioctl_alloc_mode_{thread,cpu}` we only allocate a single context aka ident 0.

br added inline comments.Oct 12 2023, 9:24 AM

sys/dev/hwt/hwt_context.c
71	I guess this may happen if two users simultaneously using hwt. This is not possible in coresight, but possible in Intel PT and probably SPE ?

zachary.leaf_arm.com added inline comments.Oct 12 2023, 12:43 PM

sys/dev/hwt/hwt_context.c
71	Ah yep - okay makes sense. It should be possible to run Coresight and SPE simultaneously at least. On SPE everything is per core, so you should be able to trace on all cores separately. Two users tracing the same core would not be possible. Not sure how that would work with `THREAD_MODE`, as would depend on where the task is scheduled - I guess no concurrent SPE users will be possible there. I've only implemented `CPU_MODE` so far.

Regenerate

zachary.leaf_arm.com added a reviewer: zachary.leaf_arm.com.Mar 14 2024, 5:24 PM

br added a child revision: D46241: HWT: Arm Statistical Profiling Extension (SPE) support.Aug 7 2024, 2:50 PM

Regenerate

br edited the summary of this revision. (Show Details)Aug 7 2024, 3:01 PM

br edited the test plan for this revision. (Show Details)

How do you handle different conflicting modes? e.g. a user is tracing a pid & another tries to enable thread tracing. The hardware may not be able to perform both operations.

br added a parent revision: D46397: amd64: Intel Processor Trace support.Aug 21 2024, 2:49 PM

br edited the summary of this revision. (Show Details)

Regenerate

bnovkov mentioned this in D46397: amd64: Intel Processor Trace support.Mon, Oct 7, 4:38 PM

Revision Contents
Changeset List

Path

Size

sys/

conf/

files

16 lines

options

3 lines

dev/

hwt/

229 lines

72 lines

230 lines

38 lines

107 lines

77 lines

140 lines

44 lines

139 lines

46 lines

98 lines

56 lines

259 lines

37 lines

374 lines

47 lines

218 lines

44 lines

147 lines

42 lines

213 lines

64 lines

215 lines

48 lines

433 lines

kern/

2 lines

2 lines

2 lines

27 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

46 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

7 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

13 lines

17 lines

2 lines

2 lines

2 lines

2 lines

21 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

subr_busdma_bufalloc.c

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

10 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

32 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

90 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

24 lines

modules/

hwt/

Makefile

23 lines

sys/

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

103 lines

56 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

4 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

152 lines

2 lines

2 lines

2 lines

2 lines

11 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

vm/

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

14 lines

3 lines

69 lines

2 lines

2 lines

2 lines

2 lines

2 lines

2 lines

Hardware Trace (HWT) frameworkNeeds ReviewPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 125528

sys/conf/files

sys/conf/options

sys/dev/hwt/hwt.c

sys/dev/hwt/hwt_backend.h

sys/dev/hwt/hwt_backend.c

sys/dev/hwt/hwt_config.h

sys/dev/hwt/hwt_config.c

sys/dev/hwt/hwt_context.h

sys/dev/hwt/hwt_context.c

sys/dev/hwt/hwt_contexthash.h

sys/dev/hwt/hwt_contexthash.c

sys/dev/hwt/hwt_cpu.h

sys/dev/hwt/hwt_cpu.c

sys/dev/hwt/hwt_hook.h

sys/dev/hwt/hwt_hook.c

sys/dev/hwt/hwt_ioctl.h

sys/dev/hwt/hwt_ioctl.c

sys/dev/hwt/hwt_owner.h

sys/dev/hwt/hwt_owner.c

sys/dev/hwt/hwt_ownerhash.h

sys/dev/hwt/hwt_ownerhash.c

sys/dev/hwt/hwt_record.h

sys/dev/hwt/hwt_record.c

sys/dev/hwt/hwt_thread.h

sys/dev/hwt/hwt_thread.c

sys/dev/hwt/hwt_vm.h

sys/dev/hwt/hwt_vm.c

sys/kern/genoffset.c

sys/kern/genoffset.sh

sys/kern/imgact_aout.c

sys/kern/imgact_elf.c

sys/kern/imgact_elf32.c

sys/kern/imgact_elf64.c

sys/kern/imgact_shell.c

sys/kern/init_main.c

sys/kern/kern_alq.c

sys/kern/kern_boottrace.c

sys/kern/kern_clocksource.c

sys/kern/kern_condvar.c

sys/kern/kern_conf.c

sys/kern/kern_context.c

sys/kern/kern_cpu.c

sys/kern/kern_cpuset.c

sys/kern/kern_ctf.c

sys/kern/kern_dtrace.c

sys/kern/kern_environment.c

sys/kern/kern_et.c

sys/kern/kern_event.c

sys/kern/kern_exec.c

sys/kern/kern_fail.c

sys/kern/kern_ffclock.c

sys/kern/kern_fork.c

sys/kern/kern_hhook.c

sys/kern/kern_hwt.c

sys/kern/kern_idle.c

sys/kern/kern_intr.c

sys/kern/kern_jail.c

sys/kern/kern_kcov.c

sys/kern/kern_khelp.c

sys/kern/kern_kthread.c

sys/kern/kern_ktr.c

sys/kern/kern_linker.c

sys/kern/kern_lock.c

sys/kern/kern_lockstat.c

sys/kern/kern_loginclass.c

sys/kern/kern_mbuf.c

sys/kern/kern_module.c

sys/kern/kern_mtxpool.c

sys/kern/kern_osd.c

sys/kern/kern_pmc.c

sys/kern/kern_poll.c

sys/kern/kern_priv.c

sys/kern/kern_racct.c

sys/kern/kern_rangelock.c

Hardware Trace (HWT) framework
Needs ReviewPublic
Actions

Revision Contents
Changeset List