Page MenuHomeFreeBSD

truss: Put the output file stream in line-buffered mode
AcceptedPublic

Authored by markj on Aug 22 2024, 8:03 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sat, Oct 26, 10:27 PM
Unknown Object (File)
Thu, Oct 17, 9:03 AM
Unknown Object (File)
Sep 23 2024, 3:14 AM
Unknown Object (File)
Sep 23 2024, 1:36 AM
Unknown Object (File)
Sep 22 2024, 5:43 AM
Unknown Object (File)
Sep 21 2024, 6:58 AM
Unknown Object (File)
Sep 21 2024, 3:14 AM
Unknown Object (File)
Sep 17 2024, 4:11 PM
Subscribers
None

Details

Reviewers
jhb
imp
Summary

Prompted by a debugging session which involved reading truss output via
tail -f. The output suggested that the tracee was stuck in the system
call preceding the one that it was actually stuck in.

While this change would seem to hurt performance, print_syscall_ret()
already unconditionally flushes the stream, so we are not getting much
benefit from block buffering for files.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 59150
Build 56037: arc lint + arc unit

Event Timeline

markj requested review of this revision.Aug 22 2024, 8:03 PM
markj created this revision.

Hmm, this is not as useful as I thought: we don't start printing the syscall line until the system call is returned. So if a thread blocks in, say, select(), truss doesn't print anything until select() returns. In general I guess it has to work this way since some system calls may return data via pointers passed as arguments. But, this way, at least we're consistent when printing to stderr vs. to a file.

  • Use setlinebuf().
  • Remove a now-unneeded fflush() call.

Alternatively, you could structure this so that we split print_syscall() into two parts, one that prints out up to the point of the syscall name and opening ( and then does an explicit fflush() when doing a syscall entry event (but you have to save off the partial len field in your current syscall state to get the padding correct later).

I've also kind of wanted a way to handle IN|OUT parameters where we generate formatted strings that we save during the syscall entry event and later when printing args we do a string compare of old vs new and if it changes do some sort of annotated "old -> new".

You could also imagine perhaps adding a flag for a new mode that always printed the name and args (and for any OUT arguments just printed the raw pointer, etc.) and then did the fflush(), and then during the return did "-> (...)" with a second copy of the system call args, maybe using "..." for IN args and only printing OUT args in the second list.

This revision is now accepted and ready to land.Aug 26 2024, 3:41 PM

BTW, I have long wanted to add a way to export info about threads blocked in select/poll via a new procstat flag, and also a way to dump the list of events for kevents in a process (useful when a thread is sleeping in kevent) so that one could see what a thread was blocked on.

In D46415#1058481, @jhb wrote:

BTW, I have long wanted to add a way to export info about threads blocked in select/poll via a new procstat flag, and also a way to dump the list of events for kevents in a process (useful when a thread is sleeping in kevent) so that one could see what a thread was blocked on.

That would be very cool. A straw proposal: before going to sleep the thread should register a callback to be invoked by the interrogating thread, together with an argument which describes the state on which the thread is going to sleep. And perhaps also a pointer to an interlock, or we use the thread lock of the sleeping thread. For poll/select it would be the seltd (and seltd->st_mtx). For kevent it would be the kqueue.

The callback would walk the structure and dump data to an sbuf which copies out to userspace. I'm not sure whether to make the output format generic or specific to the syscall.

Are there other system calls we might want to instrument this way?

The main ones I have wanted in the past are select/poll and kevent. For kevent I think you could just get by with having a 'kqueue' mode for procstat that dumped the list of notes for each kqueue belonging to the process which would be a bit easier to deal with I think. Some kind of 'struct kinfo_kevent' or something in <sys/user.h> and then a way to query them for a given <pid, kqueue fd> tuple. Perhaps kern.proc.<pid>.kqueue.<fd> sysctl MIB? Then in procstat you would first get the list of open files and iterate over that fetching the events for each queue fd.

For select/poll I do think you want something in the kernel. I'm a bit leery of using sbuf directly as it hardcodes the layout into the kernel, but that would be more extensible. umtx_op might be another case where it might be nice to get more info about what a thread is blocked on (which op and which user address might be the relevant points there).

FWIW, my kgdb scripts have things to dump the seltd already I have used for this as well as a script to walk a kqueue. For me it's really about making that info more accessible to users (and not requiring root).