Linux: Everything is a file; peekfd: peek write/read of every file

Overview

Introduction

Most of the data flows in Linux, including inter-process communication, sockets, etc., are implemented through file descriptors (fd) reading and writing. When troubleshooting, if you can peek at the flow of fd, many problems can be quickly proved/falsified. This article introduces an old tool peekfd, which can complete this task in a certain environment.

Problems I encountered

I talked about a scenario in 《I Lost to AI - A Brief Note on a Parent-Child Process Interlocking Trap, I Digged It Myself》. The following is a diagram of the parent-child relationship of the process. I want to use kill -QUIT $ (Why not jstack?) to view the jvm stack. Ideally, kill -QUIT $test_case_jvm will make test cases jvm print a stack dump to stdout, and then maven jvm will also copy and print it out. But the reality is that maven jvm eats the stack dump output of test cases jvm and does not spit it out.

[maven jvm] 
     |
     | -- [test cases jvm] (waiting for [curl] exit)
            |
            | -- [curl] (stdout pipe buffer full, waiting for pipe writeable)

1
2
3
4
5
6
7


$ sudo ll /proc/$test_case_jvm/fd
lr-x------ 1 1201 1202 64 7月   4 09:33 0 -> pipe:[1095338147]
l-wx------ 1 1201 1202 64 7月   4 09:33 1 -> pipe:[1095338148]
l-wx------ 1 1201 1202 64 7月   4 09:33 2 -> pipe:[1095338149]
lrwx------ 1 1201 1202 64 7月   4 09:33 3 -> socket:[1095340467]
lrwx------ 1 1201 1202 64 7月   4 09:33 4 -> socket:[1095340468]
lrwx------ 1 1201 1202 64 7月   4 09:33 5 -> socket:[1095340471]

Is there any other solution? Of course there is. test cases jvm shares stdout with maven jvm by writing pipe. Pipe is still a fd (file descriptor) after all. If we have a tool like tcpdump, we can see the traffic of this fd and get the data we want.

Try cat fd

At first, I wanted to use the method of cat test cases jvm’s stdout fd (fd 2):

1

sudo cat /proc/$$test_case_jvm/fd/1

I went to peek. But I soon found that the stack dump result was incomplete, with every other line missing. The reason is simple. Linux pipe supports multiple readers, but each buffer block is consumed by only one reader.

Are there any other solutions? Of course there are.

peekfd

My English vocabulary is actually limited. I learned a lot of it in my current company (thank you!). I didn’t know the meaning of the English word peek before. The first time I noticed this word was peek in Java steam. Then there is the MSG_PEEK flag in libc’s recv(int sockfd, void buf[.len], size_t len, int flags) used in Envoy Proxy’s Listener Filter.

take 1

We use the java application jconsole that comes with jdk to see the effect.

1

jconsole | grep "not match at all"

In another terminal:

1

peekfd $(pgrep jconsole) 1 # 1 is fd of stdout

In another terminal:

1

kill -QUIT `pgrep console`

The result is that peekfd outputs nothing! Disappointed! I love to tinker, of course I won’t give up just like that. I think I know a little bit about kernel and gdb.

take 2

Take a look at the peekfd source code, just a c file:

https://github.com/acg/psmisc/blob/0d03fe8ca1d494d1cc48ca26aa0d780ad0cd4898/src/peekfd.c#L95

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


void attach(pid_t pid) {
	if (num_attached_pids >= MAX_ATTACHED_PIDS)
		return;
	attached_pids[num_attached_pids] = pid;
	if (ptrace(PTRACE_ATTACH, pid, 0, 0) == -1) {
		fprintf(stderr, _("Error attaching to pid %i\n"), pid);
		return;
	}
	num_attached_pids++;
}

Using ptrace, ptrace It specifies the thread, but jvm is multi-threaded, so I have to specify the entire process.

$ peekfd --help
Usage: peekfd [-8] [-n] [-c] [-d] [-V] [-h] <pid> [<fd> ..]
    -8, --eight-bit-clean        output 8 bit clean streams.
    -n, --no-headers             don't display read/write from fd headers.
    -c, --follow                 peek at any new child processes too.
    -t, --tgid                   peek at all threads where tgid equals <pid>.
    -d, --duplicates-removed     remove duplicate read/writes from the output.
    -V, --version                prints version info.
    -h, --help                   prints this help.

--tgid seems to be able to save life, try it:

1

peekfd --follow --tgid `pgrep jconsole` 1 # 1 is fd of stdout

I don’t know why, but my luck seems to be a bit off lately, and I still haven’t peeked!

take 3

I thought I knew a little bit about jvm, and I knew that thead dump is written to stdout by a thread called VM Thread. So:

1
2


export vm_thread_id=$(ps -T -p `pgrep jconsole` | grep 'VM Thread' | cut -d ' ' -f 2)
peekfd $vm_thread_id 1 # 1 is fd of stdout

As the saying goes: Hard work pays off, it’s just that there’s no one to accompany the child. Finally, I can capture stdout this time.

What’s more interesting is that socket is also fd, and peekfd can also be used to view the read and write data of the socket!

Conclusion

peekfd uses ptrace, which may have a significant performance impact, so it is not recommended for use in production environments. peekfd is an old tool hidden in the psmisc toolkit. It is not well-known. This proves one point again: use small tools to solve difficult problems.