GDB Guides Series:

  1. GDB Guide Part 1 - Basics
  2. GDB Guide Part 2 - Breakpoints & Linux Calling Conventions
  3. GDB Guide Part 3 - Process Maps
  4. GDB Guide Part 4 - Examining Memory
  5. GDB Guide Part 5 - Stepping
  6. GDB Guide Part 6 - Automation
  7. GDB Guide Part 7 - Custom Commands
  8. GDB Guide Part 8 - Invoking Function Calls

1. Introduction

Now that we have more or less grasp most of the basics commands in GDB, it is time to start automating all these commands. Most of the time in my research, watch dog processes will monitor the performance of our target processes very closely. Any performance degradation will usually lead to the process getting killed or the entire VM getting restarted. This means that if you use interactive GDB and take your sweet time to type out the commands, the process would have held its funeral before you can even type out your 5th command.

At this point you may ask, “If we can debug processes, doesn’t that mean we are already root, and we control the VM?”. Unfortunately, we often share test beds with other researchers, and we have to be careful how we debug processes, as it can be very disruptive for other people.

2. Test Program

To make you appreciate this tutorial better, I generated a test program using ChatGPT to create a watchdog and a watched child process. In the video below, I demonstrated that the moment the watched process’s execution is paused, it will be killed by the watchdog.

Demo - Watchdog Kills Unresponsive Watched Process

It is recommended to get the source file or the static binary to follow along from this section onwards.

3. Assigning Commands to Breakpoints

To assign a set of commands to any breakpoint, use commands. According to the GDB manual:

(gdb) help commands

Set commands to be executed when the given breakpoints are hit.
Give a space-separated breakpoint list as argument after "commands".
A list element can be a breakpoint number (e.g. `5') or a range of numbers
(e.g. `5-7').
With no argument, the targeted breakpoint is the last one set.
The commands themselves follow starting on the next line.
Type a line containing "end" to indicate the end of them.
Give "silent" as the first line to make the breakpoint silent;
then no output is printed when it is hit, except what the commands print.
Commands Usage

Let’s test this out in practice by inspecting 3 arguments supplied to the write function call. To do so, we will write all our commands into a GDB script, called automation-exercise.gdb.

set pag off
set output-radix 16
set confirm off
set disassembly-flavor intel

b write
commands
  info registers rdi rsi rdx
  continue
end

# After setting the breakpoint and  commands, resume the process.
continue
GDB Script - automation-exercise.gdb

In order to attach to the process and run the commands from our file, we need to supply our GDB command with --batch and -x arguments.

Note

--batch

  • Run in batch mode. Exit with status 0 after processing all the command files specified with -x (and .gdbinit, if not inhibited). Exit with nonzero status if an error occurs in executing the GDB commands in the command files.

-x file

  • Execute GDB commands from file.

Thus, our GDB arguments will look like this:

$ gdb --batch -x automation-exercise.gdb -p $PID
Attaching GDB to Process with Script

Demo - Automating the GDB Commands on write() Function Calls

Just for practice, edit the above GDB script to only print out messages sent from the child process to the watchdog. As a start, you need to find out which argument is the one that contains the buffer. You can also filter out irrelevant write calls by the file descriptor. Since we do not want write() calls from printf(), filter out write() calls whose file descriptor is 1 (stdout fd is 1).

If you have done everything right, your output will look something similar to the example shown below.

Demo - Extra Exercise Answer

If you still cannot figure it out, that’s alright. Here’s the answer to the extra exercise.

4. Conclusion

After today’s exercise, I hope you have gained more confidence in debugging targets that cannot have their executions paused indefinitely. In the next post, I will be sharing more about more automation techinques by defining your own custom commands.

5. Resources

  1. automation-exercise.c
  2. automation-exercise
  3. automation-exercise.gdb
  4. automation-exercise-extras.gdb

GDB Guides Series:

  1. GDB Guide Part 1 - Basics
  2. GDB Guide Part 2 - Breakpoints & Linux Calling Conventions
  3. GDB Guide Part 3 - Process Maps
  4. GDB Guide Part 4 - Examining Memory
  5. GDB Guide Part 5 - Stepping
  6. GDB Guide Part 6 - Automation
  7. GDB Guide Part 7 - Custom Commands
  8. GDB Guide Part 8 - Invoking Function Calls