Linux File Monitoring With Fanotify

1. Introduction

Let’s take a break from the GDB exercises and instead look into something that I have yet to talk about in this blog, and that is - Linux file monitoring with the fanotify API.

Fanotify is a Linux kernel API that enables applications to receive notifications about file system events. It is commonly used for:

• Real-time security monitoring (e.g., antivirus scanning)
• Auditing and compliance (e.g., logging file accesses)
• File system activity tracking (e.g., synchronizing backups)

The main reason why so many antivirus software products use fanotify over inotify, is because it provides information on the triggering process PID and can even allow/deny any process trying to read a file via the FAN_ALLOW or FAN_DENY response.

However, we can also use the fanotify API for cyber research purposes! When studying a new target, it is hard to tell exactly what many of the core processes are doing, especially when it comes to file operations. That is why the fanotify API is so powerful as it gives the researcher a broad overview of what the processes are reading from and writing to.

In this post, I will show you how you can write your own fanotify user-space application to monitor for file/directory events, as long as your kernel was built with fanotify capabilities. Ensure that you have sudo or root privileges too before running your fanotify application.

You can also check out filemon, a multithreaded Linux file-monitoring tool which I have created for research purposes. With that out of the way, let’s dive right into it!

2. Checking if Fanotify is Enabled

To check if your kernel was built with fanotify enabled, we have to check the configuration options under /boot/config-<KERNEL_VERSION> or /proc/config.gz.

# Ubuntu / Debian based
$ cat /boot/config-$(uname -r) | grep FANOTIFY
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y

# Fedora / CentOS / RHEL based
$ zcat /proc/config.gz | grep FANOTIFY
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y

If CONFIG_FANOTIFY_ACCESS_PERMISSIONS=n or not defined, that just means that we will not be able to check for events such as FAN_ACCESS_PERM, FAN_OPEN_PERM and FAN_OPEN_EXEC_PERM.

As long as CONFIG_FANOTIFY=y is set, we are good to go!

3. Overall Fanotify API Usage

To begin, we need to create a fanotify file descriptor using fanotify_init(), paying special attention to the flags supplied to it (elaborated later).

We will then mark a file/directory/mount point with fanotify_mark() and proceed to read from the fanotify file descriptors to inspect events.

The snippets below shows an overview of the APIs involved for different use case of file monitoring:

• Start by initializing a fanotify file descriptor

  // Flags will be provided in other examples later
  int fan_fd = fanotify_init(...);
  

• Example to monitor a specific file for FAN_OPEN

  fanotify_mark(fan_fd, FAN_MARK_ADD, FAN_OPEN, AT_FDCWD, "test.txt");
  

• Example to monitor a directory (non-recursive, excluding files) for FAN_OPEN

  fanotify_mark(fan_fd, FAN_MARK_ADD | FAN_MARK_ONLYDIR, FAN_ONDIR | FAN_OPEN, AT_FDCWD, "/tmp/testdir");
  

• Example to monitor a directory (non-recursive, including files) for FAN_OPEN

  fanotify_mark(fan_fd, FAN_MARK_ADD, FAN_EVENT_ON_CHILD | FAN_ONDIR | FAN_OPEN, AT_FDCWD, "/tmp/testdir");
  

• Example to monitor the “/tmp” mount point (recursive, including files and new directories) for FAN_OPEN

  fanotify_mark(fan_fd, FAN_MARK_ADD | FAN_MARK_MOUNT, FAN_EVENT_ON_CHILD | FAN_ONDIR | FAN_OPEN, AT_FDCWD, "/tmp");
  

• Example to monitor the entire filesystem (all mount points)

  // Only on newer kernel versions (since 4.20)
  fanotify_mark(fan_fd, FAN_MARK_ADD | FAN_MARK_FILESYSTEM, FAN_EVENT_ON_CHILD | FAN_ONDIR | FAN_OPEN, AT_FDCWD, "/");
  

I know, I know… You must be thinking: “So what’s the deal between FAN_MARK_MOUNT and FAN_MARK_FILESYSTEM really? Why do we need two different flags?”

As it turns out, these two flags overlap just enough to be confusing, but not enough to be interchangeable. Laying them out next to each other clears up where each one shines, and where it doesn’t.

Now that you understand the APIs, we can get down to the fun of working out which flags actually go with which event masks, since of course, the fanotify man page politely declines to make this obvious.

4. Fanotify Flag-Mask Compatibility Table

I have sunk countless hours through trial and error (so that you don’t have to 😊) in order to find the out the compatibility between different fanotify_init() flags, fanotify_mark() flags and event masks.

5. Inspecting Read, Write, Execute Events

The fanotify man page has provided a template example for inspecting read, write and execute events.

Instead of specifying the FAN_NONBLOCK flag in fanotify_init(), we should omit it in order to capture all events. If FAN_NONBLOCK is specified, there is a chance that certain events may slip under the radar if our application cannot read it in time.

Therefore, with the modification applied, our initialization should look like this:

int fan_fd = fanotify_init(FAN_CLOEXEC | FAN_CLASS_CONTENT, O_RDONLY | O_LARGEFILE);

After getting a valid fanotify file descriptor (not -1), we can mark a file/directory/mount point. In this example, I will just be marking the “/” mount point with FAN_MARK_MOUNT as it is backwards compatible with older kernel versions.

uint64_t masks = FAN_EVENT_ON_CHILD | \
                 FAN_ONDIR | \
                 FAN_ACCESS | \
                 FAN_OPEN | \
                 FAN_MODIFY | \
                 FAN_OPEN_EXEC | \  // Only from Linux 5.0 onwards
                 FAN_CLOSE_WRITE | \
                 FAN_CLOSE_NOWRITE; 

fanotify_mark(fan_fd, FAN_MARK_ADD | FAN_MARK_MOUNT, masks, AT_FDCWD, "/");

We can do a poll() on our fanotify file descriptor to wait for incoming events and parse them accordingly. For the sake of simplicity, The overall code should look like this:

struct pollfd pfd;
ssize_t buflen = 0;
char buf[8192] = {0};
struct fanotify_event_metadata *metadata;

pfd.fd = fan_fd;
pfd.events = POLLIN;

int ret = poll(&fd, 1, -1); // -1 for no TIMEOUT
if (ret <= 0) {
  perror("poll");
  exit(EXIT_FAILURE);
}
for (;;) {
  if (pfd.revents & POLLIN) {
    buflen = read(fan_fd, buf, sizeof(buf));
    if (buflen <= 0) continue;
    metadata = (struct fanotify_event_metadata *)buf;
    while (FAN_EVENT_OK(metadata, buflen)) {
      ...
      if (metadata->mask & FAN_ACCESS)
        printf("FAN_ACCESS");
      ...
      // next event
      close(metadata->fd);
      metadata = FAN_EVENT_NEXT(metadata, buflen);
    }
  }
}

6. Inspecting Create, Delete, Move Events

In the same fanotify(7) man page, they provided an additional example of fanotify being used with a group that identifies objects by file handles.

As usual, we will initialize a fanotify file descriptor but this time, we will use the flags meant for file handles.

int fan_fd = fanotify_init(FAN_CLASS_NOTIF | FAN_REPORT_DFID_NAME, O_RDWR);

After getting a valid fanotify file descriptor (not -1), we can mark a file/directory/mount point. In this example, I will just be marking the “/” mount point with FAN_MARK_FILESYSTEM.

uint64_t masks = FAN_ONDIR | \
                 /* All subsequent masks only from Linux 5.1 onwards */
                 FAN_CREATE | \ 
                 FAN_DELETE | \
                 FAN_MOVED_FROM | \
                 FAN_MOVED_TO;

fanotify_mark(fan_fd, FAN_MARK_ADD | FAN_MARK_FIOLESYSTEM, masks, AT_FDCWD, "/");

Similarly, we will do a poll() on the fanotify file descriptor just like how we did it in the previous section. If you want to get the file name, you will need to make use of struct fanotify_event_info_fid and struct file_handle.

The snippet below shows how you are able to retrieve the file name:

...
struct fanotify_event_metadata *metadata;
struct fanotify_event_info_fid *fid;
unsigned char *file_name = NULL;
...
while (FAN_EVENT_OK(metadata, buflen)) {
  ...
  fid = (struct fanotify_event_info_fid *) (metadata + 1);
  file_handle = (struct file_handle *) fid->handle;

  if (fid->hdr.info_type == FAN_EVENT_INFO_TYPE_FID || fid->hdr.info_type == FAN_EVENT_INFO_TYPE_DFID) {
      file_name = NULL;
  } else if (fid->hdr.info_type == FAN_EVENT_INFO_TYPE_DFID_NAME) {
      file_name = file_handle->f_handle + file_handle->handle_bytes;
  }
  ...
}
...

7. ⭐ Bonus - Allow/Deny File Operations

If you are creating an application that prevents all users (including root) to read or execute certain files, you will need the same flags as the fanotify_init() example shown in Section 4. After getting a valid fanotify file descriptor (not -1), we can mark a file/directory/mount point.

In this example, I will be marking just 1 file - “/home/gerald/example.txt” and deny any who tries to open this file.

struct fanotify_response response;

fanotify_mark(fan_fd, FAN_MARK_ADD, FAN_OPEN_PERM, AT_FDCWD, "/home/gerald/example.txt");
...
while (FAN_EVENT_OK(metadata, buflen)) {
  ...
  if (metadata->mask & FAN_OPEN_PERM) {
    response.fd = metadata->fd;
    response.response = FAN_DENY;
    write(fan_fd, &response, sizeof(response));
  }
  ...
  // next event
  close(metadata->fd);
  metadata = FAN_EVENT_NEXT(metadata, buflen);
}
...

8. Conclusion

If everything made sense to you so far, great! If not, I would highly recommend re-visiting the sections you were unsure on, and trying the fanotify APIs for yourself!

9. Resources

1. filemon - Multithreaded Linux File Monitoring Tool
2. fanotify(7) man page - Official fanotify man page
3. fanotify_init(2) man page - Official fanotify_init man page
4. fanotify_mark(2) man page - Official fanotify_mark man page