1. Introduction

This guide is meant for anyone working with embedded systems, particularly when performing firmware analysis or vulnerability research. Embedded systems are specialized computer systems designed to perform specific tasks within a larger device or machine. Examples include routers, firewalls, or IoT devices.

Such systems often restrict both users and administrators to a limited shell and employ root file system integrity checks. Without understanding and customizing the firmware, debugging or conducting security research is extremely difficult.

This guide walks through practical steps to “jailbreak” a restricted shell and gain full access to the Linux environment, making firmware analysis and kernel debugging more approachable.

2. Mounting / Unmounting Disk Images

  1. Install required utilities.

    $ sudo apt-get install qemu-utils

  2. Allow up to 16 partitions per device.

    $ sudo modprobe nbd max_part=16

  3. By default, the Network Block Device (nbd) kernel module creates 16 devices.

    $ ls -l /dev/nbd*
brw-rw----. 1 root disk 43,   0 Aug 20 10:51 /dev/nbd0
brw-rw----. 1 root disk 43,  32 Aug 20 10:51 /dev/nbd1
brw-rw----. 1 root disk 43, 320 Aug 20 10:51 /dev/nbd10
brw-rw----. 1 root disk 43, 352 Aug 20 10:51 /dev/nbd11
brw-rw----. 1 root disk 43, 384 Aug 20 10:51 /dev/nbd12
brw-rw----. 1 root disk 43, 416 Aug 20 10:51 /dev/nbd13
brw-rw----. 1 root disk 43, 448 Aug 20 10:51 /dev/nbd14
brw-rw----. 1 root disk 43, 480 Aug 20 10:51 /dev/nbd15
brw-rw----. 1 root disk 43,  64 Aug 20 10:51 /dev/nbd2
brw-rw----. 1 root disk 43,  96 Aug 20 10:51 /dev/nbd3
brw-rw----. 1 root disk 43, 128 Aug 20 10:51 /dev/nbd4
brw-rw----. 1 root disk 43, 160 Aug 20 10:51 /dev/nbd5
brw-rw----. 1 root disk 43, 192 Aug 20 10:51 /dev/nbd6
brw-rw----. 1 root disk 43, 224 Aug 20 10:51 /dev/nbd7
brw-rw----. 1 root disk 43, 256 Aug 20 10:51 /dev/nbd8
brw-rw----. 1 root disk 43, 288 Aug 20 10:51 /dev/nbd9

  4. Connect a disk image to a nbd device.

    # Template
$ sudo qemu-nbd -c /dev/nbd1 $DISK_IMAGE_PATH

# Examples:
## Using KVM
$ sudo qemu-nbd -c /dev/nbd1 /var/lib/libvirt/images/mikrotik.qcow2
## Using VMware
$ sudo qemu-nbd -c /dev/nbd1 /home/user/vmware/mikrotik/mikrotik.vmdk
## Using VirtualBox
$ sudo qemu-nbd -c /dev/nbd1 /home/user/VirtualBox\ VMs/mikrotik/mikrotik.vdi

  5. Check the partitions that appear.

    $ ls -l /dev/nbd1p*
brw-rw----. 1 root disk 43, 33 Aug 20 10:56 /dev/nbd1p1
brw-rw----. 1 root disk 43, 34 Aug 20 10:56 /dev/nbd1p2

  6. Mount the partitions (either using a GUI file viewer or manually).

    $ sudo mkdir -p /mnt/example1 /mnt/example2
$ sudo mount /dev/nbd1p1 /mnt/example1
$ sudo mount /dev/nbd1p2 /mnt/example2

  7. Cleanup when finished.

    $ sudo umount /mnt/example1 /mnt/example2
$ sudo qemu-nbd -d /dev/nbd1

3. Identifying the Bootloader

Firmware images typically rely on a bootloader to load the kernel. Two common examples include:

Bootloader Common Files Configuration Files
GRUB2 grub64.efi, BOOTX64.efi grub.cfg
Syslinux ldlinux.sys extlinux.conf, syslinux.cfg

A simple way to identify the bootloader is through trial and error. We can rename or temporarily remove the suspected configuration file and reboot the VM. If the system fails to boot, you’ve found the active bootloader.

You can also experiment by commenting out TTY definitions in the configuration file. This often redirects logs to the current terminal, which can be very helpful for kernel debugging later on.

4. Static Analysis on the Kernel

First, we have to locate the bzImage file. This is often named with the prefix vmlinuz, but if not, you can manually inspect files using the file command.

Once you do identify it, use vmlinux-to-elf to extract and decompress the vmlinux file, populating it with symbols from the compressed kernel symbol table. Then, import it into Ghidra or IDA for analysis.

When firmware integrity checks fail, systems often reboot or hang indefinitely. On VMware, this may show up as “CPU has been stopped” errors. They may verify checksums of critical binaries or libraries and may compare absolute file paths for important system files.

Thus, the key functions and references to look out for are (non-exhaustive):

  • machine_halt()
  • kernel_halt()
  • machine_emergency_restart()
  • kernel_restart()
  • memcmp() near CPU halt functions
  • strcmp() near CPU halt functions

5. Dynamic Analysis on the Kernel

5.1 QEMU GDB Stub

You can enable the GDB stub in QEMU through libvirt. Edit the VM’s configuration:

  1. List VMs

    $ sudo virsh list --all

  2. Edit the VM configuration:

    $ sudo virsh edit <VM Name>

    <domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  ...
  <qemu:commandline>
    <qemu:arg value="-s"/> <!-- Start gdbserver on TCP port 1234 -->
    <qemu:arg value="-S"/> <!-- Wait for gdb to attach -->
  </qemu:commandline>
</domain>

    To specify a custom IP/port:

    <qemu:arg value="-gdb"/>
<qemu:arg value="tcp:<LHOST>:<LPORT>"/>

5.2 VMware GDB Stub

Edit the VM’s .vmx file:

  • For 32-bit VMs:

    debugStub.listen.guest32 = "TRUE"
debugStub.listen.guest32.remote = "TRUE"
debugStub.port.guest32 = "8832"

  • For 64-bit VMs:

    debugStub.listen.guest64 = "TRUE"
debugStub.listen.guest64.remote = "TRUE"
debugStub.port.guest64 = "8864"

  • Optional Tuning:

    debugStub.hideBreakpoints = "FALSE"   # Use software breakpoints instead of hardware
monitor.debugOnStartGuest32 = "TRUE"  # Break on startup (32-bit)
monitor.debugOnStartGuest64 = "TRUE"  # Break on startup (64-bit)

5.3 Attach to Debugee

From GDB:

# Load the kernel if available
(gdb) file vmlinux

# Attach to the remote stub. Replace $LHOST and $LPORT accordingly
(gdb) target remote $LHOST:$LPORT

6. Kernel Patching

Kernel patching without source code is challenging. The main difficulty lies not in locating the compressed kernel but in correctly replacing it with a patched version of the same size.

Thankfully, the latest binwalk (Rust-based) can reliably extract compressed sections and reveal their offsets and sizes. With this, you can replace the kernel after modifying it.

  1. Get latest installation of binwalk. Python version of binwalk has been deprecated and has migrated to Rust.

    # Get dependencies
$ sudo apt install build-essential libfontconfig1-dev liblzma-dev libssl-dev
# Install binwalk via cargo
$ cargo install binwalk

  2. Check the compression type of the vmlinuz, and identify the offset (+21196) and size (17289870 bytes).

    $ binwalk vmlinuz

---------------------------------------------------------------------------------------------------------------------------
DECIMAL                            HEXADECIMAL                        DESCRIPTION
---------------------------------------------------------------------------------------------------------------------------
21196                              0x52CC                             ZSTD compressed data, total size: 17289870 bytes
---------------------------------------------------------------------------------------------------------------------------

  3. Extract the compressed vmlinux from vmlinuz using offset and size from result previously

    $ dd if=vmlinuz of=vmlinux.zst ibs=1 skip=21196 count=17289870

  4. Decompress the extracted compressed form of vmlinux

    $ zstd -k -d vmlinux.zst

  5. Patch vmlinux however you like with ghidra or IDA

  6. Recompress vmlinux

    $ zstd -k -19 vmlinux

  7. Verify the new compressed size. Using the previous example, if the original compressed size was 17289870 bytes, then we MUST NOT exceed this value. If we do exceed it, then we need to patch out some arbitrary stuff in vmlinux again and recompress vmlinux.

    $ wc -c vmlinux.zst

  8. Zero out the original section in vmlinuz

    $ dd if=/dev/zero of=vmlinuz bs=1 seek=21196 count=17289870 conv=notrunc

  9. Overwrite original section

    $ dd if=vmlinux.zst of=vmlinuz bs=1 seek=21196 conv=notrunc

I have created a tool called kforge for this purpose. Do check it out!

7. Customizing the Root File System

To break out of restricted shells, it’s often useful to add custom binaries:

  1. Bring in BusyBox

  2. Place busybox under /bin.

  3. Add a startup script:

    #!/bin/sh
/bin/busybox telnetd -p 23 -l /bin/sh
/bin/busybox sh -i

  4. You can replace startup binaries or hook scripts that get executed automatically. Once you have a shell, bring in additional static tools such as strace, gdb, or gdbserver.

8. Bypassing User-Space Integrity Checks

User-space integrity checks are often performed by /bin/init. Look for string or path references such as

  • “integrity”, “hash”, etc.
  • Calls to reboot()
    • reboot(0x4321fedc) → LINUX_REBOOT_CMD_POWER_OFF
    • reboot(0x1234567) → LINUX_REBOOT_CMD_RESTART

You can patch these routines in Ghidra or IDA to bypass user-space checks and prevent forced reboots.

9. Conclusion

The techniques outlined here are general in nature. Your specific target may employ additional integrity checks or anti-tampering measures. Use these steps as a starting point to gain control of the system and adapt as needed.

Once you’ve broken out of the restricted shell, the real exploration begins. Happy hacking!