VR on IPsec VPN Part 1 - First Peek Into The Rabbit Hole

1. Background

1.1 What is IPsec?

IPsec (Internet Protocol Security) is a suite of protocols used to secure IP communications over untrusted networks such as the Internet. It provides:

• Confidentiality – encryption prevents eavesdropping on traffic.

• Integrity – tamper detection ensures packets have not been modified in transit.

• Authentication – both peers verify each other’s identity before exchanging data.

• Anti-replay protection – sequence numbers prevent an attacker from capturing and replaying legitimate packets.

IPsec is commonly used for site-to-site VPNs (connecting two office networks over the Internet) and remote access VPNs (allowing individual users to connect securely to a corporate network).

2 Motivations

IPsec is widely deployed on edge devices from vendors like Palo Alto, Cisco, and F5. This makes it a highly valuable attack surface:

• It is an entry point directly reachable from the Internet or other untrusted networks.
• A remote code execution (RCE) vulnerability in an IPsec implementation can give an attacker access to internal networks behind the VPN gateway.

CVE-2026-33824 is one such vulnerability: a double-free in the Windows IKEv2 implementation caused by improper ownership handling of a heap-allocated blob pointer during IKEv2 fragment reassembly.

Even FortiGate made a move to only support IPsec VPN from 7.6.3, ridding itself of the ever-vulnerable SSL VPN.

3. Internet Key Exchange (IKE)

IPsec relies on the Internet Key Exchange (IKE) protocol to automate the negotiation and establishment of Security Associations (SAs). An SA is essentially an agreement between two peers on which cryptographic algorithms and keys to use. IKE runs on UDP port 500 (or port 4500 when NAT traversal is in use).

There are two versions of IKE:

The rest of this series focuses on IKEv2.

4. Difficulty for VR

Vulnerability research against IKEv2 is hard – and “hard” is an understatement. The protocol is deeply stateful: a responder only does anything interesting after a valid IKE_SA_INIT exchange and a completed Diffie-Hellman handshake. A coverage-guided fuzzer feeding random bytes at UDP/500 will almost never reach the deeper code paths, because the very first state-machine checks reject anything that does not look like a well-formed initiation.

On top of that statefulness, there are many distinct branches an implementation has to handle:

• IKE_SA_INIT, IKE_AUTH, IKE_INTERMEDIATE, CREATE_CHILD_SA, and INFORMATIONAL exchanges.
• Fragmentation and reassembly, rekeying, EAP, redirect, and NAT-D payloads.

Then there is the combinatorial explosion of configurations: certificate-based versus PSK authentication, a wide range of Diffie-Hellman (DH) groups, and – increasingly – Post-Quantum key exchange negotiated via Additional Key Exchanges (ADDKE) per RFC 9370.

But precisely because the surface is this large and this stateful, something somewhere is bound to mishandle a transition. That is exactly what makes it worth attacking.

5. Approach

Our approach is to combine a controlled lab environment with both static and dynamic analysis of the target implementation:

• Lab environment. A strongSwan server and client on Linux, on an isolated network, give us a known-good reference implementation to drive and observe.

• Verbose debugging. With strongSwan’s ike subsystem log level set to 4 (or the save-keys plugin enabled), the daemon dumps the Sk* session keys into a Wireshark-compatible ike2_decryption_table, so we can decrypt captured IKEv2 traffic and inspect every payload.

• Packet crafting. Scapy’s ikev2 contrib layer lets us build and mutate raw IKEv2 packets – including fragments for replay and targeted fuzzing.

• Static and dynamic analysis. On the server side, we combine static reverse engineering (Ghidra, and CodeQL where source is available) with dynamic analysis of the process that handles incoming client traffic.

6. Conclusion

This first part covered the IPsec and IKEv2 background and laid out why the protocol is a worthwhile target. The rest of the series will walk through the lab setup and packet crafting – and, if we’re lucky, turn up some new bugs in strongSwan along the way!

Before we get ahead of ourselves, we need to take a deeper look at the different messaget types IKEv2 has to offer.

7. References

1. RFC 7296 – Internet Key Exchange Protocol Version 2 (IKEv2)
2. RFC 9370 – Multiple Key Exchanges in IKEv2 (ADDKE)
3. Scapy IKEv2 contrib layer
4. strongSwan Logging documentation
5. strongSwan save-keys plugin