Hacking and random thoughts

Setup VPN IPSec in Debian

It could be appear the typical post for filling content of a blog... But currently I am working in a project with VPNs and I would like to have a diary about how a VPN IPSec works and how to configure it in Linux (at least one configuration). It is not the first time I am working with a VPN IPSec, and now that I am re-reading all of the RFCs I am getting a better vision about that.


A VPN IPSec is a VPN technology that allows to create tunnels between two networks. It maybe is hard to explain in a little summary because it has a lot of different concepts.

First, to configure the server it is necessary to explain IKE (Internet Key Exchange).

IKE is a protocol which has two versions IKEv1 and IKEv2. This protocol is used to establish a Security Association (SA) for the protocol IPSec. The key exchange used by IKE is Diffie-Hellman (DH). Exists two different modes: transport and tunnel mode, the difference between both is transport uses the original IP header and tunnel mode uses a new IP header. Thus, tunnel mode will be use for security gateway to security gateway or endpoint to security gateway; and tranport mode will be use for endpoint to endpoint.

From RFC 7296 if IKEv2:

Tunnel mode:

Transport mode:

IKE consists in two phases:
Phase 1 is where two ISAKMP peers establish a secure channel to communicate, there is the autentication of the two peers (VPN GWs or VPN clients). There is negotiated the ISAKMP SAs (also called IKE SAs and Phase 1 SA). It has two modes: Main mode and aggressive mode. Here is negotiated the authentication method, encryption algorithm, hash algorithm, diffie-hellman group to use and the lifetime. These information about the SA are called the IKE SA policy. This SA is used to protect IKE traffic.
To mention that both modes are out of scope of this post. In the configuration It only be used the Main Mode since aggressive mode has not enough security because the hash of the pre-shared key (PSK) is sent in clear text.

Phase 2, also called Quick Mode (since this mode only must be used in this phase and it is the only one), is where two Security Associations negotiate the services like IPSec. This Security Associations are called IPSec SAs (or Phase 2 SA) to difference with the IKE SAs of the previous phase. In this phase are negotiated the parameters of the IPSec SA: IPSec protocol, encapsulation mode, encryption algorithm, authentication algorithm, additional DH exchange for the Perfect Forward Secrecy (PFS), and SA lifetime.
As in previous phase, explain how it works internally has enough information to only one post and it is not the purpose of this post, so the working is out of scope of this post.

As both phases can set a lifetime, the lifetime of SA ISAKMP should be greater than lifetime of IPSEC SA

Once IKE completes, an IPSec tunnel secured with Encapsulating Security Payload (ESP) is established.

Following picture shows in high level how both phases works, in Main mode and Quick Mode. For Aggressive mode the first two messages negotiate the policy, DH exchange and identities, second message authenticate the responder and third authenticates the initiator and provides a proof of participation in the exchange.

Some differences between IKEv1 and IKEv2:

IKEv2 supports nat traversal and EAP authentication method and asymmetric authentication.
IKEv2 protect the IKE messages based on ESP, rathern than a method unique to IKE.
Regarding the lifetimes, a difference between IKEv1 and IKv2 is that IKV1 SA lifetimes are negotiated and in IKEv2, each end of the SA is resposible of it own lifetimes policy. If the lifetimes of each one are different, the rekey will be done when reach the shortest lifetime.


In this example, It will be configured a VPN tunnel with L2PT/IPSec with pre-shared key.
For this, it will be use strongswan (exist others like libreswan and openswan) and xl2tpd.

  • Strongswan is the IPSec solution and is the server;
  • xl2tpd is a open source implementation of Layer 2 Tunneling Protocol (L2TP). It is the L2TP server where user will be authenticated. (in this example)

Download and install:

sudo apt install strongswan xl2tpd

Modify the following files:

For xl2pd server open and modify /etc/xl2tpd/xl2tpd.conf:

    ;port = 1701
    ;access control = no

[lns default]
        ip range =
        local ip =
        refuse pap = yes
        refuse chap = no
        require authentication = yes
        name = LinuxVPNserver
        ppp debug = yes
        pppoptfile = /etc/ppp/options.xl2tpd
        length bit = yes


# Secrets for authentication using CHAP
# client        server  secret                  IP addresses
foobar * foobar@123 *

For IPSec:


# This file holds shared secrets or RSA private keys for authentication.

# RSA private key for this host, authenticating it to any other host
# which knows the public part.
%any %any : PSK "password@123"


#config setup
        #charondebug="ike 4, knl 4, cfg 2"    #useful debugs
conn vpntest
        right=        # IP of your VPN Server

Also, add the logging in order to have all the information in the same log file and not multiple files as default (/var/log/daemon, /var/log/syslog and /var/log/messages) in strongswan configuration: /etc/strongswan.d/charon-logging.conf:

    filelog {
        charon {
            append = no
            default = 1
            flush_line = yes
            ike_name = yes
            log_level = yes
            path = /var/log/charon.log
            time_format = %b %e %T
        stderr {
            ike =2 
            knl = 3


If the log is not created; maybe it could because apparmor is blocking the newfile creation in /var/log. Thus, run the following command to allow charon:

sudo apt install apparmor-utils
sudo aa-complain /usr/lib/ipsec/charon

Start both services:

sudo systemctl start xl2tpd                         
sudo systemctl start ipsec  

Connecting from Windows

Connect there from Windows in this case is easy, just go to "VPN Settings", click on "Add a VPN conneciton" fill the fields with connection name, IP server, VPN type, pre-shared key and user/password:

There can be observer how the protocol run and traffic is encrypted.

As you can see above, the phase 1 is tagged in Wireshark as Identity Protection (Main Mode) and Phase 2 as Quick Mode.

And in both side, it is possible to observe the SA; Linux side:

sudo ipsec statusall
Status of IKE charon daemon (strongSwan 5.9.1, Linux 5.10.0-kali7-amd64, x86_64):
  uptime: 116 seconds, since Mar 13 16:14:37 2022
  malloc: sbrk 2420736, mmap 0, used 421648, free 1999088
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 3
  loaded plugins: charon aes rc2 sha2 sha1 md5 mgf1 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem fips-prf gmp xcbc hmac drbg attr kernel-netlink resolve socket-default stroke updown counters
Listening IP addresses:
     vpntest:  IKEv1
     vpntest:   local:  [] uses pre-shared key authentication
     vpntest:   remote: uses pre-shared key authentication
     vpntest:   child:  dynamic[udp] === dynamic[udp/l2f] TRANSPORT
Security Associations (1 up, 0 connecting):
     vpntest[1]: ESTABLISHED 4 seconds ago,[]...[]
     vpntest[1]: IKEv1 SPIs: 44c128d98409229b_i 1dacd3a51019f134_r*, pre-shared key reauthentication in 8 minutes
     vpntest[1]: IKE proposal: AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
     vpntest{1}:  INSTALLED, TRANSPORT, reqid 1, ESP SPIs: c7a62d02_i 33b119ce_o
     vpntest{1}:  AES_CBC_256/HMAC_SHA1_96, 3646 bytes_i (48 pkts, 0s ago), 885 bytes_o (23 pkts, 1s ago), rekeying in 44 minutes
     vpntest{1}:[udp/l2f] ===[udp/l2f]

Windows side:

PS C:\Windows\system32> Get-NetIPsecMainModeSA -All

Name                                : 1
LocalEndpoint                       :
RemoteEndpoint                      :
LocalFirstId.Identity               :
LocalFirstId.ImpersonationType      : None
LocalFirstId.AuthenticationMethod   : PresharedKey
LocalFirstId.Flags                  : None
RemoteFirstId.Identity              :
RemoteFirstId.ImpersonationType     : None
RemoteFirstId.AuthenticationMethod  : PresharedKey
RemoteFirstId.Flags                 : None
LocalSecondId.Identity              :
LocalSecondId.ImpersonationType     :
LocalSecondId.AuthenticationMethod  :
LocalSecondId.Flags                 :
RemoteSecondId.Identity             :
RemoteSecondId.ImpersonationType    :
RemoteSecondId.AuthenticationMethod :
RemoteSecondId.Flags                :
CipherAlgorithm                     : AES256
HashAlgorithm                       : SHA1
GroupId                             : DH14
KeyModule                           : IkeV1
MaxQMSAs                            : 0
LifetimeSeconds                     : 28800
LocalUdpEncapsulationPort           :
RemoteUdpEncapsulationPort          :
ExtendedFilterId                    : 0


2407 (ISAKMP)
2408 (ISAKMP)
2409 (IKE)
4109 (IKEv1)
7296 (IKEv2)
6071 (Roadmap)

Share this:

© 2022 Manu