This article is a guide to setting up redundant firewalls using Red Hat Enterprise Linux 7, that services high-availability and load-balancing requirements for the firewalls themselves as well as the services in networks behind it.

The base architecture of the network consists of at least 1 subnet on the outside of the firewalls, facing the Internet, and multiple subnets on the inside. The internal networks are supposed to use private IP space. This article does not concern itself with additional DMZ considerations, nor specific Internet connectivity or customer edge router requirements, nor hypervisor network configuration and provisioning techniques.

Let the outside network be 192.168.1.0/24. Let the gateway to the outside world be 192.168.1.254.

Create two nodes, which for the purposes of this guide we’ll name fw01.example.com and fw02.example.com. Let the system IP addresses for these nodes be 192.168.1.1 and 192.168.1.2 for fw01.example.com and fw02.example.com respectively.

Let the internal networks live in the private IP space 10.10.0.0/16. The Internet gateway will need to be configured with a route to 10.10.0.0/16 via an IP address that fails over between fw01.example.com and fw02.example.com. Let this address be 192.168.1.10.

fw01.example.com and fw02.example.com would normally be configured with the default route on eth0, a state negotiation network on eth1, and a series of 802.1q encapsulated VLANs on eth2.

Now that the basic topology is outlined, we can start configuring the services that allow these firewalls to fail over.

Using Keepalived for the Inbound Routing Address

The Internet gateway is configured with a static route for 10.10.0.0/16 via 192.168.1.10. fw01.example.com and fw02.example.com will now need to be configured such that they hold that address. This is a preliminary configuration, and we’ll be changing it soon after we configure the remainder of the services.

Install the keepalived package:

# yum -y install keepalived

Edit /etc/keepalived/keepalived.conf, and supply it with the following contents:

global_defs {
    router_id FW
}

vrrp_sync_group VG_GW {
    group {
        GWEXT
    }
}

vrrp_instance GWEXT {
    state BACKUP
    interface eth1
    virtual_router_id 1
    priority 100
    nopreempt
    advert_int 5

    authentication {
        auth_type PASS
        auth_pass SOMEPASS
    }

    virtual_ipaddress {
        192.168.1.10/24 dev eth0
    }
}

Start and enable the service:

# systemctl start keepalived
# systemctl enable keepalived

At this point, you should be able to ping 192.168.1.10.

Connection Tracking State Sharing

Install the conntrack-tools package:

# yum -y install conntrack-tools

Supply a node-specific configuration file /etc/conntrackd/conntrackd.conf. The node-specific part is the IPv4_interface in the Sync/Multicast section (in bold). Furthermore, note that the addresses for your local client network (perhaps the one in which your workstation lives) may have been excluded from connection tracking in the General/Filter From Kernelspace/Address Ignore section, per the IPv4_Address settings listed.

Sync {
    Mode ALARM {
    }

    Multicast {
        IPv4_address 225.0.0.50
        Group 3780
        IPv4_interface 10.10.1.1
        Interface eth1.1
        SndSocketBuffer 24985600
        RcvSocketBuffer 24985600
        Checksum on
    }

    Options {
        TCPWindowTracking On
        ExpectationSync On
    }
}

General {
    Nice -20
    HashSize 32768
    HashLimit 131072
    LogFile On
    Syslog On
    LockFile /var/lock/conntrack.lock

    UNIX {
        Path /var/run/conntrackd.ctl
        Backlog 20
    }

    NetlinkBufferSize 2097152
    NetlinkBufferSizeMaxGrowth 8388608

    NetlinkEventsReliable On

    Filter From Kernelspace {
        Protocol Accept {
            TCP
        }

        Address Ignore {
            IPv4_address 127.0.0.1
            IPv4_address 192.168.1.0/24
            IPv4_address 10.10.1.0/24
        }
    }
}

Additional firewall configuration will be needed to allow the traffic, but since we’re in the setup phase, it is easiest to shut off the firewall. Later, we’ll be amending the iptables service configuration, while Red Hat Enterprise Linux 7 ships with firewalld. Omit this step if you want. Please realize that conntrackd listens on two UDP ports, one of which is semi-randomly assigned (the other is 3780).

# systemctl stop firewalld
# systemctl disable firewalld

Start and enable the conntrackd service (on both nodes):

# systemctl start conntrackd
# systemctl enable conntrackd

You may verify whether conntrackd works by issuing the following command, but there’s likely not many traffic events to monitor;

# conntrack -E

The conntrack-tools package also ships with a fail-over script that keepalived can use. Find it with;

# rpm -qld conntrack-tools | grep primary-backup

It is recommended to copy this file from its original location to a known location such as /etc/conntrackd/primary-backup.sh, because updates to the conntrack-tools package could change the location.

The VRRP sync group VG_GW in /etc/keepalived/keepalived.conf can now be modified to include the fail-over script:

vrrp_sync_group VG_GW {
    group {
        GWEXT
    }

    notify_master "/etc/conntrackd/primary-backup.sh primary"
    notify_backup "/etc/conntrackd/primary-backup.sh backup"
    notify_fault "/etc/conntrackd/primary-backup.sh fault"
}

Restart the keepalived service for these changes to take effect:

# systemctl restart keepalived

Configuring IPTables

Making HAProxy a transparent proxy requires mangling traffic, and diverting it through a local routing table. Kolab Systems uses a custom firewall management suite, but it is important to ensure the following occurs when the system starts up.

A routing table needs to be added that is specific to traffic marked. For this, the following snippet could be executed:

# Add route table and route table use for marked firewall traffic.
ip rule list | grep -q fwmark || ip rule add fwmark 1 lookup 100
ip route list table 100 | grep -q ^local || ip route add local 0.0.0.0/0 dev lo table 100

# Note that a proper management suite is smarter about this.
iptables -t mangle -F PREROUTING
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT

# Flush, delete and recreate the DIVERT chain in the mangle table.
iptables -t mangle -F DIVERT 2>/dev/null
iptables -t mangle -X DIVERT 2>/dev/null
iptables -t mangle -N DIVERT 2>/dev/null
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT

For this, we may use a /etc/iptables/iptables script that is executed before the iptables service starts. We’ll replace firewalld with iptables, and in turn amend the start-up process for iptables;

Install the iptables-services package, and supply the following contents to /etc/systemd/system/iptables.service:

# yum -y install iptables-services
# cat > /etc/systemd/system/iptables.service << EOF
.include /usr/lib/systemd/system/iptables.service

[Service]
ExecStartPre=/etc/iptables/iptables --no-service
EOF
# systemctl disable firewalld
# systemctl stop firewalld
# systemctl enable iptables
# systemctl start iptables

Next, allow forwarding and binding to non-local IP addresses. Add the following to /etc/sysctl.conf:

net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1

and apply to the running system:

# sysctl -p

Create a VLAN

For this tutorial, we’ll use a VLAN 2 and an 802.1q encapsulated interface on eth2.2. Per the table above, fw01.example.com will hold system IP address 10.10.2.253, and fw02.example.com will hold system IP address 10.10.2.252. Both will fail-over the gateway address for the network, 10.10.2.254.

Repeat this process for however many VLANs you want. For each VLAN, supplement the keepalived configuration with a snippet;

vrrp_instance GW002 {
    state BACKUP
    interface eth1
    virtual_router_id 2
    priority 100
    nopreempt
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass SOMEPASS
    }

    virtual_ipaddress {
        10.10.2.254/24 dev eth2.2
    }
}

and add the new VRRP instance name to the VRRP group VG_GW. This will ensure that when the external gateway fails over, the internal gateways fail over with it.

vrrp_sync_group VG_GW {
    group {
        GWEXT
        GW002
    }
}

Restart the keepalived service;

# systemctl restart keepalived

Installing and Configuring HAProxy

Install the haproxy package:

# yum -y install haproxy

Supply /etc/haproxy/haproxy.cfg with the following configuration snippets (irrelevant details omitted, repetitive snippets omitted):

global
    (...snip...)

defaults
    (...snip...)

peers failover
    peer fw01.example.com 10.10.1.1:1347
    peer fw02.example.com 10.10.1.2:1347

frontend smtp_ext-25
    mode tcp
    bind 192.168.1.10:25 transparent

    option tcplog

use_backend smtp_ext-mx-in.example.com
    mode tcp
    source 0.0.0.0 usesrc clientip

    balance roundrobin
    stick-table type ip size 1m expire 10m peers failover
    stick on src

    option smtpchk

    server  ext-mx-in001.example.com    10.10.2.1:25 check inter 30s fastinter 10s
    server  ext-mx-in002.example.com    10.10.2.2:25 check inter 30s fastinter 10s

Start and enable the HAProxy service:

# systemctl start haproxy
# systemctl enable haproxy

Provision VLAN 2 Nodes

Provision a node or two in VLAN 2, and make Postfix listen on all IPv4 interfaces. By the time that’s complete, you should be able to telnet to 192.168.1.10 port 25, get a HELO banner, and kick the active firewall while maintaining your existing connection to the chosen backend SMTP server.

Notes & Gotchas

1. The passwords that keepalived supports have a maximum length; everything else is stripped off. Use no more than 8 characters, and keep it alphanumeric.
2. Virtualization platforms allow for a maximum number of network interfaces to be provided to a single virtual guest; be aware of this limitation before you get started. For a complete reference architecture implementation, you are almost guaranteed to require the use of 802.1q encapsulation.