Parker Samp Linux Administration and More

28Mar/103

HOWTO: Using Linux as a simple load balancer / nat router / firewall

Creating a simple network load balancer and router using Linux while maintaining redundancy for a mission critical network.

While on the job at a local cyber cafe, the owner came up to me and asked for a solution that would allow him to stop using expensive (and slow) T1 lines while giving him the option to balance between up to five cable (coax) links.

My research lead me to a low cost solution: utilize old hardware with Linux in a redundant setup. I created multiple scripts and configuration files which enabled Linux to not only load balance with a round robin configuration, but provide the proper connection and firewall support that was needed.

The following guide will enable you to setup Gentoo Linux to perform balancing; but the instructions are compatible on most distributions with little or no tweaking.

Solution

Utilize the Linux operating systems existing kernel modules and settings in conjunction with iproute2, iptables, and VRRP to allow routing and round-robin load balancing.

First we need to define multihoming. Multihoming is a method used to provide both load balancing and redundancy from a Wide Area Network, such as the internet, to a Local Area Network.

In this project you can think of it as multiple default routes to the internet. Multihoming can be accomplished in a variety of different ways, but for right now this guide will only cover round-robin load balancing with router fail-over.

Be aware that this guide assumes the following:

  • eth0 is the LAN interface with network 192.168.0.0/24
  • eth1, eth2, and eth3 are WAN interfaces each with random network and corresponding nexthop router addresses
  • You are using Gentoo Linux 2008.0 or greater, if not: you are capable of modifying the scripts to suit your particular distribution

Network Requirements

This project gives any network administrator very good flexibility on how it can be setup. Linux accepts a variety of network interface devices and as long as they can be used as an interface in ifconfig you can use them with this project.

This documentation will only cover wired ethernet, but if you modify some of the configuration you may use anything. The base requirements for this project to operate are:

  • The Gentoo router
  • A bare minimum of three network interfaces if you do not wish to vlan
  • Any regular switch*

*Note: Linux accepts vlan trunking (802.1q). You will need to purchase a switch that is IEEE 802.1q capable if you wish to utilize vlans.

Kernel Requirements

When configuring the kernel for networking modules it is fine to keep all of the defaults enabled. Use discretion when modifying these options and enable the options that aren’t listed as optional below.

If a router with the minimum amount of kernel modules is required, it is suggested that you review each modules function and decided if it is needed. This is outside the scope of this guide.

With the configuration screen open, enable [*] the following options:

  • Networking support —> Networking options —>
  • 802.1Q VLAN Support (optional)
  • Security Marking
  • Network packet filtering framework (Netfilter)
  • Core Netfilter Configuration —> Enable [*] all options here unless you know what you are doing
  • IP: Netfilter Configuration —> Enable [*] all options here unless you know what you are doing

Exit, recompile the kernel, and reboot.

Project Files

This router project consists of many configuration and program files. It is made using bash scripts that run a series of commands in order to get the router working. I have also included a script that attempts to detect dead routes and delete them from the pool if the ISP router or internal gateway cannot be pinged.

To get started, login as root and type mkdir /etc/linuxwarz . This folder will be used in all examples. If you wish to put the files elsewhere, take note of the location of the folder since some directions you are given may need to be modified. Using your favorite editor, create the following files and their contents:

balance

This file is where the magic happens. Kernel settings are tweaked, routes are setup, and calls to other scripts happen.

With some quick editing, this script can be set to handle more or less connections. You will have to add or delete lines from the “Initialization” and “Setup Balancing” sections of the script in order to support this. The editing should be pretty straightforward.

The common settings are at the top of the script:

VRRP { yes | no } – Enable or disable VRRP usage. Set this to “no” if you aren’t sure or will not utilize redundancy.

VRRPNET { string } – Set the IP address for the VRRP interface.

Syntax= A.A.A.A netmask S.S.S.S where A=IP Octet of network address, S=Subnet octet.

This MUST be different on each router!

DHCPD { yes | no } – Enable or disable DHCPd usage. Set this to “no” if you aren’t sure or will not utilize DHCPd services on this router.

DEADROUTECHK { yes | no } – Enable or disable the ping testing script for detection of unresponsive paths.

Set this to “no” if you aren’t sure or wish to keep unreliable paths in the routing pool.

LOCALNET { string } – Set this to your internal LAN network address.

Syntax= A.A.A.A/SS where A=IP Octet of network address, SS=Slash notation of the subnet.

The detection of the nexthop router for WAN links is not detected in this version. You must know the address of the nexthop router for each link and modify the script to reflect that. Be very careful and be sure to give the right interface the correct nexthop address.

NOTE: By default, this script assumes your internal interface (LAN) is eth0 while your WAN interfaces consist of eth1, eth2, and eth3.

#!/bin/bash
# Load Balancer – Version 2.2
# Script By: Parker Samp

VRRP=”yes”
VRRPNET=”192.168.0.254 netmask 255.255.255.0″
DHCPD=”yes”
DEADROUTECHK=”yes”

LOCALNET=”192.168.0.0/24″

###Begin Initialization

#Setup forwarding and settings
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 10 > /proc/sys/net/ipv4/route/gc_timeout

#Start Services
if [ $VRRP == "yes" ]
then
ip address flush dev eth0
ip address flush dev eth1
ip address flush dev eth2
ip address flush dev eth3
ifconfig eth0:1 $VRRPNET
/etc/init.d/keepalived start
sleep 3
fi
if [ $DHCPD == "yes" ]
then
/etc/init.d/dhcpd start
fi

###End Initialization

###Setup Balancing
ip route add default equalize nexthop via 192.168.10.1 dev eth1 
nexthop via 192.168.9.1 dev eth2 
nexthop via 192.168.8.1 dev eth3

ip route add $LOCALNET dev eth0 table 101
ip route add default via 192.168.10.1 dev eth1 table 101

ip route add $LOCALNET dev eth0 table 102
ip route add default via 192.168.9.1 dev eth2 table 102

ip route add $LOCALNET dev eth0 table 103
ip route add default via 192.168.8.1 dev eth3 table 103

ip rule add fwmark 1 table 101
ip rule add fwmark 2 table 102
ip rule add fwmark 3 table 103
iptables -t mangle -A POSTROUTING -o eth1 -j MARK –set-mark 1
iptables -t mangle -A POSTROUTING -o eth2 -j MARK –set-mark 2
iptables -t mangle -A POSTROUTING -o eth3 -j MARK –set-mark 3
iptables -t mangle -A POSTROUTING -j CONNMARK –save-mark
iptables -t mangle -A PREROUTING -j CONNMARK –restore-mark

###Load Firewall
/etc/linuxwarz/firewall

###Load Route Checker
if [ $DEADROUTECHK == "yes" ]
then
/etc/linuxwarz/droute &
fi
firewall

This file contains all of the firewall rules required to run a stateful firewall with NAT.

Note: These rules are from a cookie cutter firewall template that include my balancer modifications and may not fit your needs. At a minimum, you will need to copy the following lines and specifics into your custom firewall rules:

  • If a DROP policy: allow lo complete access in/out
  • If a DROP policy: allow LAN (eth0) complete access in/out
  • If a DROP policy: allow WAN (eth1,2,3) access out
  • If a DROP policy: allow forwarding between LAN/WAN networks
  • Copy every rule from the “NAT Routing” section
#Begin Firewall
/sbin/iptables -P INPUT DROP
/sbin/iptables -P FORWARD DROP
/sbin/iptables -P OUTPUT DROP
/sbin/iptables -F
/sbin/iptables -F INPUT
/sbin/iptables -F OUTPUT
/sbin/iptables -F FORWARD
/sbin/iptables -F -t mangle
/sbin/iptables -F -t nat
/sbin/iptables -X
/sbin/iptables -Z
/sbin/iptables -N DUMP
/sbin/iptables -F DUMP
/sbin/iptables -A DUMP -p icmp -m limit –limit 1/m –limit-burst 5 -j LOG –log-prefix “IPT ICMPDUMP: ”
/sbin/iptables -A DUMP -p tcp -m limit –limit 1/m –limit-burst 5 -j LOG –log-prefix “IPT TCPDUMP: ”
/sbin/iptables -A DUMP -p udp -m limit –limit 1/m –limit-burst 5 -j LOG –log-prefix “IPT UDPDUMP: ”
/sbin/iptables -A DUMP -p tcp -j REJECT –reject-with tcp-reset
/sbin/iptables -A DUMP -p udp -j REJECT –reject-with icmp-port-unreachable
/sbin/iptables -A DUMP -j DROP
/sbin/iptables -N STATEFUL
/sbin/iptables -F STATEFUL
/sbin/iptables -I STATEFUL -m state –state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -I STATEFUL -m state –state NEW -i eth0 -j ACCEPT
/sbin/iptables -A STATEFUL -j DUMP
/sbin/iptables -N SYN-FLOOD
/sbin/iptables -F SYN-FLOOD
/sbin/iptables -A SYN-FLOOD -m limit –limit 1/s –limit-burst 4 -j RETURN
/sbin/iptables -A SYN-FLOOD -j DROP
/sbin/iptables -A INPUT -i eth1 -p tcp –syn -j SYN-FLOOD
/sbin/iptables -A INPUT -i eth2 -p tcp –syn -j SYN-FLOOD
/sbin/iptables -A INPUT -i eth3 -p tcp –syn -j SYN-FLOOD
/sbin/iptables -A INPUT -p tcp -i eth1 ! –syn -m state –state NEW -j DROP
/sbin/iptables -A INPUT -p tcp -i eth2 ! –syn -m state –state NEW -j DROP
/sbin/iptables -A INPUT -p tcp -i eth3 ! –syn -m state –state NEW -j DROP
/sbin/iptables -A INPUT -i eth1 -f -j DROP
/sbin/iptables -A INPUT -i eth2 -f -j DROP
/sbin/iptables -A INPUT -i eth3 -f -j DROP

###Local Network Rules
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A OUTPUT -o lo -j ACCEPT
/sbin/iptables -A OUTPUT -o eth0 -j ACCEPT

###Enable router to WAN communication – This is a hack – In my testing, the router itself could not talk to the net while hosts could
/sbin/iptables -A OUTPUT -o eth1 -j ACCEPT
/sbin/iptables -A OUTPUT -o eth2 -j ACCEPT
/sbin/iptables -A OUTPUT -o eth3 -j ACCEPT

###NAT Routing
/sbin/iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o eth1 -j MASQUERADE
/sbin/iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o eth2 -j MASQUERADE
/sbin/iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o eth3 -j MASQUERADE

###Stateful Connection Rules
/sbin/iptables -A INPUT -j STATEFUL
/sbin/iptables -A FORWARD -j STATEFUL
/sbin/iptables -A OUTPUT -j STATEFUL

###Port fowarding (redirection)
##Enable connections to the server:
##/sbin/iptables -I INPUT -p tcp -i ${IFext} –dport 80 -j ACCEPT
##Enable connections to the LAN:
##/sbin/iptables -t nat -I PREROUTING -i ${IFext} -p tcp –dport 33335:33336  -j DNAT –to-destination 192.168.2.2
##/sbin/iptables -I FORWARD -i ${IFext} -p tcp –dport 33335:33336 -j ACCEPT
droute

This script checks for dead routes in the pool every 10 seconds by pinging the nexthop router at the ISP. It is a very crude script and requires all links to be working the second it comes up due to it needing WAN information.

If you know your WAN information does not change then enter the address of the nexthop router next to the appropriate variables.

#!/bin/bash
# Dead Route Eliminator – Version 2.1
# Script By: Parker Samp

###Get WAN Information
gweth1=`ip route | grep “nexthop via” | grep eth1 | awk {‘print $3′}`
gweth2=`ip route | grep “nexthop via” | grep eth2 | awk {‘print $3′}`
gweth3=`ip route | grep “nexthop via” | grep eth3 | awk {‘print $3′}`

###Set Variables
deadeth1=”0″
deadeth2=”0″
deadeth3=”0″
deadcount=”0″

loop=”1″
while [ $loop -le 2 ]
do
sleep 10
reroute=”0″
routes=”ip route add default equalize”

for i in eth1 eth2 eth3
do
eval GWT=$gw${i}
ping -I $i -c 2 -W 2 $GWT > /dev/null
if test $? -eq 0
then
routes=$routes” nexthop via $GWT dev $i”
eval deadcheck=$dead${i}
if test $deadcheck -eq 1
then
eval dead$i=”0″
deadcount=`expr $deadcount – 1`
reroute=”1″
fi
else
eval deadcheck=$dead${i}
if test $deadcheck -eq 1
then
eval dead$i=1
else
eval dead$i=1
deadcount=`expr $deadcount + 1`
reroute=”1″
fi
fi
done

if test $reroute -eq 1
then
ip route del default
$routes
ip route flush cache
fi
done
keepalived.conf

VRRP (keepalived) reads this configuration file in order to find out the state, keepalive interface, and virtual IP addresses of the router.

VRRP will send out keepalives through the keepalive interface at regular intervals. If the slave does not hear keepalives from the master, it will then assign all of the virtual IPs to the respective interfaces. This process usually takes about five seconds and all active connections from the systems using the internet will need to be refreshed.

I do not have a solution to keep connections active with only increased latency during the switchover at this time.

Note: This configuration assumes you wish to use eth0 as your keepalive interface and not a private interface from server to server connected with a crossover cable. If you wish to change this, modify “interface eth0:1″.

Virtual IP Addresses

keepalived uses a concept known as virtual IP addresses. These are the IP addresses, netmasks, and broadcast addresses for each interface on the router (LAN/WAN).

If you enabled VRRP in the balancer script, it will flush any existing IP addresses on the interface (required since keepalived sometimes refuses to rewrite IP configuration) and start keepalived. The script will wait 3 seconds so that keepalived can start, figure out its role, and assign the IP addresses to each interface before adding the route and firewall information.

Notification Scripts

keepalived supports running custom scripts on up/down events. I do not use these but I left them commented in the configuration because they can be useful in the future.

vrrp_instance VI_1 {
state MASTER
interface eth0:1
virtual_router_id 1
priority 100
authentication {
auth_type PASS
auth_pass Ch4Ng3
}
virtual_ipaddress {
192.168.0.1/24 brd 192.168.0.255 dev eth0
192.168.10.222/24 brd 192.168.10.255 dev eth1
192.168.9.223/24 brd 192.168.9.255 dev eth2
192.168.8.224/24 brd 192.168.8.255 dev eth3
}
#  notify_backup “/root/keepalive.sh execute”
#  notify_fault “/root/keepalive.sh fault”
}
vrrp_instance VI_1 {
state BACKUP
interface eth0:1
virtual_router_id 1
priority 50
authentication {
auth_type PASS
auth_pass Ch4Ng3
}
virtual_ipaddress {
192.168.0.1/24 brd 192.168.0.255 dev eth0
192.168.10.222/24 brd 192.168.10.255 dev eth1
192.168.9.223/24 brd 192.168.9.255 dev eth2
192.168.8.224/24 brd 192.168.8.255 dev eth3
}
# notify_backup “/root/keepalive.sh execute”
# notify_fault “/root/keepalive.sh fault”
}

Conclusions

This balancer is perfect for those on a tight budget or those looking to expand their knowledge about Linux networking.

I hope that all or parts of this guide will help you solve many problems related to balancing, firewalls, and routing issues within Linux.

Disclaimers

I provide no support or warranties for this product/guide. You are using this at your own risk and should fully test the configurations in a lab environment before deploying into any mission critical network. Parts of this guide may be used in any projects but I do not allow usage in any end products meant to be used for profit.

Comments (3) Trackbacks (0)
  1. Great tutorial. I am planning to do your setup using Arch linux for my small home office network.
    It is very helpful and an expansion for my knowledge in linux networking.
    Thanks,

  2. Keep in mind that linux caches routes. Therefore, if you were for example hoping to use a multi-threaded FTP client over 2 cable modems to double your download speed, this would not work.

    The reason is that since linux caches routes, all FTP threads would end up going over the same cable modem, thus, you would gain nothing.

    However, if all destinations are different, then this is for you. So, if you were downloading 2 files from 2 different places then this would definitely work fine.

  3. big effort…. thanks indeed for sharing …..


Leave a comment

No trackbacks yet.