Dynamic iptables firewalls

1.  Introduction

Flexible (and fun) network security

The best way to see the benefits of dynamic firewall scripts is to see them in action. To do this, let's imagine that I'm a sysadmin at an ISP, and I've recently set up a Linux-based firewall to protect my customers and internal systems from malicious users on the Internet. To do this, my firewall uses the new Linux 2.4 iptables stateful functionality to allow new outgoing connections to be established by my customers and servers, and of course to allow new incoming connections, but only to "public" services, such as web, ftp, ssh, and SMTP. Since I used a deny-by-default design, any from-Internet connections to non-public services, such as the squid proxy cache or Samba server, are automatically rejected. So far, I have a pretty decent firewall that offers a good level of protection for everyone at my ISP.

For the first week or so, the firewall works great, but then something ugly happens: Bob, my arch-nemesis (who works at a competing ISP) decides that he wants to flood my network with packets in an attempt to deny service to my customers. Unfortunately, Bob has carefully studied my firewall and knows that while I'm protecting many internal services, port 25 and 80 must be publicly accessible so that I can receive mail and serve HTTP requests. Bob decides to take advantage of this fact by launching a bandwidth-sucking attack against my web and mail server.

About a minute or so after Bob begins his attack, I notice that my uplinks start becoming saturated with packets. After taking a look at the situation with tcpdump I determine that this is yet another Bob attack, and I figure out what IP addresses he's using to launch it. Now that I have this information, all that I need to do is block these IP addresses, and that should solve the problem -- a simple solution, or so I think.

Responding to an attack

I quickly load my firewall setup script into vi and begin hacking away at my iptables rules, modifying my firewall so that it'll block those evil incoming Bob packets. After a minute or so, I find the exact place to make the appropriate DROP rule additions, and I add them. Then, I start and stop the firewall...ooops, made a bit of a mistake when I added the rules. I load up the firewall scripts again, fix the problem, and thirty seconds later the firewall has been tweaked to block Bob's attack of the month. At first, it seems like I successfully thwarted the attack...until the helpdesk phones begin ringing. Apparently, Bob was able to disrupt my network for about 10 minutes, and now my customers are calling to find out what's going on. Even worse, after a few minutes pass, I notice that our uplinks again start to become saturated. This time, Bob appears to be using a brand-new set of IP addresses for his attacks. In response, I begin feverishly hacking away at our firewall scripts, except this time, I'm a bit panicky -- maybe my solution isn't so good after all.

Here's what went wrong in the above scenario. Although I had a decent firewall in place and also quickly identified the cause of the network problem, I was unable to modify the behavior of my firewall to respond to the threat in time. Of course, when your network is under attack, you want to be able to respond immediately, and being forced to hack away at your master firewall setup script in a panicked state is not only stressful, but also very inefficient.

2.  Scripts

ipdrop

It would be far better if I had a special ipdrop script that's specifically designed to insert just the rules you need to block the IP address that I specify. With such a script, blocking a firewall is no longer a two-minute ordeal; instead, it takes five seconds. And since the script shields me from the task of editing firewall rules by hand, it eliminates a major source of errors. All that's left for me to do is to determine the IP address that I'd like to block, and then type:

# ipdrop 129.24.8.1 on
IP 129.24.8.1 drop on.

Immediately, the ipdrop script would block 129.24.8.1, Bob's current evil IP address of the week. This script dramatically improves your defenses, because now an IP block is a no-brainer. Now, let's take a look at my implementation of the ipdrop script:

#!/bin/bash

source /usr/local/share/.sh

args 2 $# "${0} IPADDR {on/off}" 

# Drops packets to/from IPADDR. Good for obnoxious
networks/hosts/DoS"

if [ "$2" == "on" ] 
then
# Rules will be appended or inserted as normal
 APPEND="-A"
  INSERT="-I"
  rec_check ipdrop $1 "$1 already blocked" on
  record ipdrop $1
elif [ "$2" == "off" ]
then
# Rules will be deleted instead
 APPEND="-D"
  INSERT="-D"
  rec_check ipdrop $1 "$1 not currently blocked" off 
  unrecord ipdrop $1
else
  echo "Error: \"off\" or \"on\" expected as second argument"
  exit 1
fi  

# Block outside IP address that's causing problems
# Attacker's incoming TCP connections will take a minute or so to time
out, reducing DoS effectiveness

iptables $INSERT INPUT   -s $1 -j DROP
iptables $INSERT OUTPUT  -d $1 -j DROP
iptables $INSERT FORWARD -d $1 -j DROP
iptables $INSERT FORWARD -s $1 -j DROP

echo "IP ${1} drop ${2}."

ipdrop: the explanation

If you take a look at the last four highlighted lines, you'll see the actual commands that insert the appropriate rules into the firewall tables. As you can see, the definition of the $INSERT environment variable varies, depending on whether we're running in "on" or "off" mode. When the iptables lines execute, the particular rules will be inserted or deleted appropriately.

Now, let's look at the function of the rules themselves, which should work perfectly with any type of existing firewall, or even on a system with no firewall; all you need is iptables support built-in to your 2.4 kernel. We block incoming packets arriving from the evil IP (first iptables line), block outgoing packets headed for the evil IP (next iptables line), and then turn off forwarding in either direction for this particular IP (last two iptables lines.) Once these rules are in place, your system will simply discard any packets that fall into one of these categories.

Another quick note: you'll also notice calls to "rec_check", "unrecord", "record", and "args". These are special helper bash functions defined in dynfw.sh. The "record" function records the blocked ip in the /root/.dynfw-ipdrop file, while the "unrecord" removes the entry from /root/.dynfw-ipdrop. The "rec_check" function is used to abort the script with an error message if you attempt to re-block an already-blocked IP, or unblock an IP that isn't currently being blocked. The "args" function takes care of making sure that we receive the correct number of command-line arguments, and also handles printing helpful usage information. I've created a dynfw-1.0.tar.gz that contains all these tools; see the Resources section at the end of this article for more information.

tcplimit

This next dynamic firewall script is useful if you need to limit the usage of a particular TCP-based network service, possibly something that generates a heavy CPU load on your end. Called "tcplimit", this script takes a TCP port, a rate, a scale, and "on" or "off" as an argument:

# tcplimit 873 5 minute on
Port 873 new connection limit (5/minute, burst=5) on.

tcplimit uses the new iptables "state" module (make sure you've enabled this in your kernel or loaded the module) to allow only a certain number of new, incoming connections in a specific period of time. In this example, the firewall will allow only five new connections to my rsync server (port 873) per minute -- and it's possible to specify the desired number of connections you'd like per second/minute/hour or day, as needed.

tcplimit offers a good way of limiting non-essential services -- so that a flood of traffic to a non-essential service doesn't disrupt your network or server. In my case, I use tcplimit to set a maximum upper bound for rsync usage to prevent my DSL line from becoming saturated by too many rsync connections. Connection-limited services are recorded in /root/.dynfw-tcplimit, and if I ever want to turn the new connection limiting off, I can simply type:

# tcplimit 873 5 minute off
Port 873 new connection limit off.

tcplimit works by creating a completely new chain in the "filter" table. This new chain will reject all packets that exceed our specified limit. Then, a single rule is inserted into the INPUT chain that redirects all incoming NEW connection packets headed to the target port (873 in this case) to this special chain, effectively placing a limit on new, incoming connections while not affecting packets that are part of an established connection.

When tcplimit is turned off, the INPUT rule and special chain are deleted. This is the kind of fancy stuff that really highlights the importance of having a well-tested, reliable script manage the firewall rules for you. As with ipblock, the tcplimit script should be compatible with any type of firewall, or even no firewall, as long as you have the proper iptables functionality enabled in your kernel.

host-tcplimit

host-tcplimit is a lot like tcplimit, but it limits new TCP connections coming in from a particular IP address and heading for a particular TCP port on your server(s). host-tcplimit is particularly useful for preventing a particular person from abusing your network resources. For example, let's say you're running a CVS server, and you discover that a particular new developer appears to have set up a script that updates his sources with the repository every 10 minutes, using up a huge amount of unnecessary network resources over the course of a day. However, while you're in the process of composing an e-mail to him explaining the error of his ways, you receive an incoming message that reads as follows:

Hi guys!

I'm really excited to be part of your development project. I just set up a
script to update my local copy of the code every ten minutes. I'm about to
leave on a two-week cruise, but when I get back, my sources will be totally
up-to-date and I'll be ready to help out! I'm heading out the door now...see
you in two weeks!

Sincerely,

Mr. Newbie

For such situations, a simple host-tcplimit command will solve the problem:

# host-tcplimit 1.1.1.1 2401 1 day on

Now, Mr. Newbie (IP address 1.1.1.1) is limited to one CVS connection (port 2401) per day, saving oodles of network bandwidth.

user-outblock

The last and possibly most intriguing of all my dynamic firewall scripts is user-outblock. This script provides an ideal way to allow a particular user to telnet or ssh into your system, yet not allow this user to establish any new outgoing connections from the command-line. Here's an example of a situation where user-outblock would come in handy. Let's say that a particular family has an account at my ISP. Mom and Dad use a graphical e-mail client to read their mail and occasionally surf the Web, but their son happens to be an aspiring hacker, and generally uses his shell access to do naughty things to other people's computers.

One day, you find that he's established ssh connections with several systems that appear to belong to the Pakistani military -- ouch. You'd like to help direct this youth towards more beneficial activities, so you do the following:

First, you do an audit of your system and make sure that you remove the suid bit from all your network binaries, like ssh:U

# chmod u-s /usr/bin/ssh

Now, any processes that he tries to use to interact with the network will be owned by his UID. You can now use user-outblock to block all outgoing TCP connections initiated by this UID (which happens to be 2049):

# user-outblock 2049 on
UID 2049 block on.

Now, he can log in and read his mail, but he's not going to be using your servers to establish ssh connections and the like. Now, he could install an ssh client on his home PC. However, it's not too hard to whip up another dynamic firewall script that limits his home PC to Web, mail, and outgoing ssh connections (to your servers only).

3.  Resources

Tarballs

Because I've found these dynamic firewall scripts so helpful, I've put together a neat little tarball (dynfw-1.0.1.tar.bz2) that you can download and install on your machine.

To install, extract the tarball and run the included install.sh script. This script will install a shared bash script to /usr/local/share/dynfw.sh, and install the dynamic firewall scripts themselves to /usr/local/sbin. If you'd like them to end up in /usr/share and /usr/sbin instead, simply type this before running install.sh:

# export PREFIX=/usr

I've also added a dynamic firewall scripts page to the Gentoo Linux Web site that you can visit to get the latest version of the tarball. I'd like to continue improving and adding to the collection, making a truly useful resource for sysadmins planetwide. Now that we have iptables in the kernel, it's time to start taking advantage of it!

If all this iptables firewall stuff is new to you, I highly recommend my Linux 2.4 stateful firewall tutorial (registration required), containing complete instructions on how to design your own iptables-based stateful firewall.

tcpdump is an essential tool for exploring low-level packet exchanges and verifying that your firewall is working correctly. If you don't have it, get it. If you've got it, start using it. If you're using it... good for you. :)

Visit the home page for the netfilter team to find lots of excellent resources, including the iptables sources, and Rusty's excellent unreliable guides. These include a basic networking concepts HOWTO, a netfilter (iptables) HOWTO, a NAT HOWTO, and a netfilter hacking HOWTO for developers. There's also a netfilter FAQ available, as well as other things.

Thankfully, there are a lot of good online netfilter resources; however, don't forget the basics. The iptables man page is very detailed and is a shining example of what a man page should be.

There's now an Advanced Linux Routing and Traffic Control HOWTO available. There's a good section that shows how to use iptables to mark packets, and then use Linux routing functionality to route the packets based on these marks.

There's a netfilter (iptables) mailing list available, as well as one for netfilter developers. You can also access the mailing list archives at these URLs.