Ban IP

Blacklist IP addresses on multiple hosts

© 2014-2016 Dennis Leeuw dleeuw at made-it dot com
License: GPLv2 or later

Index

1. Introduction
1. The solution
2. Install and setup
1. The code
2. Install
3. Configuration
4. Testing the setup
3. Intergrate into fail2ban
4. Distributed use
1. lighttpd setup
2. test the setup
5. Some important notes

Most people will probably be using fail2ban or DenyHosts to protect their machines against brute force password attacks. After a certain amount of failures the offending IP is blocked in the firewall, so no further tries can be made. If you have more then one server tied directly to the Internet, the same tools are probably running on all these machines and after a certain amount of time all will be blocking the same IP offending address. In the meantime however a distributed attack across all your machines gives the bad guys (I assume guyes here, but it could be girls also) more accounts to test, since they distribute there database across your hosts assuming you are using one account database for all you hosts. So if you have set the amount of failures to 3 and have 10 machines they could try 30 accounts with passwords before getting blocked. So what I wanted was that if a single machine detects an attempt to break in it will block the IP address on all machines that I manage and that is what the orginal ban-ip stuff was all about.

In the meantime OpenSSH removed support for hosts.allow and hosts.deny files making DenyHosts absolete. People trying to break also got smarter and started testing account/password combinations once every half our, making most of the detection tools not notice them. And the last trigger to completely rewrite ban-ip was the fact that fail2ban has no whitelist support. Not having a whitelist means I have to enable IP addresses forever since people with just a couple of public/private-key pairs are blocked really fast when they have just three changes of doing it right.

The world is not ideal, so I had to figure out a way to get arround all those items. The first thing I was pointed to on the fail2ban mailinglist is this: http://yalis.fr/cms/index.php/post/2014/11/02/Migrate-from-DenyHosts-to-Fail2ban. That kills part of the problem. But I was not happy. It all seemed too difficult, and difficult things tent to break. So I looked again at my ban-ip stuff and what is now presented here as ban-ip is a total rewrite of what I had back then. The solution is now a independant daemon that maintain a backlist and a whitelist, a commandline tool to control the daemon, some filter and action files for fail2ban, and a PHP-script for the distribution of offending IPs. Enjoy!

We want to detect brute force attacks, but allow brute force SSH keys. With the above described problems we need to make it very unattractive for people trying to break in. So let's say we will block an IP address as soon as we detect 5 failed log in attempts in 12 hours. This way testing every half our is detected. So it effectively means that you can get away with it when you try to log in on our systems once every 2 hours (just a little more). I think they will soon look for another target to attack. The downside of this setting is that regular users will probably make typos in a worse rate then that, and esp. when they also use public/private keys they will be blocked within these 12 hours, so we need a whitelist to prevent them from being blocked. I do not know about you, but I do not want to maintain a whitelist for all the external IP addresses of our users, so it needs to be automatically maintained list.

The solution is simple. As soon as a user logs in he or she is an authorized user and thus the IP address is authorized. This IP address is then added to a whitelist with a time stamp. After that a whitelisted IP adres is never added to the blacklist. As soon as 36 hours are passed the IP address is removed from the whitelist. But everytime a user logs in from that same address the timestamp is reset in de database. Regular users with daily log ins will always be on the whitelist and will thus never get blocked. Those that are less frequent should just type their password correctly within 5 tries (within a 12 hour time span)... that seems reasonable to me.

Released versions:

26 jan 2016:	0.8.2
25 jan 2016:	0.8.1
21 jan 2016:	0.8

All tools expect the ban-ip command to be available as /opt/ban-ip/ban-ip, since it is all scripting you can easily adjust it, but for now it is assumed that the provided tar-archive is extracted in /opt and that a symbolic link is present from the current version to ban-ip.

Example for version 0.8:

cd /opt
tar zxvf /path/to/downloaded/ban-ip-0.8.tar.gz
ln -s ban-ip-0.8 ban-ip

Create a /etc/ban-ip directory and copy the configuration file (ban-ip.conf) into it:

mkdir /etc/ban-ip
cp /opt/ban-ip/ban-ip.conf /etc/ban-ip

If you want to make use of the distribted ban functionality copy the ban-ip.php to the root of your webserver. More details are provided in the chapter called "Distributed use".

The configuration file is extensively documented, for the configuration options we refer to the /etc/ban-ip/ban-ip.conf file.

NOTE: Currently ban-ip only uses ipset to maintain blocklists. It can however easily be expanded to support e.g. plain iptables, but I haven't found the time to implement it.

The ban-ip tool has a couple of functions:

Tie the ipset table to iptables

To make sure iptables uses our table use:

ban-ip link

Add an IP address to the blacklist table

block an ip address:

ban-ip ban 1.1.1.1

Delete an IP address from the blacklist table

remove a blocked ip address:

ban-ip unban 1.1.1.1

Add an IP address to the whitelist table

whitelist an ip address:

ban-ip whitelist 1.1.1.1

Delete an IP address from the whitelist table

remove a whitelisted ip address:

ban-ip unwhitelist 1.1.1.1

Show the current members of a list. blacklist or whitelist can be added to the command to show a certain list. Without any name the blacklist is shown.

ban-ip show

Start and stop the ban-ipd

For this the script has a start, stop and a restart function. Create the appropriate symlinks to the script from the different runlevels and your blocklist will be preserved over a reboot. It is important that this script is started before your fail2ban server starts, since the daemon is responsible for setting up the tables. Also make sure it intergrates correctly with your already present other firewall rules

To start using ban-ip start the daemon:

/opt/ban-ip/ban-ip start

and make sure that it runs:

/opt/ban-ip/ban-ip status
Daemon: running

After starting the daemon have an empty table ready to be filled with IP addresses that you want to block. Of course they are of no use if iptables is not looking at them, so create the link between iptables and ipset by running:

/opt/ban-ip/ban-ip link

This is now a working system in which you can manually ban/unban and whitelist/unwhitelist IP addresses. Now every kind of tool you want to use can be integrated with this to call the /opt/ban-ip/ban-ip script to ban and unban IP addresses. In the next chapter we will describe how we did it using the fail2ban tools.

fail2ban is a really good tool to watch your logfiles and tell some external tool about the problems it found. To make fail2ban do what we want it to do we create a /etc/fail2ban/jail.local file:

[DEFAULT]
ignoreip = 127.0.0.1/8
bantime = 1
# findtime: 12 hours
findtime  = 43200
maxretry = 5
action = ban-ip

[sshd]
enabled = true

[sshd-ddos]
enabled = true

[pam-generic]
enabled = true

Now we need to teach fail2ban that there is also something like "good users". We do this by creating a filter that reads the SSH log file and filters out succesful ssh log ins. We stored this filter as /etc/fail2ban/filter.d/ssh-whitelist.local:

[INCLUDES]

before = common.conf

[Definition]

_daemon = sshd

failregex = ^%(__prefix_line)sAccepted \S+ for \S+ from \s.*$

ignoreregex =

Next we extend /etc/fail2ban/jail.local with an action plan:

[ssh-ok]
enabled = true
maxretry = 1
logpath = %(sshd_log)s
filter = ssh-whitelist
action = ban-ip-whitelist

Next we need to create the action that we called in the previous step. Create /etc/fail2ban/action.d/ban-ip-whitelist.local and fill it with:

[Definition]
actionstart =
actionstop =
actioncheck =
actionban = /opt/ban-ip/ban-ip whitelist 
actionunban =

[Init]
name = default

The same we need to do for the regular blocking of IP addresses. So we add an additional action as /etc/fail2ban/action.d/ban-ip.local that looks like this: [INCLUDES] [Definition] actionstart = actionstop = actionban = /opt/ban-ip/ban-ip ban actionunban = [Init] name = default So here also fail2ban is mostly doing nothing.

When we start fail2ban it will do what we want it to do: Detect succesful log ins and tell ban-ip about it and detect failed log ins and after 5 such attempts in 36 hours it also tells ban-ip about it. Sounds like a prefect detection system.

The easiest way to distribute a blacklist or an offending IP address is probably by using a well tested protcol like HTTP. No need to reinvent the wheel.

Within the code package there is a PHP-file which provides a couple of functions which makes it possible to receive an detected IP address to be blocked. To be precise it can add and delete IPs and it can list the the blacklist. The ban-ipd uses curl to send the IP address to a webserver providing the PHP-file and the PHP-program calls /opt/ban-ip/ban-ip with a command to make it all happen. To be able to call the ban-ip command from a webserver, the user under which the webserver is running should be able to call the ban-ip script. To accomplish this one could use sudo (or something else):

lighttpd	localhost=(ALL)			NOPASSWD: /opt/ban-ip/ban-ip *

Check that sudo is working correctly by using:

sudo -u lighttpd sudo /opt/ban-ip/ban-ip show

The path to ban-ip is hardcoded in the PHP file, so if you installed the script in another place then change it in the above sudoers example and also in the PHP code.

The /etc/ban-ip/ban-ip.conf file contains 2 settings for our destributed system that need to be correct. First there is the notify_array that should hold the IP-addresses or names of the servers to contact and second notify_php_script is the complete path and filename to the PHP-script on the remote server. So something like this might work:

notify_php_script="/ban-ip.php"
notify_array=(192.168.1.5:6543 192.168.1.6:6543)

The ban-ip.php file can be used on a single host that maintains a database for all your hosts, or you could add an HTTP server to all your servers and add this PHP-script to all of them. We assume you will also use lighttpd and that the PHP-script is installed in /var/www/lighttpd. If you use another HTTP-server skip this section.

Install lighttpd, spawn-cgi, php-fpm. For the different distributions we have:

	Package	php-fpm.conf dir	www.conf dir
Debian	php5-fpm	/etc/php5/fpm	/etc/php5/fpm/pool.d/
Debian	spawn-cgi
Debian	lighttpd
Red Hat	php-fpm	/etc	/etc/php-fpm.d
Red Hat	spawn-cgi
Red Hat	lighttpd

Adjust in php-fpm.conf the daemonize variable and set it to yes. Adjust the www.conf file and set:

listen = 127.0.0.1:6542
listen.allowed_clients = 127.0.0.1

On Debian based systems make sure /etc/php5/cli/php.ini and on Red Hat based systems make sure /etc/php.d/lighttpd.ini that:

cgi.fix_pathinfo = 1

lighttpd provides a /etc/lighttpd directory with the configuration, we are not going to use the distro specific setup, but create our own, just to keep this document simple. Create in the configuration directory a lighttpd.conf with the following content:

server.modules = (
	"mod_access"
)

server.username        = "lighttpd" # www-data on Debian
server.groupname       = "lighttpd" # www-data on Debian
server.port            = 6543
server.use-ipv6        = "disable"
server.max-connections = 10 # amount of servers that can connect at once

# Some variables we could use
var.www_root_dir     = "/var/www/lighttpd"
var.log_root_dir     = "/var/log/lighttpd"
var.tmp_upload_dir   = "/var/tmp/lighttpd"

# File locations
server.upload-dirs   = ( tmp_upload_dir )
server.document-root = www_root_dir
server.errorlog      = log_root_dir + "/error.log"
server.pid-file      = "/var/run/lighttpd.pid"

# Content handling
index-file.names               = ( "index.php", "index.html", "index.htm" )
url.access-deny                = ( "~", ".inc" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi", ".scgi" )
server.follow-symlink          = "disable"
server.dir-listing             = "disable"

# For 2.6 kernels and up
server.event-handler = "linux-sysepoll"

# linux-sendfile - for small files.
# writev         - for large files
server.network-backend = "linux-sendfile"

# Found on CentOS 6:
##
## disable range requests for pdf files
## workaround for a bug in the Acrobat Reader plugin.
##
$HTTP["url"] =~ "\.pdf$" {
  server.range-requests = "disable"
}

# Pick one
# Debian:  include_shell "/usr/share/lighttpd/create-mime.assign.pl"
# Red Hat: include "conf.d/mime.conf"

server.modules += ( "mod_fastcgi" )
fastcgi.server = ( ".php" =>
                   ( "localhost" =>
                     (
                       "host" => "127.0.0.1",
                       "port" => 6542
                     )
                   )
                 )

With this in place restart lighttpd and php-fpm or php5-fpm.

Create a test.php in /var/www/lighttpd and fill it with: <?php phpinfo(); ?>. Now you test your setup by accessing this file from your browser. If all is fine, remove test.php from the directory and you are ready to go.

To control the black list use:

http://localhost:6543/ban-ip.php?command=add&ip=123.123.123.123

http://localhost:6543/ban-ip.php?command=del&ip=123.123.123.123

You can also list all IPs in the table by using:

http://localhost:6543/ban-ip.php?command=list

Important NOTE: It is easy for a normal user to fool the fail2ban system by using something like the following on the command line:

logger -p auth.warning -t 'sshd[666]' 'Invalid user GuessMyName from 1.2.3.4'
logger -p auth.warning -t 'sshd[666]' 'Invalid user GuessMyName from 1.2.3.4'
logger -p auth.warning -t 'sshd[666]' 'Invalid user GuessMyName from 1.2.3.4'

Another important NOTE is that the current limit on an ipset table is 65535 entries, which you can change using the maxelem option when creating the table.