Fail2ban or csf ?

A collection of tips on howto tweak your Bubba.
Post Reply
toukie
Posts: 115
Joined: 13 Jan 2012, 12:22

Fail2ban or csf ?

Post by toukie » 16 Apr 2012, 16:44

I tried fail2ban but didn't see any action with it, http://www.configserver.com/cp/csf.html is an alternative. Would it work with B3? My first try just messed up things. I guess that I missed something. Has anyone tried these (successfully)?

Gordon
Posts: 1356
Joined: 10 Aug 2011, 03:18

Re: Fail2ban or csf ?

Post by Gordon » 17 Apr 2012, 10:46

According to the home page CSF may require rewriting some regex rules on Debian. Sounds like tricky business.

Fail2ban seems more promising to me, but that may be because I like Shorewall - I wrote a little howto on running this on the B3 just a few topics down. I'm not sure about the "TCP-Wrapper" dependency though, which appears to be required to catch certain bad actions and transform them into an IP block.

As it happens I'm currently working on a little (private) project to shield my own B3 from bad internet users. As I discovered not to long ago, something that used to be very complicated (it involved patching the kernel) has become very simple indeed. Linus hates it and Shorewall also makes it hard to implement, so I'm now back to basics and combining my own rules around the "racist patch" - an iptables rule that filters on geographic location. As a second set of rules I'm using ipset to maintain lists of IP addresses that should either be always allowed, always denied, or maybe identified as being suspicious and get limited access.

One of the challenges is to have the blacklist - those that should be blocked - automatically update itself. One way I found is to create a trap page in apache, which is actually the 404 page on my default host that has no other pages. This is a part I got working, although not yet implemented on the B3 (the whole project is still on VM). If you're interested I can share once I'm confident enough that it doesn't nuke the B3.

toukie
Posts: 115
Joined: 13 Jan 2012, 12:22

Re: Fail2ban or csf ?

Post by toukie » 17 Apr 2012, 15:58

Gordon: "If you're interested...."

For sure I am. I'm interested in tools to use for blocking certain bad activities on the server. This is what I want to get rid of:

203.211.140.185 - - [17/Apr/2012:18:09:55 +0200] "GET /muieblackcat HTTP/1.1" 404 469 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:09:55 +0200] "GET //index.php HTTP/1.1" 404 466 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:09:56 +0200] "GET //admin/index.php HTTP/1.1" 200 4563 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:09:57 +0200] "GET //admin/pma/index.php HTTP/1.1" 404 617 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:09:58 +0200] "GET //admin/phpmyadmin/index.php HTTP/1.1" 404 617 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:09:58 +0200] "GET //db/index.php HTTP/1.1" 404 469 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:09:59 +0200] "GET //dbadmin/index.php HTTP/1.1" 404 472 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:00 +0200] "GET //myadmin/index.php HTTP/1.1" 404 472 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:01 +0200] "GET //mysql/index.php HTTP/1.1" 404 472 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:01 +0200] "GET //mysqladmin/index.php HTTP/1.1" 404 475 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:02 +0200] "GET //typo3/phpmyadmin/index.php HTTP/1.1" 404 478 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:03 +0200] "GET //phpadmin/index.php HTTP/1.1" 404 472 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:03 +0200] "GET //phpMyAdmin/index.php HTTP/1.1" 404 474 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:04 +0200] "GET //phpmyadmin/index.php HTTP/1.1" 404 473 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:05 +0200] "GET //phpmyadmin1/index.php HTTP/1.1" 404 474 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:06 +0200] "GET //phpmyadmin2/index.php HTTP/1.1" 404 474 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:06 +0200] "GET //pma/index.php HTTP/1.1" 404 470 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:07 +0200] "GET //web/phpMyAdmin/index.php HTTP/1.1" 404 477 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:08 +0200] "GET //xampp/phpmyadmin/index.php HTTP/1.1" 404 478 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:08 +0200] "GET //web/index.php HTTP/1.1" 404 469 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:09 +0200] "GET //php-my-admin/index.php HTTP/1.1" 404 475 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:10 +0200] "GET //websql/index.php HTTP/1.1" 404 472 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:11 +0200] "GET //phpmyadmin/index.php HTTP/1.1" 404 473 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:11 +0200] "GET //phpMyAdmin/index.php HTTP/1.1" 404 474 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:12 +0200] "GET //phpMyAdmin-2/index.php HTTP/1.1" 404 476 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:13 +0200] "GET //php-my-admin/index.php HTTP/1.1" 404 475 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:13 +0200] "GET //phpMyAdmin-2.2.3/index.php HTTP/1.1" 404 476 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:14 +0200] "GET //phpMyAdmin-2.2.6/index.php HTTP/1.1" 404 478 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:15 +0200] "GET //phpMyAdmin-2.5.1/index.php HTTP/1.1" 404 479 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:16 +0200] "GET //phpMyAdmin-2.5.4/index.php HTTP/1.1" 404 479 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:16 +0200] "GET //phpMyAdmin-2.5.5-rc1/index.php HTTP/1.1" 404 482 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:17 +0200] "GET //phpMyAdmin-2.5.5-rc2/index.php HTTP/1.1" 404 482 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:18 +0200] "GET //phpMyAdmin-2.5.5/index.php HTTP/1.1" 404 479 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:19 +0200] "GET //phpMyAdmin-2.5.5-pl1/index.php HTTP/1.1" 404 482 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:19 +0200] "GET //phpMyAdmin-2.5.6-rc1/index.php HTTP/1.1" 404 482 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:20 +0200] "GET //phpMyAdmin-2.5.6-rc2/index.php HTTP/1.1" 404 482 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:21 +0200] "GET //phpMyAdmin-2.5.6/index.php HTTP/1.1" 404 479 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:21 +0200] "GET //phpMyAdmin-2.5.7/index.php HTTP/1.1" 404 479 "-" "-"
203.211.140.185 - - [17/Apr/2012:18:10:22 +0200] "GET //phpMyAdmin-2.5.7-pl1/index.php HTTP/1.1" 404 482 "-" "-"

Fail2ban just doesn't ban anything. It says that "If time reference is not the same everywhere, then fail2ban won't ban any IP!"

Logs have to be synchronized: http://www.fail2ban.org/wiki/index.php/FAQ_english

I just don't know how to synchronize them if that's the problem.

I was curious to know if anybody had fail2ban running. Mine is running but it doesn't do the job.

Gordon
Posts: 1356
Joined: 10 Aug 2011, 03:18

xtables-addons and ipset

Post by Gordon » 18 Apr 2012, 10:08

Okay then...

Here's a small draft.

Let's start with required software. You need 'xtables-addons' and 'ipset' and you want 'sudo' (you need to be root to run ipset). Since not all of these are in the excito repository, we'll need to update the sources list:

Code: Select all

# Create new file squeeze.list in the sources include path
cat > /etc/apt/sources.list.d/squeeze.list <<EOSL
## Debian Squeeze sources.list

## Debian NL mirror:
deb http://ftp.nl.debian.org/debian squeeze main contrib non-free
deb-src http://ftp.nl.debian.org/debian squeeze main contrib non-free

EOSL

# Get the repositories index
apt-get update
Now for some reason Debian split up the xtables-addons in a xtables-addons-common and a non-existing xtables-addons-modules:
root@b3:~# aptitude show xtables-addons-common
Package: xtables-addons-common
State: not installed
Version: 1.26-2+b1
Priority: extra
Section: admin
Maintainer: Pierre Chifflier <pollux@debian.org>
Uncompressed Size: 401 k
Depends: iptables, libc6 (>= 2.7)
Recommends: xtables-addons-modules
Conflicts: ipset
Replaces: ipset
Provides: xtables-addons-common-1.26
Description: Extensions targets and matches for iptables [tools, libs]
Xtables-addons provides extra modules for iptables not present in the kernel, and is the successor of patch-o-matic. Extensions includes new targets like
TEE, TARPIT, CHAOS, or modules like geoip, ipset, and account.

This package provides the userspace libraries for iptables to use extensions in the xtables-addons-modules package.

Note: this package is only useful with a corresponding xtables-addons-modules package, which you may produce with module-assistant:

module-assistant auto-install xtables-addons-source
Homepage: http://xtables-addons.sourceforge.net/
Since I already have build essentials and kernel source available, I opted to install from source (if I need to go through trouble, then I want the latest version also).

Get the required Debian packages:

Code: Select all

aptitude install sudo xz-utils
Now install xtables-addons:

Code: Select all

# Change to source folder
cd /usr/src

# Get the source
wget http://sourceforge.net/projects/xtables-addons/files/Xtables-addons/1.42/xtables-addons-1.42.tar.xz/download && mv download xtables-addons-1.42.tar.xz

# unpack and compile
xzcat xtables-addons-1.42.tar.xz | tar xv
cd xtables-addons-1.42.tar.xz
./configure
make && make install

Edit: as it turns out both the binary and source ipset packages provided by Debian are broken. Ipset also requires kernel modules to be compiled, but luckily the official ipset source handles all of this in a single step:

Code: Select all

cd /usr/src
wget http://ipset.netfilter.org/ipset-4.5.tar.bz2
tar xjvf ipset-4.5.tar.bz2 
cd ipset-4.5
make && make install
depmod -A
Last edited by Gordon on 19 Apr 2012, 07:37, edited 1 time in total.

Gordon
Posts: 1356
Joined: 10 Aug 2011, 03:18

xtables-addons: Geoip

Post by Gordon » 18 Apr 2012, 10:09

If you like to use the geoip match, you need to have a database (and maintain it regularly).

Create the correct folder first (the location is hardcoded in the source):

Code: Select all

mkdir /usr/share/xt_geoip
There are two scripts provided in the xtables-addons source to build the geoip database: a shell script that fetches the master data from Maxmind (which actually contains an error) and a perl script that sequences it for use with the xtables match module. Copy these scripts to a more logical location and build a wrapper for them that can be called from crontab.

Code: Select all

mkdir /usr/share/xt_geoip/bin
cp -t /usr/share/xt_geoip/bin  /usr/src/xtables-addons-1.42/geoip/xt_geoip_dl \
      /usr/src/xtables-addons-1.42/geoip/xt_geoip_build
cat > /usr/share/xt_geoip/bin/updater  << EOUPD
#!/bin/bash
cd /usr/share/xt_geoip/

# Next is the non-functional line from xt_geoip_dl (requires bash to run correctly)
rm -f GeoIPv6.csv{,.gz} GeoIPCountryCSV.zip GeoIPCountryWhois.csv;

./bin/xt_geoip_dl
./bin/xt_geoip_build -D /usr/share/xt_geoip/ GeoIP*.csv
EOUPD
chmod +x /usr/share/xt_geoip/bin/*
Next run the updater script and add a monthly schedule for it to crontab

Try it:

Code: Select all

# I want to be able to read my email when I'm away from home
iptables -A INPUT -i eth0 -p tcp --dport 993 -m geoip --src-cc NL -j ACCEPT

Gordon
Posts: 1356
Joined: 10 Aug 2011, 03:18

Example firewall script

Post by Gordon » 19 Apr 2012, 07:28

Here's an example script:
#!/bin/sh

# Which countries to allow access to private services
GEOIP_ALLOWED=NL,BE

# Names of the ipsets used by the firewall
IPSETS="blacklist whitelist"

# Which modules to load (some do not auto-load)
MODULES="nf_conntrack_ftp ip_set ip_set_hash_ip xt_geoip xt_set"


# Insert modules
for module in ${MODULES} ; do
modprobe ${module}
done

# Create ipsets
for ipset in ${IPSETS} ; do
ipset -N ${ipset} iphash
done

# Clear all rules
iptables -F
iptables -X

# Set policy (temporary set INPUT to accept all)
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD DROP


# Define chain to log blacklisted IP's
iptables -N blacklisted
iptables -A blacklisted -j LOG --log-prefix "Firewall:IP blacklisted:DROP:" --log-level 6
iptables -A blacklisted -j DROP


# What should always be open (except for blacklisted IP's)
iptables -N public
iptables -A public -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A public -m icmp -p icmp --icmp-type 3/4 -m comment --comment "Needed ICMP types" -j ACCEPT
iptables -A public -m icmp -p icmp --icmp-type 11 -m comment --comment "Needed ICMP types" -j ACCEPT
iptables -A public -m tcp -p tcp --dport 113 -m comment --comment Auth -j REJECT
iptables -A public -m tcp -p tcp --dport 25 -j ACCEPT
iptables -A public -m tcp -p tcp --dport 80 -j ACCEPT


# Define chain for private services (ftp, https, imaps)
iptables -N nonpublic
iptables -A nonpublic -m tcp -p tcp --dport 21 -j ACCEPT
iptables -A nonpublic -m tcp -p tcp --dport 443 -j ACCEPT
iptables -A nonpublic -m tcp -p tcp --dport 993 -j ACCEPT
iptables -A nonpublic -j LOG --log-prefix "Firewall:Port forbidden:DROP:" --log-level 6
iptables -A nonpublic -j DROP

# ================================================

# INPUT rules
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i br0 -j ACCEPT

iptables -A INPUT -i eth0 -m set --match-set blacklist src -j blacklisted
iptables -A INPUT -i eth0 -p tcp -j public
iptables -A INPUT -i eth0 -m set --match-set whitelist src -j nonpublic
iptables -A INPUT -i eth0 -m geoip --src-cc ${GEOIP_ALLOWED} -j nonpublic
iptables -A INPUT -j LOG --log-prefix "Firewall:IP not allowed:DROP:" --log-level 6

iptables -P INPUT DROP
Do note that this script creates empty ipset lists, which is good for testing but should be replaced by some additional logic to save and restore these lists between reboots.

Gordon
Posts: 1356
Joined: 10 Aug 2011, 03:18

Populating the blacklist

Post by Gordon » 19 Apr 2012, 08:25

I'm truly sorry. As said I have been experimenting with this setup in a VM and had not yet done any tests on the B3 itself. If you already installed ipset using aptitude then please remove it. You need to install from source to make this work.

Code: Select all

aptitude remove ipset
cd /usr/src
wget http://ipset.netfilter.org/ipset-4.5.tar.bz2
tar xjvf ipset-4.5.tar.bz2
cd ipset-4.5
make && make install
depmod -A
So now in the above example script you can instantly drop an unwanted client by issuing the following command:

Code: Select all

ipset add blacklist 10.10.10.10
And here's how you can add this to a custom 404 page:

Code: Select all

<?php 
  $clientIP=$_SERVER["REMOTE_ADDR"];
  system("/usr/bin/sudo /usr/local/sbin/ipset add blacklist $clientIP");
?>
Of course to make the 404 page work you must add the apache user to the list of sudoers:

Code: Select all

echo "www-data ALL=(root)NOPASSWD:/usr/local/sbin/ipset" > /etc/sudoers.d/apache

toukie
Posts: 115
Joined: 13 Jan 2012, 12:22

Re: Fail2ban or csf ?

Post by toukie » 19 Apr 2012, 15:48

Far too advanced for me, Gordon!
I would like to have the neat jails of fail2ban for to put all the bad bots in them.

I'll give fail2ban one more try.

It would be nice if this will fix it: https://www.ailis.de/~k/archives/59-Fix ... l2ban.html

Gordon
Posts: 1356
Joined: 10 Aug 2011, 03:18

Re: Fail2ban or csf ?

Post by Gordon » 20 Apr 2012, 09:41

How about I create a package for the software you need, so you can go straight to the example firewall script?

toukie
Posts: 115
Joined: 13 Jan 2012, 12:22

Re: Fail2ban or csf ?

Post by toukie » 21 Apr 2012, 13:59

Something smart that would end the phpMyAdmin-nuisance on the server would be very welcome.

I am quite new to things, learning to use command-line since maybe two months, so the more advanced things have to wait.

It's the unwanted behaviour that has to be stopped. The bots are all over the place, they don't come from one single country only.

Gordon
Posts: 1356
Joined: 10 Aug 2011, 03:18

Re: Fail2ban or csf ?

Post by Gordon » 22 Apr 2012, 03:20

Oh, but that's not the point. The geoip match is just a neat trick to limit as many people as possible without blocking myself when I'm in a known place with an unknown IP address. If you have no use for such a feature, then don't use it.

The part that matches your particular search is the ipsets. These are dynamic lists of IP addresses that you can also use as a firewall match rule. In the example script I used an ipset called "blacklist" to push offenders onto a different track (it's called a chain in iptables) and make them pass through a specific set of rules (which is just to log them and then drop).

The key trick here is to get offenders automatically added to that list, i.e. to redirect the service or webpage that they're trying to use to a script or active webpage that does this for us. So that's where we tell the Apache webserver not to show the regular 404 page, but one that we created ourselves and contains the bit of code shown above. Of course you probably want to run some tests, because people that mean no harm might actually trip on a error on your webpage itself or try to reach an old page on it that was indexed by a search engine (such as Google).

Post Reply