I have been attacked and don't know why.

Got problems with your B2 or B3? Share and get helped!
Post Reply
Gordon
Posts: 1356
Joined: 10 Aug 2011, 03:18

Re: I have been attacked and don't know why.

Post by Gordon » 11 Nov 2013, 03:35

Hi Johannes,

I sent you a PM

Ubi
Posts: 1547
Joined: 17 Jul 2007, 09:01

Re: I have been attacked and don't know why.

Post by Ubi » 11 Nov 2013, 03:40

The way it works is that the apache config sets an "executable" flag to a specific folder (using the ScriptAlias command). This contrasts to other folders were Apache flat-out refuses to execute any script or binary. We generally call this executable folder /cgi-bin/ (a remainder from the days of old), but if you would call the folder /ubi-is-great/ it would also work (most likely better!). So Apache exposes the cgi-bin folder to the outside and allows remote users to directly address and execute any executable file within that dir with web-server UID permissions. The only added security is that file listing is prohibited.

As an example: about a decade ago many web servers had a sendmail.cgi in their /cgi-bin/ folder (which was basically a perl script, renamed to .cgi). This script required the caller to set the sender and recipient address, making it the greatest spam machine ever. So a combination of FTP access and the possibility to execute scripts from remote made the cgi-bin a great attack vector, as you could combine a stolen FTP-password with a local privilege-escalation to effectively gain a remote-root quality exploit.

Nowadays PHP pretty much does the exact same thing as CGI scripts, but does not require the cgi-bin any more. You could argue that this bypasses a layer of security and makes the server inherently more insecure but alas. Anyway that is the reason very few ISPs offer it as a service any more (of my 600 domains nobody uses it anymore).

does this help?

johannes
Posts: 1470
Joined: 31 Dec 2006, 07:12
Location: Sweden
Contact:

Re: I have been attacked and don't know why.

Post by johannes » 11 Nov 2013, 05:23

Thanks Ubi and Gordon, and yes, one of the devs already knew this apparently and is working on a second fix right now. Will keep you informed.
/Johannes (Excito co-founder a long time ago, but now I'm just Johannes)

Gordon
Posts: 1356
Joined: 10 Aug 2011, 03:18

Re: I have been attacked and don't know why.

Post by Gordon » 11 Nov 2013, 06:48

Ubi wrote:Nowadays PHP pretty much does the exact same thing as CGI scripts, but does not require the cgi-bin any more. You could argue that this bypasses a layer of security and makes the server inherently more insecure but alas. Anyway that is the reason very few ISPs offer it as a service any more (of my 600 domains nobody uses it anymore).
Actually ASP and Java/Tomcat allow doing this as well. And I actually think that having your own code in place offers better security than a generic email script that allows setting both sender and destination.

As far as not needing cgi-bin any more, you'll find that most ISPs will actually not use mod_php but do in fact offer html preprocessing by php cgi. The reason for this is that when using mod_php this allows everyone that can put files on that server to run code as user www-data, effectively allowing them to snoop in each others (web-accessible) files and retrieve stuff like mysql database passwords in php files. Running php in cgi mode allows running it as the owner of that particular website (using suexec) and thus constrains him to his own web root.

Ubi
Posts: 1547
Joined: 17 Jul 2007, 09:01

Re: I have been attacked and don't know why.

Post by Ubi » 11 Nov 2013, 07:15

yes but this does not use the /cgi-bin/ in a way that it is user-accessible

gonk
Posts: 93
Joined: 30 May 2012, 01:53

Re: I have been attacked and don't know why.

Post by gonk » 11 Nov 2013, 14:02

Performing the update via the web interface initially showed some progress but then stuck at "0%" for over 30 minutes so I refreshed the page and got the following:

Code: Select all

Package name	Package version
bubba	2.6.0.1
bubba-album	2.6
bubba-backend	2.6
bubba-frontend	2.6.0.1+security1
bubba3-kernel	1:2.6.39.4-11
filetransferdaemon	0.55
logitechmediaserver	7.7.2.dfsg1-1
Can anyone confirm that this is a correctly patched system?

Before doing the update I removed the crontab for www-data and did some manual cleaning of /tmp but what else is recommended to be cleaned?

Is it recommended and/or needed to change the shell of www-data and denying it to have a crontab as mentioned earlier in this thread or is that of no use?

I've blocked access to port 80 on B3 on my external firewall.

gonk
Posts: 93
Joined: 30 May 2012, 01:53

Re: I have been attacked and don't know why.

Post by gonk » 11 Nov 2013, 14:53

I've found suspicious activity on the following URLs returning else than HTTP 404

1) //%63%67%69%2D%62%69%6E/%70%68%70?%2D%64+%61%6C%6C ...
2) //cgi-bin/php5?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63 ...
3) /cgi-bin/php/%63%67%69%6E/%70%68%70?%2D%64+%61%6C%75%6F%6E+%2D% ...
4) /cgi-bin/php?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63 ...
5) /cgi-bin/php5?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63 ...

Type 1 has had 10 attempts with OK (200) and 4 with 504 (timeout)
Type 2 has had no attempts with OK (200) and 4 with 504 (timeout)
Type 3 has had 14 attempts with OK (200)
Type 4 has had 50 attempts with OK (200) and 38 with 504 (timeout)
Type 5 has had 54 attempts with OK (200), 12 with internal server error (500) and 26 with 504 (timeout)

The first attaks were detected 2013-10-31 and until stopped today the hits per day was

Code: Select all

2013-10-31	 4
2013-11-01	 4
2013-11-02	 8
2013-11-03	12
2013-11-04	26
2013-11-05	20
2013-11-06	30
2013-11-07	10
2013-11-08	40
2013-11-09	16
2013-11-10	32
2013-11-11	10
In total the origin IP-addresses are 23 distinct values.

gonk
Posts: 93
Joined: 30 May 2012, 01:53

Re: I have been attacked and don't know why.

Post by gonk » 11 Nov 2013, 15:29

One picture says it all...
Attachments
Load average 2013-10-20 to 2013-11-11
Load average 2013-10-20 to 2013-11-11
load during b3 attack.png (12.12 KiB) Viewed 9052 times

Ubi
Posts: 1547
Joined: 17 Jul 2007, 09:01

Re: I have been attacked and don't know why.

Post by Ubi » 11 Nov 2013, 16:07

Its interesting. Im seeing over 1000 established connections on my machine, but no funny business in the processlist. Also, port 80 is blocked from outside, as are nearly all others.

http://kaitain.ideeel.net/munin/mail-dn ... index.html

ryz
Posts: 183
Joined: 12 Feb 2009, 06:03

Re: I have been attacked and don't know why.

Post by ryz » 12 Nov 2013, 01:17

Ubi wrote:How would you propose excito goes about thic anouncement then? There isnt exactly a registration requirement for purchase of a B2/3, so nobody knows who the customers are. Would you like an advert in all of the worlds newspapers?

Oh and when was the last time you heard Apple or Microsoft hand out a fix within 6 days after discovery? I think theyre doing a pretty good job.
I think you both miss the big picture here, I do not either see a big problem in how they have announce this threat. The problem is how they handle security fixes which is not at all. The fix for this bug was released 08 May 2012 that is over a year ago. Even when people pointing out that this exploit is used to break into B3:s they are not able to see that there is an fix already for squeeze but think they have to back port a new version of PHP. That shows unfortunately that they do not have a good way of handling security updates. How many other critical security vulnerabilities that have been fixed in Debian is still unpatched in the B3?

I which I had more time to start an community which could help Excito with these sort of things since I do realise that supporting the B2 and B3 has to take a lot of resources and hence it is very hard to make any real profit from selling these things. You more support we demand the less incitement it is for Excito to keep selling B3:s. Fortunately they seem to have an pride in there products and really want to make it as good as they can with the few resource they have.

johannes
Posts: 1470
Joined: 31 Dec 2006, 07:12
Location: Sweden
Contact:

Re: I have been attacked and don't know why.

Post by johannes » 12 Nov 2013, 02:45

rys, thanks for understanding, you are spot on the situation. One comment though, don't mistake my dumb questions over Excitos general lack of knowledge, we do have two skilled devs who know about this. The B3 is not their main project though, hence the long response time in some cases.

I fully agree though that we should have been faster letting security updates through.
/Johannes (Excito co-founder a long time ago, but now I'm just Johannes)

gonk
Posts: 93
Joined: 30 May 2012, 01:53

Re: I have been attacked and don't know why.

Post by gonk » 12 Nov 2013, 03:24

gonk wrote:Before doing the update I removed the crontab for www-data and did some manual cleaning of /tmp but what else is recommended to be cleaned?

Is it recommended and/or needed to change the shell of www-data and denying it to have a crontab as mentioned earlier in this thread or is that of no use?
I'm still highly interested in hearing your suggestions/dideas relating to these two topics.

Gordon
Posts: 1356
Joined: 10 Aug 2011, 03:18

Re: I have been attacked and don't know why.

Post by Gordon » 12 Nov 2013, 06:06

gonk wrote:Before doing the update I removed the crontab for www-data and did some manual cleaning of /tmp but what else is recommended to be cleaned?
That's hard to answer, since it is impossible to foresee what hackers will actually do once they gained access. However I think you could run `find -user www-data` to retrieve a list of files owned by www-data and determine whether some of them should not have been there.
gonk wrote:Is it recommended and/or needed to change the shell of www-data and denying it to have a crontab as mentioned earlier in this thread or is that of no use?
Changing the shell is only sensible to disallow specific users to log on directly, using a tty, ssh or telnet connection. That is not what is happening in this exploit (although I currently do not know the specifics) and will therefore not stop it. And crontab can be easily replaced by running an infinite loop with sleep, which is of course easier to detect but you'll still need to look for it to see it.

kenned
Posts: 12
Joined: 27 Feb 2011, 13:55

Re: I have been attacked and don't know why.

Post by kenned » 12 Nov 2013, 18:11

I recently found out my b3 had been compromised too. Had to happen sooner or later, since I don't show it much lovin' and it has been exposed to the internet. :)

I know it's not a fix, but I added www-data to /etc/cron.deny to disable crontab for the apache user.
But then I also blocked apache and most other services from the outside, since the easyfind service doesnt work with my 4g/LTE USB-Modem setup, so for the last many months while the evil hackers have been abusing my b3 for bitcoins and whatnot, I I have been unable to access my system from the internet. you can call that a lose-lose situation. :)

But I also searched for files owned by www-data to see if there was anything else besides the files in /tmp, and the only suspicious thing I can find is /var/lock/ttoy, a directory with a hidden subdir called ".m" and another subdir called "c". No files though, but maybe that's because whatever created the lock dir/files has been killed off.

Code: Select all

root@b3:/var/log# ls -laR /var/lock/ttoy/
/var/lock/ttoy/:
total 12
drwxr-xr-x 3 www-data www-data 4096 Nov 10 23:37 .
drwxrwxrwt 6 root     root     4096 Nov 12 23:19 ..
drwx------ 3 www-data www-data 4096 Nov 10 23:37 .m

/var/lock/ttoy/.m:
total 12
drwx------ 3 www-data www-data 4096 Nov 10 23:37 .
drwxr-xr-x 3 www-data www-data 4096 Nov 10 23:37 ..
drwxr-xr-x 2 www-data www-data 4096 Nov 10 23:37 c

/var/lock/ttoy/.m/c:
total 8
drwxr-xr-x 2 www-data www-data 4096 Nov 10 23:37 .
drwx------ 3 www-data www-data 4096 Nov 10 23:37 ..
root@b3:/var/log#
Anybody have a comment on /var/lock/ttoy? My google-fu gives me nothing at all on it.

Ubi
Posts: 1547
Joined: 17 Jul 2007, 09:01

Re: I have been attacked and don't know why.

Post by Ubi » 13 Nov 2013, 02:17

/var/lock is a nice place to hide malware, as it is a dumping ground for arbitrary files. It seems you can remove /var/lock/ttoy entirely. The fact that its owned by www-data strongly implies it is part of the attack vector.

Post Reply