New Mybubba.org servers

Announcements from Excito
Ubi
Posts: 1547
Joined: 17 Jul 2007, 09:01

New Mybubba.org servers

Post by Ubi » 30 Sep 2014, 07:39

As mentioned in another thread, the new domain mybubba.org will be taking over most functionality from the excito.org and excito.net domains.

The highest priority (copying all relevant data such as install images) as been completed.
I set up two servers (macaroni.mybubba.org and spaghetti.mybubba.org) and loaned out 2 IP addresses.

Macaroni:
Macaroni will be taking over most of the easyfind works and is mostly unconfigured. It has copies of all easyfind servers. Anyone with experience interested in porting the old easyfind infra to this server is very welcome. Current implementation uses apache, nginx, ruby and PHP (and maybe some perl somwhere).

Spaghetti:
This server has a number of http sites:
* update.mybubba.org
This takes over b3.update.excito.org. As it also listens to b3.update.excito.org we can transit this server without modifications on the client. Currently only the elvin and hugo repos are present. This service needs assistance from an experienced maintainer of debian repos, for example to upload new versions of the bash package.

* download.mybubba.org
This contains potentially relevant data from update.excito.net and download.excito.net. Also contains a copy of the excitostore.com website. Let me know if there's anything missing

* install.mybubba.org
This contains install images for the B1,B2 and B3

What else?
One thing that is missing in the current situation is off-site backups. The total installation is around 20GB, and incrementals should be very small. If you have a server that has a fixed IP, enough space and the possibility to run rsync via cron please let me know.

THanks in advance to anyone willing to help out.

MouettE
Site admin
Posts: 264
Joined: 06 Oct 2011, 19:45

Re: New Mybubba.org servers

Post by MouettE » 30 Sep 2014, 11:24

Ubi wrote:* update.mybubba.org
This takes over b3.update.excito.org. As it also listens to b3.update.excito.org we can transit this server without modifications on the client. Currently only the elvin and hugo repos are present. This service needs assistance from an experienced maintainer of debian repos, for example to upload new versions of the bash package.
I can help with that but I need to know to things :
  • What tool did excito use to maintain the repos
  • How should we sign the repo ? With excito keys or new ones ?

Ubi
Posts: 1547
Joined: 17 Jul 2007, 09:01

Re: New Mybubba.org servers

Post by Ubi » 30 Sep 2014, 11:37

1) Nobody knows the answer to this question.
2) I will supply you with new keys.

Gordon
Posts: 1350
Joined: 10 Aug 2011, 03:18

Re: New Mybubba.org servers

Post by Gordon » 30 Sep 2014, 17:16

I may be able to help with some bits and pieces of the easyfind system. Don't have a clue about ruby and rails though, but also question whether these are actually used because I don't really see the point of using a web-framework for simple text output.

I suggest you set up Nginx to serve port 80. With possibly 3000 bubbae connecting every minute to request their own IP address, Apache will simply put too much of a strain on the server. The following config file will get you going with (what seems to be all of) ef.excito.org functionality:

Code: Select all

server {
        listen 80;

        root /var/www;
        index index.html;

        server_name ef.excito.org;

        access_log /dev/null;
        error_log /dev/null;

        location /ip.json {
                default_type text/plain;
                echo "{\"ip_address\":\"$remote_addr\"}";
        }

       location /ip.xml {
                default_type text/plain;
                echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>";
                echo "<ip-addresses>";
                echo "  <ip-address>$remote_addr</ip-address>";
                echo "</ip-addresses>";
        }

        location / {
                default_type text/plain;
                echo "";
	}

}
No need for actual content here. The config file does it all and it should be blazing fast without requiring a lot of resources.

The actual DNS service and web based update routine will require some time, that currently won't fit in my schedule. Still... In the drawing you presented there was mentioning of a database being behind the DNS service. What DNS daemon is actually used here? And does it in fact reference a database (MySQL, PostgreSQL?) or is it flat file(s)?

Ubi
Posts: 1547
Joined: 17 Jul 2007, 09:01

Re: New Mybubba.org servers

Post by Ubi » 01 Oct 2014, 00:21

Thanks, nginx is probably a good idea. I'm just still contemplating whether to do it at all, considering I don't use it myself and I'm spending a lot of time already on getting the forum, wiki and updates secured. The problem is, i guess, that those that are able to migrate easyfind have almost by definition no need for it themselves. Excito had a clear business incentive to get easyfind running, but why would i spend my weekends on it? Not that i refuse to spend time on it ( i already have quite a bit), but the balance does seem a bit off at this stage.

Gordon
Posts: 1350
Joined: 10 Aug 2011, 03:18

Re: New Mybubba.org servers

Post by Gordon » 01 Oct 2014, 02:10

In that case I'd opt to build it from scratch. Unless of course it already uses what I'd use for this, which is Bind (the daemon is called 'named' and it has a command line tool called 'nsupdate' for accessing the dynamic content API). There are plenty examples to be found on the net for creating your own dyndns service and it should be easy enough to adapt them for this purpose. I could probably have a raw implementation of this in maybe as short as minutes, but it should be fine tuned and of course there is a difference between a household or small company private setup versus some 2000 legitimate users that can do updates and likely as many other servers querying the DNS at regular intervals. I simply don't have the time to do all that and if I don't you'll probably see smoke coming out of that server in no time. And of course the output must match what the client is expecting, or there's no sense in setting this up at all.

And you're right. I don't use easyfind. Not really anyway. The only reason why I have an easyfind name registered is to share some custom packages with other users. And I'll probably stop doing that when I decide to change the OS, because I won't be maintaining any Debian packages if I'm not running any Debian machines.

johannes
Posts: 1469
Joined: 31 Dec 2006, 07:12
Location: Sweden
Contact:

Re: New Mybubba.org servers

Post by johannes » 01 Oct 2014, 03:45

Gordon wrote:I may be able to help with some bits and pieces of the easyfind system. Don't have a clue about ruby and rails though, but also question whether these are actually used because I don't really see the point of using a web-framework for simple text output.
If I understood things correctly, it's a rails app producing this:
http://ef.excito.org/ip.json
(have a look in a browser, it's plain text.)
Gordon wrote: The actual DNS service and web based update routine will require some time, that currently won't fit in my schedule. Still... In the drawing you presented there was mentioning of a database being behind the DNS service. What DNS daemon is actually used here? And does it in fact reference a database (MySQL, PostgreSQL?) or is it flat file(s)?
NS1 and NS2 (the two Easyfind DNS servers) use PDNS. This is a brief overview of the easyfind process:

- once a minute, a B3 pings ef.excito.org to check it's public IP.
- only in the case of a change, B3 asks NS1 (aka easyfind.excito.org) with a change request, including MAC/secret key for authentication
- easyfind.excito.org authenticates against the "production" database, holding records of all B2/B3's ever manufactured, including MAC/Key pairs (or some scheme of that). Production is today on Amazon RDS and is quite expensive to run (> 300 EUR/month), for unknown reasons. Could be some bug triggering a lot of traffic, or stupid confguration of the RDS instance.
- if authenticated, exasyfind changes the PDNS record to the new B3 IP.

The reason for the downtime right now (if I understand thigs right) is that communication between easyfind.excito.org and production is down, due to expired certificates. So easyfind does work today, but only until your public IP changes.
/Johannes (Excito co-founder a long time ago, but now I'm just Johannes)

theWebalyst
Posts: 96
Joined: 27 May 2010, 14:53

Re: New Mybubba.org servers

Post by theWebalyst » 01 Oct 2014, 05:02

My ftpdns solution still looking "weird"? :-)

It can be up and running in a few minutes, free if you already have a website, and no load issues, or worries about who will maintain it when it goes wrong etc etc

I agree it is no good if you want to use it from an internet cafe (or other random machines), but that was not a concern for me. I use it for remote admin, and I only do that from my own machines. I think that, or not having web hosting already, are the main reasons for choosing something else.

Yes it is a hack, but if you want something that works and is easy to set up, give it a go!

https://github.com/theWebalyst/ftpdns

Mark

Gordon
Posts: 1350
Joined: 10 Aug 2011, 03:18

Re: New Mybubba.org servers

Post by Gordon » 01 Oct 2014, 07:17

Hi Johannes,

The method from the client (B2|3) point of view was known to me already. But obviously I can't look inside what are essentially black boxes somehow generating a response. Given the information you supplied here, building from scratch seems a better idea than attempting to run the current implementation on consolidated servers.
  1. The ping interval seems ridiculously short to me, and consequently the TTL that is set on the dynamic DNS entries as well. If people can wait for one minute, they can wait for 5 minutes as well if they happen to attempt to get to their Bubba right after it got a new IP. In any case, the Nginx vhost config posted above will reduce CPU and memory usage compared to the apparent current setup. You can try here for json or here for xml return. This is Nginx running on B3 on a 50M symetric line (~40M if I'm watching TV ;))
  2. I did a quick lookup on that PDNS. Seems to be highly appreciated, but I found some tests where the database backend proved to slow down its performance. On the other hand Bind was shown to be more CPU hungry, but my guess is that this is caused by the MySQL server consuming the remaining CPU time with PDNS. Serving from RAM seems like a better choice to me, specially because a maximum of 5000 hosts in two zones (forward and reverse) shouldn't take a huge amount of memory at all.
  3. I'm a bit amazed that you let the easyfind server directly reference production, rather than a local copy of the relevant data (key/mac0 combinations). The Amazon RDS issue seems disturbing. Do you get high traffic on the 'easyfind' webserver as well?
I think we should look into the clients as well. IMO we could restrict the number of accesses to the 'ef' server by only having Bubbae that are on RFC1918 addresses do it. The others can simply use their DHCP offerings to determine if an update is required.

johannes
Posts: 1469
Joined: 31 Dec 2006, 07:12
Location: Sweden
Contact:

Re: New Mybubba.org servers

Post by johannes » 01 Oct 2014, 07:52

Gordon wrote:
  1. The ping interval seems ridiculously short to me, and consequently the TTL that is set on the dynamic DNS entries as well. If people can wait for one minute, they can wait for 5 minutes as well if they happen to attempt to get to their Bubba right after it got a new IP.
Agree. The reason was for evaluation only, we did have some special cases (customers on 3G connection) where they got new leases every 30 or 60 minutes, and being down 5 of those 30 minutes was considered a to high downtime/uptime ratio. We planned a second revision where ping time was adjusted dynamically (frequent IP changes => low ping times) but never got around to it.
Gordon wrote: [*]I'm a bit amazed that you let the easyfind server directly reference production, rather than a local copy of the relevant data (key/mac0 combinations). The Amaz
on RDS issue seems disturbing. Do you get high traffic on the 'easyfind' webserver as well?[/list]
Don't know, we don't pay by traffic on that host (Glesys). Didn't find time to look into it. You are most welcome to dig if you feel like it, email me and I'll give you the credentials both to amazon and to Glesys.
/Johannes (Excito co-founder a long time ago, but now I'm just Johannes)

Ubi
Posts: 1547
Joined: 17 Jul 2007, 09:01

Re: New Mybubba.org servers

Post by Ubi » 01 Oct 2014, 10:21

There's two things here. First whether to keep easyfind funcitoning in the short term (which means copying in reality) and how to improve it.

Getting the easyfind infra transferred is not extremely difficult, My guess is around 10 hours for the whole thing. The implementation is actually done well, the code is nice and well thought through. It's just built for a much larger implementation than reality.

10 hours is basically a weekend. If someone can convince me why I (or somebody else that is capable of doing this) should spend a weekend on something I myself do not use, please do.

Ubi
Posts: 1547
Joined: 17 Jul 2007, 09:01

Re: New Mybubba.org servers

Post by Ubi » 05 Oct 2014, 02:42

ok, I set up ef.mybubba.org to reflect ef.mybubba.org using the recipe from gordon (thanks!). If Johannes can point ef.excito.org CNAME to ef.mybubba.org than we also have backwards compat.

johannes
Posts: 1469
Joined: 31 Dec 2006, 07:12
Location: Sweden
Contact:

Re: New Mybubba.org servers

Post by johannes » 05 Oct 2014, 05:04

Ubi wrote:If Johannes can point ef.excito.org CNAME to ef.mybubba.org than we also have backwards compat.
Done!
/Johannes (Excito co-founder a long time ago, but now I'm just Johannes)

Ubi
Posts: 1547
Joined: 17 Jul 2007, 09:01

Re: New Mybubba.org servers

Post by Ubi » 05 Oct 2014, 05:32

sweet.
Now the TTL on this record is 1 hour, so that's the time we need to wait at least to check the logfiles and count the number of IP addresses.

Ubi
Posts: 1547
Joined: 17 Jul 2007, 09:01

Re: New Mybubba.org servers

Post by Ubi » 05 Oct 2014, 08:17

ok, so after 3 hours we have a grand total of 708 unique IP addresses that contact easyfind. Load on the machine is exactly 0.00 thanks to gordons magic with nginx.

Post Reply