New user's registration have been closed due to high spamming and low trafic on this forum. Please contact forum admins directly if you need an account. Thanks !

understanding the power of bubba

Got problems with Bubba? Then this forum is for you.
Locked
dsp76
Posts: 76
Joined: 15 Apr 2007, 14:18

understanding the power of bubba

Post by dsp76 »

Hi,
I have a virtual debian server running at my internet provider, maintaining it with virtualmin. The Server has 128MB RAM guaranteed, however I hit the limit quite often, just with Apache, MySQL, PHP4, Dovecot, FTP and the virtualmin package...

Now I learnt from their developers, that 128MB is a bit low for that purposes. Now I read a similiar setup for Bubba, also read about PHP. Bubba has only 64MB - how can this work?

dsp76
tor
Posts: 703
Joined: 06 Dec 2006, 12:24
Contact:

Post by tor »

Hi,

I think this is much about load and most importantly how you use your mysql server. MySQL will easily eat all your memory if you have large databases and a lot of access to them.

That said its still very possible to run MySQL on Bubba as well. With moderate load and not to big quantity of data it will run just fine. And if you then tune the DB it should perform reasonably good even with a lot of data. (If anyone is interested i can share our configs.)

And sure, Bubba will run out of ram sometimes when doing everything one wants to do. But then we of course have a swapfile to solve it. But swap is always very costly timewise and should be avoided if possible.

/Tor
Co-founder OpenProducts and Ex Excito Developer
limpo
Posts: 59
Joined: 24 Mar 2007, 06:50

Post by limpo »

HI,

Yes Tor, please share the optimized mysql config. Bubba can use all the speed/optimization it can get. :)

Please share if you have any other tweaks for freeing up more ram or releaseing some cpu load.


/Limpo
dsp76
Posts: 76
Joined: 15 Apr 2007, 14:18

Post by dsp76 »

tor wrote: I think this is much about load and most importantly how you use your mysql server. MySQL will easily eat all your memory if you have large databases and a lot of access to them.
Mhm, lets see:
I think about installing a Wiki (for mainly my own usage) and maybe a SyncML Server, an IMAP accounts, FTP... (could also stop it and use SFTP via SSH)...

Difficult to say if this is considered much load? (Should be more or less my own communication machine...)

dsp76
Clive
Posts: 164
Joined: 07 Mar 2007, 07:15

Post by Clive »

dsp76 wrote:
tor wrote: I think this is much about load and most importantly how you use your mysql server. MySQL will easily eat all your memory if you have large databases and a lot of access to them.
Mhm, lets see:
I think about installing a Wiki (for mainly my own usage) and maybe a SyncML Server, an IMAP accounts, FTP... (could also stop it and use SFTP via SSH)...

Difficult to say if this is considered much load? (Should be more or less my own communication machine...)

dsp76
Well although I do not have a Wiki loaded, I do have MySQL and Wordpress running and it is perfectly respectable speed wise. One of my concerns before I bought Bubba was the speed issue and this has proved not a problem. For all the things you mention Bubba should work fine.
Indeed for only 10watts of power it works more than fine !
spoodie
Posts: 21
Joined: 24 Jan 2007, 13:32

Post by spoodie »

limpo wrote:Yes Tor, please share the optimized mysql config. Bubba can use all the speed/optimization it can get. :)
I'd like this also, I'm using mySQL for TorrentFlux and I'd like to minimise the impact. Thanks.
tor
Posts: 703
Joined: 06 Dec 2006, 12:24
Contact:

Post by tor »

Hi,

I cant give you a optimized config file that works in all cases. What i can do is try to give some pointers that most likely will save you some memory depending on how you use the system.

First of. Disable any servers that you aren't using. Ie, streaming servers, ftp, mail and print servers. Only run what you actually use.

Then when running a LAMP system there are two applications that most likely will eat a lot of memory, Apache and MySQL.

When looking at Apache the main source in saving memory is by not running more servers than you actually need. The settings for this can be found in /etc/apache2/apache2.conf and the parameters that is interesting is the section that looks like this:

Code: Select all

<IfModule prefork.c>
StartServers         5
MinSpareServers      5
MaxSpareServers     10
MaxClients          20
MaxRequestsPerChild  0
</IfModule>
If the system doesn't have many concurrent users decreasing StartServers, MinSpareServers to say 3 or possibly even 2 instead of five and lower MaxSpareServers to say 5 or 6.

If the system further more occasionally uses a lot of memory it can be good to increase the value of MaxRequestsPerChild say for example 500.

When values are changed restart Apache to apply the changes.

Code: Select all

/etc/init.d/apache2 restart
When one looks at MySQL there are quite a few to fiddle with. Most of this information is based on the provided example config for small systems from the package. More info on this is available in /usr/share/doc/mysql/my-small.cnf

MySQLs config can be found in the file /etc/mysql/my.cnf The interesting parameters can be found in the [mysqld] section.

Parameters to fiddle with and suggested new values.

key_buffer original 16M suggested change 16-32K
max_allowed_packet original 16M suggested change 1M
thread_stack original 128K suggested change 64K
query_cache_size original 16M suggested change 8M

Further more it could perhaps save even more memory by disable query_cache all together by settng query_cache_type to 0

And to do even more adding these parameters to the same section can also be beneficial:

Code: Select all

table_cache = 4
sort_buffer_size = 64K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
net_buffer_length = 2K
When changes are made, restart MySQL by issueing:

Code: Select all

/etc/init.d/mysql restart
That said. All above information wont work in all situations, they are targeted to minimize memory consumption. Everything here depends on how Bubba is used. Concurrent users, database size etc. So to find an optimum one most likely will have to test and change and test again.

Other things beyond this that can be interesting to evaluate to squeeze performance out of your Bubba is to look at php-accelerators. They can save CPU cycles if that is the problem. Example on such is Turck MMCache, eaccelerator and APC these however will of course eat memory as well and are not tested in any way by Excito.

To be able to tell if there has been any changes its good to be able to somewhat measure performance of the system. And an easy way to do that is to use the command

Code: Select all

vmstat -1
This command will write some vital statistics on the system every second. The easiest way to use this is to log in via ssh on Bubba and in the terminal start vmstat and pipe output to a file

Code: Select all

vmstat -1 > trace.log
and then while this is running make sure you have a natural load on the system.

When enough tests have been performed kill the vmstat command and evaluate the logfile produced.

There are a lot of interesting information in this file regarding memory and CPU load.

Code: Select all

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  18452   1840   1648  14532    1    6    15     7  147    42  5  1 93  0
The first two columns tells us how many processes that is runnable at any given moment and how many processes that are blocked in uninteruptable sleep.

Then comes 8 columns that are very interesting memory wise. swpd tells us how much swap that is used. free tells us how much physical memory that is free. buff and cache tells us how much memory Linux uses to buffer and cache data. Then comes si and so that tells us how much data has been swapped in or out from disk. Next is bi and bo that is a measurement of how many blocks have been read and written to and from blockdevices.(Most likely the disk in Bubbas case) The next section tells us about the system in how many interrupts per second we experience and cs the amount of process switches per second. Lastly we have information regarding CPU load. us is the time spent executing in user mode, sy is time spent in kernel mode, id is idle time not used for anything, wa is time spent waiting for IO.

This information can be used to tell if the system is IO-bound, Memory-bound or CPU bound. That will say spending its time waiting for data, waiting for available memory to or from swap or waiting for CPU time to execute.

Depending on what this shows one should take different measurements. Examples above can help when memory is in short demand or if to much time is spent parsing php-pages.

/Tor
Co-founder OpenProducts and Ex Excito Developer
bubbalibre
Posts: 117
Joined: 01 Feb 2007, 09:33
Location: Paris, France

Post by bubbalibre »

Nice piece of work. Thanks for taking the time to write it down.
limpo
Posts: 59
Joined: 24 Mar 2007, 06:50

Post by limpo »

HI,

Great job on the guide Tor:), you rule!

By the way I found the following guide to optimize, do you know if any of it is usable on the bubba?

"Below is a compilation of some methods one may use to optimize a web server to serve web pages faster and lower the overall CPU load on the box.

1. Basic Config File Editing

Make a backup of your /etc/my.cnf file, and then perform the following via SSH:

pico /etc/my.cnf
Add in the following entries or replace the current existing ones:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
skip-locking
skip-innodb
query_cache_limit=8M
query_cache_size=256M
query_cache_type=1
max_connections=500
max_user_connections=10
interactive_timeout=20
wait_timeout=20
connect_timeout=6
thread_cache_size=128
key_buffer=16M
join_buffer=1M
max_allowed_packet=16M
table_cache=1024
record_buffer=1M
sort_buffer_size=2M
read_buffer_size=2M
max_connect_errors=10
# Try number of CPU's*2 for thread_concurrency
thread_concurrency=4
myisam_sort_buffer_size=64M
#log-bin
server-id=1

[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/lib/mysql/mysql.pid
open_files_limit=8192

[mysqldump]
quick
max_allowed_packet=16M

[mysql]
no-auto-rehash
#safe-updates

[isamchk]
key_buffer=32M
sort_buffer=32M
read_buffer=16M
write_buffer=16M

[myisamchk]
key_buffer=32M
sort_buffer=32M
read_buffer=16M
write_buffer=16M

Hit CTRL + X to exit and save the file

Now to edit the httpd.conf:

pico /usr/local/apache/conf/httpd.conf (or wherever your httpd.conf is located)

Set "Timeout" value to "Timeout 300"
Change "KeepAlive on" to "KeepAlive off"
Set "MinSpareServers" to "MinSpareServers 8"
Set "MaxSpareServers" to "MaxSpareServers 13"
Set "MaxRequestsPerChild" to "MaxRequestsPerChild 50"
Set "HostnameLookups" to "HostnameLookups Off"

CTRL + X to exit and save the file

Restart Apache and MySQL with:
service httpd restart
service mysql restart OR service mysqld restart

Some people may need to restart the services with /etc/rc.d/init.d/httpd restart and /etc/rc.d/init.d/mysql restart

The above changes can be altered and played around with to suit your server's needs (i.e. if your server is fast or not).

2. Installing Mod_Perl

Mod_Perl description: "mod_perl gives you a persistent Perl interpreter embedded in your web server. This lets you avoid the overhead of starting an external interpreter and avoids the penalty of Perl start-up time, giving you super-fast dynamic content. "

Run these commands via SSH:

wget http://perl.apache.org/dist/mod_perl-1.0-current.tar.gz

tar zxvf tar zxvf mod_perl-1.0-current.tar.gz

cd mod_perl-1.29 (or whatever folder is generated)

perl Makefile.PL

If you see any errors about missing dependencies (such as CGI.pm, LWP::UserAgent & HTML::HeadParser required by mod_perl) you may install them with:

perl -MCPAN -e shell
install HTML::HeadParser

If you had to install any dependencies you must re-run "perl Makefile.PL". Back to setting up mod_perl, after perl Makefile.PL run:

make
make install

Congratulations, you've just installed mod_perl. Restart Apache with:

service httpd restart
OR
/etc/rc.d/init.d/httpd restart


3. Install Turck MMCache for PHP

Description: "Turck MMCache is a free open source PHP accelerator, optimizer, encoder and dynamic content cache for PHP. It increases performance of PHP scripts by caching them in compiled state, so that the overhead of compiling is almost completely eliminated. Also it uses some optimizations to speed up execution of PHP scripts. Turck MMCache typically reduces server load and increases the speed of your PHP code by 1-10 times. "

Turck MMCache requires: apache 1.3, mod_php 4.1, autoconf, automake, libtool, m4. You should already have most of these on your server if not use the "apt-get install <dependency here>" command to get them installed. I won't go into detail about this here, you should easily be able to search the net to get them installed.

To install Turck MMCache, perform the following commands via SSH:

wget http://aleron.dl.sourceforge.net/source ... 4.6.tar.gz

export PHP_PREFIX="/usr"

$PHP_PREFIX/bin/phpize

./configure --enable-mmcache=shared --with-php-config=$PHP_PREFIX/bin/php-config

(You must specify the real prefix where PHP is installed in the "export" command. It may be "/usr" "/usr/local", or something else.)

make

make install

Turck MMCache can be installed both as Zend or PHP extension, so you need to edit your php.ini file (usually /etc/php.ini).
To install as Zend extension:

zend_extension="/usr/lib/php4/mmcache.so"
mmcache.shm_size="16"
mmcache.cache_dir="/tmp/mmcache"
mmcache.enable="1"
mmcache.optimizer="1"
mmcache.check_mtime="1"
mmcache.debug="0"
mmcache.filter=""
mmcache.shm_max="0"
mmcache.shm_ttl="0"
mmcache.shm_prune_period="0"
mmcache.shm_only="0"
mmcache.compress="1"

To install as PHP extension:

extension="mmcache.so"
mmcache.shm_size="16"
mmcache.cache_dir="/tmp/mmcache"
mmcache.enable="1"
mmcache.optimizer="1"
mmcache.check_mtime="1"
mmcache.debug="0"
mmcache.filter=""
mmcache.shm_max="0"
mmcache.shm_ttl="0"
mmcache.shm_prune_period="0"
mmcache.shm_only="0"
mmcache.compress="1"

You may need to copy the mmcache.so file to the directory specified to the above paths in the configuration entries.

Create the cache directory:
mkdir /tmp/mmcache
chmod 0777 /tmp/mmcache

Restart Apache with:

service httpd restart
OR
/etc/rc.d/init.d/httpd restart


4. Tuning sysctl.conf

The sysctl.conf of a server is something that is seldom optimized for performance. You can get a tremendous boost in throughput by adjusting these settings. This configuration has been written by Steve from Rack911. I have applied this configuration to servers ranging from Celeron 1.7Ghz to Dual Xeon 2.8Ghz servers, and on the whole, the load on each lowered after making the changes.

First make a backup of your old /etc/sysctl.conf file by running the following command, logged in as root:

cp /etc/sysctl.conf /etc/sysctl.conf.bak

Now enter:
pico /etc/sysctl.conf

and replace the contents of the file with the following:


# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.

# Disables packet forwarding
net.ipv4.ip_forward=0

# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.eth0.log_martians = 0

# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Disables the magic-sysrq key
kernel.sysrq = 0

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time = 1800

# Turn off the tcp_window_scaling
net.ipv4.tcp_window_scaling = 0

# Turn off the tcp_sack
net.ipv4.tcp_sack = 0

# Turn off the tcp_timestamps
net.ipv4.tcp_timestamps = 0

# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1

# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 1

# Increases the size of the socket queue (effectively, q0).
net.ipv4.tcp_max_syn_backlog = 1024

# Increase the tcp-time-wait buckets pool size
net.ipv4.tcp_max_tw_buckets = 1440000

# Allowed local port range
net.ipv4.ip_local_port_range = 16384 65536

CTRL + X to exit and save the file

To make your changes take effect immediately, type this command:
/sbin/sysctl -p

You can watch your server load by entering "uptime" command via SSH.

There you have it, a quick few things you can do to your server to boost performance and lower CPU load. Please feel free to post any comments or suggestions."


/Limpo
Locked