Ridiculously high load

1. is this on dedicated or shared virual server

Two dedicated servers, one for HTTP one for MySQL

2. your server specs. For example:

cpu speed/type single or dual cpus):

HTTP: Single Pentium IV 2.8ghz, Hyperthreading enabled
MySQL: Dual G4 1.25 Ghz

how much memory installed:

HTTP: 1 gigabyte
MySQL: 2 gigabytes

hard drive type/configuration:

HTTP: 250 gigabyte SATA
MySQL: 200 gigabyte IDE

linux distributor or windows version:

HTTP: Mandrake 10
MySQL: OSX 10.3 Server

apache/IIS version:

Apache 2.0.48

PHP version: php 4.3.4
MySQL version: mysql 4.0.15

3. if you use mysql 4.x instead of mysql 3.23.x, do you have any innodb type databases/tables on your server ?

No

4. if possible how mysql was compiled/installed

Binary distribution

5. your top stats

top - 12:06:41 up 17:40, 3 users, load average: 191.26, 148.06, 103.97
Tasks: 340 total, 3 running, 337 sleeping, 0 stopped, 0 zombie
Cpu(s): 14.2% us, 51.6% sy, 0.0% ni, 33.7% id, 0.0% wa, 0.3% hi, 0.2% si
Mem: 1032328k total, 993920k used, 38408k free, 58728k buffers
Swap: 10908092k total, 24288k used, 10883804k free, 61852k cached

And for MySQL box

Processes: 74 total, 4 running, 70 sleeping... 455 threads 12:35:45
Load Avg: 0.98, 0.95, 0.82 CPU usage: 4.5% user, 11.7% sys, 83.8% idle
SharedLibs: num = 114, resident = 28.6M code, 2.69M data, 8.10M LinkEdit
MemRegions: num = 10026, resident = 183M + 10.5M private, 45.0M shared
PhysMem: 156M wired, 227M active, 1.60G inactive, 1.98G used, 21.5M free
VM: 4.64G + 79.0M 103258(0) pageins, 47506(0) pageouts

6. your mysql configuration variables located at /etc/my.cnf or c:my.cnf so post the contents inside of my.cnf (minus any passwords of course). If on Windows server if you don't have that file you need to log into telnet and as root user type

# The MySQL server
[mysqld]
port = 3306
datadir = /Volumes/web/database/
socket = /tmp/mysql.sock
skip-locking
key_buffer = 16M
join_buffer_size=2M
max_allowed_packet = 32M
table_cache = 1024
sort_buffer_size = 2M
read_buffer_size = 2M
myisam_sort_buffer_size = 128M
thread_cache = 8
query_cache_size = 32M
query_cache_limit=1M
query_cache_type=1
wait_timeout=1800
connect_timeout=10
max_connect_errors=10
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8
max_connections=500
[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

7. your mysql extended-status output which shows entire server's mysql activity status (preferably with a day or 2 worth of mysql uptime for better indication of real activity) either:

http://www.murmurs.com/talk/extended-status.php

8. is your vB the only thing on the server? or other scripts & sites which utilise php and mysql?

Yes

9. how many average and max concurrent users on your vB forum ? and what your cookie timeout is ?

Between 200 and 400, low of about 100

10. create a file named phpinfo.php and place this code in it and post the url/link to it from your web site

http://www.murmurs.com/talk/phpinfo.php

11. if you run Apache and you have your own dedicated server or access to your httpd.conf (apache configuration file) can you post the values you have set for the following :

From httpd.conf (don't know if its used):

MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 4
MaxSpareServers 20
StartServers 4
MaxClients 256
MaxRequestsPerChild 500

From httpd2.conf

(default)
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 256
MaxRequestsPerChild 0

StartServers 2
MaxClients 256
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0

12. what version of vB are you running ?

3.0.1

How are the 2 boxes connected? Just one NIC card in each? Are they 10 base?We saw something like this not too long ago and switching to dual nic's with one dedicated to comunicating between the 2 servers cleared things up rather nicely.

load average: 191.26, 148.06, 103.97

:eek:

Install a reactive firewall or some sort of anti-dos module on your webserver.

IIRC, SYN floods happen during the tcp 3-step handshake. This isn't something switching web servers will help. However, since we're using Linux, this:
iptables -A INPUT -p tcp --tcp-flags SYN,RST -m limit --limit 2/second --limit-burst 5 -j DROP
should get the job done nicely.

I totally agree. If it is SYN flood, block it with firewall rules, if it is MySQL performance issues, tune the configuration or add more memory, if it is (D)DoS attack at HTTP level, find a way to detect and fend it. The admin has to know the root cause of the problem. :)

Most time, the extremely high load is caused by excessive memory usage when system start swapping. Apache's big memory footprint and lacking of control over system resource usage is a common source of this kind of problems. Switching web server does not help in case of a SYN flood, but it may help when a web site is under attack at HTTP level. :)

Maybe you should turn off keepalive or use shorter keepalive timeout.
Good luck with Apache performance tuning. ;)

besides a firewall for reasons erwin listed above I'd do the following in this order

1. upgrade MySQL to 4.0.20 (4.0.21 seems less stable)
2. upgrade PHP to 4.3.8 and remove Zend Optimiser entry in php.ini
3. install Turck Mmcache http://www.vbulletin.com/forum/showthread.php?t=75878
4. change to use this new /etc/my.cnf on mysql database server below
[mysqld]
port = 3306
datadir = /Volumes/web/database/
socket = /tmp/mysql.sock
skip-locking
skip-innodb
max_connections = 500
key_buffer = 64M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1500
thread_cache_size = 64
wait_timeout = 1800
connect_timeout = 10
max_allowed_packet = 16M
max_connect_errors = 10
query_cache_limit = 2M
query_cache_size = 64M
query_cache_type = 1

[mysqld_safe]
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout
and restart mysql server
5. upgrade to Apache 2.0.50 on web server
6. upgrade to vB 3.03 for performance/code enhancements
7. ensure your web server's motherboard bios has Hyperthreading enabled, if it didn't have hyperthreading enabled, enable it in motherboard bios and then recompile 2.6.x latest stable smp kernel on web server

I tightened up the firewall significantly, but I'm still having performance problems. The debug mode says the scripts are executing quickly, but pages are loading slowly with images lagging.

Current Apache settings are:

StartServers 40
MinSpareServers 35
MaxSpareServers 50
MaxClients 200
MaxRequestsPerChild 0

I could not get mmcache running yet because of a problem (see that thread). I get a Failed loading /mmcache/turck-mmcache-2.4.6/modules/mmcache.so: /mmcache/turck-mmcache-2.4.6/modules/mmcache.so: undefined symbol: output_globals_id

error.

MySQL's load is very low, so I don't think its that box. Might this be caused by something in Mandrake intrinsically? I'm thinking of switching to Fedora.

Here's a current top:

top - 10:59:48 up 21:01, 2 users, load average: 1.26, 2.91, 4.83
Tasks: 172 total, 4 running, 168 sleeping, 0 stopped, 0 zombie
Cpu(s): 8.1% us, 29.4% sy, 0.3% ni, 61.9% id, 0.0% wa, 0.2% hi, 0.2% si
Mem: 1032328k total, 1000008k used, 32320k free, 115792k buffers
Swap: 10908092k total, 50804k used, 10857288k free, 435012k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6205 apache 15 0 20764 14m 12m S 20.3 1.4 0:14.61 httpd2
6188 apache 15 0 20824 13m 12m S 17.9 1.4 0:16.16 httpd2
6202 apache 15 0 20712 13m 12m S 17.9 1.4 0:20.86 httpd2
6216 apache 16 0 21876 14m 12m S 7.6 1.5 0:11.88 httpd2
6176 apache 16 0 21188 14m 12m S 4.3 1.4 0:18.34 httpd2
6201 apache 15 0 20316 13m 12m S 4.0 1.3 0:15.00 httpd2
6223 apache 15 0 20564 13m 12m R 1.0 1.4 0:19.27 httpd2
3945 ethank 34 19 97.9m 8968 95m S 0.7 0.9 5:59.71 krozat.kss
1570 root 16 0 87560 2508 78m S 0.3 0.2 4:22.14 X
6215 apache 15 0 20436 13m 12m S 0.3 1.3 0:13.55 httpd2
6242 apache 15 0 20516 13m 12m S 0.3 1.4 0:05.48 httpd2
6371 root 16 0 2176 1096 1964 R 0.3 0.1 0:00.05 top
1 root 16 0 1580 476 1424 S 0.0 0.0 0:02.72 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1

Tried that. Its very strange because even static files load slowly. Might it be a bug in this distribution of Linux? I can't figure out what the problem is and its very frustrating.

Looks like too much memory being used on your web server machine, any thing else other than Apache 2.0 on it?

If the problem persists after the upgrade, you can try our LiteSpeed Web Server, which with anti-DoS in nature, ;) , the free standard edition should work well.

An update.

I moved the site to my Mac OSX server that was acting as only a MySQL server. Everything is working much, much better now. No high load, fast response.

So I'm thinking it is something with Mandrake. I'm going to install Fedora Core and see if that helps.

ethank, just wondering, why did you choose a mac to be your database server? :)

It seems this was caused by a SYN flood.

#If you have any other info about this subject , Please add it free.#

7 January 2009 | cameltoepants.com | edit