Attention nerds: Any idea what could be wrong with this site?

[Dan Druff]

Some of you might remember about a year and a half ago (I think) I switched servers.

PFA is on a dedicated server (actually a virtual private server, but I digress), and it was running quite well. Then it started having high server load issues. At first I thought it was a DOS attack, but it wasn't. In fact, I wrote a program to thwart most forms of DOS attacks, and it wasn't catching any.

From examining the running processes, I saw that the server load was occurring mainly due to both http calls and mysql calls. While there are indeed a lot of web page requests (http) and database calls (mysql) here, nothing significant had changed, yet these calls were causing the difference between load averages of 0.2-0.8 in the past to ones like 40-100 in the then-present. I couldn't figure it out, especially because it didn't appear that one user or person was causing those high-load http/mysql stats.

I contacted the tech support people at the host company. They don't work for free, but they are better at sysadmin stuff than I am, so they looked into it. Three different techs came up blank. Nobody could figure it out. I actually got my money back for their services because everyone sheepishly admitted they were stumped.

I came up with my own idea that perhaps something had gone corrupt within the server that was very hard to diagnose/discover, and that switching to another one might be the answer. They dismissed my idea, but finally allowed me to give it a shot.

Despite the fact that I backed up the PFA software and database and restored an exact copy on the new server, the problem was fixed. Same traffic, same database calls, but suddenly the load average was back to under 1.00 and everything was cool.

Now I'm starting to see something similar happen. Load averages keep spiking up intermittently, but tonight it was as high as 188 (!!) when radio was supposed to start. I don't believe it was a DOS or DDoS attack.

This time, however, it was all mysql calls that were the problem.

Question: Is it possible for a database to be corrupt, but then operate normally when it is backed up and restored on a new machine? I would think no, but perhaps I'm wrong on this one.

Thoughts?

I really don't feel like switching again (it's a huge pain in the ass), but I can't keep having this problem with radio like I did this week.

[khalwat]

Despite the fact that I backed up the PFA software and database and restored an exact copy on the new server, the problem was fixed. Same traffic, same database calls, but suddenly the load average was back to under 1.00 and everything was cool.

How did you do the database backup and restore? mysqldump ?

Question: Is it possible for a database to be corrupt, but then operate normally when it is backed up and restored on a new machine? I would think no, but perhaps I'm wrong on this one.

If your indexes keep becoming corrupt, it's possible. Assuming you rebuilt the database by just dumping the raw sql via mysqldump and restored it the same way, the indexes would be rebuilt when you did that.

Thoughts?

I really don't feel like switching again (it's a huge pain in the ass), but I can't keep having this problem with radio like I did this week.

I'd suggest first monitoring it with mytop so you can get a look at what's actually happening:

http://jeremy.zawodny.com/mysql/mytop/mytop.html

You need more data. Also while the CPU goes through the roof, also check on the state of pageins in case the VM is thrashing.

[Dan Druff]

Yes, I'm using mysqldump.

I will try dumping it and re-creating it. I will also get mytop.

Hopefully I can get this solved.

Thanks.

[sonatine]

What is dmesg telling you?

Also consider bumping up file descriptors/ulimit, so on... man sysctl for that whole rabbit hole.

[sonatine]

Tuning a linux host for database optimization is not easy but it can be rewarding to the point of essential once you start dealing with a certain volume of data.

[khalwat]

Yes, I'm using mysqldump.

I will try dumping it and re-creating it. I will also get mytop.

Hopefully I can get this solved.

Thanks.

Is that what you did the first time? Because you can't simply copy mysql databases over... lots of bad things can happen if you do that.

You need to dump them via mysqldump and restore/recreate them the same way. Make sure you delete the database using DROP DATABASE xxxxx; in mysql> to properly delete it, otherwise you'll have stale MYD, MYI, etc. files.

Usually I just mysqldump from the original database, and create a new database via mysqldump and point the web app at the new database, to be safe, and give you an easy way to roll back.

Also consider absolutely starting from scratch, so you can reclaim the space from ibdata1 as per:

http://stackoverflow.com/questions/3...-file-in-mysql

[khalwat]

Some of you might remember about a year and a half ago (I think) I switched servers.

PFA is on a dedicated server (actually a virtual private server, but I digress), and it was running quite well. Then it started having high server load issues. At first I thought it was a DOS attack, but it wasn't. In fact, I wrote a program to thwart most forms of DOS attacks, and it wasn't catching any.

I'd just use iptables for this.

http://blog.bodhizazen.net/linux/pre...with-iptables/

You can also use it to prevent repeated login/hacking attempts:

iptables -A INPUT -p tcp -m tcp --dport 22 -m tcp -m state --state NEW -m recent --set --name SSH --rsource

iptables -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 600 --hitcount 8 --rttl --name SSH --rsource -j REJECT --reject-with icmp-host-prohibited

iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

Then make sure you whitelist your own IP(s):

iptables -I INPUT -s your_ip_address_here -j ACCEPT

[Dan Druff]

Some of you might remember about a year and a half ago (I think) I switched servers.

PFA is on a dedicated server (actually a virtual private server, but I digress), and it was running quite well. Then it started having high server load issues. At first I thought it was a DOS attack, but it wasn't. In fact, I wrote a program to thwart most forms of DOS attacks, and it wasn't catching any.

I'd just use iptables for this.

http://blog.bodhizazen.net/linux/pre...with-iptables/

You can also use it to prevent repeated login/hacking attempts:

iptables -A INPUT -p tcp -m tcp --dport 22 -m tcp -m state --state NEW -m recent --set --name SSH --rsource

iptables -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 600 --hitcount 8 --rttl --name SSH --rsource -j REJECT --reject-with icmp-host-prohibited

iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

Then make sure you whitelist your own IP(s):

iptables -I INPUT -s your_ip_address_here -j ACCEPT

Actually I am always getting hit with BS like this:

PlcmSpIp ssh:notty    92.61.39.162     Wed Feb 25 04:14 - 04:14  (00:00)
PlcmSpIp ssh:notty    92.61.39.162     Wed Feb 25 04:14 - 04:14  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)

So your suggestion will help with that. Not really related to the problems I've been experiencing, though.

I've actually just blocked large ranges through hosts.deny when I see things like this.

[Dan Druff]

Oh and I already am using something similar that I wrote myself involving iptables to prevent DDoSing.

Believe it or not, the real thing it's catching is accidental DOSing, where someone's system goes crazy and repeatedly requests the radio show archives, or something similar.

[4BET]

Word is seal team 6 is behind this, They are very good at hiding where the attack is coming from

[Daly]

Did you try and shut it off and restart it?

[hutmaster]

Did you try and shut it off and restart it?

Have you considered moving the servers to Antigua?

[abrown83]

Druff,

Does vBulletin use MyISAM or InnoDB for it's tables?

I would suggest rebuilding or repairing table by table to see if you can pinpoint which one is causing the issue.

Also ever thought about getting an account at someplace like LiquidWeb or something where they have fully Managed VPS hosting for a reasonable amount of money?

[gauchojake]

Equation Group malware imo

[sonatine]

Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.

[abrown83]

Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.

Tine is obviously right, but he only gave you half the SysAdmin info you needed not to fuck yourself.

Make sure you open the new port on your firewall prior to closing 22.

[Dan Druff]

Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.

The latter won't work for various reasons, including the fact that I go to secret locations and will want access.

I'm not so bothered about having it on 22 as long as I shut out repeated attempts from the same IP like above. There have not been any distributed hack attempts so I'm not worried.

[abrown83]

Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.

The latter won't work for various reasons, including the fact that I go to secret locations and will want access.

I'm not so bothered about having it on 22 as long as I shut out repeated attempts from the same IP like above. There have not been any distributed hack attempts so I'm not worried.

The element of risk is fairly small on a site like this.

If you wanted to do it still you could always just Remote Desktop or TeamViewer into a home computer and make changes or whitelist a new ip so you could work locally again.

[sonatine]

Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.

The latter won't work for various reasons, including the fact that I go to secret locations and will want access.

I'm not so bothered about having it on 22 as long as I shut out repeated attempts from the same IP like above. There have not been any distributed hack attempts so I'm not worried.

I hadnt considered the secret locations factor, touche.

And yeah if youre locking out tumblers, thats another huge plus.

But the trend right now is distributed cracking through botnets, which we always see after a huge password database leak. A few thousand hosts hammering port 22 is going to result in other processes stacking up in the queue and generally shite performance.

Food for thought.

Migrating off port 22 is a fairly standard play right now.

[abrown83]

Druff,

Is it mostly http requests or mysql load on the server?

You haven't been fucking around with the Apache MPM Worker settings have you?

Sometimes people think by allowing more requests that they will improve performance, but usually that's not the case at all, and it negatively impacts performance and sometimes significantly.