Host almost nuked our database

First, the view count started to look a bit higher than normal. Turned out sessions were no longer being considered by the algorithm. Tracked it down to a "table full" mysql error. Contacted the host, a company claiming to give 24/7 support. Turns out that's level 1 support - for any real issue you have to go to level 2 and beyond and you do NOT get phone support.

They've been playing with it for a few hours and not reporting any progress by mail. After a while more tables became "full" (we weren't able to post any blog entries tonight) and eventually the guy managed to first disallow remote access, then disallow even control panel access to the database, eventually even the site could no longer pull the information!

Then again the site was reading data but writing was still impossible. Fortunately by that time we had uploaded a database backup script and saved the database to a text file, which we downloaded.

We just created a new database on the host and reuploaded the downloaded data, then switched the database that the site points to. At this point everything is working again but it was very close.

The upload process didn't work so well and some tables weren't saved, but fortunately the only missing one that is needed is our notify list table which by total coincidence we had backed up earlier today! Many rows were corrupted in the old forums due to the total junk the spammers had tried to inject in the old forums; so the information was there but we had to hand stitch a 20 MB sql dump in order to get the database to accept the upload data.

We will soon need to switch to a more expensive hosting company with decent support!

If you have registered but can't access the forums or any other strange stuff happens, please let us know. Also if you have signed up for the notify list in the last 24 hours it may be worth doing that again, the system doesn't store duplicate information so it won't hurt. We were testing this throughout this incident and there were at most 5 mins of downtime (because this particular table didn't become unwritable) but just to be on the safe side.

Comments