Wordpress wp-cron.php and Bad Behavior Apache crash
March 17th 2007
Update: This is old info, and not correct anymore. The problem was a result of a bug in Apache2 (or Ubuntu’s version of it) in which apache processes swapped out to disk would not release, and would consume all memory and crash the server. After an ‘apt-get upgrade’ on 8/30/07, apache2 was upgraded to ‘2.0.55-4ubuntu2′ and the problem has disappeared. For users experiencing this issue, I recommend upgrading your version of apache to the latest.
I have just discovered the reason for the sporadic server crashes I’ve been seeing over the past month or so. The symptoms include: Server slowing to a crawl, Apache reaching max_clients and never releasing its connections, Apache processes getting “stuck” and never going away, and finally, all open requests are for wp-cron.php?check=[somelongnumber] (which you may find out through apache-status, which looks like this).
wp-cron.php is WordPress’ version of a process scheduler — a function runs selectively on a request every so often and checks to see if any scheduled things need to happen. To do them, it doesn’t just run a command — it opens an HTTP request to your local server (acting as a client, essentially) with a secret key made from a hash of your database password (”?check=”) for wp-cron.php, which runs as a new Apache and PHP process so it doesn’t interrupt or slow down the existing request. It’s a good way to “fork” a process from PHP.
Except that it’s kind of bad, in that if wp-cron.php in any way can’t be accessed, then the client is redirected to the 403 or 404 page, which are Wordpress pages, which run the whole wp-cron.php request all over again, and basically result in a good setup for an infinite loop. I’m not sure of the details of how it works, but I do know that it does happen, because my server has been brought to its knees several times as a result of this.
There are two ways a request to wp-cron.php could result in an error document– one, if it doesn’t exist, as was addressed by this wordpress patch (which I have tried and did NOT resolve the next problem), and two, if it gets redirected to a 403 Forbidden page by a plugin trying to fight spam, which has unwittingly blacklisted your own local server’s IP address for requests to wp-cron.php specifically. The plugin is called Bad Behavior, and this is some bad behavior on its part.
Here’s an easy way to reproduce this, if you have the right conditions as detailed above:
- Open an edit page for a post you’ve already written.
- Save it.
- Edit it again, and save it.
- Edit it one more time, and save it.
- Look at your server-status, if you can still load it…
On my server, that sequence resulted in Apache max_clients being hit and the server screeching to a halt, with Apache processes no longer fitting in memory and no new requests able to be served. Great. The problem was only fixed after a restart of Apache.
The solution: disable Bad Behavior until this can be resolved, and local requests to wp-cron.php kept out of its blacklist. I have already notified the developer of Bad Behavior of this problem, and I think he should be able to fix it without too much trouble.
I also consider this a WordPress bug — in no case, whether wp-cron.php fails or gets redirected or what — should any standard wordpress usage cause Apache to crash and burn so badly. I’ll be submitting this as a bug to WordPress as well and we’ll see what they say about it. Until then I hope I’ve helped someone at least, as this took me over a month to find out what was wrong, and I found no (zero) good resources on this kind of crash or any Apache crash of this variety anywhere in the depths of Google.
I too have some slowness in one projetc using Wordpress Mu and this that you pointed out make me start to look into the logs when users start to claim again.
Thanks for sharing
I bow down to trisweb.com for this post.
((bowing down))
Thanks for your posts to the SH forum about this and blogging about it. I’ve been having mad problems lately so my fingers are crossed that this will be the fix.
Hopefully that’s the problem you’re seeing, but like I said, it could be any number of things. Enable apache-status and check out what’s holding the processes when you see a crash, that’s the likely culprit.
This happened to me again the other day, even with this fix in place. It seems that wp-cron.php can overload the server to the point where apache exhibits this bug, but that other things, like natural server load, can make the same crash happen.
I’m going to say this has more to do with Apache, Xen virtualization, and swapping memory than anything else. So, if at all possible, keep your apache processes off the disk. It seems like they will not recover when brought back from virtual memory. That’s my best guess.
Yep, I’m using bad behavior and my bad behavior table is getting filled up with entries like this:
/resources/blog/wp-cron.php?check=131accfed8363d4c…
from my own server address. Argh.
Oh wait - you can just add you ip to the whitelist.php in the bad-behavior folder. Cool.
[...] This works well, but I did run into one problem wp-cron.php and bad behaviour not getting along. [...]
Added it to the TextDrive forum.
Thanks, but I think this may not be as bad a problem as I originally thought, and it turns out my Apache crash is still happening. So don’t go blaming it too harshly, even though I still think this is a bug.
Problem solved: this seems to have been an apache2 bug under Ubuntu… upgrading to the most recent version has stopped the problem completely (as of 8/30/2007).
High load now acts normally; processes quit after connections are dropped, even if swap space is used.
The working version is 2.0.55-4ubuntu2, previous version unknown.
This makes sense to me, as the original problem started after an ‘apt-get upgrade’ anyway, and did not exist before. Whatever it was I’m glad they finally fixed it.
hi is it ok for now to rename the wp-cron file to different name so it doesnt get accessed , that way for now i stop this issue , my hosting people are compalining abt this frm my files.
If you are referring to this bug, then you can disable wp-cron and issue a real cron job in two steps, here, and here.
Hi Tristan,
Thanks for this write up. I relize it’s a bit old but I’ve recently installed WP MU and am seeing this wp-cron.php?check=(big number) and I’ve read over your thread, checked out links like from Jonathan, etc. and can’t seem to solve this problem.
I’m not experienced enough with server stuff and all that. Is this purely an issue with Apache to you reckon?
You mentioned, “Problem solved: this seems to have been an apache2 bug under Ubuntu…”
Should I ask my host if my Apache version is at least 2.0.55-4ubuntu2?
Any other suggetions? Btw, I’m not using “Bad Behavior”, unless that’s the same thing as Akismet which I am using.
Thanks in advance
I’m having the same problem since about two weeks ago. Haven’t been able to pinpoint the problem yet, the cause doesn’t seem to be a plugin. (I deactivated the two that use wp-cron, activating one of them crashes the server, no matter which. It’s highly unlikely, that one of the plugins caused the problem.