I noticed recently that none of my posts were appearing on TechMeme so I emailed Gabe Rivera, TechMeme’s founder to ask what the problem was. He responded:
Your Bad Behavior plugin is blocking me, even though my crawler behavior is rather benign.
If you can whitelist my crawler (does BB let you?), it looks like this:
Mozilla/5.0 (compatible; Wazzup1.0.XXXX; http://70.86.131.10/Wazzup) …with XXXX varying (long story…).Or just uninstall it! (What are some alternatives? I’d like to do a post on this…)
Bad Behaviour is an anti-spam plugin that I have written about previously.
As I don’t see a way to whitelist, I have disabled Bad Behaviour and I advise anyone else to do so until this can be sorted.
Thanks for the speedy response Gabe.
UPDATE: – Michael Hampton, Bad Behavior’s developer has contacted me to say it is possible to Whitelist TechMeme by adding its ip address (70.86.131.10) to the Whitelist-inc.php file – this fix didn’t work for me but may be worth a try if you do want to use Bad Behavior.
If you enjoyed this post, make sure you subscribe to my RSS feed!
What do you mean, no way to whitelist? That’s what the whitelist file is for.
As for Gabe, I’m deeply disturbed that he would actually tell you to “just uninstall” Bad Behavior — or that you would take such a bizarre suggestion seriously.
And as for his bot, it’s blocked for its shoddy internal coding, which is the same shoddy internal coding I see from just about everybody’s spambots. I’ve repeatedly informed Gabe of the one-line change necessary to allow the bot to pass, but apparently, he’s not made any such change.
When it comes down to a choice between Memeorandum and killing spammers, Gabe loses. Sorry.
The IP whitelist fix isn’t a great one because I can’t guarantee that I’ll always use the same IPs. But the ones I’m using now are 70.86.131.10, 70.86.46.66, and 82.165.180.34.
Michael, your comment about “repeated informing” me seems deliberately misleading. I was the one who contacted you, and we disagreed about whether asserting Mozilla compatible was legit or not. The fact that all major search engines use that (even Google in some cases) shows you’re holding me to a higher standard than the big guys. (I also tried your tip about sending “Accept: */*”, even though I don’t accept “*/*”. Thanks, but it didn’t help.)
All things considered, yes, I recommend people avoid the BB plugin. Sorry!
Michael and Gabe – thanks for your responses.
I tried adding the ip address to the whitelist but that didn’t appear to solve the problem so for now I have disabled Bad Behavior.
I have plenty of other spam defences in place and TechMeme is more important to me than the spam stopped by Bad Behavior.
All of the requests I have in my server log indicate that Gabe’s bot was served a 400 Bad Request HTTP response, but I can’t find anything in Bad Behavior’s logs here to correlate and discover exactly why it’s still being blocked.
Gabe, if you don’t accept */*, then why would there not be an Accept: line in your headers? The HTTP specification is clear on this: If an Accept: header is not present, the server assumes that anything (i.e. */*) is acceptable.
Assuming you are sending this header, then you are being blocked for some other reason. When someone (Tom?) sends me bad_behavior logs to look at, perhaps I can figure it out.
Techmeme started aggregating my blog LiveCrunch.com but then just stopped after I sent email to feedback for saying “thanks” , so to be honest I am kind of disapointed. I also contacted Gabe (Above) no respond. I mean after all TM is not only aggregator I am/was on …it’s just makes me sick to my stomach that people follow Michael A. orders,like they are his dogs.. Sit/LayDown/Sit…if u know what I mean