Categorized | News

Get your Personal Copy of PHPkitchen ;-)

Posted on 04 April 2003 by Demian Turner

Thanks goes out to the person from Paris who yesterday pointed his/her screenscraping software (which will remain unnamed to avoid promoting further infamy) at and drained

78 MEGABYTES !!!!!!!!!!!!

off my server. Yes, your IP,, has been banned and you have been reported to your ISP and hopefully blacklisted.

It looks like the said software now has an option to override the disallow directive in robots.txt – any other webmasters know how to get around these types of nuisances?

Happy that I upgraded to a 40GB account,


Bookmark and Share

5 Comments For This Post

  1. daynah Says:

    Ick. =\\ *adds another ip to her ban list*


  2. Anonymous Says:

    seriously, the disallow in robots.txt *usually* works a lot better because even if you ban a bot, you can end up sending out 6 hours worth of 403\’s which is still bandwidth.

    but what can you do when the bots don\’t respect disallows? firewall mods in my colo are not an option, I would be greatly pleased if anyone has any advice on this 🙂

  3. jhherren Says:

    The User-Agent header is trivial to spoof. Use a .htaccess file that denys by IP address instead:

    <limit GET POST>
    order allow,deny
    allow from all
    deny from <address>

  4. demian Says:

    jhherren, thanks for the suggestion. What I was looking for was a generic way to disable certain nuisance bots.

    Banning by IP is fine but usually you only discovered you\’ve had your site siphoned *after* the fact, so it\’s not much use.

    The bot I\’m talking about, again I think it\’s best not to mention the name here, is a crappy little 2 MB program that any windows (always win98 in my experience) user can download for free and fire up.

  5. jhherren Says:

    Feel free to email me with the specifics, and I\’ll be glad to TRY to help out 🙂

Leave a Reply



Demian Turner's currently-reading book recommendations, reviews, favorite quotes, book clubs, book trivia, book lists



PHPkitchen recommends you also check out the following sites :

Accounting for Small Businesses

FreeAgent sign-up