Robots.txt

[arron]

Does it keep the Robotrons away? I thought that Eugene was the last best hope for humanity..at least until about level 11 when I usually get killed.

[piku]

Does it keep the Robotrons away? I thought that Eugene was the last best hope for humanity..at least until about level 11 when I usually get killed.

I'm dead on the first brain wave. It's a cruel and unforgiving game, just like finding posts with the search function

No, not had my whingey pills yet, but I think my blood sugar is low.

[Istar]

After our flood by google and yahoo indexers I've added a robots.txt that prevents any indexing by any search engine that obeys the convention of its usage

Code: Select all

User-agent: *
Disallow: /

Fixed that for you.

[Dio]

It was more sarcasm than whinge.

Serious question; what's wrong with it? Always works for me...

The two obvious ones I've observed are: it appears to list everything that contains the word with no attempt at ranking, so if you search for a remotely common word you get an infinite number of irrelevant matches; and it only takes you to the thread, so it doesn't appear to be any use finding a particular topic in the Hitting Refresh thread for example.

I've only used it about five times mind though, which is why I was being sarcastic, it's not really a major loss from my life .

[tufty]

It's got a "display as posts" option, which will take you to a particular post.

On the other hand, it's a sack of useless shite with regard to actually searching.

And yes, I have taken my whinge pills.

[ElfQrin]

After our flood by google and yahoo indexers I've added a robots.txt that prevents any indexing by any search engine

Code: Select all

User-agent: *
Disallow: /

I'd rather recommend:

Code: Select all

User-Agent: *
Crawl-delay: 120

The delay is in seconds.

[Stone]

Is that delay per page-request?

Stone

[ElfQrin]

It's the time in seconds between page requests.

[gravy]

It's the time in seconds between page requests.

Crawl-delay isn't recognised by Google which was one of the bots causing the trouble.

[ElfQrin]

Crawl-delay isn't recognised by Google which was one of the bots causing the trouble.

You can disallow GoogleBot only, then.

[gravy]

Yeah I know I'm sticking with this for now as we also had some problems from yahoo.

The problem is that people seem to be using search engines crawlers to trigger DOS attacks and that can come from any search engine. So for now it's going to be left that way.

We're looking and another solution we'll be able to trial soon

[chiz]

Can you not just make it so only registered logged-in peeps can view the threads? That would stop the bots going through all the pages.

[skweek]

Can you not just make it so only registered logged-in peeps can view the threads? That would stop the bots going through all the pages.

That would be a bad idea, a lot of people read threads not logged in and there there are a fair amount of lurkers who don't even have an account I think.

[gravy]

Can you not just make it so only registered logged-in peeps can view the threads? That would stop the bots going through all the pages.

That would be a bad idea, a lot of people read threads not logged in and there there are a fair amount of lurkers who don't even have an account I think.

Yep

And the robots.txt file I posted a page ago stops the bots we had a problem with.

There's not much more discussion to be had about this really

And it looks like we'll be giving PHPBB3 a go v soon anyway

[Sir_LANs-a-lot]

woohoo!