Robots.txt

[disciple]

Is it worth dumping the robots.txt temporarily and testing the waters? It's been 2 years, if there was an issue it's likely Google may have sorted out whatever it was.

[Orangy Tang]

Is it worth dumping the robots.txt temporarily and testing the waters? It's been 2 years, if there was an issue it's likely Google may have sorted out whatever it was.

I think the real question is "how did you find this thread"?

[disciple]

The search function's ok, well for my purposes at least, used to find the >3 character requirement annoying mind.

I'm just thinking about the Yakyak membership. No idea what the actual stats are and maybe I'm looking back with rose-tinted specs but it seems like Yakyak gets nowhere near as much traffic as it did, yeah it's subjective but it does feel like it's taken a significant dip over the last year and a bit.

Currently the only way people find this site (and then join up and become a new members) is through the link on the main Llamasoft site... and that's about it - and that will only have any impact when Giles and/or Yak release something new. The other main channel for users coming to the site would of course be via search engines which is no longer available, the result of which seems to be an active membership whose numbers are slowly decreasing. I remember there were times when I'd put a gaming related search term into Google and get a Yakyak thread appear in the top three results, very cool, absolutely invaluable for increasing/sustaining membership. In fact you could probably control Yakyak's active membership size and stop it getting too big by subtle use of robots.txt.

Of course I'm willing to admit rose-tinted specs may be playing a role here, it just seems kinda quiet round here nowadays.

[gumboot]

After our flood by google and yahoo indexers I've added a robots.txt that prevents any indexing by any search engine that obeys the convention of its usage

Code: Select all

User-agent: *
Disallow: /

Fixed that for you.

If you outlaw indexing, only outlaws will index... or something like that.

[disciple]

The search function's ok, well for my purposes at least, used to find the >3 character requirement annoying mind.

I'm just thinking about the Yakyak membership. No idea what the actual stats are and maybe I'm looking back with rose-tinted specs but it seems like Yakyak gets nowhere near as much traffic as it did, yeah it's subjective but it does feel like it's taken a significant dip over the last year and a bit.

Currently the only way people find this site (and then join up and become a new members) is through the link on the main Llamasoft site... and that's about it - and that will only have any impact when Giles and/or Yak release something new. The other main channel for users coming to the site would of course be via search engines which is no longer available, the result of which seems to be an active membership whose numbers are slowly decreasing. I remember there were times when I'd put a gaming related search term into Google and get a Yakyak thread appear in the top three results, very cool, absolutely invaluable for increasing/sustaining membership. In fact you could probably control Yakyak's active membership size and stop it getting too big by subtle use of robots.txt.

Of course I'm willing to admit rose-tinted specs may be playing a role here, it just seems kinda quiet round here nowadays.

Anyone any thoughts on this then?

My perception is that activity on Yakyak has been slowly decreasing for a good while now, activity on here seems low in comparison to what it was. We barely have the numbers to do the league, the photo compo gets very few entries and without being indexed in Google I don't really see this improving.

[ElfQrin]

Another vote to reopen YakYak to spiders.

[Darkflame]

yeah, try for a bit at least.

[Devlin]

Don't!

It'll be like Eternal September all over again!

[Firefox]

Anyone any thoughts on this then?

My perception is that activity on Yakyak has been slowly decreasing for a good while now, activity on here seems low in comparison to what it was. We barely have the numbers to do the league, the photo compo gets very few entries and without being indexed in Google I don't really see this improving.

Yes, it is very quiet compared to a year or two back.

[ElfQrin]

Don't!

It'll be like Eternal September all over again!

I don't think so, since this forum mainly attracts geeks and retrogamers.
We're rather becoming the forgotten brigade.

[disciple]

Don't!

It'll be like Eternal September all over again!

I don't think so, since this forum mainly attracts geeks and retrogamers.

Plus it'd take a while before anything started appearing in search engines, I don't think it's like a switch or anything similar to the above September analogy.

With the "attracting the wrong sort" thing you could control this with good use of robots.txt, enable it for all forum sections EXCEPT bleatings, that way the only stuff that gets indexed is the gaming, musician, programming and similar stuff, i.e. the core Yakyak content. Off topic wouldn't be indexed which would help focus the demographics which I don't think would be a bad thing (although it then may take longer to get up the rankings what with the bots seeing reduced traffic)

[RiK]

With the "attracting the wrong sort" thing you could control this with good use of robots.txt, enable it for all forum sections EXCEPT bleatings, that way the only stuff that gets indexed is the gaming, musician, programming and similar stuff, i.e. the core Yakyak content.

It's not really as simple as that as the forum sections aren't a simple directory structure, they're stored in a database.

[disciple]

Not really up on robots.txt but wouldn't something like : /*f=2*$ do it? I think Google does regexes. Though that would probably take out blog/league/programming and unity too.

Alternatively it could probably be done via a rewrite rule using useragent + url, 404 bleatings for Googlebot (and others). Maybe something like :

Code: Select all

RewriteCond	%{HTTP_USER_AGENT} ^.*Googlebot.*$
RewriteRule	^(\?|\&)f=2(\&|$)  /404.html

Though again am not up on Apache.

http://www.addedbytes.com/cheat-sheets/ ... eat-sheet/

[Arthur Ringworn]

I don't think so, since this forum mainly attracts geeks and retrogamers.

And talking about games has been relegated to it's own forum.

[piku]

I think they've fixed the Internet now, my site gets continually arse-raped by spiders and apart from filling my logs with rubbish and skewing my visitor stats it doesn't seem to break down. I also get a lot of visitors coming from Google search results too.

phpBB's search thing is an artform, it can take years to master, but once mastered a skilled searcher can find a target post in less than half an hour. It's cutting edge 1997 search technology at work.