Robots.txt

Inane baanter or words of wisdom - you decide...
Post Reply
User avatar
gumboot
Baaaaaaa!
Baaaaaaa!
Posts: 7652
Joined: Sun Aug 26, 2007 3:42 am 0
Location: trumpstain!

Re: Robots.txt

Post by gumboot »

RiK wrote:
disciple wrote:With the "attracting the wrong sort" thing you could control this with good use of robots.txt, enable it for all forum sections EXCEPT bleatings, that way the only stuff that gets indexed is the gaming, musician, programming and similar stuff, i.e. the core Yakyak content.
It's not really as simple as that as the forum sections aren't a simple directory structure, they're stored in a database.
Still, the interface pretends to have a hierarchy, and isn't it meant to be good form to present them to the world as if they were a directory structure? Like how it's becoming popular for news and blog sites to do things like:

Code: Select all

http://www.randomsite.com/stuff/655321/article-title-but-it-doesnt-mean-anything-and-you-can-put-whatever-you-want-here/index.html
Would I look like an utter wanker if I suggested something like this?

Code: Select all

http://www.yakyak.org/boards/Llamasoft:TNG/Bleatings/65458/Robots.txt/page2.html?stuff=1&nonsense=2
User avatar
disciple
Sheepie Whore
Sheepie Whore
Posts: 13941
Joined: Mon Oct 27, 2003 11:33 pm 0
XBL Gamertag: LiberMentis
PSN: Prime_Heretic
Location: Greenfield
Contact:

Re: Robots.txt

Post by disciple »

How would you do this?

You could mod_rewrite any old URL scheme to the new one, internal user posted links you (optionally) wouldn't have to worry about as they'd sort themselves out over time. But you'd need to update all of PHPBBs internal links to the new scheme which presumably would be right pain since you'd have to start changing the codebase.
PS4 : Prime_Heretic
Steam: bitstorm
Elite Dangerous: Cmdr Britain
User avatar
gumboot
Baaaaaaa!
Baaaaaaa!
Posts: 7652
Joined: Sun Aug 26, 2007 3:42 am 0
Location: trumpstain!

Re: Robots.txt

Post by gumboot »

disciple wrote:How would you do this?

You could mod_rewrite any old URL scheme to the new one, internal user posted links you (optionally) wouldn't have to worry about as they'd sort themselves out over time. But you'd need to update all of PHPBBs internal links to the new scheme which presumably would be right pain.
Fix the software, maybe?

I mean; fix the software rather than wrapping it in another layer of abstraction... which, now that I think about it, is entirely at odds with modern software development practices, so I'm probably being more sarcastic than I mean to be.
User avatar
disciple
Sheepie Whore
Sheepie Whore
Posts: 13941
Joined: Mon Oct 27, 2003 11:33 pm 0
XBL Gamertag: LiberMentis
PSN: Prime_Heretic
Location: Greenfield
Contact:

Re: Robots.txt

Post by disciple »

Heh yeah, updates would be a bitch though. Maybe we should build a bespoke Yakyak eh. =)
I mean; fix the software rather than wrapping it in another layer of abstraction... which, now that I think about it, is entirely at odds with modern software development practices
Well it's only one layer, besides it's a better route than making alterations to proprietary software. Saying that I'm not sure what benefit doing it would offer, obviously we don't do SEO here and people rarely manipulate URLs manually.
PS4 : Prime_Heretic
Steam: bitstorm
Elite Dangerous: Cmdr Britain
User avatar
gumboot
Baaaaaaa!
Baaaaaaa!
Posts: 7652
Joined: Sun Aug 26, 2007 3:42 am 0
Location: trumpstain!

Re: Robots.txt

Post by gumboot »

disciple wrote:Saying that I'm not sure what benefit doing it would offer, obviously we don't do SEO here and people rarely manipulate URLs manually.
It appeals to my OCD sense of order. Could there be a better reason than that?
User avatar
disciple
Sheepie Whore
Sheepie Whore
Posts: 13941
Joined: Mon Oct 27, 2003 11:33 pm 0
XBL Gamertag: LiberMentis
PSN: Prime_Heretic
Location: Greenfield
Contact:

Re: Robots.txt

Post by disciple »

A proper URL structure would be nice. It'd be cool to start adding other stuff too, flash content other than youTube would be on my list. =) Also maybe write a search v2 to run alongside the original.

We could do a dynamic sitemap as well, it wouldn't be hard to write and would help Google & co do their thing. We'd have enough people for the league in no time. ;-)
PS4 : Prime_Heretic
Steam: bitstorm
Elite Dangerous: Cmdr Britain
User avatar
gumboot
Baaaaaaa!
Baaaaaaa!
Posts: 7652
Joined: Sun Aug 26, 2007 3:42 am 0
Location: trumpstain!

Re: Robots.txt

Post by gumboot »

Well if we're reinventing the site, then I'd like to write in mediawiki markup rather than BBCode.

Notably, I'd like generic interwiki links so I don't have to do all that "url=en.wikipedia.org/bla/bla" crap.
User avatar
disciple
Sheepie Whore
Sheepie Whore
Posts: 13941
Joined: Mon Oct 27, 2003 11:33 pm 0
XBL Gamertag: LiberMentis
PSN: Prime_Heretic
Location: Greenfield
Contact:

Re: Robots.txt

Post by disciple »

Interesting, not heard of those before, it's almost like how the YouTube tags work currently but instead you'd have a generic tag taking two attributes. Not up on PHPBB but it can't be that hard to write a plugin for can it? Like maybe [link 'x:y'] or something.
PS4 : Prime_Heretic
Steam: bitstorm
Elite Dangerous: Cmdr Britain
User avatar
ElfQrin
Baaaaaa!
Baaaaaa!
Posts: 4384
Joined: Mon Jul 28, 2003 12:55 pm 0
Location: Italy
Contact:

Re: Robots.txt

Post by ElfQrin »

Eternal September is not caused by search engines, after all.
User avatar
gumboot
Baaaaaaa!
Baaaaaaa!
Posts: 7652
Joined: Sun Aug 26, 2007 3:42 am 0
Location: trumpstain!

Re: Robots.txt

Post by gumboot »

ElfQrin wrote:Eternal September is not caused by search engines, after all.
Dear god you're not going to start on about the "good old days" of web-based message boards, are you?
User avatar
disciple
Sheepie Whore
Sheepie Whore
Posts: 13941
Joined: Mon Oct 27, 2003 11:33 pm 0
XBL Gamertag: LiberMentis
PSN: Prime_Heretic
Location: Greenfield
Contact:

Re: Robots.txt

Post by disciple »

Bump. :)

Any Admins have any thoughts on this search engine stuff then? Do I need to pm someone and ask?
PS4 : Prime_Heretic
Steam: bitstorm
Elite Dangerous: Cmdr Britain
User avatar
gravy
Sheepie Whore
Sheepie Whore
Posts: 19572
Joined: Wed May 01, 2002 1:28 pm 0
Location: The Other Place

Re: Robots.txt

Post by gravy »

Indexing has been back on now for 3 days and is being monitored.

The main worry is that the forum having a bigger presence on search engines may mean we get a lot more spammers. If that happens I'll turn it back off. That hasn't been the case so far.
User avatar
disciple
Sheepie Whore
Sheepie Whore
Posts: 13941
Joined: Mon Oct 27, 2003 11:33 pm 0
XBL Gamertag: LiberMentis
PSN: Prime_Heretic
Location: Greenfield
Contact:

Re: Robots.txt

Post by disciple »

gravy wrote:Indexing has been back on now for 3 days and is being monitored.
Are you sure? I checked the other day before posting and again just now, I'm getting :

Code: Select all

User-agent: *
Disallow: /
PS4 : Prime_Heretic
Steam: bitstorm
Elite Dangerous: Cmdr Britain
User avatar
gravy
Sheepie Whore
Sheepie Whore
Posts: 19572
Joined: Wed May 01, 2002 1:28 pm 0
Location: The Other Place

Re: Robots.txt

Post by gravy »

disciple wrote:
gravy wrote:Indexing has been back on now for 3 days and is being monitored.
Are you sure? I checked the other day before posting and again just now, I'm getting :

Code: Select all

User-agent: *
Disallow: /
That's really odd.

I moved robots.txt -> robots.txt.old BUT apache now serves both robots.txt AND robots.txt.old

If I move it out of the directory complete it serves neither

Good spot :)
User avatar
Flabio
Baaaaaaa!
Baaaaaaa!
Posts: 8717
Joined: Wed Mar 03, 2004 12:37 pm 0
Location: Guildford

Re: Robots.txt

Post by Flabio »

And here comes the spam... hehe.
Post Reply