Comments on: Use Robots.txt To Prevent vBulletin Duplicate Content

By: Audax

Audax — Wed, 18 Feb 2009 11:17:27 +0000

Why disallow the faq?
Because there are thousands of vbulletins out there and most of the operators leave the faq as they are by default.
So do I.
That’s a lot of text which is exactly on thousands other sites = dupl. content.

Why don’t prevent that?
Of course if you modified the faq you should allow bot access.
That’s what I meant!

By: Todd

Todd — Fri, 06 Feb 2009 18:31:20 +0000

Also: Tink about this, even if you could stop scammer from crawling your website, you would also stop any search engine from indexing your website. so the above comment is a joke:

Disallow: /faq.php

What would that do.. ok so you would keep your .Faq all to yourself and not share it with google and the rest of the search engines ???

Oh Yea that’s good advice!!

By: Todd

Todd — Fri, 06 Feb 2009 18:28:04 +0000

Scan Robots or spiders dont follow the robots.txt this is a optional thing.. and it is very sasy to change your identity. I can rreawl any website with ease and make it believe i am a google robot. I can post instructions on this here if you need proof but just search google and you will find tons of answers. so the above will not stop crawling of your website..

general.useragent.override Googlebot/2.1 (+http://www.googlebot.com/bot.html)

This can be done using firefox also.

Just search ” changing user agent in firefox”

By: Audax

Audax — Fri, 16 Jan 2009 17:14:39 +0000

I think this is also important

Disallow: /faq.php

Why generate duplicate content as thousand other sites with boring standard faq.

By: kma

kma — Sat, 15 Nov 2008 09:38:08 +0000

It is very helpfull
thanks