Get exclusive CAP network offers from top brands

View CAP Offers

xml sitemap / robots meta tags

cat asked 1 year ago
Now that the new google site map is installed I am wondering if the metatags robots index,follow and revisit after still have any use.

Somebody even told me that the meta tags was always useless except of the decription and keyword tag and I should get rid of them.

I wonder if other search engines will crawl our entire sites without that tags.

How do you handle that?
Do you have that robots tags on your sites?

3 Answers
kwblue answered 1 year ago
From my limited knowledge, you do not need to put anything in there for the robots UNLESS you want to disallow them. The bots will typically follow your site through the links without any help from you.

Hope that helps.

kw

Coke answered 1 year ago
I have also read that it is not necessary to include them unless you don’t want the bots to follow the links or index the pages.

But then, it can’t hurt to have the tags, can it? :eh:

So all my pages still have robot tags.

kwblue answered 1 year ago
Nope, won’t hurt to have index, foll in. Only stops bots if you use disallow and that is only if you know the name of the bot AND if the bot follows the rules.