Get exclusive CAP network offers from top brands

View CAP Offers

Question about Robots.txt file & "link no follow"

casinorecommender asked 2 years ago
I am currently using Robot.txt file with (User-agent: *
Allow: /) and also for the outgoing links i use “link nofollow”.

The question is : Is using Robots.txt file with this type of code disturbs the functionality of “link no follow” codes for outgoing links ?

Thnx

8 Answers
itsblitz answered 2 years ago
dont know if im missing the question here but your robots file doesnt disallow anything and its would have no effect on outgoing links, the robots.txt just tells(suggests) bots what to crawl and what not too, the rel=”nofollow” just suggests that bots dont follow the link out of your site, they will if they want to though

Warren answered 2 years ago
You should remove the “Allow: /” from your robots.txt file as it’s not necessary and even though you nofollow links, the SE spiders can still access them through toolbars, analytics, other links, etc.

You can “Disallow:” certain sections of your site but again that is only a recommendation to the search engines, not a guarantee.

Hope this helps.

casinorecommender answered 2 years ago
Google webmaster tools recommends and suggests a robots txt file. That is why i added one. So do you say that robots txt file and nofollow links conflict each other ? Thanks

@Warren 230002 wrote:

You should remove the “Allow: /” from your robots.txt file as it’s not necessary and even though you nofollow links, the SE spiders can still access them through toolbars, analytics, other links, etc.

You can “Disallow:” certain sections of your site but again that is only a recommendation to the search engines, not a guarantee.

Hope this helps.

casinorecommender answered 2 years ago
Do you suggest that robots txt file and nofollow links conflict ?

Thanks
@itsblitz 230001 wrote:

dont know if im missing the question here but your robots file doesnt disallow anything and its would have no effect on outgoing links, the robots.txt just tells(suggests) bots what to crawl and what not too, the rel=”nofollow” just suggests that bots dont follow the link out of your site, they will if they want to though

Warren answered 2 years ago
@casinorecommender 230004 wrote:

Google webmaster tools recommends and suggests a robots txt file. That is why i added one. So do you say that robots txt file and nofollow links conflict each other ? Thanks

You should definitely have a robots.txt file. I am just telling you that “Allow: /” specifically is not needed within the robots.txt file as it’s redundant. The nofollow on a link will work in conjunction with your robots.txt file.

32x answered 2 years ago
Sorry for the double post, but I couldn’t find a way to delete it..

32x answered 2 years ago
@casinorecommender 230000 wrote:

I am currently using Robot.txt file with (User-agent: *
Allow: /) and also for the outgoing links i use “link nofollow”.

The question is : Is using Robots.txt file with this type of code disturbs the functionality of “link no follow” codes for outgoing links ?

Thnx

You need to edit your HTACCESS file on the root of your web directory. This is also the preferred Google and Yahoo method. robot.txt file is not very accurate. HTACCESS is the gate keeper the the Server, you might say. You can find information on the internet about it. I tried to post some code here for you, but this website will not allow me to put code..

I know this is not exactly what you were talking about, but your HTACCESS file is also the most accurate way to prevent search engines and other sources from archiving your pages.

Happy programming..

casinorecommender answered 2 years ago
Fixed the issue. Many thanks for your contribution.

@Warren 230006 wrote:

You should definitely have a robots.txt file. I am just telling you that “Allow: /” specifically is not needed within the robots.txt file as it’s redundant. The nofollow on a link will work in conjunction with your robots.txt file.