|
View previous topic
::
View next topic
|
| Author |
Message |
rossen
Joined:
06 Jun 2008
Posts:
4
|
Posted: Fri Jun 06, 2008 1:12 pm
Post subject: Robot.txt
|
 |
|
Hi,
is it necessary to create a robot.txt???? if yes can u tell me wht content should be there in robot.txt????
[Links deleted. They belong in your signature file. You can click on Profile and create one. Admin.] |
|
| Back to top |
  |
 |
InternetMarketing_IQ
Joined:
13 Aug 2006
Posts:
1138
|
Posted: Fri Jun 06, 2008 5:42 pm
Post subject:
|
|
|
The simple answer is NO.
If you don't have a specific reason to use the file then don't worry about it. And just because you have the file does not mean the Search Engines will follow your requests.
Not here to plug another Forum but "highrankings.com" has several good posts on the subject.
_________________
|
|
| Back to top |
 |
|
rossen
Joined:
06 Jun 2008
Posts:
4
|
Posted: Wed Jun 11, 2008 4:25 pm
Post subject:
|
|
|
Thanks, For your reply,
But my client want some that crawler doesnt Crawl one some page. Please can u tell me the formate to creare Robot.txt
_________________
Girl Lay outs vegan diet |
|
| Back to top |
|
|
kennichi
Joined:
09 Jun 2008
Posts:
190
|
Posted: Tue Sep 02, 2008 8:43 am
Post subject:
|
|
|
To exclude the spiders from any part of your site, having a robots.txt file is still a good idea.
There are a number of situations where you may wish to exclude spiders from some or all of your site.
You are still building the site, or certain pages, and do not want the unfinished work to appear in search engines
If you are using doorway pages you may wish to ensure that individual robots do not have access to all of them. This is important in order to avoid being penalized for spamming a search engine with a series of overly similar pages.
_________________
For more info
http://www.putonyourgoggles.com/blog/google-goggles/google-goggles-are-on-the-loose/ |
|
| Back to top |
|
|
kennichi
Joined:
09 Jun 2008
Posts:
190
|
|
| Back to top |
|
|
kennichi
Joined:
09 Jun 2008
Posts:
190
|
|
| Back to top |
|
|
crm911
Joined:
30 Jan 2005
Posts:
34
Location:
Australia
|
Posted: Sat Feb 21, 2009 10:52 am
Post subject:
|
|
|
robotstxt.org has the answers you seek.
I advise everyone to have such a file, even if it is empty, because not having one adds a 404 entry to the error log every time a spider asks for one.
_________________
--
Work: SEO and PPC training
Blog: Net Magellan |
|
| Back to top |
|
|
mercantile
Joined:
02 Jun 2009
Posts:
7
|
Posted: Tue Jun 02, 2009 11:28 am
Post subject: Re: Robot.txt
|
|
|
| rossen wrote: | Hi,
is it necessary to create a robot.txt???? if yes can u tell me wht content should be there in robot.txt????
[Links deleted. They belong in your signature file. You can click on Profile and create one. Admin.] |
Just if you needs to protect your website from some bots.
I usually ban alexa bot, because it publish private or commercial data about the website, such as amount of visitors etc.
_________________
My favorite affiliate program:
http://www.ubsol.com |
|
| Back to top |
|
|
hostplan4you
Joined:
11 Jun 2009
Posts:
1
|
Posted: Thu Jun 11, 2009 2:39 am
Post subject:
|
|
|
| crm911 wrote: | robotstxt.org has the answers you seek.
I advise everyone to have such a file, even if it is empty, because not having one adds a 404 entry to the error log every time a spider asks for one. |
Yes you're right, this is the sample of content of robots.txt
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/
_________________
low cost web hosting
http://www.hostplan4you.com/ |
|
| Back to top |
 |
|
Jenie0109
Joined:
10 Aug 2009
Posts:
69
|
Posted: Fri Aug 14, 2009 5:47 am
Post subject:
|
|
|
It is important from search engine optimization point of view. By specifying where search engines should look for content in directories or files you can increase the search engine ranking of your site, and is recommended by Google and all the search engines.
_________________
hydraulic valve || seo packages |
|
| Back to top |
|
|
BullSpring
Joined:
14 Aug 2009
Posts:
12
|
Posted: Sun Aug 16, 2009 11:51 pm
Post subject:
|
|
|
This is really interesting and opened some debate. I agree with the benefit having a file can bring even if its empty.
Be good to hear some more info???
_________________
Copy this easy system and make $10,000+ a week with Profit Miracle? |
|
| Back to top |
|
|
joeparera
Joined:
06 Jan 2011
Posts:
3
|
Posted: Thu Jan 06, 2011 4:11 am
Post subject:
|
|
|
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.
_________________
Tax Settlement Affiliate |
|
| Back to top |
|
|
jimkerinmar
Joined:
11 Jan 2011
Posts:
4
|
Posted: Tue Jan 11, 2011 4:39 am
Post subject:
|
|
|
Robots.txt is used by search engines to identify the direction, they must crawl and they should not. If you want your website to be crawled by the search engine and other robots you will need it, otherwise forget for now.
_________________
Study in China |
|
| Back to top |
|
|
offersking
Joined:
22 Sep 2010
Posts:
18
|
|
| Back to top |
|
|
jemswillam
Joined:
21 Feb 2011
Posts:
3
|
Posted: Mon Feb 21, 2011 12:42 pm
Post subject:
|
|
|
Robots.txt is used by search engines identify the URL that should they crawl and they should not. If your entire website to be crawled by search engines and other bots.
_________________
cpa nyc |
|
| Back to top |
|
|
| View previous topic :: View next topic |
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Private Messages
Log in
