Search engines improve XML sitemaps with robots.txt
After Google/Yahoo/MSN announced support for XML sitemaps protocol, it was only a matter of time before more would follow. Yesterday it was announced that Ask.com and IBM have been added to the list of XML sitemap protocol supporters.
One of the hottest news seem to be that it is now possible to link your xml sitemap from your robots.txt file.
You can simply add this line to your robots.txt file:
Sitemap: http://www.example.com/sitemap.xml
I think this change will help make the XML sitemaps protocol a widely used standard. The primary reason being it enables search engines to automatically discover XML sitemaps on their own.
For more information, you can check two recent blog posts by Google and Yahoo.
Comments
Comment from Thomas Schulz
Time: April 18, 2007, 1:23 am
Indeed yes, XML sitemaps and A1 Sitemap Generator have just become even more anchored into the future 
With Google, Yahoo, MSN, Ask.com and IBM supporting it, I think it is only a matter of time before adoption among webmasters and search engines will be near 100%
Pingback from robots.txt usefulness – Webmaster Forums – Webmaster forum for HTML, PHP, ASP, CSS and more
Time: May 15, 2007, 7:36 pm
[...] robots.txt usefulness You can link to your XML sitemap in robots.txt file now __________________ A1 Website Analyzer : SEO, PPC and website analysis A1 Sitemap [...]
Comment from sherri
Time: June 5, 2007, 9:19 am
I must have been doing something wrong then because I had my sitemap.xml listed in my robots.txt (after reading this information) and when I checked Google’s Webmaster tools it said that was the only page NOT CRAWLED because of a robots.txt exclusion. I can’t figure this out!
Comment from Thomas Schulz
Time: June 9, 2007, 3:24 am
Hi Sherri,
How did your robots.txt file look like?
I have an idea of what might have gone wrong, but difficult to say for sure without seeing your robots.txt file
Pingback from robots.txt file is compulsary in web sites codingfor for SEO ? – Webmaster Forums – Webmaster forum for HTML, PHP, ASP, CSS and more
Time: June 20, 2007, 3:17 pm
[...] compulsary in web sites codingfor for SEO ? Well, you can read my blog post about having your robots.txt link xml sitemap . That is kinda neat, and likely to help you get indexed and save time in doing so. [...]
Pingback from can i submit sitemap to msn? – WebProWorld
Time: August 15, 2007, 11:04 pm
[...] can i submit sitemap to msn? Ask also understands robots.txt sitemap discovery. In general, it allows any search engine to discover your XML sitemap. __________________ A1 [...]
Comment from DiscussHub.Com
Time: April 1, 2008, 7:36 pm
Nice Software, I really appreciate it, From two years my forum was having only 16 indexed paged now more than 16 pages indexed and my CTR is high…
Comment from John
Time: August 5, 2008, 7:25 pm
How can I add more than one xml sitemap file to my robots.txt? I have a site named http://www.ReviewPage.com, but struggle to get my sitemaps working properly. Any advice?
Thanks!
Comment from Thomas Schulz
Time: August 6, 2008, 2:37 pm
There two solutions:
Can list each sitemap in the robots.txt file:
Sitemap: http://www.example.com/sitemap.xml
Sitemap: http://www.example.com/sitemap-1.xml
Or refer to the sitemap index file instead:
Sitemap: http://www.example.com/sitemap-index.xml
Comment from saqib
Time: December 13, 2008, 5:15 pm
Is robots file is Important in the Sitmaps
Comment from saqib
Time: December 13, 2008, 5:23 pm
I have facing some problems with my sitemap some one here who help me my site address is http://www.urdua.com/
Comment from Thomas Schulz
Time: December 19, 2008, 2:07 am
I always recommend people contact Microsys support directly with questions concerning our sitemap generator product as blog comments can be a little delayed
Anyways, as you do not describe the nature of your problem, I suggest you first check the sitemap generator tutorials and sitemap generator “quick answers”.
Comment from Nerdlab
Time: April 13, 2007, 2:46 pm
This is a step in the good direction for XML sitemaps. This is a thing we will think more about as having as default out-of-the-box function in our upcomming relases of e-CMS etc.
As well great that A1 Sitemap Generator have support for XML sitemaps since version 1.0. Should we call it a future-proof application???