Article about solving website crawling problems
I have recently written an article about crawling of websites which explains and shows how to solve many common crawling problems.
Things covered include:
- Having multiple domains with mirrored and/or distributed content.
- Consistent usage of www (or not) in front of domain names.
- Usage of Javascript for navigation.
- Broken links and redirects.
And many other things, about 10 in total, that can result in problems for a website crawler. The article focuses on solving all website crawl and scan problems using A1 Sitemap Generator. For those who have websites where crawler problems can not be solved, the article also shows how you can still create complete HTML and XML sitemaps for your website by configuring the sitemap creator program.
Read the article and make your website crawlable by search engines now! 
Post new comments in forum
Discuss articles and get help at Web Help Forums.
