However, it depends how the server responds to those directory requests. If it gives response 200 A1SG will believe it’s an URL you will want included. You can get around this by using list filters. Under all circumstances, feel free to email me/support directly

I am new to website design, and url’s design and i am therefore doing a lot of experimenting.

I have the following link structure on my website.

http:/example.com/1/2/3/4/

When i run the sitemap generator for the *HTML template sitemap* i find that links are created to directories 2 and 3 which might not contain anything other than the directory which follows. Directory #4 would then contain my index.php page.

Is there anyway i could prevent a link being created to dirctories 2 and 3.

Tahnks in advance.

Jay

A short explanation is that “recrawl” will recrawl entire website and not just “missing” URLs. Imagine you have a large website where it takes a lot of time to crawl the entire website. In those cases it’s advantageous to have a crawler recrawl the website by prioritizing pages it hasn’t crawled in a long time. (The latter part about prioritize is not fully implemented yet, but it is going to happen at some point.)