How do I know which pages have been added or removed with the latest crawl? (Check History)

Answer

Use Check History within the Quality Assurance module to identify pages and links that have been added or removed between crawls. Check History compares the current crawl with the previous crawl and categorizes pages and links as known, new, or removed. It also helps explain unexpected increases or decreases in page and link counts and can help determine whether changes are caused by website updates, crawl settings, or accessibility issues.

When to Use Check History

Use Check History when:

• You notice an increase or decrease in page count.
• You notice an increase or decrease in link count.
• You want to identify newly discovered pages or links.
• You want to identify pages or links that are no longer being detected.
• You want to investigate crawl fluctuations.
• You want to determine whether content should be included or excluded from your crawl.

Check History

The Check History functionality within Quality Assurance provides information for each report that was sent at the end of a full crawl. The Check History overview provides information including report date and time, link count, the number of broken links, misspellings, potential misspellings, and page count. More detailed information can be accessed by clicking on a number in the page count and link count.

Note: More detailed information on page count and link count can only be accessed for the five most recent crawls, older crawl report information will become non-clickable.

Page Count Definitions

Known pages: Pages that have been seen during the previous report crawl and that have also been seen in this specific report crawl.

New pages: New pages seen during this specific report crawl that were not seen in the previous report crawl.

Removed pages: Pages that were seen on the previous report crawl but not seen on this specific report crawl.

Link Count Definitions

Known links: Links that Siteimprove crawlers have detected during the previous report crawl and that have also been seen in this specific report crawl.

New Links: New links detected during the specifically chosen report crawl.

Removed Links: Links that were on the previous report but are not detected on this particular crawl.

How Check History Can Help

Check History can be useful to detect a decrease or increase in page count and/or link count. Often you will see that Siteimprove's crawler has picked up a new part of the website (for example, a subdomain, calendars, events list, suppliers list, planning applications, etc.) which can explain a large increase in page/link count. The Check History functionality also helps determine if any sections or pages of your website should be included or excluded from the crawl.

Causes of Page Count Changes

The Page No Longer Exists or New Pages Were Created

A change in page count can simply be the result of new pages being added or removed from your site. Review the Check History page to determine what pages were added or removed.

The Entrance (or Landing) Page To A Section Of Your Site Was Removed or is Inaccessible

You may have sections of your site with a large amount of content that is only accessible through a few entrance links. If our crawler doesn't find the entrance link to this section, then we'll never find the pages contained within this section. Review the page that contains the entrance link to the section and determines if it still exists or is broken. It is not possible for our crawler to find orphan pages. An orphan page is one that is not linked to by another page on the site.

Recent Change of Site Content Settings

Site content settings tell our crawler what content should be included as a page, and what should be considered "external" content. Please see the following article for additional information: Site Content Settings

Recent Changes To Your Site's robots.txt Disallows

By default, our crawler respects the robots.txt file placed on your root directory

https://www.example.com/robots.txt

The robots.txt file may contain disallow instructions telling our crawler to not follow sections of your site.