- Free
- Windows desktop application
- Site map generator
- Link checker
- Duplicate content checker
- Analyse URLs, page title, description, keywords
- Crawler
| Functionality | 4/5 | just enough features to make the tool fully functional and not bloated |
| Interface | 3/5 | easy to use but dated and not intuitive in places |
| Value | 5/5 | free and without ads or nags- give the guy a donation! |
| Support | 4/5 | active Google Groups support but not actively developed |
GSiteCrawler has been around for a few years but still stands the test of time as a versatile, free tool to sanity check a website for common problems and
create search engine site maps.
In the context of SEO it’s as simple as plugging in the URL, letting the crawler spider the site and you’ll immediately expose several things that could harm SEO.
My tests reveal it handles cookies and session state with the default crawl configuration and is pretty much platform agnostic for the target website.
Site Map Generator
Templates are provided for Google Sitemaps and Yahoo! URL list file generation. Custom templates can be easily written for reporting, or you can plug it into a SQL Server database and do a direct query.
The interface also acts as a visual site map editor without the need to work with the raw XML. Site maps can be uploaded by FTP after a crawl has completed. Pair this with the basic automation (using a command line switch) and schedule a task and you’ve got a perfectly good way to create an automatically updated site map.
Link Checker
Broken internal links (404 – not found) are reported on after the crawl. What’s really smart is it’ll still work if your target site badly handles not found pages, by not returning the correct HTTP response code. The wizard detects the bad setup and instead looks for a string in the ‘page not found’ page, logging this as a not found.
Duplicate Content Checker
This feature is useful for the most basic of duplicate content- where 100% of the content is duplicated under a different URL. It can’t handle the case where a page contains the same content but is skinned differently and therefore may be treated as duplicate content by search engines.
Analyse URLs, page title, description, keywords
The URL list tab is invaluable for getting a bird’s eye view of your site’s URLs, titles, descriptions and keywords.
The interface is a bit cumbersome for performing analysis on a site with 100’s or 1000’s of URLs, but it at least allows you to pick out the SEO worst offenders of “all gallery pages have the same page title!” Without having to trawl the site yourself.
Crawler
After struggling with several other crawlers that require cookie configuration, custom settings for handling session state etc. GSiteCrawler was a breeze. Plugging it into classic ASP, PHP, .NET and extension-less sites couldn’t trip it up.
Don’t worry about overloading your site or interfering with others’ browsing experience as the number and rate of crawlers is completely configurable.
Being able to view the crawler is also hugely useful. Other crawlers silently go about their business, but GSiteCrawler gives you a detailed rundown of what’s currently happening and the projected remaining duration.
Conclusion
If you own a website, have been called in to work on one, or are responsible for its SEO then GSiteCrawler gives you a no-nonsense free tool in your arsenal.
By no means is it the only tool you’ll ever need for SEO, but it ticks a few of the SEO foundation tasks off the list.
The final plus is that
support through Google Groups is active and lively. The mere fact the tool has weathered several years without much development (last release 28th December 2007) is evidence of a stable, reliable and functional piece of software.
Improvements
- Option to ignore robots.txt and nofollow links when I want to check my entire site, not just the indexed portions
- Lots of knowledge in the Group, but sparse documentation on the website
- Bringing the tool back into active development after being mothballed- web-based version anyone?
