You say your website traffic isn’t exactly setting records? Well, you could be showing important visitors the exit door. Unless these guests enter, SEO value will sink. And inbound traffic will dwindle.
The visitors in question are known as ‘bots’. It’s incumbent upon digital brand marketers to know the bot and ensure each one is shown the welcome mat.
Basically, bots are digital detectives launched into cyberspace by Google and other search engines. Their stated purpose – do a little snooping; see what your website is all about.
This investigative process, known as crawling, gives bots the raw material necessary to uncover all meaningful pages, scrupulously index them, and fire the findings back to Google for further use. When search terms match the info culled from a crawl, search engines display the appropriate page or pages in search results. If, however, the crawl fails, thereby stifling or limiting info transmission to Google, website visibility is compromised. Often seriously. Upshot – your brand never makes it to the search results.
Crawl failures often are the result of glaring internal website errors. At their core, these errors serve to confuse, stymie, repel, and discourage the curious bot. Instead of gathering information, the frustrated detectives are given no choice but to leave empty-handed. Collectively, these errors easily can be called website spoilers. Because that’s what they do – spoil the available opportunities to capture website traffic.
The dreaded DNS error is one of these culprits. A common issue generally caused by coding flaws, the DNS error prevents servers from loading web pages with sufficient speed. Eventually, the loading process will time out. Thus, page access is denied to the well-intentioned bot, which never will know about and therefore never will index the unseen content.
The robots.txt file is another problem zone. This data sector indicates any web pages that should not be indexed. Before conducting a full crawl, bots prowl the robots.txt file. If the file can’t be accessed, the crawl can’t commence – and won’t commence until access is granted. If website traffic isn’t up to par, make sure a flawed .txt file isn’t blocking the bots.
URL errors complete the trio of known website spoilers. While these have many causes, one of the most common is the presence of links to pages that have been removed. These empty links, which lead nowhere, do a great job at confounding bots. The wisest of brand marketers will make sure all internal links lead to an existing destination.
If you have questions or comments about website crawl errors, or about any other brand-related topic, feel free to send them our way.