You may well be pleased to hear that Web Spiders are not of the Arachnid type. This term was coined because the search engines crawled the Worldwide Web indexing pages and what do you normally find crawling on a web, a spider!
A web spider (or sometimes referred to as robot or agent) doesn't actually go anywhere it is simply a program on a Search Engine's indexing server which basically goes through the mundane job of loading a web page which has been submitted for indexing or has been linked to by one of it's existing indexed pages. This software program will then read the code in the page and strip out any HTML and Javascript code leaving META tags and content.
Some Search Engines, not all, will compare the META keywords tag with the content of the page. If it feels that it is relevant enough, it will rank it accordingly or alternatively, if the web developer has used underhand methods to get indexed, the page could be blacklisted.
So there you are, there are no hairy legged things crawling over your web pages!
All fields are required. Email addresses are not displayed on our website. All comments are moderated before appearing on our website.
We have filled the SEO and developer jobs - welcome Tim2 and Hannah!
Thanks for posting a comment.
Your comment will appear once its been approved by an admin.