How search engines crawl and index websites

When a website is discovered by a search engine it goes through a process from being discovered to being indexed in the search engine results.

Search engines use automated robots (spiders) to locate and crawl websites. These spiders navigate through the internet by following the network of hyperlinks between pages on the internet.

Once the spider has followed a link to the website it then begins to crawl it. Crawling is just gathering the data contained on the web page (the source code) and fetching it for the search engine.

After fetching the information, the search engines will index it. Indexing is making sense of the information contained on the page and then placing it in the relative area. It’s important to note that indexing and crawling are different processes.

Once a search engine has crawled and indexed a page it has the opportunity to appear in the search engine results pages (SERPs). The position of the page in the results is dependent on how relevant the search engine deems the particular page to be to the search query. The relevancy is made up of many factors, for more information on this see how search engines rank pages.