How search engines work – Web Crawlers

These are the search engines that finally bring your website to the notice of potential customers. Therefore, it is better to know how these search engines actually work and how to present information to clients to initiate a search.

There are basically two types of search engines. The first is by robots called crawlers or spiders.

Search engines use spiders to index websites. When you submit your website pages to a search engine by completing their required submission page, the search engine spider to index your entire site. A 'spider' is an automated program that is run by the search engine system. Spider visits a web site to read the content on the same site, the site's Meta tags and follow the links that the site connects. La araña devuelve toda la información de vuelta a un depósito central, donde los datos se indexan. Will visit each link you have in your website and index those sites as well. Some spiders only index a certain number of pages on your site, so do not create a site with 500 pages!

Spider periodically revisit sites to verify all information that has changed. The frequency with which this happens is determined by the moderators of the search engines.

A spider is almost like a book which contains the table of contents, content and links and references to all the websites it finds during its search, and can index up to a million pages a day.

Example: Excite, Lycos, Altavista and Google.

When you ask a search engine to locate information, it is actually searching the index has been created and not really looking for the Web. Different search engines produce different rankings because not every search engine uses the same algorithm to search through the indices.

One of the things that a search engine algorithm is looking for the frequency and location of keywords on a website, but can also detect artificial keyword stuffing or spamdexing. Then the algorithms analyze the way that pages link to other web pages. Checking how pages link to each other, the engine can determine what is a page where the keywords of the linked pages are similar to keywords in the original page.