Welcome to Spiders
Main search engines have automated
mechanisms that are searching the web to know what is new and what is
changing.
The so called “Spiders” or “Robots” that visit your site, crawl it and
eventually index some (or all) of your new pages. In order to do that,
the engine must know of the existence of your site. So it’s very
important that you submit your domain at least once a month, and no
more, to each engine.
This visit will also be started by a link from an established site. Be
only aware to have links from sites with good reputations, other wise it
may be harmful.
Program your site to be polite with the spider’s visit. A well formatted
“robot.txt” is a good welcome. Clean codes, light pages, and error free
intra-linking will benefit your site’s indexing.
Another important piece for describing the site is with a “sitemap”
page. Its construction takes no efforts and may be read by both humans
and machines.
Different pages must have different content. Try to avoid that two or
more pages may seem to be clones of each other. Instead, specialize
every page with some aspect of the site theme and use different key
words for different pages.
Try to identify the spider visits in your logs and keep track of them,
once you are sure that the engine considers your site friendly and
acceptable, you may stop worrying about that spider, and it will keep
visiting your pages unless you commit some intolerable mistake.
A good place to begin to understand engines’ preferences are the Google
guidelines for webmasters. And more importantly, there, you may also
understand what they do not like.
Use of Keywords
