Bot term explained
Bot (from robot) is a robot search engine, which scans websites, indexing their content. This is an automated script running on a regular schedule.
Search bots have a specific function. Search bots have their own user-agents for which they are easy to identify. For example, there are the bots indexing the text content, video and audio content, images, blog bots, bots for news.
In addition search engines are have other bots whose function is also related work with content.
It can be various services or auto crawlers. Typically bots are intended to do the work monotonous and repeat as fast as possible (obviously far above human capabilities).
Major malicious bots:
- Spam bots that collect e-mail addresses from contact forms and guest books;
- A program that loads the Internet channel the flow of unnecessary information, usually advertising;
- Bots collect information about harmless sites for use automatically generated doorway pages;
- Some viruses and worms;
- DoS and DDoS attack;
- Botnets.
For the search bots to limit the indexing of the site by using the file robots.txt however, some unscrupulous bots can ignore the existence of the file. Full protection from indexation can be secured by other mechanisms, for example, by using the htaccess file.
