A web crawler is a "bot" generally used by search engines to look for new websites, download their data, and index it. They power most of the popular search engines like Google, Yahoo!, and Bing. These bots are called crawlers as this is the technical term to define what they do which is automatically opening a website and obtaining its data.
You can learn more about web crawlers from Wikipedia .