{"id":6003,"date":"2015-07-15T15:02:09","date_gmt":"2015-07-15T15:02:09","guid":{"rendered":"https:\/\/www.techopedia.com\/definition\/web-crawler\/"},"modified":"2017-01-17T15:24:35","modified_gmt":"2017-01-17T15:24:35","slug":"web-crawler","status":"publish","type":"definition","link":"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler","title":{"rendered":"Web Crawler"},"content":{"rendered":"

What Does Web Crawler Mean?<\/span><\/h2>\n

A Web crawler is an Internet bot which helps in Web indexing. They crawl one page at a time through a website until all pages have been indexed. Web crawlers help in collecting information about a website and the links related to them, and also help in validating the HTML code and hyperlinks.\n<\/p>\n

A Web crawler is also known as a Web spider, automatic indexer or simply crawler.\n<\/p>\n

Techopedia Explains Web Crawler<\/span><\/h2>\n

Web crawlers collect information such the URL of the website, the meta tag information, the Web page content, the links in the webpage and the destinations leading from those links, the web page title and any other relevant information. They keep track of the URLs which have already been downloaded to avoid downloading the same page again. A combination of policies such as re-visit policy, selection policy, parallelization policy and politeness policy determines the behavior of the Web crawler. There are many challenges for web crawlers, namely the large and continuously evolving World Wide Web, content selection tradeoffs, social obligations and dealing with adversaries.\n<\/p>\n

Web crawlers are the key components of Web search engines and systems that look into web pages. They help in indexing the Web entries and allow users to send queries against the index and also provide the webpages that match the queries. Another use of Web crawlers is in Web archiving, which involves large sets of webpages to be periodically collected and archived. Web crawlers are also used in data mining, wherein pages are analyzed for different properties like statistics, and data analytics are then performed on them.<\/p>\n","protected":false},"excerpt":{"rendered":"

What Does Web Crawler Mean? A Web crawler is an Internet bot which helps in Web indexing. They crawl one page at a time through a website until all pages have been indexed. Web crawlers help in collecting information about a website and the links related to them, and also help in validating the HTML […]<\/p>\n","protected":false},"author":7813,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","format":"standard","meta":{"_acf_changed":false,"_lmt_disableupdate":"","_lmt_disable":"","om_disable_all_campaigns":false,"footnotes":""},"definitioncat":[245],"class_list":["post-6003","definition","type-definition","status-publish","format-standard","hentry","definitioncat-internet"],"acf":[],"yoast_head":"\nWhat is a Web Crawler? - Definition from Techopedia<\/title>\n<meta name=\"description\" content=\"This definition explains the meaning of Web Crawler and why it matters.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Web Crawler\" \/>\n<meta property=\"og:description\" content=\"This definition explains the meaning of Web Crawler and why it matters.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler\" \/>\n<meta property=\"og:site_name\" content=\"Techopedia\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/techopedia\/\" \/>\n<meta property=\"article:modified_time\" content=\"2017-01-17T15:24:35+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@techopedia\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler\"},\"author\":{\"name\":\"Margaret Rouse\",\"@id\":\"https:\/\/www.techopedia.com\/#\/schema\/person\/f5dd538e31ee352d105b8af36c4268a5\"},\"headline\":\"Web Crawler\",\"datePublished\":\"2015-07-15T15:02:09+00:00\",\"dateModified\":\"2017-01-17T15:24:35+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler\"},\"wordCount\":271,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.techopedia.com\/#organization\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler#respond\"]}],\"articleSection\":\"\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler\",\"url\":\"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler\",\"name\":\"What is a Web Crawler? - Definition from Techopedia\",\"isPartOf\":{\"@id\":\"https:\/\/www.techopedia.com\/#website\"},\"datePublished\":\"2015-07-15T15:02:09+00:00\",\"dateModified\":\"2017-01-17T15:24:35+00:00\",\"description\":\"This definition explains the meaning of Web Crawler and why it matters.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.techopedia.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Internet\",\"item\":\"https:\/\/www.techopedia.com\/topic\/102\/internet\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Term\",\"item\":\"https:\/\/www.techopedia.com\/definition\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Web Crawler\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.techopedia.com\/#website\",\"url\":\"https:\/\/www.techopedia.com\/\",\"name\":\"Techopedia\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.techopedia.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.techopedia.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.techopedia.com\/#organization\",\"name\":\"Techopedia\",\"url\":\"https:\/\/www.techopedia.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.techopedia.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.techopedia.com\/wp-content\/uploads\/2023\/08\/techopedia-light.svg\",\"contentUrl\":\"https:\/\/www.techopedia.com\/wp-content\/uploads\/2023\/08\/techopedia-light.svg\",\"width\":209,\"height\":37,\"caption\":\"Techopedia\"},\"image\":{\"@id\":\"https:\/\/www.techopedia.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/techopedia\/\",\"https:\/\/x.com\/techopedia\",\"https:\/\/www.linkedin.com\/company\/techopedia\/\",\"https:\/\/www.youtube.com\/c\/Techopedia\"],\"publishingPrinciples\":\"https:\/\/www.techopedia.com\/about\/editorial-policy\",\"ownershipFundingInfo\":\"https:\/\/www.techopedia.com\/about\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.techopedia.com\/#\/schema\/person\/f5dd538e31ee352d105b8af36c4268a5\",\"name\":\"Margaret Rouse\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.techopedia.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/www.techopedia.com\/wp-content\/uploads\/2023\/02\/margaret-rouse-headshot.jpeg\",\"contentUrl\":\"https:\/\/www.techopedia.com\/wp-content\/uploads\/2023\/02\/margaret-rouse-headshot.jpeg\",\"caption\":\"Margaret Rouse\"},\"description\":\"Margaret is an award-winning writer and educator known for her ability to explain complex technical topics to a non-technical business audience. Over the past twenty years, her IT definitions have been published by Que in an encyclopedia of technology terms and cited in articles in the New York Times, Time Magazine, USA Today, ZDNet, PC Magazine, and Discovery Magazine. She joined Techopedia in 2011. Margaret\u2019s idea of \u200b\u200ba fun day is to help IT and business professionals to learn to speak each other\u2019s highly specialized languages.\",\"sameAs\":[\"https:\/\/www.linkedin.com\/in\/margaretrouse\/\",\"https:\/\/x.com\/https:\/\/twitter.com\/@techdefinitions\"],\"knowsAbout\":[\"Technology expert\"],\"url\":\"https:\/\/www.techopedia.com\/contributors\/margaret-rouse\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"What is a Web Crawler? - Definition from Techopedia","description":"This definition explains the meaning of Web Crawler and why it matters.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler","og_locale":"en_US","og_type":"article","og_title":"Web Crawler","og_description":"This definition explains the meaning of Web Crawler and why it matters.","og_url":"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler","og_site_name":"Techopedia","article_publisher":"https:\/\/www.facebook.com\/techopedia\/","article_modified_time":"2017-01-17T15:24:35+00:00","twitter_card":"summary_large_image","twitter_site":"@techopedia","twitter_misc":{"Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler#article","isPartOf":{"@id":"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler"},"author":{"name":"Margaret Rouse","@id":"https:\/\/www.techopedia.com\/#\/schema\/person\/f5dd538e31ee352d105b8af36c4268a5"},"headline":"Web Crawler","datePublished":"2015-07-15T15:02:09+00:00","dateModified":"2017-01-17T15:24:35+00:00","mainEntityOfPage":{"@id":"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler"},"wordCount":271,"commentCount":0,"publisher":{"@id":"https:\/\/www.techopedia.com\/#organization"},"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.techopedia.com\/definition\/10008\/web-crawler#respond"]}],"articleSection":""},{"@type":"WebPage","@id":"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler","url":"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler","name":"What is a Web Crawler? - Definition from Techopedia","isPartOf":{"@id":"https:\/\/www.techopedia.com\/#website"},"datePublished":"2015-07-15T15:02:09+00:00","dateModified":"2017-01-17T15:24:35+00:00","description":"This definition explains the meaning of Web Crawler and why it matters.","breadcrumb":{"@id":"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.techopedia.com\/definition\/10008\/web-crawler"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.techopedia.com\/definition\/10008\/web-crawler#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.techopedia.com\/"},{"@type":"ListItem","position":2,"name":"Internet","item":"https:\/\/www.techopedia.com\/topic\/102\/internet"},{"@type":"ListItem","position":3,"name":"Term","item":"https:\/\/www.techopedia.com\/definition"},{"@type":"ListItem","position":4,"name":"Web Crawler"}]},{"@type":"WebSite","@id":"https:\/\/www.techopedia.com\/#website","url":"https:\/\/www.techopedia.com\/","name":"Techopedia","description":"","publisher":{"@id":"https:\/\/www.techopedia.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.techopedia.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.techopedia.com\/#organization","name":"Techopedia","url":"https:\/\/www.techopedia.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.techopedia.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.techopedia.com\/wp-content\/uploads\/2023\/08\/techopedia-light.svg","contentUrl":"https:\/\/www.techopedia.com\/wp-content\/uploads\/2023\/08\/techopedia-light.svg","width":209,"height":37,"caption":"Techopedia"},"image":{"@id":"https:\/\/www.techopedia.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/techopedia\/","https:\/\/x.com\/techopedia","https:\/\/www.linkedin.com\/company\/techopedia\/","https:\/\/www.youtube.com\/c\/Techopedia"],"publishingPrinciples":"https:\/\/www.techopedia.com\/about\/editorial-policy","ownershipFundingInfo":"https:\/\/www.techopedia.com\/about"},{"@type":"Person","@id":"https:\/\/www.techopedia.com\/#\/schema\/person\/f5dd538e31ee352d105b8af36c4268a5","name":"Margaret Rouse","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.techopedia.com\/#\/schema\/person\/image\/","url":"https:\/\/www.techopedia.com\/wp-content\/uploads\/2023\/02\/margaret-rouse-headshot.jpeg","contentUrl":"https:\/\/www.techopedia.com\/wp-content\/uploads\/2023\/02\/margaret-rouse-headshot.jpeg","caption":"Margaret Rouse"},"description":"Margaret is an award-winning writer and educator known for her ability to explain complex technical topics to a non-technical business audience. Over the past twenty years, her IT definitions have been published by Que in an encyclopedia of technology terms and cited in articles in the New York Times, Time Magazine, USA Today, ZDNet, PC Magazine, and Discovery Magazine. She joined Techopedia in 2011. Margaret\u2019s idea of \u200b\u200ba fun day is to help IT and business professionals to learn to speak each other\u2019s highly specialized languages.","sameAs":["https:\/\/www.linkedin.com\/in\/margaretrouse\/","https:\/\/x.com\/https:\/\/twitter.com\/@techdefinitions"],"knowsAbout":["Technology expert"],"url":"https:\/\/www.techopedia.com\/contributors\/margaret-rouse"}]}},"_links":{"self":[{"href":"https:\/\/www.techopedia.com\/wp-json\/wp\/v2\/definition\/6003"}],"collection":[{"href":"https:\/\/www.techopedia.com\/wp-json\/wp\/v2\/definition"}],"about":[{"href":"https:\/\/www.techopedia.com\/wp-json\/wp\/v2\/types\/definition"}],"author":[{"embeddable":true,"href":"https:\/\/www.techopedia.com\/wp-json\/wp\/v2\/users\/7813"}],"replies":[{"embeddable":true,"href":"https:\/\/www.techopedia.com\/wp-json\/wp\/v2\/comments?post=6003"}],"version-history":[{"count":0,"href":"https:\/\/www.techopedia.com\/wp-json\/wp\/v2\/definition\/6003\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.techopedia.com\/wp-json\/wp\/v2\/media?parent=6003"}],"wp:term":[{"taxonomy":"definitioncat","embeddable":true,"href":"https:\/\/www.techopedia.com\/wp-json\/wp\/v2\/definitioncat?post=6003"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}