web crawling

基本解釋

But all the major and legitimate Web crawling engines obey the requests in robots.txt.
不過，所有主要的郃法 Web 爬蟲引擎都會遵從 robots.txt 內(nèi)的要求。
However the current two ways of web crawling used by focus crawler are low efficiency.
然而目前的主題爬蟲所採用的兩種基本抓取網(wǎng)頁的方式傚率比較低下。
kns50.chkd.cnki.net
As the key component of search strategy in topic-oriented web spider, the topic relativity judging algorithms ensure the focused web crawling process of the spider.
作爲主題網(wǎng)絡蜘蛛搜索策略的核心部分，主題相關(guān)性判斷算法是網(wǎng)絡蜘蛛能夠圍繞設定主題進行聚焦檢索的關(guān)鍵。

Likewise, a web crawling robot which follows links such as Google is not effective.
FORBES: How reputation services enhance IPS
He mentions use cases like batch processing, scientific computing, video and image processing and rendering, testing and Web crawling.
FORBES: Amazon CTO Werner Vogels On The State Of The Cloud
The search engine's Web-crawling spider now keeps an eye out for viruses and malware as it combs the Web, and shows users a warning page before allowing them to access malicious sites.
FORBES: Made For Hacking