web crawling

基本解釋

But all the major and legitimate Web crawling engines obey the requests in robots.txt.
不過(guò)，所有主要的合法 Web 爬蟲(chóng)引擎都會(huì)遵從 robots.txt 內(nèi)的要求。
However the current two ways of web crawling used by focus crawler are low efficiency.
然而目前的主題爬蟲(chóng)所采用的兩種基本抓取網(wǎng)頁(yè)的方式效率比較低下。
kns50.chkd.cnki.net
As the key component of search strategy in topic-oriented web spider, the topic relativity judging algorithms ensure the focused web crawling process of the spider.
作為主題網(wǎng)絡(luò)蜘蛛搜索策略的核心部分，主題相關(guān)性判斷算法是網(wǎng)絡(luò)蜘蛛能夠圍繞設(shè)定主題進(jìn)行聚焦檢索的關(guān)鍵。

Likewise, a web crawling robot which follows links such as Google is not effective.
FORBES: How reputation services enhance IPS
He mentions use cases like batch processing, scientific computing, video and image processing and rendering, testing and Web crawling.
FORBES: Amazon CTO Werner Vogels On The State Of The Cloud
The search engine's Web-crawling spider now keeps an eye out for viruses and malware as it combs the Web, and shows users a warning page before allowing them to access malicious sites.
FORBES: Made For Hacking