web crawlers

常見例句

Once the Sitemap XML is completed, you can make it available to Web crawlers by submitting a URL to the sitemap to those sites that support the protocol.
完成 Sitemap XML 文件后，便可以向支持這一協(xié)議的站點提交站點地圖的 URL，這樣 web 爬蟲程序便可以使用該文件了。
Sitemaps 0.90 is a simple and intuitive way for webmasters to provide the right level of information to Web crawlers so that they can efficiently crawl a Web site.
Sitemaps 0.90為 Web 站點管理員提供了簡單而且直觀的方法，使他們可以向 web 爬蟲程序提供正確的信息，從而使爬蟲程序能夠高效地爬行某個 Web 站點。
Also known as "invisible web", the term "deep web" refers to a vast repository of underlying content, such as documents in online databases that general-purpose web crawlers cannot reach.
“深網(wǎng)”也被稱做“隱形網(wǎng)”，指的是網(wǎng)絡中存儲數(shù)量浩繁的基本信息的空間，例如存儲于在線數(shù)據(jù)庫中的文件，一般用途的網(wǎng)絡搜索器是找不到這些文件的。

"Think algorithms and models and different software and Web crawlers, " says Christine Elliott, an American Express spokesperson.
FORBES: Magazine Article
Industry specific web crawlers or vulnerability discovery bots, network scans, and resource exhaustion attacks against web servers are rampant.
FORBES: Clean That Network Traffic
Through the use of proprietary Web crawlers, this news organization had breached the firewall to provide its readers with an upper hand.
FORBES: Data Security: A Shareholder Protection Imperative