我們在訪問網站的時候，發現有些網頁ID 是按順序排列的數字，這個時候我們就可以使用ID遍歷的方式來爬取內容。但是局限性在于有些ID數字在10位數左右，那么這樣爬取效率就會很低很低！



import itertools
from common import download


def iteration():
max_errors = 5 # maximum number of consecutive download errors allowed
num_errors = 0 # current number of consecutive download errors
for page in itertools.count(1):
url = 'http://example.webscraping.com/view/-{}'.format(page)
html = download(url)
if html is None:
# received an error trying to download this webpage
num_errors += 1
if num_errors == max_errors:
# reached maximum amount of errors in a row so exit
break
# so assume have reached the last country ID and can stop downloading
else:
# success - can scrape the result
# ...
num_errors = 0

轉載于:https://www.cnblogs.com/mrruning/p/7638459.html

總結

以上是生活随笔為你收集整理的用python写网络爬虫 -从零开始 3 编写ID遍历爬虫的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。