爬蟲篇——User-Agent爬取備用及存儲

• 代碼

代碼

本文通過抓取常見的User-Agent（用戶代理），將其寫入列表并保存為json格式文件，且將代碼進(jìn)行了封裝，方便以后抓取數(shù)據(jù)時動態(tài)的更新請求頭中的User-Agent，模擬真實(shí)的瀏覽器發(fā)送請求，從一方面避免抓取數(shù)據(jù)時反爬的干擾。

# *********************** User-Agent 爬取 ******************************** import requests from bs4 import BeautifulSoup import re import json import urllib.requestclass UserAgentSpider(object):def __init__(self):self.headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"}self.url = "https://blog.csdn.net/jamesaonier/article/details/89003053"self.user_agent_list = []self.validation_user_agent_list = []# 1、發(fā)送網(wǎng)絡(luò)請求 獲取數(shù)據(jù)def data_request(self):return requests.get(self.url,headers = self.headers).content.decode("utf-8")# 2、數(shù)據(jù)解析def data_parse(self,data):parse_data = BeautifulSoup(data,"lxml")all_agent = parse_data.select("div p")[1:12]for agent in all_agent:pattern = re.compile("\n")each_agent = pattern.split(agent.get_text())[1:]for each in each_agent:self.user_agent_list.append(each)# 3、檢驗(yàn)user-agent的可用性def test_validation(self):for agent in self.user_agent_list:try:url = "https://www.baidu.com/"headers = {}headers["User-Agent"] = agentrequests.get(url,headers = headers)self.validation_user_agent_list.append(agent)except urllib.request.HTTPError as error:print(error.code)# 4、數(shù)據(jù)存儲def data_save(self):with open("./User-Agent.json","w",encoding = "utf-8") as fp:json.dump(self.validation_user_agent_list,fp)# 4、統(tǒng)籌運(yùn)行def run(self):# 1、請求數(shù)據(jù)data = self.data_request()# 2、數(shù)據(jù)解析self.data_parse(data) # print(self.user_agent_list)# 3、檢驗(yàn)user-agent的可用性self.test_validation()# 4、數(shù)據(jù)存儲self.data_save()if __name__ == "__main__":UserAgentSpider().run()

代碼運(yùn)行結(jié)果：

["Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60", "Opera/8.0 (Windows NT 5.1; U; en)", "Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 9.50", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 9.50", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0", "Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36", "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)\"", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; SE 2.X MetaSr 1.0)", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Maxthon/4.4.3.4000 Chrome/30.0.1599.101 Safari/537.36", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 UBrowser/4.0.3214.0 Safari/537.36"]

by CyrusMay 2020 04 24

從前只想裝懂
裝作什么都懂
懂得生存的規(guī)則之后
卻只想要 都不懂
——————五月天——————

總結(jié)

以上是生活随笔為你收集整理的爬虫篇——User-Agent爬取备用及存储的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。