當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

爬虫之静态页面抓取

發布時間：2023/12/9 编程问答 29 豆豆

生活随笔收集整理的這篇文章主要介紹了爬虫之静态页面抓取小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

靜態網頁抓取

在網絡爬蟲中，靜態網頁的數據比較容易獲取，因為其所有數據都呈現在網頁的HTML代碼中

在靜態網頁抓取中，Python中的Requests庫能夠容易實現這個需求

通過requests發起Http請求

import requests url="http://www.santostang.com/" r=requests.get(url) print("文本編碼:",r.encoding) print("響應狀態碼:",r.status_code) print("響應文本內容:",r.text) 文本編碼: UTF-8 響應狀態碼: 200 響應文本內容: <!DOCTYPE html> <html lang="zh-CN"> <head> <meta charset="UTF-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1"> <title>Santos Tang</title> <meta name="description" content="Python網絡爬蟲：從入門到實踐官方網站及個人博客" /> <meta name="keywords" content="Python網絡爬蟲, Python爬蟲, Python, 爬蟲, 數據科學, 數據挖掘, 數據分析, santostang, Santos Tang, 唐松, Song Tang" /> <link rel="apple-touch-icon" href="http://www.santostang.com/wp-content/themes/SongStyle-Two/images/icon_32.png"> <link rel="apple-touch-icon" sizes="152x152" href="http://www.santostang.com/wp-content/themes/SongStyle-Two/images/icon_152.png"> <link rel="apple-touch-icon" sizes="167x167" href="http://www.santostang.com/wp-content/themes/SongStyle-Two/images/icon_167.png"> <link rel="apple-touch-icon" sizes="180x180" href="http://www.santostang.com/wp-content/themes/SongStyle-Two/images/icon_180.png"> <link rel="icon" href="http://www.santostang.com/wp-content/themes/SongStyle-Two/images/icon_32.png" type="image/x-icon"> <link rel="stylesheet" href="http://www.santostang.com/wp-content/themes/SongStyle-Two/css/bootstrap.min.css"> <link rel="stylesheet" href="http://www.santostang.com/wp-content/themes/SongStyle-Two/css/fontawesome.min.css"> <link rel="stylesheet" href="http://www.santostang.com/wp-content/themes/SongStyle-Two/style.css"> <link rel="pingback" href="http://www.santostang.com/xmlrpc.php" /> <style type="text/css"> a{color:#1e73be} a:hover{color:#2980b9!important} #header{background-color:#1e73be} .widget .widget-title::after{background-color:#1e73be} .uptop{border-left-color:#1e73be} #titleBar .toggle:before{background:#1e73be} </style> </head><body> <header id="header"><div class="avatar"><a href="http://www.santostang.com" title="Santos Tang"><img src="http://www.santostang.com/wp-content/uploads/2019/06/me.jpg" alt="Santos Tang" class="img-circle" width="50%"></a></div><h1 id="name">Santos Tang</h1><div class="sns"><a href="https://weibo.com/santostang" target="_blank" rel="nofollow" data-toggle="tooltip" data-placement="top" title="Weibo"><i class="fab fa-weibo"></i></a> <a href="https://www.linkedin.com/in/santostang" target="_blank" rel="nofollow" data-toggle="tooltip" data-placement="top" title="Linkedin"><i class="fab fa-linkedin"></i></a> <a href="https://www.zhihu.com/people/santostang" target="_blank" rel="nofollow" data-toggle="tooltip" data-placement="top" title="Zhihu"><i class="fab fa-zhihu"></i></a> <a href="https://github.com/santostang" target="_blank" rel="nofollow" data-toggle="tooltip" data-placement="top" title="GitHub"><i class="fab fa-github-alt"></i></a> </div><div class="nav"><ul><li><a href="http://www.santostang.com/">首頁</a></li> <li><a href="http://www.santostang.com/aboutme/">關于我</a></li> <li><a href="http://www.santostang.com/python%e7%bd%91%e7%bb%9c%e7%88%ac%e8%99%ab%e4%bb%a3%e7%a0%81/">爬蟲書代碼</a></li> <li><a href="http://www.santostang.com/%e5%8a%a0%e6%88%91%e5%be%ae%e4%bf%a1/">加我微信</a></li> <li><a href="https://santostang.github.io/">EnglishSite</a></li> </ul> </div><div class="weixin"><img src="http://www.santostang.com/wp-content/uploads/2019/06/qrcode_for_gh_370f70791e19_258.jpg" alt="微信公眾號" width="50%"><p>微信公眾號</p></div></header> <div id="main"><div class="row box"><div class="col-md-8"><h2 class="uptop"><i class="fas fa-arrow-circle-up"></i> <a href="http://www.santostang.com/2018/07/11/%e3%80%8a%e7%bd%91%e7%bb%9c%e7%88%ac%e8%99%ab%ef%bc%9a%e4%bb%8e%e5%85%a5%e9%97%a8%e5%88%b0%e5%ae%9e%e8%b7%b5%e3%80%8b%e4%b8%80%e4%b9%a6%e5%8b%98%e8%af%af/" target="_blank">《網絡爬蟲：從入門到實踐》一書勘誤</a></h2><article class="article-list-1 clearfix"><header class="clearfix"><h1 class="post-title"><a href="http://www.santostang.com/2018/07/15/4-3-%e9%80%9a%e8%bf%87selenium-%e6%a8%a1%e6%8b%9f%e6%b5%8f%e8%a7%88%e5%99%a8%e6%8a%93%e5%8f%96/">第四章 – 4.3 通過selenium 模擬瀏覽器抓取</a></h1><div class="post-meta"><span class="meta-span"><i class="far fa-calendar-alt"></i> 07月15日</span><span class="meta-span"><i class="far fa-folder"></i> <a href="http://www.santostang.com/category/python-%e7%bd%91%e7%bb%9c%e7%88%ac%e8%99%ab/" rel="category tag">Python 網絡爬蟲</a></span><span class="meta-span"><i class="fas fa-comments"></i> <a href="http://www.santostang.com/2018/07/15/4-3-%e9%80%9a%e8%bf%87selenium-%e6%a8%a1%e6%8b%9f%e6%b5%8f%e8%a7%88%e5%99%a8%e6%8a%93%e5%8f%96/#respond">沒有評論</a></span><span class="meta-span hidden-xs"><i class="fas fa-tags"></i> </span></div></header><div class="post-content clearfix"><p>4.3 通過selenium 模擬瀏覽器抓取在上述的例子中，使用Chrome“檢查”功能找到源地址還十分容易。但是有一些網站非常復雜，例如前面的天貓產品評論，使用“檢查”功能很難找到調用的網頁地址。除此之外，有一些數據...</p></div></article><article class="article-list-1 clearfix"><header class="clearfix"><h1 class="post-title"><a href="http://www.santostang.com/2018/07/14/4-2-%e8%a7%a3%e6%9e%90%e7%9c%9f%e5%ae%9e%e5%9c%b0%e5%9d%80%e6%8a%93%e5%8f%96/">第四章 – 4.2 解析真實地址抓取</a></h1><div class="post-meta"><span class="meta-span"><i class="far fa-calendar-alt"></i> 07月14日</span><span class="meta-span"><i class="far fa-folder"></i> <a href="http://www.santostang.com/category/python-%e7%bd%91%e7%bb%9c%e7%88%ac%e8%99%ab/" rel="category tag">Python 網絡爬蟲</a></span><span class="meta-span"><i class="fas fa-comments"></i> <a href="http://www.santostang.com/2018/07/14/4-2-%e8%a7%a3%e6%9e%90%e7%9c%9f%e5%ae%9e%e5%9c%b0%e5%9d%80%e6%8a%93%e5%8f%96/#respond">沒有評論</a></span><span class="meta-span hidden-xs"><i class="fas fa-tags"></i> <a href="http://www.santostang.com/tag/ajax/" rel="tag">ajax</a>,<a href="http://www.santostang.com/tag/python/" rel="tag">python</a>,<a href="http://www.santostang.com/tag/%e7%bd%91%e7%bb%9c%e7%88%ac%e8%99%ab/" rel="tag">網絡爬蟲</a>,<a href="http://www.santostang.com/tag/%e7%bd%91%e9%a1%b5%e7%88%ac%e8%99%ab/" rel="tag">網頁爬蟲</a>,<a href="http://www.santostang.com/tag/%e8%a7%a3%e6%9e%90%e5%9c%b0%e5%9d%80/" rel="tag">解析地址</a></span></div></header><div class="post-content clearfix"><p>由于網易云跟帖停止服務，現在已經在此處中更新了新寫的第四章。請參照文章： 4.2 解析真實地址抓取雖然數據并沒有出現在網頁源代碼中，我們也可以找到數據的真實地址，請求這個真實地址也可以獲得想要的數據。...</p></div></article><article class="article-list-1 clearfix"><header class="clearfix"><h1 class="post-title"><a href="http://www.santostang.com/2018/07/14/%e7%ac%ac%e5%9b%9b%e7%ab%a0%ef%bc%9a%e5%8a%a8%e6%80%81%e7%bd%91%e9%a1%b5%e6%8a%93%e5%8f%96-%e8%a7%a3%e6%9e%90%e7%9c%9f%e5%ae%9e%e5%9c%b0%e5%9d%80-selenium/">第四章- 動態網頁抓取 (解析真實地址 + selenium)</a></h1><div class="post-meta"><span class="meta-span"><i class="far fa-calendar-alt"></i> 07月14日</span><span class="meta-span"><i class="far fa-folder"></i> <a href="http://www.santostang.com/category/python-%e7%bd%91%e7%bb%9c%e7%88%ac%e8%99%ab/" rel="category tag">Python 網絡爬蟲</a></span><span class="meta-span"><i class="fas fa-comments"></i> <a href="http://www.santostang.com/2018/07/14/%e7%ac%ac%e5%9b%9b%e7%ab%a0%ef%bc%9a%e5%8a%a8%e6%80%81%e7%bd%91%e9%a1%b5%e6%8a%93%e5%8f%96-%e8%a7%a3%e6%9e%90%e7%9c%9f%e5%ae%9e%e5%9c%b0%e5%9d%80-selenium/#respond">沒有評論</a></span><span class="meta-span hidden-xs"><i class="fas fa-tags"></i> <a href="http://www.santostang.com/tag/ajax/" rel="tag">ajax</a>,<a href="http://www.santostang.com/tag/javascript/" rel="tag">javascript</a>,<a href="http://www.santostang.com/tag/python/" rel="tag">python</a>,<a href="http://www.santostang.com/tag/selenium/" rel="tag">selenium</a>,<a href="http://www.santostang.com/tag/%e7%bd%91%e7%bb%9c%e7%88%ac%e8%99%ab/" rel="tag">網絡爬蟲</a></span></div></header><div class="post-content clearfix"><p>由于網易云跟帖停止服務，現在已經在此處中更新了新寫的第四章。請參照文章：前面爬取的網頁均為靜態網頁，這樣的網頁在瀏覽器中展示的內容都在HTML源代碼中。但是，由于主流網站都使用JavaScript展現網頁內容，...</p></div></article><article class="article-list-1 clearfix"><header class="clearfix"><h1 class="post-title"><a href="http://www.santostang.com/2018/07/04/hello-world/">Hello world!</a></h1><div class="post-meta"><span class="meta-span"><i class="far fa-calendar-alt"></i> 07月04日</span><span class="meta-span"><i class="far fa-folder"></i> <a href="http://www.santostang.com/category/python-%e7%bd%91%e7%bb%9c%e7%88%ac%e8%99%ab/" rel="category tag">Python 網絡爬蟲</a></span><span class="meta-span"><i class="fas fa-comments"></i> <a href="http://www.santostang.com/2018/07/04/hello-world/#comments">1條評論</a></span><span class="meta-span hidden-xs"><i class="fas fa-tags"></i> </span></div></header><div class="post-content clearfix"><p>Welcome to WordPress. This is your first post. Edit or delete it, then start writing! 各位讀者，由于網易云跟帖在本書出版后已經停止服務，書中的第四章已經無法使用。所以我將本書的評論系統換成了來必力...</p></div></article><nav style="float:right"></nav></div><div class="col-md-4 hidden-xs hidden-sm"><aside class="widget clearfix"><form id="searchform" action="http://www.santostang.com"><div class="input-group"><input type="search" class="form-control" placeholder="搜索…" value="" name="s"><span class="input-group-btn"><button class="btn btn-default" type="submit"><i class="fas fa-search"></i></button></span></div></form> </aside> <aside class="widget clearfix"><h4 class="widget-title">文章分類</h4><ul class="widget-cat"><li class="cat-item cat-item-2"><a href="http://www.santostang.com/category/python-%e7%bd%91%e7%bb%9c%e7%88%ac%e8%99%ab/" >Python 網絡爬蟲</a> (5) </li></ul> </aside> <aside class="widget clearfix"><h4 class="widget-title">熱門文章</h4><ul class="widget-hot"></ul> </aside> <aside class="widget clearfix"><h4 class="widget-title">隨機推薦</h4><ul class="widget-hot"><li><a href="http://www.santostang.com/2018/07/04/hello-world/" title="Hello world!">Hello world!</a></li><li><a href="http://www.santostang.com/2018/07/11/%e3%80%8a%e7%bd%91%e7%bb%9c%e7%88%ac%e8%99%ab%ef%bc%9a%e4%bb%8e%e5%85%a5%e9%97%a8%e5%88%b0%e5%ae%9e%e8%b7%b5%e3%80%8b%e4%b8%80%e4%b9%a6%e5%8b%98%e8%af%af/" title="《網絡爬蟲：從入門到實踐》一書勘誤">《網絡爬蟲：從入門到實踐》一書勘誤</a></li><li><a href="http://www.santostang.com/2018/07/15/4-3-%e9%80%9a%e8%bf%87selenium-%e6%a8%a1%e6%8b%9f%e6%b5%8f%e8%a7%88%e5%99%a8%e6%8a%93%e5%8f%96/" title="第四章 – 4.3 通過selenium 模擬瀏覽器抓取">第四章 – 4.3 通過selenium 模擬瀏覽器抓取</a></li><li><a href="http://www.santostang.com/2018/07/14/%e7%ac%ac%e5%9b%9b%e7%ab%a0%ef%bc%9a%e5%8a%a8%e6%80%81%e7%bd%91%e9%a1%b5%e6%8a%93%e5%8f%96-%e8%a7%a3%e6%9e%90%e7%9c%9f%e5%ae%9e%e5%9c%b0%e5%9d%80-selenium/" title="第四章- 動態網頁抓取 (解析真實地址 + selenium)">第四章- 動態網頁抓取 (解析真實地址 + selenium)</a></li><li><a href="http://www.santostang.com/2018/07/14/4-2-%e8%a7%a3%e6%9e%90%e7%9c%9f%e5%ae%9e%e5%9c%b0%e5%9d%80%e6%8a%93%e5%8f%96/" title="第四章 – 4.2 解析真實地址抓取">第四章 – 4.2 解析真實地址抓取</a></li></ul> </aside> <aside class="widget clearfix"><h4 class="widget-title">標簽云</h4><div class="widget-tags"><a href="http://www.santostang.com/tag/ajax/" class="tag-cloud-link tag-link-3 tag-link-position-1" style="color:#66807;font-size: 22pt;" aria-label="ajax (2個項目);">ajax</a> <a href="http://www.santostang.com/tag/javascript/" class="tag-cloud-link tag-link-4 tag-link-position-2" style="color:#f508f0;font-size: 8pt;" aria-label="javascript (1個項目);">javascript</a> <a href="http://www.santostang.com/tag/python/" class="tag-cloud-link tag-link-5 tag-link-position-3" style="color:#cd09c1;font-size: 22pt;" aria-label="python (2個項目);">python</a> <a href="http://www.santostang.com/tag/selenium/" class="tag-cloud-link tag-link-6 tag-link-position-4" style="color:#39f9b0;font-size: 8pt;" aria-label="selenium (1個項目);">selenium</a> <a href="http://www.santostang.com/tag/%e7%bd%91%e7%bb%9c%e7%88%ac%e8%99%ab/" class="tag-cloud-link tag-link-8 tag-link-position-5" style="color:#eca091;font-size: 22pt;" aria-label="網絡爬蟲 (2個項目);">網絡爬蟲</a> <a href="http://www.santostang.com/tag/%e7%bd%91%e9%a1%b5%e7%88%ac%e8%99%ab/" class="tag-cloud-link tag-link-9 tag-link-position-6" style="color:#369ed1;font-size: 8pt;" aria-label="網頁爬蟲 (1個項目);">網頁爬蟲</a> <a href="http://www.santostang.com/tag/%e8%a7%a3%e6%9e%90%e5%9c%b0%e5%9d%80/" class="tag-cloud-link tag-link-10 tag-link-position-7" style="color:#c781f8;font-size: 8pt;" aria-label="解析地址 (1個項目);">解析地址</a> </div> </aside> <aside class="widget clearfix"><h4 class="widget-title">友情鏈接</h4><ul class="widget-links"></ul> </aside></div></div> </div> <div class="footer_search visible-xs visible-sm"><form id="searchform" action="http://www.santostang.com"><div class="input-group"><input type="search" class="form-control" placeholder="搜索…" value="" name="s"><span class="input-group-btn"><button class="btn btn-default" type="submit"><i class="fas fa-search"></i></button></span></div></form> </div> <footer id="footer"><div class="copyright"><p><i class="far fa-copyright"></i> 2019 <b>唐松-數據科學版權所有</b></p><p>Powered by <b>WordPress</b>. Theme by <a href="https://tangjie.me/jiestyle-two" data-toggle="tooltip" data-placement="top" title="WordPress 主題模板" target="_blank"><b>JieStyle Two</b></a> | <a href="http://beian.miit.gov.cn" data-toggle="tooltip" data-placement="top" target="_blank"><b>粵ICP備19068356號</b></a> </p> </p></div><div style="display:none;">代碼在頁面底部，統計標識不會顯示，但不影響統計效果</div> </footer> <script type="text/javascript" src="http://www.santostang.com/wp-content/themes/SongStyle-Two/js/jquery.min.js"></script> <script type="text/javascript" src="http://www.santostang.com/wp-content/themes/SongStyle-Two/js/bootstrap.min.js"></script> <script type="text/javascript" src="http://www.santostang.com/wp-content/themes/SongStyle-Two/js/skel.min.js"></script> <script type="text/javascript" src="http://www.santostang.com/wp-content/themes/SongStyle-Two/js/util.min.js"></script> <script type="text/javascript" src="http://www.santostang.com/wp-content/themes/SongStyle-Two/js/nav.js"></script> <script type='text/javascript' src='http://www.santostang.com/wp-includes/js/jquery/jquery.js?ver=1.12.4'></script> <script type='text/javascript' src='http://www.santostang.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.1'></script> <script type='text/javascript' src='http://www.santostang.com/wp-content/plugins/captcha-bank/assets/global/plugins/custom/js/front-end-script.js?ver=4.8.17'></script> <script> $(function() {$('[data-toggle="tooltip"]').tooltip() }); </script> <script> (function(){var bp = document.createElement('script');var curProtocol = window.location.protocol.split(':')[0];if (curProtocol === 'https') {bp.src = 'https://zz.bdstatic.com/linksubmit/push.js';}else {bp.src = 'http://push.zhanzhang.baidu.com/push.js';}var s = document.getElementsByTagName("script")[0];s.parentNode.insertBefore(bp, s); })(); </script> <script> var _hmt = _hmt || []; (function() {var hm = document.createElement("script");hm.src = "https://hm.baidu.com/hm.js?752e310cec7906ba7afeb24cd7114c48";var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(hm, s); })(); </script> </body> </html>

使用說明

r.text:服務器響應的內容,會根據響應頭部的字符編碼進行解碼
r.encoding:服務器內容使用的文本編碼
r.status_code:響應狀態碼
- 200:請求成功
- 4xx:客戶端錯誤
- 5xx:服務器錯誤響應
r.content:字節方式的響應
r.json():Requests內置的JSON解碼器

requests的用法

1.URL參數

有時我們為了請求特定的數據,需要直接在URL中加入一些數據,并且這些數據為在一個問號后面

如點擊百度中的某條新聞,其鏈接為https://baijiahao.baidu.com/s?id=1719021671421019613&wfr=spider&for=pc,其URL中的?后面會跟著一些屬性值

#示例,訪問http://httpbin.org/get？key1=value1&&key2=value2 import requests URL="http://httpbin.org/get" key_dict={"key1":"value1","key2":"value2"} r=requests.get(URL,params=key_dict) print("已經編碼的URL:",r.url) print("響應文本內容:",r.text) 已經編碼的URL: http://httpbin.org/get?key1=value1&key2=value2 響應文本內容: {"args": {"key1": "value1", "key2": "value2"}, "headers": {"Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Host": "httpbin.org", "User-Agent": "python-requests/2.25.1", "X-Amzn-Trace-Id": "Root=1-61b9c935-6c8a0f6c164028ad049e22f7"}, "origin": "115.156.142.199", "url": "http://httpbin.org/get?key1=value1&key2=value2" }

2.定制請求頭部

請求頭部Headers提供了關于請求,響應或其他發送實體的信息

如何查看請求頭:

瀏覽器點擊檢查

查看Network

查看Headers

一般只需添加瀏覽器信息即可

即user-agent

import requests URL="http://www.santostang.com/" headers={"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"} r=requests.get(URL,headers=headers) print("響應狀態:",r.status_code) 響應狀態: 200

3.發送Post請求

有時訪問網站需要提交一些表單形式的數據,這是需要發送POST請求

如果要發送POST請求,只需簡單傳遞一個字典數據給Requests中data參數

#示例 import requests URL="http://httpbin.org/post" key_dict={"key1":"value1","key2":"value2"} r=requests.post(URL,data=key_dict) print("已經編碼的URL:",r.url) print("響應文本內容:",r.text) 已經編碼的URL: http://httpbin.org/post 響應文本內容: {"args": {}, "data": "", "files": {}, "form": {"key1": "value1", "key2": "value2"}, "headers": {"Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "23", "Content-Type": "application/x-www-form-urlencoded", "Host": "httpbin.org", "User-Agent": "python-requests/2.25.1", "X-Amzn-Trace-Id": "Root=1-61b9cd28-16dce6c31b3e09c45cf7ef91"}, "json": null, "origin": "115.156.142.199", "url": "http://httpbin.org/post" }

4.超時處理

有時爬蟲會遇到服務器長時間沒有響應,這時候會造成爬蟲程序一直等待

這是可以通過Requests中的timeout參數設置定時器,如果定時器觸發后沒有響應,就會返回異常

import requests URL="http://www.santostang.com/" r=requests.get(URL,timeout=0.001) ---------------------------------------------------------------------------timeout Traceback (most recent call last)D:\ProgramData\Anaconda\lib\site-packages\urllib3\connection.py in _new_conn(self)168 try: --> 169 conn = connection.create_connection(170 (self._dns_host, self.port), self.timeout, **extra_kwD:\ProgramData\Anaconda\lib\site-packages\urllib3\util\connection.py in create_connection(address, timeout, source_address, socket_options)95 if err is not None: ---> 96 raise err97 D:\ProgramData\Anaconda\lib\site-packages\urllib3\util\connection.py in create_connection(address, timeout, source_address, socket_options)85 sock.bind(source_address) ---> 86 sock.connect(sa)87 return socktimeout: timed outDuring handling of the above exception, another exception occurred:ConnectTimeoutError Traceback (most recent call last)D:\ProgramData\Anaconda\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)698 # Make the request on the httplib connection object. --> 699 httplib_response = self._make_request(700 conn,D:\ProgramData\Anaconda\lib\site-packages\urllib3\connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)393 else: --> 394 conn.request(method, url, **httplib_request_kw)395 D:\ProgramData\Anaconda\lib\site-packages\urllib3\connection.py in request(self, method, url, body, headers)233 headers["User-Agent"] = _get_default_user_agent() --> 234 super(HTTPConnection, self).request(method, url, body=body, headers=headers)235 D:\ProgramData\Anaconda\lib\http\client.py in request(self, method, url, body, headers, encode_chunked)1254 """Send a complete request to the server.""" -> 1255 self._send_request(method, url, body, headers, encode_chunked)1256 D:\ProgramData\Anaconda\lib\http\client.py in _send_request(self, method, url, body, headers, encode_chunked)1300 body = _encode(body, 'body') -> 1301 self.endheaders(body, encode_chunked=encode_chunked)1302 D:\ProgramData\Anaconda\lib\http\client.py in endheaders(self, message_body, encode_chunked)1249 raise CannotSendHeader() -> 1250 self._send_output(message_body, encode_chunked=encode_chunked)1251 D:\ProgramData\Anaconda\lib\http\client.py in _send_output(self, message_body, encode_chunked)1009 del self._buffer[:] -> 1010 self.send(msg)1011 D:\ProgramData\Anaconda\lib\http\client.py in send(self, data)949 if self.auto_open: --> 950 self.connect()951 else:D:\ProgramData\Anaconda\lib\site-packages\urllib3\connection.py in connect(self)199 def connect(self): --> 200 conn = self._new_conn()201 self._prepare_conn(conn)D:\ProgramData\Anaconda\lib\site-packages\urllib3\connection.py in _new_conn(self)173 except SocketTimeout: --> 174 raise ConnectTimeoutError(175 self,ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x000002190B8302E0>, 'Connection to www.santostang.com timed out. (connect timeout=0.001)')During handling of the above exception, another exception occurred:MaxRetryError Traceback (most recent call last)D:\ProgramData\Anaconda\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)438 if not chunked: --> 439 resp = conn.urlopen(440 method=request.method,D:\ProgramData\Anaconda\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)754 --> 755 retries = retries.increment(756 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]D:\ProgramData\Anaconda\lib\site-packages\urllib3\util\retry.py in increment(self, method, url, response, error, _pool, _stacktrace)573 if new_retry.is_exhausted(): --> 574 raise MaxRetryError(_pool, url, error or ResponseError(cause))575 MaxRetryError: HTTPConnectionPool(host='www.santostang.com', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x000002190B8302E0>, 'Connection to www.santostang.com timed out. (connect timeout=0.001)'))During handling of the above exception, another exception occurred:ConnectTimeout Traceback (most recent call last)<ipython-input-15-c236f0bf58c5> in <module>1 import requests2 URL="http://www.santostang.com/" ----> 3 r=requests.get(URL,timeout=0.001)D:\ProgramData\Anaconda\lib\site-packages\requests\api.py in get(url, params, **kwargs)74 75 kwargs.setdefault('allow_redirects', True) ---> 76 return request('get', url, params=params, **kwargs)77 78 D:\ProgramData\Anaconda\lib\site-packages\requests\api.py in request(method, url, **kwargs)59 # cases, and look like a memory leak in others.60 with sessions.Session() as session: ---> 61 return session.request(method=method, url=url, **kwargs)62 63 D:\ProgramData\Anaconda\lib\site-packages\requests\sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)540 }541 send_kwargs.update(settings) --> 542 resp = self.send(prep, **send_kwargs)543 544 return respD:\ProgramData\Anaconda\lib\site-packages\requests\sessions.py in send(self, request, **kwargs)653 654 # Send the request --> 655 r = adapter.send(request, **kwargs)656 657 # Total elapsed time of the request (approximately)D:\ProgramData\Anaconda\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)502 # TODO: Remove this in 3.0.0: see #2811503 if not isinstance(e.reason, NewConnectionError): --> 504 raise ConnectTimeout(e, request=request)505 506 if isinstance(e.reason, ResponseError):ConnectTimeout: HTTPConnectionPool(host='www.santostang.com', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x000002190B8302E0>, 'Connection to www.santostang.com timed out. (connect timeout=0.001)'))

實踐爬取豆瓣Top250頁面

我們觀察發現,網站一頁最多顯示25部電影,且其每個頁面的url為:https://movie.douban.com/top250?start=value

其中value為0,25,50,…

import requests url="https://movie.douban.com/top250?" key_dict={"start":0} headers={"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"} for i in range(10):key_dict["start"]=25*ir=requests.get(url,params=key_dict,headers=headers)path=str(i)+".txt"print(r.status_code)fp=open(path,"w",encoding="utf-8")fp.write(r.text)fp.close() 200 200 200 403 200 200 200 200 200 403

頁面內容:

#將電影名提取出來并保存為txt from bs4 import BeautifulSoup url="https://movie.douban.com/top250?" key_dict={"start":0} headers={"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"} movie_list=[] for i in range(10):key_dict["start"]=25*ir=requests.get(url,params=key_dict,headers=headers)print(r.status_code)soup=BeautifulSoup(r.text,"lxml")content=soup.find_all("div",class_="hd")for each in content:movie=each.a.span.text.strip()movie_list.append(movie) print(movie_list) 200 200 200 200 200 200 200 200 200 200 ['肖申克的救贖', '霸王別姬', '阿甘正傳', '這個殺手不太冷', '泰坦尼克號', '美麗人生', '千與千尋', '辛德勒的名單', '盜夢空間', '忠犬八公的故事', '星際穿越', '楚門的世界', '海上鋼琴師', '三傻大鬧寶萊塢', '機器人總動員', '放牛班的春天', '無間道', '瘋狂動物城', '大話西游之大圣娶親', '熔爐', '教父', '當幸福來敲門', '控方證人', '龍貓', '怦然心動', '觸不可及', '末代皇帝', '蝙蝠俠：黑暗騎士', '尋夢環游記', '活著', '指環王3：王者無敵', '哈利·波特與魔法石', '亂世佳人', '素媛', '飛屋環游記', '摔跤吧！爸爸', '何以為家', '哈爾的移動城堡', '十二怒漢', '我不是藥神', '少年派的奇幻漂流', '鬼子來了', '大話西游之月光寶盒', '天空之城', '天堂電影院', '貓鼠游戲', '聞香識女人', '指環王2：雙塔奇兵', '羅馬假日', '鋼琴家', '讓子彈飛', '指環王1：護戒使者', '辯護人', '大鬧天宮', '教父2', '黑客帝國', '獅子王', '死亡詩社', '海蒂和爺爺', '搏擊俱樂部', '綠皮書', '飲食男女', '美麗心靈', '竊聽風暴', '本杰明·巴頓奇事', '情書', '兩桿大煙槍', '穿條紋睡衣的男孩', '西西里的美麗傳說', '看不見的客人', '飛越瘋人院', '拯救大兵瑞恩', '音樂之聲', '小鞋子', '阿凡達', '海豚灣', '致命魔術', '沉默的羔羊', '哈利·波特與死亡圣器(下)', '美國往事', '禁閉島', '蝴蝶效應', '布達佩斯大飯店', '心靈捕手', '低俗小說', '春光乍泄', '摩登時代', '七宗罪', '喜劇之王', '致命ID', '被嫌棄的松子的一生', '殺人回憶', '加勒比海盜', '紅辣椒', '狩獵', '剪刀手愛德華', '請以你的名字呼喚我', '勇敢的心', '7號房的禮物', '功夫', '超脫', '斷背山', '哈利·波特與阿茲卡班的囚徒', '天使愛美麗', '入殮師', '唐伯虎點秋香', '第六感', '幽靈公主', '重慶森林', '小森林夏秋篇', '陽光燦爛的日子', '愛在黎明破曉前', '一一', '蝙蝠俠：黑暗騎士崛起', '菊次郎的夏天', '哈利·波特與密室', '消失的愛人', '超能陸戰隊', '無人知曉', '小森林冬春篇', '完美的世界', '倩女幽魂', '愛在日落黃昏時', '側耳傾聽', '借東西的小人阿莉埃蒂', '甜蜜蜜', '螢火之森', '馴龍高手', '幸福終點站', '瑪麗和馬克思', '時空戀旅人', '大魚', '怪獸電力公司', '告白', '陽光姐妹淘', '射雕英雄傳之東成西就', '神偷奶爸', '傲慢與偏見', '教父3', '玩具總動員3', '釜山行', '恐怖直播', '一個叫歐維的男人決定去死', '哪吒鬧海', '被解救的姜戈', '血戰鋼鋸嶺', '未麻的部屋', '頭號玩家', '我是山姆', '寄生蟲', '七武士', '喜宴', '新世界', '電鋸驚魂', '哈利·波特與火焰杯', '模仿游戲', '黑客帝國3：矩陣革命', '花樣年華', '盧旺達飯店', '上帝之城', '三塊廣告牌', '風之谷', '瘋狂原始人', '你的名字。', '諜影重重3', '英雄本色', '頭腦特工隊', '達拉斯買家俱樂部', '縱橫四海', '心迷宮', '歲月神偷', '記憶碎片', '驚魂記', '忠犬八公物語', '海街日記', '荒蠻故事', '九品芝麻官', '爆裂鼓手', '貧民窟的百萬富翁', '真愛至上', '東邪西毒', '綠里奇跡', '小偷家族', '愛在午夜降臨前', '無敵破壞王', '黑天鵝', '冰川時代', '你看起來好像很好吃', '瘋狂的石頭', '螢火蟲之墓', '色，戒', '雨人', '雨中曲', '恐怖游輪', '戀戀筆記本', '魔女宅急便', '2001太空漫游', '城市之光', '可可西里', '虎口脫險', '人工智能', '二十二', '遺愿清單', '初戀這件小事', '海邊的曼徹斯特', '大佛普拉斯', '奇跡男孩', '羅生門', '終結者2：審判日', '牯嶺街少年殺人事件', '房間', '無間道2', '源代碼', '青蛇', '東京教父', '新龍門客棧', '瘋狂的麥克斯4：狂暴之路', '魂斷藍橋', '波西米亞狂想曲', '無恥混蛋', '步履不停', '血鉆', '茶館', '彗星來的那一夜', '千鈞一發', '戰爭之王', '燃情歲月', '黑客帝國2：重裝上陣', '諜影重重2', '崖上的波妞', '背靠背，臉對臉', '海洋', '小丑', '阿飛正傳', '穿越時空的少女', '諜影重重', '地球上的星星', '香水', '再次出發之紐約遇見你', '完美陌生人', '我愛你', '愛樂之城', '朗讀者', '火星救援', '聚焦', '小蘿莉的猴神大叔', '驢得水', '浪潮', '猜火車', '千年女優']

總結

以上是生活随笔為你收集整理的爬虫之静态页面抓取的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

上一篇：文字输入限制_从拼音输入法的兴起看汉字文
下一篇：《计算机应用基础》18春作业,【北语网院