2019獨角獸企業重金招聘Python工程師標準>>>

最近幾天在看平凡歲月 感覺挺不錯的一部生活劇，想看看豆瓣

對該據評論，使用scrapy爬取，基于python2.7進行實現

1. 創建scrapy項目

scrapy startproject doubanmovie

2.創建爬蟲

scrapy genspider -t crawl dbm "movie.douban.com"

需要注意的是 爬蟲名稱不能和項目名稱重復

3. 編寫item

4.編寫爬蟲（spiders-->dbm.py）

# -*- coding: utf-8 -*- import scrapyfrom scrapy.linkextractors import LinkExtractorfrom scrapy.spiders import CrawlSpider, Rulefrom doubanmovie.items import DoubanmovieItemclass DbmSpider(CrawlSpider):name = 'dbm'allowed_domains = ['movie.douban.com']start_urls = ['https://movie.douban.com/subject/26795042/reviews?start=0']rules = (Rule(LinkExtractor(allow=r'start=\d+'), follow=True),Rule(LinkExtractor(allow=r'https://movie.douban.com/review/\d+/'), callback='parse_item', follow=False),)def parse_item(self, response):item = DoubanmovieItem()item['username'] = response.xpath("//header[@class='main-hd']/a/span/text()").extract()[0]title = response.xpath("//span[@property='v:summary']/text()").extract()[0]if len(title):item['title'] = titleelse:item['title'] = 'NULL'content = response.xpath("//div[@class='review-content clearfix']/p/text()").extract()if len(content):item['content'] = ''.join(content).strip()else:content = response.xpath("//div[@class='review-content clearfix']/text()").extract()if len(content):item['content'] = ''.join(content).strip()else:item['content'] = 'NULL'item['url'] = response.urlyield item

5.編寫管道文件(該案例使用json保存) doubanmovie->pipelines.py

import json class YouyuanPipeline(object):def __init__(self):self.filename = open('youyuan.json','w')def process_item(self, item, spider):content = json.dumps(dict(item),ensure_ascii=False) + ',\n'self.filename.write(content.encode('utf-8'))return itemdef close_spider(self,spider):self.filename.close()

6.修改setting文件

USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.54 Safari/536.5' DOWNLOAD_DELAY = 3 ITEM_PIPELINES = {'doubanmovie.pipelines.DoubanmoviePipeline': 300, }

7- 執行爬蟲

scrapy crawl dbm

轉載于:https://my.oschina.net/u/3273360/blog/1579179

總結

以上是生活随笔為你收集整理的使用scrapy框架爬取豆瓣影评的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。