需求：

使用crawlSpider（全站）進行數據爬取

- 首頁: 崗位名稱，崗位類別

- 詳情頁：崗位職責

- 持久化存儲

代碼：

爬蟲文件：

from scrapy.linkextractors import LinkExtractor from scrapy.spiders import CrawlSpider, Rule from ..items import CrawlproItem,TenproItem_detailclass CrawSpider(CrawlSpider):name = 'craw'# allowed_domains = ['www.xxx.com']start_urls = ['https://hr.tencent.com/position.php?&start=0#a']# 首頁所有頁碼的連接提取器link1 = LinkExtractor(allow=r'&start=\d+#a')# 詳情頁連接提取器link2 = LinkExtractor(allow=r'position_detail.php\?id=\d+&keywords=&tid=0&lid=0$') # 問號轉義 rules = (Rule(link1, callback='parse_item', follow=True),Rule(link2, callback='parse_detail', follow=True),)def parse_item(self, response):# 崗位名稱和類別print('item',response)tr_list = response.xpath('//table[@class="tablelist"]/tr[@class="odd"] | //table[@class="tablelist"]/tr[@class="even"]')for tr in tr_list:job_name = tr.xpath('./td[1]/a/text()').extract_first()job_class = tr.xpath('./td[2]/text()').extract_first()# 實例化item類item = CrawlproItem()item['job_name'] = job_nameitem['job_class'] = job_classyield itemdef parse_detail(self, response):# 崗位職責desc = response.xpath('//ul[@class="squareli"]/li/text()').extract()desc = ''.join(desc)item = TenproItem_detail()item['desc'] = descyield item

items.py文件:

import scrapyclass CrawlproItem(scrapy.Item):job_name = scrapy.Field()job_class = scrapy.Field()class TenproItem_detail(scrapy.Item):desc = scrapy.Field()

管道文件pipelines.py:

class CrawlproPipeline(object):fp = Nonedef open_spider(self, spider):# 文件只打開一次self.fp = open('./tenxun.txt', 'w',encoding='utf-8')def process_item(self, item, spider):desc = None# 取出item中的值if item.__class__.__name__ == 'CrawlproItem':job_name = item["job_name"]job_class = item["job_class"]self.fp.write(f'{job_name}\n{job_class}\n\n')else:desc = item['desc']self.fp.write(desc)return item # 返回給下一個即將被執行的管道類def close_spider(self, spider):self.fp.close()

配置文件中注意開啟管道

?

轉載于:https://www.cnblogs.com/zwq-/p/10602093.html

總結

以上是生活随笔為你收集整理的利用Crawlspider爬取腾讯招聘数据(全站，深度)的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。