單獨(dú)爬蟲(chóng)配置 custom_settings = {'SOME_SETTING': 'some value',} 不同爬蟲(chóng)pipeline設(shè)置 custom_settings = {'ITEM_PIPELINES': {'video.pipelines.VideoPipeline': 301,} }

cookie設(shè)置

custom_settings = {'COOKIES_ENABLED':True, # 在配置文件settings中可以設(shè)置成False, 在這個(gè)spider中這樣設(shè)置就可以開(kāi)啟cookies了，其他的配置一樣適用}

?

settings/在settings同目錄下新建custom_settings.py
?

# -*- coding: utf-8 -*- custom_settings_for_spider1 = { 'LOG_LEVEL': 'INFO', 'DOWNLOAD_DELAY': 0, 'COOKIES_ENABLED': False, # enabled by default 'DOWNLOADER_MIDDLEWARES': { 'video_spider.middlewares.ProxiesMiddleware': 400, 'video_spider.middlewares.SeleniumMiddleware': 543, # 將scrapy默認(rèn)的user-agent中間件關(guān)閉 12 'scrapy.downloadmiddlewares.useragent.UserAgentMiddleware': None, }, }

在spider文件中引入custom_settings
?

import scrapy from scrapy import Request from scrapy.utils.project import get_project_settings from scrapy import signals from pydispatch import dispatcher # setting class ShanbaySpider(scrapy.Spider): name = 'shanbay' allowed_domains = ['shanbay.com'] start_urls = ['http://shanbay.com/'] custom_settings = custom_settings_for_spider1

?

總結(jié)

以上是生活随笔為你收集整理的scrapy custom_settings的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。