當我使用scrapy想進行模擬登陸, 然后抓取首頁的問題和答案時, 一直顯示重定向問題 pythonfrom scrapy.contrib.spiders import CrawlSpider, Rulefrom scrapy.selector import Selectorfrom scrapy.contrib.linkextractors.sgml import SgmlLinkExtractorfrom scrapy.http import Request, FormRequestfrom zhihu.items import ZhihuItemclass ZhihuSipder(CrawlSpider) :

name = "zhihu"

allowed_domains = ["www.zhihu.com"]

start_urls = [

"http://www.zhihu.com"

]

rules = (

Rule(SgmlLinkExtractor(allow = r'http://www\.zhihu\.com/question/\d+'), callback = 'parse_page'),

)

def start_requests(self):

return [Request("https://www.zhihu.com/login", callback = self.post_login)]

#FormRequeset出問題了

def post_login(self, response):

print 'Preparing login'

xsrf = Selector(response).xpath('//input[@name="_xsrf"]/@value').extract()[0]

print xsrf

##############

return [FormRequest.from_response(response,

#"http://www.zhihu.com/login",

formdata = {

'_xsrf': xsrf,

'email': '@qq.com',

'password': '123456',

'rememberme': 'y',

},

callback = self.parse_page

)]

def parse_page(self, response):

problem = Selector(response)

item = ZhihuItem()

item['url'] = response.url

item['title'] = problem.xpath('//h2[@class="zm-item-title zm-editable-content"]/text()').extract()

item['description'] = problem.xpath('//div[@class="zm-editable-content"]/text()').extract()

item['answer']= problem.xpath('//div[@class=" zm-editable-content clearfix"]/text()').extract()

return item 使用命令運行爬蟲, 可以正確打印xsrf, 但無法成功登陸 $ scrapy crawl zhihu錯誤結果如下2014-12-18 14:45:11+0800 [zhihu] INFO: Spider opened2014-12-18 14:45:11+0800 [zhihu] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)2014-12-18 14:45:11+0800 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:60232014-12-18 14:45:11+0800 [scrapy] DEBUG: Web service listening on 127.0.0.1:60802014-12-18 14:45:11+0800 [zhihu] DEBUG: Redirecting (301) to from 2014-12-18 14:45:11+0800 [zhihu] DEBUG: Redirecting (302) to from 2014-12-18 14:45:12+0800 [zhihu] DEBUG: Crawled (200) (referer: None)Preparing logind117e46de0dcc5e8ee2f0c7031fcafe92014-12-18 14:45:12+0800 [zhihu] DEBUG: Redirecting (302) to from 2014-12-18 14:45:12+0800 [zhihu] DEBUG: Filtered duplicate request: - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)2014-12-18 14:45:12+0800 [zhihu] INFO: Closing spider (finished)2014-12-18 14:45:12+0800 [zhihu] INFO: Dumping Scrapy stats: 希望得到解答, 為什么不能成功登陸, 非常疑惑, 非常感謝 , 上面其實已經成功登錄, 后來自己測試過了, 只不過沒有調用一個抓取頁面url的函數, 實際上已經登錄了， 在 parse_page 方法里面添加一句 print response.body 把返回的 body 復制到一個 html 文件中， 用瀏覽器打開，就知道了。

總結

以上是生活随笔為你收集整理的python模拟登陆遇到重定向_python - scrapy模拟登陆知乎出现重定向无法登陆问题的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。