六 11 六月 2016 | tags: Scrapy, Python, -- (permalink)

Scrapy 笔记

Scrapy shell

更改headers调试

scrapy shell
...
...
>>> from scrapy import Request
>>> request = Request('http://www.example.com', headers={"User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"})
>>> fetch(request)

或者

scrapy shell -s USER_AGENT='Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0' 'http://www.example.com'

Request Headers

setting.py 更改 USER_AGENT 或 DEFAULT_REQUEST_HEADERS

scrapy.http.Request 手动修改参数 headers