6.1Scrapyの準備
Scrapyがinstallされていなかったので
>conda install -c https://conda.anaconda.org/anaconda scrapy
とした。scrapyだけが」installされるのだろうとおもっていたら
新規installが18、アップデートが4個入ってきた。
attrs: 15.2.0-py35_0 anaconda 不明
automat: 0.5.0-py35_0 anaconda 不明
constantly: 15.1.0-py35_0 anaconda 不明
cssselect: 1.0.1-py35_0 anaconda 不明
hyperlink: 17.1.1-py35_0 anaconda 純粋なPythonの不変URLの実装を提供します。？
incremental: 16.10.1-py35_0 anaconda 不明
parsel: 1.2.0-py35_0 anaconda 不明
patch: 2.5.9-1 anaconda 不明
pyasn1-modules: 0.0.8-py35_0 anaconda 不明
pydispatcher: 2.0.5-py35_0 anaconda 不明
queuelib: 1.4.2-py35_0 anaconda 不明
scrapy: 1.3.3-py35_0 anaconda スクレイピングツール
service_identity: 17.0.0-py35_0 anaconda サービスID検証
twisted: 17.5.0-py35_0 anaconda 不明
w3lib: 1.17.0-py35_0 anaconda 不明
zope: 1.0-py35_0 anaconda Webアプリケーションフレームワーク
zope.interface: 4.4.2-py35_0 anaconda
anaconda: 2.4.1-np110py35_0 --> custom-py35_0 anaconda
conda: 4.3.24-py35_0 --> 4.3.25-py35_0 anaconda OSに依存しない、システムレベルのバイナリパッケージと環境マネージャ。
conda-env: 2.6.0-0 --> 2.6.0-0 anaconda 不明
spyder: 3.2.0-py35_0 --> 2.3.8-py35_1 anaconda IDE
installできたので
＞scrapy startproject nobel_winners
で実行すると本のとおりのフォルダー構造ができていた。
__init__.pyには何も記載なし
items.pyには
# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html
import scrapy
class NobelWinnersItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
pass
とありスクレイピングしたいitemを定義するファイル
middlewares.py
# -*- coding: utf-8 -*-
# Define here the models for your spider middleware
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/spider-middleware.html
from scrapy import signals
class NobelWinnersSpiderMiddleware(object):
# Not all methods need to be defined. If a method is not defined,
# scrapy acts as if the spider middleware does not modify the
# passed objects.
@classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
def process_spider_input(response, spider):
# Called for each response that goes through the spider
# middleware and into the spider.
# Should return None or raise an exception.
return None
def process_spider_output(response, result, spider):
# Called with the results returned from the Spider, after
# it has processed the response.
# Must return an iterable of Request, dict or Item objects.
for i in result:
yield i
def process_spider_exception(response, exception, spider):
# Called when a spider or process_spider_input() method
# (from other spider middleware) raises an exception.
# Should return either None or an iterable of Response, dict
# or Item objects.
pass
def process_start_requests(start_requests, spider):
# Called with the start requests of the spider, and works
# similarly to the process_spider_output() method, except
# that it doesn’t have a response associated.
# Must return only requests (not items).
for r in start_requests:
yield r
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)
spiderと接続を定義する？
pipelines.py
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
class NobelWinnersPipeline(object):
def process_item(self, item, spider):
return item
Item pipelineを定義するファイル
settings.py
# -*- coding: utf-8 -*-
# Scrapy settings for nobel_winners project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# http://doc.scrapy.org/en/latest/topics/settings.html
# http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
# http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'nobel_winners'
SPIDER_MODULES = ['nobel_winners.spiders']
NEWSPIDER_MODULE = 'nobel_winners.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'nobel_winners (+http://www.yourdomain.com)'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
# 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
# 'Accept-Language': 'en',
#}
# Enable or disable spider middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'nobel_winners.middlewares.NobelWinnersSpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'nobel_winners.middlewares.MyCustomDownloaderMiddleware': 543,
#}
# Enable or disable extensions
# See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
# 'nobel_winners.pipelines.NobelWinnersPipeline': 300,
#}
# Enable and configure the AutoThrottle extension (disabled by default)
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
プロジェクト設定ファイル
scrapy.cfg
# Automatically created by: scrapy startproject
#
# For more information about the [deploy] section see:
# https://scrapyd.readthedocs.org/en/latest/deploy.html
[settings]
default = nobel_winners.settings
[deploy]
#url = http://localhost:6800/
project = nobel_winners
これも設定ファイル
