share extractor and downloader sessions

There was never any "good" reason for the strict separation
between extractors and downloaders. This change allows for
reduced resource usage (probably unnoticeable) and less lines
of code at the "cost" of tighter coupling.
This commit is contained in:
Mike Fährmann
2017-06-30 19:38:14 +02:00
parent 4414aefe97
commit 58e95a7487
9 changed files with 13 additions and 62 deletions

View File

@@ -27,7 +27,6 @@ class ImgchiliExtractor(Extractor):
page = self.request(self.url, encoding="utf-8").text
data = self.get_job_metadata(page)
yield Message.Version, 1
yield Message.Headers, self.session.headers
yield Message.Directory, data
for url, image in self.get_images(page):
data.update(image)