Add support for instagram.com user profiles and pages (#134)

* [instagram] Add extractor for instagram.com user profiles and pages

The extractor scrapes `instagram.com/<user>' timelines and
`instagram.com/p/<shortcode>' by mimicking the behaviour of a web
browser and extracting the sharedData JSON of the single pages.

Please note that this mean that for user timelines we also do an
extra request to the `instagram.com/p/<shortcode>' page but this
permit to have consistent (and all) information about the media
fetched.

The MD5 logic used for X-Instagram-GIS was documented in

 <https://stackoverflow.com/questions/49786980/>

* [instagram] Test for keywords, not url for GraphImage and GraphSidecar

URLs returned by instagram seems not stable so avoid testing for
them and instead test for keyword returned.

* [instagram] Improve test of InstagramProfilepageExtractor

Also check the count of media returned.

* [instagram] Several cleanup and improvements

- Change description, subcategories to generate a better description in
  docs/supportedsite.rst
- Remove not needed InstagramExtractor.__init__()
- Use text.parse_int() instead of directly using int() (the former is more
  robust)
- Use self.request().json() instead of using json.loads() the
  self.request().text()
- Add `pattern:' to check the URLs where we do not have a stable URLs.
  It seems that only the subdomain is not stable.

Thanks to @mikf!
This commit is contained in:
Leonardo Taccari
2018-12-09 12:52:14 +01:00
committed by Mike Fährmann
parent e80ee77d71
commit 2655a2ea02
2 changed files with 195 additions and 0 deletions

View File

@@ -45,6 +45,7 @@ modules = [
"imgbox",
"imgth",
"imgur",
"instagram",
"jaiminisbox",
"joyreactor",
"khinsider",