github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,746 Commits 1 Branch 0 Tags
208202b9620925a92d806b3bb53dbff451ae001f
Commit Graph

1253 Commits

Author SHA1 Message Date
Mike Fährmann
208202b962 [tumblr] improve error handling (#297) 
In some cases Tumblr's API responds with an HTML document.
Trying to decode it as JSON would raise an uncaught exception.
 2019-06-04 14:02:17 +02:00
Mike Fährmann
c08c340178 [directlink] make pattern case insensitive (fixes #296) 2019-06-03 10:56:14 +02:00
Mike Fährmann
95b4a53b9c [keenspot] improve pagination (#223) 
The old code would skip the last comic page for some series.
 2019-06-02 22:12:21 +02:00
Mike Fährmann
731c7cbd5b [keenspot] support all comics and "random" access (#223) 2019-06-01 20:48:13 +02:00
Mike Fährmann
6a34f4b0c1 skip tests on read timeouts; print list of skipped tests 2019-06-01 20:47:31 +02:00
Mike Fährmann
1c36e65e9b [exhentai] choose site version depending on input URL (#278) 
Use e-hentai.org as root and cookiedomain if the input URL is from
e-hentai (or g.e-hentai), use exhentai.org otherwise.
 2019-05-31 15:34:39 +02:00
Mike Fährmann
6da3e21237 [downloader:ytdl] provide 'filename' metadata (closes #291) 2019-05-31 14:56:45 +02:00
Mike Fährmann
d33f5a7423 [wallhaven] rewrite 
- use API
- remove login support, add 'api-key' option
- remove support for "alpha" subdomain - alpha.wallhaven.cc used numeric
  IDs that can't be translated to the new ID system
- support direct links to wallpapers
 2019-05-31 14:53:02 +02:00
Mike Fährmann
5499934ae2 [ngomik] fix extraction 2019-05-30 20:18:36 +02:00
Mike Fährmann
f1893b2b5b [deviantart] add 'folders' option (#276) 2019-05-30 17:28:12 +02:00
Mike Fährmann
c849574def [keenspot] add comic extractor (#223) 
Doesn't work for
- http://brawlinthefamily.keenspot.com/
- http://flipside.keenspot.com/
- http://lastblood.keenspot.com/
- http://mysticrevolution.keenspot.com/
- http://porcelain.keenspot.com/
- http://twokinds.keenspot.com/
yet, because of custom layouts.
 2019-05-28 21:34:38 +02:00
Mike Fährmann
8bd5a19515 [hentainexus] add '_extractor' data 2019-05-28 00:20:01 +02:00
Mike Fährmann
2a085a5e96 [sankakucomplex] fix 'date' values (#258) 2019-05-28 00:18:58 +02:00
Mike Fährmann
bcd1801aa8 [sankakucomplex] add 'tag' extractor (#258) 2019-05-27 23:57:44 +02:00
Mike Fährmann
74c2415138 [sankakucomplex] move article extractor to its own module (#258) 2019-05-27 23:49:23 +02:00
Mike Fährmann
4465a3ea68 [kissmanga][readcomiconline] add 'captcha' option (#279) 
to configure how to handle CAPTCHA page redirects:
- either interactively wait for the user to solve the CAPTCHA
- or raise StopExtraction like before
 2019-05-27 22:24:48 +02:00
Mike Fährmann
1e3e15c4f3 [sankaku] add article extractor (#258) 2019-05-26 17:42:36 +02:00
Mike Fährmann
48233f00c0 [readcomiconline] detect 'AreYouHuman' redirects (#279) 2019-05-26 15:58:37 +02:00
Mike Fährmann
1cde38110d [livedoor] return 'date' as datetime object 2019-05-25 23:45:56 +02:00
Mike Fährmann
e88824e1a7 [livedoor] fix adjustments for https:// URLs 2019-05-25 23:45:22 +02:00
Mike Fährmann
b3e4664715 [hentainexus] fix extraction 2019-05-25 22:35:04 +02:00
Mike Fährmann
399e8e965a also update urllib3's cipher list for versions >= 1.25 2019-05-21 23:02:20 +02:00
Mike Fährmann
f837ea98cb [deviantart] don't call 'extend()' on folders (fixes #271) 2019-05-20 16:24:13 +02:00
Mike Fährmann
bb32a2d490 [patreon] use file extensions from original filenames (#268) 2019-05-20 15:46:59 +02:00
Mike Fährmann
efa805c5d7 [sankaku] update pagination end condition (fixes #265) 
Pagination over popular listings (`date:...+order:popular") never
terminates, not even on the site itself, and at some point returns the
same results over and over again.
 2019-05-20 15:46:06 +02:00
Mike Fährmann
a4ba34c835 [booru] prevent crash when no tags are present (#259) 2019-05-17 19:32:53 +02:00
Mike Fährmann
ca3bad1779 [patreon] small fixes and adjustments (#226) 
- fix datetime parsing
- rename 'user' to 'creator'
- convert 'id' to integer
- improve tests
 2019-05-17 19:32:41 +02:00
Leonardo Taccari
fb09dd962a [instagram] Fix extraction after `rhx_gis' field removal 2019-05-17 18:17:42 +02:00
Mike Fährmann
7a14aaed7d [luscious] fix extraction 2019-05-17 10:48:47 +02:00
Mike Fährmann
e82cadac61 [patreon] add extractors (#226) 2019-05-17 10:47:22 +02:00
Mike Fährmann
4891f4a328 [hentainexus] add search extractor (#256) 2019-05-16 23:55:30 +02:00
Mike Fährmann
c02f12ce2f avoid Cloudflare CAPTCHAs for OpenSSL < 1.1.1 
see https://github.com/Anorov/cloudflare-scrape/pull/242
 2019-05-15 12:25:20 +02:00
Mike Fährmann
0b4be57a10 [sankaku] fix error when no tags available (closes #259) 
[ci skip]
 2019-05-14 23:40:07 +02:00
Mike Fährmann
9890bfdf23 [flickr] improve code and metadata 
- simplify pagination
- add more metadata and slightly change its structure
  - convert suitable values to int or list
  - move keys from ["photo"] to the base level
- proper video support (#246)
- rename method and variable names to better fit with other extractors
 2019-05-14 22:10:50 +02:00
Mike Fährmann
aa8e366b90 [luscious] fix tag extraction 2019-05-14 17:35:52 +02:00
Mike Fährmann
ba8eb1ffec [hentainexus] add gallery extractor (#256) 2019-05-12 23:59:41 +02:00
Mike Fährmann
b1db194c14 [reactor] update and improve 
- split 'tags' into a list
- parse 'date' into a datetime object
- fix webm/mp4 URLs
 2019-05-09 23:24:49 +02:00
Mike Fährmann
b0e85a42e3 apply workaround from 4736912 in parse_datetime() itself 2019-05-09 21:53:17 +02:00
Mike Fährmann
8de5866fd2 [twitter] replace unit test URLs 
https://twitter.com/PicturesEarth was deleted
 2019-05-09 10:17:55 +02:00
Mike Fährmann
74c7304c6b [newgrounds] extract 'date', 'favorites', and 'score' 2019-05-08 18:09:17 +02:00
Mike Fährmann
4736912d4e [pixiv] work around strptime limitations in Python < 3.7 
"%z" doesn't allow a colon separator in older Python versions:
    - "+0900" is OK
    - "+09:00" raises an exception
 2019-05-08 18:08:03 +02:00
Mike Fährmann
1f7fa9dc8e [exhentai] update data extraction code 
- parse 'date' to datetime object
- use 'text.extract_from()'
 2019-05-08 15:44:29 +02:00
Mike Fährmann
80fdb11508 [pixiv] add 'date' metadata field (closes #248) 2019-05-08 15:43:59 +02:00
Mike Fährmann
049e9fd6ce [twitter] fix pagination end condition 
Some timelines would cause an endless loop because 'has_more_items' is
always True, even if it would return the same list of tweets over and
over again.
 2019-05-08 15:43:59 +02:00
Mike Fährmann
51e0e92429 [deviantart] fix GIF downloads (#242) 
The "original" download URL for GIF animations is only a preview version
of the original file.
 2019-05-08 15:43:43 +02:00
Leonardo Taccari
f347d2d152 [instagram] Fix for missing edge_media_to_comment' field and add date' metadata (#250) 
* [instagram] Remove no longer always present `comments' field

`edge_media_to_comment' is no longer always present in the response
(also for the same media sometimes is present and sometimes is not present).

* [instagram] Add `date' metadata
 2019-05-08 15:42:58 +02:00
Mike Fährmann
5fd94c6b83 import urllib3 from requests.packages 2019-05-04 22:28:07 +02:00
Mike Fährmann
35f343206c update default SSL cipher list in urllib3 < 1.25 
Cloudflare now also checks the client's SSL/TLS cipher capabilities and
produces a 403: Forbidden response with CAPTCHA if they are insufficient.

This commit replaces the default cipher list in urllib3 < 1.25 with the
one from 1.25 (1), which doesn't cause problems as long as the client
platform actually supports these ciphers. On some platforms (tested with
Python 3.4 on Linux and Python 3.7 on an outdated Windows 7 VM) it is
necessary to install pyOpenSSL to get everything to work.

Explicitly setting a minimum/maximum version for urllib3 is also no
longer necessary and installing gallery-dl will therefore not pull a
incompatible urllib3 version (#229)

Fixes the "403: Forbidden" error on Artstation (#227)

(1) 0cedb3b0f1
 2019-05-03 22:40:04 +02:00
Mike Fährmann
fc5e4f2b21 [hitomi] simplify data extraction code 2019-05-01 11:14:21 +02:00
Mike Fährmann
2756cc8dde [hitomi] set Referer header (fixes #239) 2019-05-01 10:56:00 +02:00