github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,427 Commits 1 Branch 0 Tags
e80ee77d71d55f9e186927a5df43206c1b24e016
Commit Graph

1016 Commits

Author SHA1 Message Date
HRXN
e80ee77d71 tumblr.py: update regex for video (#133) 
There seems to be another sub-domain for videos, apparently..
Not just
`vt(.media).tumblr`
`vtt(media).tumblr`
But also
`ve(.media).tumblr`
 2018-12-09 09:07:46 +01:00
Mike Fährmann
9a98b6769d use extractor.request for API calls (#130) 
... at least for OAuth1.0 based APIs (flickr, smugmug, tumblr)
 2018-12-04 21:29:06 +01:00
Mike Fährmann
ad2cefda6b [tumblr] in case of exception use filename as 'hash' (#129) 
While a filename might not be a real 'hash', or comparable to what
tumbler usually provides, it is still better than an empty string.
At least as long as "alternatives" in format strings aren't implemented.
 2018-12-04 19:15:23 +01:00
Mike Fährmann
95636418ad [tumblr] catch exception for 'hash' extraction (fixes #129) 2018-12-02 19:48:09 +01:00
Mike Fährmann
40e30694f3 [pinterest] fix pin.it redirects 2018-12-02 19:38:50 +01:00
Mike Fährmann
770200888e [gfycat] use public API endpoint 2018-12-02 18:56:53 +01:00
Mike Fährmann
5d6e219fb2 [joyreactor] update tests 2018-11-28 14:52:19 +01:00
Mike Fährmann
c59f56fe7e [gfycat] fix extraction 
/cajax/get/<id> doesn't work anymore
 2018-11-28 13:26:21 +01:00
Mike Fährmann
ba56827f36 [newgrounds] add user-, video-, image-extractors (#119) 2018-11-27 15:44:53 +01:00
Mike Fährmann
15890930ea [mangafox] fix extraction 
use mobile version since desktop version is obfuscated
 2018-11-26 16:13:41 +01:00
Mike Fährmann
a4263fb253 [luscious] add extractor for search results (closes #127) 2018-11-25 18:57:51 +01:00
Mike Fährmann
a0ae156edc [pornreactor] add tag-, user-, post-extractors (#114) 2018-11-23 14:41:26 +01:00
Mike Fährmann
bacbc2e7bd [joyreactor] try to prevent JsonDecodeErrors (#114) 2018-11-23 14:32:37 +01:00
Mike Fährmann
503d42a1c2 [joyreactor] add tag-, user-, post-extractors (#114) 2018-11-23 09:25:02 +01:00
Mike Fährmann
59bb434ba5 [flickr] add ability to download all albums of a user 
for example with 'https://www.flickr.com/photos/shona_s/albums'
 2018-11-23 09:09:37 +01:00
Mike Fährmann
9e188f6a21 [4chan] support 4channel.org domain 2018-11-21 17:40:38 +01:00
Mike Fährmann
041bd501fc [hentaifoundry] unescape YII_CSRF_TOKEN value 
This fixes the POST requests to /site/filters
 2018-11-19 21:46:17 +01:00
Mike Fährmann
b828473aa3 retry HTTP requests for more exception classes 2018-11-19 15:49:13 +01:00
Mike Fährmann
c47482b110 smaller changes, missing docs, etc. 
- make 'netrc' extractor-specific
- rename 'downloader.enable' to 'enabled'
- document 'downloader.ytdl.format'
- consistent newlines in configuration.rst
 2018-11-16 18:18:07 +01:00
Mike Fährmann
7f6a0be982 adjust some tests 2018-11-15 22:50:04 +01:00
Mike Fährmann
baad7b0fa5 [twitter] unpack API responses when logged in (closes #123) 2018-11-14 11:49:35 +01:00
Mike Fährmann
3bdfc15be1 [pinterest] don't crash on pins without image info 2018-11-14 11:46:14 +01:00
Mike Fährmann
14ee6bf611 [behance] handle external URLs with youtube-dl 2018-11-13 15:10:23 +01:00
Mike Fährmann
36425122ff [artstation] handle external URLs with youtube-dl 2018-11-13 14:27:02 +01:00
Mike Fährmann
bd8670d925 [gfycat] extend URL pattern 2018-11-11 21:19:11 +01:00
Mike Fährmann
2fa28a2609 update default user-agent string (closes #122) 2018-11-11 10:07:10 +01:00
Mike Fährmann
7e2d6bcd62 [deviantart] fix original image downloads 2018-11-10 19:16:10 +01:00
Mike Fährmann
9e12e073ab [2chan] fix extraction 2018-11-10 19:15:21 +01:00
Mike Fährmann
966a9ca3a0 update test results 2018-11-10 19:14:54 +01:00
Mike Fährmann
fd8ed35591 [turboimagehost] fix extraction 2018-10-23 21:08:24 +02:00
Mike Fährmann
c69150f715 [imagefap] fix extraction 
also adds tags to gallery-metadata and converts suitable values to int
 2018-10-20 18:32:25 +02:00
Mike Fährmann
d1f3d32eec [fallenangels] unescape chapter titles 2018-10-20 18:31:26 +02:00
Mike Fährmann
c9861ca812 adjust message for status_code based exceptions 
from: 5xx HTTP Error: Reason
to  : 5xx: Reason

The "HTTP Error" part was in there to emulate Request's error messages
from response.raise_for_status(), but it reads a lot better without.
 2018-10-18 15:09:49 +02:00
Mike Fährmann
eb1c24b286 [imagebam] detect nonexistent galleries 2018-10-17 15:21:47 +02:00
Mike Fährmann
e1d306cc48 update unit test results 2018-10-13 16:54:30 +02:00
Mike Fährmann
8faf03ed84 [pixiv] use refresh_token based authentication 
The first login will still use username and password, but everything
afterwards will use the refresh_token obtained from that.

This will prevent pixiv from sending a "New login to pixiv" email every
time a new access_token is requested.
 2018-10-12 22:26:27 +02:00
Mike Fährmann
2221cf97ff implement 'update()' for caches 2018-10-12 22:18:29 +02:00
Mike Fährmann
d8492df51b [deviantart] extend functionality of 'original' option 2018-10-11 22:11:49 +02:00
Mike Fährmann
c00dce2adc [behance] enable 'categorytransfer' 2018-10-09 23:40:49 +02:00
Mike Fährmann
1532d1b690 fix 'range' tests and update a few test results 2018-10-08 23:53:58 +02:00
Mike Fährmann
4a348990f4 adjust value resolution for retries/timeout/verify options 
This change introduces 'extractor.*.retries/timeout/verify' options
as a general way to set these values for all HTTP requests.

'downloader.http.retries/timeout/verify' is a way to override these
options for file downloads only and will fall back to 'extractor.*.…*
values if they haven't been explicitly set.

Also: downloader classes now take an extractor object as first argument
instead of a requests.session.
 2018-10-07 21:13:39 +02:00
Mike Fährmann
f647f5d9c3 use 'verify' option for regular HTTP requests 2018-10-06 16:38:43 +02:00
Mike Fährmann
ca6ac4db6a fix 'content' tests 2018-10-05 21:10:33 +02:00
Mike Fährmann
188876d814 implement youtube-dl downloader module 
URLs starting with 'ytdl:' will now be handled by youtube-dl.
There is probably a lot to fix and improve, but the basic use case
works.

TODO:
- format selection and ytdl options in general
- better filename/path handling
- ytdl support for "unsupported URLs"
- ...
 2018-10-05 18:05:11 +02:00
Mike Fährmann
f4df6c2396 [pixiv] remove 'type' and 'page' query parameter handling 
The "new and improved" /member_illust.php and /bookmark.php listings
don't quite work with how things were.
 2018-10-03 16:08:36 +02:00
Mike Fährmann
d70db2d555 Revert "[komikcast] fix extraction" 
This reverts commit 5507f5ce2e.
 2018-10-02 20:38:42 +02:00
Mike Fährmann
d69db60e2a update unit test results 2018-10-02 20:37:46 +02:00
Mike Fährmann
f8b3b00249 [twitter] add experimental 'videos' option (#99) 
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl

There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
 2018-09-30 21:52:23 +02:00
Mike Fährmann
5507f5ce2e [komikcast] fix extraction 2018-09-29 16:37:30 +02:00
Mike Fährmann
8080071174 [flickr] improve album metadata (closes #109) 2018-09-29 16:21:55 +02:00