github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,839 Commits 1 Branch 0 Tags
7ca3bf7cb0e9622d491bee5483efe4e2f6bed6d7
Commit Graph

2618 Commits

Author SHA1 Message Date
Mike Fährmann
5bcf28de93 add a 'extractor.modules' option 2020-10-25 03:05:10 +01:00
Mike Fährmann
18213dc5ba release version 1.15.2 2020-10-24 18:57:29 +02:00
Mike Fährmann
de4a1e45c9 improve 'generate_csrf_token()' 
no need to use hashlib.md5()
 2020-10-24 02:56:40 +02:00
Mike Fährmann
b788712844 [fallenangels] fix extraction of '.5' chapters 2020-10-23 16:56:08 +02:00
Mike Fährmann
28d8541cb3 [mangafox] ensure download URLs have a scheme 2020-10-23 02:45:15 +02:00
Mike Fährmann
8e3a324c91 [mangakakalot] ignore "Go Home" buttons in chapter pages 2020-10-23 02:33:35 +02:00
Mike Fährmann
c14c5d82d6 [newgrounds] use generator for fallback URLs 2020-10-23 00:39:19 +02:00
Mike Fährmann
a09f42f6b3 improve filename_from_url() performance 
Manually extracting the part between the last '/' and '?' instead of
relying on the standard libraries' 'urllib.parse.urlsplit()' increases
performance by ~400%.

urlsplit() : 3.64 secs per 1.000.000 iterations
partition(): 0.87 secs per 1.000.000 iterations
 2020-10-23 00:14:06 +02:00
Mike Fährmann
968d3e8465 remove '&' from URL patterns 
'/?&#' -> '/?#' and '?&#' -> '?#'

According to https://www.ietf.org/rfc/rfc3986.txt, URLs are
"organized hierarchically" by using "the slash ("/"), question
mark ("?"), and number sign ("#") characters to delimit components"
 2020-10-22 23:31:25 +02:00
Mike Fährmann
1686dc1757 [twitter] support media from Cards (#1005, #937) 
Can be enabled with 'extractor.twitter.cards', but for now disabled by
default because cards can redirect to rather large videos from YouTube
or Twitch.
 2020-10-22 21:33:53 +02:00
Mike Fährmann
ffd38215a4 [hitomi] fix image URLs and URL pattern 
- non-webp files are now hosted on [a-c]b.hitomi.la
- removed ampersand from invalid slug characters
 2020-10-22 15:15:34 +02:00
Mike Fährmann
286718950c [mangahere] ensure download URLs have a scheme (fixes #1070) 2020-10-17 22:43:59 +02:00
Mike Fährmann
76dfa11a65 [reddit] add 'date' metadata field (closes #1068) 2020-10-16 15:48:04 +02:00
Mike Fährmann
3f2ba629ea [newgrounds] provide fallback URLs for video downloads (#1042) 2020-10-16 01:16:12 +02:00
Mike Fährmann
a3ca2f6080 update fallback URL handling 
remove Message.Urllist and use a '_fallback' field inside a kwdict
 2020-10-16 01:09:55 +02:00
Mike Fährmann
43dab3a228 [mangadex] unescape more metadata fields (fixes #1066) 
like 'manga', 'author', 'artist', etc.
 2020-10-16 00:41:15 +02:00
Mike Fährmann
ec61696316 add 't' format string conversion (closes #1065) 
to Trim whitespace from the beginning and end of strings.
Example: '{field!t}' becomes 'foo' for 'field' == "  \nfoo\t\r"
 2020-10-16 00:37:22 +02:00
Mike Fährmann
5565025221 [xhamster] fix user profile extraction 2020-10-15 18:57:35 +02:00
Mike Fährmann
07432d6262 [seiga] fix flake8 and cookie test (#1063) 2020-10-15 15:37:58 +02:00
Mike Fährmann
b8daabc3ca [pinterest] implement login support (closes #1055) 
being logged allows access to secret/protected boards
 2020-10-15 15:14:18 +02:00
Mike Fährmann
1b1cf01d0d add a general 'generate_csrf_token()' function 2020-10-15 15:14:18 +02:00
Mike Fährmann
7a0ba370d1 [gelbooru] rewrite mp4 video URLs (fixes #1048) 2020-10-15 15:14:18 +02:00
Mike Fährmann
6491db3eaf [blogger] handle URLs with specified width/height (closes #1061) 
get highest quality for images with
/wXXX-hXXX/ instead of the usual /sXXX/
 2020-10-15 15:14:18 +02:00
Mike Fährmann
783e0af26d [hentaifoundry] update and simplify 2020-10-15 15:14:17 +02:00
Mike Fährmann
5b844a72b7 [newgrounds] handle embeds without scheme (#1033) 2020-10-15 15:13:54 +02:00
kurumigi
7e0e872f4f [seiga] Add metadata for single image downloads (#1063) 
* [seiga] Support image metadata.

* [seiga] Update test data.

* [seiga] Fix cookie check.

* [test_cookies] [seiga] Fit test_cookies.py to the last commit.
 2020-10-15 15:13:27 +02:00
Zanny
3ec60e894a [weasyl] api-key authentication (#1057) 
* [weasyl] support api keys

* [weasyl] document api-key authentication

* [weasyl] usernames can contain ~
 2020-10-15 15:12:09 +02:00
Mike Fährmann
35056a07d1 release version 1.15.1 2020-10-11 18:44:46 +02:00
Mike Fährmann
844793847c update extractor test results 2020-10-11 18:15:41 +02:00
Mike Fährmann
ddd6840509 [behance] fix 'collection' extraction 2020-10-11 18:15:41 +02:00
Mike Fährmann
c5e3971b18 [newgrounds] extract image embeds (closes #1033) 2020-10-11 18:15:40 +02:00
dawidsowa
43b156fb40 [reactor] match URLs without subdomain (#1053) 2020-10-11 18:15:06 +02:00
Mike Fährmann
fd20093c96 allow blacklist/whitelist to be empty lists/strings (#1051) 2020-10-08 14:55:21 +02:00
Mike Fährmann
3ebb174f2c add missing extractor info when spawning new ones (fixes #1051) 
Not having this information causes the blacklist/whitelist logic to
trigger and prevents things from functioning as intended when using
default settings.

Fixes issues for 8muses, deviantart, exhentai, and mangoxo.
 2020-10-08 14:34:53 +02:00
Mike Fährmann
f9c1684af7 [newgrounds] restore original video URLs (#1042) 2020-10-07 22:53:53 +02:00
Mike Fährmann
73373c06ec [weibo] handle posts with more than 9 images (closes #926) 
Responses from '/api/container/getIndex' don't list more than
9 images per 'status' object, but the embedded JSON from a
'/detail/<ID>' page does.
 2020-10-06 18:16:08 +02:00
Mike Fährmann
dd1e545597 [hentaifoundry] rename GalleryExtractor to PicturesExtractor 2020-10-04 22:53:23 +02:00
Mike Fährmann
c874071f5a [kissmanga] remove module 2020-10-04 22:46:41 +02:00
Mike Fährmann
93e04bf9a9 [500px] update query hashes 2020-10-03 19:25:28 +02:00
Mike Fährmann
844502cad5 update extractor test results 2020-10-03 19:24:19 +02:00
Mike Fährmann
fad7748b6b [xvideos] fix 'title' extraction 2020-10-01 22:04:14 +02:00
Mike Fährmann
5b927c15df [newgrounds] fix video extraction (closes #1042) 2020-10-01 20:14:16 +02:00
Mike Fährmann
bdc6c8f074 improve message for 'oauth:deviantart' etc (closes #989) 2020-09-29 21:25:24 +02:00
Mike Fährmann
430b6d6e2e [twitter] extend 'retweets' option (closes #1026) 
Setting 'retweets' to '"original"' will use metadata from the
original retweeted Tweets, and not from the Retweet entry.
 2020-09-28 23:03:35 +02:00
Mike Fährmann
b9bdd2c564 [hentaifoundry] add support for stories (closes #734) 2020-09-27 02:27:40 +02:00
Mike Fährmann
9a9d1924d8 [hentaicafe] add 'manga_id' metadata field (closes #1036) 
This field is only available when using a non-foolslide URL
like '/hc.fyi/9874' or '/hazuki-yuuto-summer-blues/'
 2020-09-26 14:34:48 +02:00
Mike Fährmann
cc4ac80302 [weasyl] add 'favorite' extractor (#1032) 2020-09-26 13:09:03 +02:00
Mike Fährmann
e9cc719497 [weasyl] update and simplify 
- simplify 'pattern' regexps
- parse 'posted_at' as 'date'
- use unaltered 'title' ({title!l:R /_/} to lowercase and replace spaces)
 2020-09-26 02:10:45 +02:00
Mike Fährmann
6514312126 [nijie] add 'include' option (closes #1018) 2020-09-25 18:18:35 +02:00
Mike Fährmann
0d43456323 [hentaifoundry] add 'include' option 2020-09-25 18:18:03 +02:00