github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,794 Commits 1 Branch 0 Tags
92b75bcdceb2555d0797e747ef3f6500c6508aeb
Commit Graph

2763 Commits

Author SHA1 Message Date
Mike Fährmann
882c614281 add album extractor for lolisafe/chibisafe instances 
- support bunkr.is (closes #2038)
- support zz.ht    (closes #2105)
 2021-12-21 19:24:17 +01:00
Mike Fährmann
d441888bfb [deviantart] adjust API endpoints 
Start all endpoints with a forward slash '/'
to be consistent with other API interfaces.
 2021-12-21 00:18:06 +01:00
Mike Fährmann
8f0cf0bf71 [deviantart] use '/browse/newest' for most-recent searches 
(#2096)
 2021-12-20 22:40:03 +01:00
Mike Fährmann
0bd7607da5 [tumblrgallery] improve 'id' extraction (#2115) 2021-12-19 05:46:02 +01:00
Mike Fährmann
0d02a7861e [tumblrgallery] fix extraction (closes #2112) 2021-12-17 19:55:53 +01:00
Mike Fährmann
62692c6842 [exhentai] add 'source' option 
setting it to "hitomi" downloads the corresponding gallery from
hitomi.la; might be extended to other sources in the future
 2021-12-16 23:16:19 +01:00
Mike Fährmann
099ed72de7 [hitomi] disable extra 'metadata' by default 
safes one HTTP request that not needed with default filename settings
 2021-12-16 22:21:07 +01:00
Mike Fährmann
9a25534490 use Extractor._check_cookies() for all cookie checks 2021-12-16 02:21:16 +01:00
Mike Fährmann
63c6bc26b5 [rule34us] extract tags per category (#1527) 
like for other boorus with 'tags': true
 2021-12-16 00:06:52 +01:00
Mike Fährmann
f587458a3c [twitter] include '4096x4096' as a default image fallback 
(closes #2107, closes #1881)
 2021-12-15 23:19:30 +01:00
Mike Fährmann
8ed282f7f2 [kemonoparty] support coomer.party URLs (#2100) 2021-12-15 16:21:05 +01:00
Mike Fährmann
87ce3fa669 [furaffinity] warn when no session cookies were found 2021-12-15 16:21:05 +01:00
Mike Fährmann
159631c808 [philomena] use a default 'filter_id' if non is given 2021-12-15 16:20:53 +01:00
Mike Fährmann
ad30653b17 allow running a BaseExtractor for any URL 
by prefixing it with '<base-category>:'

For example:
  shopify:https://partakefoods.com/products/crunchy-cookie-variety-pack
  gelbooru_v01:https://5naf.booru.org/index.php?page=post&s=view&id=46963

Available base categories are:
  mastodon, shopify, moebooru, gelbooru_v01, gelbooru_v02,
  reactor, foolslide, foolfuuka,  philomena
 2021-12-15 00:32:17 +01:00
Mike Fährmann
299bd2f1f5 [rule34us] add 'tag' and 'post' extractors (#1527) 2021-12-14 00:27:46 +01:00
Mike Fährmann
3cf1075d86 [inkbunny] add 'search' extractor (closes #2094) 2021-12-12 03:08:14 +01:00
Mike Fährmann
c6a23c26d7 [instagram] allow downloading specific stories (closes #2088) 
https://instagram.com/stories/<USER>/<ID> now only downloads the one
story specified by <ID> and not all stories from that user.
 2021-12-11 21:34:25 +01:00
Mike Fährmann
352ffcddb0 [instagran] match post URLs with usernames (fixes #2085) 2021-12-10 18:37:33 +01:00
Mike Fährmann
f4e3cee6ac use yt-dlp by default (#1850, #2028) 2021-11-29 18:24:26 +01:00
Mike Fährmann
f1b142e993 {kemonoparty[ change default 'files' order to attachments,file,inline 
(#1991)
 2021-11-29 04:41:30 +01:00
Mike Fährmann
275543b2d2 update extractor test results 2021-11-27 19:26:44 +01:00
Mike Fährmann
e7ea4f2567 [mangoxo] fix metadata extraction 2021-11-27 18:19:51 +01:00
Mike Fährmann
e298882acc [kemonoparty] match URLs with www subdomain 2021-11-26 18:58:26 +01:00
Mike Fährmann
addb72e1bb [reactor] support thatpervert.com (closes #2029) 2021-11-26 18:58:07 +01:00
Mike Fährmann
d8d9502e1e [reactor] inherit from BaseExtractor 2021-11-26 18:58:07 +01:00
Mike Fährmann
f4ea216c95 [shopify] support loungeunderwear.com (closes #2053) 2021-11-26 18:58:06 +01:00
Mike Fährmann
93cef78450 [gelbooru] workaround pagination limits 
Gelbooru only allows to retrieve the latest 20k posts for a tag search.
Add 'id:<N' to the search tags to work around that limitation, where N
is the ID of the last retrieved post.

http://gelbooru.me/index.php?page=forum&s=view&id=1467
 2021-11-26 18:56:31 +01:00
Mike Fährmann
f2ae179713 [exhentai] fix extraction for disowned galleries (closes #2055) 2021-11-24 21:26:16 +01:00
Alice
612850438e [skeb] add 'thumbnails' option (#2047) (#2051) 2021-11-23 21:16:42 +01:00
Mike Fährmann
11a3d96d13 [mangadex] load additional metadata using includes[] directives 
- always provide 'artist', 'author', and 'group' metadata fields (#2049)
- remove 'metadata' option
 2021-11-22 01:16:33 +01:00
Mike Fährmann
19e00f1322 [dynastyscans] provide 'date' as proper datetime object (#2050) 2021-11-21 22:50:52 +01:00
Mike Fährmann
af6424f398 allow testing metadata in list elements 2021-11-21 22:46:34 +01:00
Mike Fährmann
c67756e187 [kemonoparty] add 'dms' option (#2008) 2021-11-20 23:36:16 +01:00
Mike Fährmann
3a7a19c7b9 [dynastyscans] add 'manga' extractor (closes #2035) 2021-11-19 22:51:26 +01:00
Mike Fährmann
9bc83af3a6 [kemonoparty] 'postfile' -> 'file' (#1991) 
to stay consistent with the existing file types for kemono
 2021-11-19 01:50:48 +01:00
Mike Fährmann
522782c09d [subscribestar] emit metadata for posts without media (#1569) 2021-11-18 23:42:17 +01:00
Mike Fährmann
1c8aaf9318 [subscribestar] add 'num' enumeration index (closes #2040) 2021-11-18 23:38:41 +01:00
Mike Fährmann
d433735750 [kemonoparty] skip duplicate files (#2032, #1991, #1899) 
Extract the SHA-256 file hash from URLs
and skip files with the same hash in the same post.

- provide a 'hash' metadata field (empty string if not available)
- remove 'patreon-skip-file' option
 2021-11-17 22:44:15 +01:00
Mike Fährmann
d4ec245554 [kemonoparty] implement a 'files' option (#1991) 
similar to 8d676151
 2021-11-17 22:43:41 +01:00
Mike Fährmann
ab8eea1a24 [twitter] fix extractor for direct image links (fixes #2030) 2021-11-16 22:57:46 +01:00
Mike Fährmann
2076d40681 [ytdl] improve error handling (#1680) 2021-11-15 22:56:42 +01:00
Mike Fährmann
2aaac3c997 [instagram] include user metadata for 'tagged' downloads (#2024) 
Adds
- tagged_owner_id
- tagged_full_name
- tagged_username
containing the values for the user profile the URL originated from,
e.g. 'instagram' for https://www.instagram.com/instagram/tagged/.
 2021-11-15 21:21:59 +01:00
Mike Fährmann
cfa4876848 [philomena] support furbooru.org (closes #1995) 2021-11-15 20:57:51 +01:00
Mike Fährmann
4377f1c284 [twitter] distinguish between fatal & nonfatal errors (#2020) 
only show a warning for nonfatal errors
and do not raise a StopExtraction exception
 2021-11-13 22:46:40 +01:00
Kyle Anthony Williams
a14b72be21 [webtoons] Use swebtoon-phinf.pstatic.net instead of webtoon-phinf.pstatic.net (#2005) 
* [webtoons] Use swebtoon-phinf.pstatic.net instead of webtoon-phinf.pstatic.net

This trick to avoid having to set a Referer header comes from
Webtoon's RSS feeds. The two URLs below are equivalent in content:

https://webtoon-phinf.pstatic.net/20210929_153/1632867980912DmcGK_JPEG/16328679808882705182.jpg?type=q90
https://swebtoon-phinf.pstatic.net/20210929_153/1632867980912DmcGK_JPEG/16328679808882705182.jpg?type=q90

The URL with the domain "webtoon-phinf.pstatic.net" needs a Referer
header, and the domain "swebtoon-phinf.pstatic.net" does not. This
is because of the environment "swebtoon" images live in, one without
explicit network control: RSS feeds on sites such as Feedly. This change should
make it easier for gallery-dl developers to embed Webtoon comics without
worrying about headers.
 2021-11-11 20:03:34 +01:00
Mike Fährmann
6e3658ef52 [kemonoparty] provide 'date' metadata for gumroad (#2007) 
Not the 'published' or 'edited' values since they are 'null',
but still better then nothing at all.
 2021-11-11 19:38:10 +01:00
Mike Fährmann
37c9dedee1 [seisoparty] remove module 2021-11-09 22:41:04 +01:00
Mike Fährmann
efa178cc91 [ytdl] implement parsing ytdl command-line options (#1680) 
- adds 'config-file' and 'cmdline-args' options
  for both ytdl downloader and extractor
- create 'ytdl' helper module, which combines YoutubeDL creation
  and option parsing.
- most likely a buggy mess due to incompatibilities between the
  original youtube-dl and yt-dlp.
 2021-11-07 02:44:11 +01:00
Mike Fährmann
7cb303d745 [redgifs] improve URL extraction 
Fields inside 'urls' can be None, which would have caused an exception
with the old method.
 2021-11-05 20:02:43 +01:00
Mike Fährmann
2befed1a96 [redgifs] update search URL pattern (#1984) 2021-11-05 20:00:06 +01:00