Commit Graph

263 Commits

Author SHA1 Message Date
Mike Fährmann
008a08ca43 [ytdl] add 'deprecations' option
https://github.com/gdl-org/builds/issues/3
2025-07-20 19:55:24 +02:00
Mike Fährmann
1561284815 [madokami] add 'manga' extractor (#7828) 2025-07-17 20:40:26 +02:00
Mike Fährmann
bbe7faed21 [iwara] update
- add 'include' docs
- simplify code for different result types
- provide 'count' and 'num' metadata
- remove 'search_type' metadata
2025-07-14 18:47:14 +02:00
NecRaul
a7ebb835ea [iwara] Add support (#2652 #5840 #7785)
* [iwara] Add initial support
* [iwara] Add search support
* [iwara] Code cleanup
* [iwara] Small fixes and additions
* [iwara] Add tag support
* [iwara] Add mime-type to metadata
* [iwara] Refactor patterns/matching using urllib
* [iwara] Add unit tests
* [iwara] Update docs
* [iwara] Fix linting on older Python versions
* [iwara] update 'IwaraAPI' interface class
    - define endpoints inside methods
    - implement and use _call() and _pagination()
    - cache auth tokens
* [iwara] split and rename 'profile' extractor
    TODO:
    - update test results
    - simplify code
* [iwara] simplify '_user_params()' usage
* [iwara] update 'video' extractor
    and move user data extraction into 'yield_video'
* [iwara] update 'image' extractor
    and move user info extraction into 'yield_image()'
* [iwara] update 'playlist' extractor
* [iwara] update 'search' extractor
* [iwara] update 'tag' extractor
* [iwara] simplify 'yield_image' usage
    perform API calls to get full 'files' list inside the function
* [iwara] add video "image" test
* [iwara] provide 'date' metadata
* [iwara] simplify 'source()'
    remove urllib.parse usage
* [iwara] small optimizations
    * get("key", {}) -> get("key") or {}
    * split("…", 1) -> partition("…")
    * use f-strings for all patterns
* [iwara] add missing 'keyarg=1' to profile() memcache decorator
* [tests/iwara] update results
* [iwara] extract more 'user' metadata
* [iwara] update default format strings
    include 'date' in filenames to order them chronologically
* [iwara] restructure image/video handling
    - use less generators
    - make processing individual media items non-fatal
* [iwara] fix login and token handling
* [iwara] add 'favorite' extractor
* [iwara] add 'following' and 'followers' extractors

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2025-07-13 21:30:25 +02:00
Mike Fährmann
fc968ebf20 [artstation] support downloading '.mview' files (#7812) 2025-07-12 20:53:16 +02:00
Mike Fährmann
82891b4d0c [pixiv] move 'novel' extractors to a 'pixiv-novel' category (#7746)
TODO:
- restore full 'include' functionality
- allow remapping category:subcategory pairs
2025-07-04 20:13:19 +02:00
Mike Fährmann
e7922ababd [naver] change categories (#7746)
- 'naver'        -> 'naver-blog'
- 'chzzk'        -> 'naver-chzzk'
- 'naverwebtoon' -> 'naver-webtoon'
2025-07-02 23:20:40 +02:00
Mike Fährmann
b102c474c1 [bunkr] fix file downloads (#7747)
update 'root_dl', 'root_api', and default 'endpoint'
2025-06-29 17:53:13 +02:00
Mike Fährmann
22b40fc787 [sankaku] remove 'id-format' option (#5073 #6808) 2025-06-29 17:50:19 +02:00
Mike Fährmann
9dbe33b6de replace old %-formatted and .format(…) strings with f-strings (#7671)
mostly using flynt
https://github.com/ikamensh/flynt
2025-06-29 17:50:19 +02:00
enduser420
f77e98b57d [chzzk] add 'comment' and 'community' extractors (#7735 #7741)
* [chzzk] add 'comment' and 'community' extractors
* [chzzk] update
* [chzzk] add tests
* [chzzk] update docs/supportedsites
* [chzzk] add 'offset' option
* [docs] add 'offset' option to gallery-dl.conf
2025-06-28 15:27:19 +02:00
Mike Fährmann
5e61fe8668 [rule34xyz] implement login with username & password (#7736) 2025-06-27 22:35:59 +02:00
Mike Fährmann
8a93616a2d [webtoons] add 'banners' option (#6468) 2025-06-26 19:29:52 +02:00
Mike Fährmann
e7107f160d [fanbox] add 'fee-max' option (#7726) 2025-06-24 14:47:09 +02:00
Mike Fährmann
68960e29a1 [dankefuerslesen] add support (#7669) 2025-06-22 12:13:12 +02:00
Mike Fährmann
b0d7de3603 support using system certificates via 'truststore' (#6582)
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-2989290495
2025-06-20 19:55:01 +02:00
Mike Fährmann
dba45923c8 [dl:http] fail downloads with HTML content (#7697)
add 'validate-html' option
2025-06-20 14:34:40 +02:00
pocketinternet
3ea244eebb [webtoons] add 'thumbnails' option (#6468 #7441)
* Update webtoons.py
    Added thumbnail download capability which defaults to false
* Update configuration.rst
    Added documentation for webtoon thumbnail option
* extract thumbnails in GalleryExtractor.assets()
* simplify & fix flake8
* include 'type' in default filenames
* add test
* update docs

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2025-06-17 19:34:58 +02:00
hunter-gatherer8
96f5cfb305 [girlswithmuscle] add support (#4493 #6016)
* [girlswithmuscle] init
* [girlswithmuscle]: fix metadata extraction (site layout change)
* [girlswithmuscle]: fix tags extraction (site layout change)
* update login code
* update 'post' extractor
* update 'gallery' extractor, rename to 'search' extractor
* update docs
* add test cases

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2025-06-14 23:05:49 +02:00
Mike Fährmann
8e698d1a64 [ytdl] set domain as subcategory when using Generic extractor (#6582)
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-2959879730
2025-06-10 21:35:15 +02:00
Mike Fährmann
967af5eede [exhentai] add 'limits-action' option (#6504)
https://github.com/mikf/gallery-dl/issues/6504#issuecomment-2949551532
2025-06-08 12:56:56 +02:00
Mike Fährmann
3e423937d2 [misskey] implement 'include' option (#5347) 2025-06-06 20:52:03 +02:00
Mike Fährmann
38116ce04c [docs] add 'discord' options to gallery-dl.conf 2025-06-04 21:29:15 +02:00
Mike Fährmann
685836f6fd [dynastyscans] add 'anthology' extractor (#7627) 2025-06-04 21:23:49 +02:00
Mike Fährmann
75b6c8f3d8 re-implement 'category-map' (#7612) 2025-06-04 07:57:27 +02:00
Mike Fährmann
99bf92364a rename 'category-map' to 'config-map' (#7612) 2025-06-04 07:57:27 +02:00
Mike Fährmann
df4845bf60 [sexcom] update
- fix 'title' and 'type' of pictures
- remove '#' from the beginning of each tag
- add 'gifs' option
2025-06-04 07:44:43 +02:00
Mike Fährmann
783f1c1b11 add 'category-map' option
primarily to achieve some form of backwards compatibility for
922c296482
2025-05-30 17:55:49 +02:00
Mike Fährmann
922c296482 [kemono][coomer][schalenetwork] rename modules & extractors
category changes:

- kemonoparty -> kemono
- coomerparty -> coomer
- koharu      -> schalenetwork

also wanted to rename '2chan' -> 'sturdychan',
but the site's main page is still titled '2chen'
2025-05-30 17:51:49 +02:00
bradenhilton
3ba4404d21 [pixeldrain] add support for filesystem URLs (#7473) 2025-05-21 17:28:09 +02:00
Mike Fährmann
c28df68f68 [instagram:stories] add 'split' option (#7521) 2025-05-19 16:42:35 +02:00
Mike Fährmann
0285473b04 [flickr] add 'profile' option 2025-05-13 11:47:34 +02:00
Mike Fährmann
6b84de6cf7 [flickr] add 'info' option (#4720 #6817) 2025-05-12 17:07:36 +02:00
Mike Fährmann
f88e42c194 [mangadex] implement login with client credentials
unless the legacy method still works for your account,
login now requires 4 values:

- client-id
- client-secret
- username
- password

where 'client-id' and 'client-secret' are the credentials
of a personal API client.

> Public clients are not yet available.
2025-05-12 11:57:34 +02:00
Mike Fährmann
f343c8c0b6 [civitai] fix/improve video downloads (#7502)
- add 'quality-videos' option
- fix downloading static .webp files instead of video
2025-05-10 16:13:48 +02:00
Mike Fährmann
e26ae210bd [kemonoparty] restore using default creator posts endpoint
(#7438 #7450 #7462)

- revert c3fb0f53ee
- add '"endpoint": "legacy+"' option
  to use '/posts-legacy' results and with full metadata
2025-05-02 21:49:28 +02:00
Mike Fährmann
16b5d41ce5 [kemonoparty] add 'endpoint' option (#7438) 2025-04-28 16:47:48 +02:00
noxibow
4c974debf5 [naver] support video downloads (#7395)
and fix 'date' for posts newer than 24h

- fix naver blogposts directory error for posts created less than 24 hours ago
- add support for videos on naver blogs
- refactor to remove unnecessary imports
- fix video url scraping method
- rework date handling
- rework video extraction
- add 'videos' option
- add test
2025-04-22 09:14:32 +02:00
Mike Fährmann
481ad1c206 [bluesky:likes] use 'repo.listRecords' endpoint (#7194 #7287)
requires one additional API request per post,
but allows accessing likes of all users without login
and avoids the infinite loop at the end of 'getActorLikes'

add 'endpoint' option
2025-04-15 22:47:33 +02:00
Mike Fährmann
4916b4fd1f [webtoons] download JPEG files in better quality
add 'quality' option
2025-04-10 22:04:43 +02:00
Mike Fährmann
09d42b8e89 [webtoons] use a default delay pf 0.5-1.5s between requests (#7329) 2025-04-09 20:41:22 +02:00
Mike Fährmann
24bbcbcfa3 [danbooru] add 'favgroup' extractor 2025-03-26 20:58:49 +01:00
Mike Fährmann
e1aabf01e4 merge #7220: [deviantart] add subfolder support (#4988 #7185) 2025-03-24 18:43:11 +01:00
Mike Fährmann
b52c21186b [deviantart] add 'subfolders' option 2025-03-23 17:58:58 +01:00
Mike Fährmann
4a74bc6e30 [kemonoparty] extract 'archives' metadata (#7195)
add 'archives' option for additional data
2025-03-22 18:38:21 +01:00
Mike Fährmann
f8ef9a7b35 [kemonoparty] enable 'username'/'user_profile' metadata by default 2025-03-21 20:01:06 +01:00
Mike Fährmann
dbe8820b9e [arcalife] add 'gifs' option (#5657) 2025-03-14 22:34:45 +01:00
hdk5
d900e868e4 [arcalive] add support (#5657 #7100)
* [arca.live] Add extractor skeleton

* [arcalive] update names and formatting

* [arcalive] implement initial file extraction code

* [arcalive] improve '_extract_media()' performance

compile and cache regex on demand

* [arcalive] improve image extraction

- extract 'data-originalurl' URLs if available
- replace URL query strings with 'type=orig'
- ignore emoticons by default

* [arcalive] update defaults

- include 'title' in filenames
- use 0.5-1.5s delay between requests

* [arcalive] use ext from 'data-orig' if available

* [arcalive] update docs/supportedsites

* [arcalive] add tests

* [arcalive] update 'board' extractor pattern

so it doesn't also match 'post' URLs

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2025-03-14 10:52:21 +01:00
Mike Fährmann
22d46f2462 [batoto] add 'domain' option (#7174)
allow legacy domains by default
2025-03-14 10:31:49 +01:00
Mike Fährmann
cd1ddb0a67 [wikimedia] add 'subcategories' option (#2340)
https://github.com/mikf/gallery-dl/pull/2340#issuecomment-2707177295
2025-03-12 22:05:44 +01:00