Commit Graph

371 Commits

Author SHA1 Message Date
Mike Fährmann
d2477a94af [options] add 'sleep-retries' option 2026-02-13 18:04:05 +01:00
Mike Fährmann
d491564f8a [instagram] add 'user-strategy' option (#8978 #9025) 2026-02-10 16:46:33 +01:00
Mike Fährmann
a8376f2804 [instagram] add 'user-cache' option (#8978 #9025) 2026-02-10 12:01:37 +01:00
Mike Fährmann
935bdb6229 [reddit:user] implement 'only' option (#8228) 2026-02-07 21:47:17 +01:00
Mike Fährmann
2d64e76223 [job] implement 'follow' option (#8752)
Follow and process URLs found in the given format string result.
2026-02-07 21:47:17 +01:00
Mike Fährmann
98ef34a9be [twitter] support 'article' media (#8995) 2026-02-07 21:47:17 +01:00
Mike Fährmann
947085c6e3 [common] add '--xff' / 'geo-bypass' option 2026-02-04 18:45:02 +01:00
Mike Fährmann
b329e6e472 [xenforo] add 'attachments' & 'embeds' options 2026-02-04 16:16:55 +01:00
Mike Fährmann
614e1a514a [twitter] implement 'metadata-user' option (#8984) 2026-02-03 20:14:34 +01:00
Mike Fährmann
2cbd9740f2 enable 'metadata-url/-path/-parent/-extractor]' by default
assign values to "private" keys to not have them appear in the
standard JSON output, while still making it easier to access them
2026-02-03 19:28:33 +01:00
Mike Fährmann
5ddee423ed [exhentai][schalenetwork] enable 'tags' categories by default 2026-02-03 10:49:35 +01:00
Mike Fährmann
d232f0d252 [gofile][pixeldrain] enable 'recursive' by default 2026-02-03 09:11:47 +01:00
Mike Fährmann
87323fd71e rename 'image-*' & 'chapter-*' options to 'file-*' & 'child-*'
keep support for both names, e.g. file-filter & image-filter,
while prioritizing the new names
2026-02-02 18:01:45 +01:00
Mike Fährmann
44e18f9b2f [tsumino] remove module
" Tsumino - The End
  We're shutting Tsumino down. "
2026-02-01 22:15:06 +01:00
Mike Fährmann
56dcd00391 [job] add 'init' option 2026-02-01 20:48:37 +01:00
bassberry
fd5f5611f6 [tiktok] extract subtitles and all cover types (#8805)
* Make sure that `img_id`, `audio_id` and `cover_id` fields are always available.
    The values are set '' where they are not applicable.
    Having `img_id` is necessary for the default `archive_fmt`, the other fields are handled for consistency.
* Allow downloading more than one cover.
    The previous behavior is kept as-is, but setting the "covers" option to "all" now grabs all available covers.
* Add support for downloading subtitles
    Allows filtering subtitles by source type (ASR, MT) and language.
* Ensure archive uniqueness for covers and subtitles.
* Update the URL test pattern to include the `image` extension.
    Although Tiktok may serve the covers with jpeg content, the file ending can be `.image`.
    The test before 0c14b164 failed because the asserted URL did not match all cover types, but the now used pattern needs the mentioned file ending.
* Add support for "creator_caption" subtitles in "LC" format.
    These subtitles have the keys "Format" set to "creator_caption" and "Source" to "LC".
* Add "LC" (Local Captions) as a subtitle source type in the documentation
* Code deduplication and renaming subtitle metadata
    Changed the item type from singular `subtitle` to `subtitles`.
    Removed the wrong descriptor `cover` from the subtitles fallback title.
* Refactor subtitle filtering
    The filter is now prepared in `_init` to prevent parsing the same config parameter for every item.
    The `_extract_subtitles` function will still extract if either filter (source or language) matches.
* Generate a `file_id` for subtitles
    Subtitles have multiple fields that determine the unique file, so these are simply concatenated.
    This is similar to the cover types, only with more variations.
* Added tests for subtitles
* fix docs entries
* fix '"covers": "all"'
* simplify some code
* Fix fallback title for subtitles
    Added the missing "f" to the f-string and added "subtitle" to the title.
    The resulting title will look like "TikTok video subtitle #1234567"
2026-01-30 21:01:06 +01:00
Mike Fährmann
3445c51ca4 [job] add 'output.jsonl' option (#8953) 2026-01-30 09:36:28 +01:00
Mike Fährmann
56168fbc87 [weebdex] add 'lang' option, support query params (#8957)
for example '?order=asc&group=j0fsj3oem3&tlang=en'
2026-01-29 17:01:02 +01:00
Mike Fährmann
eaaa25b6e4 [job] enable all 'parent-…' options for parent extractors by default
- parent-directory
- parent-metadata
- parent-session
- parent-skip

- add general 'parent' option
2026-01-27 12:05:19 +01:00
Mike Fährmann
f869085476 [weebdex] add 'data-saver' option (#8914) 2026-01-23 09:22:41 +01:00
Mike Fährmann
18fabb9605 [batoto] remove module (#8908)
"Bato.to has shut down."

There are mirror sites, but they are unscrapeable
due to heavily obfuscated HTML and JS
2026-01-21 20:33:08 +01:00
Mike Fährmann
4798ac4836 [common] implement 'parent-session' 2026-01-21 20:33:08 +01:00
Mike Fährmann
63df6423bf [nitter] use 'gallery-dl/<version>' User-Agent (#7045 #8130 #8409) 2026-01-21 18:07:47 +01:00
brerk
e00c717b15 [turbo] update 'saint' extractors (#8893 #8896)
* Implements turbo.py & remove from domain pattern from saints.py
* Remove leftover commented pattern from saints.py
* Make turbo.py comply with flake8
* Add album support
* Improved metadata extracion for albums and single files & created turbo.py tests using saints.py test
* Align turbo.py extractor with flake8 rules
* Fix #class name on turbo.py tests
* Fix #category test
* Fix #category test x2
* Fix #category tests
* Fix #category tests
* Fix TurboMediaExtractor self.groups unpacking

* update basic module formatting
* replace 'saint' with 'turbo' in modules list
* remove saint extractors and tests
* update & simplify 'media' extractor
* update & simplify 'album' extractor
* update tests
* update supportedsites
* update 'category-map' & 'config-map'

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2026-01-19 22:20:13 +01:00
Mike Fährmann
254bb4505c [chevereto] implement password support 2026-01-16 09:33:55 +01:00
Mike Fährmann
fea4f4c30b [pixeldrain:album] add 'zip' option 2026-01-14 12:03:52 +01:00
camellia2077
084a6d73e0 [bilibili] add support for Live Photo (video) downloads (#8860)
* bilibili: add support for live photo downloads
* fix: resolve flake8 linting errors (whitespace and line length)
* fix: resolve flake8 E302 and W293 linting errors
* fix: resolve flake8 W293 and E302 linting errors

* simplify syntax
* add 'livephoto' option
* add tests
2026-01-10 19:27:34 +01:00
Mike Fährmann
60104f20af [xenforo] implement 'media' support (#8785)
add 'media-user' and 'media-item' extractors
TODO: 'media-category' extractor (?)
2026-01-09 21:21:19 +01:00
Mike Fährmann
b1a708beb9 [weibo:album] implement 'subalbum' support (#8792)
https://weibo.com/u/USER?tabtype=album-1
https://weibo.com/u/USER?tabtype=album-NAME
https://weibo.com/u/USER?tabtype=album-all
https://weibo.com/u/USER?tabtype=album-only
2026-01-07 19:23:54 +01:00
Mike Fährmann
cf96fc6ebe [koofr] refactor (#8803)
https://github.com/mikf/gallery-dl/issues/8803#issuecomment-3708215475
https://github.com/mikf/gallery-dl/issues/8803#issuecomment-3708358606

- add 'recursive' option, remove 'zip'
- recurse into subdirectories
- add 'path' metadata
- remove 'count' & 'num' metadata
- update default directory & archive format
2026-01-06 21:52:58 +01:00
Mike Fährmann
fcf6f05a0d [koofr] implement 'zip' option (#6582 #8700)
Download individual files instead of '.zip' bundle by default.

https://github.com/mikf/gallery-dl/issues/6582#issuecomment-3679504391
2026-01-01 19:13:31 +01:00
Mike Fährmann
239de3f092 [civitai] update 'quality-videos' default (#8787)
from "quality=100" to "original=true,quality=100"
to prevent a server timeout / error
2025-12-31 09:56:39 +01:00
CasualYouTuber31
a6c845bdc8 [tiktok] remove yt-dlp dependency & add support for more post types (#8715)
#7246 #8035 #8466 #8730

* [tiktok] support extracting videos directly without yt-dlp
* [tiktok] support extracting users directly without yt-dlp
* [tiktok] fixing logic, tests, linting errors
* [tiktok] implement tiktok-range support for non-yt-dlp user extractor
* [tiktok] Skip range filter if no ranges are given
* [tiktok] Remove debug code
* [tiktok] only check for faulty device IDs during the first couple of passes
    I think the original yt-dlp solution assumes that if a device ID works once, it will always work.
    Plus, my approach would cause needless retries in certain cases if hasMorePrevious does end up being wrong like the original algorithm accounts for. So let's copy the original algorithm here, too.
* [tiktok] support stories
* [tiktok] you can now extract audio without extracting photos
* [tiktok] add TiktokFollowingExtractor
* [tiktok] update supportedsites to include stories
* [tiktok] Keep tiktok-range option for no content user account test
    It acts as a nice guard against that account suddenly having lots of posts to extract
* [tiktok] TiktokUserExtractor and TiktokFollowingExtractor rewrite
* [tiktok] Fix avatar naming convention to match that of posts
* [tiktok] remove type hints for compatibility with older Python versions
* [tiktok] Improve performance of TiktokFollowingExtractor
    This was largely achieved using the story/batch/item_list endpoint
* [tiktok] Forgot to run flake8
* [tiktok] remove old constant
* [tiktok] Support order-posts config item
* [tiktok] flake8
* [tiktok] Older Python versions don't support match
* [tiktok] always ask for posts in chronological order when in "desc" mode
    We should aim to avoid having pinned posts returned before non-pinned ones
* [tiktok] Add liked posts extraction
* [tiktok] Add reposts extraction
* [tiktok] Add saved posts extraction

* cleanup imports
* remove '# MARK:' comments
* remove & simplify 'except' statements
    KeyboardInterrupt & SystemExit inherit from BaseException (not Exception)
    and therefore don't need special handling
* split 'user' extractor
* move PATTERNs into their respective functions
* use dict comprehensions
* add only-matching test URLs for split user extractors
* update config docs
    rename 'tiktok-user-extractor' to 'ytdl'
* document '"popular"' 'order-posts' value
* inline and remove 'util.chunk()'
2025-12-30 17:17:57 +01:00
Mike Fährmann
c384cd06c2 [fansly] add 'previews' option (#8686) 2025-12-25 20:25:47 +01:00
Mike Fährmann
01fe6870a3 [fansly] disable 'formats' check by default (#8757)
should allow downloading and selecting the highest resolution video format
regardless of its 'type' / format code
2025-12-25 19:15:21 +01:00
Mike Fährmann
cc8cd78103 [webtoons] download episode background music (#8733)
add 'bgm' option
2025-12-23 19:08:30 +01:00
Mike Fährmann
375ab4084b [facebook] add 'loop' option (#8696) 2025-12-21 21:27:33 +01:00
Mike Fährmann
19a27a47b3 [options] add '--sleep-skip' option (#6582)
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-3677460485
2025-12-20 19:06:11 +01:00
Mike Fährmann
717d38f38c [reddit] use '"videos": "dash"' by default (#8657)
https://github.com/mikf/gallery-dl/issues/8657#issuecomment-3663560902
2025-12-17 18:45:06 +01:00
Mike Fährmann
2980e8fcb8 [docs] remove 'twitter.username-alt' entries 2025-12-14 18:00:08 +01:00
Mike Fährmann
a53cc87b72 [misskey] implement 'order-posts' option (#8516) 2025-12-13 21:06:31 +01:00
Mike Fährmann
db90fe9b82 [docs] update 'xenforo' options 2025-12-12 16:28:41 +01:00
Mike Fährmann
5831679fd2 [blueskx] add 'api-server' option (#8668) 2025-12-09 17:52:55 +01:00
Mike Fährmann
ef9c472f94 [twitter] always use '3' as 'search-stop' default (#8613)
https://github.com/mikf/gallery-dl/issues/8613#issuecomment-3626474518
2025-12-08 20:46:59 +01:00
Mike Fährmann
b1b70528a6 [common] use 'parent' value as 'parent-metadata' default (#8525 #8604)
enable 'parent-metadata' by default for
'chevereto' & 'imagehost' (album/gallery) extractors
2025-12-08 20:07:56 +01:00
Mike Fährmann
9a8746f12d [tiktok] add 'covers' option (#8515) 2025-12-07 20:30:47 +01:00
Mike Fährmann
5da200945b implement 'post-filter' & 'post-range' options
(#8643)
2025-12-05 16:40:23 +01:00
Mike Fährmann
d15b3d4eae [twitter] add 'limit' option (#8173)
- reduce default number of requested items to 50
- allow using "fallback" values like with 'search-limit'

https://github.com/mikf/gallery-dl/issues/8173#issuecomment-3303015837
2025-12-02 22:34:53 +01:00
Mike Fährmann
5f38d8ab25 [twitter] increase 'retries-api' default to '9' (#8317) 2025-11-30 16:52:26 +01:00
Mike Fährmann
acf281a46a [twitter] implement 'retries-api' option (#8317)
retry API requests when encountering server-related errors
2025-11-27 19:23:27 +01:00