Commit Graph

1828 Commits

Author SHA1 Message Date
Amar Paul
b552cdba04 [pholder] add support (#2568 #9067)
* feat: extractor for pholder.com
    Closes #2568
* feat[pholder]: support gallery_id properly and tags
* doc[text.nameext_from_name]: minor typo in docstring

* remove '__init__' & 'request' methods and 'json' import
* use 'text.nameext_from_url' to ensure a 'filename' value
* fix 'imgur' links by disabling auto-Referer
* fix 'data["id"].partition()' call
    'partition' returns 3 elements
* use 'item["_source"]' data directly
* remove unused supportedsites overwrite
* catch all exceptions in '_thumb_resolution'
    fixes "KeyError: 'width'"
* use 'author' name for user folders

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2026-02-15 09:46:18 +01:00
Mike Fährmann
01cb378baa [imagepond] support '/i/' URLs, update root domain 2026-02-14 19:33:52 +01:00
Mike Fährmann
41998cbb8f [chevereto] combine 'image' & 'video' extractors into a 'file' extractor 2026-02-14 19:33:52 +01:00
Mike Fährmann
e9989ff7bd [options] add '--config-json' command-line option 2026-02-14 19:33:33 +01:00
Mike Fährmann
98eb857794 [pp:exec] use non-UNC path replacements (#8879)
provide '{_path_unc}' and '{_directory_unc}' replacement fields
2026-02-14 12:09:54 +01:00
Mike Fährmann
d2477a94af [options] add 'sleep-retries' option 2026-02-13 18:04:05 +01:00
Mike Fährmann
e6f2e31d45 [pp:ugoira] disable 'mkvmerge' output, add 'mkvmerge-output' option 2026-02-12 22:10:20 +01:00
Mike Fährmann
0f41f343f4 implement linear/exponential backoff for 'sleep-429' 2026-02-12 19:23:29 +01:00
Mike Fährmann
d491564f8a [instagram] add 'user-strategy' option (#8978 #9025) 2026-02-10 16:46:33 +01:00
Mike Fährmann
a8376f2804 [instagram] add 'user-cache' option (#8978 #9025) 2026-02-10 12:01:37 +01:00
wise-immersion
a8636e75a1 [fikfap] add 'hashtag' extractor (#9018)
Added functionality to extract by hashtag and save to directory named after the hashtag.
2026-02-08 11:42:48 +01:00
Mike Fährmann
56f2790626 [options] add '--config-type' command-line option
can also be set via 'GDL_CONFIG_TYPE' environment variable
9fd732afe8
2026-02-07 21:47:17 +01:00
Mike Fährmann
935bdb6229 [reddit:user] implement 'only' option (#8228) 2026-02-07 21:47:17 +01:00
Mike Fährmann
2d64e76223 [job] implement 'follow' option (#8752)
Follow and process URLs found in the given format string result.
2026-02-07 21:47:17 +01:00
Mike Fährmann
98ef34a9be [twitter] support 'article' media (#8995) 2026-02-07 21:47:17 +01:00
Mike Fährmann
71680feab9 [actions] implement 'flag … = skip' (#8960) 2026-02-06 10:10:38 +01:00
Mike Fährmann
df8aaa7e22 [options] add '--restrict-filenames' & '--windows-filenames' 2026-02-05 09:32:24 +01:00
Mike Fährmann
42407afb6d [xenforo] implement '"order-posts": "reaction"' (#8997) 2026-02-04 21:57:30 +01:00
Mike Fährmann
3d36ee0e53 [common] support multiple codes/blocks for '--xff'
for example '--xff JP,CN,105.48.0.0/12'
2026-02-04 20:58:59 +01:00
Mike Fährmann
947085c6e3 [common] add '--xff' / 'geo-bypass' option 2026-02-04 18:45:02 +01:00
Mike Fährmann
b329e6e472 [xenforo] add 'attachments' & 'embeds' options 2026-02-04 16:16:55 +01:00
Mike Fährmann
614e1a514a [twitter] implement 'metadata-user' option (#8984) 2026-02-03 20:14:34 +01:00
Mike Fährmann
2cbd9740f2 enable 'metadata-url/-path/-parent/-extractor]' by default
assign values to "private" keys to not have them appear in the
standard JSON output, while still making it easier to access them
2026-02-03 19:28:33 +01:00
Mike Fährmann
0be3383110 [formatter] add 'q' & 'Q' conversions - URL-en/decode values 2026-02-03 17:35:05 +01:00
Mike Fährmann
5ddee423ed [exhentai][schalenetwork] enable 'tags' categories by default 2026-02-03 10:49:35 +01:00
Mike Fährmann
d232f0d252 [gofile][pixeldrain] enable 'recursive' by default 2026-02-03 09:11:47 +01:00
Mike Fährmann
25d45478d6 [docs/configuration] document postprocessor /MODE and @EVENT (#8980)
4e9cb428d6
2026-02-02 22:08:10 +01:00
Mike Fährmann
e37fe053d2 [docs/configuration] use anchors for alternative/deprecated names
instead of doubled option titles
2026-02-02 19:10:20 +01:00
Mike Fährmann
87323fd71e rename 'image-*' & 'chapter-*' options to 'file-*' & 'child-*'
keep support for both names, e.g. file-filter & image-filter,
while prioritizing the new names
2026-02-02 18:01:45 +01:00
Mike Fährmann
153e589dd0 [twitter] implement '"search-pagination": "date"' 2026-02-02 16:19:48 +01:00
Mike Fährmann
17e1d25784 [scrolller] add 'user' extractor (#8961) 2026-02-02 09:09:50 +01:00
Mike Fährmann
44e18f9b2f [tsumino] remove module
" Tsumino - The End
  We're shutting Tsumino down. "
2026-02-01 22:15:06 +01:00
Mike Fährmann
c42a5dce5c [pp] add 'child' & 'child-after' events 2026-02-01 22:14:14 +01:00
Mike Fährmann
56dcd00391 [job] add 'init' option 2026-02-01 20:48:37 +01:00
Mike Fährmann
4e6e2c27d5 [xenforo] support 'forums.socialmediagirls.com' (#8964) 2026-01-31 09:55:41 +01:00
bassberry
fd5f5611f6 [tiktok] extract subtitles and all cover types (#8805)
* Make sure that `img_id`, `audio_id` and `cover_id` fields are always available.
    The values are set '' where they are not applicable.
    Having `img_id` is necessary for the default `archive_fmt`, the other fields are handled for consistency.
* Allow downloading more than one cover.
    The previous behavior is kept as-is, but setting the "covers" option to "all" now grabs all available covers.
* Add support for downloading subtitles
    Allows filtering subtitles by source type (ASR, MT) and language.
* Ensure archive uniqueness for covers and subtitles.
* Update the URL test pattern to include the `image` extension.
    Although Tiktok may serve the covers with jpeg content, the file ending can be `.image`.
    The test before 0c14b164 failed because the asserted URL did not match all cover types, but the now used pattern needs the mentioned file ending.
* Add support for "creator_caption" subtitles in "LC" format.
    These subtitles have the keys "Format" set to "creator_caption" and "Source" to "LC".
* Add "LC" (Local Captions) as a subtitle source type in the documentation
* Code deduplication and renaming subtitle metadata
    Changed the item type from singular `subtitle` to `subtitles`.
    Removed the wrong descriptor `cover` from the subtitles fallback title.
* Refactor subtitle filtering
    The filter is now prepared in `_init` to prevent parsing the same config parameter for every item.
    The `_extract_subtitles` function will still extract if either filter (source or language) matches.
* Generate a `file_id` for subtitles
    Subtitles have multiple fields that determine the unique file, so these are simply concatenated.
    This is similar to the cover types, only with more variations.
* Added tests for subtitles
* fix docs entries
* fix '"covers": "all"'
* simplify some code
* Fix fallback title for subtitles
    Added the missing "f" to the f-string and added "subtitle" to the title.
    The resulting title will look like "TikTok video subtitle #1234567"
2026-01-30 21:01:06 +01:00
Mike Fährmann
3445c51ca4 [job] add 'output.jsonl' option (#8953) 2026-01-30 09:36:28 +01:00
Mike Fährmann
532ab7112e [discord] add 'server-search' extractor
requested on Discord

https://discord.com/channels/SERVER_ID/search?from=USER_ID
2026-01-30 07:58:14 +01:00
Mike Fährmann
56168fbc87 [weebdex] add 'lang' option, support query params (#8957)
for example '?order=asc&group=j0fsj3oem3&tlang=en'
2026-01-29 17:01:02 +01:00
Mike Fährmann
6c9dff1e29 [docs/options] add Table of Contents 2026-01-27 19:27:40 +01:00
SubmarineScurvy
ef8f2869e7 [listal] add 'image' & 'people' extractors (#1589 #8921)
* listal extractor
* add listal to init
* fix flake8 & formatting & extractor names/subcategories

* remove 're' import
* remove 'datetime' import
* update & simplify extractors
* update supportedsites
* add tests

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2026-01-27 18:26:41 +01:00
Mike Fährmann
eaaa25b6e4 [job] enable all 'parent-…' options for parent extractors by default
- parent-directory
- parent-metadata
- parent-session
- parent-skip

- add general 'parent' option
2026-01-27 12:05:19 +01:00
Mike Fährmann
b67e3c15ff [xenforo] support 'titsintops.com' (#8945) 2026-01-27 10:31:26 +01:00
Mike Fährmann
93bf4ccc18 merge #8928: [mangafreak] add support 2026-01-25 19:52:34 +01:00
Mike Fährmann
7026611f31 merge #8925: [mangatown] add support 2026-01-25 18:35:39 +01:00
Duy Nguyen
8b0e8c656d feat(mangafreak): add support for MangaFreak
Add chapter and manga extractors for ww2.mangafreak.me with support
for bonus chapters (e.g., 167e suffix).
2026-01-25 15:56:52 +01:00
Mike Fährmann
adca123646 [weibo:user] add 'subalbums' include (#8792) 2026-01-25 11:16:41 +01:00
Duy Nguyen
9f2d5cbd5d docs: add mangatown to supported sites 2026-01-25 00:04:23 +01:00
Mike Fährmann
1530778bfb merge #8917: [kaliscan] add support 2026-01-23 21:04:48 +01:00
Duy Nguyen
5c71993e0b docs: add kaliscan to supported sites 2026-01-23 20:29:24 +01:00