github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6,793 Commits 1 Branch 0 Tags
52ca441dc2452c05104c4437ecd58eceba04c2fa
Commit Graph

6793 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mike Fährmann
52ca441dc2 [patreon] yield metadata for paywalled posts as well (#6582) 
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-2998535431
 2025-06-24 09:09:52 +02:00
Mike Fährmann
434bf3b1f3 use 'util.re()' in more places 
continuation of b5c88b3d3e
 2025-06-23 20:31:45 +02:00
Mike Fährmann
c08833aed9 [util] move 're' functions to text.py 2025-06-23 20:05:20 +02:00
Mike Fährmann
92ac35c932 [imgbox] fix 'TypeError' when calling 'get_image_metadata()' 
fixes regression introduced in 811b665e33
 2025-06-23 18:16:46 +02:00
Mike Fährmann
97cac544ed [facebook] fix 'TypeError' when calling 'decode_all()' 
fixes regression introduced in 811b665e33
 2025-06-23 17:55:32 +02:00
Mike Fährmann
d7103f9bdd [job:data] wrap exceptions in a dict (#7723) 
fixes exception when using 'num-to-str' with '-j'
 2025-06-23 17:17:28 +02:00
Mike Fährmann
dd759d34dd [scripts/init] support adding test results via '--url' 2025-06-23 17:16:11 +02:00
Luca Russo
1055773301 [discord] support forwarded messages + handle missing threads (#7706 #7722) 
* support forwarded messages + handle missing threads
* flake
 2025-06-23 17:14:31 +02:00
Mike Fährmann
fc1ed03484 [dankefuerslesen] fix several issues (#7669) 
- return chapters in first-to-last order
- provide 'group' as list of names
- remove hardcoded 'en' and 'English' language results
- replace 'popitem()' with non-destructive 'next(iter(…))'
- 'chapter-minor' -> 'chapter_minor'
 2025-06-22 19:52:40 +02:00
Mike Fährmann
ce4d78e8e3 [util] make NONE compare equal to the native None 2025-06-22 19:50:21 +02:00
Mike Fährmann
2cbc5cc454 [options] swap --print and --Print semantics 
--print now outputs values without downloading files
--Print now outputs values AND downloads files

add --Print-to-file with the same difference to --print-to-file
 2025-06-22 18:19:41 +02:00
Mike Fährmann
2bcc2f2a17 [formatter] implement 'M' format specifier 
Map a list of objects to a list of values

[
    {"name": "John Doe"      , "age": 42},
    {"name": "Jane Smith"    , "age": 24},
    {"name": "Max Mustermann", "age": null}
]

== :Mname =>

["John Doe", "Jane Smith", "Max Mustermann"]
 2025-06-22 16:33:20 +02:00
Mike Fährmann
71461ceaba [formatter] add 'L' conversion 
to map ISO 639-1 language codes to their full names
 2025-06-22 14:01:58 +02:00
Mike Fährmann
5e0cdac337 [formatter] change 'L'/length conversion to 'n' 2025-06-22 13:54:57 +02:00
Mike Fährmann
9c9eb7a3bb [tiktok] detect login page redirects (#7716) 2025-06-22 13:14:59 +02:00
Mike Fährmann
68960e29a1 [dankefuerslesen] add support (#7669) 2025-06-22 12:13:12 +02:00
Mike Fährmann
c65b0fb4d8 [tests] remove 'test_unique_pattern_matches' (#7671) 
no need to explicitly check for extractor pattern overlaps anymore
since extractor order (and dict insertion order in general) is
preserved since CPython 3.6
 2025-06-22 11:37:59 +02:00
Mike Fährmann
eaeabda7ac [scripts] implement 'init.py' 
Initial attempt at a helper script to generate new extractor module
files and the required boilerplate code.
 2025-06-22 10:13:06 +02:00
Mike Fährmann
60cb4468b2 [options] update --help Usage formatting 2025-06-20 23:43:22 +02:00
Mike Fährmann
b0d7de3603 support using system certificates via 'truststore' (#6582) 
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-2989290495
 2025-06-20 19:55:01 +02:00
Mike Fährmann
8a35863e86 [options] add '--Print' command-line option 2025-06-20 16:32:48 +02:00
Mike Fährmann
9e56d81292 fix Last-Modified mtime overwriting post processor mtime (#7529) 
https://github.com/mikf/gallery-dl/issues/7529#issuecomment-2989955455

- split '_mtime' into '_mtime_http' and '_mtime_meta'
- add PathFormat.set_mtime() method
 2025-06-20 16:19:37 +02:00
Mike Fährmann
74c9356442 [rule34] fix file downloads (#7697) 
replace 'api-cdn' subdomain of image files with 'wimg'
 2025-06-20 15:07:10 +02:00
Mike Fährmann
bcfce6b7db [dl:http] improve HTML signature check (#7697) 
https://github.com/mikf/gallery-dl/issues/7697#issuecomment-2990734451

ignore leading whitespace
 2025-06-20 14:39:32 +02:00
Mike Fährmann
dba45923c8 [dl:http] fail downloads with HTML content (#7697) 
add 'validate-html' option
 2025-06-20 14:34:40 +02:00
Mike Fährmann
e8d28d3d0d merge #7699: [warosu] handle missing images by checking hostname (#7698) 2025-06-20 13:17:44 +02:00
Mike Fährmann
bb73c8318c [warosu] add test for non-archived post 2025-06-20 12:38:54 +02:00
NecRaul
5ba7c98bc2 [warosu] Simpler/less costly hostname check 2025-06-20 14:27:21 +04:00
NecRaul
3c85032b9b [warosu] Handle missing images in the extractor by checking hostname 2025-06-20 09:34:49 +04:00
Mike Fährmann
4279928d0b [gelbooru_v02] extract 'total' / 'search_count' metadata (#7689) 2025-06-19 19:15:27 +02:00
Mike Fährmann
fcd1b8a155 [common] add a 'kwdict' member to extractor instances 
to allow setting general metadata at any point and without having to
rely on a manually implemented 'metadata()' method
 2025-06-19 19:08:35 +02:00
Mike Fährmann
a80d55d974 [common] improve 'user-agent' override logic 
Prevent a general 'user-agent' setting, i.e. extractor.user-agent, from
overriding the User-Agent header set by an extractor's 'browser'
option, default or otherwise, meaning only extractor-level or top-level
'user-agent' settings will override the 'browser' User-Agent.

https://github.com/mikf/gallery-dl/issues/7382#issuecomment-2985296321
 2025-06-19 18:58:39 +02:00
Mike Fährmann
6d7ca4890f [tests] add missing '\n' to stdout.write() output 2025-06-19 11:39:18 +02:00
Mike Fährmann
4f7a0e5d02 [arcalive] fix download URLs (#7678) 
https://github.com/danbooru/danbooru/pull/6041
 2025-06-18 21:12:18 +02:00
Mike Fährmann
8f79ec67f4 [text] add 'build_query()' 2025-06-18 20:49:12 +02:00
Mike Fährmann
b0580aba86 update 'match.lastindex' usage 2025-06-18 20:24:13 +02:00
Mike Fährmann
41191bb60a 'match.group(N)' -> 'match[N]' (#7671) 
2.5x faster
 2025-06-18 13:05:58 +02:00
Mike Fährmann
475506cc39 [util] more micro optimizations 2025-06-18 12:27:18 +02:00
Mike Fährmann
c73d902e13 [tests/results] include '#comment' in test output 2025-06-17 22:32:08 +02:00
Mike Fährmann
f7b1b6b5c8 [tests] replace 'print' with 'sys.stdout.write' 2025-06-17 22:05:41 +02:00
Mike Fährmann
dd299b3dad [tests/util] add LazyPrompt test 2025-06-17 21:47:21 +02:00
Mike Fährmann
acd5cb0cf0 [tests/util] add HTTPBasicAuth tests 2025-06-17 20:28:16 +02:00
Mike Fährmann
f1343894cb [tests/util] move 'datetime' tests into a separate TestCase class 2025-06-17 20:14:43 +02:00
pocketinternet
3ea244eebb [webtoons] add 'thumbnails' option (#6468 #7441) 
* Update webtoons.py
    Added thumbnail download capability which defaults to false
* Update configuration.rst
    Added documentation for webtoon thumbnail option
* extract thumbnails in GalleryExtractor.assets()
* simplify & fix flake8
* include 'type' in default filenames
* add test
* update docs

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
 2025-06-17 19:34:58 +02:00
Mike Fährmann
1f429da650 [scripts/options] make output width independent of terminal size 2025-06-17 18:52:46 +02:00
Mike Fährmann
c88d69376f [util] update code to 3.8 2025-06-17 17:06:00 +02:00
Mike Fährmann
6d928f3805 remove some pre-3.8 workarounds (#7671) 2025-06-17 12:56:47 +02:00
Mike Fährmann
d0b1da0fa7 [tiktok] handle exceptions when extracting avatars (#7682) 2025-06-17 12:21:20 +02:00
Mike Fährmann
eaed644b85 [build] update setup.py 
- increase required Python version to 3.8 (#7671)
- remove 3.4-3.7 classifiers
- add 'extra' optional dependency group
 2025-06-17 11:48:54 +02:00
Mike Fährmann
efa2ab5903 [common] allow GalleryExtractors to return additional asset files 
like thumbnails or covers
 2025-06-16 22:45:52 +02:00