github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
7,015 Commits 1 Branch 0 Tags
ba174b1a06754db4a8a44dd30c8a1c10afc121a6
Commit Graph

285 Commits

Author SHA1 Message Date
Mike Fährmann
7f414cdb4e [common] add 'interval' argument to 'request()' 
to disable 'request_interval' sleeps for "background" requests
 2025-08-01 20:40:20 +02:00
Mike Fährmann
a097a373a9 simplify if statements by using walrus operators (#7671) 2025-07-22 20:57:54 +02:00
Mike Fährmann
629db56b12 [common] detect CF & DG challenges in request_json & _xml (#7833) 
even when called when fatal=False
 2025-07-17 17:17:01 +02:00
Mike Fährmann
e68dc960c2 [pixiv] add workaround for 'novel-*' includes (#7746) 2025-07-11 22:27:45 +02:00
Mike Fährmann
267661680c add 'pixiv-novel' compat categories (#7746) 2025-07-11 11:00:19 +02:00
Mike Fährmann
096bc6f784 replace more '.format(…)' calls with f-strings (#7671) 2025-07-10 21:17:17 +02:00
Mike Fährmann
d8ef1d693f rename 'StopExtraction' to 'AbortExtraction' 
for cases where StopExtraction was used to report errors
 2025-07-09 21:07:28 +02:00
Mike Fährmann
f17ed0569a [common] raise ChallengeError for CF & DDG challenges (#1945) 
this sets bit 0x08 in the exit status bitmask
 2025-07-09 18:58:21 +02:00
Mike Fährmann
05b0a25120 [common] update 'browser' user-agents and headers 
support Firefox 140 ESR and Chrome/Chromium 138
 2025-07-06 19:41:37 +02:00
Mike Fährmann
1bbacba4ed [common] introduce 'status' attribute to Extractors 
allows reporting error codes for exceptions that are not handled
by the Job.run() try-except block

- fixes Job.status being 0 in certain situations even when errors occurred
- fixes some URLs not getting written to -e/--error-file (#7758)
 2025-07-05 21:33:01 +02:00
Mike Fährmann
95338ff0ec replace 'result' with 'results' for lists 
more consistent names
 2025-06-30 12:10:57 +02:00
Mike Fährmann
9dbe33b6de replace old %-formatted and .format(…) strings with f-strings (#7671) 
mostly using flynt
https://github.com/ikamensh/flynt
 2025-06-29 17:50:19 +02:00
Mike Fährmann
d969dbbab1 [common] fix bug in df6f4e53 when first group is empty 2025-06-28 09:02:33 +02:00
Mike Fährmann
df6f4e5307 [common] only auto-set page_url when first group starts with / 2025-06-26 23:35:53 +02:00
Mike Fährmann
26e81e4162 [common] rename 'gallery_url'/'manga_url' to 'page_url 2025-06-26 22:06:57 +02:00
Mike Fährmann
b0d7de3603 support using system certificates via 'truststore' (#6582) 
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-2989290495
 2025-06-20 19:55:01 +02:00
Mike Fährmann
fcd1b8a155 [common] add a 'kwdict' member to extractor instances 
to allow setting general metadata at any point and without having to
rely on a manually implemented 'metadata()' method
 2025-06-19 19:08:35 +02:00
Mike Fährmann
a80d55d974 [common] improve 'user-agent' override logic 
Prevent a general 'user-agent' setting, i.e. extractor.user-agent, from
overriding the User-Agent header set by an extractor's 'browser'
option, default or otherwise, meaning only extractor-level or top-level
'user-agent' settings will override the 'browser' User-Agent.

https://github.com/mikf/gallery-dl/issues/7382#issuecomment-2985296321
 2025-06-19 18:58:39 +02:00
Mike Fährmann
41191bb60a 'match.group(N)' -> 'match[N]' (#7671) 
2.5x faster
 2025-06-18 13:05:58 +02:00
Mike Fährmann
efa2ab5903 [common] allow GalleryExtractors to return additional asset files 
like thumbnails or covers
 2025-06-16 22:45:52 +02:00
Mike Fährmann
e08ec7e083 update copyright notices 2025-06-13 00:03:41 +02:00
Mike Fährmann
b4aed5e2c9 [common] allow overriding 'user-agent' when 'browser' is used (#7647) 2025-06-10 22:05:28 +02:00
Mike Fährmann
4cfddc144a [common] import 'datetime' class directly 2025-06-10 21:35:15 +02:00
Mike Fährmann
e68555defa [common] improve cookie-related logging messages 2025-06-10 21:34:27 +02:00
Mike Fährmann
511cf2363c [common] update expired cookie messages (#7644) 
- prefix with 'cookies:'
- include domain
- include exact time when it expired
 2025-06-09 18:48:04 +02:00
Mike Fährmann
72a01bc4d4 [common] use util.re_compile() in _dump_response 2025-06-05 15:24:22 +02:00
Mike Fährmann
efd49aef73 allow using predefined Firefox/Chrome 'headers' & 'ciphers' 2025-06-05 14:24:38 +02:00
Mike Fährmann
85124fe251 [common] add 'request_json()' convenience function 2025-06-05 09:13:25 +02:00
Mike Fährmann
a7bbccbd7b [common] add 'request_xml()' convenience function 2025-06-04 23:10:16 +02:00
Mike Fährmann
75b6c8f3d8 re-implement 'category-map' (#7612) 2025-06-04 07:57:27 +02:00
Mike Fährmann
8dc5794972 [common] return NullResponse for non-fatal requests (#7598) 
Make Extractor.request(…, fatal=False) actually non-fatal
by returning an empty response instead of raising an exception
when a request fails due to connection issues.
 2025-05-29 20:31:41 +02:00
Mike Fährmann
ed9c960bb9 [kemonoparty] remove '_prepare_ddosguard_cookies()' 2025-05-24 18:05:03 +02:00
Mike Fährmann
e199396872 [common] simplify 'user' extractors by using 'Dispatch' mixin 2025-05-24 18:04:53 +02:00
Mike Fährmann
88f1541a83 [common] add 'request_location()' convenience function 2025-04-19 16:45:05 +02:00
Mike Fährmann
4c8c98a14d use internal, non-caching version of re.compile for extractor patterns 
speeds up total compile time of extractor patterns by ~10ms
 2025-04-15 22:47:19 +02:00
Mike Fährmann
567f5d0bc6 [common] add 'subdomains' argument to 'cookies_check()' (#7188) 2025-03-18 14:59:41 +01:00
Mike Fährmann
c9488cee30 [util] move Cloudflare/DDoS-Guard detection into 'detect_challenge()' 2025-03-06 14:21:35 +01:00
Mike Fährmann
ddb2c4d69d [executables] fix SSLError when using HTTPAdapter (#6393) 
always load certifi certificates instead of relying on
'load_default_certs()', which might load no certs at all
 2025-01-31 20:36:41 +01:00
Mike Fährmann
1ae3ac5e39 [common] add '_extract_nextdata' method 2025-01-12 11:48:36 +01:00
Mike Fährmann
3f48e2f820 [common] add '_extract_jsonld' method (#5272) 2025-01-12 11:07:48 +01:00
Mike Fährmann
4853406fe3 [common] allow MangaExtractors to skip loading manga_url 2025-01-10 21:30:58 +01:00
Mike Fährmann
041baf8441 [common] compute and use latest Firefox UA 
instead of the latest ESR UA
 2024-12-17 22:20:37 +01:00
Mike Fährmann
0802e42c90 [common] use random unused port for '"user-agent": "browser"' 2024-12-17 21:40:20 +01:00
Mike Fährmann
e8826ed3d4 [common] simplify HTTP error messages 
[warning] HTTPSConnectionPool(host='domain.tld', port=443): Max retries
exceeded with url: /a.jpg (Caused by NameResolutionError("<urllib3.
connection.HTTPSConnection object at 0x7247fe436ea0>: Failed to resolve
'domain.tld' ([Errno -2] Name or service not known)")) (1/5)

->

[warning] NameResolutionError: Failed to resolve 'domain.tld'
([Errno -2] Name or service not known) (1/5)
 2024-12-10 17:13:44 +01:00
Mike Fährmann
86f3f3f763 [common] detect DDoS-Guard challenge pages 2024-12-08 21:39:04 +01:00
Mike Fährmann
7091904b20 [common] restore using environment proxies by default (#6553, #6609) 
change 'proxy-env' default to 'true'
 2024-12-07 17:38:44 +01:00
Mike Fährmann
57f8227473 [common] improve handling of 'user-agent' settings (#6594) 
improves 5412b22dae

ignore 'extractor.user-agent' only for extractors using a custom
'User-Agent' header
 2024-12-03 10:55:41 +01:00
Mike Fährmann
5412b22dae [common] allow overriding more default 'User-Agent' headers (#6496) 
ignore 'extractor.user-agent' if it is the default useragent value
and an extractor wants to set its own custom value
 2024-11-26 21:50:28 +01:00
Mike Fährmann
c82f3db098 [common] add 'proxy-env' option 
(#6134, #6455)
disable using environment proxies by default
 2024-11-15 18:03:56 +01:00
Mike Fährmann
0a72a5009c [common] disable Authorization header injection from .netrc auth 
(#6134, #6455)
 2024-11-15 17:37:04 +01:00