github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
7,633 Commits 1 Branch 0 Tags
cf96fc6ebef9b4400a4195ca89d214c22016a2db
Commit Graph

306 Commits

Author SHA1 Message Date
Mike Fährmann
2cfe2b3b04 [common] implement '"user-agent": "+PRESET"' 
support using a 'User-Agent' header preset, e.g. "+firefox"
 2026-01-03 21:07:45 +01:00
Mike Fährmann
00c6821a3f replace 2-element f-strings with simple '+' concatenations 
Python's 'ast' module and its 'NodeVisitor' class
were incredibly helpful in identifying these
 2025-12-22 11:26:04 +01:00
Mike Fährmann
b1b70528a6 [common] use 'parent' value as 'parent-metadata' default (#8525 #8604) 
enable 'parent-metadata' by default for
'chevereto' & 'imagehost' (album/gallery) extractors
 2025-12-08 20:07:56 +01:00
Mike Fährmann
d1de48e7b2 remove Message.Version 2025-12-07 17:13:25 +01:00
Mike Fährmann
968597a302 yield 3-tuples for Message.Directory 
adapt tuples to the same length and semantics as other messages
 2025-12-05 21:39:52 +01:00
Mike Fährmann
89ece8f900 [common] allow general ISO 8601 values for 'date-min' & '-max' 
and parse them as UTC times instead of local time
 2025-11-09 23:00:09 +01:00
Mike Fährmann
62e9d75611 [common] fix 'AttributeError' 
fixes regression introduced in 98d3354575
 2025-10-29 11:38:26 +01:00
Mike Fährmann
da4b8c1e83 [wikimedia] fix 'limit' config lookup 
rework config handling in general
 2025-10-23 20:16:46 +02:00
Mike Fährmann
98d3354575 [wikimedia] implement config lookup for fandom/wikigg sites (#7283) 
{
    "extractor": {
        "fandom": {
            "filename": "..."
        }
    }
}
 2025-10-23 20:14:56 +02:00
Mike Fährmann
085616e0a8 [dt] replace 'text.parse_datetime()' & 'text.parse_timestamp()' 2025-10-17 17:43:06 +02:00
Mike Fährmann
69f7cfdd0c [dt] replace 'datetime' imports 2025-10-16 11:42:42 +02:00
Mike Fährmann
0eb3c8a994 [dt] replace 'util' datetime functions 2025-10-15 21:30:38 +02:00
Mike Fährmann
99d5c521d1 use 'encoding="utf-8"' when opening files in text mode (#8376) 2025-10-09 09:54:18 +02:00
Mike Fährmann
c497d96c3f [common] disable 'interval' delay for 'request_loocation()' 2025-10-03 22:29:03 +02:00
Mike Fährmann
0dfc3b18d3 [common] support single quotes in '_extract_jsonld' & '…_nextdata' 2025-09-11 20:21:55 +02:00
Mike Fährmann
977e1bece3 [common] add optional 'password' argument to '_get_auth_info()' 2025-09-09 22:06:13 +02:00
Mike Fährmann
5a5857bafb [common] prevent exception when using empty 'user-agent' (#8116) 
use '"headers": {"User-Agent": ""}' to send an empty User-Agent header
 2025-08-26 09:08:49 +02:00
Mike Fährmann
a358d13062 [common] add 'expected' argument to 'request()' 2025-08-22 16:09:31 +02:00
Mike Fährmann
ea80fc80c8 [common] improve error message for non-Netscape cookie files (#8014) 2025-08-13 08:03:01 +02:00
Mike Fährmann
5bc198a7e6 [common] try subprocess.Popen when webbrowser.get fails (#7947) 2025-08-08 21:25:42 +02:00
Mike Fährmann
16acfbd1e7 [common] implement '"user-agent": "@BROWSER"' (#7947) 
support automatically fetching the 'User-Agent' header
of a specific installed browser, e.g. "@firefox"
 2025-08-07 20:58:29 +02:00
Mike Fährmann
7f414cdb4e [common] add 'interval' argument to 'request()' 
to disable 'request_interval' sleeps for "background" requests
 2025-08-01 20:40:20 +02:00
Mike Fährmann
a097a373a9 simplify if statements by using walrus operators (#7671) 2025-07-22 20:57:54 +02:00
Mike Fährmann
629db56b12 [common] detect CF & DG challenges in request_json & _xml (#7833) 
even when called when fatal=False
 2025-07-17 17:17:01 +02:00
Mike Fährmann
e68dc960c2 [pixiv] add workaround for 'novel-*' includes (#7746) 2025-07-11 22:27:45 +02:00
Mike Fährmann
267661680c add 'pixiv-novel' compat categories (#7746) 2025-07-11 11:00:19 +02:00
Mike Fährmann
096bc6f784 replace more '.format(…)' calls with f-strings (#7671) 2025-07-10 21:17:17 +02:00
Mike Fährmann
d8ef1d693f rename 'StopExtraction' to 'AbortExtraction' 
for cases where StopExtraction was used to report errors
 2025-07-09 21:07:28 +02:00
Mike Fährmann
f17ed0569a [common] raise ChallengeError for CF & DDG challenges (#1945) 
this sets bit 0x08 in the exit status bitmask
 2025-07-09 18:58:21 +02:00
Mike Fährmann
05b0a25120 [common] update 'browser' user-agents and headers 
support Firefox 140 ESR and Chrome/Chromium 138
 2025-07-06 19:41:37 +02:00
Mike Fährmann
1bbacba4ed [common] introduce 'status' attribute to Extractors 
allows reporting error codes for exceptions that are not handled
by the Job.run() try-except block

- fixes Job.status being 0 in certain situations even when errors occurred
- fixes some URLs not getting written to -e/--error-file (#7758)
 2025-07-05 21:33:01 +02:00
Mike Fährmann
95338ff0ec replace 'result' with 'results' for lists 
more consistent names
 2025-06-30 12:10:57 +02:00
Mike Fährmann
9dbe33b6de replace old %-formatted and .format(…) strings with f-strings (#7671) 
mostly using flynt
https://github.com/ikamensh/flynt
 2025-06-29 17:50:19 +02:00
Mike Fährmann
d969dbbab1 [common] fix bug in df6f4e53 when first group is empty 2025-06-28 09:02:33 +02:00
Mike Fährmann
df6f4e5307 [common] only auto-set page_url when first group starts with / 2025-06-26 23:35:53 +02:00
Mike Fährmann
26e81e4162 [common] rename 'gallery_url'/'manga_url' to 'page_url 2025-06-26 22:06:57 +02:00
Mike Fährmann
b0d7de3603 support using system certificates via 'truststore' (#6582) 
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-2989290495
 2025-06-20 19:55:01 +02:00
Mike Fährmann
fcd1b8a155 [common] add a 'kwdict' member to extractor instances 
to allow setting general metadata at any point and without having to
rely on a manually implemented 'metadata()' method
 2025-06-19 19:08:35 +02:00
Mike Fährmann
a80d55d974 [common] improve 'user-agent' override logic 
Prevent a general 'user-agent' setting, i.e. extractor.user-agent, from
overriding the User-Agent header set by an extractor's 'browser'
option, default or otherwise, meaning only extractor-level or top-level
'user-agent' settings will override the 'browser' User-Agent.

https://github.com/mikf/gallery-dl/issues/7382#issuecomment-2985296321
 2025-06-19 18:58:39 +02:00
Mike Fährmann
41191bb60a 'match.group(N)' -> 'match[N]' (#7671) 
2.5x faster
 2025-06-18 13:05:58 +02:00
Mike Fährmann
efa2ab5903 [common] allow GalleryExtractors to return additional asset files 
like thumbnails or covers
 2025-06-16 22:45:52 +02:00
Mike Fährmann
e08ec7e083 update copyright notices 2025-06-13 00:03:41 +02:00
Mike Fährmann
b4aed5e2c9 [common] allow overriding 'user-agent' when 'browser' is used (#7647) 2025-06-10 22:05:28 +02:00
Mike Fährmann
4cfddc144a [common] import 'datetime' class directly 2025-06-10 21:35:15 +02:00
Mike Fährmann
e68555defa [common] improve cookie-related logging messages 2025-06-10 21:34:27 +02:00
Mike Fährmann
511cf2363c [common] update expired cookie messages (#7644) 
- prefix with 'cookies:'
- include domain
- include exact time when it expired
 2025-06-09 18:48:04 +02:00
Mike Fährmann
72a01bc4d4 [common] use util.re_compile() in _dump_response 2025-06-05 15:24:22 +02:00
Mike Fährmann
efd49aef73 allow using predefined Firefox/Chrome 'headers' & 'ciphers' 2025-06-05 14:24:38 +02:00
Mike Fährmann
85124fe251 [common] add 'request_json()' convenience function 2025-06-05 09:13:25 +02:00
Mike Fährmann
a7bbccbd7b [common] add 'request_xml()' convenience function 2025-06-04 23:10:16 +02:00