github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
7,891 Commits 1 Branch 0 Tags
master
Commit Graph

294 Commits

Author SHA1 Message Date
Mike Fährmann
b174bb412a [job] improve 'metadata_url' handling 2026-02-07 21:47:17 +01:00
Mike Fährmann
2d64e76223 [job] implement 'follow' option (#8752) 
Follow and process URLs found in the given format string result.
 2026-02-07 21:47:17 +01:00
Mike Fährmann
7a98a93a8e [common] only call 'skip()' & 'finalize()' when defined 2026-02-07 21:47:17 +01:00
Mike Fährmann
40e4cc62c4 [common] pass job status to 'finalize()' 2026-02-07 21:47:17 +01:00
Mike Fährmann
71680feab9 [actions] implement 'flag … = skip' (#8960) 2026-02-06 10:10:38 +01:00
Mike Fährmann
b4351b8193 [job] more 'is None' checks 2026-02-04 11:10:49 +01:00
Mike Fährmann
49b2fa4b8e [job] fix overwriting '_extractor' (#8958) 2026-02-04 11:10:49 +01:00
Mike Fährmann
2cbd9740f2 enable 'metadata-url/-path/-parent/-extractor]' by default 
assign values to "private" keys to not have them appear in the
standard JSON output, while still making it easier to access them
 2026-02-03 19:28:33 +01:00
Mike Fährmann
f0f9575406 [job] fix 'AttributeError' when enabling 'init' for non-DownloadJob 
fixes bug in 56dcd00391
 2026-02-03 19:00:45 +01:00
Mike Fährmann
0c04090ff4 [util] set flags in range predicates 
use flags to stop extractors immediately when hitting the upper range
limit instead of raising an exception one predicate check later
 2026-02-02 21:48:45 +01:00
Mike Fährmann
87323fd71e rename 'image-*' & 'chapter-*' options to 'file-*' & 'child-*' 
keep support for both names, e.g. file-filter & image-filter,
while prioritizing the new names
 2026-02-02 18:01:45 +01:00
Mike Fährmann
c42a5dce5c [pp] add 'child' & 'child-after' events 2026-02-01 22:14:14 +01:00
Mike Fährmann
56dcd00391 [job] add 'init' option 2026-02-01 20:48:37 +01:00
Mike Fährmann
3445c51ca4 [job] add 'output.jsonl' option (#8953) 2026-01-30 09:36:28 +01:00
Mike Fährmann
eaaa25b6e4 [job] enable all 'parent-…' options for parent extractors by default 
- parent-directory
- parent-metadata
- parent-session
- parent-skip

- add general 'parent' option
 2026-01-27 12:05:19 +01:00
Mike Fährmann
4798ac4836 [common] implement 'parent-session' 2026-01-21 20:33:08 +01:00
Mike Fährmann
c23beee57c [util] use functions for predicates 
more lightweight and faster than classes
 2026-01-18 20:32:36 +01:00
Mike Fährmann
205a448060 [job] fix extractor 'kwdict' values being unavailable in filters 2026-01-13 16:44:50 +01:00
Mike Fährmann
19a27a47b3 [options] add '--sleep-skip' option (#6582) 
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-3677460485
 2025-12-20 19:06:11 +01:00
Mike Fährmann
b1b70528a6 [common] use 'parent' value as 'parent-metadata' default (#8525 #8604) 
enable 'parent-metadata' by default for
'chevereto' & 'imagehost' (album/gallery) extractors
 2025-12-08 20:07:56 +01:00
Mike Fährmann
968597a302 yield 3-tuples for Message.Directory 
adapt tuples to the same length and semantics as other messages
 2025-12-05 21:39:52 +01:00
Mike Fährmann
5da200945b implement 'post-filter' & 'post-range' options 
(#8643)
 2025-12-05 16:40:23 +01:00
Mike Fährmann
d9c1d15aee [job] inline 'dispatch' loop 2025-12-05 11:48:51 +01:00
Mike Fährmann
5cda64c8d8 [job] apply 'keywords-eval' to 'keywords-global' values (#6582) 
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-3527246051
 2025-11-13 20:29:31 +01:00
Mike Fährmann
691775d0fb [job] add 'keywords-global' option (#6582) 
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-3505732704
 2025-11-12 18:48:01 +01:00
Mike Fährmann
98d3354575 [wikimedia] implement config lookup for fandom/wikigg sites (#7283) 
{
    "extractor": {
        "fandom": {
            "filename": "..."
        }
    }
}
 2025-10-23 20:14:56 +02:00
Mike Fährmann
8c62be343e [output] add 'Logger.traceback()' helper 2025-10-14 18:44:29 +02:00
Mike Fährmann
cda63b94bb [job] use identity checks 
micro optimizations...
 2025-10-14 12:12:41 +02:00
Mike Fährmann
899dcc62cf [job] implement '"archive-event": "after"' (#8373) 2025-10-14 11:13:17 +02:00
Mike Fährmann
723d0dfa03 [job] reorder 'finalize' hooks (#8243) 
prevent 'sqlite3.ProgrammingError: Cannot operate on a closed database.'
when using '"event": "finalize-success"' or '"event": "finalize-error"'
with an archive after e5db6efca9
 2025-09-20 09:10:22 +02:00
Mike Fährmann
20fb1d9214 [job] split collected DataJob results 2025-08-03 19:14:01 +02:00
Mike Fährmann
55977e2be3 more f-string usage (#7671) 2025-08-03 11:36:28 +02:00
Mike Fährmann
491d70f918 [job] apply 'extension-map' to 'SimulationJob' results 2025-08-02 07:28:12 +02:00
Mike Fährmann
2eb5e52055 extend '-A / --abort' & '"skip": "abort"' functionality (#7891) 
implement ascending by more than 1 level or
up to an extractor with a specific subcategory
 2025-07-30 00:01:49 +02:00
Mike Fährmann
64de6605ce [job] improve URL 'scheme' extraction performance 2025-07-29 22:26:21 +02:00
Mike Fährmann
a097a373a9 simplify if statements by using walrus operators (#7671) 2025-07-22 20:57:54 +02:00
Mike Fährmann
dd09937d69 fix exit code for requests' JSONDecodeError (#4380) 2025-07-17 16:37:40 +02:00
Mike Fährmann
232e30f64e [actions] fix 'parse_logging' import (#7837) 
fixes regressions introduced in bccf467d19
 2025-07-17 16:30:10 +02:00
Mike Fährmann
98895b732f [reddit] improve archive IDs of fallback files (#7760) 
prevent 'DASH...' and 'HLS...' entries
 2025-07-11 22:59:44 +02:00
Mike Fährmann
d8a370da0b [signals] update FLAGS handling 2025-07-11 22:28:26 +02:00
Mike Fährmann
0210ffcdd8 initial 'signals-actions' implementation (#6582) 
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-2973285775

To stop gracefully after the current file finishes processing when
Ctrl+C was pressed, or after the current post finishes processing when
SIGUSR1 was received:

{
    "signals-actions": {
        "SIGINT" : "file",
        "SIGUSR1": "post"
    }
}
 2025-07-09 23:02:23 +02:00
Mike Fährmann
d8ef1d693f rename 'StopExtraction' to 'AbortExtraction' 
for cases where StopExtraction was used to report errors
 2025-07-09 21:07:28 +02:00
Mike Fährmann
4e9cb428d6 [pp] implement shortcuts for 'mode' and 'event' options 
This makes it possible to specify 'mode' and/or 'event' options of a
postprocessor in its 'name' as
"NAME/MODE@EVENT" or "NAME/MODE" or "NAME@EVENT"

For example
"postprocessors": "metadata/jsonl@file,skip"

is equivalent to
"postprocessors": {
    "name" : "metadata",
    "mode" : "jsonl",
    "event": ["file", "skip"]
}
 2025-07-09 12:40:37 +02:00
Mike Fährmann
1bbacba4ed [common] introduce 'status' attribute to Extractors 
allows reporting error codes for exceptions that are not handled
by the Job.run() try-except block

- fixes Job.status being 0 in certain situations even when errors occurred
- fixes some URLs not getting written to -e/--error-file (#7758)
 2025-07-05 21:33:01 +02:00
Mike Fährmann
59b266f883 [reddit] emit logging message when downloading previews (#7748) 2025-06-29 21:35:51 +02:00
Mike Fährmann
9dbe33b6de replace old %-formatted and .format(…) strings with f-strings (#7671) 
mostly using flynt
https://github.com/ikamensh/flynt
 2025-06-29 17:50:19 +02:00
Mike Fährmann
0334a7c48c [job] apply 'update_kwdict()' to Message.Queue metadata as well 2025-06-26 19:15:44 +02:00
Mike Fährmann
d7103f9bdd [job:data] wrap exceptions in a dict (#7723) 
fixes exception when using 'num-to-str' with '-j'
 2025-06-23 17:17:28 +02:00
Mike Fährmann
fcd1b8a155 [common] add a 'kwdict' member to extractor instances 
to allow setting general metadata at any point and without having to
rely on a manually implemented 'metadata()' method
 2025-06-19 19:08:35 +02:00
Mike Fährmann
e08ec7e083 update copyright notices 2025-06-13 00:03:41 +02:00