github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
7,807 Commits 1 Branch 0 Tags
f0f9575406b305110a665b801414a00a6be394d7
Commit Graph

286 Commits

Author SHA1 Message Date
Mike Fährmann
f0f9575406 [job] fix 'AttributeError' when enabling 'init' for non-DownloadJob 
fixes bug in 56dcd00391
 2026-02-03 19:00:45 +01:00
Mike Fährmann
0c04090ff4 [util] set flags in range predicates 
use flags to stop extractors immediately when hitting the upper range
limit instead of raising an exception one predicate check later
 2026-02-02 21:48:45 +01:00
Mike Fährmann
87323fd71e rename 'image-*' & 'chapter-*' options to 'file-*' & 'child-*' 
keep support for both names, e.g. file-filter & image-filter,
while prioritizing the new names
 2026-02-02 18:01:45 +01:00
Mike Fährmann
c42a5dce5c [pp] add 'child' & 'child-after' events 2026-02-01 22:14:14 +01:00
Mike Fährmann
56dcd00391 [job] add 'init' option 2026-02-01 20:48:37 +01:00
Mike Fährmann
3445c51ca4 [job] add 'output.jsonl' option (#8953) 2026-01-30 09:36:28 +01:00
Mike Fährmann
eaaa25b6e4 [job] enable all 'parent-…' options for parent extractors by default 
- parent-directory
- parent-metadata
- parent-session
- parent-skip

- add general 'parent' option
 2026-01-27 12:05:19 +01:00
Mike Fährmann
4798ac4836 [common] implement 'parent-session' 2026-01-21 20:33:08 +01:00
Mike Fährmann
c23beee57c [util] use functions for predicates 
more lightweight and faster than classes
 2026-01-18 20:32:36 +01:00
Mike Fährmann
205a448060 [job] fix extractor 'kwdict' values being unavailable in filters 2026-01-13 16:44:50 +01:00
Mike Fährmann
19a27a47b3 [options] add '--sleep-skip' option (#6582) 
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-3677460485
 2025-12-20 19:06:11 +01:00
Mike Fährmann
b1b70528a6 [common] use 'parent' value as 'parent-metadata' default (#8525 #8604) 
enable 'parent-metadata' by default for
'chevereto' & 'imagehost' (album/gallery) extractors
 2025-12-08 20:07:56 +01:00
Mike Fährmann
968597a302 yield 3-tuples for Message.Directory 
adapt tuples to the same length and semantics as other messages
 2025-12-05 21:39:52 +01:00
Mike Fährmann
5da200945b implement 'post-filter' & 'post-range' options 
(#8643)
 2025-12-05 16:40:23 +01:00
Mike Fährmann
d9c1d15aee [job] inline 'dispatch' loop 2025-12-05 11:48:51 +01:00
Mike Fährmann
5cda64c8d8 [job] apply 'keywords-eval' to 'keywords-global' values (#6582) 
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-3527246051
 2025-11-13 20:29:31 +01:00
Mike Fährmann
691775d0fb [job] add 'keywords-global' option (#6582) 
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-3505732704
 2025-11-12 18:48:01 +01:00
Mike Fährmann
98d3354575 [wikimedia] implement config lookup for fandom/wikigg sites (#7283) 
{
    "extractor": {
        "fandom": {
            "filename": "..."
        }
    }
}
 2025-10-23 20:14:56 +02:00
Mike Fährmann
8c62be343e [output] add 'Logger.traceback()' helper 2025-10-14 18:44:29 +02:00
Mike Fährmann
cda63b94bb [job] use identity checks 
micro optimizations...
 2025-10-14 12:12:41 +02:00
Mike Fährmann
899dcc62cf [job] implement '"archive-event": "after"' (#8373) 2025-10-14 11:13:17 +02:00
Mike Fährmann
723d0dfa03 [job] reorder 'finalize' hooks (#8243) 
prevent 'sqlite3.ProgrammingError: Cannot operate on a closed database.'
when using '"event": "finalize-success"' or '"event": "finalize-error"'
with an archive after e5db6efca9
 2025-09-20 09:10:22 +02:00
Mike Fährmann
20fb1d9214 [job] split collected DataJob results 2025-08-03 19:14:01 +02:00
Mike Fährmann
55977e2be3 more f-string usage (#7671) 2025-08-03 11:36:28 +02:00
Mike Fährmann
491d70f918 [job] apply 'extension-map' to 'SimulationJob' results 2025-08-02 07:28:12 +02:00
Mike Fährmann
2eb5e52055 extend '-A / --abort' & '"skip": "abort"' functionality (#7891) 
implement ascending by more than 1 level or
up to an extractor with a specific subcategory
 2025-07-30 00:01:49 +02:00
Mike Fährmann
64de6605ce [job] improve URL 'scheme' extraction performance 2025-07-29 22:26:21 +02:00
Mike Fährmann
a097a373a9 simplify if statements by using walrus operators (#7671) 2025-07-22 20:57:54 +02:00
Mike Fährmann
dd09937d69 fix exit code for requests' JSONDecodeError (#4380) 2025-07-17 16:37:40 +02:00
Mike Fährmann
232e30f64e [actions] fix 'parse_logging' import (#7837) 
fixes regressions introduced in bccf467d19
 2025-07-17 16:30:10 +02:00
Mike Fährmann
98895b732f [reddit] improve archive IDs of fallback files (#7760) 
prevent 'DASH...' and 'HLS...' entries
 2025-07-11 22:59:44 +02:00
Mike Fährmann
d8a370da0b [signals] update FLAGS handling 2025-07-11 22:28:26 +02:00
Mike Fährmann
0210ffcdd8 initial 'signals-actions' implementation (#6582) 
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-2973285775

To stop gracefully after the current file finishes processing when
Ctrl+C was pressed, or after the current post finishes processing when
SIGUSR1 was received:

{
    "signals-actions": {
        "SIGINT" : "file",
        "SIGUSR1": "post"
    }
}
 2025-07-09 23:02:23 +02:00
Mike Fährmann
d8ef1d693f rename 'StopExtraction' to 'AbortExtraction' 
for cases where StopExtraction was used to report errors
 2025-07-09 21:07:28 +02:00
Mike Fährmann
4e9cb428d6 [pp] implement shortcuts for 'mode' and 'event' options 
This makes it possible to specify 'mode' and/or 'event' options of a
postprocessor in its 'name' as
"NAME/MODE@EVENT" or "NAME/MODE" or "NAME@EVENT"

For example
"postprocessors": "metadata/jsonl@file,skip"

is equivalent to
"postprocessors": {
    "name" : "metadata",
    "mode" : "jsonl",
    "event": ["file", "skip"]
}
 2025-07-09 12:40:37 +02:00
Mike Fährmann
1bbacba4ed [common] introduce 'status' attribute to Extractors 
allows reporting error codes for exceptions that are not handled
by the Job.run() try-except block

- fixes Job.status being 0 in certain situations even when errors occurred
- fixes some URLs not getting written to -e/--error-file (#7758)
 2025-07-05 21:33:01 +02:00
Mike Fährmann
59b266f883 [reddit] emit logging message when downloading previews (#7748) 2025-06-29 21:35:51 +02:00
Mike Fährmann
9dbe33b6de replace old %-formatted and .format(…) strings with f-strings (#7671) 
mostly using flynt
https://github.com/ikamensh/flynt
 2025-06-29 17:50:19 +02:00
Mike Fährmann
0334a7c48c [job] apply 'update_kwdict()' to Message.Queue metadata as well 2025-06-26 19:15:44 +02:00
Mike Fährmann
d7103f9bdd [job:data] wrap exceptions in a dict (#7723) 
fixes exception when using 'num-to-str' with '-j'
 2025-06-23 17:17:28 +02:00
Mike Fährmann
fcd1b8a155 [common] add a 'kwdict' member to extractor instances 
to allow setting general metadata at any point and without having to
rely on a manually implemented 'metadata()' method
 2025-06-19 19:08:35 +02:00
Mike Fährmann
e08ec7e083 update copyright notices 2025-06-13 00:03:41 +02:00
Mike Fährmann
811b665e33 remove @staticmethod decorators 
There might have been a time when calling a static method was faster
than a regular method, but that is no longer the case. According to
micro-benchmarks, it is 70% slower in CPython 3.13 and it also makes
executing the code of a class definition slower.
 2025-06-12 22:50:52 +02:00
Mike Fährmann
33f3ed9f57 [job] refactor parent-child config paths (#7527) 
- fixes TypeError when enabling 'category-transfer'
- fixes 'category-transfer' not applying to early config lookups
 2025-05-31 17:41:54 +02:00
Mike Fährmann
19fc4e0ba4 [job] do not reset skip count when 'skip-filter' fails (#7433) 2025-04-27 19:16:02 +02:00
Mike Fährmann
8daf496a22 [archive] add 'archive-table' option (#6152) 2025-02-17 11:41:13 +01:00
Mike Fährmann
841bc9f66f [archive] implement support for PostgreSQL databases (#6152) 2025-02-16 17:56:52 +01:00
Mike Fährmann
5ab2ae17bc support wildcards for parent>child categories (#6673) 
For example "reddit>*" for all reddit child extractors
 2024-12-16 08:50:18 +01:00
Mike Fährmann
d8cf381904 [archive] use defaults when 'prefix'/'format' are 'null' 2024-11-29 16:36:35 +01:00
Mike Fährmann
55afd712d6 [pp] allow inheriting settings from global 'postprocessor' entries 
No idea how to properly explain/document this, so here's an example:

The extractor.postprocessors object
gets its options from postprocessor.jl
and adds 'filename' itself.

{
    "extractor": {
        "postprocessors": {
            "type": "jl",
            "filename": "meta.jsonl"
        }
    },

    "postprocessor": {
        "jl": {
            "name": "metadata",
            "mode": "jsonl",
            "open": "a"
        }
    }
}
 2024-11-16 21:16:13 +01:00