github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,001 Commits 1 Branch 0 Tags
8d124a3766c633afd9d8f00aac2a671a2a42978b
Commit Graph

155 Commits

Author SHA1 Message Date
Mike Fährmann
bf241811dd allow '_extractor' fields to be None or empty 2021-03-20 01:19:31 +01:00
Mike Fährmann
23641742a3 improve 'parent-directory' (#1364) 
Allow forwarding metadata from the top-level extractor to all children
if 'parent-directory' is enabled for all extractors along the way.

For example 'reddit' -> 'gfycat' -> 'redgifs'
 2021-03-14 17:19:57 +01:00
Mike Fährmann
df94182e11 implement 'parent-metadata' option (#1364) 
experimental, might not work as expected, etc.
 2021-03-11 01:10:34 +01:00
Mike Fährmann
b6719becf1 ensure '-s/--simulate' always prints filenames (#1360) 
by assuming a potentially wrong filename extension in cases where the
correct one would only get known after a download started
 2021-03-07 22:38:20 +01:00
Mike Fährmann
c963741860 add '-E/--extractor-info' command-line option (#875) 2021-03-02 23:59:56 +01:00
Mike Fährmann
65ca923b4e fix 'whitelist' option for BaseExtractor instances 2021-02-15 21:58:33 +01:00
Mike Fährmann
56a8968435 remove 'Message.Metadata' (#866) 2021-01-31 02:12:37 +01:00
Mike Fährmann
46323ae6ff initialize 'hooks' as empty tuple 
follow-up to 9c29fc4e

Prevent a "race" between initializing 'pathfmt' and 'hooks',
and receiving a signal in between (e.g. ctrl+c),
which would then crash in 'handle_finalize()'.
 2020-11-28 18:18:49 +01:00
Mike Fährmann
9c29fc4e55 always initialize DownloadJob.hooks (fixes #1135) 
and not just when any (potential) post processors are defined
 2020-11-28 00:09:19 +01:00
Mike Fährmann
9fffa9c343 rework post processor callbacks 2020-11-19 02:29:06 +01:00
Mike Fährmann
f99c6031e0 apply post processor blacklists/whitelists to basecategories 
(#1103)
 2020-11-17 02:02:31 +01:00
Mike Fährmann
a3ca2f6080 update fallback URL handling 
remove Message.Urllist and use a '_fallback' field inside a kwdict
 2020-10-16 01:09:55 +02:00
Mike Fährmann
fd20093c96 allow blacklist/whitelist to be empty lists/strings (#1051) 2020-10-08 14:55:21 +02:00
Mike Fährmann
d5fa716d89 fix crash when using 'skip=false' and archive (fixes #1023) 
Separating the archive check from pathfmt.exists() in b5243297
had some unintended side effects.

It is also not possible to monkey-patch a dunder method like
__contains__ because of the special method lookup that gets
performed for them.
 2020-09-23 19:07:40 +02:00
Mike Fährmann
231dd4c800 accumulate postprocessor objects (#994) 
Instead of one 'postprocessors' setting overwriting all others lower
in the hierarchy, all postprocessors along the config path will now
get collected into one big list.

For example '--mtime-from-date' will therefore no longer cause
other postprocessor settings in a config file to get ignored.
 2020-09-14 21:51:55 +02:00
Mike Fährmann
3afd362e2e add 'sleep-extractor' option (closes #964) 
(would have been nice if this were possible without code duplication)
 2020-09-12 21:04:47 +02:00
Mike Fährmann
c78aa17506 add general 'blacklist' and 'whitelist' options (#492, #844) 2020-09-11 13:17:12 +02:00
Mike Fährmann
5912727b88 support format string replacement fields in archive paths 
(closes #985)
 2020-09-10 22:09:30 +02:00
Mike Fährmann
b5243297ff write skipped files to archive (closes #550) 2020-09-03 18:37:38 +02:00
Mike Fährmann
3f73cc6855 allow 'parent-directory' to work recursively (fixes #905) 2020-07-29 00:31:23 +02:00
Mike Fährmann
d5bfb0b38c set pseudo extension for Metadata messages (#865) 
This prevents pathfmt.filename from potentially being empty.
 2020-07-04 22:14:39 +02:00
Mike Fährmann
1b3870a4be flush after writing JSON in DataJob() (#727) 
… and remove the dead handle_finalize() method,
which is never called since DataJob() overrides run().
 2020-06-19 23:05:44 +02:00
Mike Fährmann
7e8a747c56 improve output of '-K' for parent extractors 2 (#825) 
This is what bb882b8 was supposed to be, but I managed to
not include those changes in the first commit …
 2020-06-18 15:04:15 +02:00
Mike Fährmann
ece73b5b2a make 'path' and 'keywords' available in logging messages 
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.

__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.

(#574, #575)
 2020-05-18 19:04:51 +02:00
Mike Fährmann
a1e739b96c reuse connection adapters from parent extractors 2020-05-12 23:52:01 +02:00
Mike Fährmann
42f29c3e11 improve and simplify attribute access in DownloadJob.initialize() 2020-05-09 00:57:59 +02:00
Mike Fährmann
56f1c96168 implement 'parent-directory' option (#551) 2020-01-29 18:32:37 +01:00
Mike Fährmann
37247dbaff miscellaneous fixes 2020-01-19 22:53:06 +01:00
Mike Fährmann
0e9dc5c88e fix AttributeError when accessing 'temppath' 
[ci skip]
 2020-01-19 00:41:21 +01:00
Mike Fährmann
0b84068d84 remove temp files before downloading from fallback URLs 
otherwise the next call to download() with a fallback URL could see
the partially downloaded "remains" from the previous, failed download
attempt and "continue" it, writing the second half of a potentially
different version of that file.
 2020-01-18 00:47:17 +01:00
Mike Fährmann
2d4887b75b improve KeywordJob output for "parent" extractors (closes #548) 2019-12-28 22:26:49 +01:00
Mike Fährmann
2e2fc7f0ad prevent infinite recursion when spawning extractors (closes #489) 2019-12-26 23:38:16 +01:00
Mike Fährmann
1921c127a5 make OSErrors during file downloads nonfatal (closes #512) 
… except ENOSPC (No space left on device), since there is no reason to
continue downloading in that case.

All other errors that would prevent downloading data and writing it to
disk get already raised during directory creation and are therefore not
checked here.
 2019-12-19 18:34:05 +01:00
Mike Fährmann
63e6993716 merge 'bypost' functionality into metadata postprocessor 2019-12-16 17:19:23 +01:00
Gio
c0b9ad678d Separate metadata from handle_url into handle_metadata, commenting 2019-12-09 16:02:15 -06:00
Gio
6ed4fc07ff Don't print intentional metadata skips to the console. 2019-12-09 01:02:17 -06:00
Gio
cfc70a97ab Added an additional channel for downloading the metadata of an entire post or gallery. 2019-12-09 00:56:27 -06:00
Mike Fährmann
f5604492c3 update interface of config functions 2019-11-24 00:42:28 +01:00
Mike Fährmann
3fc1e12949 [postprocessor:metadata] filter private entries 
i.e. keys starting with an underscore
 2019-11-21 16:58:44 +01:00
Mike Fährmann
9e88e7a344 [postprocessor:exec] improve (#421, #413) 
- add 'final' option
- include job status in pp finalization
- improve and extend documentation
 2019-11-03 21:45:45 +01:00
Mike Fährmann
5af291ba5c include failed downloads and child extractors in exit status 2019-10-29 15:56:54 +01:00
Mike Fährmann
322c2e7ed4 renaming variables 
mostly 'keyword(s)' to 'kwdict'
 2019-10-29 15:46:35 +01:00
Mike Fährmann
4409d00141 embed error messages in StopExtraction exceptions 2019-10-28 16:39:49 +01:00
Mike Fährmann
c887493a80 overhaul exception stuff 2019-10-27 23:53:37 +01:00
Mike Fährmann
389d2d7e38 implement 'cookies-update' option (#445) 2019-10-19 15:23:55 +02:00
Mike Fährmann
03bc8adfc7 [postprocessor:exec] run after file moved to target location 
(#421)
 2019-10-06 23:12:22 +02:00
Mike Fährmann
776e9e073f close archive on job completion (#417) 2019-09-10 22:43:51 +02:00
Mike Fährmann
9178b54eae handle errors when opening download archive file (#417) 2019-09-10 16:44:47 +02:00
Mike Fährmann
682105b8ee prevent crash when loading unavailable downloader (#405) 2019-08-31 21:58:33 +02:00
Mike Fährmann
5f8621b29d improve output of active post processor modules 2019-08-15 13:31:04 +02:00