rename 'image-*' & 'chapter-*' options to 'file-*' & 'child-*'
keep support for both names, e.g. file-filter & image-filter, while prioritizing the new names
This commit is contained in:
@@ -1515,6 +1515,8 @@ Description
|
|||||||
Use fallback download URLs when a download fails.
|
Use fallback download URLs when a download fails.
|
||||||
|
|
||||||
|
|
||||||
|
extractor.*.file-range
|
||||||
|
----------------------
|
||||||
extractor.*.image-range
|
extractor.*.image-range
|
||||||
-----------------------
|
-----------------------
|
||||||
Type
|
Type
|
||||||
@@ -1546,19 +1548,23 @@ extractor.*.post-range
|
|||||||
Type
|
Type
|
||||||
``string``
|
``string``
|
||||||
Description
|
Description
|
||||||
Like `image-range <extractor.*.image-range_>`__,
|
Like `file-range <extractor.*.file-range_>`__,
|
||||||
but for posts.
|
but for posts.
|
||||||
|
|
||||||
|
|
||||||
|
extractor.*.child-range
|
||||||
|
-----------------------
|
||||||
extractor.*.chapter-range
|
extractor.*.chapter-range
|
||||||
-------------------------
|
-------------------------
|
||||||
Type
|
Type
|
||||||
``string``
|
``string``
|
||||||
Description
|
Description
|
||||||
Like `image-range <extractor.*.image-range_>`__,
|
Like `file-range <extractor.*.file-range_>`__,
|
||||||
but for child extractors handling manga chapters, external URLs, etc.
|
but for child extractors handling manga chapters, external URLs, etc.
|
||||||
|
|
||||||
|
|
||||||
|
extractor.*.file-filter
|
||||||
|
-----------------------
|
||||||
extractor.*.image-filter
|
extractor.*.image-filter
|
||||||
------------------------
|
------------------------
|
||||||
Type
|
Type
|
||||||
@@ -1584,10 +1590,14 @@ Example
|
|||||||
* ``"post['id'] > 12345"``
|
* ``"post['id'] > 12345"``
|
||||||
* ``["date >= datetime(2025, 5, 1)", "print(post_id)"]``
|
* ``["date >= datetime(2025, 5, 1)", "print(post_id)"]``
|
||||||
Description
|
Description
|
||||||
Like `image-filter <extractor.*.image-filter_>`__,
|
Like `file-filter <extractor.*.file-filter_>`__,
|
||||||
but for posts.
|
but for posts.
|
||||||
|
|
||||||
|
Available values are the directory-specific ones listed by ``-K`` or ``-j``.
|
||||||
|
|
||||||
|
|
||||||
|
extractor.*.child-filter
|
||||||
|
------------------------
|
||||||
extractor.*.chapter-filter
|
extractor.*.chapter-filter
|
||||||
--------------------------
|
--------------------------
|
||||||
Type
|
Type
|
||||||
@@ -1597,10 +1607,12 @@ Example
|
|||||||
* ``"lang == 'en'"``
|
* ``"lang == 'en'"``
|
||||||
* ``["language == 'French'", "10 <= chapter < 20"]``
|
* ``["language == 'French'", "10 <= chapter < 20"]``
|
||||||
Description
|
Description
|
||||||
Like `image-filter <extractor.*.image-filter_>`__,
|
Like `file-filter <extractor.*.file-filter_>`__,
|
||||||
but for child extractors handling manga chapters, external URLs, etc.
|
but for child extractors handling manga chapters, external URLs, etc.
|
||||||
|
|
||||||
|
|
||||||
|
extractor.*.file-unique
|
||||||
|
-----------------------
|
||||||
extractor.*.image-unique
|
extractor.*.image-unique
|
||||||
------------------------
|
------------------------
|
||||||
Type
|
Type
|
||||||
@@ -1608,10 +1620,12 @@ Type
|
|||||||
Default
|
Default
|
||||||
``false``
|
``false``
|
||||||
Description
|
Description
|
||||||
Ignore image URLs that have been encountered before during the
|
Ignore file URLs that have been encountered before during the
|
||||||
current extractor run.
|
current extractor run.
|
||||||
|
|
||||||
|
|
||||||
|
extractor.*.child-unique
|
||||||
|
------------------------
|
||||||
extractor.*.chapter-unique
|
extractor.*.chapter-unique
|
||||||
--------------------------
|
--------------------------
|
||||||
Type
|
Type
|
||||||
@@ -1619,7 +1633,7 @@ Type
|
|||||||
Default
|
Default
|
||||||
``false``
|
``false``
|
||||||
Description
|
Description
|
||||||
Like `image-unique <extractor.*.image-unique_>`__,
|
Like `file-unique <extractor.*.file-unique_>`__,
|
||||||
but applies to delegated URLs like manga chapters, etc.
|
but applies to delegated URLs like manga chapters, etc.
|
||||||
|
|
||||||
|
|
||||||
@@ -6003,7 +6017,7 @@ Note
|
|||||||
It is not possible to filter all subtitles of a specific source type,
|
It is not possible to filter all subtitles of a specific source type,
|
||||||
while also filtering for additional languages of another source type.
|
while also filtering for additional languages of another source type.
|
||||||
(e.g. any ASR subtitle + fra-FR of any source type)
|
(e.g. any ASR subtitle + fra-FR of any source type)
|
||||||
For this, refer to `extractor.*.image-filter`_.
|
For this, refer to `extractor.*.file-filter`_.
|
||||||
|
|
||||||
|
|
||||||
extractor.tiktok.videos
|
extractor.tiktok.videos
|
||||||
@@ -6601,8 +6615,8 @@ Note
|
|||||||
use the ``/with_replies`` timeline while logged in. For example,
|
use the ``/with_replies`` timeline while logged in. For example,
|
||||||
media from Tweets which the user replied to will also be downloaded.
|
media from Tweets which the user replied to will also be downloaded.
|
||||||
|
|
||||||
It is possible to exclude unwanted Tweets using `image-filter
|
It is possible to exclude unwanted Tweets using `file-filter
|
||||||
<extractor.*.image-filter_>`__.
|
<extractor.*.file-filter_>`__.
|
||||||
|
|
||||||
|
|
||||||
extractor.twitter.retries-api
|
extractor.twitter.retries-api
|
||||||
@@ -9976,7 +9990,7 @@ Description
|
|||||||
post-processor type, as well as any of its `options <Postprocessor Options_>`__.
|
post-processor type, as well as any of its `options <Postprocessor Options_>`__.
|
||||||
|
|
||||||
It is possible to set a ``"filter"`` Condition_ similar to
|
It is possible to set a ``"filter"`` Condition_ similar to
|
||||||
`image-filter <extractor.*.image-filter_>`_
|
`file-filter <extractor.*.file-filter_>`_
|
||||||
to only run a post-processor conditionally.
|
to only run a post-processor conditionally.
|
||||||
|
|
||||||
It is also possible set a ``"whitelist"`` or ``"blacklist"`` to
|
It is also possible set a ``"whitelist"`` or ``"blacklist"`` to
|
||||||
|
|||||||
@@ -44,14 +44,14 @@
|
|||||||
"cookies-select": null,
|
"cookies-select": null,
|
||||||
"cookies-update": true,
|
"cookies-update": true,
|
||||||
|
|
||||||
"image-filter" : null,
|
"file-filter" : null,
|
||||||
"image-range" : null,
|
"file-range" : null,
|
||||||
"image-unique" : false,
|
"file-unique" : false,
|
||||||
"post-filter" : null,
|
"post-filter" : null,
|
||||||
"post-range" : null,
|
"post-range" : null,
|
||||||
"chapter-filter": null,
|
"child-filter": null,
|
||||||
"chapter-range" : null,
|
"child-range" : null,
|
||||||
"chapter-unique": false,
|
"child-unique": false,
|
||||||
|
|
||||||
"keywords" : {},
|
"keywords" : {},
|
||||||
"keywords-default" : null,
|
"keywords-default" : null,
|
||||||
|
|||||||
@@ -162,7 +162,7 @@
|
|||||||
download. These can be either a constant value,
|
download. These can be either a constant value,
|
||||||
range, or slice (e.g. '5', '8-20', or '1:24:3')
|
range, or slice (e.g. '5', '8-20', or '1:24:3')
|
||||||
--post-range RANGE Like '--range', but for posts
|
--post-range RANGE Like '--range', but for posts
|
||||||
--chapter-range RANGE Like '--range', but for child extractors
|
--child-range RANGE Like '--range', but for child extractors
|
||||||
handling manga chapters, external URLs, etc.
|
handling manga chapters, external URLs, etc.
|
||||||
--filter EXPR Python expression controlling which files to
|
--filter EXPR Python expression controlling which files to
|
||||||
download. Files for which the expression
|
download. Files for which the expression
|
||||||
@@ -171,7 +171,7 @@
|
|||||||
Example: --filter "image_width >= 1000 and
|
Example: --filter "image_width >= 1000 and
|
||||||
rating in ('s', 'q')"
|
rating in ('s', 'q')"
|
||||||
--post-filter EXPR Like '--filter', but for posts
|
--post-filter EXPR Like '--filter', but for posts
|
||||||
--chapter-filter EXPR Like '--filter', but for child extractors
|
--child-filter EXPR Like '--filter', but for child extractors
|
||||||
handling manga chapters, external URLs, etc.
|
handling manga chapters, external URLs, etc.
|
||||||
|
|
||||||
## Post-processing Options:
|
## Post-processing Options:
|
||||||
|
|||||||
@@ -270,34 +270,39 @@ class Job():
|
|||||||
|
|
||||||
def _init(self):
|
def _init(self):
|
||||||
self.extractor.initialize()
|
self.extractor.initialize()
|
||||||
self.pred_url = self._prepare_predicates("image", True)
|
self.pred_url = self._prepare_predicates(
|
||||||
self.pred_post = self._prepare_predicates("post", False)
|
"file", "image", True)
|
||||||
self.pred_queue = self._prepare_predicates("chapter", False)
|
self.pred_post = self._prepare_predicates(
|
||||||
|
"post", None, False)
|
||||||
|
self.pred_queue = self._prepare_predicates(
|
||||||
|
"child", "chapter", False)
|
||||||
|
|
||||||
init = self.extractor.config("init", False)
|
init = self.extractor.config("init", False)
|
||||||
if init and init != "lazy":
|
if init and init != "lazy":
|
||||||
self.initialize()
|
self.initialize()
|
||||||
|
|
||||||
def _prepare_predicates(self, target, skip):
|
def _prepare_predicates(self, target, alt=None, skip=None):
|
||||||
predicates = []
|
predicates = []
|
||||||
extr = self.extractor
|
extr = self.extractor
|
||||||
|
|
||||||
if extr.config(target + "-unique"):
|
if extr.config(target + "-unique") or \
|
||||||
|
alt is not None and extr.config(alt + "-unique"):
|
||||||
predicates.append(util.predicate_unique())
|
predicates.append(util.predicate_unique())
|
||||||
|
|
||||||
if pfilter := extr.config(target + "-filter"):
|
if (pfilter := extr.config(target + "-filter")) or \
|
||||||
|
alt is not None and (pfilter := extr.config(alt + "-filter")):
|
||||||
try:
|
try:
|
||||||
predicates.append(util.predicate_filter(pfilter, target))
|
predicates.append(util.predicate_filter(pfilter, target))
|
||||||
except (SyntaxError, ValueError, TypeError) as exc:
|
except (SyntaxError, ValueError, TypeError) as exc:
|
||||||
extr.log.warning(exc)
|
extr.log.warning(exc)
|
||||||
|
|
||||||
if prange := extr.config(target + "-range"):
|
if (prange := extr.config(target + "-range")) or \
|
||||||
|
alt is not None and (prange := extr.config(alt + "-range")):
|
||||||
try:
|
try:
|
||||||
skip = extr.skip if skip and not pfilter else None
|
skip = extr.skip if skip and not pfilter else None
|
||||||
predicates.append(util.predicate_range(prange, skip))
|
predicates.append(util.predicate_range(prange, skip))
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
extr.log.warning(
|
extr.log.warning("invalid %s range: %s", target, exc)
|
||||||
"invalid %s range: %s", target, exc)
|
|
||||||
|
|
||||||
return util.predicate_build(predicates)
|
return util.predicate_build(predicates)
|
||||||
|
|
||||||
|
|||||||
@@ -235,6 +235,7 @@ def _parse_option(opt):
|
|||||||
|
|
||||||
def build_parser():
|
def build_parser():
|
||||||
"""Build and configure an ArgumentParser object"""
|
"""Build and configure an ArgumentParser object"""
|
||||||
|
SUPPRESS = argparse.SUPPRESS
|
||||||
parser = argparse.ArgumentParser(
|
parser = argparse.ArgumentParser(
|
||||||
formatter_class=Formatter,
|
formatter_class=Formatter,
|
||||||
add_help=False,
|
add_help=False,
|
||||||
@@ -318,7 +319,7 @@ def build_parser():
|
|||||||
input.add_argument(
|
input.add_argument(
|
||||||
"urls",
|
"urls",
|
||||||
metavar="URL", nargs="*",
|
metavar="URL", nargs="*",
|
||||||
help=argparse.SUPPRESS,
|
help=SUPPRESS,
|
||||||
)
|
)
|
||||||
input.add_argument(
|
input.add_argument(
|
||||||
"-i", "--input-file",
|
"-i", "--input-file",
|
||||||
@@ -620,7 +621,7 @@ def build_parser():
|
|||||||
configuration.add_argument(
|
configuration.add_argument(
|
||||||
"--ignore-config",
|
"--ignore-config",
|
||||||
dest="config_load", action="store_false",
|
dest="config_load", action="store_false",
|
||||||
help=argparse.SUPPRESS,
|
help=SUPPRESS,
|
||||||
)
|
)
|
||||||
|
|
||||||
authentication = parser.add_argument_group("Authentication Options")
|
authentication = parser.add_argument_group("Authentication Options")
|
||||||
@@ -698,7 +699,7 @@ def build_parser():
|
|||||||
)
|
)
|
||||||
selection.add_argument(
|
selection.add_argument(
|
||||||
"--range",
|
"--range",
|
||||||
dest="image-range", metavar="RANGE", action=ConfigAction,
|
dest="file-range", metavar="RANGE", action=ConfigAction,
|
||||||
help=("Index range(s) specifying which files to download. "
|
help=("Index range(s) specifying which files to download. "
|
||||||
"These can be either a constant value, range, or slice "
|
"These can be either a constant value, range, or slice "
|
||||||
"(e.g. '5', '8-20', or '1:24:3')"),
|
"(e.g. '5', '8-20', or '1:24:3')"),
|
||||||
@@ -709,14 +710,14 @@ def build_parser():
|
|||||||
help=("Like '--range', but for posts"),
|
help=("Like '--range', but for posts"),
|
||||||
)
|
)
|
||||||
selection.add_argument(
|
selection.add_argument(
|
||||||
"--chapter-range",
|
"--child-range",
|
||||||
dest="chapter-range", metavar="RANGE", action=ConfigAction,
|
dest="child-range", metavar="RANGE", action=ConfigAction,
|
||||||
help=("Like '--range', but for child extractors handling "
|
help=("Like '--range', but for child extractors handling "
|
||||||
"manga chapters, external URLs, etc."),
|
"manga chapters, external URLs, etc."),
|
||||||
)
|
)
|
||||||
selection.add_argument(
|
selection.add_argument(
|
||||||
"--filter",
|
"--filter",
|
||||||
dest="image-filter", metavar="EXPR", action=ConfigAction,
|
dest="file-filter", metavar="EXPR", action=ConfigAction,
|
||||||
help=("Python expression controlling which files to download. "
|
help=("Python expression controlling which files to download. "
|
||||||
"Files for which the expression evaluates to False are ignored. "
|
"Files for which the expression evaluates to False are ignored. "
|
||||||
"Available keys are the filename-specific ones listed by '-K'. "
|
"Available keys are the filename-specific ones listed by '-K'. "
|
||||||
@@ -729,11 +730,23 @@ def build_parser():
|
|||||||
help=("Like '--filter', but for posts"),
|
help=("Like '--filter', but for posts"),
|
||||||
)
|
)
|
||||||
selection.add_argument(
|
selection.add_argument(
|
||||||
"--chapter-filter",
|
"--child-filter",
|
||||||
dest="chapter-filter", metavar="EXPR", action=ConfigAction,
|
dest="child-filter", metavar="EXPR", action=ConfigAction,
|
||||||
help=("Like '--filter', but for child extractors handling "
|
help=("Like '--filter', but for child extractors handling "
|
||||||
"manga chapters, external URLs, etc."),
|
"manga chapters, external URLs, etc."),
|
||||||
)
|
)
|
||||||
|
selection.add_argument(
|
||||||
|
"--file-range", "--image-range",
|
||||||
|
dest="file-range", action=ConfigAction, help=SUPPRESS)
|
||||||
|
selection.add_argument(
|
||||||
|
"--chapter-range",
|
||||||
|
dest="child-range", action=ConfigAction, help=SUPPRESS)
|
||||||
|
selection.add_argument(
|
||||||
|
"--file-filter", "--image-filter",
|
||||||
|
dest="file-filter", action=ConfigAction, help=SUPPRESS)
|
||||||
|
selection.add_argument(
|
||||||
|
"--chapter-filter",
|
||||||
|
dest="child-filter", action=ConfigAction, help=SUPPRESS)
|
||||||
|
|
||||||
infojson = {
|
infojson = {
|
||||||
"name" : "metadata",
|
"name" : "metadata",
|
||||||
@@ -773,7 +786,7 @@ def build_parser():
|
|||||||
"--write-infojson",
|
"--write-infojson",
|
||||||
dest="postprocessors",
|
dest="postprocessors",
|
||||||
action="append_const", const=infojson,
|
action="append_const", const=infojson,
|
||||||
help=argparse.SUPPRESS,
|
help=SUPPRESS,
|
||||||
)
|
)
|
||||||
postprocessor.add_argument(
|
postprocessor.add_argument(
|
||||||
"--write-tags",
|
"--write-tags",
|
||||||
@@ -806,7 +819,7 @@ def build_parser():
|
|||||||
"--mtime-from-date",
|
"--mtime-from-date",
|
||||||
dest="postprocessors", nargs=0, action=MtimeAction,
|
dest="postprocessors", nargs=0, action=MtimeAction,
|
||||||
const="date|status[date]",
|
const="date|status[date]",
|
||||||
help=argparse.SUPPRESS,
|
help=SUPPRESS,
|
||||||
)
|
)
|
||||||
postprocessor.add_argument(
|
postprocessor.add_argument(
|
||||||
"--rename",
|
"--rename",
|
||||||
@@ -830,18 +843,18 @@ def build_parser():
|
|||||||
postprocessor.add_argument(
|
postprocessor.add_argument(
|
||||||
"--ugoira-conv",
|
"--ugoira-conv",
|
||||||
dest="postprocessors", nargs=0, action=UgoiraAction, const="vp8",
|
dest="postprocessors", nargs=0, action=UgoiraAction, const="vp8",
|
||||||
help=argparse.SUPPRESS,
|
help=SUPPRESS,
|
||||||
)
|
)
|
||||||
postprocessor.add_argument(
|
postprocessor.add_argument(
|
||||||
"--ugoira-conv-lossless",
|
"--ugoira-conv-lossless",
|
||||||
dest="postprocessors", nargs=0, action=UgoiraAction,
|
dest="postprocessors", nargs=0, action=UgoiraAction,
|
||||||
const="vp9-lossless",
|
const="vp9-lossless",
|
||||||
help=argparse.SUPPRESS,
|
help=SUPPRESS,
|
||||||
)
|
)
|
||||||
postprocessor.add_argument(
|
postprocessor.add_argument(
|
||||||
"--ugoira-conv-copy",
|
"--ugoira-conv-copy",
|
||||||
dest="postprocessors", nargs=0, action=UgoiraAction, const="copy",
|
dest="postprocessors", nargs=0, action=UgoiraAction, const="copy",
|
||||||
help=argparse.SUPPRESS,
|
help=SUPPRESS,
|
||||||
)
|
)
|
||||||
postprocessor.add_argument(
|
postprocessor.add_argument(
|
||||||
"--exec",
|
"--exec",
|
||||||
|
|||||||
Reference in New Issue
Block a user