No idea how to properly explain/document this, so here's an example:
The extractor.postprocessors object
gets its options from postprocessor.jl
and adds 'filename' itself.
{
"extractor": {
"postprocessors": {
"type": "jl",
"filename": "meta.jsonl"
}
},
"postprocessor": {
"jl": {
"name": "metadata",
"mode": "jsonl",
"open": "a"
}
}
}
For example, instead of 'url-metadata' it is now also possible to use
'metadata-url' as option name.
- metadata-url
- metadata-path
- metadata-http
- metadata-version
- metadata-parent
This is a pretty ugly hack as the internal infrastructure doesn't
really support switching from external URL to regular download in
case the former fails, but it kind of works ...
Can be disabled by setting 'reddit.fallback' to 'false'.
Add 'finalize-error' and 'finalize-success' events that trigger
depending on whether error(s) did or did not happen.
'finalize' itself now always triggers regardless of error status.
(was supposed to have the same behavior as the new 'finalize-success')
continuation of ed21908f
allow for children to have an arbitrary distance from their parent,
e.g. reddit -> danbooru -> imgur:gallery -> imgur:album
would still be covered by 'reddit>imgur' or even 'danbooru>imgur'
Using "parent-category>child-category" as extractor category in a config
file allows to set options for a child extractor when it was spawned by
that parent.
For example "reddit>gfycat" to set gfycat options for when it was found
in a reddit post.
{
"extractor": {
"gfycat": {
"filename": "regular filename"
},
"reddit>gfycat": {
"filename": "reddit-specific filename"
}
}
}
Note: This does currently not work for most imgur links due to how its
extractor hierarchy is structured.
continuation of d37e7f48
but more versatile and extendable
Example:
"actions": [
# change debug messages to info
["debug", "level ~info"],
# change exit status to a non-zero value
["info:^No results for", "status |= 1"],
# exit with status 2 on 429
["warning:429", "exit 2"],
# restart extractor when no cookies found
["warning:^[Nn]o .*cookies", "restart"]
]
Very much a work in progress.
At the moment, it allows to
- wait and restart an extractor (#3338)
- change the exit code (#3630)
- change the log level of a logging message
based on the contents of a logging message
