[pp] allow inheriting settings from global 'postprocessor' entries
No idea how to properly explain/document this, so here's an example:
The extractor.postprocessors object
gets its options from postprocessor.jl
and adds 'filename' itself.
{
"extractor": {
"postprocessors": {
"type": "jl",
"filename": "meta.jsonl"
}
},
"postprocessor": {
"jl": {
"name": "metadata",
"mode": "jsonl",
"open": "a"
}
}
}
This commit is contained in:
@@ -622,6 +622,14 @@ class DownloadJob(Job):
|
||||
for pp_dict in postprocessors:
|
||||
if isinstance(pp_dict, str):
|
||||
pp_dict = pp_conf.get(pp_dict) or {"name": pp_dict}
|
||||
elif "type" in pp_dict:
|
||||
pp_type = pp_dict["type"]
|
||||
if pp_type in pp_conf:
|
||||
pp = pp_conf[pp_type].copy()
|
||||
pp.update(pp_dict)
|
||||
pp_dict = pp
|
||||
if "name" not in pp_dict:
|
||||
pp_dict["name"] = pp_type
|
||||
if pp_opts:
|
||||
pp_dict = pp_dict.copy()
|
||||
pp_dict.update(pp_opts)
|
||||
|
||||
Reference in New Issue
Block a user