Commit Graph

80 Commits

Author SHA1 Message Date
Mike Fährmann
8ef84a6823 add option to enable/disable specific downloader modules
... and write URLs with no (active) downloader to unsupported-file
2018-11-13 18:06:36 +01:00
Mike Fährmann
2fa28a2609 update default user-agent string (closes #122) 2018-11-11 10:07:10 +01:00
Mike Fährmann
655549df7c [downloader:ytdl] add several options
The "default" downloader options (rate, retries, timeout, verify) are
mapped to corresponding youtube-dl options.

downloader.ytdl.logging tells the downloader to pass youtube-dl's output
to a Logger object.

downloader.ytdl.raw-options allows to pass arbitrary options to the
YoutubeDL constructor.
2018-10-20 18:26:49 +02:00
Mike Fährmann
6ed629f2b6 allow specifying number of skips before abort/exit (closes #115)
In addition to 'abort' and 'exit', it is now possible to specify
'abort:N' and 'exit:N' (where N is any integer) as value for 'skip'
to abort/exit after consecutively skipping N downloads.
2018-10-13 17:21:55 +02:00
Mike Fährmann
d8492df51b [deviantart] extend functionality of 'original' option 2018-10-11 22:11:49 +02:00
Mike Fährmann
392a081657 update 'docs/configuration.rst' 2018-10-08 23:09:01 +02:00
Mike Fährmann
f8b3b00249 [twitter] add experimental 'videos' option (#99)
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl

There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
2018-09-30 21:52:23 +02:00
Mike Fährmann
09d2f3e5e7 [postprocessor:ugoira] improve libx264 detection 2018-09-24 13:12:54 +02:00
Mike Fährmann
55f5c87160 [postprocessor:ugoira] add 'libx264-prevent-odd' option
A rather crude workaround for "width/height not divisible by 2"
errors when using libx264.
2018-09-21 20:01:14 +02:00
Mike Fährmann
7742cf8601 [tumblr] change 'reblogs' option (#103)
- rename "deleted" to "same-blog"
- change test for deleted original post to test if
  original post owner has the same UUID (full blog name) as the one
  being downloaded from
- add 'blog[uuid]' metadata to allow comparison with
  'reblogged_from_uuid'
2018-09-10 15:40:25 +02:00
Mike Fährmann
a666ddd16b [tumblr] extend 'reblogs' functionality (#103)
Setting 'reblogs' to "deleted" will check if the parent post of a
reblog has been deleted and download its media content if that is the
case, otherwise it will be skipped.

This is a rather costly operation (1 API request per reblogged post)
and should therefore be used with care.
2018-09-07 19:13:52 +02:00
Mike Fährmann
6ecb36d88c [postprocessor:ugoira] add 'ffmpeg-output' option 2018-08-31 17:37:35 +02:00
Mike Fährmann
10365394d7 [twitter] add support for user-timelines (closes #96)
also adds a 'retweets' option to filter retweeted content
2018-08-17 20:04:11 +02:00
Mike Fährmann
b164231bca [sankaku] increase default values for 'wait-min/-max' 2018-08-03 17:06:51 +02:00
Mike Fährmann
5f27cfeff6 [deviantart] remove prefer-public option
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
2018-07-26 19:43:46 +02:00
Mike Fährmann
212130b048 [deviantart] improve public-private token switching
- rename option to `prefer-public`
- now also works for galleries with less than 24 items
2018-07-25 12:52:36 +02:00
Mike Fährmann
886d662582 [deviantart] add option to minimize refresh-token usage
Always trying with a public token first and repeating the API request
with a private token if deviations are missing doesn't quite work for
galleries and folders with less than 25 items, so its an option and
not the default.
2018-07-24 21:44:57 +02:00
Mike Fährmann
54a0d72dc8 [postprocessor:ugoira] improve frame rate handling
By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped
frames if the source requires a higher frame rate than that.

This commit adds a `framerate` option (default "auto"), which allows to
automatically assign a (more or less) fitting frame rate based on
delays between ugoira frames and avoids dropped frames.
2018-07-23 21:33:20 +02:00
Mike Fährmann
ff436692bf ["deviantart] add 'journals' option 2018-07-16 18:14:41 +02:00
Mike Fährmann
00032b828c [deviantart] add 'wait-min' option 2018-07-14 11:52:21 +02:00
Mike Fährmann
269dc2bbd5 [sankaku] add 'tags' option (#94) 2018-07-14 09:53:01 +02:00
Mike Fährmann
1d43cbbf52 [gelbooru] tag-splitting for non-api mode 2018-07-06 15:24:19 +02:00
Mike Fährmann
4a57509392 generalize tag-splitting option (#92)
- extend functionality to other booru sites:
  - http://behoimi.org/
  - https://konachan.com/
  - https://e621.net/
  - https://rule34.xxx/
  - https://safebooru.org/
  - https://yande.re/
2018-07-04 12:21:16 +02:00
Mike Fährmann
87853538b4 [yandere] add option to split tags by type (#92) 2018-06-29 19:38:53 +02:00
Mike Fährmann
4eb94aca17 [postprocessor:ugoira] pass '-f' if not present 2018-06-22 13:26:17 +02:00
Mike Fährmann
0c1c4557dd [postprocessor:ugoira] add option for two-pass encoding 2018-06-20 18:48:10 +02:00
Mike Fährmann
eb7a1f3b98 [pixiv] rework ugoira handling
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
2018-06-18 17:57:57 +02:00
Mike Fährmann
b84e71da91 add postprocessor documentation to configuration.rst 2018-06-16 15:46:41 +02:00
Mike Fährmann
12797e3b1f update configuration.rst
... again

- some more 'Path' references
- fixed some inconsistencies and errors
- added note about logging config for files
2018-05-28 22:14:38 +02:00
Mike Fährmann
c43f02245f update configuration.rst
- fix default values for 'log' and 'unsupportedfile'

[ci skip]
2018-05-27 17:12:57 +02:00
Mike Fährmann
dacda69c9e update configuration.rst
- document logging options
- add a section for "custom types"

[ci skip]
2018-05-27 16:50:35 +02:00
Mike Fährmann
53f36176fd update configuration.rst
- update the API Tokens & IDs section
  - mention redirect URIs for deviantart
  - include api-secret for tumblr
  - add instructions for smugmug
- [ci skip]
2018-05-26 11:26:50 +02:00
Mike Fährmann
0f1e07f627 [pinterest] scrap OAuth implementation; code improvements
OAuth authentication isn't needed anymore and other tools
like Postman are better suited for this job anyway.
2018-04-25 16:04:30 +02:00
Mike Fährmann
d10579edb5 [pinterest] improve PinterestAPI code; remove OAuth mentions
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
2018-04-17 17:12:42 +02:00
Mike Fährmann
4bd182c107 [pinterest] implement oauth:pinterest (#83)
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
2018-04-16 20:03:28 +02:00
Mike Fährmann
a1fa4b43b0 Revert "[tumblr] add option to sort photosets by upload order"
This reverts commit 4a26ae32df.
2018-04-09 16:08:08 +02:00
Mike Fährmann
4a26ae32df [tumblr] add option to sort photosets by upload order 2018-04-07 15:57:55 +02:00
Mike Fährmann
f0e3861ec7 update configuration.rst
- move `base-directory` entry to extractor options
  (it should've been there since v1.0.0 / commit e3bfb83)
- rename "General" to "Miscellaneous" and move it down a bit, since it
  has only two options left
- mention the ability to use a list for tumblr.posts
- replace any appearance of bato.to
2018-03-16 12:00:55 +01:00
Mike Fährmann
723cc66bb1 [artstation] add user-, image- and likes-extractors 2018-03-14 14:05:14 +01:00
Mike Fährmann
8cdce21dcb make archive keys user-configurable 2018-02-25 21:57:01 +01:00
Mike Fährmann
e1e0668ca8 add option to set default replacement field value
Missing or undefined keywords will now be replaced with the value
set for 'keywords-default'. The default is Python's 'None', which
is equivalent to setting this option to JSON's 'null'.
2018-02-23 00:59:20 +01:00
Mike Fährmann
8704d850bf add explicit proxy support (#76)
- '--proxy' as command-line argument
- 'extractor.*.proxy' as config option
2018-02-19 18:45:06 +01:00
Mike Fährmann
3cec533c28 Merge branch 'archive' 2018-02-12 18:07:58 +01:00
Mike Fährmann
7f7c16ae37 add option to specify additional key-value pairs 2018-02-08 23:10:58 +01:00
Mike Fährmann
c0dd922c13 add '--download-archive' cmdline option
… as well as a config file equivalent
2018-02-01 22:00:44 +01:00
Mike Fährmann
d951f13e37 add config option for unsupported-URL file
for consistency's sake
2018-01-28 18:42:10 +01:00
Mike Fährmann
97f4f15ec0 add option to write logging output to a file
- '--write-log FILE' as cmdline argument
- 'output.logfile' as config file option
2018-01-26 18:51:51 +01:00
Mike Fährmann
03b8a548cb [tumblr] change reblogs default value to true (#61) 2018-01-06 15:52:08 +01:00
Mike Fährmann
d235f68f59 [tumblr] add option to filter reblogged posts (#61)
Reblogs are ignored by default, but can be included by setting
'extractor.tumblr.reblogs' to 'true'.
2018-01-05 13:05:57 +01:00
Mike Fährmann
d0886f411e [gelbooru] re-enable API use (closes #56)
Gelbooru's API allows access to all images and is not restricted
to the first 20000.

This also adds an option to select between API use and manual
information extraction in case their API gets disabled again.
2017-12-21 21:42:40 +01:00