Commit Graph

86 Commits

Author SHA1 Message Date
Mike Fährmann
1e4d351ad3 [danbooru] add authentication support (closes #151)
... via HTTP Basic Auth with username and "password".

The password value in this case is not the account password itself,
but the"api_key" found in your user profile.
2019-01-09 14:19:07 +01:00
Mike Fährmann
06cbf5f9c4 implement 'chapter-reverse' option (#149)
Setting it to `true` will start with the latest chapter instead of the
first one.
2019-01-07 18:22:33 +01:00
Mike Fährmann
e95b24f056 [reactor] add wait-min & -max options (#148) 2019-01-07 18:04:47 +01:00
Mike Fährmann
28f9539551 [tumblr] change default values for post types and inline media 2018-12-26 18:55:59 +01:00
Mike Fährmann
5be95034ba [tumblr] add option to download avatars (#137) 2018-12-26 14:29:30 +01:00
Mike Fährmann
c47482b110 smaller changes, missing docs, etc.
- make 'netrc' extractor-specific
- rename 'downloader.enable' to 'enabled'
- document 'downloader.ytdl.format'
- consistent newlines in configuration.rst
2018-11-16 18:18:07 +01:00
Mike Fährmann
8ef84a6823 add option to enable/disable specific downloader modules
... and write URLs with no (active) downloader to unsupported-file
2018-11-13 18:06:36 +01:00
Mike Fährmann
2fa28a2609 update default user-agent string (closes #122) 2018-11-11 10:07:10 +01:00
Mike Fährmann
655549df7c [downloader:ytdl] add several options
The "default" downloader options (rate, retries, timeout, verify) are
mapped to corresponding youtube-dl options.

downloader.ytdl.logging tells the downloader to pass youtube-dl's output
to a Logger object.

downloader.ytdl.raw-options allows to pass arbitrary options to the
YoutubeDL constructor.
2018-10-20 18:26:49 +02:00
Mike Fährmann
6ed629f2b6 allow specifying number of skips before abort/exit (closes #115)
In addition to 'abort' and 'exit', it is now possible to specify
'abort:N' and 'exit:N' (where N is any integer) as value for 'skip'
to abort/exit after consecutively skipping N downloads.
2018-10-13 17:21:55 +02:00
Mike Fährmann
d8492df51b [deviantart] extend functionality of 'original' option 2018-10-11 22:11:49 +02:00
Mike Fährmann
392a081657 update 'docs/configuration.rst' 2018-10-08 23:09:01 +02:00
Mike Fährmann
f8b3b00249 [twitter] add experimental 'videos' option (#99)
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl

There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
2018-09-30 21:52:23 +02:00
Mike Fährmann
09d2f3e5e7 [postprocessor:ugoira] improve libx264 detection 2018-09-24 13:12:54 +02:00
Mike Fährmann
55f5c87160 [postprocessor:ugoira] add 'libx264-prevent-odd' option
A rather crude workaround for "width/height not divisible by 2"
errors when using libx264.
2018-09-21 20:01:14 +02:00
Mike Fährmann
7742cf8601 [tumblr] change 'reblogs' option (#103)
- rename "deleted" to "same-blog"
- change test for deleted original post to test if
  original post owner has the same UUID (full blog name) as the one
  being downloaded from
- add 'blog[uuid]' metadata to allow comparison with
  'reblogged_from_uuid'
2018-09-10 15:40:25 +02:00
Mike Fährmann
a666ddd16b [tumblr] extend 'reblogs' functionality (#103)
Setting 'reblogs' to "deleted" will check if the parent post of a
reblog has been deleted and download its media content if that is the
case, otherwise it will be skipped.

This is a rather costly operation (1 API request per reblogged post)
and should therefore be used with care.
2018-09-07 19:13:52 +02:00
Mike Fährmann
6ecb36d88c [postprocessor:ugoira] add 'ffmpeg-output' option 2018-08-31 17:37:35 +02:00
Mike Fährmann
10365394d7 [twitter] add support for user-timelines (closes #96)
also adds a 'retweets' option to filter retweeted content
2018-08-17 20:04:11 +02:00
Mike Fährmann
b164231bca [sankaku] increase default values for 'wait-min/-max' 2018-08-03 17:06:51 +02:00
Mike Fährmann
5f27cfeff6 [deviantart] remove prefer-public option
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
2018-07-26 19:43:46 +02:00
Mike Fährmann
212130b048 [deviantart] improve public-private token switching
- rename option to `prefer-public`
- now also works for galleries with less than 24 items
2018-07-25 12:52:36 +02:00
Mike Fährmann
886d662582 [deviantart] add option to minimize refresh-token usage
Always trying with a public token first and repeating the API request
with a private token if deviations are missing doesn't quite work for
galleries and folders with less than 25 items, so its an option and
not the default.
2018-07-24 21:44:57 +02:00
Mike Fährmann
54a0d72dc8 [postprocessor:ugoira] improve frame rate handling
By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped
frames if the source requires a higher frame rate than that.

This commit adds a `framerate` option (default "auto"), which allows to
automatically assign a (more or less) fitting frame rate based on
delays between ugoira frames and avoids dropped frames.
2018-07-23 21:33:20 +02:00
Mike Fährmann
ff436692bf ["deviantart] add 'journals' option 2018-07-16 18:14:41 +02:00
Mike Fährmann
00032b828c [deviantart] add 'wait-min' option 2018-07-14 11:52:21 +02:00
Mike Fährmann
269dc2bbd5 [sankaku] add 'tags' option (#94) 2018-07-14 09:53:01 +02:00
Mike Fährmann
1d43cbbf52 [gelbooru] tag-splitting for non-api mode 2018-07-06 15:24:19 +02:00
Mike Fährmann
4a57509392 generalize tag-splitting option (#92)
- extend functionality to other booru sites:
  - http://behoimi.org/
  - https://konachan.com/
  - https://e621.net/
  - https://rule34.xxx/
  - https://safebooru.org/
  - https://yande.re/
2018-07-04 12:21:16 +02:00
Mike Fährmann
87853538b4 [yandere] add option to split tags by type (#92) 2018-06-29 19:38:53 +02:00
Mike Fährmann
4eb94aca17 [postprocessor:ugoira] pass '-f' if not present 2018-06-22 13:26:17 +02:00
Mike Fährmann
0c1c4557dd [postprocessor:ugoira] add option for two-pass encoding 2018-06-20 18:48:10 +02:00
Mike Fährmann
eb7a1f3b98 [pixiv] rework ugoira handling
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
2018-06-18 17:57:57 +02:00
Mike Fährmann
b84e71da91 add postprocessor documentation to configuration.rst 2018-06-16 15:46:41 +02:00
Mike Fährmann
12797e3b1f update configuration.rst
... again

- some more 'Path' references
- fixed some inconsistencies and errors
- added note about logging config for files
2018-05-28 22:14:38 +02:00
Mike Fährmann
c43f02245f update configuration.rst
- fix default values for 'log' and 'unsupportedfile'

[ci skip]
2018-05-27 17:12:57 +02:00
Mike Fährmann
dacda69c9e update configuration.rst
- document logging options
- add a section for "custom types"

[ci skip]
2018-05-27 16:50:35 +02:00
Mike Fährmann
53f36176fd update configuration.rst
- update the API Tokens & IDs section
  - mention redirect URIs for deviantart
  - include api-secret for tumblr
  - add instructions for smugmug
- [ci skip]
2018-05-26 11:26:50 +02:00
Mike Fährmann
0f1e07f627 [pinterest] scrap OAuth implementation; code improvements
OAuth authentication isn't needed anymore and other tools
like Postman are better suited for this job anyway.
2018-04-25 16:04:30 +02:00
Mike Fährmann
d10579edb5 [pinterest] improve PinterestAPI code; remove OAuth mentions
on another note: access_tokens have been set to only allow for
10 requests per hour (from 200 yesterday)
2018-04-17 17:12:42 +02:00
Mike Fährmann
4bd182c107 [pinterest] implement oauth:pinterest (#83)
Pinterest access tokens are rate limited at 200 requests per
hour (or maybe per 2 or 3 hours?) so having just one access token
for all users isn't going to work in the long run.
2018-04-16 20:03:28 +02:00
Mike Fährmann
a1fa4b43b0 Revert "[tumblr] add option to sort photosets by upload order"
This reverts commit 4a26ae32df.
2018-04-09 16:08:08 +02:00
Mike Fährmann
4a26ae32df [tumblr] add option to sort photosets by upload order 2018-04-07 15:57:55 +02:00
Mike Fährmann
f0e3861ec7 update configuration.rst
- move `base-directory` entry to extractor options
  (it should've been there since v1.0.0 / commit e3bfb83)
- rename "General" to "Miscellaneous" and move it down a bit, since it
  has only two options left
- mention the ability to use a list for tumblr.posts
- replace any appearance of bato.to
2018-03-16 12:00:55 +01:00
Mike Fährmann
723cc66bb1 [artstation] add user-, image- and likes-extractors 2018-03-14 14:05:14 +01:00
Mike Fährmann
8cdce21dcb make archive keys user-configurable 2018-02-25 21:57:01 +01:00
Mike Fährmann
e1e0668ca8 add option to set default replacement field value
Missing or undefined keywords will now be replaced with the value
set for 'keywords-default'. The default is Python's 'None', which
is equivalent to setting this option to JSON's 'null'.
2018-02-23 00:59:20 +01:00
Mike Fährmann
8704d850bf add explicit proxy support (#76)
- '--proxy' as command-line argument
- 'extractor.*.proxy' as config option
2018-02-19 18:45:06 +01:00
Mike Fährmann
3cec533c28 Merge branch 'archive' 2018-02-12 18:07:58 +01:00
Mike Fährmann
7f7c16ae37 add option to specify additional key-value pairs 2018-02-08 23:10:58 +01:00