Commit Graph

905 Commits

Author SHA1 Message Date
Mike Fährmann
c9290d8212 [wallhaven] add wallpaper and search extractors
todo:
- login support to gain access to NSFW wallpapers
- extractors for tag-, similar-, latest-listings
- skip() support
2018-09-17 21:26:13 +02:00
Mike Fährmann
22ab509a70 [bobx] rename "model" to "idol" extractor 2018-09-14 18:11:36 +02:00
Mike Fährmann
69fd61ea86 [bobx] add gallery and model extractors 2018-09-13 20:13:12 +02:00
Mike Fährmann
7742cf8601 [tumblr] change 'reblogs' option (#103)
- rename "deleted" to "same-blog"
- change test for deleted original post to test if
  original post owner has the same UUID (full blog name) as the one
  being downloaded from
- add 'blog[uuid]' metadata to allow comparison with
  'reblogged_from_uuid'
2018-09-10 15:40:25 +02:00
Mike Fährmann
f1695567e8 adjust values in template config file
(#104)
2018-09-09 14:10:55 +02:00
Mike Fährmann
a666ddd16b [tumblr] extend 'reblogs' functionality (#103)
Setting 'reblogs' to "deleted" will check if the parent post of a
reblog has been deleted and download its media content if that is the
case, otherwise it will be skipped.

This is a rather costly operation (1 API request per reblogged post)
and should therefore be used with care.
2018-09-07 19:13:52 +02:00
Mike Fährmann
c9b8e6aefc [reddit] fix submission-ID parsing (#104)
Uppercase characters caused a ValueError exception
2018-09-07 18:27:54 +02:00
Mike Fährmann
3ecea4cf36 [hentaicafe] add chapter and manga extractors (#101) 2018-09-05 21:08:40 +02:00
Mike Fährmann
75862715ac [behance] add user extractor 2018-08-31 17:42:09 +02:00
Mike Fährmann
6ecb36d88c [postprocessor:ugoira] add 'ffmpeg-output' option 2018-08-31 17:37:35 +02:00
Mike Fährmann
ab2bfaeb46 [ngomik] add replacement for 'subapics'
http://subapics.com/ got discontinued and replaced by http://ngomik.in/.

ngomik.in is still displaying a link to the "old site" showing a big
"Account Suspended" sign.
2018-08-23 15:29:53 +02:00
Mike Fährmann
e9dd2eff1d [twitter] add extractor for media-tweet timelines (#96)
For example "https://twitter.com/PicturesEarth/media".
They are different from normal timelines in that they do not contain
any (re)tweets from other users and feature all media the user ever
posted, including responses to other tweets.
2018-08-19 20:46:12 +02:00
Mike Fährmann
9b1c39032c [twitter] changes and improvements
- rename User- to TimelineExtractor
- rename 'userid' to 'user_id' to conform to the other ..._id values
- adjust archive_fmt to deal with retweets
- emulate browser behavior for API calls
2018-08-18 23:04:45 +02:00
Mike Fährmann
10365394d7 [twitter] add support for user-timelines (closes #96)
also adds a 'retweets' option to filter retweeted content
2018-08-17 20:04:11 +02:00
Mike Fährmann
63fa0b2006 [pinterest] add extractors for related pins
Related pins can not be accessed by adding a "#related" fragment
to the end of a Pinterest URL, for example:
- https://www.pinterest.com/pin/858146903966145189/#related
- https://www.pinterest.com/g1952849/test-/#related

There are no explicit real URLs for related pins,
using an option to enable them results in "clunky" code,
and a custom "related:<URL>" scheme doesn't feel right either.
2018-08-15 21:49:45 +02:00
Mike Fährmann
3c90df6635 [piczel] add user, folder and image extractors 2018-08-08 10:53:01 +02:00
Mike Fährmann
b164231bca [sankaku] increase default values for 'wait-min/-max' 2018-08-03 17:06:51 +02:00
Mike Fährmann
df082e923c [behance] add gallery extractor (#95) 2018-08-01 21:46:55 +02:00
Mike Fährmann
5f27cfeff6 [deviantart] remove prefer-public option
All API requests now always use a public token and only switch to
a private token for pagination results if `refresh-token` is set
and less deviations than requested were returned.
2018-07-26 19:43:46 +02:00
Mike Fährmann
212130b048 [deviantart] improve public-private token switching
- rename option to `prefer-public`
- now also works for galleries with less than 24 items
2018-07-25 12:52:36 +02:00
Mike Fährmann
886d662582 [deviantart] add option to minimize refresh-token usage
Always trying with a public token first and repeating the API request
with a private token if deviations are missing doesn't quite work for
galleries and folders with less than 25 items, so its an option and
not the default.
2018-07-24 21:44:57 +02:00
Mike Fährmann
54a0d72dc8 [postprocessor:ugoira] improve frame rate handling
By default FFmpeg assumes a 25 FPS input frame rate, leading to dropped
frames if the source requires a higher frame rate than that.

This commit adds a `framerate` option (default "auto"), which allows to
automatically assign a (more or less) fitting frame rate based on
delays between ugoira frames and avoids dropped frames.
2018-07-23 21:33:20 +02:00
Mike Fährmann
84854fcad7 [myportfolio] add user and gallery extractors (#95) 2018-07-19 18:56:45 +02:00
Mike Fährmann
c9f70e0a19 [paheal] use HTTPS 2018-07-17 21:25:03 +02:00
Mike Fährmann
ff436692bf ["deviantart] add 'journals' option 2018-07-16 18:14:41 +02:00
Mike Fährmann
00032b828c [deviantart] add 'wait-min' option 2018-07-14 11:52:21 +02:00
Mike Fährmann
269dc2bbd5 [sankaku] add 'tags' option (#94) 2018-07-14 09:53:01 +02:00
Mike Fährmann
1d43cbbf52 [gelbooru] tag-splitting for non-api mode 2018-07-06 15:24:19 +02:00
Mike Fährmann
c20c0a4820 [safebooru] add pool extractor 2018-07-04 12:24:57 +02:00
Mike Fährmann
4a57509392 generalize tag-splitting option (#92)
- extend functionality to other booru sites:
  - http://behoimi.org/
  - https://konachan.com/
  - https://e621.net/
  - https://rule34.xxx/
  - https://safebooru.org/
  - https://yande.re/
2018-07-04 12:21:16 +02:00
Mike Fährmann
188e956c4e [imagefap] use HTTPS + update test results 2018-06-30 19:40:46 +02:00
Mike Fährmann
87853538b4 [yandere] add option to split tags by type (#92) 2018-06-29 19:38:53 +02:00
Mike Fährmann
4eb94aca17 [postprocessor:ugoira] pass '-f' if not present 2018-06-22 13:26:17 +02:00
Mike Fährmann
0c1c4557dd [postprocessor:ugoira] add option for two-pass encoding 2018-06-20 18:48:10 +02:00
Mike Fährmann
eb7a1f3b98 [pixiv] rework ugoira handling
Frame information now gets attached to the ZIP file's keyword dict
instead of being written to a separate text file.
2018-06-18 17:57:57 +02:00
Mike Fährmann
b84e71da91 add postprocessor documentation to configuration.rst 2018-06-16 15:46:41 +02:00
Mike Fährmann
6ac403c5d3 add postprocessor config example 2018-06-08 18:31:59 +02:00
Mike Fährmann
cdcc3427a0 [simplyhentai] add video extractor (#89)
All videos hosted on their own servers seem be to dead,
but myhentai.tv embeds, which are most of the videos, work fine.
2018-05-30 11:25:23 +02:00
Mike Fährmann
12797e3b1f update configuration.rst
... again

- some more 'Path' references
- fixed some inconsistencies and errors
- added note about logging config for files
2018-05-28 22:14:38 +02:00
Mike Fährmann
c43f02245f update configuration.rst
- fix default values for 'log' and 'unsupportedfile'

[ci skip]
2018-05-27 17:12:57 +02:00
Mike Fährmann
dacda69c9e update configuration.rst
- document logging options
- add a section for "custom types"

[ci skip]
2018-05-27 16:50:35 +02:00
Mike Fährmann
55b0913412 [simplyhentai] add gallery extractor (#89) 2018-05-27 15:25:04 +02:00
Mike Fährmann
53f36176fd update configuration.rst
- update the API Tokens & IDs section
  - mention redirect URIs for deviantart
  - include api-secret for tumblr
  - add instructions for smugmug
- [ci skip]
2018-05-26 11:26:50 +02:00
Mike Fährmann
ecdc3475b8 [pixhost] support .to TLDs 2018-05-23 18:32:34 +02:00
Mike Fährmann
909d105ae6 [pixiv] add extractor for illusts from followed users 2018-05-15 13:05:15 +02:00
Mike Fährmann
f3483a2b7c [smugmug] add OAuth support 2018-05-10 18:58:05 +02:00
Mike Fährmann
69a5e6ddb3 Merge branch 'master' into 1.4-dev 2018-05-04 10:19:02 +02:00
Mike Fährmann
82c50fa609 release version 1.3.5 2018-05-04 10:03:20 +02:00
Mike Fährmann
8bf3cdd82b implement logging options
Standard logging to stderr, logfiles, and unsupported URL files (which
are now handled through the logging module) can now be configured by
setting their respective option keys (log, logfile, unsupportedfile)
to a dict and specifying the following options;

- format:
    format string for logging messages
    available keys: see [1]
    default: "[{name}][{levelname}] {message}"
- format-date:
    format string for {asctime} fields in logging messages
    available keys: see [2]
    default: "%Y-%m-%d %H:%M:%S"
- level:
    the lowercase levelname until which the logger should activate;
    available levels are debug, info, warning, error, exception
    default: "info"
- path:
    path of the file to be written to
- mode:
    'mode' argument when opening the specified file
    can be either "w" to truncate the file or "a" to append to it (see [3])

If 'output.log', '.logfile', or '.unsupportedfile' is a string, it will
be interpreted, as it has been, as the filepath
(or as format string for .log)

[1] https://docs.python.org/3/library/logging.html#logrecord-attributes
[2] https://docs.python.org/3/library/time.html#time.strftime
[3] https://docs.python.org/3/library/functions.html#open
2018-05-01 17:54:52 +02:00
Mike Fährmann
16e014baaa [smugmug] added image and album extractor
just some initial code that still requires a lot of work ...

TODO:
- folders
- old-style albums (which are nearly all of them ...)
- images from users
- OAuth

It could also happen that the API credentials used will become invalid
whenever my 14 day trial period ends (7 days remaining), but that
would just require users to supply their own.
2018-04-29 21:27:25 +02:00