Commit Graph

1507 Commits

Author SHA1 Message Date
Mike Fährmann
9e188f6a21 [4chan] support 4channel.org domain 2018-11-21 17:40:38 +01:00
Mike Fährmann
041bd501fc [hentaifoundry] unescape YII_CSRF_TOKEN value
This fixes the POST requests to /site/filters
2018-11-19 21:46:17 +01:00
Mike Fährmann
b828473aa3 retry HTTP requests for more exception classes 2018-11-19 15:49:13 +01:00
Mike Fährmann
c2e59b9a7d update CHANGELOG.md
[ci skip]
2018-11-18 22:33:35 +01:00
Mike Fährmann
d4b2b73bef release version 1.6.0 2018-11-17 18:28:02 +01:00
Mike Fährmann
ea9d1b6501 update README.rst
- point to pip3/python3 in installation-instructions (#118, #121)
- add dependency list
- update URLs to external resources
- remove incomplete list of supported sites
2018-11-17 17:46:19 +01:00
Mike Fährmann
c47482b110 smaller changes, missing docs, etc.
- make 'netrc' extractor-specific
- rename 'downloader.enable' to 'enabled'
- document 'downloader.ytdl.format'
- consistent newlines in configuration.rst
2018-11-16 18:18:07 +01:00
Mike Fährmann
b17a5d6f3b give downloader classes proper names 2018-11-16 14:40:05 +01:00
Mike Fährmann
3c25fa2dad update build_testresult_db.py script 2018-11-15 22:58:14 +01:00
Mike Fährmann
7f6a0be982 adjust some tests 2018-11-15 22:50:04 +01:00
Mike Fährmann
baad7b0fa5 [twitter] unpack API responses when logged in (closes #123) 2018-11-14 11:49:35 +01:00
Mike Fährmann
3bdfc15be1 [pinterest] don't crash on pins without image info 2018-11-14 11:46:14 +01:00
Mike Fährmann
8ef84a6823 add option to enable/disable specific downloader modules
... and write URLs with no (active) downloader to unsupported-file
2018-11-13 18:06:36 +01:00
Mike Fährmann
14ee6bf611 [behance] handle external URLs with youtube-dl 2018-11-13 15:10:23 +01:00
Mike Fährmann
36425122ff [artstation] handle external URLs with youtube-dl 2018-11-13 14:27:02 +01:00
Mike Fährmann
bd8670d925 [gfycat] extend URL pattern 2018-11-11 21:19:11 +01:00
Mike Fährmann
2fa28a2609 update default user-agent string (closes #122) 2018-11-11 10:07:10 +01:00
Mike Fährmann
7e2d6bcd62 [deviantart] fix original image downloads 2018-11-10 19:16:10 +01:00
Mike Fährmann
9e12e073ab [2chan] fix extraction 2018-11-10 19:15:21 +01:00
Mike Fährmann
966a9ca3a0 update test results 2018-11-10 19:14:54 +01:00
Mike Fährmann
e26ba682a2 enforce utf-8 encoding for input files (#120) 2018-11-10 18:27:01 +01:00
Mike Fährmann
a36259d8f1 update setup.py
- add Python version check
- add classifiers
- simplify sys.exit() usage
2018-10-24 14:43:37 +02:00
Mike Fährmann
fd8ed35591 [turboimagehost] fix extraction 2018-10-23 21:08:24 +02:00
Mike Fährmann
c69150f715 [imagefap] fix extraction
also adds tags to gallery-metadata and converts suitable values to int
2018-10-20 18:32:25 +02:00
Mike Fährmann
d1f3d32eec [fallenangels] unescape chapter titles 2018-10-20 18:31:26 +02:00
Mike Fährmann
655549df7c [downloader:ytdl] add several options
The "default" downloader options (rate, retries, timeout, verify) are
mapped to corresponding youtube-dl options.

downloader.ytdl.logging tells the downloader to pass youtube-dl's output
to a Logger object.

downloader.ytdl.raw-options allows to pass arbitrary options to the
YoutubeDL constructor.
2018-10-20 18:26:49 +02:00
Mike Fährmann
d3d7f01543 add 'prepare()' step for post-processors
This allows post-processors to modify the destination path before
checking if a file already exists.
2018-10-18 22:32:03 +02:00
Mike Fährmann
c9861ca812 adjust message for status_code based exceptions
from: 5xx HTTP Error: Reason
to  : 5xx: Reason

The "HTTP Error" part was in there to emulate Request's error messages
from response.raise_for_status(), but it reads a lot better without.
2018-10-18 15:09:49 +02:00
Mike Fährmann
eb1c24b286 [imagebam] detect nonexistent galleries 2018-10-17 15:21:47 +02:00
Mike Fährmann
6ed629f2b6 allow specifying number of skips before abort/exit (closes #115)
In addition to 'abort' and 'exit', it is now possible to specify
'abort:N' and 'exit:N' (where N is any integer) as value for 'skip'
to abort/exit after consecutively skipping N downloads.
2018-10-13 17:21:55 +02:00
Mike Fährmann
e1d306cc48 update unit test results 2018-10-13 16:54:30 +02:00
Mike Fährmann
8faf03ed84 [pixiv] use refresh_token based authentication
The first login will still use username and password, but everything
afterwards will use the refresh_token obtained from that.

This will prevent pixiv from sending a "New login to pixiv" email every
time a new access_token is requested.
2018-10-12 22:26:27 +02:00
Mike Fährmann
2221cf97ff implement 'update()' for caches 2018-10-12 22:18:29 +02:00
Mike Fährmann
d8492df51b [deviantart] extend functionality of 'original' option 2018-10-11 22:11:49 +02:00
Mike Fährmann
c00dce2adc [behance] enable 'categorytransfer' 2018-10-09 23:40:49 +02:00
Mike Fährmann
1532d1b690 fix 'range' tests and update a few test results 2018-10-08 23:53:58 +02:00
Mike Fährmann
392a081657 update 'docs/configuration.rst' 2018-10-08 23:09:01 +02:00
Mike Fährmann
48a8717a7c add 'output.num-to-str' option
... to convert any numeric values to string when outputting them as JSON
(during '--dump-json' or otherwise)
2018-10-08 20:28:54 +02:00
Mike Fährmann
af3f81c7d9 add '--no-check-certificate' command-line option 2018-10-07 21:46:00 +02:00
Mike Fährmann
0514d6a0ae make --filter and --range config-file options
The functionality of --(chapter-)filter and --(chapter-)range are now
also exposed as the following config-file options:

- extractor.*.image-filter
- extractor.*.image-range
- extractor.*.chapter-filter
- extractor.*.chapter-range

TODO: update configuration.rst
2018-10-07 21:39:56 +02:00
Mike Fährmann
4a348990f4 adjust value resolution for retries/timeout/verify options
This change introduces 'extractor.*.retries/timeout/verify' options
as a general way to set these values for all HTTP requests.

'downloader.http.retries/timeout/verify' is a way to override these
options for file downloads only and will fall back to 'extractor.*.…*
values if they haven't been explicitly set.

Also: downloader classes now take an extractor object as first argument
instead of a requests.session.
2018-10-07 21:13:39 +02:00
Mike Fährmann
f647f5d9c3 use 'verify' option for regular HTTP requests 2018-10-06 16:38:43 +02:00
Mike Fährmann
ca6ac4db6a fix 'content' tests 2018-10-05 21:10:33 +02:00
Mike Fährmann
188876d814 implement youtube-dl downloader module
URLs starting with 'ytdl:' will now be handled by youtube-dl.
There is probably a lot to fix and improve, but the basic use case
works.

TODO:
- format selection and ytdl options in general
- better filename/path handling
- ytdl support for "unsupported URLs"
- ...
2018-10-05 18:05:11 +02:00
Mike Fährmann
f4df6c2396 [pixiv] remove 'type' and 'page' query parameter handling
The "new and improved" /member_illust.php and /bookmark.php listings
don't quite work with how things were.
2018-10-03 16:08:36 +02:00
Mike Fährmann
d70db2d555 Revert "[komikcast] fix extraction"
This reverts commit 5507f5ce2e.
2018-10-02 20:38:42 +02:00
Mike Fährmann
d69db60e2a update unit test results 2018-10-02 20:37:46 +02:00
Mike Fährmann
f8b3b00249 [twitter] add experimental 'videos' option (#99)
Enabling this option will detect videos in tweets and output them as
"unsupported" URLs, so that these can then be downloaded with youtube-dl

There are a lot of improvements to be made to the current
implementation, but it works and does what it is supposed to, even if
inefficient as can be ...
2018-09-30 21:52:23 +02:00
Mike Fährmann
5507f5ce2e [komikcast] fix extraction 2018-09-29 16:37:30 +02:00
Mike Fährmann
8080071174 [flickr] improve album metadata (closes #109) 2018-09-29 16:21:55 +02:00