github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
923 Commits 1 Branch 0 Tags
e6814aebe295384f2938ced2b47d5b2338efd137
Commit Graph

923 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mike Fährmann
e6814aebe2 add 'extractor.*.user-agent' config option 2017-11-15 14:01:33 +01:00
Mike Fährmann
6913eeaa40 [powermanga] replace manga extractor unit test 
My Hero Academia is gone
 2017-11-15 14:01:24 +01:00
Mike Fährmann
7e0d9257a7 [hbrowse] fix manga extraction 2017-11-15 13:59:50 +01:00
Mike Fährmann
3c576d10c0 [seiga] better metadata + 'skip()' support 2017-11-15 13:58:35 +01:00
Mike Fährmann
f72318e593 [seiga] support more than 200 images 
Due to API restrictions and/or missing knowledge about and
documentation of API usage, it was only possible to retrieve the
latest 200 images of a niconico seiga user with said API.

The new approach manually visits each HTML page and gets its
information from there.
 2017-11-13 20:46:24 +01:00
Mike Fährmann
baf8094868 improve Extractor.request()'s retry behavior 2017-11-13 20:37:11 +01:00
Mike Fährmann
2457b71633 skip tests on 5xx status codes 2017-11-12 20:51:12 +01:00
Mike Fährmann
7e7b64162b [batoto] handle error 10031 2017-11-12 20:49:37 +01:00
Mike Fährmann
79bcaa8726 improve downloader retry behavior 
- only retry download on 5xx and 429 status codes
- immediately fail on 4xx status codes
 2017-11-10 21:46:18 +01:00
Mike Fährmann
5ee8ca0319 release version 1.0.1 2017-11-10 08:54:33 +01:00
Mike Fährmann
2c1adda784 update release.sh script 
- update CHANGELOG on new releases
  - change issue references to actual links
  - replace "Unreleased" with new version and date
- fix filenames of old Windows executables

[no ci]
 2017-11-08 17:47:52 +01:00
Mike Fährmann
e913e5ec77 add a CHANGELOG 
This is basically just a copy&paste from the Releases page, but it
has the benefits of (1) better visibility and (2) "forcing" me to
write a changelog section before releasing a new version and not
several days after.
 2017-11-08 16:54:13 +01:00
Mike Fährmann
42e948584d fix downloader error handling 
RequestException being a subclass of OSError caused all exceptions
during file downloads to be ignored/re-raised.
 2017-11-07 15:23:07 +01:00
Mike Fährmann
92027f67f9 use consistent names for URL constants 
root := <scheme>://<host>
base_url := <root>/<common path>
 2017-11-06 20:56:49 +01:00
Mike Fährmann
69cbc0619f [mangastream] fix 'next-page' URLs (fixes #49) 2017-11-04 11:50:40 +01:00
Mike Fährmann
980fd3616d [tumblr] use API v2 (#48) 2017-11-03 22:16:57 +01:00
Mike Fährmann
d6bed9f36f [tumblr] prevent premature exit to get all images (fixes #48) 2017-11-03 14:59:31 +01:00
Mike Fährmann
305da540c3 [mangahere] fix metadata extraction 2017-11-03 14:54:46 +01:00
Mike Fährmann
2d0cfb33e1 [xvideos] add user profile extractor (#45) 2017-11-02 17:28:35 +01:00
Mike Fährmann
a393e6e538 [xvideos] add gallery extractor (#45) 2017-11-02 15:36:53 +01:00
Mike Fährmann
3a8a0c1f35 [imgbox] rewrite / fix extraction (closes #47) 2017-11-01 13:01:59 +01:00
Mike Fährmann
f97207a8e6 release version 1.0.0 2017-10-27 16:22:51 +02:00
Mike Fährmann
a4bc5a3491 update setup.py and README.rst 2017-10-27 16:08:57 +02:00
Mike Fährmann
707b15b586 create missing directories for 'part-directory' 
also some code improvements regarding downloader config values
 2017-10-27 12:22:45 +02:00
Mike Fährmann
035ef655f1 [imagefap] update unit tests 
old gallery/image has been deleted
 2017-10-27 12:22:16 +02:00
Mike Fährmann
caf26412dd add option to set alternate location of .part files (#29) 
Note: The path set for 'downloader.*.part-directory' needs to point to an
already existing directory.
 2017-10-26 00:16:48 +02:00
Mike Fährmann
ea8ca4cfa4 add 'util.expand_path()' 2017-10-26 00:04:28 +02:00
Mike Fährmann
9a41002b77 fix partial downloads for 'text:' URLs 
Using a filesize in bytes as offset into a Python string is not
a good idea if said file contains non-ASCII characters.
 2017-10-25 15:04:45 +02:00
Mike Fährmann
239d7afea7 [hosturimage] fix extraction of larger images 2017-10-25 12:56:16 +02:00
Mike Fährmann
27c026543f re-enable download unit tests 2017-10-25 12:55:36 +02:00
Mike Fährmann
963670d73b add options to control usage of .part files (#29) 
- '--no-part' command line option to disable them
- 'downloader.http.part' and 'downloader.text.part' config options

Disabling .part files restores the behaviour of the old downloader
implementation.
 2017-10-24 23:33:44 +02:00
Mike Fährmann
158e60ee89 [3dbooru] enable download continuation 
behoimi.org doesn't respect 'Range' headers and doesn't report
'Content-Length' for compressed content encodings.
 2017-10-24 13:05:31 +02:00
Mike Fährmann
b0353aa02d rewrite download modules (#29) 
- use '.part' files during file-download
- implement continuation of incomplete downloads
- check if file size matches the one reported by server
 2017-10-24 12:53:03 +02:00
Mike Fährmann
c4fcdf2691 Revert "[senmanga] fix extraction and download" 
This reverts commit 2ace5c7b3c.
 2017-10-24 00:22:05 +02:00
Mike Fährmann
81a7788b40 replace space characters in unit test URLs 2017-10-23 17:00:53 +02:00
Mike Fährmann
bf82181359 [jaiminisbox] fix extraction 2017-10-22 13:26:09 +02:00
Mike Fährmann
2e982f56af use 'Content-Length' to determine incomplete downloads (#29) 2017-10-20 18:56:18 +02:00
Mike Fährmann
16783e327f [common] fix UnboundLocalError in Extractor.request() 2017-10-20 18:51:06 +02:00
Mike Fährmann
2ace5c7b3c [senmanga] fix extraction and download 2017-10-19 18:25:31 +02:00
Mike Fährmann
4d8387f93b [pixiv] support mobile URLs (https://touch.pixiv.net/) 2017-10-17 16:49:42 +02:00
Mike Fährmann
ab2bf0b0dd [deviantart] replace collection unittest 2017-10-17 15:58:16 +02:00
Mike Fährmann
289d6b65d2 [danbooru] extend and improve URL regex 
- add support for danbooru mirrors:
  - hijiribe.donmai.us
  - sonohara.donmai.us
  - todo: actually use these domains instead of redirecting everything
          to danbooru itself
- improve handling of query string parameters
 2017-10-16 21:21:19 +02:00
Mike Fährmann
5fa42336a2 [sankaku] add warning for unauthenticated users 
also improve URL pattern and add missing options to default config file
 2017-10-16 21:21:08 +02:00
Mike Fährmann
6af921a952 [sankaku] rewrite/improve (fixes #44) 
- add wait-time between HTTP requests similar to exhentai
- add 'wait-min' and 'wait-max' options
- increase retry-count for HTTP requests to 10
- implement user authentication (non-authenticated users can only view
  images up to page 25)
- implement 'skip()' functionality (only works up to page 50)
- implement image-retrieval for pages >= 51
- fix issue with multiple tags
 2017-10-14 23:01:33 +02:00
Mike Fährmann
9aecc67841 [common] explicitly handle HTTP status code 429 2017-10-14 21:37:59 +02:00
Mike Fährmann
abfd2924a9 update configuration.rst 
- add instructions on how to register your own applications
- combine descriptions of a few config keys
 2017-10-12 23:37:28 +02:00
Mike Fährmann
d68a24aa70 [kissmanga] fix extraction 
site changed '\n' to '\r\n' for newlines
 2017-10-12 14:49:27 +02:00
Mike Fährmann
864a63ed33 fix typo 
[skip ci]
 2017-10-10 17:42:06 +02:00
Mike Fährmann
f3fbaa5c3e [reddit] allow users to override the API User-Agent 
Only overriding the Client-ID is not enough if you want to follow
Reddit's API access rules [1].

[1] https://github.com/reddit/reddit/wiki/API#rules
 2017-10-10 17:29:46 +02:00
Mike Fährmann
31ea6001e8 [dynastyscans] improve metadata and filename formats 2017-10-10 17:14:39 +02:00