github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,429 Commits 1 Branch 0 Tags
da22ea8ced3340700c4ed5c3e5ccdb1ceabcd832
Commit Graph

2429 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mike Fährmann
da22ea8ced use %APPDATA%\gallery-dl for config/cache on Windows 2020-05-28 02:56:38 +02:00
Mike Fährmann
275cceeb6a [redgifs] fix extraction (#724) 
… and prepare for more potential extractors
 2020-05-28 02:18:42 +02:00
Mike Fährmann
45baa13615 update extractor test results 
- don't run Instagram tests on Travis anymore
- replace Twitter test because timeline was made private
- update Hiperdex domain to '.com' (again ...)
 2020-05-28 02:18:06 +02:00
Mike Fährmann
b489f4de12 fix internal links in configuration.rst 2020-05-28 01:51:16 +02:00
Mike Fährmann
dfcf2a2c91 write OAuth token to cache by default (#616) 2020-05-25 22:35:45 +02:00
Mike Fährmann
ddc253cf9a implement a 'path-replace' option (#662, #755) 2020-05-25 22:21:58 +02:00
Mike Fährmann
15c3d29062 move dump_response() into a separate function (#737) 2020-05-25 22:21:58 +02:00
Mike Fährmann
a363da4b43 include redirects and headers in --write-pages dumps (#737) 2020-05-25 22:21:57 +02:00
Mike Fährmann
6bcdb264e0 [imgur] treat 't/unmuted' URLs as galleries 2020-05-25 22:21:57 +02:00
Mike Fährmann
b6cee3e45b [imgur] fix extraction of animated images without 'mp4' entry 2020-05-25 22:21:57 +02:00
Mike Fährmann
7003e61d1d add a few more examples to gallery-dl-example.conf 
- include 'igneous' and 'hath_perks' in Exhentai cookies
- add an example of how to write DeviantArt description to file
- add a 'path-restrict' mapping from invalid characters in Windows
  paths to Unicode alternatives (taken from #662)
 2020-05-25 22:21:57 +02:00
Mike Fährmann
bc53302ad6 extend 'path-restrict' option 
Allow its value to be a JSON object / Python dict that specifies
a mapping from invalid/unwanted input characters to specific
output characters.

For example {"/": "-", "*": "+"} will transform
"foo / ***bar***" into "foo - +++bar+++"

(closes #662, #755)
 2020-05-25 22:21:56 +02:00
Leonardo Taccari
bcac31b7c7 [webtoons] make archive_fmt unique (#779) 
close #778
 2020-05-25 21:23:54 +02:00
Mike Fährmann
34929f673f readd 'session' to base downloader class (fixes #768) 2020-05-20 20:04:46 +02:00
Mike Fährmann
e19f665a44 [danbooru] change default for 'ugoira' to 'false' 
Downloading the pre-rendered versions should be a better default
than .zip files with individual frames.
 2020-05-20 19:57:28 +02:00
Mike Fährmann
3201fe3521 add global SENTINEL object 2020-05-19 22:32:53 +02:00
Mike Fährmann
c8787647ed add global WINDOWS bool 2020-05-19 22:32:53 +02:00
Mike Fährmann
6294e2c540 add 'text.ensure_http_scheme()' 2020-05-19 22:32:53 +02:00
Starz0r
4df2cadf60 Updated README to include additional Windows installation method (#763) 2020-05-18 20:34:08 +02:00
Mike Fährmann
0378d079a5 [webtoons] fixes and simplifications (#593, #761) 
- fix episode listings for french comics
- allow input URLs without explicit scheme
- add 'lang'/'language' metadata
- use str.format() instead of '+' to assemble URLs
 2020-05-18 20:20:03 +02:00
Mike Fährmann
ab11b1c896 [imagechest] simplify code (#750) 2020-05-18 19:11:26 +02:00
Mike Fährmann
ece73b5b2a make 'path' and 'keywords' available in logging messages 
Wrap all loggers used by job, extractor, downloader, and postprocessor
objects into a (custom) LoggerAdapter that provides access to the
underlying job, extractor, pathfmt, and kwdict objects and their
properties.

__init__() signatures for all downloader and postprocessor classes have
been changed to take the current Job object as their first argument,
instead of the current extractor or pathfmt.

(#574, #575)
 2020-05-18 19:04:51 +02:00
Mike Fährmann
846d3a2466 [sexcom] replace 404ed test 2020-05-18 19:04:51 +02:00
Mike Fährmann
9b4635917f [gelbooru] simplify and fix pool extraction 
use 'pool:<pool id>' as search tag to get pool posts
 2020-05-18 19:04:51 +02:00
Mike Fährmann
abbd8fbbd9 reset filenames on empty file extensions (#733) 2020-05-18 19:04:50 +02:00
Mike Fährmann
d17e9628b3 fix/improve Cloudflare bypass code (#728, #757) 
- support changing values for 'k'
- use XML parser to get request parameters
  (some input fields are now embedded in an HTML comment)
 2020-05-18 19:04:50 +02:00
Leonardo Taccari
39cd389679 [webtoons] Add a new extractor for webtoons.com (#761) 
The webtoons extractor can extract episode and entire comic (all
episodes) from webtoons.com.

All the logic of the extractors should be trivial except for a couple
of kludges needed:

 - `ageGatePass' cookie is always set to avoid possible redirect and stop of
    extraction, especially in the comic extractor
 - The image URLs returned by the episode extractor could not be fetched
   directly and the `Referer:' HTTP header needs to be passed to fetch them

Close #593.
 2020-05-18 19:04:20 +02:00
Bepis
7b5711ee04 [imagechest] Add new extractor for ImageChest (#750) 
* [imagechest] Add new extractor for ImageChest

* [imagechest] Fix flake8 compliance issues
 2020-05-18 19:02:56 +02:00
Unit 193
b7ebf51229 Fix typo: defaut → default. (#754) 2020-05-14 23:37:15 +02:00
Mike Fährmann
f8661c6578 [downloader:ytdl] fix file extensions when merging into mkv 2020-05-13 22:35:33 +02:00
Mike Fährmann
a1e739b96c reuse connection adapters from parent extractors 2020-05-12 23:52:01 +02:00
Mike Fährmann
f8f95e68a7 improve '--write-pages' (#737) 
- move code into its own function
- add enumeration index to filenames
- dump responses regardless of status code
 2020-05-12 20:40:25 +02:00
Mike Fährmann
dba87ca99e [downloader:ytdl] change 'forward-cookies' default to 'false' 
There are currently no situations where forwarding gallery-dl's cookies
to youtube-dl is necessary, and it only causes problems when forcing
youtube-dl for Twitter video downloads while logged in.
 2020-05-12 20:17:08 +02:00
Mike Fährmann
09cc9dbec0 prevent flake8 errors from comments looking like type annotations 2020-05-12 20:08:05 +02:00
Mike Fährmann
2d6724180b [hiperdex] update domain to hiperdex.info 2020-05-12 17:00:51 +02:00
Mike Fährmann
c31e7dd573 add MANIFEST.in (#730) 
include LICENSE, CHANGELOG.md, and example config files
in source tarballs created by 'python setup.py sdist'
 2020-05-12 17:00:51 +02:00
Mike Fährmann
be16d77073 simplify and improve documentation markup (#742) 
- use | to get a more "compact" representation
- link directly to example config files
 2020-05-12 17:00:34 +02:00
Vrihub
4cc761c730 Implement --write-pages option (#736) 
* Implement --write-pages option

* Fix long lines

* Fix file mode to binary

* Fix pattern for Windows compatibility
 2020-05-12 14:25:21 +02:00
Vrihub
fe224416bf Some documentation improvements (#742) 
* Link configuration examples in the intro, see #712

* Clarify overriding conf files by options, see #678
 2020-05-10 23:47:20 +02:00
Mike Fährmann
4b606b68e4 skip OAuth tests when server is unreachable 2020-05-10 00:33:00 +02:00
Mike Fährmann
f557cac074 [redgifs] add image extractor (#724) 2020-05-10 00:31:42 +02:00
Mike Fährmann
11d3247c85 use 'util.Formatter' when formatting logging output 2020-05-10 00:09:06 +02:00
Mike Fährmann
8b60bd6a91 mock 'time()' in cache tests 
instead of calling 'sleep()' to let time advance.

This shortens the time needed to run those tests,
and ensures consistent results.
(Tests would randomly fail when using 'sleep()')
 2020-05-09 23:55:14 +02:00
Mike Fährmann
42f29c3e11 improve and simplify attribute access in DownloadJob.initialize() 2020-05-09 00:57:59 +02:00
Mike Fährmann
8f2c1da041 skip example config tests if files are not available (#730) 2020-05-08 22:56:00 +02:00
Mike Fährmann
65b1cb7acd [deviantart] use private access tokens for Journals (fixes #738) 2020-05-08 21:45:01 +02:00
Mike Fährmann
0bf0146bfe [reddit] don't send OAuth headers for file downloads (fixes #729) 2020-05-08 21:42:52 +02:00
Mike Fährmann
ba42ec321c release version 1.13.6 2020-05-02 22:18:56 +02:00
Mike Fährmann
d6a480682f update test results 2020-05-02 21:13:00 +02:00
Mike Fährmann
5df8f2959b insert local directory into PYTHONPATH when running tests 2020-05-02 01:15:50 +02:00