github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5,081 Commits 1 Branch 0 Tags
5ff7106d4f907775f8902c5aded0a68b0e4cf528
Commit Graph

73 Commits

Author SHA1 Message Date
Mike Fährmann
e256434c9e use custom HTTPBasicAuth class 
to support LazyPrompt as password
and to generate the Authorization header only once
instead of for every request
 2023-12-07 01:40:31 +01:00
Mike Fährmann
50e2ebaff0 [danbooru] support 'donmai.moe' URLs 2023-09-22 20:58:38 +02:00
Mike Fährmann
a453335a9f remove test results in extractor modules 
and add generic example URLs
 2023-09-11 16:30:55 +02:00
Mike Fährmann
a383eca7f6 decouple extractor initialization 
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().

This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
 2023-07-25 22:16:16 +02:00
Mike Fährmann
63326e3168 [danbooru] add tests for booruvar 2023-07-18 18:29:57 +02:00
thatfuckingbird
822a77d846 [danbooru] add support for booru.borvar.art instance 2023-05-24 19:06:40 +02:00
Mike Fährmann
494acabd38 [danbooru] refactor pagination logic (#4002) 
- only use 'b<ID>' when no other order is specified
- support 'a<ID>' when  using 'order:id' as tag
 2023-05-14 18:39:59 +02:00
Mike Fährmann
fd0e1ffd6e [danbooru] improve 75666cf9 (#4002) 
Search for direct post IDs instead of trying to
replicate the same results as the initial request.
 2023-05-14 16:35:31 +02:00
Mike Fährmann
708f478d15 [danbooru][e621] add 'date' metadata field (#4047) 2023-05-11 15:59:42 +02:00
Mike Fährmann
75666cf9c3 [danbooru] reduce API requests for fetching extended 'metadata' 
Instead of using one additional API request per post object (N+1),
this requires only one request per 200-post batch.
 2023-04-02 20:11:52 +02:00
Mike Fährmann
3d8777fbc1 move user agent string to util.py 2023-02-22 11:09:17 +01:00
Mike Fährmann
cbe4769246 [danbooru] use gallery-dl UA (#3665) 
this removes the ability to set a custom UA via 'user-agent' option
for extractor requests
 2023-02-18 22:28:35 +01:00
Mike Fährmann
bbf0911a46 [e621] implement 'notes' and 'pools' metadata extraction 
(#3425)
 2023-02-04 21:20:38 +01:00
Mike Fährmann
925b467496 split e621 from danbooru module (#3425) 2023-02-03 19:24:31 +01:00
Mike Fährmann
c87bd1a752 [danbooru] extend 'metadata' option 
make it possible to specify a custom list of metadata includes
 2023-01-13 16:20:01 +01:00
blankie
f82ee93676 [danbooru] extract uploader metadata (if metadata is set) 2022-12-25 11:00:59 +07:00
ClosedPort22
dd4a4a3fa6 [e621] softcode the pagination threshold 2022-12-17 21:06:47 +08:00
ClosedPort22
9faa4ed738 [e621] refactor pagination control 
as suggested by @mikf
 2022-12-17 20:05:34 +08:00
ClosedPort22
d0ad6d0e67 [e621] implement manual pagination mode 2022-12-16 16:15:36 +08:00
Mike Fährmann
e99ce99284 [danbooru] remove stray 'print()' 2022-11-25 18:27:45 +01:00
Mike Fährmann
4e26bf98f5 [aibooru] support 'safe' subdomain (#3110) 2022-10-28 12:10:18 +02:00
thatfuckingbird
062ef238a6 add support for aibooru (using danbooru extractor) (#3075) 2022-10-19 11:53:59 +02:00
Mike Fährmann
d0d4ce1a13 [danbooru] fix ugoira metadata extraction (#3056) 2022-10-15 13:29:09 +02:00
Mike Fährmann
f362d4a3c7 [e621] fix 'popular' extraction 2022-08-26 17:57:17 +02:00
Mike Fährmann
c6a9bab019 update extractor test results 2022-07-12 15:49:22 +02:00
Mike Fährmann
3f02e483c6 [e621] fix applying request_interval_min (#2533) 
Setting this property after calling Extractor.__init__() has no effect.
 2022-04-27 21:10:34 +02:00
Mike Fährmann
563bd0ecf4 [danbooru] inherit from BaseExtractor 
- merge danbooru and e621 code
- support booru.allthefallen.moe (closes #2283)
- remove support for old e621 tag search URLs
 2022-02-11 21:01:51 +01:00
Mike Fährmann
cadfad4eea [danbooru] add 'external' option (closes #1747) 2021-08-08 19:18:41 +02:00
Mike Fährmann
221015e586 [downloader:http] disable filename extension changes for ugoira 
(#1507)
 2021-04-27 01:29:09 +02:00
thatfuckingbird
224b883ff4 [danbooru] add option for extended metadata extraction (#1458) 
* [danbooru] add option for extended metadata extraction

* appease linter

* [danbooru] update docs/configuration.rst

* [danbooru] rename extended-metadata -> metadata
 2021-04-13 23:41:30 +02:00
Mike Fährmann
d781e6ac44 [e621] return pool posts in order (closes #1195) 
… and add a 'num' enumeration index.

A bit more code than the PR version, but it prints some helpful messages
and doesn't call 'metadata()' twice.
 2020-12-26 19:00:29 +01:00
Mike Fährmann
e7d446a8f7 [danbooru] slight code refactoring 2020-12-25 22:06:25 +01:00
Mike Fährmann
1e3dd7330e merge SharedConfigMixin functionality into Extractor 2020-11-17 00:34:07 +01:00
Mike Fährmann
2626629117 [danbooru] handle posts without 'id' (fixes #1004) 2020-09-16 21:35:27 +02:00
Mike Fährmann
e19f665a44 [danbooru] change default for 'ugoira' to 'false' 
Downloading the pre-rendered versions should be a better default
than .zip files with individual frames.
 2020-05-20 19:57:28 +02:00
Mike Fährmann
ad2efa8509 [e621] derive from Danbooru extractors (#651) 
- use extractor implementations from 'danbooru'
- use "page": "b[ID]" to paginate over results instead of
  "tags": "id:<[ID]", avoiding infinite loops with certain
  post orders
- bump User-Agent version
 2020-03-22 21:08:45 +01:00
Mike Fährmann
5bcc7184c9 [danbooru][e621] increase page limits 2020-03-17 15:53:28 +01:00
Mike Fährmann
f117e32910 [danbooru] restore 'popular' functionality 2020-02-29 23:37:53 +01:00
Mike Fährmann
86c00f9e66 [danbooru] move extractor logic from booru.py 2020-02-28 22:53:45 +01:00
Mike Fährmann
f02a768b5c [danbooru] add 'ugoira' option (#406) 
to choose between ZIP archives or converted video files
for Ugoira posts
 2019-08-31 21:51:52 +02:00
Mike Fährmann
6284731107 simplify extractor constants 
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
 2019-02-08 13:45:40 +01:00
Mike Fährmann
1e4d351ad3 [danbooru] add authentication support (closes #151) 
... via HTTP Basic Auth with username and "password".

The password value in this case is not the account password itself,
but the"api_key" found in your user profile.
 2019-01-09 14:19:07 +01:00
Mike Fährmann
e1d306cc48 update unit test results 2018-10-13 16:54:30 +02:00
Mike Fährmann
6ac403c5d3 add postprocessor config example 2018-06-08 18:31:59 +02:00
Mike Fährmann
3905474805 [booru] call update_page() with correct dict (closes #82) 2018-03-19 11:33:19 +01:00
Mike Fährmann
974e73bdbb [booru] smaller code adjustments 2018-01-06 17:48:49 +01:00
Mike Fährmann
1219ebb7f5 [danbooru] use alternate subdomains; support safebooru 2018-01-04 00:51:04 +01:00
Mike Fährmann
9e8a84ab6c [booru] rewrite using Mixin classes (#59) 
- improved code structure
- improved URL patterns
- better pagination to work around page limits on
  - Danbooru
  - e621
  - 3dbooru
 2018-01-04 00:01:39 +01:00
Mike Fährmann
289d6b65d2 [danbooru] extend and improve URL regex 
- add support for danbooru mirrors:
  - hijiribe.donmai.us
  - sonohara.donmai.us
  - todo: actually use these domains instead of redirecting everything
          to danbooru itself
- improve handling of query string parameters
 2017-10-16 21:21:19 +02:00
Mike Fährmann
65997d835b replace popular/ranking tests with older ones 
Metadata of several year old lists shouldn't change as much as it
would for newer ones, which makes metadata-comparisons of the output
of build_testresult_db.oy easier.
 2017-08-31 15:09:18 +02:00