github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6,261 Commits 1 Branch 0 Tags
d110dfd2da0d12fb2cb9abb0495d20eaf5afa939
Commit Graph

83 Commits

Author SHA1 Message Date
Mike Fährmann
939cf51b01 [danbooru] add missing ':' to 'md5' tag prefix 2024-12-07 17:39:25 +01:00
Mike Fährmann
63e042dec7 [e621] fix 'TypeError' when 'metadata' is enabled (#6587) 
fixes regression introduced in 9184a564
 2024-12-02 14:09:38 +01:00
Mike Fährmann
5412b22dae [common] allow overriding more default 'User-Agent' headers (#6496) 
ignore 'extractor.user-agent' if it is the default useragent value
and an extractor wants to set its own custom value
 2024-11-26 21:50:28 +01:00
Mike Fährmann
50acf2ac84 [danbooru] add 'artist-search' extractor (#5348) 2024-11-17 16:58:54 +01:00
Mike Fährmann
ca4b2a0760 [danbooru] add 'artist' extractor (#5348) 2024-11-17 16:23:16 +01:00
Mike Fährmann
9184a5643a [danbooru] move all initialization code into '_init()' 2024-11-17 16:15:53 +01:00
Mike Fährmann
f8bb0fd6cb [pixiv][danbooru] set '_ugoira_original' for ugoira results 2024-10-11 13:19:45 +02:00
Mike Fährmann
cc38042500 [pp:ugoira] forward frame data as '_ugoira_frame_data' (#6154, #6285) 2024-10-08 11:28:44 +02:00
Mike Fährmann
627ed794a2 [danbooru] provide 'tags' as list (#4942) 
keep the old 'tag_string' values around, similar to sankaku

a lot of repeat code ...
would be a lot less bad if "".split(" ") returned an empty list
 2023-12-21 14:39:38 +01:00
Mike Fährmann
1f9b16a70b replace static 'sleep-request' defaults with dynamic ones 2023-12-18 22:06:26 +01:00
Mike Fährmann
e256434c9e use custom HTTPBasicAuth class 
to support LazyPrompt as password
and to generate the Authorization header only once
instead of for every request
 2023-12-07 01:40:31 +01:00
Mike Fährmann
50e2ebaff0 [danbooru] support 'donmai.moe' URLs 2023-09-22 20:58:38 +02:00
Mike Fährmann
a453335a9f remove test results in extractor modules 
and add generic example URLs
 2023-09-11 16:30:55 +02:00
Mike Fährmann
a383eca7f6 decouple extractor initialization 
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().

This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
 2023-07-25 22:16:16 +02:00
Mike Fährmann
63326e3168 [danbooru] add tests for booruvar 2023-07-18 18:29:57 +02:00
thatfuckingbird
822a77d846 [danbooru] add support for booru.borvar.art instance 2023-05-24 19:06:40 +02:00
Mike Fährmann
494acabd38 [danbooru] refactor pagination logic (#4002) 
- only use 'b<ID>' when no other order is specified
- support 'a<ID>' when  using 'order:id' as tag
 2023-05-14 18:39:59 +02:00
Mike Fährmann
fd0e1ffd6e [danbooru] improve 75666cf9 (#4002) 
Search for direct post IDs instead of trying to
replicate the same results as the initial request.
 2023-05-14 16:35:31 +02:00
Mike Fährmann
708f478d15 [danbooru][e621] add 'date' metadata field (#4047) 2023-05-11 15:59:42 +02:00
Mike Fährmann
75666cf9c3 [danbooru] reduce API requests for fetching extended 'metadata' 
Instead of using one additional API request per post object (N+1),
this requires only one request per 200-post batch.
 2023-04-02 20:11:52 +02:00
Mike Fährmann
3d8777fbc1 move user agent string to util.py 2023-02-22 11:09:17 +01:00
Mike Fährmann
cbe4769246 [danbooru] use gallery-dl UA (#3665) 
this removes the ability to set a custom UA via 'user-agent' option
for extractor requests
 2023-02-18 22:28:35 +01:00
Mike Fährmann
bbf0911a46 [e621] implement 'notes' and 'pools' metadata extraction 
(#3425)
 2023-02-04 21:20:38 +01:00
Mike Fährmann
925b467496 split e621 from danbooru module (#3425) 2023-02-03 19:24:31 +01:00
Mike Fährmann
c87bd1a752 [danbooru] extend 'metadata' option 
make it possible to specify a custom list of metadata includes
 2023-01-13 16:20:01 +01:00
blankie
f82ee93676 [danbooru] extract uploader metadata (if metadata is set) 2022-12-25 11:00:59 +07:00
ClosedPort22
dd4a4a3fa6 [e621] softcode the pagination threshold 2022-12-17 21:06:47 +08:00
ClosedPort22
9faa4ed738 [e621] refactor pagination control 
as suggested by @mikf
 2022-12-17 20:05:34 +08:00
ClosedPort22
d0ad6d0e67 [e621] implement manual pagination mode 2022-12-16 16:15:36 +08:00
Mike Fährmann
e99ce99284 [danbooru] remove stray 'print()' 2022-11-25 18:27:45 +01:00
Mike Fährmann
4e26bf98f5 [aibooru] support 'safe' subdomain (#3110) 2022-10-28 12:10:18 +02:00
thatfuckingbird
062ef238a6 add support for aibooru (using danbooru extractor) (#3075) 2022-10-19 11:53:59 +02:00
Mike Fährmann
d0d4ce1a13 [danbooru] fix ugoira metadata extraction (#3056) 2022-10-15 13:29:09 +02:00
Mike Fährmann
f362d4a3c7 [e621] fix 'popular' extraction 2022-08-26 17:57:17 +02:00
Mike Fährmann
c6a9bab019 update extractor test results 2022-07-12 15:49:22 +02:00
Mike Fährmann
3f02e483c6 [e621] fix applying request_interval_min (#2533) 
Setting this property after calling Extractor.__init__() has no effect.
 2022-04-27 21:10:34 +02:00
Mike Fährmann
563bd0ecf4 [danbooru] inherit from BaseExtractor 
- merge danbooru and e621 code
- support booru.allthefallen.moe (closes #2283)
- remove support for old e621 tag search URLs
 2022-02-11 21:01:51 +01:00
Mike Fährmann
cadfad4eea [danbooru] add 'external' option (closes #1747) 2021-08-08 19:18:41 +02:00
Mike Fährmann
221015e586 [downloader:http] disable filename extension changes for ugoira 
(#1507)
 2021-04-27 01:29:09 +02:00
thatfuckingbird
224b883ff4 [danbooru] add option for extended metadata extraction (#1458) 
* [danbooru] add option for extended metadata extraction

* appease linter

* [danbooru] update docs/configuration.rst

* [danbooru] rename extended-metadata -> metadata
 2021-04-13 23:41:30 +02:00
Mike Fährmann
d781e6ac44 [e621] return pool posts in order (closes #1195) 
… and add a 'num' enumeration index.

A bit more code than the PR version, but it prints some helpful messages
and doesn't call 'metadata()' twice.
 2020-12-26 19:00:29 +01:00
Mike Fährmann
e7d446a8f7 [danbooru] slight code refactoring 2020-12-25 22:06:25 +01:00
Mike Fährmann
1e3dd7330e merge SharedConfigMixin functionality into Extractor 2020-11-17 00:34:07 +01:00
Mike Fährmann
2626629117 [danbooru] handle posts without 'id' (fixes #1004) 2020-09-16 21:35:27 +02:00
Mike Fährmann
e19f665a44 [danbooru] change default for 'ugoira' to 'false' 
Downloading the pre-rendered versions should be a better default
than .zip files with individual frames.
 2020-05-20 19:57:28 +02:00
Mike Fährmann
ad2efa8509 [e621] derive from Danbooru extractors (#651) 
- use extractor implementations from 'danbooru'
- use "page": "b[ID]" to paginate over results instead of
  "tags": "id:<[ID]", avoiding infinite loops with certain
  post orders
- bump User-Agent version
 2020-03-22 21:08:45 +01:00
Mike Fährmann
5bcc7184c9 [danbooru][e621] increase page limits 2020-03-17 15:53:28 +01:00
Mike Fährmann
f117e32910 [danbooru] restore 'popular' functionality 2020-02-29 23:37:53 +01:00
Mike Fährmann
86c00f9e66 [danbooru] move extractor logic from booru.py 2020-02-28 22:53:45 +01:00
Mike Fährmann
f02a768b5c [danbooru] add 'ugoira' option (#406) 
to choose between ZIP archives or converted video files
for Ugoira posts
 2019-08-31 21:51:52 +02:00