Commit Graph

17 Commits

Author SHA1 Message Date
Mike Fährmann
a383eca7f6 decouple extractor initialization
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().

This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
2023-07-25 22:16:16 +02:00
Mike Fährmann
d97b8c2fba consistent cookie-related names
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
2023-07-22 01:20:50 +02:00
Mike Fährmann
b0cb4a1b9c replace 'text.extract()' with 'text.extr()' where possible 2022-11-05 01:14:09 +01:00
Mike Fährmann
a3bf878329 [idolcomplex] improve and fix pagination (#1601)
always rely on the 'next-page-url' value and its query parameters
2021-06-04 20:31:08 +02:00
Mike Fährmann
78f89d2e61 [idolcomplex] fix pagination (closes #1594) 2021-06-02 15:32:46 +02:00
Mike Fährmann
bae874f370 replace 'wait-min/-max' with 'sleep-request'
on exhentai, idolcomplex, reactor
2021-03-02 22:55:45 +01:00
Mike Fährmann
b2c55f0a72 [sankaku] remove login support
The old login method for 'https://chan.sankakucomplex.com/user/login'
and the cookies it produces have no effect on the results from
'beta.sankakucomplex.com'.
2020-12-08 21:05:47 +01:00
Mike Fährmann
ecdea799dd [sankaku] use 'beta.sankakucomplex.com' API endpoints 2020-12-05 22:08:58 +01:00
Mike Fährmann
6284731107 simplify extractor constants
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
2019-02-08 13:45:40 +01:00
Mike Fährmann
ecad69100a [photobucket] add 'image' extractor (#117) 2019-01-22 17:24:43 +01:00
Mike Fährmann
d69db60e2a update unit test results 2018-10-02 20:37:46 +02:00
Mike Fährmann
d98e47817d [deviantart] reduce refresh-token usage
Instead of using a refresh-token-based access-token for every API
request, they are now only used for paginated results.

API requests to get a user's profile and the original download URL
now always use a public access-token.
2018-07-24 17:32:46 +02:00
Mike Fährmann
269dc2bbd5 [sankaku] add 'tags' option (#94) 2018-07-14 09:53:01 +02:00
Mike Fährmann
829ddf4ac1 [sankaku] general improvements
- simplify regex
- unquote search tags
- increase default wait-time between HTTP requests
  - downloading several hundreds of images always resulted
    in '429 Too Many Requests' eventually
- circumvent paging restrictions for unauthenticated users by only
  using the 'next' parameter
  - setting 'page' to a constant, low value (or simply omitting it)
    does the trick
2018-02-27 16:51:14 +01:00
Jad
49463f76bb support multi-page URL (#79)
* support multi-page URL

* fix

* all done.

* fix, again
2018-02-26 11:13:49 +01:00
Mike Fährmann
e420a28bbc fix cookie tests 2018-01-09 21:43:52 +01:00
Mike Fährmann
b33efc99a4 [idolcomplex] add support for idol.sankakucomplex.com 2018-01-09 17:54:37 +01:00