Commit Graph

93 Commits

Author SHA1 Message Date
Mike Fährmann
c7c3fef0bc [exhentai] support '/tag/' URLs (closes #1363) 2021-03-08 22:40:51 +01:00
Mike Fährmann
90830daf85 [exhentai] improve 'favorites' extraction (closes #1360)
add special cases for when the favorite count is 0 (Never) or 1 (Once)
2021-03-07 22:40:25 +01:00
Mike Fährmann
bae874f370 replace 'wait-min/-max' with 'sleep-request'
on exhentai, idolcomplex, reactor
2021-03-02 22:55:45 +01:00
Mike Fährmann
61fbbd2dae [exhentai] rename metadata fields to match API results (#1325)
- gallery_id    -> gid
- gallery_token -> token
- title_jp      -> title_jpn
- visible       -> expunged
- gallery_size  -> filesize
- count         -> filecount

Also changes the function of the 'metadata' option.
It is now boolean and causes extra data fields from the API to be added
instead of completely replacing the data from HTML when activated.
2021-02-26 18:21:04 +01:00
Mike Fährmann
e5735361ed [exhentai] add 'metadata' option (#1325)
to select between gallery metadata from 'api' or 'html'
2021-02-23 00:23:42 +01:00
Mike Fährmann
8f095a0980 [exhentai] extract more metadata from gallery pages (#1325) 2021-02-22 22:58:31 +01:00
Mike Fährmann
9b2e5f72d6 [exhentai] update image URL parsing (#1094) 2020-11-02 15:28:54 +01:00
Mike Fährmann
3ebb174f2c add missing extractor info when spawning new ones (fixes #1051)
Not having this information causes the blacklist/whitelist logic to
trigger and prevents things from functioning as intended when using
default settings.

Fixes issues for 8muses, deviantart, exhentai, and mangoxo.
2020-10-08 14:34:53 +02:00
Mike Fährmann
da87a5fb7e [exhentai] fix accessing config before main constructor
bug introduced with 055c32e0

Making 'Extractor.config()' quite  a bit faster is worth the "cost"
of having to set _cfgpath in exhentai constructors, I think.
2020-09-15 18:09:50 +02:00
Mike Fährmann
a0d916ed41 [exhentai] update wait time before original image download (#978)
depend on 'wait-max', don't use a hard-coded value
2020-09-07 23:48:28 +02:00
Mike Fährmann
0f55b8e80a [exhentai] fix type check from dbbbb21 (#940)
'bool' is a subclass of 'int', and therefore
'isinstance(self.limits, int)' also returns True when
'self.limits' has a boolean value
2020-08-30 20:51:22 +02:00
Mike Fährmann
dbbbb21180 [exhentai] add ability to specify custom image limit (#940) 2020-08-17 22:29:20 +02:00
Mike Fährmann
cd9de613a2 [exhentai] adjust image limit costs (#940)
Each original file costs 10 points per 10^6 bytes,
not 10 per 2^20 == 1048576 bytes.
2020-08-15 18:19:33 +02:00
Mike Fährmann
ecaecc4064 [exhentai] add 'domain' option (#897) 2020-07-17 22:17:46 +02:00
Mike Fährmann
6b373cb7e2 [exhentai] restrict default directory name length (#545) 2019-12-27 21:29:00 +01:00
Mike Fährmann
4409d00141 embed error messages in StopExtraction exceptions 2019-10-28 16:39:49 +01:00
Mike Fährmann
1848788970 update test results etc 2019-09-08 11:33:35 +02:00
Mike Fährmann
9ae58a6b3e [exhentai] update image limit checks
- adjust cost of original images
- delay limit initialization until gallery and first image page have
  been requested and all cookies are available
2019-09-07 13:29:01 +02:00
Mike Fährmann
fa60109e97 [exhentai] don't use e-hentai.org for exhentai URLs 2019-08-02 21:10:09 +02:00
Mike Fährmann
beb4fab2e6 [exhentai] improve limit and error handling (#360)
- check image limit before opening the first gallery or image page
- prevent any further exhentai extractors from running after the image
  limit has been reached
2019-07-30 22:58:35 +02:00
Mike Fährmann
81b35ed3cb [exhentai] catch more error states (#356, #360)
- warn on MPV-enabled galleries
- catch parsing errors for gallery pages and image info
- write page content to debug output
2019-07-29 16:54:31 +02:00
Mike Fährmann
6ce22f606b [exhentai] update login procedure and tests
Logging in now follows the natural login flow that also happens in a
browser more closely and collects more cookies than just ipb_member_id
and ipb_pass_hash.

Test URLs have been updated and now point to the e-hentai.org domain.
2019-07-28 16:51:05 +02:00
Mike Fährmann
dc73d02d87 [exhentai] always use e-hentai.org as domain + set nw cookie 2019-07-28 10:54:17 +02:00
Mike Fährmann
fdec59f8e2 replace extractor.request() 'expect' argument
with
- 'fatal': allow 4xx status codes
- 'notfound': raise NotFoundError on 404
2019-07-05 00:42:16 +02:00
Mike Fährmann
1c36e65e9b [exhentai] choose site version depending on input URL (#278)
Use e-hentai.org as root and cookiedomain if the input URL is from
e-hentai (or g.e-hentai), use exhentai.org otherwise.
2019-05-31 15:34:39 +02:00
Mike Fährmann
1f7fa9dc8e [exhentai] update data extraction code
- parse 'date' to datetime object
- use 'text.extract_from()'
2019-05-08 15:44:29 +02:00
Mike Fährmann
5398bfbd69 [exhentai] fix search and favorite extraction
removes basically all metadata, but that can be compensated for with the
right search query. writing "parsers" for all 4 possible views that have
been introduced in the latest changes is too much of a hassle ...
2019-03-28 16:22:02 +01:00
Mike Fährmann
a2af2d2965 adjust cache maxage values 2019-03-14 22:21:49 +01:00
Mike Fährmann
5530871b5a change results of text.nameext_from_url()
Instead of getting a complete 'filename' from an URL and splitting that
into 'name' and 'extension', the new approach gets rid of the complete
version and renames 'name' to 'filename'. (Using anything other than
{extension} for a filename extension doesn't really work anyway)

Example: "https://example.org/path/filename.ext"

before:
- filename : filename.ext
- name     : filename
- extension: ext

now:
- filename : filename
- extension: ext
2019-02-14 16:07:17 +01:00
Mike Fährmann
61741d7333 provide type information for Queue messages
Child extractors are now directly constructed with Extractor.from_url()
if the extractor class is known beforehand, instead of using
extractor.find() and searching through all possible extractor classes.
2019-02-12 21:32:32 +01:00
Mike Fährmann
2e516a1e3e store the full original URL in Extractor.url 2019-02-12 18:46:48 +01:00
Mike Fährmann
4b1880fa5e propagate 'match' to base extractor constructor 2019-02-11 13:31:10 +01:00
Mike Fährmann
6284731107 simplify extractor constants
- single strings for URL patterns
- tuples instead of lists for 'directory_fmt' and 'test'
- single-tuple tests where applicable
2019-02-08 13:45:40 +01:00
Mike Fährmann
dd358b4564 improve cookie handling during logins 2019-01-30 17:09:32 +01:00
Mike Fährmann
134487ffb0 [exhentai] stop extraction if image limit is exceeded (#141)
can be turned off with the `exhentai.limits' option
2019-01-26 18:40:39 +01:00
Mike Fährmann
e868fb4393 [exhentai] improve gallery extraction
- match image page URLs and extract galleries from that point onward
- add a few more metadata entries: 'parent', 'visible', 'cost'
2019-01-26 18:23:25 +01:00
Mike Fährmann
2ffc105887 [exhentai] extract tag metadata 2019-01-15 18:08:17 +01:00
Mike Fährmann
2801a0d997 [exhentai] skip "Content Warning" page when not logged in
(closes #97)
2018-08-16 09:17:22 +02:00
Mike Fährmann
b8c97d2295 use 'extractor.request()' for more HTTP requests 2018-06-25 23:40:59 +02:00
Mike Fährmann
017188d268 improve extractor.request()
Replace the 'fatal' parameter with 'expect', which is a list/range
of HTTP status codes >= 400 that should also be accepted.
2018-06-18 16:29:56 +02:00
Mike Fährmann
7a58151566 fix util.parse_bytes invocations
(should be text.parse_bytes)
2018-05-10 22:07:55 +02:00
Mike Fährmann
cc36f88586 rename safe_int to parse_int; move parse_* to text module 2018-04-20 14:53:21 +02:00
Mike Fährmann
64d7c85b55 [exhentai] improve metadata
- add 'width', 'height' and 'size' (in bytes) for each image
- change the former 'size' and 'size_units' into 'gallery_size'
2018-04-03 18:59:53 +02:00
Mike Fährmann
52d41c41e7 [exhentai] add extractor for favorited galleries 2018-03-27 18:58:42 +02:00
Mike Fährmann
63cc2599c4 [exhentai] add extractor for search results 2018-03-27 16:50:47 +02:00
Mike Fährmann
34873dbd90 set 'archive_fmt' values
These are going to be used to create an unique id for each image.
2018-02-01 15:30:49 +01:00
Mike Fährmann
e6814aebe2 add 'extractor.*.user-agent' config option 2017-11-15 14:01:33 +01:00
Mike Fährmann
9fc1d0c901 implement and use 'util.safe_int()'
same as Python's 'int()', except it doesn't raise any exceptions and
accepts a default value
2017-09-24 15:59:25 +02:00
Mike Fährmann
6f30cf4c64 change keyword names to valid Python identifiers
This commit mostly replaces all minus-signs ('-') in keyword names with
underscores ('_') to allow them to be used in filter-expressions. For
example 'gallery-id' got renamed to 'gallery_id'.

(It is theoretically possible to access any variable, regardless of its
name, with 'locals()["NAME"]', but that seems a bit too convoluted if
just 'NAME' could be enough)
2017-09-10 22:20:47 +02:00
Mike Fährmann
c0755a4d5e [exhentai] revert login-method to its old version (#37)
Additional cookies don't seem to help and have to be manually set
anyway. The older method is more likely to succeed, so I'd rather
use this one.
2017-08-29 22:10:38 +02:00