Commit Graph

6261 Commits

Author SHA1 Message Date
Mike Fährmann
d110dfd2da [tests] update extractor results 2025-01-27 17:15:32 +01:00
Mike Fährmann
b9567ae3e0 [pornpics] fix pagination results from HTML page
prevent URLs containing the root URL twice
2025-01-27 16:12:19 +01:00
Mike Fährmann
b201f91d8d [pornpics] prevent redirect when retrieving gallery page 2025-01-27 16:01:34 +01:00
Mike Fährmann
65863239a0 [webtoons] fix 'username' and 'author_name' extraction 2025-01-27 12:05:40 +01:00
Mike Fährmann
44f4902544 [toyhouse] fix metadata extraction
- date
- artists
- characters
2025-01-27 11:33:17 +01:00
Mike Fährmann
019f9068bf [saint] fix metadata names/IDs of '/d/' URLs 2025-01-27 11:04:41 +01:00
Mike Fährmann
aec5827248 [rule34xyz] fix 'date' and 'tags' extraction 2025-01-27 10:48:11 +01:00
Mike Fährmann
db9cee20c2 [pixiv] fix 'sanity_level' workaround (#4327)
The App API now returns
https://s.pximg.net/common/images/limit_unviewable_360.png
as URL for "Work cannot be displayed" artworks.
2025-01-27 10:10:38 +01:00
Mike Fährmann
61d80df509 [mangaread] fix manga metadata extraction 2025-01-26 21:28:24 +01:00
Mike Fährmann
4bd9b2672c [mangahere] fix chapter extraction 2025-01-26 21:16:24 +01:00
Mike Fährmann
6b6a4942b6 [mangafox] fix chapter extraction 2025-01-26 21:11:36 +01:00
Mike Fährmann
83e50e43a8 [hiperdex] update domain to 'hiperdex.com' 2025-01-26 19:26:03 +01:00
Mike Fährmann
2cdb7e86ca [hiperdex] fix 'description' extraction 2025-01-26 19:17:10 +01:00
Mike Fährmann
21e5a84b6b [architizer] fix extraction
switch " to '
2025-01-26 17:28:06 +01:00
Mike Fährmann
bf361ec7d3 [urlgalleries] support new URL format
... but the site itself is broken, i.e. image pages are empty.
2025-01-26 17:20:28 +01:00
Mike Fährmann
254ffd3fcd [shimmie2] remove 'tentaclerape.net'
"Site Not Found"
2025-01-26 17:02:07 +01:00
Mike Fährmann
d2164af63d [komikcast] update domain to 'komikcast.la' 2025-01-26 16:54:14 +01:00
Mike Fährmann
804fd048ef [szurubooru] remove 'booru.foalcon.com'
DNS record of foalcon.com no longer exists
2025-01-26 16:42:49 +01:00
Mike Fährmann
b271a874ed [fanleaks] remove module
DNS record of fanleaks.club no longer exists
2025-01-26 16:35:46 +01:00
Mike Fährmann
98c068a379 [artstation] simplify '_no_cache()' 2025-01-26 16:12:25 +01:00
Mike Fährmann
954796a466 [artstation] prevent CF challenges (#5817, #5658, #5564, #5554) 2025-01-26 16:00:16 +01:00
Mike Fährmann
4003e4b688 [4archive] fix 'TypeError' when processing 'date' metadata 2025-01-26 12:55:01 +01:00
Mike Fährmann
d9f59ec56e [adultempire] bypass age confirmation check
set 'ageConfirmed' cookie
2025-01-26 11:24:48 +01:00
Mike Fährmann
acaae40254 [facebook] evaluate 'directory' per photo (#6874) 2025-01-24 21:37:20 +01:00
Mike Fährmann
4904186b70 [deviantart] prevent crash when accessing premium_folder_data (#6873)
e.g. for inactive accounts
2025-01-24 19:08:32 +01:00
Mike Fährmann
f5add4048e [nekohouse] fix pagination (#6871)
use distinct names for URL values
2025-01-24 10:18:52 +01:00
Mike Fährmann
783cb83258 merge #6865: [lofter] improve error handling
- Add 'NotFoundError' when the lofter blog returns a 404
- Stop extractor if the blog is empty, because this returns
  {offset: -1} in the data which previously infinity-looped
  the extractor.
- Prevent errors when the blog is locked, the posts in
  self.posts() are None
2025-01-21 19:23:48 +01:00
Mike Fährmann
4d609e284a merge #6833: [kemonoparty] Support /posts endpoint and Creator Tag Calls 2025-01-21 19:22:00 +01:00
BishopRed
b11434a069 [kemonoparty] Support /posts endpoint and Creator Tag Calls
- Adding support for calling a creator with a tag selected.
    It is using a legacy endpoint but there is no other way currently
    documented to get the users post filtered by a tag.
- Fixing the User Tags feature to be paginated
    offset is not defined in the API but it is supported.
- Fixed the `/posts` endpoint not working:
    1. Added check along with metadata to make sure there is a
       creator/service information as that is a requirement
    2. Fixed the parameter from tags -> tag.
    3. Fixed the _paginate call to exit correctly when there is
       a key required for the data (it was prematurely exiting)
- Adding a type of caching mechanism for the metadata/user information.
    The current logic would work just fine if looking up for a
    singular user, however for the multiple posts via normal
    filtering would cause it to either:
    This builds a local cache during the process so it should
    only make a call for the user info once during the process.
- Updating to meet standards
    Fixes
      1. Reset formatting for unnecessary line changes
      2. Removed Type Hinting
      3.Replaced f-string with "".format
   Updates
     Renamed function creator_posts_tags -> creator_tagged_posts
     for clarity of what it does (get posts tags vs get tagged posts)
- Fixing check for the length of response:
    1. If it is list - just check len
    2. If there is a key - check that the key length is less
       than the batch.
- add test for '?tag=...' user URLs
    plus some code simplifications
2025-01-21 19:20:22 +01:00
Mike Fährmann
cb1a75eefc [twitter] handle errors during file extraction (#6647) 2025-01-21 18:23:54 +01:00
Mike Fährmann
d9c4fcc7fa [twitter] generate longer CSRF token values 2025-01-21 18:19:25 +01:00
Jakaline
b05fa669bc Add error messages in lofter extractor 2025-01-21 21:38:20 +09:00
Mike Fährmann
105c027411 [path] handle exception when using --rename-to --no-download (#6861)
Catch a possible FileExistsError exception when attempting to create a
new directory during handling of a FileNotFoundError exception.
FileNotFoundError may also occur when the file at self.temppath is
missing because it hasn't been downloaded due to --no-download.
2025-01-20 20:50:31 +01:00
Mike Fährmann
05fa6dd354 [nekohouse] add initial support (#5241, #6738) 2025-01-20 20:15:34 +01:00
Mike Fährmann
6ce310d865 [weebcentral] fix extraction (#6860) 2025-01-19 18:14:03 +01:00
Mike Fährmann
f867e690c1 merge #6855: [turboimagehost] add support for galleries 2025-01-19 17:51:48 +01:00
Mike Fährmann
0f50dd17ba merge #6606: [docs] add nix docs to README 2025-01-19 17:50:05 +01:00
arebokert
556fbb1a44 [turboimagehost] add support for galleries
- added support
- raise error if gallery not found
- fix test
- fix lint issues
- simplify
2025-01-19 17:28:45 +01:00
DontEatOreo
b15283cf6d README.rst: add nix docs 2025-01-19 17:46:57 +02:00
Mike Fährmann
bb2f9b8443 [release] include 'scripts/run_tests.py' in release tarball (#6856) 2025-01-19 15:58:23 +01:00
Mike Fährmann
438c61601b [xfolio] add initial support (#5514, #6351, #6837) 2025-01-18 15:57:56 +01:00
Mike Fährmann
dc7b46be21 [khinsider] add 'covers' option (#6844) 2025-01-18 15:57:56 +01:00
Mike Fährmann
5a31a2ad22 [khinsider] extract more 'album' metadata (#6844)
- year
- catalog
- developer
- publisher
- uploader
2025-01-18 15:57:55 +01:00
Mike Fährmann
3849b3fa92 [batoto] use 'chapter_id' in default archive IDs (#6835)
instead of '{chapter}{chapter_minor}' since some chapters have no actual
chapter number and end up as '0', potentially causing ID overlap
2025-01-15 14:52:18 +01:00
Mike Fährmann
6e919a3695 [e621] support e621.cc and e621.anthro.fr frontend URLs (#6809) 2025-01-15 14:35:37 +01:00
Mike Fährmann
843a39a6c6 [bunkr] extract correct 'filename' data (#6824) 2025-01-14 19:45:48 +01:00
Mike Fährmann
d17a423245 [xhamster] fix 'gallery' extractor (#6818) 2025-01-13 18:58:08 +01:00
Mike Fährmann
bde99cc6ce [cohost] remove module
cohost.org  now redirects to archive.org
2025-01-13 14:38:35 +01:00
Mike Fährmann
42070240ae [tests] allow testing for types + values 2025-01-12 20:55:37 +01:00
Mike Fährmann
2b46b82f9c [release] prevent overwriting ${CHANGELOG}.orig with truncated file
to avoid deleting most of CHANGELOG.md by accident when the release.sh
script gets interrupted halfway through, as happened during the v1.28.3
release in commit 7e8ca377fc
2025-01-12 18:05:35 +01:00