hdk5 d900e868e4 [arcalive] add support (#5657 #7100) ...

* [arca.live] Add extractor skeleton

* [arcalive] update names and formatting

* [arcalive] implement initial file extraction code

* [arcalive] improve '_extract_media()' performance

compile and cache regex on demand

* [arcalive] improve image extraction

- extract 'data-originalurl' URLs if available
- replace URL query strings with 'type=orig'
- ignore emoticons by default

* [arcalive] update defaults

- include 'title' in filenames
- use 0.5-1.5s delay between requests

* [arcalive] use ext from 'data-orig' if available

* [arcalive] update docs/supportedsites

* [arcalive] add tests

* [arcalive] update 'board' extractor pattern

so it doesn't also match 'post' URLs

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>

2025-03-14 10:52:21 +01:00

..

__init__.py

[arcalive] add support (#5657 #7100)

2025-03-14 10:52:21 +01:00

2ch.py

[2ch] update

2024-01-15 04:09:05 +01:00

2chan.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

2chen.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

3dbooru.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

4archive.py

[4archive] fix 'TypeError' when processing 'date' metadata

2025-01-26 12:55:01 +01:00

4chan.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

4chanarchives.py

[4chanarchives] disable Referer headers by default (#4686)

2023-10-20 17:39:50 +02:00

8chan.py

[8chan] don't make network requests in '_init()' (#6387)

2024-10-27 19:36:49 +01:00

8muses.py

[8muses] skip albums without valid 'permalink' (#6717)

2024-12-24 13:49:19 +01:00

35photo.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

500px.py

send Referer headers by default

2023-09-19 00:02:04 +02:00

adultempire.py

[adultempire] bypass age confirmation check

2025-01-26 11:24:48 +01:00

agnph.py

[agnph] implement 'tags' option (#5284)

2024-07-30 14:16:19 +02:00

ao3.py

[ao3] add 'subscriptions' extractor (#6247)

2024-09-29 13:01:51 +02:00

arcalive.py

[arcalive] add support (#5657 #7100)

2025-03-14 10:52:21 +01:00

architizer.py

[architizer] fix extraction

2025-01-26 17:28:06 +01:00

artstation.py

[artstation] restore 'browser' and 'tls12' defaults

2025-01-28 11:36:06 +01:00

aryion.py

[aryion] update favorite extractor

2024-07-21 12:34:06 +02:00

batoto.py

[batoto] add 'domain' option (#7174)

2025-03-14 10:31:49 +01:00

bbc.py

[common] add '_extract_jsonld' method (#5272)

2025-01-12 11:07:48 +01:00

behance.py

[dl:ytdl] implement explicit HLS/DASH handling

2024-10-16 15:16:21 +02:00

bilibili.py

add bilibili User Articles FavList support

2025-02-23 18:18:45 +01:00

blogger.py

[blogger] use default API key when 'api-key' is empty

2024-11-20 16:02:16 +01:00

bluesky.py

[bluesky] default to /posts if reposts/quoted is enabled (#6583)

2024-12-13 22:24:37 +01:00

booru.py

[booru] allow multiple 'url' keys (#5859)

2024-07-17 20:49:52 +02:00

boosty.py

[boosty] added new direct message extractor

2025-02-23 18:14:59 +01:00

bunkr.py

[bunkr] add 'endpoint' option (#7097)

2025-03-08 08:50:16 +01:00

catbox.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

chevereto.py

[util] implement 'decrypt_xor()'

2025-03-04 16:33:42 +01:00

cien.py

[common] add '_extract_jsonld' method (#5272)

2025-01-12 11:07:48 +01:00

civitai.py

[civitai] fix/improve query parameter handling (#7138)

2025-03-10 20:18:57 +01:00

comicvine.py

fix bug in test_init and update example URLs

2023-09-14 13:27:03 +02:00

common.py

[util] move Cloudflare/DDoS-Guard detection into 'detect_challenge()'

2025-03-06 14:21:35 +01:00

cyberdrop.py

[cyberdrop] add extractor for media URLs (#2496)

2024-12-08 20:57:12 +01:00

danbooru.py

[danbooru:pool] download posts in pool order (#7091)

2025-03-03 16:46:43 +01:00

desktopography.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

deviantart.py

[deviantart] prevent crash when accessing premium_folder_data (#6873)

2025-01-24 19:08:32 +01:00

directlink.py

[directlink] use domain as 'subcategory' (#6703)

2024-12-22 17:19:56 +01:00

discord.py

[discord] add single message support

2025-02-26 22:16:53 +01:00

dynastyscans.py

[dynastyscans] extract chapter 'tags'

2024-07-29 16:50:47 +05:30

e621.py

[danbooru][e621] remove constructors

2025-03-02 20:38:12 +01:00

erome.py

[erome:search] recognize all input URL query parameters (#7125)

2025-03-07 08:23:02 +01:00

everia.py

[everia] update

2024-11-03 17:51:04 +01:00

exhentai.py

[exhentai] implement 'tags' option (#2117)

2024-11-15 21:47:13 +01:00

facebook.py

[facebook] improve 'date' extraction (#7151)

2025-03-10 17:35:32 +01:00

fanbox.py

[fanbox] add 'comments' option, extend 'metadata' option (#6287)

2024-10-06 22:31:41 +02:00

fantia.py

[fantia] simplify 'tags' to a list of strings (#4752)

2023-11-02 15:25:01 +01:00

fapachi.py

[fapachi] fix extraction (#6881)

2025-01-28 11:49:55 +01:00

fapello.py

fapello.py Fullsize image

2024-03-21 18:08:18 +01:00

flickr.py

[flickr] fix video download URLs (#6464)

2024-11-13 20:56:37 +01:00

foolfuuka.py

[b4k] keep support for previous 'arch.b4k.co' domain

2025-02-09 11:11:38 +01:00

foolslide.py

[foolslide] remove 'powermanga.org'

2023-11-26 23:19:05 +01:00

furaffinity.py

[furaffinity] extract 'scraps' metadata (#7015)

2025-03-12 16:29:16 +01:00

furry34.py

[furry34] add support (#1078 #7018)

2025-02-19 16:35:48 +01:00

fuskator.py

[fuskator] make metadata extraction non-fatal (#5039)

2024-01-08 19:18:12 +01:00

gelbooru_v01.py

fix bug in test_init and update example URLs

2023-09-14 13:27:03 +02:00

gelbooru_v02.py

[realbooru] fix extraction (#6543)

2024-12-07 17:39:25 +01:00

gelbooru.py

[gelbooru] improve pagination logic for meta tags (#5478)

2024-04-15 23:14:48 +02:00

generic.py

[generic] fix config lookups by subcategory

2025-02-22 10:08:59 +01:00

gofile.py

[gofile] fix website token extraction (#6596)

2024-12-03 12:04:05 +01:00

hatenablog.py

[hatenablog] implement additional suggestion

2024-01-13 10:43:25 +11:00

hentai2read.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

hentaicosplays.py

[hentaicosplays] update domains (#6578)

2024-12-03 13:56:32 +01:00

hentaifoundry.py

[hentaifoundry] add 'tag' extractor (#6465)

2024-11-13 20:56:37 +01:00

hentaifox.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

hentaihand.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

hentaihere.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

hentainexus.py

[hentainexus] fix error for spread pages (#5827)

2024-07-05 21:36:29 +02:00

hiperdex.py

[hiperdex] update domain to 'hiperdex.com'

2025-01-26 19:26:03 +01:00

hitomi.py

[hitomi] provide 'search_tags' metadata for search/tag results

2025-01-02 17:49:30 +01:00

hotleak.py

[hotleak] fix AttributeError (#5950)

2024-08-07 08:26:23 +02:00

idolcomplex.py

[idolcomplex] fix metadata extraction

2024-03-22 01:43:05 +01:00

imagebam.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

imagechest.py

[imagechest] simplify

2024-11-16 09:17:13 +01:00

imagefap.py

[common] add '_extract_jsonld' method (#5272)

2025-01-12 11:07:48 +01:00

imagehosts.py

[turboimagehost] add support for galleries

2025-01-19 17:28:45 +01:00

imgbb.py

replace '24*3600' with '86400'

2023-12-18 23:57:22 +01:00

imgbox.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

imgth.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

imgur.py

[imgur] replace AuthorizationError exception with logging message

2025-02-15 15:36:39 +01:00

imhentai.py

[imhentai] improve pagination duplicate filtering

2025-02-20 20:53:35 +01:00

inkbunny.py

[inkbunny] fix re-login loop (#6618)

2024-12-07 17:39:25 +01:00

instagram.py

[instagram] fix 'pinned' values for '/reels' results (#6719)

2024-12-25 19:42:50 +01:00

issuu.py

[issuu] unescape HTML entities

2025-02-02 18:33:18 +01:00

itaku.py

[text] add second argument to 'parse_query_list()' (#7138)

2025-03-10 09:36:50 +01:00

itchio.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

jschan.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

kabeuchi.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

keenspot.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

kemonoparty.py

[kemonoparty] Support /posts endpoint and Creator Tag Calls

2025-01-21 19:20:22 +01:00

khinsider.py

[khinsider] add 'covers' option (#6844)

2025-01-18 15:57:56 +01:00

koharu.py

[koharu] implement 'tags' option

2024-11-15 23:49:58 +01:00

komikcast.py

[komikcast] update domain to 'komikcast.la'

2025-01-26 16:54:14 +01:00

lensdump.py

[lensdump] fix extraction (#6313)

2024-10-24 13:59:39 +02:00

lexica.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

lightroom.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

livedoor.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

lofter.py

Add error messages in lofter extractor

2025-01-21 21:38:20 +09:00

lolisafe.py

[saint] fix metadata names/IDs of '/d/' URLs

2025-01-27 11:04:41 +01:00

luscious.py

[luscious] fix IndexError for files without thumbnail (#5122)

2024-01-28 01:43:29 +01:00

lynxchan.py

[lynxchan] update 'bbw-chan' domain (#4970)

2023-12-25 15:29:05 +01:00

mangadex.py

[common] allow overriding more default 'User-Agent' headers (#6496)

2024-11-26 21:50:28 +01:00

mangafox.py

[mangafox] fix chapter extraction

2025-01-26 21:11:36 +01:00

mangahere.py

[mangahere] fix chapter extraction

2025-01-26 21:16:24 +01:00

mangakakalot.py

use 'v[0] == "c"' instead of 'v.startswith("c")'

2024-10-15 08:24:06 +02:00

manganelo.py

[manganelo] fix extraction & recognize '.to' TLDs (#5005)

2024-01-01 22:05:21 +01:00

mangapark.py

[common] add '_extract_nextdata' method

2025-01-12 11:48:36 +01:00

mangaread.py

[mangaread] fix manga metadata extraction

2025-01-26 21:28:24 +01:00

mangasee.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

mangoxo.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

mastodon.py

allow passing cookies to OAuth extractors

2024-11-09 18:06:27 +01:00

message.py

remove 'Message.Metadata' (#866)

2021-01-31 02:12:37 +01:00

misskey.py

[misskey] support 'misskey.design' (#4713)

2023-10-25 17:47:03 +02:00

moebooru.py

[moebooru] add 'metadata' option (#4646)

2023-10-12 21:34:25 +02:00

motherless.py

[motherless] add 'media' and 'gallery' extractors

2024-11-22 21:06:32 +01:00

myhentaigallery.py

[myhentaigallery] fix 'tags' extraction

2024-10-10 17:06:17 +02:00

myportfolio.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

naver.py

[naver] unescape post 'title' and 'description'

2024-03-06 00:46:19 +01:00

naverwebtoon.py

[naverwebtoon] support '/webtoon/' paths for all comics (#5123)

2024-02-04 21:38:46 +01:00

nekohouse.py

[nekohouse] fix pagination (#6871)

2025-01-24 10:18:52 +01:00

newgrounds.py

[newgrounds] provide 'comment_html' metadata (#7038)

2025-02-22 10:08:58 +01:00

nhentai.py

[nhentai] select random file servers for download URLs (#6620)

2024-12-07 17:39:25 +01:00

nijie.py

[nijie] fix image URLs for single image posts (#5842)

2024-07-10 15:06:25 +02:00

nitter.py

[nitter] remove instances

2024-06-02 18:15:53 +02:00

noop.py

[noop] add 'noop' extractor

2024-10-28 19:45:24 +01:00

nozomi.py

use 'True if COND else False' for bool conversions

2024-10-14 20:57:47 +02:00

nsfwalbum.py

[nsfwalbum] detect '/error.jpg' images (#4598)

2023-10-02 15:38:22 +02:00

oauth.py

[oauth] improve wording when opening URL

2025-02-25 20:12:54 +01:00

paheal.py

[paheal] implement fast '--range' support (#5905)

2024-07-30 12:55:57 +02:00

patreon.py

[patreon] extract 'campaign' metadata (#6989)

2025-02-13 14:13:43 +01:00

pexels.py

[pexels] add support (#2286, #4214, #6769)

2025-01-12 16:50:12 +01:00

philomena.py

[philomena] download 'full' URLs (#6922)

2025-02-02 18:23:46 +01:00

photovogue.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

picarto.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

piczel.py

[piczel] fix extraction (#6735)

2024-12-27 15:08:08 +01:00

pillowfort.py

[pillowfort] provide 'count' metadata field (#6478)

2024-11-15 08:27:52 +01:00

pinterest.py

[pinterest] update API headers (#6513)

2024-11-22 08:41:10 +01:00

pixeldrain.py

[pixeldrain] integrate into 'album' extractor

2024-05-25 23:42:23 +02:00

pixiv.py

[pixiv] prevent exceptions during 'comments' extraction (#6965)

2025-02-10 09:56:09 +01:00

pixnet.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

plurk.py

[plurk] fix 'user' data extraction and make it non-fatal (#6742)

2025-01-06 20:27:37 +01:00

poipiku.py

[poipiku] always query 'ShowAppendFileF' when post has warning (#6736)

2024-12-27 20:32:50 +01:00

poringa.py

[poringa] improvements and fixes

2023-12-30 20:37:09 +01:00

pornhub.py

[pornhub] fix KeyError when album images are missing (#6299)

2024-10-09 17:18:06 +02:00

pornpics.py

[pornpics] fix pagination results from HTML page

2025-01-27 16:12:19 +01:00

postmill.py

use 'v[0] == "c"' instead of 'v.startswith("c")'

2024-10-15 08:24:06 +02:00

reactor.py

replace static 'sleep-request' defaults with dynamic ones

2023-12-18 22:06:26 +01:00

readcomiconline.py

[readcomiconline] fix chapter extractor (#6070, #6335)

2024-12-03 10:54:58 +01:00

realbooru.py

[realbooru] fix extraction (#6543)

2024-12-07 17:39:25 +01:00

recursive.py

[recursive] add 'https://' to URLs if not present

2024-12-10 17:16:52 +01:00

reddit.py

[reddit] add 'selftext' option (#7111)

2025-03-08 09:01:50 +01:00

redgifs.py

[redgifs:search] support '/search?query=...' URLs (#7118)

2025-03-07 12:35:35 +01:00

rule34us.py

[rule34us] make 'file_url' extraction more robust (#6289)

2024-10-10 09:11:38 +02:00

rule34vault.py

[rule34vault] update

2024-11-03 09:59:25 +01:00

rule34xyz.py

[rule34xyz] fix 'date' and 'tags' extraction

2025-01-27 10:48:11 +01:00

saint.py

[saint] fix metadata names/IDs of '/d/' URLs

2025-01-27 11:04:41 +01:00

sankaku.py

[sankaku] fix potential infinite loop (#7155)

2025-03-14 08:35:54 +01:00

sankakucomplex.py

[sankakucomplex] update domain to 'news.sankakucomplex.com'

2024-07-26 20:39:55 +02:00

scrolller.py

[scrolller] ignore posts without 'mediaSources' (#5051)

2024-11-26 21:50:28 +01:00

seiga.py

[seiga] re-implement login with username & password

2024-05-23 18:52:53 +02:00

senmanga.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

sexcom.py

[sexcom] add 'likes' extractor (#6149)

2024-09-06 07:44:06 +02:00

shimmie2.py

[shimmie2] remove 'tentaclerape.net'

2025-01-26 17:02:07 +01:00

shopify.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

simplyhentai.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

skeb.py

[skeb] prevent 429 errors and need for 'request_key' cookie

2024-09-26 15:15:37 +02:00

slickpic.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

slideshare.py

[common] add '_extract_nextdata' method

2025-01-12 11:48:36 +01:00

smugmug.py

allow passing cookies to OAuth extractors

2024-11-09 18:06:27 +01:00

soundgasm.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

speakerdeck.py

[speakerdeck] inherit from GalleryExtractor

2024-06-15 21:56:51 +02:00

steamgriddb.py

[steamgriddb] disable 'adjust-extensions' for fake-png files (#5274)

2024-11-20 19:19:23 +01:00

subscribestar.py

[subscribestar] fix 'post' extractor (#6582)

2025-02-22 10:08:59 +01:00

szurubooru.py

[szurubooru] remove 'booru.foalcon.com'

2025-01-26 16:42:49 +01:00

tapas.py

[tapas] fix TypeError for locked episodes (#6700)

2024-12-21 15:17:51 +01:00

tcbscans.py

[tcbscans] support other domains

2024-06-21 21:29:11 +05:30

telegraph.py

use 'v[0] == "c"' instead of 'v.startswith("c")'

2024-10-15 08:24:06 +02:00

tenor.py

[tenor] relax '/view/' URL pattern (#6075)

2025-03-08 15:54:46 +01:00

tiktok.py

[tiktok] implement audio extraction without ytdl

2025-03-12 22:05:24 +01:00

tmohentai.py

[tmohentai] 'categories' -> 'genres'

2023-11-21 22:11:43 +01:00

toyhouse.py

[toyhouse] fix metadata extraction

2025-01-27 11:33:17 +01:00

tsumino.py

use 'v[0] == "c"' instead of 'v.startswith("c")'

2024-10-15 08:24:06 +02:00

tumblr.py

allow passing cookies to OAuth extractors

2024-11-09 18:06:27 +01:00

tumblrgallery.py

[tumblrgallery] fix file downloads (#6391)

2024-10-29 08:38:59 +01:00

twibooru.py

[twibooru] match URLs with 'www' subdomain (#6903)

2025-01-30 19:20:02 +01:00

twitter.py

[twitter] prevent exception in '_extract_components()' (#7139)

2025-03-09 10:15:18 +01:00

unsplash.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

uploadir.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

urlgalleries.py

[urlgalleries] support new URL format

2025-01-26 17:20:28 +01:00

urlshortener.py

add 'config_instance' method

2024-01-18 03:20:36 +01:00

vanillarock.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

vichan.py

[vichan] remove 'wikieat.club'

2024-06-02 18:16:24 +02:00

vipergirls.py

[vipergirls] change default 'domain' to 'viper.click' (4166)

2025-02-26 10:41:51 +01:00

vk.py

[vk] document 'offset', add '--range' support

2024-10-17 21:20:21 +02:00

vsco.py

[vsco] fix extracting videos from '/gallery' results (#7113)

2025-03-04 16:10:48 +01:00

wallhaven.py

[wallhaven] extract 'search[tags]' and 'search[tag_id]' metadata

2025-01-06 17:18:04 +01:00

wallpapercave.py

[wallpapercave] support 'album' listings (#5925)

2024-08-02 16:13:09 +02:00

warosu.py

[warosu] fix extraction

2024-07-26 21:09:07 +02:00

weasyl.py

[weasyl:favorite] update

2024-09-25 20:02:01 +02:00

webmshare.py

fix bug in test_init and update example URLs

2023-09-14 13:27:03 +02:00

webtoons.py

[webtoons] fix 'username' and 'author_name' extraction

2025-01-27 12:05:40 +01:00

weebcentral.py

[weebcentral] fix extracting wrong number of chapter pages (#6966)

2025-02-10 16:00:13 +01:00

weibo.py

[weibo] add 'movies' option (#6988)

2025-02-13 18:01:20 +01:00

wikiart.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

wikifeet.py

remove test results in extractor modules

2023-09-11 16:30:55 +02:00

wikimedia.py

[wikimedia] add 'subcategories' option (#2340)

2025-03-12 22:05:44 +01:00

xfolio.py

[xfolio] add initial support (#5514, #6351, #6837)

2025-01-18 15:57:56 +01:00

xhamster.py

[xhamster] fix 'gallery' extractor (#6818)

2025-01-13 18:58:08 +01:00

xvideos.py

[xvideos] fix galleries with more than 500 images (#5244)

2024-02-26 15:36:41 +01:00

yiffverse.py

[yiffverse] add support (#6611)

2024-12-11 10:57:21 +01:00

ytdl.py

[ytdl] simplify '_process_entries()'

2024-09-03 21:17:31 +02:00

zerochan.py

[zerochan] fix parsing regular JSON

2024-12-14 20:41:08 +01:00

zzup.py

[zzup] support 'up.zzup.com' galleries (#6181)

2024-09-15 11:01:27 +02:00