Commit Graph

1420 Commits

Author SHA1 Message Date
Deer-Spangle
859f1e7d04 [furaffinity] Adding a FuraffinityFolderExtractor, which extracts a single folder
- Ensure FuraffinityGalleryExtractor doesn't detect folder links
- Fix example URL for folder extractor
- Reordering classes a bit
- Another tweak of the regex
- One more go at the regex..
- cleanup
2025-03-12 14:00:50 +01:00
Mike Fährmann
486e307ecd [reddit] add 'selftext' option (#7111) 2025-03-08 09:01:50 +01:00
Mike Fährmann
3b81d89fef [bunkr] add 'endpoint' option (#7097) 2025-03-08 08:50:16 +01:00
Mike Fährmann
f5073605f6 [tenor] add 'user' extractor (#6075) 2025-03-04 21:47:16 +01:00
Mike Fährmann
2f3265a8ae [tenor] add initial support (#6075) 2025-03-03 19:04:50 +01:00
Mike Fährmann
f232a07faf [danbooru:pool] download posts in pool order (#7091)
- add 'order-posts' option
- add 'num' metadata field for pool position
- update default filenames to order by pool position
2025-03-03 16:46:43 +01:00
Mike Fährmann
4ecc40ce4b [docs] fix 'tiktok-range' default value (#7098) 2025-03-02 14:34:30 +01:00
Mike Fährmann
fa7114ee20 [docs] update supportedsites 2025-02-28 10:48:28 +01:00
Mike Fährmann
afde4ad343 [tiktok] add 'avatar' option 2025-02-26 21:09:57 +01:00
Mike Fährmann
5e87aee32d [tiktok] add 'audio' option (#7060) 2025-02-26 21:02:33 +01:00
Mike Fährmann
13c3fa45f7 [docs] add 'tiktok' options (#7060) 2025-02-26 20:45:25 +01:00
Mike Fährmann
203c2e3492 [vipergirls] change default 'domain' to 'viper.click' (4166)
https://github.com/mikf/gallery-dl/issues/4166#issuecomment-2684014628

and update general 'domain' handling
2025-02-26 10:41:51 +01:00
Mike Fährmann
876169ded5 [furaffinity] use a 1s delay between requests by default (#7054) 2025-02-25 20:12:54 +01:00
CasualYouTuber31
daac2c6e04 [tiktok] add support (#3061 #4177 #5646 #6878 #6708)
* Add TikTok photo support

#3061
#4177

* Address linting errors

* Fix more test failures

* Forgot to update category names in tests

* Looking into re issue

* Follow default yt-dlp output template

* Fix format string error on 3.5

* Support downloading videos and audio

Respond to comments
Improve archiving and file naming

* Forgot to update supportedsites.md

* Support user profiles

* Fix indentation

* Prevent matching with more than one TikTok extractor

* Fix TikTok regex

* Support TikTok profile avatars

* Fix supportedsites.md

* TikTok: Ignore no formats error

In my limited experience, this doesn't mean that gallery-dl can't download the photo post (but this could mean that you can't download the audio)

* Fix error reporting message

* TikTok: Support more URL formats

vt.tiktok.com
www.tiktok.com/t/

* TikTok: Only download avatar when extracting user profile

* TikTok: Document profile avatar limitation

* TikTok: Add support for www.tiktokv.com/share links

* Address Share -> Sharepost issue

* TikTok: Export post's creation date in JSON (ISO 8601)

* [tiktok] update

* [tiktok] update 'vmpost' handling

just perform a HEAD request and handle its response

* [tiktok] build URLs from post IDs

instead of reusing unchanged input URLs

* [tiktok] combine 'post' and 'sharepost' extractors

* [tiktok] update default filenames

put 'id' and 'num' first to ensure better file order

* [tiktok] improve ytdl usage

- speed up extraction by passing '"extract_flat": True'
- pass more user options and cookies
- pre-define 'TikTokUser' extractor usage

* [tiktok] Add _COOKIES entry to AUTH_MAP

* [tiktok] Always download user avatars

* [tiktok] Add more documentation to supportedsites.md

* [tiktok] Address review comments

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2025-02-25 20:10:48 +01:00
Mike Fährmann
a9853cd273 merge #6781: [bilibili] add 'user-articles-favorite' extractor (#6725) 2025-02-23 18:19:51 +01:00
mmmpipi
e4cc3419c5 add bilibili User Articles FavList support
- fix whitespace
- fix extractor names
- Add favlist url user check
- apply changes
- add test
- update docs/supportedsites
2025-02-23 18:18:45 +01:00
Dominik Prange
ff5f6fe70f [boosty] added new direct message extractor
- formatting
- fixed linting formatting errors
- fixed E999 SyntaxError: invalid syntax
- fixed class naming
- fixed mandatory extractor.boosty.metadata as true requirement
- update
  - apply changes
  - add test
  - update docs/supportedsites
- improve 'dialog' pagination logic
2025-02-23 18:14:59 +01:00
Mike Fährmann
52d4e1a100 [imhentai] inherit from BaseExtractor
combine all imhentai-like sites into one module
2025-02-19 22:14:52 +01:00
Mike Fährmann
d4c56b08d7 [hentaiera] add support (#3046 #6952 #7020) 2025-02-19 17:42:04 +01:00
Mike Fährmann
4396029d36 [furry34] add support (#1078 #7018) 2025-02-19 16:35:48 +01:00
Mike Fährmann
82493a6672 [hentairox] add support (#7003) 2025-02-18 21:45:30 +01:00
Luca Russo
95c446fcd1 [discord] add support (#6836)
* first commit

* add --

* skip video embeds

* fix typo

* removed ambiguity

* add category support

* code tweaks

* more reliable embed extraction

* handle 403 errors (testing done)

* added "parent_id" keyword

* added "parent", "parent_type" keywords

the extractor should be now ready to merge!

* removed unnecessary dict unpacking

* added empty text messages extraction

* added "channel_topic"

* even more metadata extraction

can now extract all embeds images & text, as well as server banners. also code is much better.

* added user avatar and banner

* better pagination

* fix regression

* minor tweaks

* Made requested changes
2025-02-18 18:45:39 +01:00
Mike Fährmann
fd4de02e67 [archive] support PostgreSQL archives for post processors (#6152) 2025-02-17 14:58:14 +01:00
Mike Fährmann
8daf496a22 [archive] add 'archive-table' option (#6152) 2025-02-17 11:41:13 +01:00
Mike Fährmann
841bc9f66f [archive] implement support for PostgreSQL databases (#6152) 2025-02-16 17:56:52 +01:00
Mike Fährmann
35307608f2 [dl:http] add 'sleep-429' option (#6996) 2025-02-15 17:42:03 +01:00
Mike Fährmann
182b544217 [ytdl] support specifying filesystem paths as 'module' (#6991) 2025-02-14 19:58:25 +01:00
Mike Fährmann
7ae09c6b29 [imgur] add support for (hidden) personal posts (#6990)
https://imgur.com/user/me
https://imgur.com/user/me/hidden
2025-02-14 19:28:55 +01:00
Mike Fährmann
51f978e027 [weibo] add 'movies' option (#6988)
disable 'movie' downloads by default
2025-02-13 18:01:20 +01:00
Mike Fährmann
f1f27eb2ab [vsco] support '/video/' URLs (#4295 #6973)
requires yt-dlp/youtube-dl to handle m3u8 manifests
2025-02-12 19:12:00 +01:00
Mike Fährmann
55034d9638 [imhentai] add support (#1660 #3046 #3824 #4338 #5936) 2025-02-10 21:42:07 +01:00
NecRaul
dae82f1519 [b4k] update domain to arch.b4k.dev 2025-02-09 01:28:23 +04:00
Mike Fährmann
873cbf6b36 [docs] add more details to 'user-agent' and 'browser' docs (#6917) 2025-02-03 20:54:27 +01:00
Mike Fährmann
4874c8e1d1 [artstation] restore 'browser' and 'tls12' defaults
partially revert 954796a466
2025-01-28 11:36:06 +01:00
Mike Fährmann
83e50e43a8 [hiperdex] update domain to 'hiperdex.com' 2025-01-26 19:26:03 +01:00
Mike Fährmann
254ffd3fcd [shimmie2] remove 'tentaclerape.net'
"Site Not Found"
2025-01-26 17:02:07 +01:00
Mike Fährmann
d2164af63d [komikcast] update domain to 'komikcast.la' 2025-01-26 16:54:14 +01:00
Mike Fährmann
804fd048ef [szurubooru] remove 'booru.foalcon.com'
DNS record of foalcon.com no longer exists
2025-01-26 16:42:49 +01:00
Mike Fährmann
b271a874ed [fanleaks] remove module
DNS record of fanleaks.club no longer exists
2025-01-26 16:35:46 +01:00
Mike Fährmann
954796a466 [artstation] prevent CF challenges (#5817, #5658, #5564, #5554) 2025-01-26 16:00:16 +01:00
Mike Fährmann
05fa6dd354 [nekohouse] add initial support (#5241, #6738) 2025-01-20 20:15:34 +01:00
Mike Fährmann
f867e690c1 merge #6855: [turboimagehost] add support for galleries 2025-01-19 17:51:48 +01:00
arebokert
556fbb1a44 [turboimagehost] add support for galleries
- added support
- raise error if gallery not found
- fix test
- fix lint issues
- simplify
2025-01-19 17:28:45 +01:00
Mike Fährmann
438c61601b [xfolio] add initial support (#5514, #6351, #6837) 2025-01-18 15:57:56 +01:00
Mike Fährmann
dc7b46be21 [khinsider] add 'covers' option (#6844) 2025-01-18 15:57:56 +01:00
Mike Fährmann
6e919a3695 [e621] support e621.cc and e621.anthro.fr frontend URLs (#6809) 2025-01-15 14:35:37 +01:00
Mike Fährmann
bde99cc6ce [cohost] remove module
cohost.org  now redirects to archive.org
2025-01-13 14:38:35 +01:00
Mike Fährmann
91bd3e37f2 [pexels] add support (#2286, #4214, #6769) 2025-01-12 16:50:12 +01:00
Mike Fährmann
1d75c8308c [weebcentral] add support (#6778) 2025-01-10 23:04:51 +01:00
Mike Fährmann
b1ffb62644 [docs] update 'sleep-request' value for 'wallhaven' 2025-01-06 17:24:04 +01:00