Commit Graph

400 Commits

Author SHA1 Message Date
Mike Fährmann
83e50e43a8 [hiperdex] update domain to 'hiperdex.com' 2025-01-26 19:26:03 +01:00
Mike Fährmann
6e919a3695 [e621] support e621.cc and e621.anthro.fr frontend URLs (#6809) 2025-01-15 14:35:37 +01:00
Mike Fährmann
2b46b82f9c [release] prevent overwriting ${CHANGELOG}.orig with truncated file
to avoid deleting most of CHANGELOG.md by accident when the release.sh
script gets interrupted halfway through, as happened during the v1.28.3
release in commit 7e8ca377fc
2025-01-12 18:05:35 +01:00
Mike Fährmann
1d75c8308c [weebcentral] add support (#6778) 2025-01-10 23:04:51 +01:00
Mike Fährmann
167a726972 [szurubooru] support 'visuabusters.com/booru' (#6729) 2024-12-26 19:04:16 +01:00
Mike Fährmann
998f949db1 [civitai] add 'user-videos' extractor (#6644) 2024-12-26 10:18:54 +01:00
Mike Fährmann
a3fb03c943 [release] ensure executables have a minimum size
to prevent issues like #6699 from happening again
2024-12-23 16:07:41 +01:00
Mike Fährmann
63008f77e2 merge #6607: [lofter] add initial support
(#650, #2294, #4095, #4728, #5656)
2024-12-11 20:41:52 +01:00
Mike Fährmann
717081dabd [lofter] update
- add tests
- update docs/supportedsites
- provide 'date' metadata
- simplify/restructure some code
2024-12-11 20:39:01 +01:00
Mike Fährmann
86334f9c4a [yiffverse] add support (#6611) 2024-12-11 10:57:21 +01:00
Mike Fährmann
d96717e2e6 [hentaicosplays] update domains (#6578)
inherit from BaseExtractor to make differentiating between sites easier
2024-12-03 13:56:32 +01:00
Mike Fährmann
bc22c56c90 merge #6501: [docs] update gallery-dl.conf 2024-11-30 09:59:42 +01:00
Mike Fährmann
75c463bb18 [docs] update gallery-dl.conf
add simple script that compares configuration.rst and gallery-dl.conf
2024-11-30 09:58:11 +01:00
Luca Russo
e9370b7b8a merge #5626: [facebook] add support (#470, #2612)
* [facebook] add initial support

* renamed extractors & subcategories

* better stability, modularity & naming

* added single photo extractor, warnings & retries

* more metadata + extract author followups

* renamed "album" mentions to "set" for consistency

* cookies are now only used when necessary

also added author followups for singular images

* removed f-strings

* added way to continue extraction from where it left off

also fixed some bugs

* fixed bug wrong subcategory

* added individual video extraction

* extract audio + added ytdl option

* updated setextract regex

* added option to disable start warning

the extractor should be ready :)

* fixed description metadata bug

* removed cookie "safeguard" + fixed for private profiles

I have removed the cookie "safeguard" (not using cookies until they are necessary) as I've come to the conclusion that it does more harm than good. There is no way to detect whether the extractor has skipped private images, that could have been possibly extracted otherwise. Also, doing this provides little to no advantages.

* fixed a few bugs regarding profile parsing

* a few bugfixes

Fixed some metadata attributes from not decoding correctly from non-latin languages, or not showing at all.
Also improved few patterns.

* retrigger checks

* Final cleanups

-Added tests
-Fixed video extractor giving incorrect URLs
-Removed start warning
-Listed supported site correctly

* fixed regex

* trigger checks

* fixed livestream playback extraction + bugfixes

I've chosen to remove the "reactions", "comments" and "views" attributes as I've felt that they require additional maintenance even though nobody would ever actually use them to order files.
I've also removed the "title" and "caption" video attributes for their inconsistency across different videos.
Feel free to share your thoughts.

* fixed regex

* fixed filename fallback

* fixed retrying when a photo url is not found

* fixed end line

* post url fix + better naming

* fix posts

* fixed tests

* added profile.php url

* made most of the requested changes

* flake

* archive: false

* removed unnecessary url extract

* [facebook] update

- more 'Sec-Fetch-…' headers
- simplify 'text.nameext_from_url()' calls
- replace 'sorted(…)[-1]' with 'max(…)'
- fix '_interval_429' usage
- use replacement fields in logging messages

* [facebook] update URL patterns

get rid of '.*' and '.*?'

* added few remaining tests

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2024-11-26 21:49:11 +01:00
Mike Fährmann
50acf2ac84 [danbooru] add 'artist-search' extractor (#5348) 2024-11-17 16:58:54 +01:00
Mike Fährmann
e5c2882320 [koharu] cleanup
- update BASE_PATTERN formatting
- fix groups indices
- add tests for new domains
- update docs/supportedsites
2024-11-15 22:41:40 +01:00
Mike Fährmann
82d561e825 [bilibili] update
- use self.groups[…] to access matched values
- extract more metadata (count, width, height, size)
- remove type hint
- add tests
- update docs/supportedsites
2024-11-10 17:59:24 +01:00
Mike Fährmann
0b3ddd01af [hiperdex] update domain to 'hipertoon.com' (#6420)
and fix 'description' extraction
2024-11-05 15:54:42 +01:00
Mike Fährmann
cb0d8cae77 merge #6227: [everia] add support (#1067, #2472, #4091) 2024-11-03 17:52:17 +01:00
Mike Fährmann
cea062ffc5 [everia] update
- implement general _pagination method
- simplify code
- adjust URL patterns
- update test results
2024-11-03 17:51:04 +01:00
Mike Fährmann
d787c0c4ea [rule34xyz] add support (#1078, #4960) 2024-11-03 10:12:26 +01:00
Mike Fährmann
655e42dc92 merge #6240: [rule34vault] add support (#5708) 2024-10-28 22:31:05 +01:00
ssdaniel24
3d0263b3ab [rule34vault] Added initial support for rule34vault.com
- Added playlists support for rule34vault
- Added support for posts in rule34vault
- Fixed supported sites with script
- Fixed posts pattern in rule34vault
- Added tests for rule34vault
- Clean
- Fixed lint warnings
2024-10-28 22:26:47 +01:00
Mike Fährmann
5de8576ff6 [noop] add 'noop' extractor 2024-10-28 19:45:24 +01:00
Mike Fährmann
66aa514c25 [scrolller] add initial support (#295, #3418, #5051) 2024-10-21 14:17:18 +02:00
Mike Fährmann
69a75b1de2 [civitai] add extractors for global 'models' and 'images' (#6310) 2024-10-16 23:00:51 +02:00
Mike Fährmann
1df09292b5 consistent __exit__() argument names 2024-10-11 16:43:28 +02:00
Mike Fährmann
9757eacce1 [civitai] add 'post' extractors (#6279)
- https://civitai.com/posts/12345
- https://civitai.com/user/USER/posts
2024-10-06 17:48:48 +02:00
Mike Fährmann
7f945c44f5 [pixiv] support unlisted artworks (#5162) 2024-10-05 17:10:03 +02:00
Mike Fährmann
1ad58cab84 [boosty] add initial support (#2387) 2024-10-02 20:39:55 +02:00
Mike Fährmann
92bbb9bf31 [civitai] add 'image' extractors (#3706, #6220) 2024-09-23 10:13:30 +02:00
Mike Fährmann
3348b05df0 [ao3] implement login with username & password (#6013) 2024-09-21 13:15:50 +02:00
Mike Fährmann
638a676495 [ao3] add initial support (#6013) 2024-09-15 22:38:21 +02:00
Mike Fährmann
df0d7d4a12 [cohost] add 'user' and 'post' extractors (#4483) 2024-09-11 18:03:33 +02:00
Mike Fährmann
f52cf54e16 [docs] update 'wikimedia' capabilities in supportedsites.md 2024-08-31 17:04:22 +02:00
Mike Fährmann
846512f6cd [furaffinity] add 'submissions' extractor (#5954) 2024-08-07 16:34:32 +02:00
Mike Fährmann
aa6d00613f [cien] initial support (#2885, #4103, #5240) 2024-07-28 19:27:12 +02:00
Mike Fährmann
c83c812a1e [instagram][twitter] rename 'profile' to 'info' (#5262, #3623) 2024-07-11 00:22:39 +02:00
Mike Fährmann
44896b0296 [instagram] add 'profile' extractor (#5262)
https://github.com/mikf/gallery-dl/issues/5262#issuecomment-2188915210
2024-06-28 22:54:07 +02:00
Mike Fährmann
9c65db2a92 consistent 'with open(…) as fp:' syntax 2024-06-14 01:22:00 +02:00
Mike Fährmann
9abeab5ecf [shimmie2] support 'vidya.pics' (#5632) 2024-06-06 15:08:56 +02:00
Mike Fährmann
8fce9ea6d5 [hentainexus] restore module (#5275)
revert 97641cd151
2024-06-05 16:48:25 +02:00
Mike Fährmann
cbaf041cea [docs:options] add --update arguments 2024-06-02 18:16:25 +02:00
Mike Fährmann
035d882c6f [docs:supportedsites] include base categories without instances 2024-06-02 18:16:24 +02:00
Mike Fährmann
3d033a48ea [workflows:executables] change dev/linux binary name to …_linux
This reverts commit 05f8435e09.
2024-05-29 02:46:29 +02:00
Mike Fährmann
05f8435e09 [workflows:executables] restore 'gallery-dl_ubuntu' name 2024-05-28 22:20:13 +02:00
Mike Fährmann
601f5becc8 [version] add __variant__
Specifies origin and OS of executable files.
For example 'stable/windows'.
2024-05-27 21:37:01 +02:00
Mike Fährmann
141a93c8fd [docs] update docs/configuration links (#5059, #5369, #5423) 2024-04-13 02:18:44 +02:00
Mike Fährmann
b57051719f [wikimedia] support wiki.gg wikis 2024-04-09 19:24:01 +02:00
Mike Fährmann
28a795ca61 [build] include only the latest CHANGELOG entries
The CHANGELOG is now at a size where it takes up roughly 50kB or 10% of
an sdist or wheel package.
2024-04-03 23:25:36 +02:00