Commit Graph

472 Commits

Author SHA1 Message Date
Mike Fährmann
3dd1372a0b [scripts/init] use 'args' directly 2025-08-05 21:06:24 +02:00
Mike Fährmann
0364092241 [scripts/init] add '-M/--no-module' command-line option 2025-08-05 19:52:00 +02:00
Mike Fährmann
ef8044f546 [imgdrive] add 'image' extractor (#7976) 2025-08-05 19:50:06 +02:00
Mike Fährmann
3b5d4e0753 [scripts/pyprint] fix various issues
- reintroduce 'lmin' argument
- don't use one-line dict representation
- replace list.extend with actual for loop
2025-08-05 09:27:47 +02:00
Mike Fährmann
86e703f479 [scripts/generate_result] collect URLs 2025-08-03 19:14:01 +02:00
Mike Fährmann
d1f9e8f16b [scripts/generate_result] use 'pyprint' 2025-08-03 19:14:01 +02:00
Mike Fährmann
00c298a8bd [scripts/generate_result] improve insert position 2025-08-03 19:14:01 +02:00
Mike Fährmann
dd14dbe8c6 [scripts/pyprint] implement 'sort' argument 2025-08-02 22:48:46 +02:00
Mike Fährmann
8aa2fe33d5 [booth] add 'item' extractor (#7920) 2025-07-29 16:52:27 +02:00
Mike Fährmann
64f6547aef [scripts/run_tests] use 'TestResult.wasSuccessful()' 2025-07-29 16:52:26 +02:00
Mike Fährmann
1f08f81e67 [scripts/pyprint] fix typo in list generation 2025-07-28 21:50:28 +02:00
Mike Fährmann
876df379f4 [scripts/init] add '-C' command-line option 2025-07-28 21:50:28 +02:00
Mike Fährmann
0a7546201c [scripts/pre-commit] detect debug 'elif' statements 2025-07-28 18:49:08 +02:00
Mike Fährmann
022f216217 [itaku] add 'posts' extractors (#7707) 2025-07-27 19:30:44 +02:00
CautiouslyKate
ed2d07181a [kemono] update extractor to use new domains (#7902 #7907 #7904)
* Update kemono.py
* Update coomer.py
* Update kemono.py
* Fixed TypeError
    - Fixed a regex mistake I made
* Make legacy domains resolve as current domains
* Update gallery_dl/extractor/kemono.py
* remove legacy .su domain support
* fixed accidental regex change
* update docs/supportedsites
* update test results
    - (re)add legacy domain URLs
    - update URL pattern TLDs
* restore support for '.su' TLDs

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2025-07-27 10:10:35 +02:00
Mike Fährmann
c3d385bce0 [scripts/pyprint] support datetime objects 2025-07-27 09:37:46 +02:00
Mike Fährmann
07af386700 [scripts/pyprint] support sets 2025-07-27 09:37:46 +02:00
Mike Fährmann
14c29fe3e0 [scripts/pyprint] restructure dict/list handling 2025-07-27 09:37:46 +02:00
Mike Fährmann
11127726e4 [scripts/pyprint] update maxlen selection
- consider only values smaller than 'lmax' instead of taking the max of
  all lengths and trimming that
- remove 'lmin'
2025-07-27 09:37:45 +02:00
Mike Fährmann
4138ac06e2 [scripts/pyprint] fix indentation 2025-07-25 23:43:08 +02:00
Mike Fährmann
d99b2ab47a [imgadult] add 'image' extractor (#7893) 2025-07-25 18:58:29 +02:00
Mike Fährmann
3eb0b28d6d [facebook] implement 'include' option & add 'avatar' extractor (#7848)
rename 'profile' extractor to 'photos'
2025-07-25 18:20:05 +02:00
Farahat
cf2e5a1619 [leakgallery] add support (#7872)
* add new extractor for leakgallery.com

    Added support for downloading photo and video posts from leakgallery.com.

    Supports:
    * Individual post URLs
    * User profile URLs with pagination via AJAX
    * Optional type/sort filters (e.g. /Photos/MostRecent)
    * Proper file extension handling
    * Creator-based folder structure
    * Compatibility with --download-archive

    Tested locally and functional, but may still need review or improvement.
    
* [leakgallery] add support
    Added leakgallery to extractor module imports so it's recognized and used.
* [leakgallery] update extractor structure
    - Refactored using LeakGalleryExtractorBase to remove duplication
    - Moved init logic into items() using self.groups
    - Replaced re with text.re as per upstream guidance
    - Added creator fallback and media deduplication
    - Aligned structure with gallery-dl maintainer review tips
* [leakgallery] add support
    - Added leakgallery entry to supportedsites.md
    - Includes post, user, trending, and most-liked subcategories
* add exported extractor results
* [leakgallery] fix flake8 style issues
    Cleaned up code to comply with flake8 rules, especially:
    - removed unused imports
    - split long lines >79 chars
    - ensured newline at EOF
    No functional changes made; purely formatting to satisfy CI checks.
* [tests] update extractor results
* [leakgallery] fix flake8 style issues (part 2)
    Fix remaining flake8 issues in leakgallery.py:
    - Reformat line breaks to avoid W503 (line break before binary operator)
    - Wrap long lines to respect E501 (line too long > 79 characters)
    - Cleaned up exception logging for better clarity
    - Confirmed all flake8 checks now pass successfully
    This superseedes the previous commit which partially fixed formatting violations.
* [leakgallery] fix flake8 style issues (part 3)
* [leakgallery] rename extractor classes
* [tests] update extractor results
* [tests] rename extractor results
* [leakgallery] rename extractor classes (part 2)
* [leakgallery] rename example
* update docs/supportedsites
* update test results
    and convert line endings to '\n'
* update
    - convert line endings to '\n'
    - use _pagination method
    - fix logging calls
* return more metadata for _pagination() results
2025-07-22 22:50:25 +02:00
Mike Fährmann
e8b2a496ba [scripts] ensure files use 'utf-8' encoding and '\n' newlines (#7872) 2025-07-22 20:57:54 +02:00
Mike Fährmann
0b991148a1 [civitai] rename 'generate' to 'generated' (#7796) 2025-07-19 18:30:29 +02:00
Mike Fährmann
67a4472bc2 [civitai] add 'generate' extractor (#7796) 2025-07-18 18:34:17 +02:00
Mike Fährmann
1561284815 [madokami] add 'manga' extractor (#7828) 2025-07-17 20:40:26 +02:00
Mike Fährmann
493fc483c6 [scripts/init] handle subdomains when building BASE_PATTERN 2025-07-17 18:38:54 +02:00
Mike Fährmann
df946faf40 [scripts/init] fix extra blank line without copyright
1686f32a0d (commitcomment-162021403)
2025-07-14 16:54:21 +02:00
NecRaul
a7ebb835ea [iwara] Add support (#2652 #5840 #7785)
* [iwara] Add initial support
* [iwara] Add search support
* [iwara] Code cleanup
* [iwara] Small fixes and additions
* [iwara] Add tag support
* [iwara] Add mime-type to metadata
* [iwara] Refactor patterns/matching using urllib
* [iwara] Add unit tests
* [iwara] Update docs
* [iwara] Fix linting on older Python versions
* [iwara] update 'IwaraAPI' interface class
    - define endpoints inside methods
    - implement and use _call() and _pagination()
    - cache auth tokens
* [iwara] split and rename 'profile' extractor
    TODO:
    - update test results
    - simplify code
* [iwara] simplify '_user_params()' usage
* [iwara] update 'video' extractor
    and move user data extraction into 'yield_video'
* [iwara] update 'image' extractor
    and move user info extraction into 'yield_image()'
* [iwara] update 'playlist' extractor
* [iwara] update 'search' extractor
* [iwara] update 'tag' extractor
* [iwara] simplify 'yield_image' usage
    perform API calls to get full 'files' list inside the function
* [iwara] add video "image" test
* [iwara] provide 'date' metadata
* [iwara] simplify 'source()'
    remove urllib.parse usage
* [iwara] small optimizations
    * get("key", {}) -> get("key") or {}
    * split("…", 1) -> partition("…")
    * use f-strings for all patterns
* [iwara] add missing 'keyarg=1' to profile() memcache decorator
* [tests/iwara] update results
* [iwara] extract more 'user' metadata
* [iwara] update default format strings
    include 'date' in filenames to order them chronologically
* [iwara] restructure image/video handling
    - use less generators
    - make processing individual media items non-fatal
* [iwara] fix login and token handling
* [iwara] add 'favorite' extractor
* [iwara] add 'following' and 'followers' extractors

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2025-07-13 21:30:25 +02:00
Mike Fährmann
1686f32a0d [scripts/init] split into separate scripts
- init.py:
    - generate initial extractor module code and test result file
    - insert new entries into modules list and site names
- generate_test_result.py:
    - generate test result for a given URL
    - insert it into the test result file generated by init.py
      (or an already existing one)
2025-07-12 21:14:29 +02:00
Mike Fährmann
b6bd675a9e [scripts/pre-commit] disable user site-packages when running flake8 2025-07-07 15:07:56 +02:00
Mike Fährmann
82891b4d0c [pixiv] move 'novel' extractors to a 'pixiv-novel' category (#7746)
TODO:
- restore full 'include' functionality
- allow remapping category:subcategory pairs
2025-07-04 20:13:19 +02:00
Mike Fährmann
e7922ababd [naver] change categories (#7746)
- 'naver'        -> 'naver-blog'
- 'chzzk'        -> 'naver-chzzk'
- 'naverwebtoon' -> 'naver-webtoon'
2025-07-02 23:20:40 +02:00
Mike Fährmann
5e61fe8668 [rule34xyz] implement login with username & password (#7736) 2025-06-27 22:35:59 +02:00
Mike Fährmann
c1db879b6c [scripts] publish 'pre-commit' hook script (#6582)
https://github.com/mikf/gallery-dl/issues/6582#issuecomment-3010067010
2025-06-26 23:55:12 +02:00
Mike Fährmann
dd759d34dd [scripts/init] support adding test results via '--url' 2025-06-23 17:16:11 +02:00
Mike Fährmann
68960e29a1 [dankefuerslesen] add support (#7669) 2025-06-22 12:13:12 +02:00
Mike Fährmann
eaeabda7ac [scripts] implement 'init.py'
Initial attempt at a helper script to generate new extractor module
files and the required boilerplate code.
2025-06-22 10:13:06 +02:00
Mike Fährmann
60cb4468b2 [options] update --help Usage formatting 2025-06-20 23:43:22 +02:00
Mike Fährmann
1f429da650 [scripts/options] make output width independent of terminal size 2025-06-17 18:52:46 +02:00
SpiffyChatterbox
e0f65be36b [nudostar] add support (#5735 #6556)
* Drafting initial basic extractor layout
* Better debug logging
* Update nudostar.py
    Still tinkering
* Update nudostar.py
    Basic extractor is working. Now starting on Gallery
* Update nudostar.py
    Still a work in progress.
    Got individual posts working, galleries are not.
* Update nudostar.py
* Site now appears working. Added Tests.
* PEP Updates
* PEP - Line Length Updates
* Update nudostar.py
    Resolving PEP8 issues.
* update 'gallery' extractor, rename to 'model'
* update 'image' extractor
* expand tests
* update docs/supportedsites

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2025-06-16 19:21:49 +02:00
SpiffyChatterbox
48ac41605d [redbust] add support (#6759 #6918 #7043)
* init - Redbust.com Support
* Added Test
    Could use a second set of eyes on this
* update 'gallery' extractor
    - extract more metadata
    - simplify image extraction
    - support legacy galleries
* add tests
* update 'image' extractor
* add 'tag' extractor
* add 'archive' extractor
* restrict 'image' extractor pattern
* update docs/supportedsites
* replace quotes inside f-string

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2025-06-16 12:10:42 +02:00
hunter-gatherer8
96f5cfb305 [girlswithmuscle] add support (#4493 #6016)
* [girlswithmuscle] init
* [girlswithmuscle]: fix metadata extraction (site layout change)
* [girlswithmuscle]: fix tags extraction (site layout change)
* update login code
* update 'post' extractor
* update 'gallery' extractor, rename to 'search' extractor
* update docs
* add test cases

---------

Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
2025-06-14 23:05:49 +02:00
Mike Fährmann
3e423937d2 [misskey] implement 'include' option (#5347) 2025-06-06 20:52:03 +02:00
Mike Fährmann
09eed2a515 [docs:supportedsites] update 'civitai' entries (#7608 #7609) 2025-06-01 10:33:31 +02:00
Mike Fährmann
ec523c2c2c [mangasee] remove module 2025-05-30 18:04:55 +02:00
Mike Fährmann
922c296482 [kemono][coomer][schalenetwork] rename modules & extractors
category changes:

- kemonoparty -> kemono
- coomerparty -> coomer
- koharu      -> schalenetwork

also wanted to rename '2chan' -> 'sturdychan',
but the site's main page is still titled '2chen'
2025-05-30 17:51:49 +02:00
Mike Fährmann
129fc00962 [pyinstaller] exclude 'pkg_resources' module (#7592) 2025-05-28 09:30:11 +02:00
bradenhilton
3ba4404d21 [pixeldrain] add support for filesystem URLs (#7473) 2025-05-21 17:28:09 +02:00