github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
7,891 Commits 1 Branch 0 Tags
master
Commit Graph

322 Commits

Author SHA1 Message Date
Mike Fährmann
6cfbc1071f workaround for requests 2.32.3 (#5665) 
manually call 'load_default_certs()' for SSLContexts
in custom HTTPAdapter instances
 2024-06-01 16:02:18 +02:00
Mike Fährmann
28039229fe [common] use 'create_urllib3_context' for creating SSLContexts 
enables dumping TLS session keys by setting SSLKEYLOGFILE (#5215)
as well as other potentially useful settings.
 2024-05-10 22:59:29 +02:00
Mike Fährmann
33006fe126 [common] disable 'check_hostname' for non-urllib3 SSLContexts 
e.g. when 'browser' is set to a non-empty value and gallery-dl creates
its own SSLContext instance instead of using requests' and urllib3's
defaults.

urllib3 disables this option for its default contexts,
since it does this check on its own.

Fixes "ValueError: Cannot set verify_mode to CERT_NONE when
check_hostname is enabled" when using --no-check-certificate.

(#3614, #4891, #5576)
 2024-05-10 18:20:08 +02:00
Mike Fährmann
d11ec00908 [common] fix _cfgpath for BaseExtractor objects 
After the changes in 0c17884673,
_cfgpath was mssing its 'category' value
since that hadn't been initialized yet.
 2024-05-01 16:00:07 +02:00
Mike Fährmann
a7d8cbab0e [common] show full URL in Extractor.request() error messages 2024-04-18 15:45:36 +02:00
Mike Fährmann
a5071c9ca0 [common] fix NameError in Extractor.request() 
… when accessing 'code' after an requests exception was raised.

Caused by the changes in 566472f080
 2024-04-18 15:42:53 +02:00
Mike Fährmann
566472f080 [common] add 'sleep-429' option (#5160) 2024-04-16 18:41:28 +02:00
Mike Fährmann
923c6f3214 [common] simplify 'status_code' check in Extractor.request() 2024-04-16 18:39:47 +02:00
Mike Fährmann
68f4208251 [common] update Extractor.wait() message format 2024-04-16 17:51:14 +02:00
Mike Fährmann
b38a917355 [common] add Extractor.input() method 2024-04-16 00:02:48 +02:00
Mike Fährmann
0d72789aa3 merge #5461: [cookies] use tempfile when saving cookies.txt files 2024-04-13 19:02:39 +02:00
Mike Fährmann
63ac06643f compute tempfile path only once 2024-04-13 18:59:18 +02:00
Mike Fährmann
0c17884673 store 'match' and 'groups' in Extractor objects 2024-04-01 03:07:52 +02:00
Mike Fährmann
106dfdb4c3 cleanup sleep-request retry delay code 
more lines but easier to read I'd say
 2024-03-11 21:38:06 +01:00
Mike Fährmann
89066844f4 add 'config_instance' method 
to allow for a more streamlined access to BaseExtractor instance options
 2024-01-18 03:20:36 +01:00
Mike Fährmann
f36dafad06 improve 'include' handling (#4982) 
- remove spaces when given as string
- warn about invalid vales
 2023-12-28 19:07:04 +01:00
Luc Ritchie
7dd79eee93 save cookies to tempfile, then rename 
avoids wiping the cookies file if the disk is full
 2023-12-11 00:47:42 -05:00
Mike Fährmann
6a4218aa23 handle 'json' parameter in Extractor.request() manually 
Mainly to allow passing custom classes like util.LazyPrompt,
but also to simplify and streamline how requests handles it.
 2023-12-06 22:13:13 +01:00
Mike Fährmann
9dd5cb8c8a interactively prompt for passwords on login when none is provided 2023-12-06 22:12:59 +01:00
Mike Fährmann
34a387b6e2 support 'metadata-*' names for '*-metadata' options 
For example, instead of 'url-metadata' it is now also possible to use
'metadata-url' as option name.

- metadata-url
- metadata-path
- metadata-http
- metadata-version
- metadata-parent
 2023-11-18 23:52:10 +01:00
Mike Fährmann
61d6558322 [exhentai] try to avoid 'DH_KEY_TOO_SMALL' errors (#1021, #4593) 2023-11-04 17:30:27 +01:00
Mike Fährmann
eb230e4b77 [nsfwalbum] disable Referer headers by default (#4598) 2023-10-01 13:55:17 +02:00
Mike Fährmann
3ecb512722 send Referer headers by default 2023-09-19 00:02:04 +02:00
Mike Fährmann
4cdab8074e update/fix --list-extractors 2023-09-11 17:32:59 +02:00
Mike Fährmann
a453335a9f remove test results in extractor modules 
and add generic example URLs
 2023-09-11 16:30:55 +02:00
Mike Fährmann
ceb59e176f fix default Firefox user agent string 
note to self: do not trust some random third-party website
 2023-09-02 22:22:23 +02:00
Mike Fährmann
a4f7f7da17 add '_dump()' convenience method to Extractor 2023-08-06 17:03:09 +02:00
Mike Fährmann
48ef062867 fix issues with 'Extractor.finalize()' 
- prevent crash in InstagramUserExtractor (#4359)
- call it at the end of every DownloadJob
- add it to tests
 2023-07-29 13:43:27 +02:00
Mike Fährmann
ed21908fda initial support for child extractor options 
Using "parent-category>child-category" as extractor category in a config
file allows to set options for a child extractor when it was spawned by
that parent.

For example "reddit>gfycat" to set gfycat options for when it was found
in a reddit post.

{
    "extractor": {
        "gfycat": {
            "filename": "regular filename"
        },
        "reddit>gfycat": {
            "filename": "reddit-specific filename"
        }
    }
}

Note: This does currently not work for most imgur links due to how its
extractor hierarchy is structured.
 2023-07-28 17:07:25 +02:00
Mike Fährmann
a383eca7f6 decouple extractor initialization 
Introduce an 'initialize()' function that does the actual init
(session, cookies, config options) and can called separately from
the constructor __init__().

This allows, for example, to adjust config access inside a Job
before most of it already happened when calling 'extractor.find()'.
 2023-07-25 22:16:16 +02:00
Mike Fährmann
d97b8c2fba consistent cookie-related names 
- rename every cookie variable or method to 'cookies_*'
- simplify '.session.cookies' to just '.cookies'
- more consistent 'login()' structure
 2023-07-22 01:20:50 +02:00
Mike Fährmann
ceebacc9e1 remove 'pyopenssl' option 2023-07-19 20:44:07 +02:00
Mike Fährmann
5b59a0d143 update default User-Agent header to Firefox 115 ESR 2023-07-05 15:12:50 +02:00
Mike Fährmann
856f6c10cd allow for GalleryExtractors to skip loading gallery_url 2023-05-22 22:29:30 +02:00
Mike Fährmann
3ca5dac8b6 extend 'cookies-update' functionality 
Allow writing cookies to a different file than a given cookies.txt,
making it possible to export cookies imported with --cookies-from-browser

To convert browser cookies to cookies.txt format:
  gallery-dl --cookies-fr chromium \
             -o cookies-update=cookies.txt \
             --no-download \
             http://example.org/file.jpg
 2023-05-04 15:10:47 +02:00
Mike Fährmann
bc6d65d203 implement 'Extractor.config_deprecated()' 
a version of 'Extractor.config()'
that logs a warning when using a deprecated option name
 2023-05-04 10:49:14 +02:00
Mike Fährmann
076380e079 remove '*' indicating keyword-only arguments 
they are kind of unnecessary and
cause a non-insignificant function call overhead (~10%)
 2023-05-02 22:23:33 +02:00
Mike Fährmann
9abcb2b6e5 update headers and ciphers for '"browser": "chrome"' 2023-03-08 17:19:59 +01:00
Mike Fährmann
00b94946b3 [instagram] show -o cursor=… after every error (#3440) 2023-01-23 13:00:44 +01:00
Mike Fährmann
80a2ff2d38 support setting 'write-pages' to "ALL" 
to show authentication header, cookies, etc
 2023-01-14 22:34:46 +01:00
Mike Fährmann
c881548a27 add 'extractor.retry-codes' option (#3313) 
do not retry 429 and 430 by default
 2023-01-14 17:25:30 +01:00
Mike Fährmann
9695c4e88d emit debug logging message when loading cookies from file 
attempt nr. 2
no idea how I managed to remove 6514828d in a918ce29
 2023-01-06 11:13:44 +01:00
Mike Fährmann
a918ce29b5 run tests on ubuntu-20.04 
and remove Python 3.4, since that's no longer available
on this test runner
 2023-01-05 13:33:27 +01:00
Mike Fährmann
6514828d4e emit debug logging message when loading cookies from file 2023-01-05 12:40:22 +01:00
Mike Fährmann
9f06e79868 implement '"user-agent": "browser"' (#2636) 2022-11-13 19:17:39 +01:00
Mike Fährmann
86790da2d5 update Cloudflare IUAM detection 
again
 2022-10-31 18:33:52 +01:00
Mike Fährmann
8b1fe0bcf1 emit debug logging messages before calling time.sleep() (#2982) 2022-10-08 15:41:39 +02:00
Mike Fährmann
73a52a95b0 update Cloudflare IUAM detection 2022-09-12 11:40:06 +02:00
Mike Fährmann
eb68d45544 add global 'warnings' option (#2762) 2022-07-18 22:20:30 +02:00
Mike Fährmann
e4f48cc810 make it easier to disable default 'browser' settings 
Previously it was necessary to set 'browser' to a non-empty, non-string
value to disable any default 'browser' value.
Now '-o browser=' or '-o browser=false' is enough.
 2022-07-07 11:17:43 +02:00