github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4,717 Commits 1 Branch 0 Tags
a27dbe8c820d82291df76afd4fdcefb306c72513
Commit Graph

190 Commits

Author SHA1 Message Date
Mike Fährmann
5b59a0d143 update default User-Agent header to Firefox 115 ESR 2023-07-05 15:12:50 +02:00
Mike Fährmann
856f6c10cd allow for GalleryExtractors to skip loading gallery_url 2023-05-22 22:29:30 +02:00
Mike Fährmann
3ca5dac8b6 extend 'cookies-update' functionality 
Allow writing cookies to a different file than a given cookies.txt,
making it possible to export cookies imported with --cookies-from-browser

To convert browser cookies to cookies.txt format:
  gallery-dl --cookies-fr chromium \
             -o cookies-update=cookies.txt \
             --no-download \
             http://example.org/file.jpg
 2023-05-04 15:10:47 +02:00
Mike Fährmann
bc6d65d203 implement 'Extractor.config_deprecated()' 
a version of 'Extractor.config()'
that logs a warning when using a deprecated option name
 2023-05-04 10:49:14 +02:00
Mike Fährmann
076380e079 remove '*' indicating keyword-only arguments 
they are kind of unnecessary and
cause a non-insignificant function call overhead (~10%)
 2023-05-02 22:23:33 +02:00
Mike Fährmann
9abcb2b6e5 update headers and ciphers for '"browser": "chrome"' 2023-03-08 17:19:59 +01:00
Mike Fährmann
00b94946b3 [instagram] show -o cursor=… after every error (#3440) 2023-01-23 13:00:44 +01:00
Mike Fährmann
80a2ff2d38 support setting 'write-pages' to "ALL" 
to show authentication header, cookies, etc
 2023-01-14 22:34:46 +01:00
Mike Fährmann
c881548a27 add 'extractor.retry-codes' option (#3313) 
do not retry 429 and 430 by default
 2023-01-14 17:25:30 +01:00
Mike Fährmann
9695c4e88d emit debug logging message when loading cookies from file 
attempt nr. 2
no idea how I managed to remove 6514828d in a918ce29
 2023-01-06 11:13:44 +01:00
Mike Fährmann
a918ce29b5 run tests on ubuntu-20.04 
and remove Python 3.4, since that's no longer available
on this test runner
 2023-01-05 13:33:27 +01:00
Mike Fährmann
6514828d4e emit debug logging message when loading cookies from file 2023-01-05 12:40:22 +01:00
Mike Fährmann
9f06e79868 implement '"user-agent": "browser"' (#2636) 2022-11-13 19:17:39 +01:00
Mike Fährmann
86790da2d5 update Cloudflare IUAM detection 
again
 2022-10-31 18:33:52 +01:00
Mike Fährmann
8b1fe0bcf1 emit debug logging messages before calling time.sleep() (#2982) 2022-10-08 15:41:39 +02:00
Mike Fährmann
73a52a95b0 update Cloudflare IUAM detection 2022-09-12 11:40:06 +02:00
Mike Fährmann
eb68d45544 add global 'warnings' option (#2762) 2022-07-18 22:20:30 +02:00
Mike Fährmann
e4f48cc810 make it easier to disable default 'browser' settings 
Previously it was necessary to set 'browser' to a non-empty, non-string
value to disable any default 'browser' value.
Now '-o browser=' or '-o browser=false' is enough.
 2022-07-07 11:17:43 +02:00
Mike Fährmann
92b75bcdce limit path length for --write-pages output on Windows (#2733) 2022-07-06 18:56:23 +02:00
Mike Fährmann
de20cadc68 add 'brotli' as optional dependency (#2716) 
only send 'Accept-Encoding: br' if supported
 2022-06-29 15:10:05 +02:00
Mike Fährmann
3a5d5c3a91 update default User-Agent header to Firefox 102 ESR 
snd update headers and ciphers for "browser": "firefox"
 2022-06-28 17:38:58 +02:00
Mike Fährmann
535cbcb185 cache extracted browser cookies 
(in memory, for as long as gallery-dl is running)

Extracting encrypted cookies from a chromium-based browser can take a
long time, so repeating this process for each extractor should be
avoided.

Same goes for creating a temporary copy of the entire cookie database.
 2022-06-04 12:38:38 +02:00
Mike Fährmann
6742f3bc1e implement --cookies-from-browser (#1606) 
most of the code is adapted from yt-dlp's implementation
and *should* work the same.
 2022-05-07 23:06:37 +02:00
Mike Fährmann
c4b9f7bab8 update functions working with cookies.txt files 
- rename
  - load_cookiestxt -> cookiestxt_load
  - save_cookiestxt -< cookiestxt_store
- in cookiestxt_load, add cookies directly to a cookie jar
  instead of storing them in a list first
- other unnoticeable performance increases
 2022-05-06 13:21:29 +02:00
Mike Fährmann
3f02e483c6 [e621] fix applying request_interval_min (#2533) 
Setting this property after calling Extractor.__init__() has no effect.
 2022-04-27 21:10:34 +02:00
Mike Fährmann
29db716a63 implement 'datetime_to_timestamp()' 
and rename 'to_timestamp()'
to the more descriptive 'datetime_to_timestamp_string()'
 2022-03-23 22:36:01 +01:00
Mike Fährmann
500a479026 fix a third(!) bug in _check_cookies() (#2372) 
turns out tests are worthless if you get em wrong ...
 2022-03-18 19:52:37 +01:00
Mike Fährmann
47cf05c4ab refactor proxy handling code (#2357) 
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
  - add 'downloader.proxy' option
  - '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
    now has the same effect as youtube-dl's '--geo-verification-proxy'
 2022-03-10 23:55:35 +01:00
Mike Fährmann
bddcec49f1 implement 'text.root_from_url()' 
use domain from input URL for kemono
 2022-03-01 03:09:57 +01:00
Mike Fährmann
f5b2b9333f fix another bug in _check:cookies (#2160) 
regression introduced in ed317bfc

Added a couple of tests to hopefully catch such bugs
before they land in a release.
 2022-02-16 22:58:57 +01:00
Mike Fährmann
ed317bfcf1 warn about cookies expiring in less than 24 hours 
requires an expiration timestamp,
so this only works with cookies from a cookies.txt file
 2022-02-13 23:00:49 +01:00
Mike Fährmann
b4f8e15a1f allow BaseExtractors to use the domain pf the matched URL 2022-02-10 01:38:50 +01:00
Mike Fährmann
f58364f6a8 update Firefox cipher list 2022-02-01 02:33:01 +01:00
Mike Fährmann
7e6981dda6 rename 'disabletls12' to 'tls12' 
and let config options override any default settings
 2022-02-01 01:37:03 +01:00
Mike Fährmann
bb3e182562 overhaul session initialization 
- share adapter & connection pool across sessions with the same
  ssl options, ssl ciphers, and source address
- simplify browser emulation to just a list of headers and ciphers
 2022-01-31 23:12:08 +01:00
Robert Pendell
4c651f6252 [patreon] Disable TLS 1.2 by default (#2249) 
Disables TLS 1.2 on Patreon by default.
 2022-01-30 23:30:44 +01:00
Robert Pendell
392cf079f7 Add ability to disable TLS 1.2 (#2243) 
Fix for Patreon Cloudflare issues by having only TLS v1.3 or higher establish HTTPS connections

This now allows you to disable it on a per-host or global basis.  Add disabletls12 as a config option either under extractor.(host) or just under extractor.  Option is false by default.

Example:
        "patreon":
        {
            "disabletls12": true,
            "cookies": {
                "session_id": "X"
            }
        }
 2022-01-30 22:14:43 +01:00
Mike Fährmann
de754590e0 add --source-address command-line option (closes #2206) 2022-01-21 17:07:56 +01:00
Mike Fährmann
6f2e0c9c3d fix cookie checks for patreon, fanbox, fantia 
The changes in 9a255344 caused a warning about missing cookies to be
displayed even if those cookies were present, because _check_cookies()
did not account for an empty cookiedomain.
 2022-01-01 17:55:58 +01:00
Mike Fährmann
ad30653b17 allow running a BaseExtractor for any URL 
by prefixing it with '<base-category>:'

For example:
  shopify:https://partakefoods.com/products/crunchy-cookie-variety-pack
  gelbooru_v01:https://5naf.booru.org/index.php?page=post&s=view&id=46963

Available base categories are:
  mastodon, shopify, moebooru, gelbooru_v01, gelbooru_v02,
  reactor, foolslide, foolfuuka,  philomena
 2021-12-15 00:32:17 +01:00
Mike Fährmann
dad2875a3e fix calculating retry sleep times (fixes #1990) 2021-10-29 23:53:48 +02:00
Mike Fährmann
e69ee41f25 implement 'page-reverse' option (#1854) 2021-09-23 18:02:19 +02:00
Mike Fährmann
c9e6693530 allow specifying a minimum/maximum for 'sleep-*' options (#1835) 
for example '"sleep-request": [5.0, 10.0]' to wait between 5 and 10
seconds between each HTTP request
 2021-09-14 17:40:05 +02:00
Mike Fährmann
2ff2974353 [common] update default argument handling in Extractor.request() 
more lines of code, but slightly less execution time
 2021-09-11 01:26:11 +02:00
Mike Fährmann
d79bcb6236 allow extractors to register a 'finalize()' method 2021-09-07 21:15:30 +02:00
Mike Fährmann
bb6a130942 automatically set required DDoS-GUARD cookies (#1779) 
for kemono.party and seiso.party
 2021-08-16 17:40:29 +02:00
Mike Fährmann
bd08ee2859 remove most 'yield Message.Version' statements 
only leave them in oauth.py as noop results
 2021-08-16 03:10:48 +02:00
Mike Fährmann
9cb5ea5eda update default User-Agent headers 2021-08-14 04:01:41 +02:00
Mike Fährmann
0179581340 add 'T' format string conversion (#1646) 
to convert 'date'/datetime to timestamp
 2021-06-25 22:35:45 +02:00
Mike Fährmann
94faf8c85a add type check before applying 'browser' option (fixes #1358) 2021-03-06 18:15:32 +01:00