github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6,320 Commits 1 Branch 0 Tags
e4cc3419c58ca951d0ef9a5fc07ec611944e6ef2
Commit Graph

158 Commits

Author SHA1 Message Date
Mike Fährmann
18ed39c1cf implement 'downloader' options per extractor category 
by setting options inside 'http' or 'ytdl' inside extractor options
or inside subcategory options

{
    "extractor": {
        "mastodon": {
            "http": {
                "rate": "10k"
            }
        },
        "mastodon.social": {
            "http": {
                "rate": "100k"
            }
        }
    },
    "downloader": {
        "rate": "100m"
    }
}

Sets download speed to
-  10k for mastodon.social URLs
- 100k for mastodon sites in general
- 100m for all other sites
 2025-02-22 10:08:59 +01:00
Mike Fährmann
800cf5beb5 replace 'print()' with 'output.stderr_write("\n")' 2025-02-15 18:01:05 +01:00
Mike Fährmann
35307608f2 [dl:http] add 'sleep-429' option (#6996) 2025-02-15 17:42:03 +01:00
Mike Fährmann
e8826ed3d4 [common] simplify HTTP error messages 
[warning] HTTPSConnectionPool(host='domain.tld', port=443): Max retries
exceeded with url: /a.jpg (Caused by NameResolutionError("<urllib3.
connection.HTTPSConnection object at 0x7247fe436ea0>: Failed to resolve
'domain.tld' ([Errno -2] Name or service not known)")) (1/5)

->

[warning] NameResolutionError: Failed to resolve 'domain.tld'
([Errno -2] Name or service not known) (1/5)
 2024-12-10 17:13:44 +01:00
Mike Fährmann
f79e57b71e [dl:ytdl] change 'forward-cookies' default to 'true' (#6401, #6348) 
revert dba87ca99e
 2024-10-31 17:35:08 +01:00
Mike Fährmann
1824267447 [dl:ytdl] implement explicit HLS/DASH handling 
add '_ytdl_manifest' to specify a manifest type to process
 2024-10-16 15:16:21 +02:00
Mike Fährmann
a051e1c955 directly pass exception instances as 'exc_info' logger argument 2024-09-19 14:50:08 +02:00
Mike Fährmann
127aa45834 [ytdl] ignore SyntaxErrors when trying to import a module 2024-08-29 19:28:08 +02:00
Mike Fährmann
9c65db2a92 consistent 'with open(…) as fp:' syntax 2024-06-14 01:22:00 +02:00
Mike Fährmann
3ec7ee6baa [downloader:ytdl] fix exception due to missing 'ext' (#5675) 2024-06-02 18:16:53 +02:00
Mike Fährmann
cd241bea0a [downloader:http] add MIME type and signature for .m4v files (#5505) 2024-04-25 01:01:35 +02:00
Mike Fährmann
bffadf35b7 [hotleak] download files with 404 status code (#5395) 2024-04-19 16:08:31 +02:00
Mike Fährmann
a8027745e3 [downloader:http] add MIME type and signature for .mov files (#5287) 2024-03-06 14:00:24 +01:00
Mike Fährmann
0dacb2b24c [downloader:http] remove 'pyopenssl' import (#5156) 2024-02-04 21:06:54 +01:00
Mike Fährmann
ea78f67860 [downloader:http] skip files not passing filesize-min/-max (#4821) 
instead of failing the download
 2023-11-17 22:54:20 +01:00
ClosedPort22
5448268d5c [downloader:http] close connection when file already exists (#3748) 2023-08-08 23:35:43 +08:00
Mike Fährmann
c182094ebf merge #3748: [downloader:http] add 'consume-content' option 2023-04-26 23:03:18 +02:00
ClosedPort22
6f4a843fba [downloader:http] release connection before logging messages 
This allows connections to be properly released when using 'actions'
feature.
 2023-04-24 23:59:36 +08:00
Mike Fährmann
2edcdee32f [downloader:http] add MIME type and signature for .heic files 
(#3915)
https://github.com/strukturag/libheif/issues/83
 2023-04-15 17:09:22 +02:00
ClosedPort22
775d2ac999 [downloader:http] improve error logging when releasing connection 2023-03-31 20:08:38 +08:00
ClosedPort22
1a977f0f62 [downloader:http] handle exceptions in 'validate' 
This isn't strictly necessary for 'exhentai.py', but it improves
efficiency when the adapter is reused
 2023-03-23 19:57:13 +08:00
ClosedPort22
fcaeaf539c [downloader:http] handle exceptions while consuming content 2023-03-11 21:36:37 +08:00
Mike Fährmann
67ec91cdbd [downloader:http] change '_http_retry' to accept a Python function 
and rename '_http_retry_codes' to '_http_retry'

(#3569)
 2023-03-09 23:30:15 +01:00
ClosedPort22
df77271438 [downloader:http] add 'consume-content' option 
* fix connection not being released when the response is neither
  successful nor retried
* add the ability to consume the HTTP response body instead of closing
  the connection

reference:

https://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
 2023-03-09 21:07:10 +08:00
Mike Fährmann
8148c2a097 [downloader:ytdl] prevent exception on empty results 
a7c7953107 (commitcomment-92042240)
 2023-03-06 12:25:12 +01:00
Mike Fährmann
d16873941c [downloader:http] use 'time.monotonic()' 2023-01-31 15:32:12 +01:00
Mike Fährmann
ec9ff7640d merge #3535: [downloader:http] add signature checks for .blend, .obj, and .clip files 2023-01-16 15:09:10 +01:00
ClosedPort22
b6706b373a [downloader:http] add signature checks for some formats 
also add the MIME type for .obj files
 2023-01-15 23:40:55 +08:00
Mike Fährmann
c881548a27 add 'extractor.retry-codes' option (#3313) 
do not retry 429 and 430 by default
 2023-01-14 17:25:30 +01:00
Mike Fährmann
c0d7d2be35 [downloader:http] add 'validate' option 2023-01-11 15:37:40 +01:00
Mike Fährmann
80102fa367 [downloader:http] add 'retry-codes' option (#3313) 2022-12-01 11:08:23 +01:00
Mike Fährmann
b4253f69c9 [downloader:http] fix ZeroDivisionError (#3328) 
ensure 'time_elapsed' only get used as divisor
when it is greater than zero
 2022-11-30 21:56:18 +01:00
Mike Fährmann
f87cfa5f66 [downloader:http] add signature check for .mp4 files 2022-11-16 21:45:26 +01:00
Mike Fährmann
a4ff20cf16 [downloader:http] fix issues from inaccurate 'time.sleep()' 
(#3143)

Reverts part of c59b98c8 by going back to using a global timer
instead of a per-chunk one.

Reintroduces the issue of ignoring rate limits after
suspending and resuming the process.
 2022-11-10 13:24:02 +01:00
Mike Fährmann
550f90ab56 delay enabling .part files when 'http-metadata' is set 
otherwise 'build_path' gets called before all metadata is collected
 2022-11-09 13:23:52 +01:00
Mike Fährmann
8124c16a50 split 'build_path' from 'set_filename' and 'set_extension' 
Do not automatically build a new path
when setting file metadata or updating its extension.
 2022-11-08 17:03:24 +01:00
Mike Fährmann
39d9c362e4 include 'http-metadata' in '-K' output 2022-11-07 16:33:26 +01:00
Mike Fährmann
870e6a48a0 implement 'http-metadata' option 
or at least attempt to.
 2022-11-05 18:29:29 +01:00
Mike Fährmann
bca9f965e5 [downloader:http] add 'chunk-size' option (#3143) 
and double the previous default from 16384 (2**14) to 32768 (2**15)
 2022-11-02 16:50:26 +01:00
Mike Fährmann
0059e2bfe7 [downloader:http] add MIME type and signature for .avif files 2022-11-01 17:25:21 +01:00
Mike Fährmann
f687e64513 [downloader:http] refactor file signature checks 
use functions/lambdas instead of startswith()
 2022-11-01 17:09:13 +01:00
Mike Fährmann
1aae9f2b71 [downloader:ytdl] update _set_outtmpl() (fixes #2692) 
bf1824b391
 2022-06-20 11:32:02 +02:00
Mike Fährmann
c0c1277c5f [downloader:http] support sending POST data (#2433) 
by setting the '_http_data' metadata field for a file

needed in addition to be3492776b
to download files with POST requests
 2022-03-23 21:48:38 +01:00
Mike Fährmann
be3492776b [downloader:http] support using a different method than GET (#2433) 
by setting the '_http_method' metadata field for a file
 2022-03-20 10:09:05 +01:00
Mike Fährmann
47cf05c4ab refactor proxy handling code (#2357) 
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
  - add 'downloader.proxy' option
  - '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
    now has the same effect as youtube-dl's '--geo-verification-proxy'
 2022-03-10 23:55:35 +01:00
Mike Fährmann
c0fddcefc5 [downloader:ytdl] make ImportErrors non-fatal (#2273) 2022-02-08 19:30:29 +01:00
Mike Fährmann
ebd3d5c1cc [bunkr] fix .mp4 downloads (closes #2239) 2022-01-28 23:21:16 +01:00
Mike Fährmann
f4e3cee6ac use yt-dlp by default (#1850, #2028) 2021-11-29 18:24:26 +01:00
Mike Fährmann
19403a7fff [downloader:ytdl] prevent crash in '_progress_hook()' (#1680) 
'speed' is not guaranteed to be defined or convertible to 'int'
 2021-11-12 18:54:04 +01:00
Mike Fährmann
efa178cc91 [ytdl] implement parsing ytdl command-line options (#1680) 
- adds 'config-file' and 'cmdline-args' options
  for both ytdl downloader and extractor
- create 'ytdl' helper module, which combines YoutubeDL creation
  and option parsing.
- most likely a buggy mess due to incompatibilities between the
  original youtube-dl and yt-dlp.
 2021-11-07 02:44:11 +01:00