github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6,563 Commits 1 Branch 0 Tags
eba44b8f78b612aaf476ea01d3985331f4b7cb21
Commit Graph

165 Commits

Author SHA1 Message Date
Mike Fährmann
1772c233e4 [dl:ytdl] fix exception when ytdl reports an error (#7301) 
when using the HLS/DASH download functionality
 2025-04-08 17:02:35 +02:00
Mike Fährmann
81a1b647bb [dl:ytdl] support processing inline HLS/DASH manifests (#6379 #7006) 2025-03-26 22:41:40 +01:00
Mike Fährmann
4a2c8606a5 [dl:http] detect Cloudflare/DDoS-Guard challenges (#7066 #7121) 2025-03-06 14:31:32 +01:00
Mike Fährmann
b5c1bf3f59 [dl] improve invalid 'subcategory' value warning (#7103) 2025-03-04 16:17:53 +01:00
Mike Fährmann
4d2037f6c6 [dl] warn about invalid 'subcategory' values (#7103) 
prevent fatal exception when collecting downloader options
 2025-03-03 16:51:13 +01:00
Mike Fährmann
6c8c264cf5 [dl:ytdl] log ytdl module and version when importing 2025-02-28 21:18:42 +01:00
Mike Fährmann
613f05afa3 fix cmdline arguments not overriding extractor-downloader options 2025-02-22 17:40:27 +01:00
Mike Fährmann
18ed39c1cf implement 'downloader' options per extractor category 
by setting options inside 'http' or 'ytdl' inside extractor options
or inside subcategory options

{
    "extractor": {
        "mastodon": {
            "http": {
                "rate": "10k"
            }
        },
        "mastodon.social": {
            "http": {
                "rate": "100k"
            }
        }
    },
    "downloader": {
        "rate": "100m"
    }
}

Sets download speed to
-  10k for mastodon.social URLs
- 100k for mastodon sites in general
- 100m for all other sites
 2025-02-22 10:08:59 +01:00
Mike Fährmann
800cf5beb5 replace 'print()' with 'output.stderr_write("\n")' 2025-02-15 18:01:05 +01:00
Mike Fährmann
35307608f2 [dl:http] add 'sleep-429' option (#6996) 2025-02-15 17:42:03 +01:00
Mike Fährmann
e8826ed3d4 [common] simplify HTTP error messages 
[warning] HTTPSConnectionPool(host='domain.tld', port=443): Max retries
exceeded with url: /a.jpg (Caused by NameResolutionError("<urllib3.
connection.HTTPSConnection object at 0x7247fe436ea0>: Failed to resolve
'domain.tld' ([Errno -2] Name or service not known)")) (1/5)

->

[warning] NameResolutionError: Failed to resolve 'domain.tld'
([Errno -2] Name or service not known) (1/5)
 2024-12-10 17:13:44 +01:00
Mike Fährmann
f79e57b71e [dl:ytdl] change 'forward-cookies' default to 'true' (#6401, #6348) 
revert dba87ca99e
 2024-10-31 17:35:08 +01:00
Mike Fährmann
1824267447 [dl:ytdl] implement explicit HLS/DASH handling 
add '_ytdl_manifest' to specify a manifest type to process
 2024-10-16 15:16:21 +02:00
Mike Fährmann
a051e1c955 directly pass exception instances as 'exc_info' logger argument 2024-09-19 14:50:08 +02:00
Mike Fährmann
127aa45834 [ytdl] ignore SyntaxErrors when trying to import a module 2024-08-29 19:28:08 +02:00
Mike Fährmann
9c65db2a92 consistent 'with open(…) as fp:' syntax 2024-06-14 01:22:00 +02:00
Mike Fährmann
3ec7ee6baa [downloader:ytdl] fix exception due to missing 'ext' (#5675) 2024-06-02 18:16:53 +02:00
Mike Fährmann
cd241bea0a [downloader:http] add MIME type and signature for .m4v files (#5505) 2024-04-25 01:01:35 +02:00
Mike Fährmann
bffadf35b7 [hotleak] download files with 404 status code (#5395) 2024-04-19 16:08:31 +02:00
Mike Fährmann
a8027745e3 [downloader:http] add MIME type and signature for .mov files (#5287) 2024-03-06 14:00:24 +01:00
Mike Fährmann
0dacb2b24c [downloader:http] remove 'pyopenssl' import (#5156) 2024-02-04 21:06:54 +01:00
Mike Fährmann
ea78f67860 [downloader:http] skip files not passing filesize-min/-max (#4821) 
instead of failing the download
 2023-11-17 22:54:20 +01:00
ClosedPort22
5448268d5c [downloader:http] close connection when file already exists (#3748) 2023-08-08 23:35:43 +08:00
Mike Fährmann
c182094ebf merge #3748: [downloader:http] add 'consume-content' option 2023-04-26 23:03:18 +02:00
ClosedPort22
6f4a843fba [downloader:http] release connection before logging messages 
This allows connections to be properly released when using 'actions'
feature.
 2023-04-24 23:59:36 +08:00
Mike Fährmann
2edcdee32f [downloader:http] add MIME type and signature for .heic files 
(#3915)
https://github.com/strukturag/libheif/issues/83
 2023-04-15 17:09:22 +02:00
ClosedPort22
775d2ac999 [downloader:http] improve error logging when releasing connection 2023-03-31 20:08:38 +08:00
ClosedPort22
1a977f0f62 [downloader:http] handle exceptions in 'validate' 
This isn't strictly necessary for 'exhentai.py', but it improves
efficiency when the adapter is reused
 2023-03-23 19:57:13 +08:00
ClosedPort22
fcaeaf539c [downloader:http] handle exceptions while consuming content 2023-03-11 21:36:37 +08:00
Mike Fährmann
67ec91cdbd [downloader:http] change '_http_retry' to accept a Python function 
and rename '_http_retry_codes' to '_http_retry'

(#3569)
 2023-03-09 23:30:15 +01:00
ClosedPort22
df77271438 [downloader:http] add 'consume-content' option 
* fix connection not being released when the response is neither
  successful nor retried
* add the ability to consume the HTTP response body instead of closing
  the connection

reference:

https://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
 2023-03-09 21:07:10 +08:00
Mike Fährmann
8148c2a097 [downloader:ytdl] prevent exception on empty results 
a7c7953107 (commitcomment-92042240)
 2023-03-06 12:25:12 +01:00
Mike Fährmann
d16873941c [downloader:http] use 'time.monotonic()' 2023-01-31 15:32:12 +01:00
Mike Fährmann
ec9ff7640d merge #3535: [downloader:http] add signature checks for .blend, .obj, and .clip files 2023-01-16 15:09:10 +01:00
ClosedPort22
b6706b373a [downloader:http] add signature checks for some formats 
also add the MIME type for .obj files
 2023-01-15 23:40:55 +08:00
Mike Fährmann
c881548a27 add 'extractor.retry-codes' option (#3313) 
do not retry 429 and 430 by default
 2023-01-14 17:25:30 +01:00
Mike Fährmann
c0d7d2be35 [downloader:http] add 'validate' option 2023-01-11 15:37:40 +01:00
Mike Fährmann
80102fa367 [downloader:http] add 'retry-codes' option (#3313) 2022-12-01 11:08:23 +01:00
Mike Fährmann
b4253f69c9 [downloader:http] fix ZeroDivisionError (#3328) 
ensure 'time_elapsed' only get used as divisor
when it is greater than zero
 2022-11-30 21:56:18 +01:00
Mike Fährmann
f87cfa5f66 [downloader:http] add signature check for .mp4 files 2022-11-16 21:45:26 +01:00
Mike Fährmann
a4ff20cf16 [downloader:http] fix issues from inaccurate 'time.sleep()' 
(#3143)

Reverts part of c59b98c8 by going back to using a global timer
instead of a per-chunk one.

Reintroduces the issue of ignoring rate limits after
suspending and resuming the process.
 2022-11-10 13:24:02 +01:00
Mike Fährmann
550f90ab56 delay enabling .part files when 'http-metadata' is set 
otherwise 'build_path' gets called before all metadata is collected
 2022-11-09 13:23:52 +01:00
Mike Fährmann
8124c16a50 split 'build_path' from 'set_filename' and 'set_extension' 
Do not automatically build a new path
when setting file metadata or updating its extension.
 2022-11-08 17:03:24 +01:00
Mike Fährmann
39d9c362e4 include 'http-metadata' in '-K' output 2022-11-07 16:33:26 +01:00
Mike Fährmann
870e6a48a0 implement 'http-metadata' option 
or at least attempt to.
 2022-11-05 18:29:29 +01:00
Mike Fährmann
bca9f965e5 [downloader:http] add 'chunk-size' option (#3143) 
and double the previous default from 16384 (2**14) to 32768 (2**15)
 2022-11-02 16:50:26 +01:00
Mike Fährmann
0059e2bfe7 [downloader:http] add MIME type and signature for .avif files 2022-11-01 17:25:21 +01:00
Mike Fährmann
f687e64513 [downloader:http] refactor file signature checks 
use functions/lambdas instead of startswith()
 2022-11-01 17:09:13 +01:00
Mike Fährmann
1aae9f2b71 [downloader:ytdl] update _set_outtmpl() (fixes #2692) 
bf1824b391
 2022-06-20 11:32:02 +02:00
Mike Fährmann
c0c1277c5f [downloader:http] support sending POST data (#2433) 
by setting the '_http_data' metadata field for a file

needed in addition to be3492776b
to download files with POST requests
 2022-03-23 21:48:38 +01:00