github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6,757 Commits 1 Branch 0 Tags
41191bb60aa73df4e95cf31ede90fba76bc2796f
Commit Graph

179 Commits

Author SHA1 Message Date
Mike Fährmann
e08ec7e083 update copyright notices 2025-06-13 00:03:41 +02:00
Mike Fährmann
811b665e33 remove @staticmethod decorators 
There might have been a time when calling a static method was faster
than a regular method, but that is no longer the case. According to
micro-benchmarks, it is 70% slower in CPython 3.13 and it also makes
executing the code of a class definition slower.
 2025-06-12 22:50:52 +02:00
Mike Fährmann
8b6bc54e95 [dl:http] add MIME type and signature for .html files 2025-06-12 21:16:34 +02:00
Mike Fährmann
e0a961047c [dl:http] fix potential FileExistsError when '.part' file moved (#5385) 
when the downloader detects a '.part' file and attempts to resume its
download, a FileExistsError was raised while trying to create an assumed
missing directory path when said '.part' file was (re)moved during the
time it was detected and the downloader attempted to open it in 'a+b' mode.
 2025-06-12 16:44:41 +02:00
Mike Fährmann
7a48b696a6 [dl] improve maximum 'rate' detection (#7638) 2025-06-09 12:15:39 +02:00
Mike Fährmann
a25e14e776 [dl:http] implement dynamic download 'rate' limits (#7638) 2025-06-08 20:04:31 +02:00
Mike Fährmann
483e30d846 [dl:http] allow '_http_signature' to return a custom error message 
… and fail the download immediately like '_http_validate'
 2025-06-07 22:06:00 +02:00
Mike Fährmann
839a43fe5c [dl:http] implement '_http_signature' (#4902) 
allows passing a function to validate a file's signature bytes,
similar to '_http_validate'
 2025-06-07 20:27:59 +02:00
Mike Fährmann
f51511155e [dl:http] include traceback in exception debug log 2025-06-07 20:24:08 +02:00
Mike Fährmann
a798b8971e [dl:ytdl] fix postprocessing/merge errors (#7581) 
don't use a hardcoded '.part' filename extension

fixes regression introduced in 29d315e495
 2025-05-26 18:56:40 +02:00
Mike Fährmann
e05b82253a [dl:ytdl] support custom headers when fetching HLS/DASH manifests 2025-05-26 17:17:37 +02:00
Mike Fährmann
3e7a5fcb00 [dl:ytdl] fix crash when using legacy 'youtube_dl' (#6949) 
https://github.com/mikf/gallery-dl/issues/6949#issuecomment-2906920039

fixes regression introduced in 29d315e495
 2025-05-24 19:58:09 +02:00
Mike Fährmann
b76e7de1a7 [dl:http] fix setting 'mtime' per file (#7529) 
introduce '_http_lastmodified' meta field
 2025-05-21 13:50:51 +02:00
Mike Fährmann
29d315e495 [dl:ytdl] improve temp/part file handling (#6949 #7494) 
- fixes files not being moved from 'part-directory' (#6949)
- fixes 'skip: enumerate' overwriting files (#7494)
 2025-05-21 13:50:07 +02:00
Mike Fährmann
1772c233e4 [dl:ytdl] fix exception when ytdl reports an error (#7301) 
when using the HLS/DASH download functionality
 2025-04-08 17:02:35 +02:00
Mike Fährmann
81a1b647bb [dl:ytdl] support processing inline HLS/DASH manifests (#6379 #7006) 2025-03-26 22:41:40 +01:00
Mike Fährmann
4a2c8606a5 [dl:http] detect Cloudflare/DDoS-Guard challenges (#7066 #7121) 2025-03-06 14:31:32 +01:00
Mike Fährmann
b5c1bf3f59 [dl] improve invalid 'subcategory' value warning (#7103) 2025-03-04 16:17:53 +01:00
Mike Fährmann
4d2037f6c6 [dl] warn about invalid 'subcategory' values (#7103) 
prevent fatal exception when collecting downloader options
 2025-03-03 16:51:13 +01:00
Mike Fährmann
6c8c264cf5 [dl:ytdl] log ytdl module and version when importing 2025-02-28 21:18:42 +01:00
Mike Fährmann
613f05afa3 fix cmdline arguments not overriding extractor-downloader options 2025-02-22 17:40:27 +01:00
Mike Fährmann
18ed39c1cf implement 'downloader' options per extractor category 
by setting options inside 'http' or 'ytdl' inside extractor options
or inside subcategory options

{
    "extractor": {
        "mastodon": {
            "http": {
                "rate": "10k"
            }
        },
        "mastodon.social": {
            "http": {
                "rate": "100k"
            }
        }
    },
    "downloader": {
        "rate": "100m"
    }
}

Sets download speed to
-  10k for mastodon.social URLs
- 100k for mastodon sites in general
- 100m for all other sites
 2025-02-22 10:08:59 +01:00
Mike Fährmann
800cf5beb5 replace 'print()' with 'output.stderr_write("\n")' 2025-02-15 18:01:05 +01:00
Mike Fährmann
35307608f2 [dl:http] add 'sleep-429' option (#6996) 2025-02-15 17:42:03 +01:00
Mike Fährmann
e8826ed3d4 [common] simplify HTTP error messages 
[warning] HTTPSConnectionPool(host='domain.tld', port=443): Max retries
exceeded with url: /a.jpg (Caused by NameResolutionError("<urllib3.
connection.HTTPSConnection object at 0x7247fe436ea0>: Failed to resolve
'domain.tld' ([Errno -2] Name or service not known)")) (1/5)

->

[warning] NameResolutionError: Failed to resolve 'domain.tld'
([Errno -2] Name or service not known) (1/5)
 2024-12-10 17:13:44 +01:00
Mike Fährmann
f79e57b71e [dl:ytdl] change 'forward-cookies' default to 'true' (#6401, #6348) 
revert dba87ca99e
 2024-10-31 17:35:08 +01:00
Mike Fährmann
1824267447 [dl:ytdl] implement explicit HLS/DASH handling 
add '_ytdl_manifest' to specify a manifest type to process
 2024-10-16 15:16:21 +02:00
Mike Fährmann
a051e1c955 directly pass exception instances as 'exc_info' logger argument 2024-09-19 14:50:08 +02:00
Mike Fährmann
127aa45834 [ytdl] ignore SyntaxErrors when trying to import a module 2024-08-29 19:28:08 +02:00
Mike Fährmann
9c65db2a92 consistent 'with open(…) as fp:' syntax 2024-06-14 01:22:00 +02:00
Mike Fährmann
3ec7ee6baa [downloader:ytdl] fix exception due to missing 'ext' (#5675) 2024-06-02 18:16:53 +02:00
Mike Fährmann
cd241bea0a [downloader:http] add MIME type and signature for .m4v files (#5505) 2024-04-25 01:01:35 +02:00
Mike Fährmann
bffadf35b7 [hotleak] download files with 404 status code (#5395) 2024-04-19 16:08:31 +02:00
Mike Fährmann
a8027745e3 [downloader:http] add MIME type and signature for .mov files (#5287) 2024-03-06 14:00:24 +01:00
Mike Fährmann
0dacb2b24c [downloader:http] remove 'pyopenssl' import (#5156) 2024-02-04 21:06:54 +01:00
Mike Fährmann
ea78f67860 [downloader:http] skip files not passing filesize-min/-max (#4821) 
instead of failing the download
 2023-11-17 22:54:20 +01:00
ClosedPort22
5448268d5c [downloader:http] close connection when file already exists (#3748) 2023-08-08 23:35:43 +08:00
Mike Fährmann
c182094ebf merge #3748: [downloader:http] add 'consume-content' option 2023-04-26 23:03:18 +02:00
ClosedPort22
6f4a843fba [downloader:http] release connection before logging messages 
This allows connections to be properly released when using 'actions'
feature.
 2023-04-24 23:59:36 +08:00
Mike Fährmann
2edcdee32f [downloader:http] add MIME type and signature for .heic files 
(#3915)
https://github.com/strukturag/libheif/issues/83
 2023-04-15 17:09:22 +02:00
ClosedPort22
775d2ac999 [downloader:http] improve error logging when releasing connection 2023-03-31 20:08:38 +08:00
ClosedPort22
1a977f0f62 [downloader:http] handle exceptions in 'validate' 
This isn't strictly necessary for 'exhentai.py', but it improves
efficiency when the adapter is reused
 2023-03-23 19:57:13 +08:00
ClosedPort22
fcaeaf539c [downloader:http] handle exceptions while consuming content 2023-03-11 21:36:37 +08:00
Mike Fährmann
67ec91cdbd [downloader:http] change '_http_retry' to accept a Python function 
and rename '_http_retry_codes' to '_http_retry'

(#3569)
 2023-03-09 23:30:15 +01:00
ClosedPort22
df77271438 [downloader:http] add 'consume-content' option 
* fix connection not being released when the response is neither
  successful nor retried
* add the ability to consume the HTTP response body instead of closing
  the connection

reference:

https://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
 2023-03-09 21:07:10 +08:00
Mike Fährmann
8148c2a097 [downloader:ytdl] prevent exception on empty results 
a7c7953107 (commitcomment-92042240)
 2023-03-06 12:25:12 +01:00
Mike Fährmann
d16873941c [downloader:http] use 'time.monotonic()' 2023-01-31 15:32:12 +01:00
Mike Fährmann
ec9ff7640d merge #3535: [downloader:http] add signature checks for .blend, .obj, and .clip files 2023-01-16 15:09:10 +01:00
ClosedPort22
b6706b373a [downloader:http] add signature checks for some formats 
also add the MIME type for .obj files
 2023-01-15 23:40:55 +08:00
Mike Fährmann
c881548a27 add 'extractor.retry-codes' option (#3313) 
do not retry 429 and 430 by default
 2023-01-14 17:25:30 +01:00