github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6,757 Commits 1 Branch 0 Tags
41191bb60aa73df4e95cf31ede90fba76bc2796f
Commit Graph

116 Commits

Author SHA1 Message Date
Mike Fährmann
e08ec7e083 update copyright notices 2025-06-13 00:03:41 +02:00
Mike Fährmann
811b665e33 remove @staticmethod decorators 
There might have been a time when calling a static method was faster
than a regular method, but that is no longer the case. According to
micro-benchmarks, it is 70% slower in CPython 3.13 and it also makes
executing the code of a class definition slower.
 2025-06-12 22:50:52 +02:00
Mike Fährmann
8b6bc54e95 [dl:http] add MIME type and signature for .html files 2025-06-12 21:16:34 +02:00
Mike Fährmann
e0a961047c [dl:http] fix potential FileExistsError when '.part' file moved (#5385) 
when the downloader detects a '.part' file and attempts to resume its
download, a FileExistsError was raised while trying to create an assumed
missing directory path when said '.part' file was (re)moved during the
time it was detected and the downloader attempted to open it in 'a+b' mode.
 2025-06-12 16:44:41 +02:00
Mike Fährmann
7a48b696a6 [dl] improve maximum 'rate' detection (#7638) 2025-06-09 12:15:39 +02:00
Mike Fährmann
a25e14e776 [dl:http] implement dynamic download 'rate' limits (#7638) 2025-06-08 20:04:31 +02:00
Mike Fährmann
483e30d846 [dl:http] allow '_http_signature' to return a custom error message 
… and fail the download immediately like '_http_validate'
 2025-06-07 22:06:00 +02:00
Mike Fährmann
839a43fe5c [dl:http] implement '_http_signature' (#4902) 
allows passing a function to validate a file's signature bytes,
similar to '_http_validate'
 2025-06-07 20:27:59 +02:00
Mike Fährmann
f51511155e [dl:http] include traceback in exception debug log 2025-06-07 20:24:08 +02:00
Mike Fährmann
b76e7de1a7 [dl:http] fix setting 'mtime' per file (#7529) 
introduce '_http_lastmodified' meta field
 2025-05-21 13:50:51 +02:00
Mike Fährmann
4a2c8606a5 [dl:http] detect Cloudflare/DDoS-Guard challenges (#7066 #7121) 2025-03-06 14:31:32 +01:00
Mike Fährmann
800cf5beb5 replace 'print()' with 'output.stderr_write("\n")' 2025-02-15 18:01:05 +01:00
Mike Fährmann
35307608f2 [dl:http] add 'sleep-429' option (#6996) 2025-02-15 17:42:03 +01:00
Mike Fährmann
e8826ed3d4 [common] simplify HTTP error messages 
[warning] HTTPSConnectionPool(host='domain.tld', port=443): Max retries
exceeded with url: /a.jpg (Caused by NameResolutionError("<urllib3.
connection.HTTPSConnection object at 0x7247fe436ea0>: Failed to resolve
'domain.tld' ([Errno -2] Name or service not known)")) (1/5)

->

[warning] NameResolutionError: Failed to resolve 'domain.tld'
([Errno -2] Name or service not known) (1/5)
 2024-12-10 17:13:44 +01:00
Mike Fährmann
cd241bea0a [downloader:http] add MIME type and signature for .m4v files (#5505) 2024-04-25 01:01:35 +02:00
Mike Fährmann
bffadf35b7 [hotleak] download files with 404 status code (#5395) 2024-04-19 16:08:31 +02:00
Mike Fährmann
a8027745e3 [downloader:http] add MIME type and signature for .mov files (#5287) 2024-03-06 14:00:24 +01:00
Mike Fährmann
0dacb2b24c [downloader:http] remove 'pyopenssl' import (#5156) 2024-02-04 21:06:54 +01:00
Mike Fährmann
ea78f67860 [downloader:http] skip files not passing filesize-min/-max (#4821) 
instead of failing the download
 2023-11-17 22:54:20 +01:00
ClosedPort22
5448268d5c [downloader:http] close connection when file already exists (#3748) 2023-08-08 23:35:43 +08:00
Mike Fährmann
c182094ebf merge #3748: [downloader:http] add 'consume-content' option 2023-04-26 23:03:18 +02:00
ClosedPort22
6f4a843fba [downloader:http] release connection before logging messages 
This allows connections to be properly released when using 'actions'
feature.
 2023-04-24 23:59:36 +08:00
Mike Fährmann
2edcdee32f [downloader:http] add MIME type and signature for .heic files 
(#3915)
https://github.com/strukturag/libheif/issues/83
 2023-04-15 17:09:22 +02:00
ClosedPort22
775d2ac999 [downloader:http] improve error logging when releasing connection 2023-03-31 20:08:38 +08:00
ClosedPort22
1a977f0f62 [downloader:http] handle exceptions in 'validate' 
This isn't strictly necessary for 'exhentai.py', but it improves
efficiency when the adapter is reused
 2023-03-23 19:57:13 +08:00
ClosedPort22
fcaeaf539c [downloader:http] handle exceptions while consuming content 2023-03-11 21:36:37 +08:00
Mike Fährmann
67ec91cdbd [downloader:http] change '_http_retry' to accept a Python function 
and rename '_http_retry_codes' to '_http_retry'

(#3569)
 2023-03-09 23:30:15 +01:00
ClosedPort22
df77271438 [downloader:http] add 'consume-content' option 
* fix connection not being released when the response is neither
  successful nor retried
* add the ability to consume the HTTP response body instead of closing
  the connection

reference:

https://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
 2023-03-09 21:07:10 +08:00
Mike Fährmann
d16873941c [downloader:http] use 'time.monotonic()' 2023-01-31 15:32:12 +01:00
Mike Fährmann
ec9ff7640d merge #3535: [downloader:http] add signature checks for .blend, .obj, and .clip files 2023-01-16 15:09:10 +01:00
ClosedPort22
b6706b373a [downloader:http] add signature checks for some formats 
also add the MIME type for .obj files
 2023-01-15 23:40:55 +08:00
Mike Fährmann
c881548a27 add 'extractor.retry-codes' option (#3313) 
do not retry 429 and 430 by default
 2023-01-14 17:25:30 +01:00
Mike Fährmann
c0d7d2be35 [downloader:http] add 'validate' option 2023-01-11 15:37:40 +01:00
Mike Fährmann
80102fa367 [downloader:http] add 'retry-codes' option (#3313) 2022-12-01 11:08:23 +01:00
Mike Fährmann
b4253f69c9 [downloader:http] fix ZeroDivisionError (#3328) 
ensure 'time_elapsed' only get used as divisor
when it is greater than zero
 2022-11-30 21:56:18 +01:00
Mike Fährmann
f87cfa5f66 [downloader:http] add signature check for .mp4 files 2022-11-16 21:45:26 +01:00
Mike Fährmann
a4ff20cf16 [downloader:http] fix issues from inaccurate 'time.sleep()' 
(#3143)

Reverts part of c59b98c8 by going back to using a global timer
instead of a per-chunk one.

Reintroduces the issue of ignoring rate limits after
suspending and resuming the process.
 2022-11-10 13:24:02 +01:00
Mike Fährmann
550f90ab56 delay enabling .part files when 'http-metadata' is set 
otherwise 'build_path' gets called before all metadata is collected
 2022-11-09 13:23:52 +01:00
Mike Fährmann
8124c16a50 split 'build_path' from 'set_filename' and 'set_extension' 
Do not automatically build a new path
when setting file metadata or updating its extension.
 2022-11-08 17:03:24 +01:00
Mike Fährmann
39d9c362e4 include 'http-metadata' in '-K' output 2022-11-07 16:33:26 +01:00
Mike Fährmann
870e6a48a0 implement 'http-metadata' option 
or at least attempt to.
 2022-11-05 18:29:29 +01:00
Mike Fährmann
bca9f965e5 [downloader:http] add 'chunk-size' option (#3143) 
and double the previous default from 16384 (2**14) to 32768 (2**15)
 2022-11-02 16:50:26 +01:00
Mike Fährmann
0059e2bfe7 [downloader:http] add MIME type and signature for .avif files 2022-11-01 17:25:21 +01:00
Mike Fährmann
f687e64513 [downloader:http] refactor file signature checks 
use functions/lambdas instead of startswith()
 2022-11-01 17:09:13 +01:00
Mike Fährmann
c0c1277c5f [downloader:http] support sending POST data (#2433) 
by setting the '_http_data' metadata field for a file

needed in addition to be3492776b
to download files with POST requests
 2022-03-23 21:48:38 +01:00
Mike Fährmann
be3492776b [downloader:http] support using a different method than GET (#2433) 
by setting the '_http_method' metadata field for a file
 2022-03-20 10:09:05 +01:00
Mike Fährmann
47cf05c4ab refactor proxy handling code (#2357) 
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
  - add 'downloader.proxy' option
  - '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
    now has the same effect as youtube-dl's '--geo-verification-proxy'
 2022-03-10 23:55:35 +01:00
Mike Fährmann
ebd3d5c1cc [bunkr] fix .mp4 downloads (closes #2239) 2022-01-28 23:21:16 +01:00
Mike Fährmann
d0761454b1 implement a download progress indicator (#1519) 2021-09-28 22:48:58 +02:00
Mike Fährmann
b5b1cf22b7 [downloader:http] reorder HTTP header sources 
so that any header can be overwritten by a user, except Range
 2021-08-05 23:01:54 +02:00