github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5,089 Commits 1 Branch 0 Tags
7dd79eee936b1b30312303251e5597febf298159
Commit Graph

204 Commits

Author SHA1 Message Date
Mike Fährmann
e256434c9e use custom HTTPBasicAuth class 
to support LazyPrompt as password
and to generate the Authorization header only once
instead of for every request
 2023-12-07 01:40:31 +01:00
Mike Fährmann
bdebe4597a fix util.dump_response to work with bytes as header values 2023-12-06 23:03:34 +01:00
Mike Fährmann
9dd5cb8c8a interactively prompt for passwords on login when none is provided 2023-12-06 22:12:59 +01:00
Mike Fährmann
4700051562 rework and extend input file processing (#4732) 
- add 2 command-line options to modify input file contents
  - -I/--input-file-comment
  - -x/--input-file-delete
- implement InputManager class
- move code from util.py to __init__.py
  (mainly to avoid import cycles)
 2023-11-14 20:50:11 +01:00
Mike Fährmann
72b18d701f represent util.NONE as 'null' in JSON output 
was '"None"' before
 2023-11-02 15:23:28 +01:00
Mike Fährmann
899df8f237 remove another '*' for keyword-only arguments 
076380e0
 2023-09-13 21:38:31 +02:00
Mike Fährmann
26d06e0bb2 move executable check into util.py 2023-02-28 23:10:23 +01:00
Mike Fährmann
d788e6c60c implement 'globals' option 2023-02-28 18:18:55 +01:00
Mike Fährmann
3d8777fbc1 move user agent string to util.py 2023-02-22 11:09:17 +01:00
Mike Fährmann
56039d2456 add 'hash_md5' and 'hash_sha1' functions (#3679) 
... to global eval namespace
 2023-02-22 10:58:44 +01:00
Mike Fährmann
d4232f3a8b implement restarting an extractor (#3338) 2023-02-11 21:06:14 +01:00
Mike Fährmann
1aae72773f put argument init on separate lines 2023-02-10 17:05:20 +01:00
Mike Fährmann
5503ac4d5e replace json.dumps with direct calls to JSONEncoder.encode 2023-02-09 15:51:40 +01:00
Mike Fährmann
dd884b02ee replace json.loads with direct calls to JSONDecoder.decode 2023-02-09 15:22:00 +01:00
Mike Fährmann
762a68996b implement 'archive-pragma' option 2023-02-05 17:00:31 +01:00
Mike Fährmann
d6793b2c7d include request body in 'write-pages=all' output 2023-01-14 22:33:59 +01:00
Mike Fährmann
d651d45239 implement specifying ranges in slice notation (#918, #2865) 
e.g.
- '1:101'   or ':101' or ':101:'  for files 1 to 100
- '1::2'    or '::2'              for every second file
- '1:101:5' or ':101:5'           for files 1, 6, 11, ..., 91, 96

(the second argument specifies the first index NOT included)
 2022-12-27 18:21:12 +01:00
Mike Fährmann
3616adfc75 implement '--range' with Python ranges 2022-12-26 18:32:34 +01:00
Mike Fährmann
1800bd7d14 allow '*-filter' options to be a list of expressions 2022-12-23 22:20:21 +01:00
Mike Fährmann
43c211f1a7 extend and rename util.CustomNone 2022-12-06 22:08:51 +01:00
Mike Fährmann
39d9c362e4 include 'http-metadata' in '-K' output 2022-11-07 16:33:26 +01:00
Mike Fährmann
597b63d922 move git head functionality to function in util.py 2022-11-04 19:49:39 +01:00
ClosedPort22
455e34113e Improve compatibility of DownloadArchive (#3078) 
Other programs can add additional columns to the table without affecting
gallery-dl
 2022-10-21 15:56:40 +02:00
Mike Fährmann
a6e2d96dde fix bug when processing input file comments (#2808) 
and move 'parse_inputfile()' to util.py
 2022-10-08 15:41:39 +02:00
Mike Fährmann
dd3a6a9fd1 make 'enumerate_reversed()' work with generators (#2795) 2022-08-01 14:08:44 +02:00
Mike Fährmann
a96f9d29b5 create missing directories for archive files (#2597) 2022-05-17 12:57:01 +02:00
Mike Fährmann
c4b9f7bab8 update functions working with cookies.txt files 
- rename
  - load_cookiestxt -> cookiestxt_load
  - save_cookiestxt -< cookiestxt_store
- in cookiestxt_load, add cookies directly to a cookie jar
  instead of storing them in a list first
- other unnoticeable performance increases
 2022-05-06 13:21:29 +02:00
Mike Fährmann
4c8e28db82 avoid circular import in util.py 2022-05-04 17:42:37 +02:00
Mike Fährmann
61afa973b7 use extended format strings for archive keys 2022-04-28 20:26:12 +02:00
Mike Fährmann
ca3a364db7 fix build_duration_func() (#2533) 
for extractors with request_interval_min > 0
 2022-04-27 20:28:14 +02:00
Mike Fährmann
7fe54bab2a attempt to fix some issues with 'contains()' (#2446) 
add a third argument that gets used
when the values o search are given as a string
 2022-04-08 14:40:26 +02:00
Mike Fährmann
d78a2c7163 re.escape() arguments for 'contains()' (#2446) 2022-04-07 15:35:54 +02:00
Mike Fährmann
413b77757b implement 'contains()' (#2446) 
and add it to globals() in compiled expressions for --filter etc
 2022-03-30 16:18:33 +02:00
Mike Fährmann
29db716a63 implement 'datetime_to_timestamp()' 
and rename 'to_timestamp()'
to the more descriptive 'datetime_to_timestamp_string()'
 2022-03-23 22:36:01 +01:00
Mike Fährmann
9bd27b1b8d [postprocessor:metadata] implement archive options (#2421) 
'archive', 'archive-format', and 'archive-prefix'
 2022-03-20 21:16:46 +01:00
Mike Fährmann
8295bc6d97 fix loading/storing cookies without domain 2022-03-19 15:14:55 +01:00
Mike Fährmann
47cf05c4ab refactor proxy handling code (#2357) 
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
  - add 'downloader.proxy' option
  - '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
    now has the same effect as youtube-dl's '--geo-verification-proxy'
 2022-03-10 23:55:35 +01:00
Mike Fährmann
00fbc942e5 add 'timedelta' class to the global namespace in filter expressions 
(https://github.com/mikf/gallery-dl/issues/2308#issuecomment-1046135924)
 2022-02-23 00:02:10 +01:00
Mike Fährmann
7958995398 [postprocessor:metadata] implement 'mtime' option (#2307) 2022-02-22 23:02:13 +01:00
Mike Fährmann
64cf26eaf4 allow specifying sleep-* options as string 
either as single value or as range: "3.5", "2.1 - 5.0"
 2021-12-18 23:28:56 +01:00
Mike Fährmann
010d65dcec extend blacklist/whitelist syntax (#2025) 
Each entry in such a list can now also include a subcategory
'<category>:<subcategory>'
and it is possible to use '*' or an empty string as placeholder
'*:<subcategory>', ':<subcategory>', '<category>:*'

For example
  "blacklist": "imgur,*:tag,gfycat:user" or
  "blacklist": ["imgur", "*:tag", "gfycat:user"]
will filter all 'imgur' extractors, all extractors  with a 'tag'
subcategory (e.g. https://danbooru.donmai.us/posts?tags=bonocho),
and all 'gfycat' user extractors.
 2021-11-23 20:31:43 +01:00
Mike Fährmann
c22ff97743 remove 'unit' argument from 'util.format_value()' 2021-09-28 23:07:55 +02:00
Mike Fährmann
cad85640de move 'util.PathFormat' into its own 'path' module 
to prevent circular imports between 'formatter' and 'util'
 2021-09-27 21:29:37 +02:00
Mike Fährmann
74145467dd move 'util.Formatter' into its own 'formatter' module 2021-09-27 02:37:04 +02:00
Mike Fährmann
e69ee41f25 implement 'page-reverse' option (#1854) 2021-09-23 18:02:19 +02:00
Mike Fährmann
c9e6693530 allow specifying a minimum/maximum for 'sleep-*' options (#1835) 
for example '"sleep-request": [5.0, 10.0]' to wait between 5 and 10
seconds between each HTTP request
 2021-09-14 17:40:05 +02:00
Mike Fährmann
292fffc83c add 'j' format string conversion 
to convert to a JSON formatted string
 2021-08-28 01:19:36 +02:00
Mike Fährmann
d3eab417ed implement a 'path-strip' option 2021-08-24 23:23:12 +02:00
Mike Fährmann
2792ed6e4b implement 'util.format_value()' 2021-07-26 02:11:22 +02:00
Mike Fährmann
13d4045a8a add 'archive-prefix' option (#1711) 2021-07-20 20:21:33 +02:00