github/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5,148 Commits 1 Branch 0 Tags
63f649cd92a1aa2e70df8a2edbdb180ccae49f21
Commit Graph

205 Commits

Author SHA1 Message Date
Mike Fährmann
a24b82e67d add 'util.repeat()' 2023-12-11 23:32:28 +01:00
Mike Fährmann
e256434c9e use custom HTTPBasicAuth class 
to support LazyPrompt as password
and to generate the Authorization header only once
instead of for every request
 2023-12-07 01:40:31 +01:00
Mike Fährmann
bdebe4597a fix util.dump_response to work with bytes as header values 2023-12-06 23:03:34 +01:00
Mike Fährmann
9dd5cb8c8a interactively prompt for passwords on login when none is provided 2023-12-06 22:12:59 +01:00
Mike Fährmann
4700051562 rework and extend input file processing (#4732) 
- add 2 command-line options to modify input file contents
  - -I/--input-file-comment
  - -x/--input-file-delete
- implement InputManager class
- move code from util.py to __init__.py
  (mainly to avoid import cycles)
 2023-11-14 20:50:11 +01:00
Mike Fährmann
72b18d701f represent util.NONE as 'null' in JSON output 
was '"None"' before
 2023-11-02 15:23:28 +01:00
Mike Fährmann
899df8f237 remove another '*' for keyword-only arguments 
076380e0
 2023-09-13 21:38:31 +02:00
Mike Fährmann
26d06e0bb2 move executable check into util.py 2023-02-28 23:10:23 +01:00
Mike Fährmann
d788e6c60c implement 'globals' option 2023-02-28 18:18:55 +01:00
Mike Fährmann
3d8777fbc1 move user agent string to util.py 2023-02-22 11:09:17 +01:00
Mike Fährmann
56039d2456 add 'hash_md5' and 'hash_sha1' functions (#3679) 
... to global eval namespace
 2023-02-22 10:58:44 +01:00
Mike Fährmann
d4232f3a8b implement restarting an extractor (#3338) 2023-02-11 21:06:14 +01:00
Mike Fährmann
1aae72773f put argument init on separate lines 2023-02-10 17:05:20 +01:00
Mike Fährmann
5503ac4d5e replace json.dumps with direct calls to JSONEncoder.encode 2023-02-09 15:51:40 +01:00
Mike Fährmann
dd884b02ee replace json.loads with direct calls to JSONDecoder.decode 2023-02-09 15:22:00 +01:00
Mike Fährmann
762a68996b implement 'archive-pragma' option 2023-02-05 17:00:31 +01:00
Mike Fährmann
d6793b2c7d include request body in 'write-pages=all' output 2023-01-14 22:33:59 +01:00
Mike Fährmann
d651d45239 implement specifying ranges in slice notation (#918, #2865) 
e.g.
- '1:101'   or ':101' or ':101:'  for files 1 to 100
- '1::2'    or '::2'              for every second file
- '1:101:5' or ':101:5'           for files 1, 6, 11, ..., 91, 96

(the second argument specifies the first index NOT included)
 2022-12-27 18:21:12 +01:00
Mike Fährmann
3616adfc75 implement '--range' with Python ranges 2022-12-26 18:32:34 +01:00
Mike Fährmann
1800bd7d14 allow '*-filter' options to be a list of expressions 2022-12-23 22:20:21 +01:00
Mike Fährmann
43c211f1a7 extend and rename util.CustomNone 2022-12-06 22:08:51 +01:00
Mike Fährmann
39d9c362e4 include 'http-metadata' in '-K' output 2022-11-07 16:33:26 +01:00
Mike Fährmann
597b63d922 move git head functionality to function in util.py 2022-11-04 19:49:39 +01:00
ClosedPort22
455e34113e Improve compatibility of DownloadArchive (#3078) 
Other programs can add additional columns to the table without affecting
gallery-dl
 2022-10-21 15:56:40 +02:00
Mike Fährmann
a6e2d96dde fix bug when processing input file comments (#2808) 
and move 'parse_inputfile()' to util.py
 2022-10-08 15:41:39 +02:00
Mike Fährmann
dd3a6a9fd1 make 'enumerate_reversed()' work with generators (#2795) 2022-08-01 14:08:44 +02:00
Mike Fährmann
a96f9d29b5 create missing directories for archive files (#2597) 2022-05-17 12:57:01 +02:00
Mike Fährmann
c4b9f7bab8 update functions working with cookies.txt files 
- rename
  - load_cookiestxt -> cookiestxt_load
  - save_cookiestxt -< cookiestxt_store
- in cookiestxt_load, add cookies directly to a cookie jar
  instead of storing them in a list first
- other unnoticeable performance increases
 2022-05-06 13:21:29 +02:00
Mike Fährmann
4c8e28db82 avoid circular import in util.py 2022-05-04 17:42:37 +02:00
Mike Fährmann
61afa973b7 use extended format strings for archive keys 2022-04-28 20:26:12 +02:00
Mike Fährmann
ca3a364db7 fix build_duration_func() (#2533) 
for extractors with request_interval_min > 0
 2022-04-27 20:28:14 +02:00
Mike Fährmann
7fe54bab2a attempt to fix some issues with 'contains()' (#2446) 
add a third argument that gets used
when the values o search are given as a string
 2022-04-08 14:40:26 +02:00
Mike Fährmann
d78a2c7163 re.escape() arguments for 'contains()' (#2446) 2022-04-07 15:35:54 +02:00
Mike Fährmann
413b77757b implement 'contains()' (#2446) 
and add it to globals() in compiled expressions for --filter etc
 2022-03-30 16:18:33 +02:00
Mike Fährmann
29db716a63 implement 'datetime_to_timestamp()' 
and rename 'to_timestamp()'
to the more descriptive 'datetime_to_timestamp_string()'
 2022-03-23 22:36:01 +01:00
Mike Fährmann
9bd27b1b8d [postprocessor:metadata] implement archive options (#2421) 
'archive', 'archive-format', and 'archive-prefix'
 2022-03-20 21:16:46 +01:00
Mike Fährmann
8295bc6d97 fix loading/storing cookies without domain 2022-03-19 15:14:55 +01:00
Mike Fährmann
47cf05c4ab refactor proxy handling code (#2357) 
- allow gallery-dl proxy settings to overwrite environment proxies
- allow specifying different proxies for data extraction and download
  - add 'downloader.proxy' option
  - '-o extractor.proxy=–PROXY_URL -o downloader.proxy=null'
    now has the same effect as youtube-dl's '--geo-verification-proxy'
 2022-03-10 23:55:35 +01:00
Mike Fährmann
00fbc942e5 add 'timedelta' class to the global namespace in filter expressions 
(https://github.com/mikf/gallery-dl/issues/2308#issuecomment-1046135924)
 2022-02-23 00:02:10 +01:00
Mike Fährmann
7958995398 [postprocessor:metadata] implement 'mtime' option (#2307) 2022-02-22 23:02:13 +01:00
Mike Fährmann
64cf26eaf4 allow specifying sleep-* options as string 
either as single value or as range: "3.5", "2.1 - 5.0"
 2021-12-18 23:28:56 +01:00
Mike Fährmann
010d65dcec extend blacklist/whitelist syntax (#2025) 
Each entry in such a list can now also include a subcategory
'<category>:<subcategory>'
and it is possible to use '*' or an empty string as placeholder
'*:<subcategory>', ':<subcategory>', '<category>:*'

For example
  "blacklist": "imgur,*:tag,gfycat:user" or
  "blacklist": ["imgur", "*:tag", "gfycat:user"]
will filter all 'imgur' extractors, all extractors  with a 'tag'
subcategory (e.g. https://danbooru.donmai.us/posts?tags=bonocho),
and all 'gfycat' user extractors.
 2021-11-23 20:31:43 +01:00
Mike Fährmann
c22ff97743 remove 'unit' argument from 'util.format_value()' 2021-09-28 23:07:55 +02:00
Mike Fährmann
cad85640de move 'util.PathFormat' into its own 'path' module 
to prevent circular imports between 'formatter' and 'util'
 2021-09-27 21:29:37 +02:00
Mike Fährmann
74145467dd move 'util.Formatter' into its own 'formatter' module 2021-09-27 02:37:04 +02:00
Mike Fährmann
e69ee41f25 implement 'page-reverse' option (#1854) 2021-09-23 18:02:19 +02:00
Mike Fährmann
c9e6693530 allow specifying a minimum/maximum for 'sleep-*' options (#1835) 
for example '"sleep-request": [5.0, 10.0]' to wait between 5 and 10
seconds between each HTTP request
 2021-09-14 17:40:05 +02:00
Mike Fährmann
292fffc83c add 'j' format string conversion 
to convert to a JSON formatted string
 2021-08-28 01:19:36 +02:00
Mike Fährmann
d3eab417ed implement a 'path-strip' option 2021-08-24 23:23:12 +02:00
Mike Fährmann
2792ed6e4b implement 'util.format_value()' 2021-07-26 02:11:22 +02:00