implement linear/exponential backoff for 'sleep-429'

docs/configuration.rst

@@ -540,13 +540,22 @@ Description
 extractor.*.sleep-429
 ---------------------
 Type
-    |Duration|_
+    |Duration+|_
 Default
     ``60``
+Example
+    * ``"30-50"``
+    * ``"exp=40"``
+    * ``"lin:20=30-60"``
 Description
-    Number of seconds to sleep when receiving a `429 Too Many Requests`
+    Number of seconds to sleep when receiving a
+    `429 Too Many Requests <https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429>`__
     response before `retrying <extractor.*.retries_>`__ the request.
 
+    If this is a ``string``, its |Duration|_ value can be prefixed with
+    ``lin[:START[:MAX]]`` or ``exp[:BASE[:START[:MAX]]]``
+    for `linear` or `exponential` backoff respectively.
+
 
 extractor.*.sleep-request
 -------------------------
@@ -9965,6 +9974,24 @@ Description
       value (``"2.85"``) or a range  (``"1.5-3.0"``).
 
 
+Duration+
+---------
+Type
+    * |Duration|_
+    * ``string``
+Example
+    ``"1.5-3.0"``
+    * ``"lin=5"``
+    * ``"lin:20=30-60"``
+    * ``"exp:1.8=40"``
+Description
+    A |Duration|_ value.
+
+    When given as ``string``, it can optionally be prefixed with
+    ``lin[:START[:MAX]]=`` for `linear` or
+    ``exp[:BASE[:START[:MAX]]]=`` for `exponential` growth.
+
+
 Module
 ------
 Type
@@ -10316,6 +10343,7 @@ Reference
 .. |datetime.max| replace:: ``datetime.max``
 .. |Date| replace:: ``Date``
 .. |Duration| replace:: ``Duration``
+.. |Duration+| replace:: ``Duration+``
 .. |Module| replace:: ``Module``
 .. |Path| replace:: ``Path``
 .. |Last-Modified| replace:: ``Last-Modified``

docs/options.md

@@ -11,6 +11,7 @@
 * [Output Options](#output-options)
 * [Networking Options](#networking-options)
 * [Downloader Options](#downloader-options)
+* [Sleep Options](#sleep-options)
 * [Configuration Options](#configuration-options)
 * [Authentication Options](#authentication-options)
 * [Cookie Options](#cookie-options)
@@ -109,34 +110,40 @@
     -r, --limit-rate RATE       Maximum download rate (e.g. 500k, 2.5M, or
                                 800k-2M)
     --chunk-size SIZE           Size of in-memory data chunks (default: 32k)
-    --sleep SECONDS             Number of seconds to wait before each download.
-                                This can be either a constant value or a range
-                                (e.g. 2.7 or 2.0-3.5)
-    --sleep-skip SECONDS        Number of seconds to wait after skipping a file
-                                download
-    --sleep-request SECONDS     Number of seconds to wait between HTTP requests
-                                during data extraction
-    --sleep-429 SECONDS         Number of seconds to wait when receiving a '429
-                                Too Many Requests' response
-    --sleep-extractor SECONDS   Number of seconds to wait before starting data
-                                extraction for an input URL
     --no-part                   Do not use .part files
     --no-skip                   Do not skip downloads; overwrite existing files
     --no-mtime                  Do not set file modification times according to
                                 Last-Modified HTTP response headers
     --no-download               Do not download any files
 
+## Sleep Options:
+    --sleep SECONDS             Number of seconds to wait before each download.
+                                This can be either a constant value or a range
+                                (e.g. 2.7 or 2.0-3.5)
+    --sleep-skip SECONDS        Number of seconds to wait after skipping a file
+                                download
+    --sleep-extractor SECONDS   Number of seconds to wait before starting data
+                                extraction for an input URL
+    --sleep-request SECONDS     Number of seconds to wait between HTTP requests
+                                during data extraction
+    --sleep-429 [TYPE=]SECONDS  Number of seconds to wait when receiving a '429
+                                Too Many Requests' response. Can be prefixed
+                                with 'lin[:START[:MAX]]' or
+                                'exp[:BASE[:START[:MAX]]]' for linear or
+                                exponential growth (e.g. '30', 'exp=40',
+                                'lin:20=30-60'
+
 ## Configuration Options:
     -o, --option KEY=VALUE      Additional options. Example: -o browser=firefox
-    -c, --config FILE           Additional configuration files
+    -c, --config FILE           Additional configuration files in JSON format
     --config-yaml FILE          Additional configuration files in YAML format
     --config-toml FILE          Additional configuration files in TOML format
+    --config-type TYPE          Set filetype of default configuration files
+                                (json, yaml, toml)
+    --config-ignore             Do not load default configuration files
     --config-create             Create a basic configuration file
     --config-status             Show configuration file status
     --config-open               Open configuration file in external application
-    --config-type TYPE          Set filetype of default configuration files
-                                (json, yaml, toml)
-    --config-ignore             Do not read default configuration files
 
 ## Authentication Options:
     -u, --username USER         Username to login with