Intermittent ChunkedEncodingErrors occurs while fetching URLs. This change try to fix it by retrying.
According to https://stackoverflow.com/a/44511691/374236, most servers transmit all data, but that's not what was observed.
For future reference the traceback was:
```
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/runner/work/release-data/release-data/src/firefox.py", line 36, in <module>
for response in endoflife.fetch_urls(urls):
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/release-data/release-data/src/common/endoflife.py", line 55, in fetch_urls
return [future.result() for future in as_completed(futures)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/release-data/release-data/src/common/endoflife.py", line 55, in <listcomp>
return [future.result() for future in as_completed(futures)]
^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requests/sessions.py", line 747, in send
r.content
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requests/models.py", line 899, in content
self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requests/models.py", line 818, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
```
Move the parallel URL fetching from firefox.py to endoflife.py to make available parallel URL fetching for all scripts.
Also a a fix found on https://stackoverflow.com/a/44511691/374236 to avoid ChunkedEncodingError.
The purpose of this new script is to be alerted of new runtimes, while not making updates to the original product file (because release dates cannot be fetched from AWS documentation).
The main reason for doing this is to have some common code between scripts, so that it is easier to change the JSON schema globally and normalize a few things (such as release order).
The Ruby code was kept as is so we can quickly roll back if necessary.
Recently links to web.archive.org were added for various products. Such links are very long to load. This increases the default timeout value so that such links do not make the update fail.
The script only get versions 4.x and above, see script comments for more informations.
Note that the code related to git has been extracted to a common script so that it can be reused for the Debian script.
Create a common function to write resulting JSON files to the releases directory.
It makes this task simpler to read and maintain, while making it modifiable at a central point in the future.
One example of such modification could be the sorting of the versions in a uniform way for all the scripts.
This creates a common function to fetch HTTP URLs, with enhanced capabilities (retry, use of a known User-Agent).
It makes scripts that need those capabilities simpler, while improving other scripts.
This commit also fixes some scripts that did not log properly (cos.py, eks.py, haproxy.py, palo-alto-networks.py, rhel.py, ros.py, unrealircd.py).
Generic scripts are scripts that handle multiple product based on some identifier (URL, coordinates...).
This creates a common function to list and load product configurations for those scripts.
It makes them simpler to read and maintain, while making the way they work much more consistent.
