Use the new endoflife.fetch_urls to fetch URLs. This is a bit more efficient, especially when network is slow.
Also update the way rows without a proper date format are excluded by checking the format first. I don't know why, but this fixed a bit the script, as now the 6.2.1 is properly retrieved and parsed.
Intermittent ChunkedEncodingErrors occurs while fetching URLs. This change try to fix it by retrying.
According to https://stackoverflow.com/a/44511691/374236, most servers transmit all data, but that's not what was observed.
For future reference the traceback was:
```
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/runner/work/release-data/release-data/src/firefox.py", line 36, in <module>
for response in endoflife.fetch_urls(urls):
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/release-data/release-data/src/common/endoflife.py", line 55, in fetch_urls
return [future.result() for future in as_completed(futures)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/release-data/release-data/src/common/endoflife.py", line 55, in <listcomp>
return [future.result() for future in as_completed(futures)]
^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requests/sessions.py", line 747, in send
r.content
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requests/models.py", line 899, in content
self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requests/models.py", line 818, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
```
Move the parallel URL fetching from firefox.py to endoflife.py to make available parallel URL fetching for all scripts.
Also a a fix found on https://stackoverflow.com/a/44511691/374236 to avoid ChunkedEncodingError.
The purpose of this new script is to be alerted of new runtimes, while not making updates to the original product file (because release dates cannot be fetched from AWS documentation).
The main reason for doing this is to have some common code between scripts, so that it is easier to change the JSON schema globally and normalize a few things (such as release order).
The Ruby code was kept as is so we can quickly roll back if necessary.
Fetch Looker versions with their dates from the Google Cloud release notes RSS feed: https://cloud.google.com/feeds/looker-release-notes.xml.
Only version above 23.0 are retrieved, as prior version were not announced in the Google Cloud release notes.
Using tags misses all ELTS releases, which are relevant for ELTS-only release cycles.
Important changes:
- very old v3 and v4 releases completely dropped except for 3.3.0, and 3.5.0.
- We track only as far back as v7, so dropping v4 and earlier should be fine.
- 7.0.1 and 9.3.2 are missing, which is okay because both are superseded.
Fixes#92.
Recently links to web.archive.org were added for various products. Such links are very long to load. This increases the default timeout value so that such links do not make the update fail.
GraalVM for JDK versions has to be prefixed as their release cycle collide with older GraalVM release cycles.
Example: GraalVM for JDK 20 and 20.0 (will be the same with JDK 21 and 22).
The script only get versions 4.x and above, see script comments for more informations.
Note that the code related to git has been extracted to a common script so that it can be reused for the Debian script.
AWS is no longer publishing the history on GitHub, so we
instead rely on older copies of the website to track the old
release dates. This is done at 2 cutoff points right now for
1.19 and 1.20.
Older releases are available on the web archive, but those
pages didn't include release dates so not helpful.
