Make the script more readable, mostly by:
- using the Product classes,
- removing the use of functions when unnecessary,
- a little bit of renaming and documentation.
Make the script more readable, mostly by:
- using the Product classes,
- removing the use of functions when unnecessary,
- a little bit of renaming and documentation.
Make the script more readable, mostly by:
- using the Product and AutoConfig classes,
- removing the use of functions when unnecessary,
- a little bit of renaming and documentation.
Note that this also changed the module used for regexes in endoflife.py. The regex module is now used because the Python re module does not support identically named groups (as used in the mariadb product). The regex module being backwards-compatible with the standard re module, this should not be an issue.
Make the script more readable, mostly by:
- using the Product and AutoConfig classes,
- removing the use of functions when unnecessary,
- a little bit of renaming and documentation.
Make the script more readable, mostly by:
- changing slightly the logic,
- using the Product and AutoConfig classes,
- removing the use of functions when unnecessary,
- a little bit of renaming and documentation.
Disabled web.archive.org has also been re-enabled because HTTP retry mechanism has been improved and should handle timeouts a lot better.
Make the script more readable, mostly by:
- using the Product and AutoConfig classes,
- removing the use of functions when unnecessary,
- a little bit of renaming and documentation.
Make the script more readable, mostly by:
- using the endoflife.Product class,
- removing the use of functions when unnecessary,
- a little bit of renaming and documentation.
Make the script more readable, mostly by:
- using the endoflife.Product class,
- removing some unnecessary use of functions,
- a little bit of renaming and documentation.
Make the script more readable, mostly by:
- using the endoflife.Product class,
- introducing the endoflife.AutoConfig class to make it easier to manage such configuration,
- removing the unnecessary use of functions,
- a little bit of renaming.
Make the script more readable, mostly by:
- using the endoflife.Product class,
- removing the unnecessary use of functions,
- a little bit of renaming.
Make the script more readable, mostly by:
- using the endoflife.Product class,
- removing the unnecessary use of functions,
- a little bit of renaming.
Make the script more readable, mostly by:
- using the endoflife.Product class,
- removing the unnecessary use of functions,
- a little bit of renaming.
Make the script more readable, mostly by:
- using the endoflife.Product class,
- removing the unnecessary use of functions,
- renaming a few variables,
- declaring a constant for regexes.
This will be useful in future PRs:
- add a few more supported formats,
- cleanup string so that we don't have to deal with some special characters in formats.
Automation is only for the latest 3 majors version (2022, 2019 and 2017) as there is no release history published for 2015 and below.
This is not a big deal because there was no version for those release in a very long time.
Create common functions parse_date, parse_month_year_date and parse_datetime.
Those functions support trying multiple formats, and come with default formats lists that support most of the date format encountered so far.
Notable change: year-month dates are now set to the end of month (impacted couchbase-server and ibm-aix).
When a ChunkedEncodingError occurs, request and response are not set and there is no way to get the URL that causes the error.
With this change all URLs are retried. The max_retries parameter is decreased each time so that we do not get stuck in an infinite loop.
I also considered to also wait before retrying, but for now I don't see any benefit to it.
Relates to #188.
Use the new endoflife.fetch_urls to fetch URLs. This is a bit more efficient, especially when network is slow.
Also update the way rows without a proper date format are excluded by checking the format first. I don't know why, but this fixed a bit the script, as now the 6.2.1 is properly retrieved and parsed.
Intermittent ChunkedEncodingErrors occurs while fetching URLs. This change try to fix it by retrying.
According to https://stackoverflow.com/a/44511691/374236, most servers transmit all data, but that's not what was observed.
For future reference the traceback was:
```
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/runner/work/release-data/release-data/src/firefox.py", line 36, in <module>
for response in endoflife.fetch_urls(urls):
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/release-data/release-data/src/common/endoflife.py", line 55, in fetch_urls
return [future.result() for future in as_completed(futures)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/release-data/release-data/src/common/endoflife.py", line 55, in <listcomp>
return [future.result() for future in as_completed(futures)]
^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requests/sessions.py", line 747, in send
r.content
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requests/models.py", line 899, in content
self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requests/models.py", line 818, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
```
Move the parallel URL fetching from firefox.py to endoflife.py to make available parallel URL fetching for all scripts.
Also a a fix found on https://stackoverflow.com/a/44511691/374236 to avoid ChunkedEncodingError.
The purpose of this new script is to be alerted of new runtimes, while not making updates to the original product file (because release dates cannot be fetched from AWS documentation).
The main reason for doing this is to have some common code between scripts, so that it is easier to change the JSON schema globally and normalize a few things (such as release order).
The Ruby code was kept as is so we can quickly roll back if necessary.
