- remove the use of environment variables to get directory paths,
- make use of arguments / argparse instead of environment variables in `update.py` and `report.py`,
- automatically guess the data directory in `latest.py` based on the script's location,
- propagate log level to auto scripts,
- move `list_configs_from_argv` from `endoflife` module to `releasedata` module,
- use `list_products` in `latest.py` to load the product's frontmatters.
Allow stale responses in case of error in order to reduce the number of temporary errors.
Note that the cache will be reset on the first build of every week in any cases, see .github/workflows/update.yml.
Align custom scripts with generic scripts, making them configurable. This has a few advantages:
- script code is more unified,
- no more hard-coded method names in scripts, which is less error prone and make it easier to rename scripts,
- no more hard coded product names in scripts, which is less error prone and make it easier to rename products,
- less hard-coded URLs and regexes in scripts, which makes auto-configuration more expressive / updatable,
Also added method `endoflife.list_configs_from_argv()` so that it is easier to manipulate scripts arguments.
Make the Apple script compatible with the way update.py now works, which is 'product' oriented, meaning the script will be called once for each product.
To minimize the impacts the responses are now cached to avoid rate-limiting by support.apple.com.
Version patterns have also been moved to product's auto configuration to make future changes simpler.
Until now products could declare multiple auto-update methods, but they all had to be of the same kind.
For example if you used the git auto-update method, you could not use an additional github_releases or custom auto-update method.
This is an issue as it prevents us to extend the auto-update process, for example by having a product using the 'git' auto-update method to retrieve all the versions, and a custom script to retrieve support and EOL dates.
This improve the scripts execution orchestration to be able to support auto configurations using a mix of methods, meaning:
- multiple kind of methods, such as git and github_release,
- or multiple custom methods.
A side-effect of those changes is that now a failure in a generic script does not cancel the update of subsequent products.
Another side-effect, unwanted this time, is that now custom scripts managing multiple products, such as apple.py, are now executed multiple times instead of once.
This way the writing of the JSON file is handled automatically if the update does not fail.
It pave the way to further global improvements, such as a better error handling.
Up to now extra version fields were ignored: only name and date fields were accepted. This changes that by retaining the full JSON data when reading the file, making it possible in the future to support custom fields.
This also fixes a bug with versions having released on the same date: they was not ordered as expected (reverse order).
It may not be the best place for that (gha.py would have been better), but it's the shorter / faster way to do it for now.
Moreover it now uses logging for writing the group. The logger format has been updated for this to work. This was done to fix issues on GitHub Action logs, where groups were declared after the logs.
Make the script more readable, mostly by:
- using the endoflife.Product class,
- removing the unnecessary use of functions,
- a little bit of renaming.
Make the script more readable, mostly by:
- using the endoflife.Product class,
- removing the unnecessary use of functions,
- a little bit of renaming.
Create common functions parse_date, parse_month_year_date and parse_datetime.
Those functions support trying multiple formats, and come with default formats lists that support most of the date format encountered so far.
Notable change: year-month dates are now set to the end of month (impacted couchbase-server and ibm-aix).
Use the new endoflife.fetch_urls to fetch URLs. This is a bit more efficient, especially when network is slow.
Also update the way rows without a proper date format are excluded by checking the format first. I don't know why, but this fixed a bit the script, as now the 6.2.1 is properly retrieved and parsed.
Create a common function to write resulting JSON files to the releases directory.
It makes this task simpler to read and maintain, while making it modifiable at a central point in the future.
One example of such modification could be the sorting of the versions in a uniform way for all the scripts.
This creates a common function to fetch HTTP URLs, with enhanced capabilities (retry, use of a known User-Agent).
It makes scripts that need those capabilities simpler, while improving other scripts.
This commit also fixes some scripts that did not log properly (cos.py, eks.py, haproxy.py, palo-alto-networks.py, rhel.py, ros.py, unrealircd.py).
There were two issues with the macOS regexes:
- The regexex accepted only spaces separator. But non-breakable spaces are also used from time to time. Replacing spaces by \s fixed the issue (other product regexes were also updated - never know).
- The first regex, the one that accepted major releases with no minor version, were not useful (covered by the second regex) and partly catched valid versions (such as 13.1), who was then ignored. I removed it.
