Commit Graph

13 Commits

Author SHA1 Message Date
Marc Wrobel
1f7a3772d6 [http] Improve fetch_javascript_url (#462)
Replace `click_selector` by `wait_for,` which is a selector that we must wait for before considering the page loaded.

Also added `select_wait_for`, which returns the waited for element. Oddly this may be needed in some case (such as `artifactory.py`) where the `page.content()` does not contain the waited for element.
2025-07-09 23:20:39 +02:00
Marc Wrobel
3beb0f6e34 Respect HTTP Cache-Control header if available (#450)
Allow stale responses in case of error in order to reduce the number of temporary errors.

Note that the cache will be reset on the first build of every week in any cases, see .github/workflows/update.yml.
2025-06-28 12:57:41 +02:00
Marc Wrobel
312ce078bb Improve HTTP handling in scripts (#449)
Provide some helper methods to hide the complexity of parsing HTML, JSON, YAML, XML or Markdown.
2025-06-28 12:26:10 +02:00
Marc Wrobel
08d4ea469e Do not use wait_until='networkidle' by default with fetch_javascript_url (#340)
It does not work in all situation, especially with GitHub-rendered markdown files.
2024-04-02 23:17:51 +02:00
Marc Wrobel
2f35f1a8c5 [release_table] Allow clicking on page after the JavaScript rendering (#329)
This allows more elements to be rendered before parsing the page.
2024-03-08 23:57:48 +01:00
Marc Wrobel
2d5145444b [apple] Adapt script for single-product usage (#317)
Make the Apple script compatible with the way update.py now works, which is 'product' oriented, meaning the script will be called once for each product.

To minimize the impacts the responses are now cached to avoid rate-limiting by support.apple.com.

Version patterns have also been moved to product's auto configuration to make future changes simpler.
2024-02-21 00:21:07 +01:00
Marc Wrobel
9cf243a10e Fix scripts requiring rendering pages with javascript (#310)
Replace request_html by playwright, as request_html, as it is [not maintained anymore](https://pypi.org/project/requests-html/) and scripts using it, such as artifactory.py, started to fail.
2024-02-16 23:13:42 +01:00
Marc Wrobel
d72008ae9c Update user-agent (#283)
Update user-agent to the latest Firefox ESR.
2024-01-24 22:58:56 +01:00
Marc Wrobel
801859d78a Enable flake8-return linting rules (#267)
See https://docs.astral.sh/ruff/rules/#flake8-raise-rse.
2023-12-30 14:29:24 +01:00
Marc Wrobel
f49e3dff15 Enable flake8-annotations linting rules (#267)
See https://docs.astral.sh/ruff/rules/#flake8-annotations-ann.
2023-12-30 14:29:24 +01:00
Marc Wrobel
6002cbdb26 Enable isort linting rules (#267)
See https://docs.astral.sh/ruff/rules/#isort-i.
2023-12-30 14:29:24 +01:00
Marc Wrobel
d80504aea8 Add some basic logging (#209)
Add some basic logging when using common.git or common.http.
2023-12-09 23:19:18 +01:00
Marc Wrobel
7cf62c2355 Refactor HTTP common code (#207)
- move to common/http.py,
- make fetch_url return a Response.
2023-12-02 22:52:12 +01:00