add 'browser' option (#1117)

- change default user agent to Firefox ESR 78 on Windows 10
- remove 'ciphers' option
This commit is contained in:
Mike Fährmann
2021-02-25 23:39:34 +01:00
parent 92071d02f4
commit cf5fa75d4c
4 changed files with 146 additions and 65 deletions

View File

@@ -380,7 +380,7 @@ extractor.*.user-agent
Type
``string``
Default
``"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"``
``"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0"``
Description
User-Agent header value to be used for HTTP requests.
@@ -388,6 +388,24 @@ Description
as these need specific values to function correctly.
extractor.*.browser
-------------------
Type
``string``
Example
* ``"firefox"``
* ``"chrome:macos"``
Description
Try to emulate a real browser (``firefox`` or ``chrome``)
by using their default HTTP headers and TLS ciphers for HTTP requests.
Optionally, the operating system used in the ``User-Agent`` header can be
specified after a ``:`` (``windows``, ``linux``, or ``macos``).
Note: ``requests`` and ``urllib3`` only support HTTP/1.1, while a real
browser would use HTTP/2.
extractor.*.keywords
--------------------
Type
@@ -2457,20 +2475,6 @@ Description
this cache.
ciphers
-------
Type
``bool`` or ``string``
Default
``true``
Description
* ``true``: Update urllib3's default cipher list
* ``false``: Leave the default cipher list as is
* Any ``string``: Replace urllib3's default ciphers with these
(See `SSLContext.set_ciphers() <https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_ciphers>`__
for details)
pyopenssl
---------
Type