[common] add 'googlebot' User-Agent preset

docs/configuration.rst

@@ -830,7 +830,11 @@ Description
       will use the latest ``User-Agent`` header of this preset target,
       e.g. ``"+ff"``.
     | (Supported values:
-      ``firefox`` | ``ff`` | ``chrome`` | ``cr`` | ``gallery-dl`` | ``gdl``)
+      ``firefox`` | ``ff`` |
+      ``chrome`` | ``cr`` |
+      ``gallery-dl`` | ``gdl`` |
+      ``google-bot`` | ``bot``
+      )
 
     | Starting this value with an ``@``
      will try to automatically detect and use the ``User-Agent`` header

gallery_dl/extractor/common.py

@@ -485,6 +485,8 @@ class Extractor():
                 headers["User-Agent"] = util.USERAGENT_CHROME
             elif custom_ua in {"gallery-dl", "gallerydl", "gdl"}:
                 headers["User-Agent"] = util.USERAGENT_GALLERYDL
+            elif custom_ua in {"google-bot", "googlebot", "bot"}:
+                headers["User-Agent"] = "Googlebot-Image/1.0"
             else:
                 self.log.warning(
                     "Unsupported User-Agent preset '%s'", custom_ua)