17 Commits

Author SHA1 Message Date
Mike Fährmann
968597a302 yield 3-tuples for Message.Directory
adapt tuples to the same length and semantics as other messages
2025-12-05 21:39:52 +01:00
Mike Fährmann
9bf76c1352 replace 'util.re()' with 'text.re()'
remove unnecessary 'util' imports
2025-10-20 17:44:58 +02:00
Mike Fährmann
41191bb60a 'match.group(N)' -> 'match[N]' (#7671)
2.5x faster
2025-06-18 13:05:58 +02:00
Mike Fährmann
b5c88b3d3e replace standard library 're' uses with 'util.re()' 2025-06-06 13:24:52 +02:00
Mike Fährmann
4906541f7d [generic] fix config lookups by subcategory
'subcategory' needs to be set before Extractor.__init__() runs
to be included in '_cfgpath'
2025-02-22 10:08:59 +01:00
Mike Fährmann
73f77a543a [generic] ensure 'path' metadata is always defined
fixes None directory names when 'path' is empty and got removed
2024-10-10 14:06:47 +02:00
Allen
a552fb5bcd [generic] better directory formatting/md 2024-08-29 22:30:48 +02:00
Mike Fährmann
a453335a9f remove test results in extractor modules
and add generic example URLs
2023-09-11 16:30:55 +02:00
Mike Fährmann
1614c5c4bf [generic] write regular expressions without 'x' flags 2023-04-10 20:45:23 +02:00
Mike Fährmann
4883420e67 [generic] revert pattern change 2023-03-09 22:25:23 +01:00
ClosedPort22
34a7fab0e2 [generic] add support for IDNs
(internationalized domain name)
2023-03-06 22:42:36 +08:00
thatfuckingbird
8cfeed78b1 [generic] fix regex for non-src image URLs 2023-01-21 22:32:42 +01:00
Mike Fährmann
b0cb4a1b9c replace 'text.extract()' with 'text.extr()' where possible 2022-11-05 01:14:09 +01:00
Mike Fährmann
b8d268f57e allow '/' and '?' in URL queries 2022-10-02 19:02:05 +02:00
Vrihub
96fcff182c generic extractor (#735)
* Generic extractor, see issue #683

* Fix failed test_names test, no subcategory needed

* Prefix directory_fmt with "generic"

* Relax regex (would break some urls)

* Flake8 compliance

* pattern: don't require a scheme

This fixes a bug when we force the generic extractor on urls without a
scheme (that are allowed by all other extractors).

* Fix using g: and r: on urls without http(s) scheme

Almost all extractors accept urls without an initial http(s) scheme.

Many extractors also allow for generic subdomains in their "pattern"
variable; some of them implement this with the regex character class
"[^.]+" (everything but a dot).

This leads to a problem when the extractor is given a url starting
with g: or r: (to force using the generic or recursive extractor)
and without the http(s) scheme: e.g. with "r:foobar.tumblr.com"
the "r:" is wrongly considered part of the subdomain.

This commit fixes the bug, replacing the too generic "[^.]+" with the
more specific "[\w-]+" (letters, digits and "-", the only characters
allowed in domain names), which is already used by some extractors.

* Relax imageurl_pattern_ext: allow relative urls

* First round of small suggested changes

* Support image urls starting with "//"

* self.baseurl: remove trailing slash

* Relax regexp (didn't catch some image urls)

* Some fixes and cleanup

* Fix domain pattern; option to enable extractor

Fixed the domain section for "pattern", to pass "test_add" and
"test_add_module" tests.
Added the "enabled" configuration option (default False) to enable the
generic extractor. Using "g(eneric):URL" forces using the extractor.
2021-12-29 22:39:29 +01:00
Mike Fährmann
d31ccb16f2 rename 'generic' to 'recursive' 2016-10-01 15:54:27 +02:00
Mike Fährmann
65c8566326 add provisional generic extractor 2015-12-03 00:50:34 +01:00