Presearch responds with a Cloudflare captcha on each request when using HTTP2.
Using HTTP1.1, everything seems to work fine.
- other engines with the same issue: pixabay, uxwing
- closes https://github.com/searxng/searxng/issues/5438
The results from the recoll engine were not displaying the usual
toggle for showing media previews. After the changes described bellow,
the toggle is displayed and works as expected.
In the JSON returned by recoll-webui, the field containing the
mimetype is actually `mtype`, not `mime`.
Furthermore, according to the documentation for the `File` class in
`searx/result_types/file.py`, `embedded` should contain the URL to the
media itself. The embedding of the media into the page for preview is
done in `searx/templates/simple/result_templates/file.html`.
- official website: https://devicon.dev/
- the engine contains a lot of icons of popular software frameworks (e.g. pytest),
so they could for example be useful for visualizing a diagram of the tech stack used in an app
Yandex engine will return parsing error instead of informing that a CAPTCHA was found. It is confusing for the admin and the users (#5415).
This patch fixes an issue where the CAPTCHA response from Yandex wouldn't be detected, resulting in `ParserError` when trying to parse the response to DOM.
In this fix, I replaced the url condition and instead is checking if the `x-yandex-captcha` header is set, and is equal to `captcha`.
Alternatively, maybe something like `resp.headers.get('Location', '').startswith("https://yandex.com/showcaptcha")` could be done instead. Lastly, setting `params['allow_redirects'] = True` can also work, but this will waste an extra request. Just let me know.
Closes: https://github.com/searxng/searxng/issues/5415
SourceHut uses a foss bot protection tool called `go-away` (which I can
recommend BTW). It blocks common crawler user agents, such as the standard
Firefox user agent. Hence, we're now using our custom SearXNG user agent to
clarify we're not a crawler.
Closes: https://github.com/searxng/searxng/issues/5270
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Fixes an issue where startpage engine would display parsing error
(`json.decoder.JSONDecodeError`) when returning CAPTCHA redirect page.
The fix simply checks if response header has `Location` set, and if it starts
with `https://www.startpage.com/sp/captcha`, it will raise a CAPTCHA exception
before trying to parse the data.
This PR adds a new result type: File
Python class: searx/result_types/file.py
Jinja template: searx/templates/simple/result_templates/file.html
CSS (less) client/simple/src/less/result_types/file.less
Class 'File' (singular) replaces template 'files.html' (plural). The renaming
was carried out because there is only one file (singular) in a result. Not to be
confused with the category 'files' where in multiple results can exist.
As mentioned in issue [1], the class '.category-files' was removed from the CSS
and the stylesheet was adopted in result_types/file.less (there based on the
templates and no longer based on the category).
[1] https://github.com/searxng/searxng/issues/5198
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The size of the full-size images from ``thumbnail.url`` is usually several
MB. By reducing the full-size image to 80 pixels, the data size for a thumb is
reduced from MB to a few KB.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
It doesn't matter if you're using Mullvad's VPN and a proper browser, you'll
still get blocked for specific searches [1] with a 403 or 429 HTTP status code.
Mullvad only blocks the search request and doesn't prevent you from doing more
searches.
The logic should handle the blocked requests (403, 429), but not put the engine
on a cooldown.
[1] https://leta.mullvad.net/search?q=site%3Afoo+bar&engine=brave
Closes: https://github.com/searxng/searxng/issues/5328
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Apparently, in China, Bing redirects from `www.bing.com` to `cn.bing.com`.
So in order to make Bing work for chinese users by default, we have to follow that redirect.
related: https://github.com/searxng/searxng/issues/5243
Adds a new engine `searx/engines/azure.py` to search cloud resources on Azure.
A lot of enterprise users have to deal with Azure Public Cloud. This helps them
easily search for cloud resources without logging in to the Portal first
How to test this PR locally?
You should create an App Registration on Azure Entra Id with Reader access on
the resources you want to search for. You should create a Secret for the App
Registration. After that, you should set up appropriate values in the
`settings.yml` file [1]::
- name: azure
engine: azure
...
azure_tenant_id: "your_tenant_id"
azure_client_id: "your_client_id"
azure_client_secret: "your_client_secret"
azure_token_expiration_seconds: 5000
[1] https://github.com/searxng/searxng/pull/5235#issuecomment-3397664928
Co-authored-by: Bnyro <bnyro@tutanota.com>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
The Name of the option is *disable_family_filter* -> we have to reverse the
meaning of the ascending safe-search filter level.
Closes: https://github.com/searxng/searxng/issues/5287
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the case of .. response, for example, an HTTP 302 is returned by Google
Scholar::
Our systems have detected unusual traffic from your computer
network. Please try again later.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
cppreference has replaced its search (``mwiki/index.php?title=``) with a DDG
search.
The engine was first introduced in SearXNG with PR-3274 [1], and even back then
the mediawiki proved to be incompatible, which is why the API could not be used
at the time. Now there isn't even a dedicated search function anymore.. I think
the cppreference project suffers from a lack of maintenance.
[1] https://github.com/searxng/searxng/pull/3247
Closes: https://github.com/searxng/searxng/issues/5271
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In some engines, under certain circumstances, the content field can also have
the value ``None``; in these cases, a length check results in an exception::
File "/usr/local/searxng/searx/results.py", line 360, in merge_two_main_results
if len(other.content) > len(origin.content):
^^^^^^^^^^^^^^^^^^
TypeError: object of type 'NoneType' has no len()
[1] https://github.com/searxng/searxng/issues/5250#issuecomment-3352863488
Reported-by: @scross01 [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The ``JSONEncoder`` (``format="json"``) must perform a conversion to the
built-in types for the ``msgspec.Struct``::
if isinstance(o, msgspec.Struct):
return msgspec.to_builtins(o)
The result types are already of type ``msgspec.Struct``, so they can be
converted into built-in types.
The field types (in the result type) that were not yet of type ``msgspec.Struct``
have been converted to::
searx.weather.GeoLocation@dataclass -> msgspec.Struct
searx.weather.DateTime -> msgspec.Struct
searx.weather.Temperature -> msgspec.Struct
searx.weather.PressureUnits -> msgspec.Struct
searx.weather.WindSpeed -> msgspec.Struct
searx.weather.RelativeHumidity -> msgspec.Struct
searx.weather.Compass -> msgspec.Struct
BTW: Wherever it seemed sensible, the typing was also modernized in the modified
files.
Closes: https://github.com/searxng/searxng/issues/5250
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Revision of the Astrophysics Data System (ADS) engine / use of the result type
Paper as well as other typifications.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Revision of the engine / use of the result type Paper as well as other
typifications.
The engine has been placed on inactive because no service is currently
available, or at least not known in the SearXNG community [1]
[1] https://github.com/searxng/searxng/issues/3610
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Repology_ monitors a huge number of package repositories and other sources
comparing packages versions across them and gathering other information.
Repology_ shows you in which repositories a given project is packaged, which
version is the latest and which needs updating, who maintains the package, and
other related information.
.. _Repology: https://repology.org/docs/about
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Format: reST
