Revision of the Astrophysics Data System (ADS) engine / use of the result type
Paper as well as other typifications.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Revision of the engine / use of the result type Paper as well as other
typifications.
The engine has been placed on inactive because no service is currently
available, or at least not known in the SearXNG community [1]
[1] https://github.com/searxng/searxng/issues/3610
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Repology_ monitors a huge number of package repositories and other sources
comparing packages versions across them and gathering other information.
Repology_ shows you in which repositories a given project is packaged, which
version is the latest and which needs updating, who maintains the package, and
other related information.
.. _Repology: https://repology.org/docs/about
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Format: reST
This patch adds a new result type: Code
- Python class: searx/result_types/code.py
- Jinja template: searx/templates/simple/result_templates/code.html
- CSS (less) client/simple/src/less/result_types/code.less
Signed-of-by: Markus Heiser <markus.heiser@darmarIT.de>
Apparently the layout of https://annas-archive.org has changed, making changes necessary.
The issue has been reported in #5146, see there for more details.
- closes#5146
- Adds a new engine `searx/engines/openalex.py` that integrates the OpenAlex
Works API to return scientific paper results using the `paper.html` template.
- Uses the official API (no auth required); supports OpenAlex polite pool via `mailto`.
- Adds developer docs at `docs/dev/engines/online/openalex.rst`.
OpenAlex API reference: https://docs.openalex.org/how-to-use-the-api/api-overview
- the previous CDN icon list url no longer works
- a list of all icons is mirrored to the JSDelivr CDN however
- there's no reason to set the engine to inactive now that we use public CDNs
This patch adds GitHub Code Search [1] engine to allow querying the codebases.
Template code.html is changed to allow passthrough of strip and highlighting
options.
Engine Searchcode is adjusted to pass filename and not rely on hardcoded
extensions.
GitHub search code API does not return the exact code line indices, this
implementation assigns the code arbitrary numbers starting from 1
(effectively relabeling the code).
The API allows for unauth calls, and the default engine settings default to
that, although the calls are heavily rate limited.
The 'text' lexer is the default pygments lexer when parsing fails.
[1] https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28#search-code
Co-authored-by: Markus Heiser <markus.heiser@darmarIT.de>
pyrightconfig.json :
for the paths searx, searxng_extra and tests, individual rules were
defined (for example, in test fewer / different rules are needed than in the
searx package
searx/engines/__builtins__.pyi :
The builtin types that are added to the global namespace of a module by the
intended monkey patching of the engine modules / replaces the previous
filtering of the stdout using grep.
test.pyright_modified (utils/lib_sxng_test.sh) :
static type check of local modified files not yet commited
make test :
prerequisite 'test.pyright' has been replaced by 'test.pyright_modified'
searx/engines/__init__.py, searx/enginelib/__init__.py :
First, minimal typifications that were considered necessary.
FIxes publishedDate format in reuters engine to encompass ISO 8601 times both with and without milliseconds.
Why is this change important?
Previously, the engine would sometimes fail saying:
2025-08-12 21:13:23,091 ERROR:searx.engines.reuters: exception : time data '2024-04-15T19:08:30.833Z' does not match format '%Y-%m-%dT%H:%M:%SZ'
Traceback (most recent call last):
...
File "/usr/local/searxng/searx/engines/reuters.py", line 87, in response
publishedDate=datetime.strptime(result["display_time"], "%Y-%m-%dT%H:%M:%SZ"),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
Note that most queries seem to work with Reuters, but there are some results that have the additional milliseconds and fail. Regardless, the change is backwards compatible as both the formats (with and without the ms) should now parse correctly.
Add Baidu Captcha detection to reduce `JSONDecodeError` error
Baidu will redirect to `wappass.baidu.com` and return a captcha challenge.
Current behavior will get the data from `wappass.baidu.com` then return a
`json.decoder.JSONDecodeError` error.
Sometimes, there's only an `adaptivestreaming` field in `streams`, which
is usually an m3u8 file. That's however not supported by the video player
of any browser, so we can't use and must build a different url instead.
- not 100% sure about the condition code mapping, there are no real matches for most of the codes from Apple WeatherKit to the weather codes we have in SearXNG
- related: https://github.com/searxng/searxng/issues/4885
* [fix] CI task "update_engine_traits.py" fails
To catch all problems with an HTTP request, the more general class
``httpx.HTTPError`` must be caught, for your test use::
$ ./manage dev.env
$ python ./searxng_extra/update/update_engine_traits.py
Closes: https://github.com/searxng/searxng/issues/5068
* [data] update searx.data - update_engine_traits.py
The error message in case the vqd value could not be determined was incorrect
and triggered an exception::
File "/usr/local/searxng/searxng-src/searx/engines/duckduckgo.py", line 132, in get_vqd
logger.error("vqd value from duckduckgo.com ", resp.status_code)
Message: 'vqd value from duckduckgo.com '
Arguments: (202,)
The current google_videos.py in the master branch is completely non functional, due to it not parsing the returned video search results correctly. The result is searxng saying that no results were found. This commit is a new updated google_videos.py that's designed to fix that and is confirmed to be working.
Implementing the suggestions by Bnyro.
Re-formatted with `black` for compatibility. After failing automated checks, ran the command:
black --line-length 120 --skip-string-normalization --target-version py311 google_videos.py
Wordnik is now an answerer and not in the infobox anymore: it uses the
translations answerer, because it provides all the features needed. By default,
only its first results is shown
Additionally a new "define" category is added - I know, it's the same as the
"dictionaries" category, but I don't think we can alias categories. This allows
to search e.g. for `!define tree`, the idea is to allow easy searches for
definitions of words.
Related:
- https://github.com/searxng/searxng/issues/4111
These engines override the user agent manually using `gen_useragent`, although that's already done in the online preprocessor that runs before the actual `request(query, params)` method is called. Hence, this call is duplicated.
Related:
- https://github.com/searxng/searxng/pull/4990#discussion_r2195142838
- apparently, PDIA switched from Angolia to AWS :/
- we no longer require an API key, but the AWS node might change, so we still have to extract the API url of the node
- the response format is still the same, so no changes needed in that regard
- closes#4989
This patch migrates from `redis==5.2.1` [1] to `valkey==6.1.0` [2].
The migration to valkey is necessary because the company behind Redis has decided
to abandon the open source license. After experiencing a drop in user numbers,
they now want to run it under a dual license again. But this move demonstrates
once again how unreliable the company is and how it treats open source
developers.
To review first, read the docs::
$ make docs.live
Follow the instructions to remove redis:
- http://0.0.0.0:8000/admin/settings/settings_redis.html
Config and install a local valkey DB:
- http://0.0.0.0:8000/admin/settings/settings_valkey.html
[1] https://pypi.org/project/redis/
[2] https://pypi.org/project/valkey/
Co-authored-by: HLFH <gaspard@dhautefeuille.eu>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
What's changed?
- this PR adds Pixabay, a collection of royalty free images
- additionaly it seems to have some videos, so there's an engine for it too
Author Notes
- when using SearXNG's transport, all our requests will get blocked, probably due to fingerprinting
- we should find an alternative solution because this is just a hacky change to make things work for now, but idk how ...
Tube Archivist [1] is a self-hosted project which archives youtube videos on
your own local server. This engine connects with Tube Archivist's search API to
allow searching from SearXNG into your own hosted videos.
[1] https://www.tubearchivist.com/
Signed-off-by: Robert M. Clabough <robert@claobugh.tech>
Co-authored-by: Bnyro <bnyro@tutanota.com>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
