FIxes publishedDate format in reuters engine to encompass ISO 8601 times both with and without milliseconds.
Why is this change important?
Previously, the engine would sometimes fail saying:
2025-08-12 21:13:23,091 ERROR:searx.engines.reuters: exception : time data '2024-04-15T19:08:30.833Z' does not match format '%Y-%m-%dT%H:%M:%SZ'
Traceback (most recent call last):
...
File "/usr/local/searxng/searx/engines/reuters.py", line 87, in response
publishedDate=datetime.strptime(result["display_time"], "%Y-%m-%dT%H:%M:%SZ"),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
Note that most queries seem to work with Reuters, but there are some results that have the additional milliseconds and fail. Regardless, the change is backwards compatible as both the formats (with and without the ms) should now parse correctly.
Add Baidu Captcha detection to reduce `JSONDecodeError` error
Baidu will redirect to `wappass.baidu.com` and return a captcha challenge.
Current behavior will get the data from `wappass.baidu.com` then return a
`json.decoder.JSONDecodeError` error.
The HTTP X-Forwarded-Proto (XFP) request header is a *de-facto* standard header
for identifying the protocol (HTTP or HTTPS) that a client used to connect to a
proxy or load balancer.[1]
The ``X-Scheme`` header was added 10 years ago, why ``X-Scheme`` was used back
then and not ``X-Forwarded-Proto``, nobody knows today / possibly because
``X-Forwarded-Proto`` wasn't a *de-facto* standard back then.
[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Forwarded-Proto
[2] https://github.com/searx/searx/commit/6ef7c3276
Sometimes, there's only an `adaptivestreaming` field in `streams`, which
is usually an m3u8 file. That's however not supported by the video player
of any browser, so we can't use and must build a different url instead.
WhiteNoise requires all headers to be strings, however it's common to use
other primitive types (e.g. numbers) in the header, e.g. `X-XSS-Protection: 0`.
Thus, we must convert all types of values (i.e. numbers) to strings.
- closes https://github.com/searxng/searxng/issues/5091
Building the container currently does not work properly.
When rebuilding several times with `make container`, `version_frozen.py`
is recreated, which wouldn't be an issue if the file’s timestamp was constant.
Now, when creating `version_frozen.py`, it will have the same timestamp as the
commit when it was created. (`version_frozen.py` is moved to a dedicated layer).
Reusing "builder" cache when building "dist" could be slow
(CD reports 2 seconds, but locally I've seen it take up to 10 seconds),
so the Dockerfile is now split and we save a couple steps
by importing the "builder" image directly.
The last changes made it possible to remove the layer cache in "builder",
since the overhead is now greater than building the layers from scratch.
Until now, all "dist" layers were squashed into a single layer,
which in most cases is a good idea
(except for storage/delivery pricing/overhead), but in our case,
since we manage the entire pipeline, we can ignore this
and share layers between builds.
This means (for example) that if we change files unrelated to the container
in several consecutive commits (documentation changes), we don't have to push
the entire image to registry, but only the different layers
(`version_frozen.py` in this example).
The same applies when pulling, as only the layers that have changed
compared to the local layers will be downloaded (that's the theory,
we'll see if this works as expected or if we need to tweak something else).
- not 100% sure about the condition code mapping, there are no real matches for most of the codes from Apple WeatherKit to the weather codes we have in SearXNG
- related: https://github.com/searxng/searxng/issues/4885
* [fix] CI task "update_engine_traits.py" fails
To catch all problems with an HTTP request, the more general class
``httpx.HTTPError`` must be caught, for your test use::
$ ./manage dev.env
$ python ./searxng_extra/update/update_engine_traits.py
Closes: https://github.com/searxng/searxng/issues/5068
* [data] update searx.data - update_engine_traits.py
The error message in case the vqd value could not be determined was incorrect
and triggered an exception::
File "/usr/local/searxng/searxng-src/searx/engines/duckduckgo.py", line 132, in get_vqd
logger.error("vqd value from duckduckgo.com ", resp.status_code)
Message: 'vqd value from duckduckgo.com '
Arguments: (202,)
If the ``on_result`` handle returns False, then the ``else`` was always jumped
to, which throws the NotImplementedError exception::
File "/usr/local/searxng/searxng-src/searx/results.py", line 99, in extend
raise NotImplementedError(f"no handler implemented to process the result of type {result}")
NotImplementedError: no handler implemented to process the result of type MainResult(title=...
The current google_videos.py in the master branch is completely non functional, due to it not parsing the returned video search results correctly. The result is searxng saying that no results were found. This commit is a new updated google_videos.py that's designed to fix that and is confirmed to be working.
Implementing the suggestions by Bnyro.
Re-formatted with `black` for compatibility. After failing automated checks, ran the command:
black --line-length 120 --skip-string-normalization --target-version py311 google_videos.py
The path to static should be relative (If sxng is served under "/sxng", the static route passed to the client won't be "/sxng/static/..." as expected but "/static/...")
Closes https://github.com/searxng/searxng/issues/5042
Wordnik is now an answerer and not in the infobox anymore: it uses the
translations answerer, because it provides all the features needed. By default,
only its first results is shown
Additionally a new "define" category is added - I know, it's the same as the
"dictionaries" category, but I don't think we can alias categories. This allows
to search e.g. for `!define tree`, the idea is to allow easy searches for
definitions of words.
Related:
- https://github.com/searxng/searxng/issues/4111
While looking at ways to better handle static files, I saw a package that replaces Flask `static_folder` functionality. Not only it's considerably faster, but already includes the capability to serve sidecars without having to intercept. This also replaces the uWSGI folder mapping functionality.
Closes https://github.com/searxng/searxng/issues/4977
Whether the query is a real calculation tasks is currently only detected in the
AST, resulting in unnecessary creatins of subprocesses. This problem is
mitigated with this patch: if the query contains letters, it is obviously not a
math problem, and the plugin can return without further action.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In customizing it should be decided which plugin modules should be loaded and
which should not.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
These engines override the user agent manually using `gen_useragent`, although that's already done in the online preprocessor that runs before the actual `request(query, params)` method is called. Hence, this call is duplicated.
Related:
- https://github.com/searxng/searxng/pull/4990#discussion_r2195142838
- apparently, PDIA switched from Angolia to AWS :/
- we no longer require an API key, but the AWS node might change, so we still have to extract the API url of the node
- the response format is still the same, so no changes needed in that regard
- closes#4989
