The requests changed here all run outside of the network context timeout,
thereby preventing the engine's timeout from being applied (the engine's timeout
can become longer than it was configured).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Presearch responds with a Cloudflare captcha on each request when using HTTP2.
Using HTTP1.1, everything seems to work fine.
- other engines with the same issue: pixabay, uxwing
- closes https://github.com/searxng/searxng/issues/5438
On a long-running server, the tracebacks below can be found (albeit rarely),
which indicate problems with NoneType where a string or another data type is
expected.
result.img_src::
File "/usr/local/searxng/searxng-src/searx/templates/simple/result_templates/images.html", line 13, in top-level template code
<img src="" data-src="{{ image_proxify(result.img_src) }}" alt="{{ result.title|striptags }}">{{- "" -}}
^
File "/usr/local/searxng/searxng-src/searx/webapp.py", line 284, in image_proxify
if url.startswith('//'):
^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'startswith'
result.content::
File "/usr/local/searxng/searxng-src/searx/result_types/_base.py", line 105, in _normalize_text_fields
result.content = WHITESPACE_REGEX.sub(" ", result.content).strip()
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'NoneType'
html_to_text, when html_str is a NoneType::
File "/usr/local/searxng/searxng-src/searx/engines/wikipedia.py", line 190, in response
title = utils.html_to_text(api_result.get('titles', {}).get('display') or api_result.get('title'))
File "/usr/local/searxng/searxng-src/searx/utils.py", line 158, in html_to_text
html_str = html_str.replace('\n', ' ').replace('\r', ' ')
^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'replace'
presearch engine, when json_resp is a NoneType::
File "/usr/local/searxng/searxng-src/searx/engines/presearch.py", line 221, in response
results = parse_search_query(json_resp.get('results'))
File "/usr/local/searxng/searxng-src/searx/engines/presearch.py", line 161, in parse_search_query
for item in json_results.get('specialSections', {}).get('topStoriesCompact', {}).get('data', []):
^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The use of img_src AND thumbnail in the default results makes no sense (only a
thumbnail is needed). In the current state this is rather confusing, because
img_src is displayed like a thumbnail (small) and thumbnail is displayed like an
image (large).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the past, some files were tested with the standard profile, others with a
profile in which most of the messages were switched off ... some files were not
checked at all.
- ``PYLINT_SEARXNG_DISABLE_OPTION`` has been abolished
- the distinction ``# lint: pylint`` is no longer necessary
- the pylint tasks have been reduced from three to two
1. ./searx/engines -> lint engines with additional builtins
2. ./searx ./searxng_extra ./tests -> lint all other python files
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In Presearch there are languages for the UI and regions for narrowing down the
search. With this change the SearXNG engine supports a search by region. The
details can be found in the documentation of the source code.
To test, you can search terms like::
!presearch bmw :zh-TW
!presearch bmw :en-CA
1. You should get results corresponding to the region (Taiwan, Canada)
2. and in the language (Chinese, Englisch).
3. The context in info box content is in the same language.
Exceptions:
1. Region or language is not supported by Presearch or
2. SearXNG user did not selected a region tag, example::
!presearch bmw :en
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
