mirror of
https://github.com/searxng/searxng.git
synced 2025-10-31 10:37:06 -04:00
8f8343dc0d
Continuation of #5147 .. typification of the engine processors. BTW: - removed obsolete engine property https_support - fixed & improved currency_convert - engine instances can now implement a engine.setup method [#5147] https://github.com/searxng/searxng/pull/5147 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
252 lines
11 KiB
ReStructuredText
252 lines
11 KiB
ReStructuredText
.. _engines-dev:
|
|
|
|
===============
|
|
Engine Overview
|
|
===============
|
|
|
|
.. contents::
|
|
:depth: 3
|
|
:local:
|
|
:backlinks: entry
|
|
|
|
.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine
|
|
|
|
.. sidebar:: Further reading ..
|
|
|
|
- :ref:`configured engines`
|
|
- :ref:`settings engines`
|
|
|
|
SearXNG is a metasearch-engine_, so it uses different search engines to provide
|
|
better results.
|
|
|
|
Because there is no general search API which could be used for every search
|
|
engine, an adapter has to be built between SearXNG and the external search
|
|
engines. Adapters are stored under the folder :origin:`searx/engines`.
|
|
|
|
.. _general engine configuration:
|
|
|
|
General Engine Configuration
|
|
============================
|
|
|
|
It is required to tell SearXNG the type of results the engine provides. The
|
|
arguments can be set in the engine file or in the settings file (normally
|
|
``settings.yml``). The arguments in the settings file override the ones in the
|
|
engine file.
|
|
|
|
It does not matter if an option is stored in the engine file or in the settings.
|
|
However, the standard way is the following:
|
|
|
|
.. _engine file:
|
|
|
|
Engine File
|
|
-----------
|
|
|
|
.. table:: Common options in the engine module
|
|
:width: 100%
|
|
|
|
======================= =========== ========================================================
|
|
argument type information
|
|
======================= =========== ========================================================
|
|
categories list categories, in which the engine is working
|
|
paging boolean support multiple pages
|
|
time_range_support boolean support search time range
|
|
engine_type str - ``online`` :ref:`[ref] <online engines>` by
|
|
default, other possibles values are:
|
|
- ``offline`` :ref:`[ref] <offline engines>`
|
|
- ``online_dictionary`` :ref:`[ref] <online dictionary>`
|
|
- ``online_currency`` :ref:`[ref] <online currency>`
|
|
- ``online_url_search`` :ref:`[ref] <online url search>`
|
|
======================= =========== ========================================================
|
|
|
|
.. _engine settings:
|
|
|
|
Engine ``settings.yml``
|
|
-----------------------
|
|
|
|
For a more detailed description, see :ref:`settings engines` in the :ref:`settings.yml`.
|
|
|
|
.. table:: Common options in the engine setup (``settings.yml``)
|
|
:width: 100%
|
|
|
|
======================= =========== ==================================================
|
|
argument type information
|
|
======================= =========== ==================================================
|
|
name string name of search-engine
|
|
engine string name of searxng-engine (file name without ``.py``)
|
|
enable_http bool enable HTTP (by default only HTTPS is enabled).
|
|
shortcut string shortcut of search-engine
|
|
timeout string specific timeout for search-engine
|
|
display_error_messages boolean display error messages on the web UI
|
|
proxies dict set proxies for a specific engine
|
|
(e.g. ``proxies : {http: socks5://proxy:port,
|
|
https: socks5://proxy:port}``)
|
|
======================= =========== ==================================================
|
|
|
|
.. _engine overrides:
|
|
|
|
Overrides
|
|
---------
|
|
|
|
A few of the options have default values in the namespace of the engine's python
|
|
module, but are often overwritten by the settings. If ``None`` is assigned to an
|
|
option in the engine file, it has to be redefined in the settings, otherwise
|
|
SearXNG will not start with that engine (global names with a leading underline can
|
|
be ``None``).
|
|
|
|
Here is an very simple example of the global names in the namespace of engine's
|
|
module:
|
|
|
|
.. code:: python
|
|
|
|
# engine dependent config
|
|
categories = ['general']
|
|
paging = True
|
|
_non_overwritten_global = 'foo'
|
|
|
|
|
|
.. table:: The naming of overrides is arbitrary / recommended overrides are:
|
|
:width: 100%
|
|
|
|
======================= =========== ===========================================
|
|
argument type information
|
|
======================= =========== ===========================================
|
|
base_url string base-url, can be overwritten to use same
|
|
engine on other URL
|
|
number_of_results int maximum number of results per request
|
|
language string ISO code of language and country like en_US
|
|
api_key string api-key if required by engine
|
|
======================= =========== ===========================================
|
|
|
|
.. _engine request:
|
|
|
|
Making a Request
|
|
================
|
|
|
|
To perform a search an URL have to be specified. In addition to specifying an
|
|
URL, arguments can be passed to the query.
|
|
|
|
.. _engine request arguments:
|
|
|
|
Passed Arguments (request)
|
|
--------------------------
|
|
|
|
These arguments can be used to construct the search query. Furthermore,
|
|
parameters with default value can be redefined for special purposes.
|
|
|
|
.. _engine request online:
|
|
|
|
.. table:: If the ``engine_type`` is :py:obj:`online
|
|
<searx.search.processors.online.OnlineProcessor.get_params>`
|
|
:width: 100%
|
|
|
|
====================== ============== ========================================================================
|
|
argument type default-value, information
|
|
====================== ============== ========================================================================
|
|
url str ``''``
|
|
method str ``'GET'``
|
|
headers dict ``{}``
|
|
data dict ``{}``
|
|
cookies dict ``{}``
|
|
verify bool ``True``
|
|
headers.User-Agent str a random User-Agent
|
|
category str current category, like ``'general'``
|
|
safesearch int ``0``, between ``0`` and ``2`` (normal, moderate, strict)
|
|
time_range Optional[str] ``None``, can be ``day``, ``week``, ``month``, ``year``
|
|
pageno int current pagenumber
|
|
searxng_locale str SearXNG's locale selected by user. Specific language code like
|
|
``'en'``, ``'en-US'``, or ``'all'`` if unspecified.
|
|
====================== ============== ========================================================================
|
|
|
|
|
|
.. _engine request online_dictionary:
|
|
|
|
.. table:: If the ``engine_type`` is :py:obj:`online_dictionary
|
|
<searx.search.processors.online_dictionary.OnlineDictionaryProcessor.get_params>`,
|
|
in addition to the :ref:`online <engine request online>` arguments:
|
|
:width: 100%
|
|
|
|
====================== ============== ========================================================================
|
|
argument type default-value, information
|
|
====================== ============== ========================================================================
|
|
from_lang str specific language code like ``'en_US'``
|
|
to_lang str specific language code like ``'en_US'``
|
|
query str the text query without the languages
|
|
====================== ============== ========================================================================
|
|
|
|
.. _engine request online_currency:
|
|
|
|
.. table:: If the ``engine_type`` is :py:obj:`online_currency
|
|
<searx.search.processors.online_currency.OnlineCurrencyProcessor.get_params>`,
|
|
in addition to the :ref:`online <engine request online>` arguments:
|
|
:width: 100%
|
|
|
|
====================== ============== ========================================================================
|
|
argument type default-value, information
|
|
====================== ============== ========================================================================
|
|
amount float the amount to convert
|
|
from str ISO 4217 code
|
|
to str ISO 4217 code
|
|
from_name str currency name
|
|
to_name str currency name
|
|
====================== ============== ========================================================================
|
|
|
|
.. _engine request online_url_search:
|
|
|
|
.. table:: If the ``engine_type`` is :py:obj:`online_url_search
|
|
<searx.search.processors.online_url_search.OnlineUrlSearchProcessor.get_params>`,
|
|
in addition to the :ref:`online <engine request online>` arguments:
|
|
:width: 100%
|
|
|
|
====================== ============== ========================================================================
|
|
argument type default-value, information
|
|
====================== ============== ========================================================================
|
|
search_url dict URLs from the search query:
|
|
|
|
.. code:: python
|
|
|
|
{
|
|
'http': str,
|
|
'ftp': str,
|
|
'data:image': str
|
|
}
|
|
====================== ============== ========================================================================
|
|
|
|
Specify Request
|
|
---------------
|
|
|
|
The function :py:func:`def request(query, params):
|
|
<searx.engines.demo_online.request>` always returns the ``params`` variable, the
|
|
following parameters can be used to specify a search request:
|
|
|
|
.. table::
|
|
:width: 100%
|
|
|
|
=================== =========== ==========================================================================
|
|
argument type information
|
|
=================== =========== ==========================================================================
|
|
url str requested url
|
|
method str HTTP request method
|
|
headers dict HTTP header information
|
|
data dict HTTP data information
|
|
cookies dict HTTP cookies
|
|
verify bool Performing SSL-Validity check
|
|
allow_redirects bool Follow redirects
|
|
max_redirects int maximum redirects, hard limit
|
|
soft_max_redirects int maximum redirects, soft limit. Record an error but don't stop the engine
|
|
raise_for_httperror bool True by default: raise an exception if the HTTP code of response is >= 300
|
|
=================== =========== ==========================================================================
|
|
|
|
|
|
Making a Response
|
|
=================
|
|
|
|
In the ``response`` function of the engine, the HTTP response (``resp``) is
|
|
parsed and a list of results is returned.
|
|
|
|
A engine can append result-items of different media-types and different
|
|
result-types to the result list. The list of the result items is render to HTML
|
|
by templates. For more details read section:
|
|
|
|
- :ref:`simple theme templates`
|
|
- :ref:`result types`
|