Cutlery/searxng
1
0
Fork 0
mirror of https://github.com/searxng/searxng.git synced 2025-11-20 05:23:09 -05:00
Code Issues Packages Projects Releases Wiki Activity
9,042 Commits 6 Branches 0 Tags
Commit Graph

542 Commits

Author SHA1 Message Date
Markus Heiser
9c2b8f2f93 [data] update searx.data - update_ahmia_blacklist.py 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-10-30 07:40:20 +01:00
github-actions[bot]
dce383881d
[data] update searx.data - update_firefox_version.py (#5388) 2025-10-29 07:15:11 +01:00
github-actions[bot]
1ebedcbc17
[data] update searx.data - update_wikidata_units.py (#5389) 2025-10-29 07:14:31 +01:00
github-actions[bot]
5d99877d8d
[data] update searx.data - update_currencies.py (#5390) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-10-29 07:13:53 +01:00
github-actions[bot]
adc1a2a1ea
[data] update searx.data - update_engine_descriptions.py (#5391) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-10-29 07:13:19 +01:00
Markus Heiser
33e798b01b
[fix] TrackerPatternsDB.clean_url: don't delete query argument from new_url (#5339) 
The query argument for URLs like:

- 'http://example.org?q='       --> query_str is 'q='
- 'http://example.org?/foo/bar' --> query_str is 'foo/bar'

is a *simple string* and not a key/value dict.  This string may only be removed
from the URL if one of the patterns matches.

BTW get_pretty_url(): keep such a *simple string* in the path element.

Closes: https://github.com/searxng/searxng/issues/5299

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-10-20 11:20:33 +02:00
searxng-bot
0f3ef5da59 [data] update searx.data - update_engine_traits.py 2025-09-30 16:48:59 +02:00
github-actions[bot]
77fd3ee534
[data] update searx.data - update_currencies.py (#5261) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-09-29 07:37:30 +02:00
github-actions[bot]
b8e60542d9
[data] update searx.data - update_engine_descriptions.py (#5262) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-09-29 07:36:51 +02:00
github-actions[bot]
5513549adc
[data] update searx.data - update_wikidata_units.py (#5259) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-09-29 07:35:10 +02:00
github-actions[bot]
cd2ea7c5f5
[data] update searx.data - update_ahmia_blacklist.py (#5257) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-09-29 07:34:38 +02:00
searxng-bot
38b80133f7 [data] update searx.data - update_firefox_version.py 2025-09-29 07:33:19 +02:00
Markus Heiser
18a58943cc [mod] ExpireCacheSQLite - implement .setmany() for bulk loading 
[1] https://github.com/searxng/searxng/issues/5223#issuecomment-3328597147

Suggested-by: Ivan G <igabaldon@inetol.net> [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-09-28 07:32:41 +02:00
Markus Heiser
62b0b3f697 [mod] debug-log the bootload of the CURRENCIES caches (CurrenciesDB) 
The time for the bootload is measured and recorded.  To have an eye on the
bootload time is motivated by: In [1] it can be seen that the bootload of the
CURRENCIES cache takes about 2/3 of the time required to initialize SearXNG::

    6.068 <module>  searx/webapp.py:1
    └─ 6.068 wrapper  pyinstrument/context_manager.py:52
       ├─ 5.538 init  searx/webapp.py:1373
       │  ├─ 4.822 initialize  searx/search/__init__.py:34
       │  │  ├─ 4.631 ProcessorMap.init  searx/search/processors/__init__.py:47
       │  │  │  ├─ 4.607 OnlineCurrencyProcessor.initialize  searx/search/processors/online_currency.py:55
       │  │  │  │  └─ 4.607 CurrenciesDB.init  searx/data/currencies.py:25
       │  │  │  │     ├─ 4.601 CurrenciesDB.load  searx/data/currencies.py:34
       │  │  │  │     │  ├─ 4.572 ExpireCacheSQLite.set  searx/cache.py:334

In the example, the CurrenciesDB.init call takes 4.6 seconds... on my laptop,
CurrenciesDB.init takes only 0.7 seconds.  The absolute numerical values depend
on external conditions, where I already find 4-5 seconds very long. Test::

    $ rm /tmp/sxng_cache_*.db*
    $ make run 2>&1 | grep "searx.data.CURRENCIES"
    DEBUG   searx.data : init searx.data.CURRENCIES
    DEBUG   searx.data : init searx.data.CURRENCIES added 9089 items in 0.7623255252838135 sec.

[1] https://github.com/searxng/searxng/issues/5223#issuecomment-3323083411

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-09-28 07:32:41 +02:00
Markus Heiser
81cbe0befe
[upd] pypi: Bump black from 24.3.0 to 25.9.0 (#5251) 
In 25.1.0 [2] an old bug has been fixed: "Docstring formatting does not apply to
module docstrings" [3].

[1] https://github.com/psf/black/blob/main/CHANGES.md#2590
[2] https://github.com/psf/black/blob/main/CHANGES.md#2510
[3] https://github.com/psf/black/issues/4094

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-09-26 12:35:57 +02:00
Markus Heiser
8f8343dc0d [mod] addition of various type hints / engine processors 
Continuation of #5147 .. typification of the engine processors.

BTW:

- removed obsolete engine property https_support
- fixed & improved currency_convert
- engine instances can now implement a engine.setup method

[#5147] https://github.com/searxng/searxng/pull/5147

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-09-18 19:40:03 +02:00
Markus Heiser
f24d85bc4b [mod] drop: from __future__ import annotations 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-09-03 13:37:36 +02:00
Markus Heiser
57b9673efb [mod] addition of various type hints / tbc 
- pyright configuration [1]_
- stub files: types-lxml [2]_
- addition of various type hints
- enable use of new type system features on older Python versions [3]_
- ``.tool-versions`` - set python to lowest version we support (3.10.18) [4]_:
  Older versions typically lack some typing features found in newer Python
  versions.  Therefore, for local type checking (before commit), it is necessary
  to use the older Python interpreter.

.. [1] https://docs.basedpyright.com/v1.20.0/configuration/config-files/
.. [2] https://pypi.org/project/types-lxml/
.. [3] https://typing-extensions.readthedocs.io/en/latest/#
.. [4] https://mise.jdx.dev/configuration.html#tool-versions

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Format: reST
 2025-09-03 13:37:36 +02:00
github-actions[bot]
fde4fac3ca
[data] update searx.data - update_external_bangs.py (#5158) 2025-08-29 07:07:46 +02:00
github-actions[bot]
a9263fd063
[data] update searx.data - update_wikidata_units.py (#5159) 2025-08-29 07:07:00 +02:00
github-actions[bot]
146cefe7fa
[data] update searx.data - update_firefox_version.py (#5160) 2025-08-29 07:05:08 +02:00
github-actions[bot]
2a7d26b714
[data] update searx.data - update_ahmia_blacklist.py (#5161) 2025-08-29 07:04:41 +02:00
github-actions[bot]
480914d736
[data] update searx.data - update_currencies.py (#5162) 2025-08-29 07:04:12 +02:00
github-actions[bot]
edc50768cc
[data] update searx.data - update_engine_traits.py (#5163) 2025-08-29 07:03:29 +02:00
github-actions[bot]
a519a13bfa
[data] update searx.data - update_engine_descriptions.py (#5164) 2025-08-29 07:02:31 +02:00
Markus Heiser
664aab0ec9
[fix] CI task "update_engine_traits.py" fails (#5069) 
* [fix] CI task "update_engine_traits.py" fails

To catch all problems with an HTTP request, the more general class
``httpx.HTTPError`` must be caught, for your test use::

    $ ./manage dev.env
    $ python ./searxng_extra/update/update_engine_traits.py

Closes: https://github.com/searxng/searxng/issues/5068

* [data] update searx.data - update_engine_traits.py
 2025-08-01 12:08:27 +02:00
github-actions[bot]
40b78ad06c
[data] update searx.data - update_wikidata_units.py (#5062) 2025-07-29 07:26:01 +02:00
github-actions[bot]
db83a39544
[data] update searx.data - update_external_bangs.py (#5061) 2025-07-29 07:25:26 +02:00
github-actions[bot]
bb3bea829d
[data] update searx.data - update_ahmia_blacklist.py (#5064) 2025-07-29 07:24:09 +02:00
github-actions[bot]
dc9ad0a493
[data] update searx.data - update_currencies.py (#5065) 2025-07-29 07:23:38 +02:00
github-actions[bot]
5db7b70dc7
[data] update searx.data - update_engine_descriptions.py (#5066) 2025-07-29 07:22:58 +02:00
github-actions[bot]
2ad35421d7
[data] update searx.data - update_firefox_version.py (#5063) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-07-29 07:22:21 +02:00
mggh0139
54a2b553f4
[fix] tracker pattern: let startup continue if url fetch fails (#5055) 
Use Python exception to prevent startup crash in case of fetch ClearURL
failure. Also add some logs.

Closes: https://github.com/searxng/searxng/issues/5054
 2025-07-28 07:03:01 +02:00
Bnyro
5cbf422621 [fix] tracker url remover + external bangs: use standard network config 
Using plain `httpx` directly doesn't use SearXNG's additional network config, including proxies, http2 config, ...

Related issues:
- https://github.com/searxng/searxng/issues/5027
 2025-07-22 10:25:33 +02:00
Markus Heiser
2fe8540903
[fix] prevent multiple, parallel initializations of tables in the cache DB (#4991) 
Depending on the respective runtime behavior, it could happen that the initial
loading of the DB tables in the cache was performed multiple times and in
parallel. The concurrent accesses then led to the `sqlite3.OperationalError:
database is locked` exception as in #4951.

Since this problem depends significantly on the runtimes (e.g., how long it
takes to retrieve the content for a table), this error could not be observed in
all installations.

Closes: https://github.com/searxng/searxng/issues/4951

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-07-09 17:32:10 +02:00
github-actions[bot]
60be0f453e
[data] update searx.data - update_firefox_version.py (#4954) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:17:46 +02:00
github-actions[bot]
4fd3217786
[data] update searx.data - update_external_bangs.py (#4955) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:17:06 +02:00
github-actions[bot]
a8cc4ad877
[data] update searx.data - update_wikidata_units.py (#4956) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:16:07 +02:00
github-actions[bot]
c475508a1b
[data] update searx.data - update_ahmia_blacklist.py (#4957) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:15:38 +02:00
github-actions[bot]
2c0bda729a
[data] update searx.data - update_currencies.py (#4958) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:15:05 +02:00
github-actions[bot]
8de4119179
[data] update searx.data - update_engine_traits.py (#4959) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:14:42 +02:00
github-actions[bot]
6df862a4c7
[data] update searx.data - update_engine_descriptions.py (#4960) 
Co-authored-by: searxng-bot <searxng-bot@users.noreply.github.com>
 2025-06-29 08:14:18 +02:00
Markus Heiser
2dd4f7b972 [mod] data: implement a simple tracker URL (SQL) database 
On demand, the tracker data is loaded directly into the cache, so that the
maintenance of this data via PRs is no longer necessary.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2025-06-23 22:12:18 +02:00
Bnyro
2001efbbab [data] update searx.data - add tracker_patterns.json 2025-06-23 22:12:18 +02:00
Bnyro
8f7eee2473 [feat] tracker url plugin: use ClearURL tracking param list 2025-06-23 22:12:18 +02:00
Markus Heiser
913dfbde3c [fix] plugins: currency processor bug 
Fix C&P typo from  https://github.com/searxng/searxng/pull/4836

Closes: https://github.com/searxng/searxng/issues/4861
 2025-05-29 07:37:59 +02:00
searxng-bot
7448a18efa [data] update searx.data - update_ahmia_blacklist.py 2025-05-29 07:11:55 +02:00
searxng-bot
23396f5a1d [data] update searx.data - update_firefox_version.py 2025-05-29 07:10:22 +02:00
searxng-bot
dbe6a6f4fa [data] update searx.data - update_wikidata_units.py 2025-05-29 07:08:32 +02:00
searxng-bot
b2727e1be9 [data] update searx.data - update_currencies.py 2025-05-29 07:07:55 +02:00