The requests changed here all run outside of the network context timeout,
thereby preventing the engine's timeout from being applied (the engine's timeout
can become longer than it was configured).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
These engines override the user agent manually using `gen_useragent`, although that's already done in the online preprocessor that runs before the actual `request(query, params)` method is called. Hence, this call is duplicated.
Related:
- https://github.com/searxng/searxng/pull/4990#discussion_r2195142838
Fixes the semantic scholar engine by extracting a ui version token.
BTW: remove html tags from the content.
Author's checklist:
- they are ratelimiting very fast, if you do approx more than 2 requests per
minute, you have to wait some time again...
- they also have an official api at api.semanticscholar.org, but it's ratelimits
are even harder
Closes: https://github.com/searxng/searxng/issues/4685
