* feat: add OCR functionality and related configurations
* chore: update labeler configuration for machine learning files
* feat(i18n): enhance OCR model descriptions and add orientation classification and unwarping features
* chore: update Dockerfile to include ccache for improved build performance
* feat(ocr): enhance OCR model configuration with orientation classification and unwarping options, update PaddleOCR integration, and improve response structure
* refactor(ocr): remove OCR_CLEANUP job from enum and type definitions
* refactor(ocr): remove obsolete OCR entity and migration files, and update asset job status and schema to accommodate new OCR table structure
* refactor(ocr): update OCR schema and response structure to use individual coordinates instead of bounding box, and adjust related service and repository files
* feat: enhance OCR configuration and functionality
- Updated OCR settings to include minimum detection box score, minimum detection score, and minimum recognition score.
- Refactored PaddleOCRecognizer to utilize new scoring parameters.
- Introduced new database tables for asset OCR data and search functionality.
- Modified related services and repositories to support the new OCR features.
- Updated translations for improved clarity in settings UI.
* sql changes
* use rapidocr
* change dto
* update web
* update lock
* update api
* store positions as normalized floats
* match column order in db
* update admin ui settings descriptions
fix max resolution key
set min threshold to 0.1
fix bind
* apply config correctly, adjust defaults
* unnecessary model type
* unnecessary sources
* fix(ocr): switch RapidOCR lang type from LangDet to LangRec
* fix(ocr): expose lang_type (LangRec.CH) and font_path on OcrOptions for RapidOCR
* fix(ocr): make OCR text search case- and accent-insensitive using ILIKE + unaccent
* fix(ocr): add OCR search fields
* fix: Add OCR database migration and update ML prediction logic.
* trigrams are already case insensitive
* add tests
* format
* update migrations
* wrong uuid function
* linting
* maybe fix medium tests
* formatting
* fix weblate check
* openapi
* sql
* minor fixes
* maybe fix medium tests part 2
* passing medium tests
* format web
* readd sql
* format dart
* disabled in e2e
* chore: translation ordering
---------
Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
* duplicate detection job, entity, config
* queueing
* job panel, update api
* use embedding in db instead of fetching
* disable concurrency
* only queue visible assets
* handle multiple duplicateIds
* update concurrent queue check
* add provider
* add web placeholder, server endpoint, migration, various fixes
* update sql
* select embedding by default
* rename variable
* simplify
* remove separate entity, handle re-running with different threshold, set default back to 0.02
* fix tests
* add tests
* add index to entity
* formatting
* update asset mock
* fix `upsertJobStatus` signature
* update sql
* formatting
* default to 0.03
* optimize clustering
* use asset's `duplicateId` if present
* update sql
* update tests
* expose admin setting
* refactor
* formatting
* skip if ml is disabled
* debug trash e2e
* remove from web
* remove from sidebar
* test if ml is disabled
* update sql
* separate duplicate detection from clip in config, disable by default for now
* fix doc
* lower minimum `maxDistance`
* update api
* Add and Use Duplicate Detection Feature Flag (#9364)
* Add Duplicate Detection Flag
* Use Duplicate Detection Flag
* Attempt Fixes for Failing Checks
* lower minimum `maxDistance`
* fix tests
---------
Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>
* chore: fixes and additions after rebase
* chore: update api (remove new Role enum)
* fix: left join smart search so getAll works without machine learning
* test: trash e2e go back to checking length of assets is zero
* chore: regen api after rebase
* test: fix tests after rebase
* redundant join
---------
Co-authored-by: Nicholas Flamy <30300649+NicholasFlamy@users.noreply.github.com>
Co-authored-by: Zack Pollard <zackpollard@ymail.com>
Co-authored-by: Zack Pollard <zack@futo.org>
* consolidated endpoints, added live configuration
* added ml settings to server
* added settings dashboard
* updated deps, fixed typos
* simplified modelconfig
updated tests
* Added ml setting accordion for admin page
updated tests
* merge `clipText` and `clipVision`
* added face distance setting
clarified setting
* add clip mode in request, dropdown for face models
* polished ml settings
updated descriptions
* update clip field on error
* removed unused import
* add description for image classification threshold
* pin safetensors for arm wheel
updated poetry lock
* moved dto
* set model type only in ml repository
* revert form-data package install
use fetch instead of axios
* added slotted description with link
updated facial recognition description
clarified effect of disabling tasks
* validation before model load
* removed unnecessary getconfig call
* added migration
* updated api
updated api
updated api
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
