46 Commits

Author SHA1 Message Date
Jan
4ea15a5b0c
fix(server): check if sidecarPath exists (#6293)
* check if sidecarPath exists

* Revert "check if sidecarPath exists"

This reverts commit 954a1097b870585afee34974d466e51c5172fed9.

* sidecar sync remove dead sidecarPaths and discover new ones

* tests and minor cleanup

* chore: linting

---------

Co-authored-by: Daniel Dietzler <mail@ddietzler.dev>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-02-07 17:30:38 +00:00
Jonathan Jogenfors
f44fa45aa0
chore(server,cli,web): housekeeping and stricter code style (#6751)
* add unicorn to eslint

* fix lint errors for cli

* fix merge

* fix album name extraction

* Update cli/src/commands/upload.command.ts

Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>

* es2k23

* use lowercase os

* return undefined album name

* fix bug in asset response dto

* auto fix issues

* fix server code style

* es2022 and formatting

* fix compilation error

* fix test

* fix config load

* fix last lint errors

* set string type

* bump ts

* start work on web

* web formatting

* Fix UUIDParamDto as UUIDParamDto

* fix library service lint

* fix web errors

* fix errors

* formatting

* wip

* lints fixed

* web can now start

* alphabetical package json

* rename error

* chore: clean up

---------

Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-02-01 22:18:00 -05:00
Jonathan Jogenfors
068e703e88
feat(server): Automatic watching of library folders (#6192)
* feat: initial watch support

* allow offline files

* chore: ignore query errors when resetting e2e db

* revert db query

* add savepoint

* guard the user query

* chore: openapi and db migration

* wip

* support multiple libraries

* fix tests

* wip

* can now cleanup chokidar watchers

* fix unit tests

* add library watch queue

* add missing init from merge

* wip

* can now filter file extensions

* remove watch api from non job client

* Fix e2e test

* watch library with updated import path and exclusion pattern

* add library watch frontend ui

* case sensitive watching extensions

* can auto watch libraries

* move watcher e2e tests to separate file

* don't watch libraries from a queue

* use event emitters

* shorten e2e test timeout

* refactor chokidar code to filesystem provider

* expose chokidar parameters to config file

* fix storage mock

* set default config for library watching

* add fs provider mocks

* cleanup

* add more unit tests for watcher

* chore: fix format + sql

* add more tests

* move unwatch feature back to library service

* add file event unit tests

* chore: formatting

* add documentation

* fix e2e tests

* chore: fix e2e tests

* fix library updating

* test cleanup

* fix typo

* cleanup

* fixing as per pr comments

* reduce library watch config file

* update storage config and mocks

* move negative event tests to unit tests

* fix library watcher e2e

* make watch configuration global

* remove the feature flag

* refactor watcher teardown

* fix microservices init

* centralize asset scan job queue

* improve docs

* add more tests

* chore: open api

* initialize app service

* fix docs

* fix library watch feature flag

* Update docs/docs/features/libraries.md

Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>

* fix: import right app service

* don't be truthy

* fix test speling

* stricter library update tests

* move fs watcher mock to external file

* subscribe to config changes

* docker does not need polling

* make library watch() private

* feat: add configuration ui

---------

Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-01-31 08:15:54 +00:00
Mert
ae7f174948
refactor: rename clip -> smart search (#6713) 2024-01-29 09:51:22 -05:00
Mert
68f52818ae
feat(server): separate face clustering job (#5598)
* separate facial clustering job

* update api

* fixed some tests

* invert clustering

* hdbscan

* update api

* remove commented code

* wip dbscan

* cleanup

removed cluster endpoint

remove commented code

* fixes

updated tests

minor fixes and formatting

fixed queuing

refinements

* scale search range based on library size

* defer non-core faces

* optimizations

removed unused query option

* assign faces individually for correctness

fixed unit tests

remove unused method

* don't select face embedding

update sql

linting

fixed ml typing

* updated job mock

* paginate people query

* select face embeddings because typeorm

* fix setting face detection concurrency

* update sql

formatting

linting

* simplify logic

remove unused imports

* more specific delete signature

* more accurate typing for face stubs

* add migration

formatting

* chore: better typing

* don't select embedding by default

remove unused import

* updated sql

* use normal try/catch

* stricter concurrency typing and enforcement

* update api

* update job concurrency panel to show disabled queues

formatting

* check jobId in queueAll

fix tests

* remove outdated comment

* better facial recognition icon

* wording

wording

formatting

* fixed tests

* fix

* formatting & sql

* try to fix sql check

* more detailed description

* update sql

* formatting

* wording

* update `minFaces` description

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-01-18 00:08:48 -05:00
cfitzw
deb1f970a8
feat(server, web): quotas (#4471)
* feat: quotas

* chore: open api

* chore: update status box and upload error message

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-12 19:43:36 -05:00
Jason Rasmussen
12dc7c48c9
refactor(server): domain and infra modules (#6301) 2024-01-09 17:07:01 -05:00
Jason Rasmussen
03eb5ffc5c
refactor(server): simplify config init process (#5702) 2024-01-01 13:16:44 -05:00
Mert
cc2dc12f6c
fix(server): run migrations after database checks (#5832)
* run migrations after checks

* optional migrations

* only run checks in server and e2e

* re-add migrations for microservices

* refactor

* move e2e init

* remove assert from migration

* update providers

* update microservices app service

* fixed logging

* refactored version check, added unit tests

* more version tests

* don't use mocks for sut

* refactor tests

* suggest image only if postgres is 14, 15 or 16

* review suggestions

* fixed regexp escape

* fix typing

* update migration
2023-12-21 10:06:26 -06:00
Mert
092a23fd7f
feat(server,ml): remove image tagging (#5903)
* remove image tagging

* updated lock

* fixed tests, improved logging

* be nice

* fixed tests
2023-12-20 20:47:56 -05:00
Mert
de1514a441
chore(server): startup check for pgvecto.rs (#5815)
* startup check for pgvecto.rs

* prefilter after assertion

* formatting

* add assert to migration

* more specific import

* use runner
2023-12-18 10:38:25 -06:00
Jason Rasmussen
9768931275
feat(web,server)!: runtime log level (#5672)
* feat: change log level at runtime

* chore: open api

* chore: prefer env over runtime

* chore: remove default env value
2023-12-14 16:55:40 +00:00
Jason Rasmussen
b34abf25f0
feat(server): server-side events (#5669) 2023-12-13 12:23:51 -05:00
Jason Rasmussen
92b4284b5a
feat(server): use postgres-adapter for websockets (#5569)
* feat(server): use postgres-adapter for websockets

* refactor: create attachment table via migration
2023-12-08 20:38:25 -05:00
Jason Rasmussen
1e99ba8167
feat: use pgvecto.rs (#3605) 2023-12-08 11:15:46 -05:00
YFrendo
644e52b153
feat: Edit metadata (#5066)
* chore: rebase and clean-up

* feat: sync description, add e2e tests

* feat: simplify web code

* chore: unit tests

* fix: linting

* Bug fix with the arrows key

* timezone typeahead filter

timezone typeahead filter

* small stlying

* format fix

* Bug fix in the map selection

Bug fix in the map selection

* Websocket basic

Websocket basic

* Update metadata visualisation through the websocket

* Update timeline

* fix merge

* fix web

* fix web

* maplibre system

* format fix

* format fix

* refactor: clean up

* Fix small bug in the hour/timezone

* Don't diplay modify for readOnly asset

* Add log in case of failure

* Formater + try/catch error

* Remove everything related to websocket

* Revert "Remove everything related to websocket"

This reverts commit 14bcb9e1e4398e8211adfe6c14348ef8f3f5fce4.

* remove notification

* fix test

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-11-30 03:52:28 +00:00
Zack Pollard
698226634e
feat: postgres reverse geocoding (#5301)
* feat: add system metadata repository for storing key values for internal usage

* feat: add database entities for geodata

* feat: move reverse geocoding from local-reverse-geocoder to postgresql

* infra: disable synchronization for geodata_places table until typeorm supports earth column

* feat: remove cities override config as we will default all instances to cities500 now

* test: e2e tests don't clear geodata tables on reset
2023-11-25 18:53:30 +00:00
martin
1aae29a0b8
refactor(server, web)!: store latest immich version available on the server (#3565)
* refactor: store latest immich version available on the server

* don't store admins acknowledgement

* merge main

* fix: api

* feat: custom interval

* pr feedback

* remove unused code

* update environment-variables

* pr feedback

* ci: fix server tests

* fix: dart number

* pr feedback

* remove proxy

* pr feedback

* feat: make stringToVersion more flexible

* feat(web): disable check

* feat: working version

* remove env

* fix: check if interval exists when updating the interval

* feat: show last check

* fix: tests

* fix: remove availableVersion when updated

* fix merge

* fix: web

* fix e2e tests

* merge main

* merge main

* pr feedback

* pr feedback

* fix: tests

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* fix: migration

* regenerate api

* fix: typo

* fix: compare versions

* pr feedback

* fix

* pr feedback

* fix: checkIntervalTime on startup

* refactor: websockets and interval logic

* chore: open api

* chore: remove unused code

* fix: use interval instead of cron

* mobile: handle WS event data as json object

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: shalong-tanwen <139912620+shalong-tanwen@users.noreply.github.com>
2023-10-24 11:05:42 -04:00
Jason Rasmussen
4b59f83288
refactor: e2e tests (#4536) 2023-10-18 17:02:42 -05:00
Jonathan Jogenfors
f0bb50b61a
fix(server,cli): don't float promises (#4433)
* fix: don't allow floating promises

* fix: await all promises

* fix: download archives

* fix cli tests

* fix: skip web
2023-10-13 01:22:40 -04:00
Jonathan Jogenfors
56eb7bf0fc
fix(server): improve library scan queuing performance (#4418)
* fix: inline mark asset as offline

* fix: improve log message

* chore: lint

* fix: offline asset algorithm

* fix: use set comparison to check what to import

* fix: only mark new offline files as offline

* fix: compare the correct array

* fix: set default library concurrency to 5

* fix: remove one db call when scanning new files

* chore: remove unused import

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-10-10 18:59:13 -04:00
shenlong
4a8887f37b
feat(server): trash asset (#4015)
* refactor(server): delete assets endpoint

* fix: formatting

* chore: cleanup

* chore: open api

* chore(mobile): replace DeleteAssetDTO with BulkIdsDTOs

* feat: trash an asset

* chore(server): formatting

* chore: open api

* chore: wording

* chore: open-api

* feat(server): add withDeleted to getAssets queries

* WIP: mobile-recycle-bin

* feat(server): recycle-bin to system config

* feat(web): use recycle-bin system config

* chore(server): domain assetcore removed

* chore(server): rename recycle-bin to trash

* chore(web): rename recycle-bin to trash

* chore(server): always send soft deleted assets for getAllByUserId

* chore(web): formatting

* feat(server): permanent delete assets older than trashed period

* feat(web): trash empty placeholder image

* feat(server): empty trash

* feat(web): empty trash

* WIP: mobile-recycle-bin

* refactor(server): empty / restore trash to separate endpoint

* test(server): handle failures

* test(server): fix e2e server-info test

* test(server): deletion test refactor

* feat(mobile): use map settings from server-config to enable / disable map

* feat(mobile): trash asset

* fix(server): operations on assets in trash

* feat(web): show trash statistics

* fix(web): handle trash enabled

* fix(mobile): restore updates from trash

* fix(server): ignore trashed assets for person

* fix(server): add / remove search index when trashed / restored

* chore(web): format

* fix(server): asset service test

* fix(server): include trashed assts for duplicates from uploads

* feat(mobile): no dialog for trash, always dialog for permanent delete

* refactor(mobile): use isar where instead of dart filter

* refactor(mobile): asset provide - handle deletes in single db txn

* chore(mobile): review changes

* feat(web): confirmation before empty trash

* server: review changes

* fix(server): handle library changes

* fix: filter external assets from getting trashed / deleted

* fix(server): empty-bin

* feat: broadcast config update events through ws

* change order of trash button on mobile

* styling

* fix(mobile): do not show trashed toast for local only assets

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-10-06 02:01:14 -05:00
Daniel Dietzler
98db9331d8
fix(server): delete face thumbnails when merging people (#4310)
* new job for person deletion, including face thumbnail deletion

* fix tests, delete files directly instead queueing jobs
2023-10-02 21:15:11 -04:00
Daniel Dietzler
0a22e64799
refactor(server): merge facial-recognition and person (#4237)
* move facial recognition service into person service

* merge face repository and person repository

* fix imports
2023-09-27 16:46:46 -04:00
Daniel Dietzler
3a44e8f8d3
refactor(server): Move metadata extraction to domain (#4243)
* use storageRepository in metadata extraction

* move metadata extraction processor to domain

* cleanup infra/domain

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-09-27 18:44:51 +00:00
Daniel Dietzler
9bada51d56
feat(server, web)!: Move reverse geocoding settings to the UI (#4222)
* feat: reverse geocoding settings

* chore: open api

* re-init geocoder if precision has been updated

* update docs

* chore: update verbiage

* fix: re-init logic

* fix: reset to default

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-09-26 14:03:57 +07:00
Jason Rasmussen
7bc6e9ef64
refactor(server): person thumbnail job (#4233)
* refactor(server): person thumbnail job

* fix(server): set feature photo
2023-09-26 14:03:22 +07:00
Mert
56cf9464af
fix(server): use srgb pipeline for srgb images (#4101)
* added color-related exif fields

* remove metadata check, conditional pipe colorspace

* check exif metadata for srgb

* added migration

* updated e2e fixture

* uncased srgb check, search substrings

* extracted exif logic into separate function

* handle images with no bit depth or color metadata

* added unit tests
2023-09-25 19:18:47 -04:00
Daniel Dietzler
3053cbd4c8
chore(server): Store generated files (thumbnails, encoded video) in subdirectories (#4112)
* save thumbnails in subdirectories

* migration job, migrate assets and face thumbnails

* fix tests

* directory depth of two instead of three

* cleanup empty dirs after migration

* clean up empty dirs after migration, migrate people without assetId

* add job card for new migration job

* fix removeEmptyDirs race condition because of missing await

* cleanup empty directories after asset deletion

* move ensurePath to storage core

* rename jobs

* remove unnecessary property of IEntityJob

* use updated person getById, minor refactoring

* ensure that directory cleanup doesn't interfere with migration

* better description for job in ui

* fix remove directories when migration is done

* cleanup empty folders at start of migration

* fix: actually persist concurrency setting

* add comment explaining regex

* chore: cleanup

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-09-25 11:07:21 -04:00
Jason Rasmussen
07069c3b1e
fix(server): android live motion (#4218) 2023-09-25 09:42:23 -04:00
Jonathan Jogenfors
acdc66413c
feat(server,web): libraries (#3124)
* feat: libraries

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex <alex.tran1502@gmail.com>
2023-09-20 13:16:33 +02:00
Thomas
49ef86173f
fix(server): use exiftool decoded values and unify metadata extraction (#2908)
* refactor(server): metadata extraction

* chore: upgrade exiftool

* chore: remove log

* fix: other rotation cases

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-09-18 09:06:03 -04:00
Jason Rasmussen
74d34b4f6c
refactor(server): android motion photos (#3711) 2023-09-13 10:46:37 +07:00
Jason Rasmussen
454737ca79
refactor(server): update asset endpoint (#3973)
* refactor(server): update asset

* chore: open api
2023-09-04 22:25:31 -04:00
Alex
e02817362c
fix(server): initialization search service (#3879) 2023-08-26 13:29:34 -05:00
Fynn Petersen-Frey
cf9e04c8ec
feat(server): asset entity audit (#3824)
* feat(server): audit log

* feedback

* Insert to database

* migration

* test

* controller/repository/service

* test

* module

* feat(server): implement audit endpoint

* directly return changed assets

* add daily cleanup of audit table

* fix tests

* review feedback

* ci

* refactor(server): audit implementation

* chore: open api

---------

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
Co-authored-by: Fynn Petersen-Frey <zoodyy@users.noreply.github.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-08-24 15:28:50 -04:00
Jason Rasmussen
4762fd83d4
fix(server): link live photos after metadata extraction finishes (#3702)
* fix(server): link live photos after metadata extraction finishes

* chore: fix test

---------

Co-authored-by: Alex <alex.tran1502@gmail.com>
2023-08-15 20:34:57 -05:00
Russell Tan
f1b8a7ab54
fix(server): Does not assign lat/lon if they are at 0,0 #2991 (#3669)
* fix(server): Does not assign lat/lon if they are at 0,0 #2991

* Adds migration file to fix null island rows

* Removed down migration

* Leave empty down function
2023-08-14 20:37:17 -05:00
Dmitry Brazhenko
e071b82e8a
feat (web/server) 360 degrees Web panoramas [attempt 2] (#3412)
* commit 1 (isPanorama: boolean)

* working solution for projectiontypeenum

* fix

* format fix

* fix

* fix

* fix

* fix

* enum projectiontype

* working solution with exif

* fix

* reverted >

* fix format

* reverted auto-magic api.ts prettification

* fix

* reverted api.ts autogenerated

* api ts regenerated

* Update web/src/lib/components/assets/thumbnail/thumbnail.svelte

Co-authored-by: Sergey Kondrikov <sergey.kondrikov@gmail.com>

* Update web/src/lib/components/asset-viewer/asset-viewer.svelte

Co-authored-by: Sergey Kondrikov <sergey.kondrikov@gmail.com>

* exifProjectionType

* Update server/src/microservices/processors/metadata-extraction.processor.ts

Co-authored-by: Sergey Kondrikov <sergey.kondrikov@gmail.com>

* projectionType?: string = ProjectionType.NONE;

* not null

* projectionType!: ProjectionType;

* opeapi generator fix

* fixes

* fix

* fix

* generate api

* asset.exifInifo?.projectionType

* Update server/src/domain/asset/response-dto/exif-response.dto.ts

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>

* Update server/src/microservices/processors/metadata-extraction.processor.ts

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>

* enum -> varchar;projectiontypeenum->projectiontype

* asset-viewer fixed prettiffier

* @Column({}) single line

* enum | string

* make api

* enum | string

* enum | str fix

* fix

* chore: use string instead of enum

* chore: open api

* fix: checks

---------

Co-authored-by: Sergey Kondrikov <sergey.kondrikov@gmail.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-07-27 23:29:09 -05:00
Jason Rasmussen
f3edf43158
chore: log listen address (#3428) 2023-07-26 18:29:35 +00:00
Simon Shields
4f59e6c7ab
feat(server): Google Pixel motion photos (#3175)
* feat(server): Google Pixel motion photos

Add support for motion photos taken on Pixel phones. They have the exif
property 'MotionPhoto' set to 1, and an embedded mp4 file appended to
the JPEG file.

The implementation works like this:

- on metadata extraction, if a live photo is detected, examine the
  metadata to determine where in the file the embedded MP4 is.
- extract this MP4 and write it next to the JPEG.
- link it using the existing mechanism for live photos.

There is a "MotionPhotoPresentationTimestampUs" exif property, which we
don't do anything with - I imagine that it refers to the timepoint in
the video that the photo was taken at, but it probably warrants more
investigation.

* fix format

* fix test

---------

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-07-21 22:52:59 -05:00
oddlama
c415ee82d1
chore: adjust loglevel of reverse geocoding intializer to LOG (#3303) 2023-07-16 21:57:20 -05:00
Mert
8349a28ed8
refactor(server): modularize getFfmpegOptions (#3138)
* refactored `getFfmpegOptions`

refactor transcoding, make separate service

* fixed enum casing

* use `Logger` instead of `console.log`

* review suggestions

* use enum for `getHandler`

* fixed formatting

* Update server/src/domain/media/media.util.ts

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>

* Update server/src/domain/media/media.util.ts

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>

* More specific imports, renamed codec classes

* simplified code

* removed unused import

* added tests

* added base implementation for bitrate and threads

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-07-08 21:43:11 -05:00
Covalent
3e804f16df
feat(web,server): add thumbhash support (#2649)
* add thumbhash: server generation and web impl

* move logic to infra & use byta in db

* remove unnecesary logs

* update generated API and simplify thumbhash gen

* fix check errors

* removed unnecessary library and css tag

* style edits

* syntax mistake

* update server test, change thumbhash job name

* fix tests

* Update server/src/domain/asset/response-dto/asset-response.dto.ts

Co-authored-by: Thomas <9749173+uhthomas@users.noreply.github.com>

* add unit test, change migration date

* change to official thumbhash impl

* update call method to not use eval

* "generate missing" looks for thumbhash

* improve queue & improve syntax

* update syntax again

* update tests

* fix thumbhash generation

* consolidate queueing to avoid duplication

* cover all types of incorrect thumbnail cases

* split out jest tasks

* put back thumbnail duration loading for images without thumbhash

* Remove stray package.json

---------

Co-authored-by: Luke McCarthy <mail@lukehmcc.com>
Co-authored-by: Thomas <9749173+uhthomas@users.noreply.github.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-06-17 22:22:31 -05:00
Jason Rasmussen
bff6914a73
chore(server): organize imports (#2779)
* feat: lint rule for organize imports

* chore: organize imports
2023-06-16 19:54:17 +00:00
Jason Rasmussen
8ebac41318
refactor(server)*: tsconfigs (#2689)
* refactor(server): tsconfigs

* chore: dummy commit

* fix: start.sh

* chore: restore original entry scripts
2023-06-08 10:01:07 -05:00