* Add --delete-duplicates option to delete local assets that already exist on the server, fixes#12181
* Update docs
* Fix `--delete-duplicates` implying `--delete`
* fix the test, break the english
* format
* also ran the formatter on the e2e folder :)
* early return, fewer allocations
* add back log
---------
Co-authored-by: Robin Jacobs <robin.jacobs@beeline.com>
Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>
Set default concurrency number to #CPU cores - 1
Co-authored-by: Alex <alex.tran1502@gmail.com>
Co-authored-by: Mert <101130780+mertalev@users.noreply.github.com>
* fix(docs): update the cli upload usage
The cli upload usage is missing some options compared to what is the current
output of `immich upload --help`. Update the docs accordingly.
Signed-off-by: Bence Ferdinandy <bence@ferdinandy.com>
* feat(cli): add --json-output option to upload command
Add an option that allows retrieving per-file information about the
upload process. The output includes the newFiles, duplicates and
newAssets lists, but could accommodate more information later if needed.
One use case this allows for is using --dry-run to get a list of all the
files that would be uploaded, and checking them manually before an
upload. This can be particularly useful when a curated subset of images
have already been uploaded to immich and we want to double check for
some stragglers without uploading everything to immich.
The upload command has a few lines of logging, so to get an actually
parsable json one needs to strip those lines:
immich upload --dry-run * | tail -n +4 | jq .newFiles[]
Signed-off-by: Bence Ferdinandy <bence@ferdinandy.com>
---------
Signed-off-by: Bence Ferdinandy <bence@ferdinandy.com>
* feat(cli): watch paths for auto uploading daemon
* chore: update package-lock
* test(cli): Batcher util calss
* feat(cli): expose batcher params from startWatch()
* test(cli): startWatch() for `--watch`
* refactor(cli): more reliable watcher
* feat(cli): disable progress bar on --no-progress or --watch
* fix(cli): extensions match when upload with watch
* feat(cli): basic logs without progress on upload
* feat(cli): hide progress in uploadFiles()
* refactor(cli): use promise-based setTimeout() instead of hand crafted sleep()
* refactor(cli): unexport UPLOAD_WATCH consts
* refactor(cli): rename fsWatchListener() to onFile()
* test(cli): prefix dot to mocked getSupportedMediaTypes()
* test(cli): add tests for ignored patterns/ unsupported exts
* refactor(cli): minor changes for code reviews
* feat(cli): disable onFile logs when progress bar is enabled
* feat(cli): use a queue for duplicate and upload
Using a queue to process the files makes the file duplicate detection and asset upload more stable and tolerant of network errors. If an error occurs, the whole command will not stop; the task will be retried (3 times) before logging the error and moving to the next step.
The new queue abstraction is using [fastq](https://www.npmjs.com/package/fastq) internally.
* chore(cli): queue.push return promise which resolve with task
* test(cli): add spec for uploadFiles and checkForDuplicates
Modify the handling of patterns in the `crawl` function to correctly
convert the current path to a pattern when it contains backslash on
Windows, in according to fast-glob's docs.
* refactor: asset media endpoints
* refactor: mobile upload livePhoto as separate request
* refactor: change mobile backup flow to use new asset upload endpoints
* chore: format and analyze dart code
* feat: mark motion as hidden when linked
* feat: upload video portion of live photo before image portion
* fix: incorrect assetApi calls in mobile code
* fix: download asset
---------
Co-authored-by: shenlong-tanwen <139912620+shalong-tanwen@users.noreply.github.com>
Co-authored-by: Zack Pollard <zackpollard@ymail.com>
* refactor(server): user endpoints
* fix repos
* fix unit tests
---------
Co-authored-by: Daniel Dietzler <mail@ddietzler.dev>
Co-authored-by: Alex <alex.tran1502@gmail.com>
* concurrent cli upload
* added concurrency flag, progress bar refinements
* no data property 🦀
* use lodash-es
* rebase
* linting
* typing
* album bug fixes
* dev dependency for lodash typing
* fixed not deleting assets if album isn't specified
* formatting
* fixed tests
* use `arrayContaining`
* add more checks
* assert updates existing assets
* add unicorn to eslint
* fix lint errors for cli
* fix merge
* fix album name extraction
* Update cli/src/commands/upload.command.ts
Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>
* es2k23
* use lowercase os
* return undefined album name
* fix bug in asset response dto
* auto fix issues
* fix server code style
* es2022 and formatting
* fix compilation error
* fix test
* fix config load
* fix last lint errors
* set string type
* bump ts
* start work on web
* web formatting
* Fix UUIDParamDto as UUIDParamDto
* fix library service lint
* fix web errors
* fix errors
* formatting
* wip
* lints fixed
* web can now start
* alphabetical package json
* rename error
* chore: clean up
---------
Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* feat(web): onboarding
* feat: openapi
* feat: modulization
* feat: page advancing
* Animation
* Add storage templaete settings
* sql
* more style
* Theme
* information and styling
* hide/show table
* Styling
* Update user property
* fix test
* fix test:
* fix e2e
* test
* Update web/src/lib/components/onboarding-page/onboarding-hello.svelte
Co-authored-by: bo0tzz <git@bo0tzz.me>
* naming
* use System Metadata
* better return type
* onboarding using server metadata
* revert previous changes in user entity
* sql
* test web
* fix test server
* server/web test
* more test
* consolidate color theme change logic
* consolidate save button to storage template
* merge main
* fix web
---------
Co-authored-by: bo0tzz <git@bo0tzz.me>
* fix: pgvecto.rs extension breaks typeorm schema:drop command
* fix: parse postgres bigints to javascript number types when selecting data
* feat: verify file size is the same as original asset after copying file for storage template job
* feat: allow disabling of storage template job, defaults to disabled for new instances
* fix: don't allow setting concurrency for storage template migration, can cause race conditions above 1
* feat: add checksum verification when file is copied for storage template job
* fix: extract metadata for assets that aren't visible on timeline
