diff --git a/docs/docs/features/libraries.md b/docs/docs/features/libraries.md index cdea1a11a51e1..17555469543c8 100644 --- a/docs/docs/features/libraries.md +++ b/docs/docs/features/libraries.md @@ -1,18 +1,14 @@ -# Libraries +# External Libraries -## Overview +External libraries track assets stored in the filesystem outside of Immich. When the external library is scanned, Immich will load videos and photos from disk and create the corresponding assets. These assets will then be shown in the main timeline, and they will look and behave like any other asset, including viewing on the map, adding to albums, etc. Later, if a file is modified outside of Immich, you need to scan the library for the changes to show up. -Immich supports the creation of libraries which is a top-level asset container. Currently, there are two types of libraries: traditional upload libraries that can sync with a mobile device, and external libraries, that keeps up to date with files on disk. Libraries are different from albums in that an asset can belong to multiple albums but only one library, and deleting a library deletes all assets contained within. As of August 2023, this is a new feature and libraries have a lot of potential for future development beyond what is documented here. This document attempts to describe the current state of libraries. +If an external asset is deleted from disk, Immich will move it to trash on rescan. To restore the asset, you need to restore the original file. After 30 days the file will be removed from trash, and any changes to metadata within Immich will be lost. -## External Libraries +:::caution -External libraries tracks assets stored outside of Immich, i.e. in the file system. When the external library is scanned, Immich will read the metadata from the file and create an asset in the library for each image or video file. These items will then be shown in the main timeline, and they will look and behave like any other asset, including viewing on the map, adding to albums, etc. +If you add metadata to an external asset in any way (i.e. add it to an album or edit the description), that metadata is only stored inside Immich and will not be persisted to the external asset file. If you move an asset to another location within the library all such metadata will be lost upon rescan. This is because the asset is considered a new asset after the move. This is a known issue and will be fixed in a future release. -If a file is modified outside of Immich, the changes will not be reflected in immich until the library is scanned again. There are different ways to scan a library depending on the use case: - -- Scan Library Files: This is the default scan method and also the quickest. It will scan all files in the library and add new files to the library. It will notice if any files are missing (see below) but not check existing assets -- Scan All Library Files: Same as above, but will check each existing asset to see if the modification time has changed. If it has, the asset will be updated. Since it has to check each asset, this is slower than Scan Library Files. -- Force Scan All Library Files: Same as above, but will read each asset from disk no matter the modification time. This is useful in some cases where an asset has been modified externally but the modification time has not changed. This is the slowest way to scan because it reads each asset from disk. +::: :::caution @@ -20,22 +16,6 @@ Due to aggressive caching it can take some time for a refreshed asset to appear ::: -In external libraries, the file path is used for duplicate detection. This means that if a file is moved to a different location, it will be added as a new asset. If the file is moved back to its original location, it will be added as a new asset. In contrast to upload libraries, two identical files can be uploaded if they are in different locations. This is a deliberate design choice to make Immich reflect the file system as closely as possible. Remember that duplication detection is only done within the same library, so if you have multiple external libraries, the same file can be added to multiple libraries. - -:::caution - -If you add assets from an external library to an album and then move the asset to another location within the library, the asset will be removed from the album upon rescan. This is because the asset is considered a new asset after the move. This is a known issue and will be fixed in a future release. - -::: - -### Deleted External Assets - -Note: Either a manual or scheduled library scan must have been performed to identify offline assets before this process will work. - -In all above scan methods, Immich will check if any files are missing. This can happen if files are deleted, or if they are on a storage location that is currently unavailable, like a network drive that is not mounted, or a USB drive that has been unplugged. In order to prevent accidental deletion of assets, Immich will not immediately delete an asset from the library if the file is missing. Instead, the asset will be internally marked as offline and will still be visible in the main timeline. If the file is moved back to its original location and the library is scanned again, the asset will be restored. - -Finally, files can be deleted from Immich via the `Remove Offline Files` job. This job can be found by the three dots menu for the associated external storage that was configured under Administration > Libraries (the same location described at [create external libraries](#create-external-libraries)). When this job is run, any assets marked as offline will then be removed from Immich. Run this job whenever files have been deleted from the file system and you want to remove them from Immich. - ### Import Paths External libraries use import paths to determine which files to scan. Each library can have multiple import paths so that files from different locations can be added to the same library. Import paths are scanned recursively, and if a file is in multiple import paths, it will only be added once. Each import file must be a readable directory that exists on the filesystem; the import path dialog will alert you of any paths that are not accessible. @@ -66,9 +46,13 @@ Some basic examples: - `**/Raw/**` will exclude all files in any directory named `Raw` - `**/*.{tif,jpg}` will exclude all files with the extension `.tif` or `.jpg` +Special characters such as @ should be escaped, for instance: + +- `**/\@eadir/**` will exclude all files in any directory named `@eadir` + ### Automatic watching (EXPERIMENTAL) -This feature - currently hidden in the config file - is considered experimental and for advanced users only. If enabled, it will allow automatic watching of the filesystem which means new assets are automatically imported to Immich without needing to rescan. Deleted assets are, as always, marked as offline and can be removed with the "Remove offline files" button. +This feature - currently hidden in the config file - is considered experimental and for advanced users only. If enabled, it will allow automatic watching of the filesystem which means new assets are automatically imported to Immich without needing to rescan. If your photos are on a network drive, automatic file watching likely won't work. In that case, you will have to rely on a periodic library refresh to pull in your changes. @@ -84,7 +68,7 @@ In rare cases, the library watcher can hang, preventing Immich from starting up. ### Nightly job -There is an automatic job that's run once a day and refreshes all modified files in all libraries as well as cleans up any libraries stuck in deletion. +There is an automatic scan job that is scheduled to run once a day. This job also cleans up any libraries stuck in deletion. ## Usage @@ -120,7 +104,7 @@ This will disallow the images from being deleted in the web UI, or adding metada _Remember to run `docker compose up -d` to register the changes. Make sure you can see the mounted path in the container._ ::: -### Create External Libraries +### Create A New Library These actions must be performed by the Immich administrator. @@ -144,7 +128,7 @@ Next, we'll add an exclusion pattern to filter out raw files. - Enter `**/Raw/**` and click save. - Click save - Click the drop-down menu on the newly created library -- Click on Scan Library Files +- Click on Scan The christmas trip library will now be scanned in the background. In the meantime, let's add the videos and old photos to another library. @@ -161,7 +145,7 @@ If you get an error here, please rename the other external library to something - Click on Add Path - Enter `/mnt/media/videos` then click Add - Click Save -- Click on Scan Library Files +- Click on Scan Within seconds, the assets from the old-pics and videos folders should show up in the main timeline. diff --git a/e2e/src/api/specs/library.e2e-spec.ts b/e2e/src/api/specs/library.e2e-spec.ts index 8d98e866301f8..20bd230159c28 100644 --- a/e2e/src/api/specs/library.e2e-spec.ts +++ b/e2e/src/api/specs/library.e2e-spec.ts @@ -1,11 +1,4 @@ -import { - LibraryResponseDto, - LoginResponseDto, - ScanLibraryDto, - getAllLibraries, - removeOfflineFiles, - scanLibrary, -} from '@immich/sdk'; +import { LibraryResponseDto, LoginResponseDto, getAllLibraries, scanLibrary } from '@immich/sdk'; import { cpSync, existsSync } from 'node:fs'; import { Socket } from 'socket.io-client'; import { userDto, uuidDto } from 'src/fixtures'; @@ -15,8 +8,7 @@ import request from 'supertest'; import { utimes } from 'utimes'; import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest'; -const scan = async (accessToken: string, id: string, dto: ScanLibraryDto = {}) => - scanLibrary({ id, scanLibraryDto: dto }, { headers: asBearerAuth(accessToken) }); +const scan = async (accessToken: string, id: string) => scanLibrary({ id }, { headers: asBearerAuth(accessToken) }); describe('/libraries', () => { let admin: LoginResponseDto; @@ -293,14 +285,19 @@ describe('/libraries', () => { expect(body).toEqual(errorDto.unauthorized); }); - it('should scan external library', async () => { + it('should import new asset when scanning external library', async () => { const library = await utils.createLibrary(admin.accessToken, { ownerId: admin.userId, importPaths: [`${testAssetDirInternal}/temp/directoryA`], }); - await scan(admin.accessToken, library.id); - await utils.waitForWebsocketEvent({ event: 'assetUpload', total: 1 }); + const { status } = await request(app) + .post(`/libraries/${library.id}/scan`) + .set('Authorization', `Bearer ${admin.accessToken}`) + .send(); + expect(status).toBe(204); + + await utils.waitForQueueFinish(admin.accessToken, 'library'); const { assets } = await utils.metadataSearch(admin.accessToken, { originalPath: `${testAssetDirInternal}/temp/directoryA/assetA.png`, @@ -315,8 +312,13 @@ describe('/libraries', () => { exclusionPatterns: ['**/directoryA'], }); - await scan(admin.accessToken, library.id); - await utils.waitForWebsocketEvent({ event: 'assetUpload', total: 1 }); + const { status } = await request(app) + .post(`/libraries/${library.id}/scan`) + .set('Authorization', `Bearer ${admin.accessToken}`) + .send(); + expect(status).toBe(204); + + await utils.waitForQueueFinish(admin.accessToken, 'library'); const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); @@ -330,8 +332,13 @@ describe('/libraries', () => { importPaths: [`${testAssetDirInternal}/temp/directoryA`, `${testAssetDirInternal}/temp/directoryB`], }); - await scan(admin.accessToken, library.id); - await utils.waitForWebsocketEvent({ event: 'assetUpload', total: 2 }); + const { status } = await request(app) + .post(`/libraries/${library.id}/scan`) + .set('Authorization', `Bearer ${admin.accessToken}`) + .send(); + expect(status).toBe(204); + + await utils.waitForQueueFinish(admin.accessToken, 'library'); const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); @@ -340,95 +347,144 @@ describe('/libraries', () => { expect(assets.items.find((asset) => asset.originalPath.includes('directoryB'))).toBeDefined(); }); - it('should pick up new files', async () => { + it('should reimport a modified file', async () => { const library = await utils.createLibrary(admin.accessToken, { ownerId: admin.userId, importPaths: [`${testAssetDirInternal}/temp`], }); - await scan(admin.accessToken, library.id); - await utils.waitForWebsocketEvent({ event: 'assetUpload', total: 2 }); - - const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); - - expect(assets.count).toBe(2); - - utils.createImageFile(`${testAssetDir}/temp/directoryA/assetC.png`); + utils.createImageFile(`${testAssetDir}/temp/directoryA/assetB.jpg`); + await utimes(`${testAssetDir}/temp/directoryA/assetB.jpg`, 447_775_200_000); await scan(admin.accessToken, library.id); - await utils.waitForWebsocketEvent({ event: 'assetUpload', total: 3 }); + await utils.waitForQueueFinish(admin.accessToken, 'library'); - const { assets: newAssets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); + cpSync(`${testAssetDir}/albums/nature/tanners_ridge.jpg`, `${testAssetDir}/temp/directoryA/assetB.jpg`); + await utimes(`${testAssetDir}/temp/directoryA/assetB.jpg`, 447_775_200_001); - expect(newAssets.count).toBe(3); - utils.removeImageFile(`${testAssetDir}/temp/directoryA/assetC.png`); + const { status } = await request(app) + .post(`/libraries/${library.id}/scan`) + .set('Authorization', `Bearer ${admin.accessToken}`) + .send({ refreshModifiedFiles: true }); + expect(status).toBe(204); + + await utils.waitForQueueFinish(admin.accessToken, 'library'); + await utils.waitForQueueFinish(admin.accessToken, 'metadataExtraction'); + utils.removeImageFile(`${testAssetDir}/temp/directoryA/assetB.jpg`); + + const { assets } = await utils.metadataSearch(admin.accessToken, { + libraryId: library.id, + model: 'NIKON D750', + }); + expect(assets.count).toBe(1); }); - it('should offline a file missing from disk', async () => { - utils.createImageFile(`${testAssetDir}/temp/directoryA/assetC.png`); + it('should not reimport unmodified files', async () => { const library = await utils.createLibrary(admin.accessToken, { ownerId: admin.userId, importPaths: [`${testAssetDirInternal}/temp`], }); + utils.createImageFile(`${testAssetDir}/temp/directoryA/assetB.jpg`); + await utimes(`${testAssetDir}/temp/directoryA/assetB.jpg`, 447_775_200_000); + + await scan(admin.accessToken, library.id); + await utils.waitForQueueFinish(admin.accessToken, 'library'); + + cpSync(`${testAssetDir}/albums/nature/tanners_ridge.jpg`, `${testAssetDir}/temp/directoryA/assetB.jpg`); + await utimes(`${testAssetDir}/temp/directoryA/assetB.jpg`, 447_775_200_000); + + const { status } = await request(app) + .post(`/libraries/${library.id}/scan`) + .set('Authorization', `Bearer ${admin.accessToken}`) + .send({ refreshModifiedFiles: true }); + expect(status).toBe(204); + + await utils.waitForQueueFinish(admin.accessToken, 'library'); + await utils.waitForQueueFinish(admin.accessToken, 'metadataExtraction'); + utils.removeImageFile(`${testAssetDir}/temp/directoryA/assetB.jpg`); + + const { assets } = await utils.metadataSearch(admin.accessToken, { + libraryId: library.id, + model: 'NIKON D750', + }); + expect(assets.count).toBe(0); + }); + + it('should set an asset offline if its file is missing', async () => { + const library = await utils.createLibrary(admin.accessToken, { + ownerId: admin.userId, + importPaths: [`${testAssetDirInternal}/temp/offline`], + }); + + utils.createImageFile(`${testAssetDir}/temp/offline/offline.png`); + await scan(admin.accessToken, library.id); await utils.waitForQueueFinish(admin.accessToken, 'library'); const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); - expect(assets.count).toBe(3); + expect(assets.count).toBe(1); - utils.removeImageFile(`${testAssetDir}/temp/directoryA/assetC.png`); + utils.removeImageFile(`${testAssetDir}/temp/offline/offline.png`); + + const { status } = await request(app) + .post(`/libraries/${library.id}/scan`) + .set('Authorization', `Bearer ${admin.accessToken}`) + .send(); + expect(status).toBe(204); - await scan(admin.accessToken, library.id); await utils.waitForQueueFinish(admin.accessToken, 'library'); + const trashedAsset = await utils.getAssetInfo(admin.accessToken, assets.items[0].id); + expect(trashedAsset.originalPath).toBe(`${testAssetDirInternal}/temp/offline/offline.png`); + expect(trashedAsset.isOffline).toEqual(true); + const { assets: newAssets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); - expect(newAssets.count).toBe(3); - - expect(newAssets.items).toEqual( - expect.arrayContaining([ - expect.objectContaining({ - isOffline: true, - originalFileName: 'assetC.png', - }), - ]), - ); + expect(newAssets.items).toEqual([]); }); - it('should offline a file outside of import paths', async () => { + it('should set an asset offline its file is not in any import path', async () => { const library = await utils.createLibrary(admin.accessToken, { ownerId: admin.userId, - importPaths: [`${testAssetDirInternal}/temp`], + importPaths: [`${testAssetDirInternal}/temp/offline`], }); + utils.createImageFile(`${testAssetDir}/temp/offline/offline.png`); + await scan(admin.accessToken, library.id); await utils.waitForQueueFinish(admin.accessToken, 'library'); + const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); + expect(assets.count).toBe(1); + + utils.createDirectory(`${testAssetDir}/temp/another-path/`); + await request(app) .put(`/libraries/${library.id}`) .set('Authorization', `Bearer ${admin.accessToken}`) - .send({ importPaths: [`${testAssetDirInternal}/temp/directoryA`] }); + .send({ importPaths: [`${testAssetDirInternal}/temp/another-path/`] }); + + const { status } = await request(app) + .post(`/libraries/${library.id}/scan`) + .set('Authorization', `Bearer ${admin.accessToken}`) + .send(); + expect(status).toBe(204); - await scan(admin.accessToken, library.id); await utils.waitForQueueFinish(admin.accessToken, 'library'); - const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); + const trashedAsset = await utils.getAssetInfo(admin.accessToken, assets.items[0].id); + expect(trashedAsset.originalPath).toBe(`${testAssetDirInternal}/temp/offline/offline.png`); + expect(trashedAsset.isOffline).toBe(true); - expect(assets.items).toEqual( - expect.arrayContaining([ - expect.objectContaining({ - isOffline: false, - originalFileName: 'assetA.png', - }), - expect.objectContaining({ - isOffline: true, - originalFileName: 'assetB.png', - }), - ]), - ); + const { assets: newAssets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); + + expect(newAssets.items).toEqual([]); + + utils.removeImageFile(`${testAssetDir}/temp/offline/offline.png`); + utils.removeDirectory(`${testAssetDir}/temp/another-path/`); }); - it('should offline a file covered by an exclusion pattern', async () => { + it('should set an asset offline if its file is covered by an exclusion pattern', async () => { const library = await utils.createLibrary(admin.accessToken, { ownerId: admin.userId, importPaths: [`${testAssetDirInternal}/temp`], @@ -437,6 +493,12 @@ describe('/libraries', () => { await scan(admin.accessToken, library.id); await utils.waitForQueueFinish(admin.accessToken, 'library'); + const { assets } = await utils.metadataSearch(admin.accessToken, { + libraryId: library.id, + originalFileName: 'assetB.png', + }); + expect(assets.count).toBe(1); + await request(app) .put(`/libraries/${library.id}`) .set('Authorization', `Bearer ${admin.accessToken}`) @@ -445,282 +507,21 @@ describe('/libraries', () => { await scan(admin.accessToken, library.id); await utils.waitForQueueFinish(admin.accessToken, 'library'); - const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); + const trashedAsset = await utils.getAssetInfo(admin.accessToken, assets.items[0].id); + expect(trashedAsset.isTrashed).toBe(true); + expect(trashedAsset.originalPath).toBe(`${testAssetDirInternal}/temp/directoryB/assetB.png`); + expect(trashedAsset.isOffline).toBe(true); - expect(assets.count).toBe(2); + const { assets: newAssets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); - expect(assets.items).toEqual( - expect.arrayContaining([ - expect.objectContaining({ - isOffline: false, - originalFileName: 'assetA.png', - }), - expect.objectContaining({ - isOffline: true, - originalFileName: 'assetB.png', - }), - ]), - ); + expect(newAssets.items).toEqual([ + expect.objectContaining({ + originalFileName: 'assetA.png', + }), + ]); }); - it('should not try to delete offline files', async () => { - utils.createImageFile(`${testAssetDir}/temp/offline1/assetA.png`); - - const library = await utils.createLibrary(admin.accessToken, { - ownerId: admin.userId, - importPaths: [`${testAssetDirInternal}/temp/offline1`], - }); - - await scan(admin.accessToken, library.id); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - - const { assets: initialAssets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); - expect(initialAssets).toEqual({ - count: 1, - total: 1, - facets: [], - items: [expect.objectContaining({ originalFileName: 'assetA.png' })], - nextPage: null, - }); - - utils.removeImageFile(`${testAssetDir}/temp/offline1/assetA.png`); - - await scan(admin.accessToken, library.id); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - - const { assets: offlineAssets } = await utils.metadataSearch(admin.accessToken, { - libraryId: library.id, - isOffline: true, - }); - expect(offlineAssets).toEqual({ - count: 1, - total: 1, - facets: [], - items: [expect.objectContaining({ originalFileName: 'assetA.png' })], - nextPage: null, - }); - - utils.createImageFile(`${testAssetDir}/temp/offline1/assetA.png`); - await removeOfflineFiles({ id: library.id }, { headers: asBearerAuth(admin.accessToken) }); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - await utils.waitForWebsocketEvent({ event: 'assetDelete', total: 1 }); - - expect(existsSync(`${testAssetDir}/temp/offline1/assetA.png`)).toBe(true); - - utils.removeImageFile(`${testAssetDir}/temp/offline1/assetA.png`); - }); - - it('should scan new files', async () => { - const library = await utils.createLibrary(admin.accessToken, { - ownerId: admin.userId, - importPaths: [`${testAssetDirInternal}/temp`], - }); - - await scan(admin.accessToken, library.id); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - - utils.createImageFile(`${testAssetDir}/temp/directoryC/assetC.png`); - - await scan(admin.accessToken, library.id); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - - const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); - - expect(assets.count).toBe(3); - expect(assets.items).toEqual( - expect.arrayContaining([ - expect.objectContaining({ - originalFileName: 'assetC.png', - }), - ]), - ); - - utils.removeImageFile(`${testAssetDir}/temp/directoryC/assetC.png`); - }); - - describe('with refreshModifiedFiles=true', () => { - it('should reimport modified files', async () => { - const library = await utils.createLibrary(admin.accessToken, { - ownerId: admin.userId, - importPaths: [`${testAssetDirInternal}/temp`], - }); - - utils.createImageFile(`${testAssetDir}/temp/directoryA/assetB.jpg`); - await utimes(`${testAssetDir}/temp/directoryA/assetB.jpg`, 447_775_200_000); - - await scan(admin.accessToken, library.id); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - - cpSync(`${testAssetDir}/albums/nature/tanners_ridge.jpg`, `${testAssetDir}/temp/directoryA/assetB.jpg`); - await utimes(`${testAssetDir}/temp/directoryA/assetB.jpg`, 447_775_200_001); - - await scan(admin.accessToken, library.id, { refreshModifiedFiles: true }); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - await utils.waitForQueueFinish(admin.accessToken, 'metadataExtraction'); - utils.removeImageFile(`${testAssetDir}/temp/directoryA/assetB.jpg`); - - const { assets } = await utils.metadataSearch(admin.accessToken, { - libraryId: library.id, - model: 'NIKON D750', - }); - expect(assets.count).toBe(1); - }); - - it('should not reimport unmodified files', async () => { - const library = await utils.createLibrary(admin.accessToken, { - ownerId: admin.userId, - importPaths: [`${testAssetDirInternal}/temp`], - }); - - utils.createImageFile(`${testAssetDir}/temp/directoryA/assetB.jpg`); - await utimes(`${testAssetDir}/temp/directoryA/assetB.jpg`, 447_775_200_000); - - await scan(admin.accessToken, library.id); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - - cpSync(`${testAssetDir}/albums/nature/tanners_ridge.jpg`, `${testAssetDir}/temp/directoryA/assetB.jpg`); - await utimes(`${testAssetDir}/temp/directoryA/assetB.jpg`, 447_775_200_000); - - await scan(admin.accessToken, library.id, { refreshModifiedFiles: true }); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - await utils.waitForQueueFinish(admin.accessToken, 'metadataExtraction'); - utils.removeImageFile(`${testAssetDir}/temp/directoryA/assetB.jpg`); - - const { assets } = await utils.metadataSearch(admin.accessToken, { - libraryId: library.id, - model: 'NIKON D750', - }); - expect(assets.count).toBe(0); - }); - }); - - describe('with refreshAllFiles=true', () => { - it('should reimport all files', async () => { - const library = await utils.createLibrary(admin.accessToken, { - ownerId: admin.userId, - importPaths: [`${testAssetDirInternal}/temp`], - }); - - utils.createImageFile(`${testAssetDir}/temp/directoryA/assetB.jpg`); - await utimes(`${testAssetDir}/temp/directoryA/assetB.jpg`, 447_775_200_000); - - await scan(admin.accessToken, library.id); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - - cpSync(`${testAssetDir}/albums/nature/tanners_ridge.jpg`, `${testAssetDir}/temp/directoryA/assetB.jpg`); - await utimes(`${testAssetDir}/temp/directoryA/assetB.jpg`, 447_775_200_000); - - await scan(admin.accessToken, library.id, { refreshAllFiles: true }); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - await utils.waitForQueueFinish(admin.accessToken, 'metadataExtraction'); - utils.removeImageFile(`${testAssetDir}/temp/directoryA/assetB.jpg`); - - const { assets } = await utils.metadataSearch(admin.accessToken, { - libraryId: library.id, - model: 'NIKON D750', - }); - expect(assets.count).toBe(1); - }); - }); - }); - - describe('POST /libraries/:id/removeOffline', () => { - it('should require authentication', async () => { - const { status, body } = await request(app).post(`/libraries/${uuidDto.notFound}/removeOffline`).send({}); - - expect(status).toBe(401); - expect(body).toEqual(errorDto.unauthorized); - }); - - it('should remove offline files', async () => { - const library = await utils.createLibrary(admin.accessToken, { - ownerId: admin.userId, - importPaths: [`${testAssetDirInternal}/temp/offline`], - }); - - utils.createImageFile(`${testAssetDir}/temp/offline/online.png`); - utils.createImageFile(`${testAssetDir}/temp/offline/offline.png`); - - await scan(admin.accessToken, library.id); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - - const { assets: initialAssets } = await utils.metadataSearch(admin.accessToken, { - libraryId: library.id, - }); - expect(initialAssets.count).toBe(2); - - utils.removeImageFile(`${testAssetDir}/temp/offline/offline.png`); - - await scan(admin.accessToken, library.id); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - - const { assets: offlineAssets } = await utils.metadataSearch(admin.accessToken, { - libraryId: library.id, - isOffline: true, - }); - expect(offlineAssets.count).toBe(1); - - const { status } = await request(app) - .post(`/libraries/${library.id}/removeOffline`) - .set('Authorization', `Bearer ${admin.accessToken}`) - .send(); - expect(status).toBe(204); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - await utils.waitForQueueFinish(admin.accessToken, 'backgroundTask'); - - const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); - - expect(assets.count).toBe(1); - - utils.removeImageFile(`${testAssetDir}/temp/offline/online.png`); - }); - - it('should remove offline files from trash', async () => { - const library = await utils.createLibrary(admin.accessToken, { - ownerId: admin.userId, - importPaths: [`${testAssetDirInternal}/temp/offline`], - }); - - utils.createImageFile(`${testAssetDir}/temp/offline/online.png`); - utils.createImageFile(`${testAssetDir}/temp/offline/offline.png`); - - await scan(admin.accessToken, library.id); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - - const { assets: initialAssets } = await utils.metadataSearch(admin.accessToken, { - libraryId: library.id, - }); - - expect(initialAssets.count).toBe(2); - utils.removeImageFile(`${testAssetDir}/temp/offline/offline.png`); - - await scan(admin.accessToken, library.id); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - - const { assets: offlineAssets } = await utils.metadataSearch(admin.accessToken, { - libraryId: library.id, - isOffline: true, - }); - expect(offlineAssets.count).toBe(1); - - const { status } = await request(app) - .post(`/libraries/${library.id}/removeOffline`) - .set('Authorization', `Bearer ${admin.accessToken}`) - .send(); - expect(status).toBe(204); - await utils.waitForQueueFinish(admin.accessToken, 'library'); - await utils.waitForQueueFinish(admin.accessToken, 'backgroundTask'); - - const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); - - expect(assets.count).toBe(1); - expect(assets.items[0].isOffline).toBe(false); - expect(assets.items[0].originalPath).toEqual(`${testAssetDirInternal}/temp/offline/online.png`); - - utils.removeImageFile(`${testAssetDir}/temp/offline/online.png`); - }); - - it('should not remove online files', async () => { + it('should not trash an online asset', async () => { const library = await utils.createLibrary(admin.accessToken, { ownerId: admin.userId, importPaths: [`${testAssetDirInternal}/temp`], @@ -733,10 +534,11 @@ describe('/libraries', () => { expect(assetsBefore.count).toBeGreaterThan(1); const { status } = await request(app) - .post(`/libraries/${library.id}/removeOffline`) + .post(`/libraries/${library.id}/scan`) .set('Authorization', `Bearer ${admin.accessToken}`) .send(); expect(status).toBe(204); + await utils.waitForQueueFinish(admin.accessToken, 'library'); const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); @@ -828,7 +630,7 @@ describe('/libraries', () => { }); await scan(admin.accessToken, library.id); - await utils.waitForWebsocketEvent({ event: 'assetUpload', total: 2 }); + await utils.waitForQueueFinish(admin.accessToken, 'library'); const { status, body } = await request(app) .delete(`/libraries/${library.id}`) diff --git a/e2e/src/api/specs/search.e2e-spec.ts b/e2e/src/api/specs/search.e2e-spec.ts index beeaf1cc01ca5..0e5d882f80e50 100644 --- a/e2e/src/api/specs/search.e2e-spec.ts +++ b/e2e/src/api/specs/search.e2e-spec.ts @@ -181,7 +181,7 @@ describe('/search', () => { dto: { size: -1.5 }, expected: ['size must not be less than 1', 'size must be an integer number'], }, - ...['isArchived', 'isFavorite', 'isEncoded', 'isMotion', 'isOffline', 'isVisible'].map((value) => ({ + ...['isArchived', 'isFavorite', 'isEncoded', 'isOffline', 'isMotion', 'isVisible'].map((value) => ({ should: `should reject ${value} not a boolean`, dto: { [value]: 'immich' }, expected: [`${value} must be a boolean value`], diff --git a/e2e/src/api/specs/trash.e2e-spec.ts b/e2e/src/api/specs/trash.e2e-spec.ts index 17bb568c61cef..0bfc0ec19be21 100644 --- a/e2e/src/api/specs/trash.e2e-spec.ts +++ b/e2e/src/api/specs/trash.e2e-spec.ts @@ -1,10 +1,13 @@ -import { LoginResponseDto, getAssetInfo, getAssetStatistics } from '@immich/sdk'; +import { LoginResponseDto, getAssetInfo, getAssetStatistics, scanLibrary } from '@immich/sdk'; +import { existsSync } from 'node:fs'; import { Socket } from 'socket.io-client'; import { errorDto } from 'src/responses'; -import { app, asBearerAuth, utils } from 'src/utils'; +import { app, asBearerAuth, testAssetDir, testAssetDirInternal, utils } from 'src/utils'; import request from 'supertest'; import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +const scan = async (accessToken: string, id: string) => scanLibrary({ id }, { headers: asBearerAuth(accessToken) }); + describe('/trash', () => { let admin: LoginResponseDto; let ws: Socket; @@ -44,6 +47,8 @@ describe('/trash', () => { const after = await getAssetStatistics({ isTrashed: true }, { headers: asBearerAuth(admin.accessToken) }); expect(after.total).toBe(0); + + expect(existsSync(before.originalPath)).toBe(false); }); it('should empty the trash with archived assets', async () => { @@ -64,6 +69,46 @@ describe('/trash', () => { const after = await getAssetStatistics({ isTrashed: true }, { headers: asBearerAuth(admin.accessToken) }); expect(after.total).toBe(0); + + expect(existsSync(before.originalPath)).toBe(false); + }); + + it('should not delete offline-trashed assets from disk', async () => { + const library = await utils.createLibrary(admin.accessToken, { + ownerId: admin.userId, + importPaths: [`${testAssetDirInternal}/temp/offline`], + }); + + utils.createImageFile(`${testAssetDir}/temp/offline/offline.png`); + + await scan(admin.accessToken, library.id); + await utils.waitForQueueFinish(admin.accessToken, 'library'); + + const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); + expect(assets.items.length).toBe(1); + const asset = assets.items[0]; + + utils.removeImageFile(`${testAssetDir}/temp/offline/offline.png`); + + await scan(admin.accessToken, library.id); + await utils.waitForQueueFinish(admin.accessToken, 'library'); + + const assetBefore = await utils.getAssetInfo(admin.accessToken, asset.id); + expect(assetBefore).toMatchObject({ isTrashed: true, isOffline: true }); + + utils.createImageFile(`${testAssetDir}/temp/offline/offline.png`); + + const { status } = await request(app).post('/trash/empty').set('Authorization', `Bearer ${admin.accessToken}`); + expect(status).toBe(200); + + await utils.waitForQueueFinish(admin.accessToken, 'backgroundTask'); + + const assetAfter = await utils.getAssetInfo(admin.accessToken, asset.id); + expect(assetAfter).toMatchObject({ isTrashed: true, isOffline: true }); + + expect(existsSync(`${testAssetDir}/temp/offline/offline.png`)).toBe(true); + + utils.removeImageFile(`${testAssetDir}/temp/offline/offline.png`); }); }); @@ -91,6 +136,37 @@ describe('/trash', () => { const after = await getAssetInfo({ id: assetId }, { headers: asBearerAuth(admin.accessToken) }); expect(after).toStrictEqual(expect.objectContaining({ id: assetId, isTrashed: false })); }); + + it('should not restore offline-trashed assets', async () => { + const library = await utils.createLibrary(admin.accessToken, { + ownerId: admin.userId, + importPaths: [`${testAssetDirInternal}/temp/offline`], + }); + + utils.createImageFile(`${testAssetDir}/temp/offline/offline.png`); + + await scan(admin.accessToken, library.id); + await utils.waitForQueueFinish(admin.accessToken, 'library'); + + const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); + expect(assets.count).toBe(1); + const assetId = assets.items[0].id; + + utils.removeImageFile(`${testAssetDir}/temp/offline/offline.png`); + + await scan(admin.accessToken, library.id); + + await utils.waitForQueueFinish(admin.accessToken, 'library'); + + const before = await getAssetInfo({ id: assetId }, { headers: asBearerAuth(admin.accessToken) }); + expect(before).toStrictEqual(expect.objectContaining({ id: assetId, isOffline: true })); + + const { status } = await request(app).post('/trash/restore').set('Authorization', `Bearer ${admin.accessToken}`); + expect(status).toBe(200); + + const after = await getAssetInfo({ id: assetId }, { headers: asBearerAuth(admin.accessToken) }); + expect(after).toStrictEqual(expect.objectContaining({ id: assetId, isOffline: true })); + }); }); describe('POST /trash/restore/assets', () => { @@ -118,5 +194,38 @@ describe('/trash', () => { const after = await utils.getAssetInfo(admin.accessToken, assetId); expect(after.isTrashed).toBe(false); }); + + it('should not restore an offline-trashed asset', async () => { + const library = await utils.createLibrary(admin.accessToken, { + ownerId: admin.userId, + importPaths: [`${testAssetDirInternal}/temp/offline`], + }); + + utils.createImageFile(`${testAssetDir}/temp/offline/offline.png`); + + await scan(admin.accessToken, library.id); + await utils.waitForQueueFinish(admin.accessToken, 'library'); + + const { assets } = await utils.metadataSearch(admin.accessToken, { libraryId: library.id }); + expect(assets.count).toBe(1); + const assetId = assets.items[0].id; + + utils.removeImageFile(`${testAssetDir}/temp/offline/offline.png`); + + await scan(admin.accessToken, library.id); + await utils.waitForQueueFinish(admin.accessToken, 'library'); + + const before = await utils.getAssetInfo(admin.accessToken, assetId); + expect(before.isTrashed).toBe(true); + + const { status } = await request(app) + .post('/trash/restore/assets') + .set('Authorization', `Bearer ${admin.accessToken}`) + .send({ ids: [assetId] }); + expect(status).toBe(200); + + const after = await utils.getAssetInfo(admin.accessToken, assetId); + expect(after.isTrashed).toBe(true); + }); }); }); diff --git a/e2e/src/utils.ts b/e2e/src/utils.ts index 3c9d4284ce49c..e21b3bfd14934 100644 --- a/e2e/src/utils.ts +++ b/e2e/src/utils.ts @@ -372,6 +372,12 @@ export const utils = { writeFileSync(path, makeRandomImage()); }, + createDirectory: (path: string) => { + if (!existsSync(dirname(path))) { + mkdirSync(dirname(path), { recursive: true }); + } + }, + removeImageFile: (path: string) => { if (!existsSync(path)) { return; @@ -380,6 +386,14 @@ export const utils = { rmSync(path); }, + removeDirectory: (path: string) => { + if (!existsSync(path)) { + return; + } + + rmSync(path); + }, + getAssetInfo: (accessToken: string, id: string) => getAssetInfo({ id }, { headers: asBearerAuth(accessToken) }), checkExistingAssets: (accessToken: string, checkExistingAssetsDto: CheckExistingAssetsDto) => diff --git a/mobile/lib/entities/asset.entity.g.dart b/mobile/lib/entities/asset.entity.g.dart index 23bf23604635d..8be636efb659b 100644 --- a/mobile/lib/entities/asset.entity.g.dart +++ b/mobile/lib/entities/asset.entity.g.dart @@ -57,69 +57,64 @@ const AssetSchema = CollectionSchema( name: r'isFavorite', type: IsarType.bool, ), - r'isOffline': PropertySchema( - id: 8, - name: r'isOffline', - type: IsarType.bool, - ), r'isTrashed': PropertySchema( - id: 9, + id: 8, name: r'isTrashed', type: IsarType.bool, ), r'livePhotoVideoId': PropertySchema( - id: 10, + id: 9, name: r'livePhotoVideoId', type: IsarType.string, ), r'localId': PropertySchema( - id: 11, + id: 10, name: r'localId', type: IsarType.string, ), r'ownerId': PropertySchema( - id: 12, + id: 11, name: r'ownerId', type: IsarType.long, ), r'remoteId': PropertySchema( - id: 13, + id: 12, name: r'remoteId', type: IsarType.string, ), r'stackCount': PropertySchema( - id: 14, + id: 13, name: r'stackCount', type: IsarType.long, ), r'stackId': PropertySchema( - id: 15, + id: 14, name: r'stackId', type: IsarType.string, ), r'stackPrimaryAssetId': PropertySchema( - id: 16, + id: 15, name: r'stackPrimaryAssetId', type: IsarType.string, ), r'thumbhash': PropertySchema( - id: 17, + id: 16, name: r'thumbhash', type: IsarType.string, ), r'type': PropertySchema( - id: 18, + id: 17, name: r'type', type: IsarType.byte, enumMap: _AssettypeEnumValueMap, ), r'updatedAt': PropertySchema( - id: 19, + id: 18, name: r'updatedAt', type: IsarType.dateTime, ), r'width': PropertySchema( - id: 20, + id: 19, name: r'width', type: IsarType.int, ) @@ -244,19 +239,18 @@ void _assetSerialize( writer.writeInt(offsets[5], object.height); writer.writeBool(offsets[6], object.isArchived); writer.writeBool(offsets[7], object.isFavorite); - writer.writeBool(offsets[8], object.isOffline); - writer.writeBool(offsets[9], object.isTrashed); - writer.writeString(offsets[10], object.livePhotoVideoId); - writer.writeString(offsets[11], object.localId); - writer.writeLong(offsets[12], object.ownerId); - writer.writeString(offsets[13], object.remoteId); - writer.writeLong(offsets[14], object.stackCount); - writer.writeString(offsets[15], object.stackId); - writer.writeString(offsets[16], object.stackPrimaryAssetId); - writer.writeString(offsets[17], object.thumbhash); - writer.writeByte(offsets[18], object.type.index); - writer.writeDateTime(offsets[19], object.updatedAt); - writer.writeInt(offsets[20], object.width); + writer.writeBool(offsets[8], object.isTrashed); + writer.writeString(offsets[9], object.livePhotoVideoId); + writer.writeString(offsets[10], object.localId); + writer.writeLong(offsets[11], object.ownerId); + writer.writeString(offsets[12], object.remoteId); + writer.writeLong(offsets[13], object.stackCount); + writer.writeString(offsets[14], object.stackId); + writer.writeString(offsets[15], object.stackPrimaryAssetId); + writer.writeString(offsets[16], object.thumbhash); + writer.writeByte(offsets[17], object.type.index); + writer.writeDateTime(offsets[18], object.updatedAt); + writer.writeInt(offsets[19], object.width); } Asset _assetDeserialize( @@ -275,20 +269,19 @@ Asset _assetDeserialize( id: id, isArchived: reader.readBoolOrNull(offsets[6]) ?? false, isFavorite: reader.readBoolOrNull(offsets[7]) ?? false, - isOffline: reader.readBoolOrNull(offsets[8]) ?? false, - isTrashed: reader.readBoolOrNull(offsets[9]) ?? false, - livePhotoVideoId: reader.readStringOrNull(offsets[10]), - localId: reader.readStringOrNull(offsets[11]), - ownerId: reader.readLong(offsets[12]), - remoteId: reader.readStringOrNull(offsets[13]), - stackCount: reader.readLongOrNull(offsets[14]) ?? 0, - stackId: reader.readStringOrNull(offsets[15]), - stackPrimaryAssetId: reader.readStringOrNull(offsets[16]), - thumbhash: reader.readStringOrNull(offsets[17]), - type: _AssettypeValueEnumMap[reader.readByteOrNull(offsets[18])] ?? + isTrashed: reader.readBoolOrNull(offsets[8]) ?? false, + livePhotoVideoId: reader.readStringOrNull(offsets[9]), + localId: reader.readStringOrNull(offsets[10]), + ownerId: reader.readLong(offsets[11]), + remoteId: reader.readStringOrNull(offsets[12]), + stackCount: reader.readLongOrNull(offsets[13]) ?? 0, + stackId: reader.readStringOrNull(offsets[14]), + stackPrimaryAssetId: reader.readStringOrNull(offsets[15]), + thumbhash: reader.readStringOrNull(offsets[16]), + type: _AssettypeValueEnumMap[reader.readByteOrNull(offsets[17])] ?? AssetType.other, - updatedAt: reader.readDateTime(offsets[19]), - width: reader.readIntOrNull(offsets[20]), + updatedAt: reader.readDateTime(offsets[18]), + width: reader.readIntOrNull(offsets[19]), ); return object; } @@ -319,29 +312,27 @@ P _assetDeserializeProp

( case 8: return (reader.readBoolOrNull(offset) ?? false) as P; case 9: - return (reader.readBoolOrNull(offset) ?? false) as P; + return (reader.readStringOrNull(offset)) as P; case 10: return (reader.readStringOrNull(offset)) as P; case 11: - return (reader.readStringOrNull(offset)) as P; - case 12: return (reader.readLong(offset)) as P; - case 13: + case 12: return (reader.readStringOrNull(offset)) as P; - case 14: + case 13: return (reader.readLongOrNull(offset) ?? 0) as P; + case 14: + return (reader.readStringOrNull(offset)) as P; case 15: return (reader.readStringOrNull(offset)) as P; case 16: return (reader.readStringOrNull(offset)) as P; case 17: - return (reader.readStringOrNull(offset)) as P; - case 18: return (_AssettypeValueEnumMap[reader.readByteOrNull(offset)] ?? AssetType.other) as P; - case 19: + case 18: return (reader.readDateTime(offset)) as P; - case 20: + case 19: return (reader.readIntOrNull(offset)) as P; default: throw IsarError('Unknown property with id $propertyId'); @@ -1362,16 +1353,6 @@ extension AssetQueryFilter on QueryBuilder { }); } - QueryBuilder isOfflineEqualTo( - bool value) { - return QueryBuilder.apply(this, (query) { - return query.addFilterCondition(FilterCondition.equalTo( - property: r'isOffline', - value: value, - )); - }); - } - QueryBuilder isTrashedEqualTo( bool value) { return QueryBuilder.apply(this, (query) { @@ -2647,18 +2628,6 @@ extension AssetQuerySortBy on QueryBuilder { }); } - QueryBuilder sortByIsOffline() { - return QueryBuilder.apply(this, (query) { - return query.addSortBy(r'isOffline', Sort.asc); - }); - } - - QueryBuilder sortByIsOfflineDesc() { - return QueryBuilder.apply(this, (query) { - return query.addSortBy(r'isOffline', Sort.desc); - }); - } - QueryBuilder sortByIsTrashed() { return QueryBuilder.apply(this, (query) { return query.addSortBy(r'isTrashed', Sort.asc); @@ -2913,18 +2882,6 @@ extension AssetQuerySortThenBy on QueryBuilder { }); } - QueryBuilder thenByIsOffline() { - return QueryBuilder.apply(this, (query) { - return query.addSortBy(r'isOffline', Sort.asc); - }); - } - - QueryBuilder thenByIsOfflineDesc() { - return QueryBuilder.apply(this, (query) { - return query.addSortBy(r'isOffline', Sort.desc); - }); - } - QueryBuilder thenByIsTrashed() { return QueryBuilder.apply(this, (query) { return query.addSortBy(r'isTrashed', Sort.asc); @@ -3121,12 +3078,6 @@ extension AssetQueryWhereDistinct on QueryBuilder { }); } - QueryBuilder distinctByIsOffline() { - return QueryBuilder.apply(this, (query) { - return query.addDistinctBy(r'isOffline'); - }); - } - QueryBuilder distinctByIsTrashed() { return QueryBuilder.apply(this, (query) { return query.addDistinctBy(r'isTrashed'); @@ -3263,12 +3214,6 @@ extension AssetQueryProperty on QueryBuilder { }); } - QueryBuilder isOfflineProperty() { - return QueryBuilder.apply(this, (query) { - return query.addPropertyName(r'isOffline'); - }); - } - QueryBuilder isTrashedProperty() { return QueryBuilder.apply(this, (query) { return query.addPropertyName(r'isTrashed'); diff --git a/mobile/lib/extensions/collection_extensions.dart b/mobile/lib/extensions/collection_extensions.dart index 769bec472b210..f71b0aacd3a7d 100644 --- a/mobile/lib/extensions/collection_extensions.dart +++ b/mobile/lib/extensions/collection_extensions.dart @@ -72,13 +72,14 @@ extension AssetListExtension on Iterable { } /// Filters out offline assets and returns those that are still accessible by the Immich server + /// TODO: isOffline is removed from Immich, so this method is not useful anymore Iterable nonOfflineOnly({ void Function()? errorCallback, }) { - final bool onlyLive = every((e) => !e.isOffline); + final bool onlyLive = every((e) => false); if (!onlyLive) { if (errorCallback != null) errorCallback(); - return where((a) => !a.isOffline); + return where((a) => false); } return this; } diff --git a/mobile/lib/widgets/asset_viewer/bottom_gallery_bar.dart b/mobile/lib/widgets/asset_viewer/bottom_gallery_bar.dart index 7e6136c256192..8b5684d0fa241 100644 --- a/mobile/lib/widgets/asset_viewer/bottom_gallery_bar.dart +++ b/mobile/lib/widgets/asset_viewer/bottom_gallery_bar.dart @@ -172,29 +172,12 @@ class BottomGalleryBar extends ConsumerWidget { } shareAsset() { - if (asset.isOffline) { - ImmichToast.show( - durationInSecond: 1, - context: context, - msg: 'asset_action_share_err_offline'.tr(), - gravity: ToastGravity.BOTTOM, - ); - return; - } ref.read(imageViewerStateProvider.notifier).shareAsset(asset, context); } void handleEdit() async { final image = Image(image: ImmichImage.imageProvider(asset: asset)); - if (asset.isOffline) { - ImmichToast.show( - durationInSecond: 1, - context: context, - msg: 'asset_action_edit_err_offline'.tr(), - gravity: ToastGravity.BOTTOM, - ); - return; - } + Navigator.of(context).push( MaterialPageRoute( builder: (context) => EditImagePage( @@ -219,16 +202,6 @@ class BottomGalleryBar extends ConsumerWidget { if (asset.isLocal) { return; } - if (asset.isOffline) { - ImmichToast.show( - durationInSecond: 1, - context: context, - msg: 'asset_action_share_err_offline'.tr(), - gravity: ToastGravity.BOTTOM, - ); - return; - } - ref.read(imageViewerStateProvider.notifier).downloadAsset( asset, context, diff --git a/mobile/lib/widgets/asset_viewer/top_control_app_bar.dart b/mobile/lib/widgets/asset_viewer/top_control_app_bar.dart index 2157a1aebbf36..984b61f50cc05 100644 --- a/mobile/lib/widgets/asset_viewer/top_control_app_bar.dart +++ b/mobile/lib/widgets/asset_viewer/top_control_app_bar.dart @@ -183,8 +183,7 @@ class TopControlAppBar extends HookConsumerWidget { if (asset.isRemote && isOwner) buildFavoriteButton(a), if (asset.livePhotoVideoId != null) buildLivePhotoButton(), if (asset.isLocal && !asset.isRemote) buildUploadButton(), - if (asset.isRemote && !asset.isLocal && !asset.isOffline && isOwner) - buildDownloadButton(), + if (asset.isRemote && !asset.isLocal && isOwner) buildDownloadButton(), if (asset.isRemote && (isOwner || isPartner) && !asset.isTrashed) buildAddToAlbumButton(), if (asset.isTrashed) buildRestoreButton(), diff --git a/mobile/openapi/README.md b/mobile/openapi/README.md index b6b0897e8f5e0..e337c4831f5eb 100644 --- a/mobile/openapi/README.md +++ b/mobile/openapi/README.md @@ -133,7 +133,6 @@ Class | Method | HTTP request | Description *LibrariesApi* | [**getAllLibraries**](doc//LibrariesApi.md#getalllibraries) | **GET** /libraries | *LibrariesApi* | [**getLibrary**](doc//LibrariesApi.md#getlibrary) | **GET** /libraries/{id} | *LibrariesApi* | [**getLibraryStatistics**](doc//LibrariesApi.md#getlibrarystatistics) | **GET** /libraries/{id}/statistics | -*LibrariesApi* | [**removeOfflineFiles**](doc//LibrariesApi.md#removeofflinefiles) | **POST** /libraries/{id}/removeOffline | *LibrariesApi* | [**scanLibrary**](doc//LibrariesApi.md#scanlibrary) | **POST** /libraries/{id}/scan | *LibrariesApi* | [**updateLibrary**](doc//LibrariesApi.md#updatelibrary) | **PUT** /libraries/{id} | *LibrariesApi* | [**validate**](doc//LibrariesApi.md#validate) | **POST** /libraries/{id}/validate | @@ -385,7 +384,6 @@ Class | Method | HTTP request | Description - [ReactionLevel](doc//ReactionLevel.md) - [ReactionType](doc//ReactionType.md) - [ReverseGeocodingStateResponseDto](doc//ReverseGeocodingStateResponseDto.md) - - [ScanLibraryDto](doc//ScanLibraryDto.md) - [SearchAlbumResponseDto](doc//SearchAlbumResponseDto.md) - [SearchAssetResponseDto](doc//SearchAssetResponseDto.md) - [SearchExploreItem](doc//SearchExploreItem.md) diff --git a/mobile/openapi/lib/api.dart b/mobile/openapi/lib/api.dart index d08b6fc52138b..22b48df2fbcb1 100644 --- a/mobile/openapi/lib/api.dart +++ b/mobile/openapi/lib/api.dart @@ -197,7 +197,6 @@ part 'model/ratings_update.dart'; part 'model/reaction_level.dart'; part 'model/reaction_type.dart'; part 'model/reverse_geocoding_state_response_dto.dart'; -part 'model/scan_library_dto.dart'; part 'model/search_album_response_dto.dart'; part 'model/search_asset_response_dto.dart'; part 'model/search_explore_item.dart'; diff --git a/mobile/openapi/lib/api/assets_api.dart b/mobile/openapi/lib/api/assets_api.dart index bd1d5b84847a1..fd899869803fd 100644 --- a/mobile/openapi/lib/api/assets_api.dart +++ b/mobile/openapi/lib/api/assets_api.dart @@ -833,14 +833,12 @@ class AssetsApi { /// /// * [bool] isFavorite: /// - /// * [bool] isOffline: - /// /// * [bool] isVisible: /// /// * [String] livePhotoVideoId: /// /// * [MultipartFile] sidecarData: - Future uploadAssetWithHttpInfo(MultipartFile assetData, String deviceAssetId, String deviceId, DateTime fileCreatedAt, DateTime fileModifiedAt, { String? key, String? xImmichChecksum, String? duration, bool? isArchived, bool? isFavorite, bool? isOffline, bool? isVisible, String? livePhotoVideoId, MultipartFile? sidecarData, }) async { + Future uploadAssetWithHttpInfo(MultipartFile assetData, String deviceAssetId, String deviceId, DateTime fileCreatedAt, DateTime fileModifiedAt, { String? key, String? xImmichChecksum, String? duration, bool? isArchived, bool? isFavorite, bool? isVisible, String? livePhotoVideoId, MultipartFile? sidecarData, }) async { // ignore: prefer_const_declarations final path = r'/assets'; @@ -896,10 +894,6 @@ class AssetsApi { hasFields = true; mp.fields[r'isFavorite'] = parameterToString(isFavorite); } - if (isOffline != null) { - hasFields = true; - mp.fields[r'isOffline'] = parameterToString(isOffline); - } if (isVisible != null) { hasFields = true; mp.fields[r'isVisible'] = parameterToString(isVisible); @@ -951,15 +945,13 @@ class AssetsApi { /// /// * [bool] isFavorite: /// - /// * [bool] isOffline: - /// /// * [bool] isVisible: /// /// * [String] livePhotoVideoId: /// /// * [MultipartFile] sidecarData: - Future uploadAsset(MultipartFile assetData, String deviceAssetId, String deviceId, DateTime fileCreatedAt, DateTime fileModifiedAt, { String? key, String? xImmichChecksum, String? duration, bool? isArchived, bool? isFavorite, bool? isOffline, bool? isVisible, String? livePhotoVideoId, MultipartFile? sidecarData, }) async { - final response = await uploadAssetWithHttpInfo(assetData, deviceAssetId, deviceId, fileCreatedAt, fileModifiedAt, key: key, xImmichChecksum: xImmichChecksum, duration: duration, isArchived: isArchived, isFavorite: isFavorite, isOffline: isOffline, isVisible: isVisible, livePhotoVideoId: livePhotoVideoId, sidecarData: sidecarData, ); + Future uploadAsset(MultipartFile assetData, String deviceAssetId, String deviceId, DateTime fileCreatedAt, DateTime fileModifiedAt, { String? key, String? xImmichChecksum, String? duration, bool? isArchived, bool? isFavorite, bool? isVisible, String? livePhotoVideoId, MultipartFile? sidecarData, }) async { + final response = await uploadAssetWithHttpInfo(assetData, deviceAssetId, deviceId, fileCreatedAt, fileModifiedAt, key: key, xImmichChecksum: xImmichChecksum, duration: duration, isArchived: isArchived, isFavorite: isFavorite, isVisible: isVisible, livePhotoVideoId: livePhotoVideoId, sidecarData: sidecarData, ); if (response.statusCode >= HttpStatus.badRequest) { throw ApiException(response.statusCode, await _decodeBodyBytes(response)); } diff --git a/mobile/openapi/lib/api/libraries_api.dart b/mobile/openapi/lib/api/libraries_api.dart index 53ab0e19ce77b..36d98d9a88a78 100644 --- a/mobile/openapi/lib/api/libraries_api.dart +++ b/mobile/openapi/lib/api/libraries_api.dart @@ -243,13 +243,13 @@ class LibrariesApi { return null; } - /// Performs an HTTP 'POST /libraries/{id}/removeOffline' operation and returns the [Response]. + /// Performs an HTTP 'POST /libraries/{id}/scan' operation and returns the [Response]. /// Parameters: /// /// * [String] id (required): - Future removeOfflineFilesWithHttpInfo(String id,) async { + Future scanLibraryWithHttpInfo(String id,) async { // ignore: prefer_const_declarations - final path = r'/libraries/{id}/removeOffline' + final path = r'/libraries/{id}/scan' .replaceAll('{id}', id); // ignore: prefer_final_locals @@ -276,52 +276,8 @@ class LibrariesApi { /// Parameters: /// /// * [String] id (required): - Future removeOfflineFiles(String id,) async { - final response = await removeOfflineFilesWithHttpInfo(id,); - if (response.statusCode >= HttpStatus.badRequest) { - throw ApiException(response.statusCode, await _decodeBodyBytes(response)); - } - } - - /// Performs an HTTP 'POST /libraries/{id}/scan' operation and returns the [Response]. - /// Parameters: - /// - /// * [String] id (required): - /// - /// * [ScanLibraryDto] scanLibraryDto (required): - Future scanLibraryWithHttpInfo(String id, ScanLibraryDto scanLibraryDto,) async { - // ignore: prefer_const_declarations - final path = r'/libraries/{id}/scan' - .replaceAll('{id}', id); - - // ignore: prefer_final_locals - Object? postBody = scanLibraryDto; - - final queryParams = []; - final headerParams = {}; - final formParams = {}; - - const contentTypes = ['application/json']; - - - return apiClient.invokeAPI( - path, - 'POST', - queryParams, - postBody, - headerParams, - formParams, - contentTypes.isEmpty ? null : contentTypes.first, - ); - } - - /// Parameters: - /// - /// * [String] id (required): - /// - /// * [ScanLibraryDto] scanLibraryDto (required): - Future scanLibrary(String id, ScanLibraryDto scanLibraryDto,) async { - final response = await scanLibraryWithHttpInfo(id, scanLibraryDto,); + Future scanLibrary(String id,) async { + final response = await scanLibraryWithHttpInfo(id,); if (response.statusCode >= HttpStatus.badRequest) { throw ApiException(response.statusCode, await _decodeBodyBytes(response)); } diff --git a/mobile/openapi/lib/api_client.dart b/mobile/openapi/lib/api_client.dart index c62d1c5b2e2b2..3db3297acb091 100644 --- a/mobile/openapi/lib/api_client.dart +++ b/mobile/openapi/lib/api_client.dart @@ -448,8 +448,6 @@ class ApiClient { return ReactionTypeTypeTransformer().decode(value); case 'ReverseGeocodingStateResponseDto': return ReverseGeocodingStateResponseDto.fromJson(value); - case 'ScanLibraryDto': - return ScanLibraryDto.fromJson(value); case 'SearchAlbumResponseDto': return SearchAlbumResponseDto.fromJson(value); case 'SearchAssetResponseDto': diff --git a/mobile/openapi/lib/model/scan_library_dto.dart b/mobile/openapi/lib/model/scan_library_dto.dart deleted file mode 100644 index 8ff978be05321..0000000000000 --- a/mobile/openapi/lib/model/scan_library_dto.dart +++ /dev/null @@ -1,125 +0,0 @@ -// -// AUTO-GENERATED FILE, DO NOT MODIFY! -// -// @dart=2.18 - -// ignore_for_file: unused_element, unused_import -// ignore_for_file: always_put_required_named_parameters_first -// ignore_for_file: constant_identifier_names -// ignore_for_file: lines_longer_than_80_chars - -part of openapi.api; - -class ScanLibraryDto { - /// Returns a new [ScanLibraryDto] instance. - ScanLibraryDto({ - this.refreshAllFiles, - this.refreshModifiedFiles, - }); - - /// - /// Please note: This property should have been non-nullable! Since the specification file - /// does not include a default value (using the "default:" property), however, the generated - /// source code must fall back to having a nullable type. - /// Consider adding a "default:" property in the specification file to hide this note. - /// - bool? refreshAllFiles; - - /// - /// Please note: This property should have been non-nullable! Since the specification file - /// does not include a default value (using the "default:" property), however, the generated - /// source code must fall back to having a nullable type. - /// Consider adding a "default:" property in the specification file to hide this note. - /// - bool? refreshModifiedFiles; - - @override - bool operator ==(Object other) => identical(this, other) || other is ScanLibraryDto && - other.refreshAllFiles == refreshAllFiles && - other.refreshModifiedFiles == refreshModifiedFiles; - - @override - int get hashCode => - // ignore: unnecessary_parenthesis - (refreshAllFiles == null ? 0 : refreshAllFiles!.hashCode) + - (refreshModifiedFiles == null ? 0 : refreshModifiedFiles!.hashCode); - - @override - String toString() => 'ScanLibraryDto[refreshAllFiles=$refreshAllFiles, refreshModifiedFiles=$refreshModifiedFiles]'; - - Map toJson() { - final json = {}; - if (this.refreshAllFiles != null) { - json[r'refreshAllFiles'] = this.refreshAllFiles; - } else { - // json[r'refreshAllFiles'] = null; - } - if (this.refreshModifiedFiles != null) { - json[r'refreshModifiedFiles'] = this.refreshModifiedFiles; - } else { - // json[r'refreshModifiedFiles'] = null; - } - return json; - } - - /// Returns a new [ScanLibraryDto] instance and imports its values from - /// [value] if it's a [Map], null otherwise. - // ignore: prefer_constructors_over_static_methods - static ScanLibraryDto? fromJson(dynamic value) { - upgradeDto(value, "ScanLibraryDto"); - if (value is Map) { - final json = value.cast(); - - return ScanLibraryDto( - refreshAllFiles: mapValueOfType(json, r'refreshAllFiles'), - refreshModifiedFiles: mapValueOfType(json, r'refreshModifiedFiles'), - ); - } - return null; - } - - static List listFromJson(dynamic json, {bool growable = false,}) { - final result = []; - if (json is List && json.isNotEmpty) { - for (final row in json) { - final value = ScanLibraryDto.fromJson(row); - if (value != null) { - result.add(value); - } - } - } - return result.toList(growable: growable); - } - - static Map mapFromJson(dynamic json) { - final map = {}; - if (json is Map && json.isNotEmpty) { - json = json.cast(); // ignore: parameter_assignments - for (final entry in json.entries) { - final value = ScanLibraryDto.fromJson(entry.value); - if (value != null) { - map[entry.key] = value; - } - } - } - return map; - } - - // maps a json object with a list of ScanLibraryDto-objects as value to a dart map - static Map> mapListFromJson(dynamic json, {bool growable = false,}) { - final map = >{}; - if (json is Map && json.isNotEmpty) { - // ignore: parameter_assignments - json = json.cast(); - for (final entry in json.entries) { - map[entry.key] = ScanLibraryDto.listFromJson(entry.value, growable: growable,); - } - } - return map; - } - - /// The list of required keys that must be present in a JSON. - static const requiredKeys = { - }; -} - diff --git a/open-api/immich-openapi-specs.json b/open-api/immich-openapi-specs.json index 1a070f126b9b9..d0864675a172b 100644 --- a/open-api/immich-openapi-specs.json +++ b/open-api/immich-openapi-specs.json @@ -2853,41 +2853,6 @@ ] } }, - "/libraries/{id}/removeOffline": { - "post": { - "operationId": "removeOfflineFiles", - "parameters": [ - { - "name": "id", - "required": true, - "in": "path", - "schema": { - "format": "uuid", - "type": "string" - } - } - ], - "responses": { - "204": { - "description": "" - } - }, - "security": [ - { - "bearer": [] - }, - { - "cookie": [] - }, - { - "api_key": [] - } - ], - "tags": [ - "Libraries" - ] - } - }, "/libraries/{id}/scan": { "post": { "operationId": "scanLibrary", @@ -2902,16 +2867,6 @@ } } ], - "requestBody": { - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/ScanLibraryDto" - } - } - }, - "required": true - }, "responses": { "204": { "description": "" @@ -8287,9 +8242,6 @@ "isFavorite": { "type": "boolean" }, - "isOffline": { - "type": "boolean" - }, "isVisible": { "type": "boolean" }, @@ -10628,17 +10580,6 @@ ], "type": "object" }, - "ScanLibraryDto": { - "properties": { - "refreshAllFiles": { - "type": "boolean" - }, - "refreshModifiedFiles": { - "type": "boolean" - } - }, - "type": "object" - }, "SearchAlbumResponseDto": { "properties": { "count": { diff --git a/open-api/typescript-sdk/src/fetch-client.ts b/open-api/typescript-sdk/src/fetch-client.ts index f2f946f2626e2..85710af49c294 100644 --- a/open-api/typescript-sdk/src/fetch-client.ts +++ b/open-api/typescript-sdk/src/fetch-client.ts @@ -366,7 +366,6 @@ export type AssetMediaCreateDto = { fileModifiedAt: string; isArchived?: boolean; isFavorite?: boolean; - isOffline?: boolean; isVisible?: boolean; livePhotoVideoId?: string; sidecarData?: Blob; @@ -579,10 +578,6 @@ export type UpdateLibraryDto = { importPaths?: string[]; name?: string; }; -export type ScanLibraryDto = { - refreshAllFiles?: boolean; - refreshModifiedFiles?: boolean; -}; export type LibraryStatsResponseDto = { photos: number; total: number; @@ -2066,24 +2061,14 @@ export function updateLibrary({ id, updateLibraryDto }: { body: updateLibraryDto }))); } -export function removeOfflineFiles({ id }: { +export function scanLibrary({ id }: { id: string; }, opts?: Oazapfts.RequestOpts) { - return oazapfts.ok(oazapfts.fetchText(`/libraries/${encodeURIComponent(id)}/removeOffline`, { + return oazapfts.ok(oazapfts.fetchText(`/libraries/${encodeURIComponent(id)}/scan`, { ...opts, method: "POST" })); } -export function scanLibrary({ id, scanLibraryDto }: { - id: string; - scanLibraryDto: ScanLibraryDto; -}, opts?: Oazapfts.RequestOpts) { - return oazapfts.ok(oazapfts.fetchText(`/libraries/${encodeURIComponent(id)}/scan`, oazapfts.json({ - ...opts, - method: "POST", - body: scanLibraryDto - }))); -} export function getLibraryStatistics({ id }: { id: string; }, opts?: Oazapfts.RequestOpts) { diff --git a/server/src/controllers/library.controller.ts b/server/src/controllers/library.controller.ts index a45617fc2a503..b8959ca28875c 100644 --- a/server/src/controllers/library.controller.ts +++ b/server/src/controllers/library.controller.ts @@ -4,7 +4,6 @@ import { CreateLibraryDto, LibraryResponseDto, LibraryStatsResponseDto, - ScanLibraryDto, UpdateLibraryDto, ValidateLibraryDto, ValidateLibraryResponseDto, @@ -43,6 +42,13 @@ export class LibraryController { return this.service.update(id, dto); } + @Delete(':id') + @HttpCode(HttpStatus.NO_CONTENT) + @Authenticated({ permission: Permission.LIBRARY_DELETE, admin: true }) + deleteLibrary(@Param() { id }: UUIDParamDto): Promise { + return this.service.delete(id); + } + @Post(':id/validate') @HttpCode(200) @Authenticated({ admin: true }) @@ -51,13 +57,6 @@ export class LibraryController { return this.service.validate(id, dto); } - @Delete(':id') - @HttpCode(HttpStatus.NO_CONTENT) - @Authenticated({ permission: Permission.LIBRARY_DELETE, admin: true }) - deleteLibrary(@Param() { id }: UUIDParamDto): Promise { - return this.service.delete(id); - } - @Get(':id/statistics') @Authenticated({ permission: Permission.LIBRARY_STATISTICS, admin: true }) getLibraryStatistics(@Param() { id }: UUIDParamDto): Promise { @@ -66,15 +65,8 @@ export class LibraryController { @Post(':id/scan') @HttpCode(HttpStatus.NO_CONTENT) - @Authenticated({ admin: true }) - scanLibrary(@Param() { id }: UUIDParamDto, @Body() dto: ScanLibraryDto) { - return this.service.queueScan(id, dto); - } - - @Post(':id/removeOffline') - @HttpCode(HttpStatus.NO_CONTENT) - @Authenticated({ admin: true }) - removeOfflineFiles(@Param() { id }: UUIDParamDto) { - return this.service.queueRemoveOffline(id); + @Authenticated({ permission: Permission.LIBRARY_UPDATE, admin: true }) + scanLibrary(@Param() { id }: UUIDParamDto) { + return this.service.queueScan(id); } } diff --git a/server/src/dtos/asset-media.dto.ts b/server/src/dtos/asset-media.dto.ts index e9e346c4cb593..c62857da65042 100644 --- a/server/src/dtos/asset-media.dto.ts +++ b/server/src/dtos/asset-media.dto.ts @@ -56,9 +56,6 @@ export class AssetMediaCreateDto extends AssetMediaBase { @ValidateBoolean({ optional: true }) isVisible?: boolean; - @ValidateBoolean({ optional: true }) - isOffline?: boolean; - @ValidateUUID({ optional: true }) livePhotoVideoId?: string; diff --git a/server/src/dtos/library.dto.ts b/server/src/dtos/library.dto.ts index c2c3ac9d27546..7fb363dd9a5e1 100644 --- a/server/src/dtos/library.dto.ts +++ b/server/src/dtos/library.dto.ts @@ -1,7 +1,7 @@ import { ApiProperty } from '@nestjs/swagger'; import { ArrayMaxSize, ArrayUnique, IsNotEmpty, IsString } from 'class-validator'; import { LibraryEntity } from 'src/entities/library.entity'; -import { Optional, ValidateBoolean, ValidateUUID } from 'src/validation'; +import { Optional, ValidateUUID } from 'src/validation'; export class CreateLibraryDto { @ValidateUUID() @@ -89,14 +89,6 @@ export class LibrarySearchDto { userId?: string; } -export class ScanLibraryDto { - @ValidateBoolean({ optional: true }) - refreshModifiedFiles?: boolean; - - @ValidateBoolean({ optional: true }) - refreshAllFiles?: boolean; -} - export class LibraryResponseDto { id!: string; ownerId!: string; diff --git a/server/src/interfaces/asset.interface.ts b/server/src/interfaces/asset.interface.ts index 387fa27185fc6..c6808e3aa87a6 100644 --- a/server/src/interfaces/asset.interface.ts +++ b/server/src/interfaces/asset.interface.ts @@ -36,8 +36,6 @@ export enum WithoutProperty { export enum WithProperty { SIDECAR = 'sidecar', - IS_ONLINE = 'isOnline', - IS_OFFLINE = 'isOffline', } export enum TimeBucketSize { @@ -176,7 +174,6 @@ export interface IAssetRepository { ): Paginated; getRandom(userIds: string[], count: number): Promise; getLastUpdatedAssetForAlbumId(albumId: string): Promise; - getExternalLibraryAssetPaths(pagination: PaginationOptions, libraryId: string): Paginated; getByLibraryIdAndOriginalPath(libraryId: string, originalPath: string): Promise; deleteAll(ownerId: string): Promise; getAll(pagination: PaginationOptions, options?: AssetSearchOptions): Paginated; diff --git a/server/src/interfaces/job.interface.ts b/server/src/interfaces/job.interface.ts index 3e7b0b9d08140..8b6e2c289bd28 100644 --- a/server/src/interfaces/job.interface.ts +++ b/server/src/interfaces/job.interface.ts @@ -76,12 +76,12 @@ export enum JobName { FACIAL_RECOGNITION = 'facial-recognition', // library management - LIBRARY_SCAN = 'library-refresh', - LIBRARY_SCAN_ASSET = 'library-refresh-asset', - LIBRARY_REMOVE_OFFLINE = 'library-remove-offline', - LIBRARY_CHECK_OFFLINE = 'library-check-offline', + LIBRARY_QUEUE_SYNC_FILES = 'library-queue-sync-files', + LIBRARY_QUEUE_SYNC_ASSETS = 'library-queue-sync-assets', + LIBRARY_SYNC_FILE = 'library-sync-file', + LIBRARY_SYNC_ASSET = 'library-sync-asset', LIBRARY_DELETE = 'library-delete', - LIBRARY_QUEUE_SCAN_ALL = 'library-queue-all-refresh', + LIBRARY_QUEUE_SYNC_ALL = 'library-queue-sync-all', LIBRARY_QUEUE_CLEANUP = 'library-queue-cleanup', // cleanup @@ -137,16 +137,11 @@ export interface ILibraryFileJob extends IEntityJob { assetPath: string; } -export interface ILibraryOfflineJob extends IEntityJob { +export interface ILibraryAssetJob extends IEntityJob { importPaths: string[]; exclusionPatterns: string[]; } -export interface ILibraryRefreshJob extends IEntityJob { - refreshModifiedFiles: boolean; - refreshAllFiles: boolean; -} - export interface IBulkEntityJob extends IBaseJob { ids: string[]; } @@ -277,12 +272,12 @@ export type JobItem = | { name: JobName.ASSET_DELETION_CHECK; data?: IBaseJob } // Library Management - | { name: JobName.LIBRARY_SCAN_ASSET; data: ILibraryFileJob } - | { name: JobName.LIBRARY_SCAN; data: ILibraryRefreshJob } - | { name: JobName.LIBRARY_REMOVE_OFFLINE; data: IEntityJob } + | { name: JobName.LIBRARY_SYNC_FILE; data: ILibraryFileJob } + | { name: JobName.LIBRARY_QUEUE_SYNC_FILES; data: IEntityJob } + | { name: JobName.LIBRARY_QUEUE_SYNC_ASSETS; data: IEntityJob } + | { name: JobName.LIBRARY_SYNC_ASSET; data: IEntityJob } | { name: JobName.LIBRARY_DELETE; data: IEntityJob } - | { name: JobName.LIBRARY_QUEUE_SCAN_ALL; data: IBaseJob } - | { name: JobName.LIBRARY_CHECK_OFFLINE; data: IEntityJob } + | { name: JobName.LIBRARY_QUEUE_SYNC_ALL; data?: IBaseJob } | { name: JobName.LIBRARY_QUEUE_CLEANUP; data: IBaseJob } // Notification diff --git a/server/src/queries/asset.repository.sql b/server/src/queries/asset.repository.sql index 5b5730717910f..69309325848ed 100644 --- a/server/src/queries/asset.repository.sql +++ b/server/src/queries/asset.repository.sql @@ -268,35 +268,6 @@ DELETE FROM "assets" WHERE "ownerId" = $1 --- AssetRepository.getExternalLibraryAssetPaths -SELECT DISTINCT - "distinctAlias"."AssetEntity_id" AS "ids_AssetEntity_id" -FROM - ( - SELECT - "AssetEntity"."id" AS "AssetEntity_id", - "AssetEntity"."originalPath" AS "AssetEntity_originalPath", - "AssetEntity"."isOffline" AS "AssetEntity_isOffline" - FROM - "assets" "AssetEntity" - LEFT JOIN "libraries" "AssetEntity__AssetEntity_library" ON "AssetEntity__AssetEntity_library"."id" = "AssetEntity"."libraryId" - AND ( - "AssetEntity__AssetEntity_library"."deletedAt" IS NULL - ) - WHERE - ( - ( - ((("AssetEntity__AssetEntity_library"."id" = $1))) - AND ("AssetEntity"."isExternal" = $2) - ) - ) - AND ("AssetEntity"."deletedAt" IS NULL) - ) "distinctAlias" -ORDER BY - "AssetEntity_id" ASC -LIMIT - 2 - -- AssetRepository.getByLibraryIdAndOriginalPath SELECT DISTINCT "distinctAlias"."AssetEntity_id" AS "ids_AssetEntity_id" @@ -366,18 +337,6 @@ WHERE AND "originalPath" = path ); --- AssetRepository.updateOfflineLibraryAssets -UPDATE "assets" -SET - "isOffline" = $1, - "updatedAt" = CURRENT_TIMESTAMP -WHERE - ( - "libraryId" = $2 - AND NOT ("originalPath" IN ($3)) - AND "isOffline" = $4 - ) - -- AssetRepository.getAllByDeviceId SELECT "AssetEntity"."deviceAssetId" AS "AssetEntity_deviceAssetId", diff --git a/server/src/repositories/asset.repository.ts b/server/src/repositories/asset.repository.ts index 4ec5523df11b0..43e765d00b678 100644 --- a/server/src/repositories/asset.repository.ts +++ b/server/src/repositories/asset.repository.ts @@ -13,7 +13,6 @@ import { AssetDeltaSyncOptions, AssetExploreFieldOptions, AssetFullSyncOptions, - AssetPathEntity, AssetStats, AssetStatsOptions, AssetUpdateAllOptions, @@ -177,14 +176,6 @@ export class AssetRepository implements IAssetRepository { return this.getAll(pagination, { ...options, userIds: [userId] }); } - @GenerateSql({ params: [{ take: 1, skip: 0 }, DummyValue.UUID] }) - getExternalLibraryAssetPaths(pagination: PaginationOptions, libraryId: string): Paginated { - return paginate(this.repository, pagination, { - select: { id: true, originalPath: true, isOffline: true }, - where: { library: { id: libraryId }, isExternal: true }, - }); - } - @GenerateSql({ params: [DummyValue.UUID, DummyValue.STRING] }) getByLibraryIdAndOriginalPath(libraryId: string, originalPath: string): Promise { return this.repository.findOne({ @@ -198,24 +189,16 @@ export class AssetRepository implements IAssetRepository { async getPathsNotInLibrary(libraryId: string, originalPaths: string[]): Promise { const result = await this.repository.query( ` - WITH paths AS (SELECT unnest($2::text[]) AS path) - SELECT path FROM paths - WHERE NOT EXISTS (SELECT 1 FROM assets WHERE "libraryId" = $1 AND "originalPath" = path); - `, + WITH paths AS (SELECT unnest($2::text[]) AS path) + SELECT path + FROM paths + WHERE NOT EXISTS (SELECT 1 FROM assets WHERE "libraryId" = $1 AND "originalPath" = path); + `, [libraryId, originalPaths], ); return result.map((row: { path: string }) => row.path); } - @GenerateSql({ params: [DummyValue.UUID, [DummyValue.STRING]] }) - @ChunkedArray({ paramIndex: 1 }) - async updateOfflineLibraryAssets(libraryId: string, originalPaths: string[]): Promise { - await this.repository.update( - { library: { id: libraryId }, originalPath: Not(In(originalPaths)), isOffline: false }, - { isOffline: true }, - ); - } - getAll(pagination: PaginationOptions, options: AssetSearchOptions = {}): Paginated { let builder = this.repository.createQueryBuilder('asset').leftJoinAndSelect('asset.files', 'files'); builder = searchAssetBuilder(builder, options); @@ -373,12 +356,10 @@ export class AssetRepository implements IAssetRepository { } @GenerateSql( - ...Object.values(WithProperty) - .filter((property) => property !== WithProperty.IS_OFFLINE && property !== WithProperty.IS_ONLINE) - .map((property) => ({ - name: property, - params: [DummyValue.PAGINATION, property], - })), + ...Object.values(WithProperty).map((property) => ({ + name: property, + params: [DummyValue.PAGINATION, property], + })), ) getWithout(pagination: PaginationOptions, property: WithoutProperty): Paginated { let relations: FindOptionsRelations = {}; @@ -531,26 +512,16 @@ export class AssetRepository implements IAssetRepository { where = [{ sidecarPath: Not(IsNull()), isVisible: true }]; break; } - case WithProperty.IS_OFFLINE: { - if (!libraryId) { - throw new Error('Library id is required when finding offline assets'); - } - where = [{ isOffline: true, libraryId }]; - break; - } - case WithProperty.IS_ONLINE: { - if (!libraryId) { - throw new Error('Library id is required when finding online assets'); - } - where = [{ isOffline: false, libraryId }]; - break; - } default: { throw new Error(`Invalid getWith property: ${property}`); } } + if (libraryId) { + where = [{ ...where, libraryId }]; + } + return paginate(this.repository, pagination, { where, withDeleted, @@ -750,7 +721,10 @@ export class AssetRepository implements IAssetRepository { builder.andWhere(`asset.deletedAt ${options.isTrashed ? 'IS NOT NULL' : 'IS NULL'}`).withDeleted(); if (options.isTrashed) { - builder.andWhere('asset.status = :status', { status: AssetStatus.TRASHED }); + // TODO: Temporarily inverted to support showing offline assets in the trash queries. + // Once offline assets are handled in a separate screen, this should be set back to status = TRASHED + // and the offline screens should use a separate isOffline = true parameter in the timeline query. + builder.andWhere('asset.status != :status', { status: AssetStatus.DELETED }); } } diff --git a/server/src/repositories/job.repository.ts b/server/src/repositories/job.repository.ts index 2b4c1f6dc1b3b..cd4c7135be80f 100644 --- a/server/src/repositories/job.repository.ts +++ b/server/src/repositories/job.repository.ts @@ -79,12 +79,12 @@ export const JOBS_TO_QUEUE: Record = { [JobName.SIDECAR_WRITE]: QueueName.SIDECAR, // Library management - [JobName.LIBRARY_SCAN_ASSET]: QueueName.LIBRARY, - [JobName.LIBRARY_SCAN]: QueueName.LIBRARY, + [JobName.LIBRARY_SYNC_FILE]: QueueName.LIBRARY, + [JobName.LIBRARY_QUEUE_SYNC_FILES]: QueueName.LIBRARY, + [JobName.LIBRARY_QUEUE_SYNC_ASSETS]: QueueName.LIBRARY, [JobName.LIBRARY_DELETE]: QueueName.LIBRARY, - [JobName.LIBRARY_CHECK_OFFLINE]: QueueName.LIBRARY, - [JobName.LIBRARY_REMOVE_OFFLINE]: QueueName.LIBRARY, - [JobName.LIBRARY_QUEUE_SCAN_ALL]: QueueName.LIBRARY, + [JobName.LIBRARY_SYNC_ASSET]: QueueName.LIBRARY, + [JobName.LIBRARY_QUEUE_SYNC_ALL]: QueueName.LIBRARY, [JobName.LIBRARY_QUEUE_CLEANUP]: QueueName.LIBRARY, // Notification diff --git a/server/src/repositories/trash.repository.ts b/server/src/repositories/trash.repository.ts index 9e0f6728f19ed..d24f4f709afac 100644 --- a/server/src/repositories/trash.repository.ts +++ b/server/src/repositories/trash.repository.ts @@ -3,7 +3,7 @@ import { AssetEntity } from 'src/entities/asset.entity'; import { AssetStatus } from 'src/enum'; import { ITrashRepository } from 'src/interfaces/trash.interface'; import { Paginated, paginatedBuilder, PaginationOptions } from 'src/utils/pagination'; -import { In, IsNull, Not, Repository } from 'typeorm'; +import { In, Repository } from 'typeorm'; export class TrashRepository implements ITrashRepository { constructor(@InjectRepository(AssetEntity) private assetRepository: Repository) {} @@ -26,7 +26,7 @@ export class TrashRepository implements ITrashRepository { async restore(userId: string): Promise { const result = await this.assetRepository.update( - { ownerId: userId, deletedAt: Not(IsNull()) }, + { ownerId: userId, status: AssetStatus.TRASHED }, { status: AssetStatus.ACTIVE, deletedAt: null }, ); @@ -35,7 +35,7 @@ export class TrashRepository implements ITrashRepository { async empty(userId: string): Promise { const result = await this.assetRepository.update( - { ownerId: userId, deletedAt: Not(IsNull()), status: AssetStatus.TRASHED }, + { ownerId: userId, status: AssetStatus.TRASHED }, { status: AssetStatus.DELETED }, ); @@ -43,7 +43,10 @@ export class TrashRepository implements ITrashRepository { } async restoreAll(ids: string[]): Promise { - const result = await this.assetRepository.update({ id: In(ids) }, { status: AssetStatus.ACTIVE, deletedAt: null }); + const result = await this.assetRepository.update( + { id: In(ids), status: AssetStatus.TRASHED }, + { status: AssetStatus.ACTIVE, deletedAt: null }, + ); return result.affected ?? 0; } } diff --git a/server/src/services/asset-media.service.ts b/server/src/services/asset-media.service.ts index 5321c335a7e5d..d3dce323f0bb7 100644 --- a/server/src/services/asset-media.service.ts +++ b/server/src/services/asset-media.service.ts @@ -427,7 +427,6 @@ export class AssetMediaService { livePhotoVideoId: dto.livePhotoVideoId, originalFileName: file.originalName, sidecarPath: sidecarFile?.originalPath, - isOffline: dto.isOffline ?? false, }); if (sidecarFile) { diff --git a/server/src/services/job.service.ts b/server/src/services/job.service.ts index 5ed9f3202457b..f978f334108d1 100644 --- a/server/src/services/job.service.ts +++ b/server/src/services/job.service.ts @@ -164,7 +164,7 @@ export class JobService { } case QueueName.LIBRARY: { - return this.jobRepository.queue({ name: JobName.LIBRARY_QUEUE_SCAN_ALL, data: { force } }); + return this.jobRepository.queue({ name: JobName.LIBRARY_QUEUE_SYNC_ALL, data: { force } }); } default: { diff --git a/server/src/services/library.service.spec.ts b/server/src/services/library.service.spec.ts index 36bdfd05dc1db..8b14c76cbcc1d 100644 --- a/server/src/services/library.service.spec.ts +++ b/server/src/services/library.service.spec.ts @@ -10,9 +10,8 @@ import { ICryptoRepository } from 'src/interfaces/crypto.interface'; import { IDatabaseRepository } from 'src/interfaces/database.interface'; import { IJobRepository, + ILibraryAssetJob, ILibraryFileJob, - ILibraryOfflineJob, - ILibraryRefreshJob, JobName, JOBS_LIBRARY_PAGINATION_SIZE, JobStatus, @@ -37,6 +36,10 @@ import { makeMockWatcher, newStorageRepositoryMock } from 'test/repositories/sto import { newSystemMetadataRepositoryMock } from 'test/repositories/system-metadata.repository.mock'; import { Mocked, vitest } from 'vitest'; +async function* mockWalk() { + yield await Promise.resolve(['/data/user1/photo.jpg']); +} + describe(LibraryService.name, () => { let sut: LibraryService; @@ -91,7 +94,7 @@ describe(LibraryService.name, () => { enabled: true, cronExpression: '0 1 * * *', }, - watch: { enabled: false }, + watch: { enabled: true }, }, } as SystemConfig); @@ -163,102 +166,29 @@ describe(LibraryService.name, () => { describe('handleQueueAssetRefresh', () => { it('should queue refresh of a new asset', async () => { - const mockLibraryJob: ILibraryRefreshJob = { - id: libraryStub.externalLibrary1.id, - refreshModifiedFiles: false, - refreshAllFiles: false, - }; - assetMock.getWith.mockResolvedValue({ items: [], hasNextPage: false }); libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1); - // eslint-disable-next-line @typescript-eslint/require-await - storageMock.walk.mockImplementation(async function* generator() { - yield ['/data/user1/photo.jpg']; - }); - assetMock.getExternalLibraryAssetPaths.mockResolvedValue({ items: [], hasNextPage: false }); + storageMock.walk.mockImplementation(mockWalk); - await sut.handleQueueAssetRefresh(mockLibraryJob); + await sut.handleQueueSyncFiles({ id: libraryStub.externalLibrary1.id }); expect(jobMock.queueAll).toHaveBeenCalledWith([ { - name: JobName.LIBRARY_SCAN_ASSET, + name: JobName.LIBRARY_SYNC_FILE, data: { id: libraryStub.externalLibrary1.id, ownerId: libraryStub.externalLibrary1.owner.id, assetPath: '/data/user1/photo.jpg', - force: false, - }, - }, - ]); - }); - - it('should queue offline check of existing online assets', async () => { - const mockLibraryJob: ILibraryRefreshJob = { - id: libraryStub.externalLibrary1.id, - refreshModifiedFiles: false, - refreshAllFiles: false, - }; - - assetMock.getWith.mockResolvedValue({ items: [], hasNextPage: false }); - libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1); - storageMock.walk.mockImplementation(async function* generator() {}); - assetMock.getWith.mockResolvedValue({ items: [assetStub.external], hasNextPage: false }); - - await sut.handleQueueAssetRefresh(mockLibraryJob); - - expect(jobMock.queueAll).toHaveBeenCalledWith([ - { - name: JobName.LIBRARY_CHECK_OFFLINE, - data: { - id: assetStub.external.id, - importPaths: libraryStub.externalLibrary1.importPaths, - exclusionPatterns: [], }, }, ]); }); it("should fail when library can't be found", async () => { - const mockLibraryJob: ILibraryRefreshJob = { - id: libraryStub.externalLibrary1.id, - refreshModifiedFiles: false, - refreshAllFiles: false, - }; - libraryMock.get.mockResolvedValue(null); - await expect(sut.handleQueueAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.SKIPPED); - }); - - it('should force queue new assets', async () => { - const mockLibraryJob: ILibraryRefreshJob = { - id: libraryStub.externalLibrary1.id, - refreshModifiedFiles: false, - refreshAllFiles: true, - }; - - assetMock.getWith.mockResolvedValue({ items: [], hasNextPage: false }); - libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1); - // eslint-disable-next-line @typescript-eslint/require-await - storageMock.walk.mockImplementation(async function* generator() { - yield ['/data/user1/photo.jpg']; - }); - assetMock.getExternalLibraryAssetPaths.mockResolvedValue({ items: [], hasNextPage: false }); - - await sut.handleQueueAssetRefresh(mockLibraryJob); - - expect(jobMock.queueAll).toHaveBeenCalledWith([ - { - name: JobName.LIBRARY_SCAN_ASSET, - data: { - id: libraryStub.externalLibrary1.id, - ownerId: libraryStub.externalLibrary1.owner.id, - assetPath: '/data/user1/photo.jpg', - force: true, - }, - }, - ]); + await expect(sut.handleQueueSyncFiles({ id: libraryStub.externalLibrary1.id })).resolves.toBe(JobStatus.SKIPPED); }); it('should ignore import paths that do not exist', async () => { @@ -276,16 +206,9 @@ describe(LibraryService.name, () => { assetMock.getWith.mockResolvedValue({ items: [], hasNextPage: false }); - const mockLibraryJob: ILibraryRefreshJob = { - id: libraryStub.externalLibraryWithImportPaths1.id, - refreshModifiedFiles: false, - refreshAllFiles: false, - }; - libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1); - assetMock.getExternalLibraryAssetPaths.mockResolvedValue({ items: [], hasNextPage: false }); - await sut.handleQueueAssetRefresh(mockLibraryJob); + await sut.handleQueueSyncFiles({ id: libraryStub.externalLibraryWithImportPaths1.id }); expect(storageMock.walk).toHaveBeenCalledWith({ pathsToCrawl: [libraryStub.externalLibraryWithImportPaths1.importPaths[1]], @@ -296,9 +219,36 @@ describe(LibraryService.name, () => { }); }); - describe('handleOfflineCheck', () => { + describe('handleQueueRemoveDeleted', () => { + it('should queue online check of existing assets', async () => { + libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1); + storageMock.walk.mockImplementation(async function* generator() {}); + assetMock.getAll.mockResolvedValue({ items: [assetStub.external], hasNextPage: false }); + + await sut.handleQueueSyncAssets({ id: libraryStub.externalLibrary1.id }); + + expect(jobMock.queueAll).toHaveBeenCalledWith([ + { + name: JobName.LIBRARY_SYNC_ASSET, + data: { + id: assetStub.external.id, + importPaths: libraryStub.externalLibrary1.importPaths, + exclusionPatterns: [], + }, + }, + ]); + }); + + it("should fail when library can't be found", async () => { + libraryMock.get.mockResolvedValue(null); + + await expect(sut.handleQueueSyncAssets({ id: libraryStub.externalLibrary1.id })).resolves.toBe(JobStatus.SKIPPED); + }); + }); + + describe('handleSyncAsset', () => { it('should skip missing assets', async () => { - const mockAssetJob: ILibraryOfflineJob = { + const mockAssetJob: ILibraryAssetJob = { id: assetStub.external.id, importPaths: ['/'], exclusionPatterns: [], @@ -306,41 +256,31 @@ describe(LibraryService.name, () => { assetMock.getById.mockResolvedValue(null); - await expect(sut.handleOfflineCheck(mockAssetJob)).resolves.toBe(JobStatus.SKIPPED); + await expect(sut.handleSyncAsset(mockAssetJob)).resolves.toBe(JobStatus.SKIPPED); - expect(assetMock.update).not.toHaveBeenCalled(); - }); - - it('should do nothing with already-offline assets', async () => { - const mockAssetJob: ILibraryOfflineJob = { - id: assetStub.external.id, - importPaths: ['/'], - exclusionPatterns: [], - }; - - assetMock.getById.mockResolvedValue(assetStub.offline); - - await expect(sut.handleOfflineCheck(mockAssetJob)).resolves.toBe(JobStatus.SUCCESS); - - expect(assetMock.update).not.toHaveBeenCalled(); + expect(assetMock.remove).not.toHaveBeenCalled(); }); it('should offline assets no longer on disk', async () => { - const mockAssetJob: ILibraryOfflineJob = { + const mockAssetJob: ILibraryAssetJob = { id: assetStub.external.id, importPaths: ['/'], exclusionPatterns: [], }; assetMock.getById.mockResolvedValue(assetStub.external); + storageMock.stat.mockRejectedValue(new Error('ENOENT, no such file or directory')); - await expect(sut.handleOfflineCheck(mockAssetJob)).resolves.toBe(JobStatus.SUCCESS); + await expect(sut.handleSyncAsset(mockAssetJob)).resolves.toBe(JobStatus.SUCCESS); - expect(assetMock.update).toHaveBeenCalledWith({ id: assetStub.external.id, isOffline: true }); + expect(assetMock.updateAll).toHaveBeenCalledWith([assetStub.external.id], { + isOffline: true, + deletedAt: expect.any(Date), + }); }); it('should offline assets matching an exclusion pattern', async () => { - const mockAssetJob: ILibraryOfflineJob = { + const mockAssetJob: ILibraryAssetJob = { id: assetStub.external.id, importPaths: ['/'], exclusionPatterns: ['**/user1/**'], @@ -348,13 +288,15 @@ describe(LibraryService.name, () => { assetMock.getById.mockResolvedValue(assetStub.external); - await expect(sut.handleOfflineCheck(mockAssetJob)).resolves.toBe(JobStatus.SUCCESS); - - expect(assetMock.update).toHaveBeenCalledWith({ id: assetStub.external.id, isOffline: true }); + await expect(sut.handleSyncAsset(mockAssetJob)).resolves.toBe(JobStatus.SUCCESS); + expect(assetMock.updateAll).toHaveBeenCalledWith([assetStub.external.id], { + isOffline: true, + deletedAt: expect.any(Date), + }); }); it('should set assets outside of import paths as offline', async () => { - const mockAssetJob: ILibraryOfflineJob = { + const mockAssetJob: ILibraryAssetJob = { id: assetStub.external.id, importPaths: ['/data/user2'], exclusionPatterns: [], @@ -363,28 +305,74 @@ describe(LibraryService.name, () => { assetMock.getById.mockResolvedValue(assetStub.external); storageMock.checkFileExists.mockResolvedValue(true); - await expect(sut.handleOfflineCheck(mockAssetJob)).resolves.toBe(JobStatus.SUCCESS); + await expect(sut.handleSyncAsset(mockAssetJob)).resolves.toBe(JobStatus.SUCCESS); - expect(assetMock.update).toHaveBeenCalledWith({ id: assetStub.external.id, isOffline: true }); + expect(assetMock.updateAll).toHaveBeenCalledWith([assetStub.external.id], { + isOffline: true, + deletedAt: expect.any(Date), + }); }); it('should do nothing with online assets', async () => { - const mockAssetJob: ILibraryOfflineJob = { + const mockAssetJob: ILibraryAssetJob = { id: assetStub.external.id, importPaths: ['/'], exclusionPatterns: [], }; assetMock.getById.mockResolvedValue(assetStub.external); - storageMock.checkFileExists.mockResolvedValue(true); + storageMock.stat.mockResolvedValue({ mtime: assetStub.external.fileModifiedAt } as Stats); - await expect(sut.handleOfflineCheck(mockAssetJob)).resolves.toBe(JobStatus.SUCCESS); + await expect(sut.handleSyncAsset(mockAssetJob)).resolves.toBe(JobStatus.SUCCESS); - expect(assetMock.update).not.toHaveBeenCalled(); + expect(assetMock.updateAll).not.toHaveBeenCalled(); + }); + + it('should un-trash an asset previously marked as offline', async () => { + const mockAssetJob: ILibraryAssetJob = { + id: assetStub.external.id, + importPaths: ['/'], + exclusionPatterns: [], + }; + + assetMock.getById.mockResolvedValue(assetStub.trashedOffline); + storageMock.stat.mockResolvedValue({ mtime: assetStub.trashedOffline.fileModifiedAt } as Stats); + + await expect(sut.handleSyncAsset(mockAssetJob)).resolves.toBe(JobStatus.SUCCESS); + + expect(assetMock.updateAll).toHaveBeenCalledWith([assetStub.trashedOffline.id], { + deletedAt: null, + fileCreatedAt: assetStub.trashedOffline.fileModifiedAt, + fileModifiedAt: assetStub.trashedOffline.fileModifiedAt, + isOffline: false, + originalFileName: 'path.jpg', + }); }); }); - describe('handleAssetRefresh', () => { + it('should update file when mtime has changed', async () => { + const mockAssetJob: ILibraryAssetJob = { + id: assetStub.external.id, + importPaths: ['/'], + exclusionPatterns: [], + }; + + const newMTime = new Date(); + assetMock.getById.mockResolvedValue(assetStub.external); + storageMock.stat.mockResolvedValue({ mtime: newMTime } as Stats); + + await expect(sut.handleSyncAsset(mockAssetJob)).resolves.toBe(JobStatus.SUCCESS); + + expect(assetMock.updateAll).toHaveBeenCalledWith([assetStub.external.id], { + fileModifiedAt: newMTime, + fileCreatedAt: newMTime, + isOffline: false, + originalFileName: 'photo.jpg', + deletedAt: null, + }); + }); + + describe('handleSyncFile', () => { let mockUser: UserEntity; beforeEach(() => { @@ -397,42 +385,18 @@ describe(LibraryService.name, () => { } as Stats); }); - it('should reject an unknown file extension', async () => { - const mockLibraryJob: ILibraryFileJob = { - id: libraryStub.externalLibrary1.id, - ownerId: mockUser.id, - assetPath: '/data/user1/file.xyz', - force: false, - }; - - assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(null); - - await expect(sut.handleAssetRefresh(mockLibraryJob)).rejects.toBeInstanceOf(BadRequestException); - }); - - it('should reject an unknown file type', async () => { - const mockLibraryJob: ILibraryFileJob = { - id: libraryStub.externalLibrary1.id, - ownerId: mockUser.id, - assetPath: '/data/user1/file.xyz', - force: false, - }; - - await expect(sut.handleAssetRefresh(mockLibraryJob)).rejects.toBeInstanceOf(BadRequestException); - }); - - it('should add a new image', async () => { + it('should import a new asset', async () => { const mockLibraryJob: ILibraryFileJob = { id: libraryStub.externalLibrary1.id, ownerId: mockUser.id, assetPath: '/data/user1/photo.jpg', - force: false, }; assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(null); assetMock.create.mockResolvedValue(assetStub.image); + libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1); - await expect(sut.handleAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); + await expect(sut.handleSyncFile(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); expect(assetMock.create.mock.calls).toEqual([ [ @@ -467,19 +431,19 @@ describe(LibraryService.name, () => { ]); }); - it('should add a new image with sidecar', async () => { + it('should import a new asset with sidecar', async () => { const mockLibraryJob: ILibraryFileJob = { id: libraryStub.externalLibrary1.id, ownerId: mockUser.id, assetPath: '/data/user1/photo.jpg', - force: false, }; assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(null); assetMock.create.mockResolvedValue(assetStub.image); storageMock.checkFileExists.mockResolvedValue(true); + libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1); - await expect(sut.handleAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); + await expect(sut.handleSyncFile(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); expect(assetMock.create.mock.calls).toEqual([ [ @@ -514,18 +478,18 @@ describe(LibraryService.name, () => { ]); }); - it('should add a new video', async () => { + it('should import a new video', async () => { const mockLibraryJob: ILibraryFileJob = { id: libraryStub.externalLibrary1.id, ownerId: mockUser.id, assetPath: '/data/user1/video.mp4', - force: false, }; assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(null); assetMock.create.mockResolvedValue(assetStub.video); + libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1); - await expect(sut.handleAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); + await expect(sut.handleSyncFile(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); expect(assetMock.create.mock.calls).toEqual([ [ @@ -568,29 +532,27 @@ describe(LibraryService.name, () => { ]); }); - it('should not add an image to a soft deleted library', async () => { + it('should not import an asset to a soft deleted library', async () => { const mockLibraryJob: ILibraryFileJob = { id: libraryStub.externalLibrary1.id, ownerId: mockUser.id, assetPath: '/data/user1/photo.jpg', - force: false, }; assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(null); assetMock.create.mockResolvedValue(assetStub.image); libraryMock.get.mockResolvedValue({ ...libraryStub.externalLibrary1, deletedAt: new Date() }); - await expect(sut.handleAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.FAILED); + await expect(sut.handleSyncFile(mockLibraryJob)).resolves.toBe(JobStatus.FAILED); expect(assetMock.create.mock.calls).toEqual([]); }); - it('should not import an asset when mtime matches db asset', async () => { + it('should not refresh a file whose mtime matches existing asset', async () => { const mockLibraryJob: ILibraryFileJob = { id: libraryStub.externalLibrary1.id, ownerId: mockUser.id, assetPath: assetStub.hasFileExtension.originalPath, - force: false, }; storageMock.stat.mockResolvedValue({ @@ -601,190 +563,52 @@ describe(LibraryService.name, () => { assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.hasFileExtension); - await expect(sut.handleAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.SKIPPED); + await expect(sut.handleSyncFile(mockLibraryJob)).resolves.toBe(JobStatus.SKIPPED); expect(jobMock.queue).not.toHaveBeenCalled(); expect(jobMock.queueAll).not.toHaveBeenCalled(); }); - it('should import an asset when mtime differs from db asset', async () => { + it('should skip existing asset', async () => { const mockLibraryJob: ILibraryFileJob = { id: libraryStub.externalLibrary1.id, ownerId: mockUser.id, assetPath: '/data/user1/photo.jpg', - force: false, }; assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.image); - assetMock.create.mockResolvedValue(assetStub.image); - await expect(sut.handleAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); - - expect(jobMock.queue).toHaveBeenCalledWith({ - name: JobName.METADATA_EXTRACTION, - data: { - id: assetStub.image.id, - source: 'upload', - }, - }); - - expect(jobMock.queue).not.toHaveBeenCalledWith({ - name: JobName.VIDEO_CONVERSION, - data: { - id: assetStub.image.id, - }, - }); + await expect(sut.handleSyncFile(mockLibraryJob)).resolves.toBe(JobStatus.SKIPPED); }); - it('should import an asset that is missing a file extension', async () => { - // This tests for the case where the file extension is missing from the asset path. - // This happened in previous versions of Immich + it('should not refresh an asset trashed by user', async () => { const mockLibraryJob: ILibraryFileJob = { id: libraryStub.externalLibrary1.id, ownerId: mockUser.id, - assetPath: assetStub.missingFileExtension.originalPath, - force: false, - }; - - assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.missingFileExtension); - - await expect(sut.handleAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); - - expect(assetMock.updateAll).toHaveBeenCalledWith( - [assetStub.missingFileExtension.id], - expect.objectContaining({ originalFileName: 'photo.jpg' }), - ); - }); - - it('should set a missing asset to offline', async () => { - storageMock.stat.mockRejectedValue(new Error('Path not found')); - - const mockLibraryJob: ILibraryFileJob = { - id: assetStub.image.id, - ownerId: mockUser.id, - assetPath: '/data/user1/photo.jpg', - force: false, - }; - - assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.image); - assetMock.create.mockResolvedValue(assetStub.image); - - await expect(sut.handleAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); - - expect(assetMock.update).toHaveBeenCalledWith({ id: assetStub.image.id, isOffline: true }); - expect(jobMock.queue).not.toHaveBeenCalled(); - expect(jobMock.queueAll).not.toHaveBeenCalled(); - }); - - it('should online a previously-offline asset', async () => { - const mockLibraryJob: ILibraryFileJob = { - id: assetStub.offline.id, - ownerId: mockUser.id, - assetPath: '/data/user1/photo.jpg', - force: false, - }; - - assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.offline); - assetMock.create.mockResolvedValue(assetStub.offline); - - await expect(sut.handleAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); - - expect(assetMock.update).toHaveBeenCalledWith({ id: assetStub.offline.id, isOffline: false }); - - expect(jobMock.queue).toHaveBeenCalledWith({ - name: JobName.METADATA_EXTRACTION, - data: { - id: assetStub.offline.id, - source: 'upload', - }, - }); - - expect(jobMock.queue).not.toHaveBeenCalledWith({ - name: JobName.VIDEO_CONVERSION, - data: { - id: assetStub.offline.id, - }, - }); - }); - - it('should do nothing when mtime matches existing asset', async () => { - const mockLibraryJob: ILibraryFileJob = { - id: assetStub.image.id, - ownerId: assetStub.image.ownerId, - assetPath: '/data/user1/photo.jpg', - force: false, - }; - - assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.image); - assetMock.create.mockResolvedValue(assetStub.image); - - expect(assetMock.update).not.toHaveBeenCalled(); - - await expect(sut.handleAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); - }); - - it('should refresh an existing asset if forced', async () => { - const mockLibraryJob: ILibraryFileJob = { - id: assetStub.image.id, - ownerId: assetStub.hasFileExtension.ownerId, assetPath: assetStub.hasFileExtension.originalPath, - force: true, }; - assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.hasFileExtension); - assetMock.create.mockResolvedValue(assetStub.hasFileExtension); + assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.trashed); - await expect(sut.handleAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); + await expect(sut.handleSyncFile(mockLibraryJob)).resolves.toBe(JobStatus.SKIPPED); - expect(assetMock.updateAll).toHaveBeenCalledWith([assetStub.hasFileExtension.id], { - fileCreatedAt: new Date('2023-01-01'), - fileModifiedAt: new Date('2023-01-01'), - originalFileName: assetStub.hasFileExtension.originalFileName, - }); + expect(jobMock.queue).not.toHaveBeenCalled(); + expect(jobMock.queueAll).not.toHaveBeenCalled(); }); - it('should refresh an existing asset with modified mtime', async () => { - const filemtime = new Date(); - filemtime.setSeconds(assetStub.image.fileModifiedAt.getSeconds() + 10); - - const mockLibraryJob: ILibraryFileJob = { - id: libraryStub.externalLibrary1.id, - ownerId: userStub.admin.id, - assetPath: '/data/user1/photo.jpg', - force: false, - }; - - storageMock.stat.mockResolvedValue({ - size: 100, - mtime: filemtime, - ctime: new Date('2023-01-01'), - } as Stats); - - assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(null); - assetMock.create.mockResolvedValue(assetStub.image); - - await expect(sut.handleAssetRefresh(mockLibraryJob)).resolves.toBe(JobStatus.SUCCESS); - - expect(assetMock.create).toHaveBeenCalled(); - const createdAsset = assetMock.create.mock.calls[0][0]; - - expect(createdAsset.fileModifiedAt).toEqual(filemtime); - }); - - it('should throw error when asset does not exist', async () => { + it('should throw BadRequestException when asset does not exist', async () => { storageMock.stat.mockRejectedValue(new Error("ENOENT, no such file or directory '/data/user1/photo.jpg'")); const mockLibraryJob: ILibraryFileJob = { id: libraryStub.externalLibrary1.id, ownerId: userStub.admin.id, assetPath: '/data/user1/photo.jpg', - force: false, }; assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(null); assetMock.create.mockResolvedValue(assetStub.image); - await expect(sut.handleAssetRefresh(mockLibraryJob)).rejects.toBeInstanceOf(BadRequestException); + await expect(sut.handleSyncFile(mockLibraryJob)).resolves.toBe(JobStatus.FAILED); }); }); @@ -857,7 +681,6 @@ describe(LibraryService.name, () => { describe('getStatistics', () => { it('should return library statistics', async () => { - libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1); libraryMock.getStatistics.mockResolvedValue({ photos: 10, videos: 0, total: 10, usage: 1337 }); await expect(sut.getStatistics(libraryStub.externalLibrary1.id)).resolves.toEqual({ photos: 10, @@ -1092,12 +915,11 @@ describe(LibraryService.name, () => { expect(jobMock.queueAll).toHaveBeenCalledWith([ { - name: JobName.LIBRARY_SCAN_ASSET, + name: JobName.LIBRARY_SYNC_FILE, data: { id: libraryStub.externalLibraryWithImportPaths1.id, assetPath: '/foo/photo.jpg', ownerId: libraryStub.externalLibraryWithImportPaths1.owner.id, - force: false, }, }, ]); @@ -1114,30 +936,16 @@ describe(LibraryService.name, () => { expect(jobMock.queueAll).toHaveBeenCalledWith([ { - name: JobName.LIBRARY_SCAN_ASSET, + name: JobName.LIBRARY_SYNC_FILE, data: { id: libraryStub.externalLibraryWithImportPaths1.id, assetPath: '/foo/photo.jpg', ownerId: libraryStub.externalLibraryWithImportPaths1.owner.id, - force: false, }, }, ]); }); - it('should handle a file unlink event', async () => { - libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1); - libraryMock.getAll.mockResolvedValue([libraryStub.externalLibraryWithImportPaths1]); - assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.external); - storageMock.watch.mockImplementation( - makeMockWatcher({ items: [{ event: 'unlink', value: '/foo/photo.jpg' }] }), - ); - - await sut.watchAll(); - - expect(assetMock.update).toHaveBeenCalledWith({ id: assetStub.external.id, isOffline: true }); - }); - it('should handle an error event', async () => { libraryMock.get.mockResolvedValue(libraryStub.externalLibraryWithImportPaths1); assetMock.getByLibraryIdAndOriginalPath.mockResolvedValue(assetStub.external); @@ -1232,72 +1040,23 @@ describe(LibraryService.name, () => { }); describe('queueScan', () => { - it('should queue a library scan of external library', async () => { + it('should queue a library scan', async () => { libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1); - await sut.queueScan(libraryStub.externalLibrary1.id, {}); + await sut.queueScan(libraryStub.externalLibrary1.id); expect(jobMock.queue.mock.calls).toEqual([ [ { - name: JobName.LIBRARY_SCAN, + name: JobName.LIBRARY_QUEUE_SYNC_FILES, data: { id: libraryStub.externalLibrary1.id, - refreshModifiedFiles: false, - refreshAllFiles: false, }, }, ], - ]); - }); - - it('should queue a library scan of all modified assets', async () => { - libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1); - - await sut.queueScan(libraryStub.externalLibrary1.id, { refreshModifiedFiles: true }); - - expect(jobMock.queue.mock.calls).toEqual([ [ { - name: JobName.LIBRARY_SCAN, - data: { - id: libraryStub.externalLibrary1.id, - refreshModifiedFiles: true, - refreshAllFiles: false, - }, - }, - ], - ]); - }); - - it('should queue a forced library scan', async () => { - libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1); - - await sut.queueScan(libraryStub.externalLibrary1.id, { refreshAllFiles: true }); - - expect(jobMock.queue.mock.calls).toEqual([ - [ - { - name: JobName.LIBRARY_SCAN, - data: { - id: libraryStub.externalLibrary1.id, - refreshModifiedFiles: false, - refreshAllFiles: true, - }, - }, - ], - ]); - }); - }); - - describe('queueEmptyTrash', () => { - it('should queue the trash job', async () => { - await sut.queueRemoveOffline(libraryStub.externalLibrary1.id); - - expect(jobMock.queue.mock.calls).toEqual([ - [ - { - name: JobName.LIBRARY_REMOVE_OFFLINE, + name: JobName.LIBRARY_QUEUE_SYNC_ASSETS, data: { id: libraryStub.externalLibrary1.id, }, @@ -1311,7 +1070,7 @@ describe(LibraryService.name, () => { it('should queue the refresh job', async () => { libraryMock.getAll.mockResolvedValue([libraryStub.externalLibrary1]); - await expect(sut.handleQueueAllScan({})).resolves.toBe(JobStatus.SUCCESS); + await expect(sut.handleQueueSyncAll()).resolves.toBe(JobStatus.SUCCESS); expect(jobMock.queue.mock.calls).toEqual([ [ @@ -1323,48 +1082,32 @@ describe(LibraryService.name, () => { ]); expect(jobMock.queueAll).toHaveBeenCalledWith([ { - name: JobName.LIBRARY_SCAN, + name: JobName.LIBRARY_QUEUE_SYNC_FILES, data: { id: libraryStub.externalLibrary1.id, - refreshModifiedFiles: true, - refreshAllFiles: false, - }, - }, - ]); - }); - - it('should queue the force refresh job', async () => { - libraryMock.getAll.mockResolvedValue([libraryStub.externalLibrary1]); - - await expect(sut.handleQueueAllScan({ force: true })).resolves.toBe(JobStatus.SUCCESS); - - expect(jobMock.queue).toHaveBeenCalledWith({ - name: JobName.LIBRARY_QUEUE_CLEANUP, - data: {}, - }); - - expect(jobMock.queueAll).toHaveBeenCalledWith([ - { - name: JobName.LIBRARY_SCAN, - data: { - id: libraryStub.externalLibrary1.id, - refreshModifiedFiles: false, - refreshAllFiles: true, }, }, ]); }); }); - describe('handleRemoveOfflineFiles', () => { - it('should queue trash deletion jobs', async () => { - assetMock.getWith.mockResolvedValue({ items: [assetStub.image1], hasNextPage: false }); + describe('handleQueueAssetOfflineCheck', () => { + it('should queue removal jobs', async () => { + libraryMock.get.mockResolvedValue(libraryStub.externalLibrary1); + assetMock.getAll.mockResolvedValue({ items: [assetStub.image1], hasNextPage: false }); assetMock.getById.mockResolvedValue(assetStub.image1); - await expect(sut.handleRemoveOffline({ id: libraryStub.externalLibrary1.id })).resolves.toBe(JobStatus.SUCCESS); + await expect(sut.handleQueueSyncAssets({ id: libraryStub.externalLibrary1.id })).resolves.toBe(JobStatus.SUCCESS); expect(jobMock.queueAll).toHaveBeenCalledWith([ - { name: JobName.ASSET_DELETION, data: { id: assetStub.image1.id, deleteOnDisk: false } }, + { + name: JobName.LIBRARY_SYNC_ASSET, + data: { + id: assetStub.image1.id, + importPaths: libraryStub.externalLibrary1.importPaths, + exclusionPatterns: libraryStub.externalLibrary1.exclusionPatterns, + }, + }, ]); }); }); diff --git a/server/src/services/library.service.ts b/server/src/services/library.service.ts index 3dd81dd61377b..52b786089cbaf 100644 --- a/server/src/services/library.service.ts +++ b/server/src/services/library.service.ts @@ -1,6 +1,5 @@ import { BadRequestException, Inject, Injectable } from '@nestjs/common'; import { R_OK } from 'node:constants'; -import { Stats } from 'node:fs'; import path, { basename, parse } from 'node:path'; import picomatch from 'picomatch'; import { StorageCore } from 'src/cores/storage.core'; @@ -10,27 +9,26 @@ import { CreateLibraryDto, LibraryResponseDto, LibraryStatsResponseDto, - ScanLibraryDto, + mapLibrary, UpdateLibraryDto, ValidateLibraryDto, ValidateLibraryImportPathResponseDto, ValidateLibraryResponseDto, - mapLibrary, } from 'src/dtos/library.dto'; +import { AssetEntity } from 'src/entities/asset.entity'; +import { LibraryEntity } from 'src/entities/library.entity'; import { AssetType } from 'src/enum'; -import { IAssetRepository, WithProperty } from 'src/interfaces/asset.interface'; +import { IAssetRepository } from 'src/interfaces/asset.interface'; import { ICryptoRepository } from 'src/interfaces/crypto.interface'; import { DatabaseLock, IDatabaseRepository } from 'src/interfaces/database.interface'; import { ArgOf } from 'src/interfaces/event.interface'; import { - IBaseJob, IEntityJob, IJobRepository, + ILibraryAssetJob, ILibraryFileJob, - ILibraryOfflineJob, - ILibraryRefreshJob, - JOBS_LIBRARY_PAGINATION_SIZE, JobName, + JOBS_LIBRARY_PAGINATION_SIZE, JobStatus, } from 'src/interfaces/job.interface'; import { ILibraryRepository } from 'src/interfaces/library.interface'; @@ -78,11 +76,7 @@ export class LibraryService { this.jobRepository.addCronJob( 'libraryScan', scan.cronExpression, - () => - handlePromiseError( - this.jobRepository.queue({ name: JobName.LIBRARY_QUEUE_SCAN_ALL, data: { force: false } }), - this.logger, - ), + () => handlePromiseError(this.jobRepository.queue({ name: JobName.LIBRARY_QUEUE_SYNC_ALL }), this.logger), scan.enabled, ); @@ -143,7 +137,7 @@ export class LibraryService { const handler = async () => { this.logger.debug(`File add event received for ${path} in library ${library.id}}`); if (matcher(path)) { - await this.scanAssets(library.id, [path], library.ownerId, false); + await this.syncFiles(library, [path]); } }; return handlePromiseError(handler(), this.logger); @@ -151,9 +145,13 @@ export class LibraryService { onChange: (path) => { const handler = async () => { this.logger.debug(`Detected file change for ${path} in library ${library.id}`); + const asset = await this.assetRepository.getByLibraryIdAndOriginalPath(library.id, path); + if (asset) { + await this.syncAssets(library, [asset.id]); + } if (matcher(path)) { // Note: if the changed file was not previously imported, it will be imported now. - await this.scanAssets(library.id, [path], library.ownerId, false); + await this.syncFiles(library, [path]); } }; return handlePromiseError(handler(), this.logger); @@ -162,8 +160,8 @@ export class LibraryService { const handler = async () => { this.logger.debug(`Detected deleted file at ${path} in library ${library.id}`); const asset = await this.assetRepository.getByLibraryIdAndOriginalPath(library.id, path); - if (asset && matcher(path)) { - await this.assetRepository.update({ id: asset.id, isOffline: true }); + if (asset) { + await this.syncAssets(library, [asset.id]); } }; return handlePromiseError(handler(), this.logger); @@ -216,7 +214,7 @@ export class LibraryService { async getStatistics(id: string): Promise { const statistics = await this.repository.getStatistics(id); if (!statistics) { - throw new BadRequestException('Library not found'); + throw new BadRequestException(`Library ${id} not found`); } return statistics; } @@ -250,20 +248,28 @@ export class LibraryService { return mapLibrary(library); } - private async scanAssets(libraryId: string, assetPaths: string[], ownerId: string, force = false) { + private async syncFiles({ id, ownerId }: LibraryEntity, assetPaths: string[]) { await this.jobRepository.queueAll( assetPaths.map((assetPath) => ({ - name: JobName.LIBRARY_SCAN_ASSET, + name: JobName.LIBRARY_SYNC_FILE, data: { - id: libraryId, + id, assetPath, ownerId, - force, }, })), ); } + private async syncAssets({ importPaths, exclusionPatterns }: LibraryEntity, assetIds: string[]) { + await this.jobRepository.queueAll( + assetIds.map((assetId) => ({ + name: JobName.LIBRARY_SYNC_ASSET, + data: { id: assetId, importPaths, exclusionPatterns }, + })), + ); + } + private async validateImportPath(importPath: string): Promise { const validation = new ValidateLibraryImportPathResponseDto(); validation.importPath = importPath; @@ -366,258 +372,182 @@ export class LibraryService { return JobStatus.SUCCESS; } - async handleAssetRefresh(job: ILibraryFileJob): Promise { + async handleSyncFile(job: ILibraryFileJob): Promise { + // Only needs to handle new assets const assetPath = path.normalize(job.assetPath); - const existingAssetEntity = await this.assetRepository.getByLibraryIdAndOriginalPath(job.id, assetPath); - - let stats: Stats; - try { - stats = await this.storageRepository.stat(assetPath); - } catch (error: Error | any) { - // Can't access file, probably offline - if (existingAssetEntity) { - // Mark asset as offline - this.logger.debug(`Marking asset as offline: ${assetPath}`); - - await this.assetRepository.update({ id: existingAssetEntity.id, isOffline: true }); - return JobStatus.SUCCESS; - } else { - // File can't be accessed and does not already exist in db - throw new BadRequestException('Cannot access file', { cause: error }); - } - } - - let doImport = false; - let doRefresh = false; - - if (job.force) { - doRefresh = true; - } - - const originalFileName = parse(assetPath).base; - - if (!existingAssetEntity) { - // This asset is new to us, read it from disk - this.logger.debug(`Importing new asset: ${assetPath}`); - doImport = true; - } else if (stats.mtime.toISOString() !== existingAssetEntity.fileModifiedAt.toISOString()) { - // File modification time has changed since last time we checked, re-read from disk - this.logger.debug( - `File modification time has changed, re-importing asset: ${assetPath}. Old mtime: ${existingAssetEntity.fileModifiedAt}. New mtime: ${stats.mtime}`, - ); - doRefresh = true; - } else if (existingAssetEntity.originalFileName !== originalFileName) { - // TODO: We can likely remove this check in the second half of 2024 when all assets have likely been re-imported by all users - this.logger.debug( - `Asset is missing file extension, re-importing: ${assetPath}. Current incorrect filename: ${existingAssetEntity.originalFileName}.`, - ); - doRefresh = true; - } else if (!job.force && stats && !existingAssetEntity.isOffline) { - // Asset exists on disk and in db and mtime has not changed. Also, we are not forcing refresn. Therefore, do nothing - this.logger.debug(`Asset already exists in database and on disk, will not import: ${assetPath}`); - } - - if (stats && existingAssetEntity?.isOffline) { - // File was previously offline but is now online - this.logger.debug(`Marking previously-offline asset as online: ${assetPath}`); - await this.assetRepository.update({ id: existingAssetEntity.id, isOffline: false }); - doRefresh = true; - } - - if (!doImport && !doRefresh) { - // If we don't import, exit here + let asset = await this.assetRepository.getByLibraryIdAndOriginalPath(job.id, assetPath); + if (asset) { return JobStatus.SKIPPED; } - let assetType: AssetType; - - if (mimeTypes.isImage(assetPath)) { - assetType = AssetType.IMAGE; - } else if (mimeTypes.isVideo(assetPath)) { - assetType = AssetType.VIDEO; - } else { - throw new BadRequestException(`Unsupported file type ${assetPath}`); + let stat; + try { + stat = await this.storageRepository.stat(assetPath); + } catch (error: any) { + if (error.code === 'ENOENT') { + this.logger.error(`File not found: ${assetPath}`); + return JobStatus.SKIPPED; + } + this.logger.error(`Error reading file: ${assetPath}. Error: ${error}`); + return JobStatus.FAILED; } + this.logger.log(`Importing new library asset: ${assetPath}`); + + const library = await this.repository.get(job.id, true); + if (!library || library.deletedAt) { + this.logger.error('Cannot import asset into deleted library'); + return JobStatus.FAILED; + } + + // TODO: device asset id is deprecated, remove it + const deviceAssetId = `${basename(assetPath)}`.replaceAll(/\s+/g, ''); + + const pathHash = this.cryptoRepository.hashSha1(`path:${assetPath}`); + // TODO: doesn't xmp replace the file extension? Will need investigation let sidecarPath: string | null = null; if (await this.storageRepository.checkFileExists(`${assetPath}.xmp`, R_OK)) { sidecarPath = `${assetPath}.xmp`; } - // TODO: device asset id is deprecated, remove it - const deviceAssetId = `${basename(assetPath)}`.replaceAll(/\s+/g, ''); + const assetType = mimeTypes.isVideo(assetPath) ? AssetType.VIDEO : AssetType.IMAGE; - let assetId; - if (doImport) { - const library = await this.repository.get(job.id, true); - if (library?.deletedAt) { - this.logger.error('Cannot import asset into deleted library'); - return JobStatus.FAILED; - } + const mtime = stat.mtime; - const pathHash = this.cryptoRepository.hashSha1(`path:${assetPath}`); + asset = await this.assetRepository.create({ + ownerId: job.ownerId, + libraryId: job.id, + checksum: pathHash, + originalPath: assetPath, + deviceAssetId, + deviceId: 'Library Import', + fileCreatedAt: mtime, + fileModifiedAt: mtime, + localDateTime: mtime, + type: assetType, + originalFileName: parse(assetPath).base, - // TODO: In wait of refactoring the domain asset service, this function is just manually written like this - const addedAsset = await this.assetRepository.create({ - ownerId: job.ownerId, - libraryId: job.id, - checksum: pathHash, - originalPath: assetPath, - deviceAssetId, - deviceId: 'Library Import', - fileCreatedAt: stats.mtime, - fileModifiedAt: stats.mtime, - localDateTime: stats.mtime, - type: assetType, - originalFileName, - sidecarPath, - isExternal: true, - }); - assetId = addedAsset.id; - } else if (doRefresh && existingAssetEntity) { - assetId = existingAssetEntity.id; - await this.assetRepository.updateAll([existingAssetEntity.id], { - fileCreatedAt: stats.mtime, - fileModifiedAt: stats.mtime, - originalFileName, - }); - } else { - // Not importing and not refreshing, do nothing - return JobStatus.SKIPPED; - } + sidecarPath, + isExternal: true, + }); - this.logger.debug(`Queueing metadata extraction for: ${assetPath}`); - - await this.jobRepository.queue({ name: JobName.METADATA_EXTRACTION, data: { id: assetId, source: 'upload' } }); - - if (assetType === AssetType.VIDEO) { - await this.jobRepository.queue({ name: JobName.VIDEO_CONVERSION, data: { id: assetId } }); - } + await this.queuePostSyncJobs(asset); return JobStatus.SUCCESS; } - async queueScan(id: string, dto: ScanLibraryDto) { + async queuePostSyncJobs(asset: AssetEntity) { + this.logger.debug(`Queueing metadata extraction for: ${asset.originalPath}`); + + await this.jobRepository.queue({ name: JobName.METADATA_EXTRACTION, data: { id: asset.id, source: 'upload' } }); + + if (asset.type === AssetType.VIDEO) { + await this.jobRepository.queue({ name: JobName.VIDEO_CONVERSION, data: { id: asset.id } }); + } + } + + async queueScan(id: string) { await this.findOrFail(id); await this.jobRepository.queue({ - name: JobName.LIBRARY_SCAN, + name: JobName.LIBRARY_QUEUE_SYNC_FILES, data: { id, - refreshModifiedFiles: dto.refreshModifiedFiles ?? false, - refreshAllFiles: dto.refreshAllFiles ?? false, }, }); + await this.jobRepository.queue({ name: JobName.LIBRARY_QUEUE_SYNC_ASSETS, data: { id } }); } - async queueRemoveOffline(id: string) { - this.logger.verbose(`Queueing offline file removal from library ${id}`); - await this.jobRepository.queue({ name: JobName.LIBRARY_REMOVE_OFFLINE, data: { id } }); - } - - async handleQueueAllScan(job: IBaseJob): Promise { - this.logger.debug(`Refreshing all external libraries: force=${job.force}`); + async handleQueueSyncAll(): Promise { + this.logger.debug(`Refreshing all external libraries`); await this.jobRepository.queue({ name: JobName.LIBRARY_QUEUE_CLEANUP, data: {} }); const libraries = await this.repository.getAll(true); await this.jobRepository.queueAll( libraries.map((library) => ({ - name: JobName.LIBRARY_SCAN, + name: JobName.LIBRARY_QUEUE_SYNC_FILES, + data: { + id: library.id, + }, + })), + ); + await this.jobRepository.queueAll( + libraries.map((library) => ({ + name: JobName.LIBRARY_QUEUE_SYNC_ASSETS, data: { id: library.id, - refreshModifiedFiles: !job.force, - refreshAllFiles: job.force ?? false, }, })), ); return JobStatus.SUCCESS; } - async handleOfflineCheck(job: ILibraryOfflineJob): Promise { + async handleSyncAsset(job: ILibraryAssetJob): Promise { const asset = await this.assetRepository.getById(job.id); - if (!asset) { - // Asset is no longer in the database, skip return JobStatus.SKIPPED; } - if (asset.isOffline) { - this.logger.verbose(`Asset is already offline: ${asset.originalPath}`); - return JobStatus.SUCCESS; - } + const markOffline = async (explanation: string) => { + if (!asset.isOffline) { + this.logger.debug(`${explanation}, removing: ${asset.originalPath}`); + await this.assetRepository.updateAll([asset.id], { isOffline: true, deletedAt: new Date() }); + } + }; const isInPath = job.importPaths.find((path) => asset.originalPath.startsWith(path)); if (!isInPath) { - this.logger.debug(`Asset is no longer in an import path, marking offline: ${asset.originalPath}`); - await this.assetRepository.update({ id: asset.id, isOffline: true }); + await markOffline('Asset is no longer in an import path'); return JobStatus.SUCCESS; } const isExcluded = job.exclusionPatterns.some((pattern) => picomatch.isMatch(asset.originalPath, pattern)); if (isExcluded) { - this.logger.debug(`Asset is covered by an exclusion pattern, marking offline: ${asset.originalPath}`); - await this.assetRepository.update({ id: asset.id, isOffline: true }); + await markOffline('Asset is covered by an exclusion pattern'); return JobStatus.SUCCESS; } - const fileExists = await this.storageRepository.checkFileExists(asset.originalPath, R_OK); - if (!fileExists) { - this.logger.debug(`Asset is no longer found on disk, marking offline: ${asset.originalPath}`); - await this.assetRepository.update({ id: asset.id, isOffline: true }); + let stat; + try { + stat = await this.storageRepository.stat(asset.originalPath); + } catch { + await markOffline('Asset is no longer on disk or is inaccessible because of permissions'); return JobStatus.SUCCESS; } - this.logger.verbose( - `Asset is found on disk, not covered by an exclusion pattern, and is in an import path, keeping online: ${asset.originalPath}`, - ); + const mtime = stat.mtime; + const isAssetModified = mtime.toISOString() !== asset.fileModifiedAt.toISOString(); + if (asset.isOffline || isAssetModified) { + this.logger.debug(`Asset was offline or modified, updating asset record ${asset.originalPath}`); + //TODO: When we have asset status, we need to leave deletedAt as is when status is trashed + await this.assetRepository.updateAll([asset.id], { + isOffline: false, + deletedAt: null, + fileCreatedAt: mtime, + fileModifiedAt: mtime, + originalFileName: parse(asset.originalPath).base, + }); + } + + if (isAssetModified) { + this.logger.debug(`Asset was modified, queuing metadata extraction for: ${asset.originalPath}`); + await this.queuePostSyncJobs(asset); + } return JobStatus.SUCCESS; } - async handleRemoveOffline(job: IEntityJob): Promise { - this.logger.debug(`Removing offline assets for library ${job.id}`); - - const assetPagination = usePagination(JOBS_LIBRARY_PAGINATION_SIZE, (pagination) => - this.assetRepository.getWith(pagination, WithProperty.IS_OFFLINE, job.id, true), - ); - - let offlineAssets = 0; - for await (const assets of assetPagination) { - offlineAssets += assets.length; - if (assets.length > 0) { - this.logger.debug(`Discovered ${offlineAssets} offline assets in library ${job.id}`); - await this.jobRepository.queueAll( - assets.map((asset) => ({ - name: JobName.ASSET_DELETION, - data: { - id: asset.id, - deleteOnDisk: false, - }, - })), - ); - this.logger.verbose(`Queued deletion of ${assets.length} offline assets in library ${job.id}`); - } - } - - if (offlineAssets) { - this.logger.debug(`Finished queueing deletion of ${offlineAssets} offline assets for library ${job.id}`); - } else { - this.logger.debug(`Found no offline assets to delete from library ${job.id}`); - } - - return JobStatus.SUCCESS; - } - - async handleQueueAssetRefresh(job: ILibraryRefreshJob): Promise { + async handleQueueSyncFiles(job: IEntityJob): Promise { const library = await this.repository.get(job.id); if (!library) { + this.logger.debug(`Library ${job.id} not found, skipping refresh`); return JobStatus.SKIPPED; } - this.logger.log(`Refreshing library ${library.id}`); + this.logger.log(`Refreshing library ${library.id} for new assets`); const validImportPaths: string[] = []; @@ -630,55 +560,66 @@ export class LibraryService { } } - if (validImportPaths.length === 0) { + if (validImportPaths) { + const assetsOnDisk = this.storageRepository.walk({ + pathsToCrawl: validImportPaths, + includeHidden: false, + exclusionPatterns: library.exclusionPatterns, + take: JOBS_LIBRARY_PAGINATION_SIZE, + }); + + let count = 0; + + for await (const assetBatch of assetsOnDisk) { + count += assetBatch.length; + this.logger.debug(`Discovered ${count} asset(s) on disk for library ${library.id}...`); + await this.syncFiles(library, assetBatch); + this.logger.verbose(`Queued scan of ${assetBatch.length} crawled asset(s) in library ${library.id}...`); + } + + if (count > 0) { + this.logger.debug(`Finished queueing scan of ${count} assets on disk for library ${library.id}`); + } else { + this.logger.debug(`No non-excluded assets found in any import path for library ${library.id}`); + } + } else { this.logger.warn(`No valid import paths found for library ${library.id}`); } - const assetsOnDisk = this.storageRepository.walk({ - pathsToCrawl: validImportPaths, - includeHidden: false, - exclusionPatterns: library.exclusionPatterns, - take: JOBS_LIBRARY_PAGINATION_SIZE, - }); + await this.repository.update({ id: job.id, refreshedAt: new Date() }); - let crawledAssets = 0; + return JobStatus.SUCCESS; + } - for await (const assetBatch of assetsOnDisk) { - crawledAssets += assetBatch.length; - this.logger.debug(`Discovered ${crawledAssets} asset(s) on disk for library ${library.id}...`); - await this.scanAssets(job.id, assetBatch, library.ownerId, job.refreshAllFiles ?? false); - this.logger.verbose(`Queued scan of ${assetBatch.length} crawled asset(s) in library ${library.id}...`); + async handleQueueSyncAssets(job: IEntityJob): Promise { + const library = await this.repository.get(job.id); + if (!library) { + return JobStatus.SKIPPED; } - if (crawledAssets) { - this.logger.debug(`Finished queueing scan of ${crawledAssets} assets on disk for library ${library.id}`); - } else { - this.logger.debug(`No non-excluded assets found in any import path for library ${library.id}`); - } + this.logger.log(`Scanning library ${library.id} for removed assets`); const onlineAssets = usePagination(JOBS_LIBRARY_PAGINATION_SIZE, (pagination) => - this.assetRepository.getWith(pagination, WithProperty.IS_ONLINE, job.id), + this.assetRepository.getAll(pagination, { libraryId: job.id }), ); - let onlineAssetCount = 0; + let assetCount = 0; for await (const assets of onlineAssets) { - onlineAssetCount += assets.length; - this.logger.debug(`Discovered ${onlineAssetCount} asset(s) in library ${library.id}...`); + assetCount += assets.length; + this.logger.debug(`Discovered ${assetCount} asset(s) in library ${library.id}...`); await this.jobRepository.queueAll( assets.map((asset) => ({ - name: JobName.LIBRARY_CHECK_OFFLINE, - data: { id: asset.id, importPaths: validImportPaths, exclusionPatterns: library.exclusionPatterns }, + name: JobName.LIBRARY_SYNC_ASSET, + data: { id: asset.id, importPaths: library.importPaths, exclusionPatterns: library.exclusionPatterns }, })), ); - this.logger.debug(`Queued online check of ${assets.length} asset(s) in library ${library.id}...`); + this.logger.debug(`Queued check of ${assets.length} asset(s) in library ${library.id}...`); } - if (onlineAssetCount) { - this.logger.log(`Finished queueing online check of ${onlineAssetCount} assets for library ${library.id}`); + if (assetCount) { + this.logger.log(`Finished queueing check of ${assetCount} assets for library ${library.id}`); } - await this.repository.update({ id: job.id, refreshedAt: new Date() }); - return JobStatus.SUCCESS; } diff --git a/server/src/services/microservices.service.ts b/server/src/services/microservices.service.ts index 25bfc0fdd29e1..80f1b2be415aa 100644 --- a/server/src/services/microservices.service.ts +++ b/server/src/services/microservices.service.ts @@ -86,12 +86,12 @@ export class MicroservicesService { [JobName.SIDECAR_DISCOVERY]: (data) => this.metadataService.handleSidecarDiscovery(data), [JobName.SIDECAR_SYNC]: (data) => this.metadataService.handleSidecarSync(data), [JobName.SIDECAR_WRITE]: (data) => this.metadataService.handleSidecarWrite(data), - [JobName.LIBRARY_SCAN_ASSET]: (data) => this.libraryService.handleAssetRefresh(data), - [JobName.LIBRARY_SCAN]: (data) => this.libraryService.handleQueueAssetRefresh(data), + [JobName.LIBRARY_QUEUE_SYNC_ALL]: () => this.libraryService.handleQueueSyncAll(), + [JobName.LIBRARY_QUEUE_SYNC_FILES]: (data) => this.libraryService.handleQueueSyncFiles(data), //Queues all files paths on disk + [JobName.LIBRARY_SYNC_FILE]: (data) => this.libraryService.handleSyncFile(data), //Handles a single path on disk //Watcher calls for new files + [JobName.LIBRARY_QUEUE_SYNC_ASSETS]: (data) => this.libraryService.handleQueueSyncAssets(data), //Queues all library assets + [JobName.LIBRARY_SYNC_ASSET]: (data) => this.libraryService.handleSyncAsset(data), //Handles all library assets // Watcher calls for unlink and changed [JobName.LIBRARY_DELETE]: (data) => this.libraryService.handleDeleteLibrary(data), - [JobName.LIBRARY_CHECK_OFFLINE]: (data) => this.libraryService.handleOfflineCheck(data), - [JobName.LIBRARY_REMOVE_OFFLINE]: (data) => this.libraryService.handleRemoveOffline(data), - [JobName.LIBRARY_QUEUE_SCAN_ALL]: (data) => this.libraryService.handleQueueAllScan(data), [JobName.LIBRARY_QUEUE_CLEANUP]: () => this.libraryService.handleQueueCleanup(), [JobName.SEND_EMAIL]: (data) => this.notificationService.handleSendEmail(data), [JobName.NOTIFY_ALBUM_INVITE]: (data) => this.notificationService.handleAlbumInvite(data), diff --git a/server/src/services/trash.service.spec.ts b/server/src/services/trash.service.spec.ts index 87821f028a08a..d0c719ae48e73 100644 --- a/server/src/services/trash.service.spec.ts +++ b/server/src/services/trash.service.spec.ts @@ -67,7 +67,7 @@ describe(TrashService.name, () => { }); it('should restore', async () => { - trashMock.getDeletedIds.mockResolvedValue({ items: ['asset-id'], hasNextPage: false }); + trashMock.getDeletedIds.mockResolvedValue({ items: ['asset-1'], hasNextPage: false }); trashMock.restore.mockResolvedValue(1); await expect(sut.restore(authStub.user1)).resolves.toEqual({ count: 1 }); expect(trashMock.restore).toHaveBeenCalledWith('user-id'); @@ -83,7 +83,7 @@ describe(TrashService.name, () => { }); it('should empty the trash', async () => { - trashMock.getDeletedIds.mockResolvedValue({ items: ['asset-id'], hasNextPage: false }); + trashMock.getDeletedIds.mockResolvedValue({ items: ['asset-1'], hasNextPage: false }); trashMock.empty.mockResolvedValue(1); await expect(sut.empty(authStub.user1)).resolves.toEqual({ count: 1 }); expect(trashMock.empty).toHaveBeenCalledWith('user-id'); diff --git a/server/src/utils/database.ts b/server/src/utils/database.ts index f3232eb78bb2e..5f4577f4df59b 100644 --- a/server/src/utils/database.ts +++ b/server/src/utils/database.ts @@ -80,7 +80,7 @@ export function searchAssetBuilder( }); } - const status = _.pick(options, ['isFavorite', 'isOffline', 'isVisible', 'type']); + const status = _.pick(options, ['isFavorite', 'isVisible', 'type']); const { isArchived, isEncoded, diff --git a/server/test/fixtures/asset.stub.ts b/server/test/fixtures/asset.stub.ts index a9b5167909db8..119c0b6e5ab76 100644 --- a/server/test/fixtures/asset.stub.ts +++ b/server/test/fixtures/asset.stub.ts @@ -70,9 +70,9 @@ export const assetStub = { faces: [], sidecarPath: null, deletedAt: null, - isOffline: false, isExternal: false, duplicateId: null, + isOffline: false, }), noWebpPath: Object.freeze({ @@ -104,13 +104,13 @@ export const assetStub = { originalFileName: 'IMG_456.jpg', faces: [], sidecarPath: null, - isOffline: false, isExternal: false, exifInfo: { fileSizeInByte: 123_000, } as ExifEntity, deletedAt: null, duplicateId: null, + isOffline: false, }), noThumbhash: Object.freeze({ @@ -133,7 +133,6 @@ export const assetStub = { localDateTime: new Date('2023-02-23T05:06:29.716Z'), isFavorite: true, isArchived: false, - isOffline: false, duration: null, isVisible: true, isExternal: false, @@ -146,6 +145,7 @@ export const assetStub = { sidecarPath: null, deletedAt: null, duplicateId: null, + isOffline: false, }), primaryImage: Object.freeze({ @@ -173,7 +173,6 @@ export const assetStub = { isExternal: false, livePhotoVideo: null, livePhotoVideoId: null, - isOffline: false, tags: [], sharedLinks: [], originalFileName: 'asset-id.jpg', @@ -191,6 +190,7 @@ export const assetStub = { { id: 'stack-child-asset-2' } as AssetEntity, ]), duplicateId: null, + isOffline: false, }), image: Object.freeze({ @@ -218,7 +218,6 @@ export const assetStub = { isExternal: false, livePhotoVideo: null, livePhotoVideoId: null, - isOffline: false, tags: [], sharedLinks: [], originalFileName: 'asset-id.jpg', @@ -231,9 +230,50 @@ export const assetStub = { exifImageWidth: 2160, } as ExifEntity, duplicateId: null, + isOffline: false, }), trashed: Object.freeze({ + id: 'asset-id', + deviceAssetId: 'device-asset-id', + fileModifiedAt: new Date('2023-02-23T05:06:29.716Z'), + fileCreatedAt: new Date('2023-02-23T05:06:29.716Z'), + owner: userStub.user1, + ownerId: 'user-id', + deviceId: 'device-id', + originalPath: '/original/path.jpg', + checksum: Buffer.from('file hash', 'utf8'), + type: AssetType.IMAGE, + files, + thumbhash: Buffer.from('blablabla', 'base64'), + encodedVideoPath: null, + createdAt: new Date('2023-02-23T05:06:29.716Z'), + updatedAt: new Date('2023-02-23T05:06:29.716Z'), + deletedAt: new Date('2023-02-24T05:06:29.716Z'), + localDateTime: new Date('2023-02-23T05:06:29.716Z'), + isFavorite: false, + isArchived: false, + duration: null, + isVisible: true, + isExternal: false, + livePhotoVideo: null, + livePhotoVideoId: null, + tags: [], + sharedLinks: [], + originalFileName: 'asset-id.jpg', + faces: [], + sidecarPath: null, + exifInfo: { + fileSizeInByte: 5000, + exifImageHeight: 3840, + exifImageWidth: 2160, + } as ExifEntity, + duplicateId: null, + isOffline: false, + status: AssetStatus.TRASHED, + }), + + trashedOffline: Object.freeze({ id: 'asset-id', status: AssetStatus.ACTIVE, deviceAssetId: 'device-asset-id', @@ -259,7 +299,6 @@ export const assetStub = { isExternal: false, livePhotoVideo: null, livePhotoVideoId: null, - isOffline: false, tags: [], sharedLinks: [], originalFileName: 'asset-id.jpg', @@ -271,8 +310,8 @@ export const assetStub = { exifImageWidth: 2160, } as ExifEntity, duplicateId: null, + isOffline: true, }), - archived: Object.freeze({ id: 'asset-id', status: AssetStatus.ACTIVE, @@ -298,7 +337,6 @@ export const assetStub = { isExternal: false, livePhotoVideo: null, livePhotoVideoId: null, - isOffline: false, tags: [], sharedLinks: [], originalFileName: 'asset-id.jpg', @@ -311,6 +349,7 @@ export const assetStub = { exifImageWidth: 2160, } as ExifEntity, duplicateId: null, + isOffline: false, }), external: Object.freeze({ @@ -338,97 +377,19 @@ export const assetStub = { isVisible: true, livePhotoVideo: null, livePhotoVideoId: null, + libraryId: 'library-id', + library: libraryStub.externalLibrary1, + tags: [], + sharedLinks: [], + originalFileName: 'asset-id.jpg', + faces: [], + deletedAt: null, + sidecarPath: null, + exifInfo: { + fileSizeInByte: 5000, + } as ExifEntity, + duplicateId: null, isOffline: false, - libraryId: 'library-id', - library: libraryStub.externalLibrary1, - tags: [], - sharedLinks: [], - originalFileName: 'asset-id.jpg', - faces: [], - deletedAt: null, - sidecarPath: null, - exifInfo: { - fileSizeInByte: 5000, - } as ExifEntity, - duplicateId: null, - }), - - offline: Object.freeze({ - id: 'asset-id', - status: AssetStatus.ACTIVE, - deviceAssetId: 'device-asset-id', - fileModifiedAt: new Date('2023-02-23T05:06:29.716Z'), - fileCreatedAt: new Date('2023-02-23T05:06:29.716Z'), - owner: userStub.user1, - ownerId: 'user-id', - deviceId: 'device-id', - originalPath: '/original/path.jpg', - checksum: Buffer.from('file hash', 'utf8'), - type: AssetType.IMAGE, - files, - thumbhash: Buffer.from('blablabla', 'base64'), - encodedVideoPath: null, - createdAt: new Date('2023-02-23T05:06:29.716Z'), - updatedAt: new Date('2023-02-23T05:06:29.716Z'), - localDateTime: new Date('2023-02-23T05:06:29.716Z'), - isFavorite: true, - isArchived: false, - isExternal: false, - duration: null, - isVisible: true, - livePhotoVideo: null, - livePhotoVideoId: null, - isOffline: true, - tags: [], - sharedLinks: [], - originalFileName: 'asset-id.jpg', - faces: [], - sidecarPath: null, - exifInfo: { - fileSizeInByte: 5000, - } as ExifEntity, - deletedAt: null, - duplicateId: null, - }), - - externalOffline: Object.freeze({ - id: 'asset-id', - status: AssetStatus.ACTIVE, - deviceAssetId: 'device-asset-id', - fileModifiedAt: new Date('2023-02-23T05:06:29.716Z'), - fileCreatedAt: new Date('2023-02-23T05:06:29.716Z'), - owner: userStub.user1, - ownerId: 'user-id', - deviceId: 'device-id', - originalPath: '/data/user1/photo.jpg', - checksum: Buffer.from('path hash', 'utf8'), - type: AssetType.IMAGE, - files, - thumbhash: Buffer.from('blablabla', 'base64'), - encodedVideoPath: null, - createdAt: new Date('2023-02-23T05:06:29.716Z'), - updatedAt: new Date('2023-02-23T05:06:29.716Z'), - localDateTime: new Date('2023-02-23T05:06:29.716Z'), - isFavorite: true, - isArchived: false, - isExternal: true, - duration: null, - isVisible: true, - livePhotoVideo: null, - livePhotoVideoId: null, - isOffline: true, - libraryId: 'library-id', - library: libraryStub.externalLibrary1, - tags: [], - sharedLinks: [], - originalFileName: 'asset-id.jpg', - faces: [], - sidecarPath: null, - exifInfo: { - fileSizeInByte: 5000, - } as ExifEntity, - deletedAt: null, - duplicateId: null, }), image1: Object.freeze({ @@ -457,7 +418,6 @@ export const assetStub = { livePhotoVideo: null, livePhotoVideoId: null, isExternal: false, - isOffline: false, tags: [], sharedLinks: [], originalFileName: 'asset-id.ext', @@ -467,6 +427,7 @@ export const assetStub = { fileSizeInByte: 5000, } as ExifEntity, duplicateId: null, + isOffline: false, }), imageFrom2015: Object.freeze({ @@ -490,7 +451,6 @@ export const assetStub = { isFavorite: true, isArchived: false, isExternal: false, - isOffline: false, duration: null, isVisible: true, livePhotoVideo: null, @@ -505,6 +465,7 @@ export const assetStub = { } as ExifEntity, deletedAt: null, duplicateId: null, + isOffline: false, }), video: Object.freeze({ @@ -529,7 +490,6 @@ export const assetStub = { isFavorite: true, isArchived: false, isExternal: false, - isOffline: false, duration: null, isVisible: true, livePhotoVideo: null, @@ -545,6 +505,7 @@ export const assetStub = { } as ExifEntity, deletedAt: null, duplicateId: null, + isOffline: false, }), livePhotoMotionAsset: Object.freeze({ @@ -664,7 +625,6 @@ export const assetStub = { isFavorite: false, isArchived: false, isExternal: false, - isOffline: false, duration: null, isVisible: true, livePhotoVideo: null, @@ -683,6 +643,7 @@ export const assetStub = { } as ExifEntity, deletedAt: null, duplicateId: null, + isOffline: false, }), sidecar: Object.freeze({ id: 'asset-id', @@ -705,7 +666,6 @@ export const assetStub = { isFavorite: true, isArchived: false, isExternal: false, - isOffline: false, duration: null, isVisible: true, livePhotoVideo: null, @@ -717,6 +677,7 @@ export const assetStub = { sidecarPath: '/original/path.ext.xmp', deletedAt: null, duplicateId: null, + isOffline: false, }), sidecarWithoutExt: Object.freeze({ id: 'asset-id', @@ -739,7 +700,6 @@ export const assetStub = { isFavorite: true, isArchived: false, isExternal: false, - isOffline: false, duration: null, isVisible: true, livePhotoVideo: null, @@ -751,41 +711,7 @@ export const assetStub = { sidecarPath: '/original/path.xmp', deletedAt: null, duplicateId: null, - }), - - readOnly: Object.freeze({ - id: 'read-only-asset', - status: AssetStatus.ACTIVE, - deviceAssetId: 'device-asset-id', - fileModifiedAt: new Date('2023-02-23T05:06:29.716Z'), - fileCreatedAt: new Date('2023-02-23T05:06:29.716Z'), - owner: userStub.user1, - ownerId: 'user-id', - deviceId: 'device-id', - originalPath: '/original/path.ext', - thumbhash: null, - checksum: Buffer.from('file hash', 'utf8'), - type: AssetType.IMAGE, - files: [previewFile], - encodedVideoPath: null, - createdAt: new Date('2023-02-23T05:06:29.716Z'), - updatedAt: new Date('2023-02-23T05:06:29.716Z'), - localDateTime: new Date('2023-02-23T05:06:29.716Z'), - isFavorite: true, - isArchived: false, - isExternal: false, isOffline: false, - duration: null, - isVisible: true, - livePhotoVideo: null, - livePhotoVideoId: null, - tags: [], - sharedLinks: [], - originalFileName: 'asset-id.ext', - faces: [], - sidecarPath: '/original/path.ext.xmp', - deletedAt: null, - duplicateId: null, }), hasEncodedVideo: Object.freeze({ @@ -810,7 +736,6 @@ export const assetStub = { isFavorite: true, isArchived: false, isExternal: false, - isOffline: false, duration: null, isVisible: true, livePhotoVideo: null, @@ -824,6 +749,7 @@ export const assetStub = { } as ExifEntity, deletedAt: null, duplicateId: null, + isOffline: false, }), missingFileExtension: Object.freeze({ id: 'asset-id', @@ -850,7 +776,6 @@ export const assetStub = { isVisible: true, livePhotoVideo: null, livePhotoVideoId: null, - isOffline: false, libraryId: 'library-id', library: libraryStub.externalLibrary1, tags: [], @@ -863,6 +788,7 @@ export const assetStub = { fileSizeInByte: 5000, } as ExifEntity, duplicateId: null, + isOffline: false, }), hasFileExtension: Object.freeze({ id: 'asset-id', @@ -889,7 +815,6 @@ export const assetStub = { isVisible: true, livePhotoVideo: null, livePhotoVideoId: null, - isOffline: false, libraryId: 'library-id', library: libraryStub.externalLibrary1, tags: [], @@ -902,6 +827,7 @@ export const assetStub = { fileSizeInByte: 5000, } as ExifEntity, duplicateId: null, + isOffline: false, }), imageDng: Object.freeze({ id: 'asset-id', @@ -928,7 +854,6 @@ export const assetStub = { isExternal: false, livePhotoVideo: null, livePhotoVideoId: null, - isOffline: false, tags: [], sharedLinks: [], originalFileName: 'asset-id.jpg', @@ -941,6 +866,7 @@ export const assetStub = { bitsPerSample: 14, } as ExifEntity, duplicateId: null, + isOffline: false, }), hasEmbedding: Object.freeze({ id: 'asset-id-embedding', @@ -967,7 +893,6 @@ export const assetStub = { isExternal: false, livePhotoVideo: null, livePhotoVideoId: null, - isOffline: false, tags: [], sharedLinks: [], originalFileName: 'asset-id.jpg', @@ -982,6 +907,7 @@ export const assetStub = { assetId: 'asset-id', embedding: Array.from({ length: 512 }, Math.random), }, + isOffline: false, }), hasDupe: Object.freeze({ id: 'asset-id-dupe', @@ -1008,7 +934,6 @@ export const assetStub = { isExternal: false, livePhotoVideo: null, livePhotoVideoId: null, - isOffline: false, tags: [], sharedLinks: [], originalFileName: 'asset-id.jpg', @@ -1023,5 +948,6 @@ export const assetStub = { assetId: 'asset-id', embedding: Array.from({ length: 512 }, Math.random), }, + isOffline: false, }), }; diff --git a/server/test/repositories/asset.repository.mock.ts b/server/test/repositories/asset.repository.mock.ts index 9ac568af306b8..ba2f5e10d98cc 100644 --- a/server/test/repositories/asset.repository.mock.ts +++ b/server/test/repositories/asset.repository.mock.ts @@ -25,7 +25,6 @@ export const newAssetRepositoryMock = (): Mocked => { getLivePhotoCount: vitest.fn(), updateAll: vitest.fn(), updateDuplicates: vitest.fn(), - getExternalLibraryAssetPaths: vitest.fn(), getByLibraryIdAndOriginalPath: vitest.fn(), deleteAll: vitest.fn(), update: vitest.fn(), diff --git a/server/vitest.config.mjs b/server/vitest.config.mjs index 3c0ea00c84c7a..1013b4606df3f 100644 --- a/server/vitest.config.mjs +++ b/server/vitest.config.mjs @@ -13,7 +13,7 @@ export default defineConfig({ lines: 80, statements: 80, branches: 85, - functions: 85, + functions: 80, }, }, server: { diff --git a/web/src/lib/components/asset-viewer/asset-viewer-nav-bar.svelte b/web/src/lib/components/asset-viewer/asset-viewer-nav-bar.svelte index db216641d5c2f..d19b428750fda 100644 --- a/web/src/lib/components/asset-viewer/asset-viewer-nav-bar.svelte +++ b/web/src/lib/components/asset-viewer/asset-viewer-nav-bar.svelte @@ -59,7 +59,6 @@ export let onClose: () => void; const sharedLink = getSharedLink(); - $: isOwner = $user && asset.ownerId === $user?.id; $: showDownloadButton = sharedLink ? sharedLink.allowDownload : !asset.isOffline; // $: showEditorButton = @@ -87,7 +86,7 @@ {/if} {#if asset.isOffline} - + {/if} {#if asset.livePhotoVideoId} diff --git a/web/src/lib/components/asset-viewer/detail-panel.svelte b/web/src/lib/components/asset-viewer/detail-panel.svelte index 9e32927fc3605..88ea98778faca 100644 --- a/web/src/lib/components/asset-viewer/detail-panel.svelte +++ b/web/src/lib/components/asset-viewer/detail-panel.svelte @@ -148,12 +148,21 @@ {#if asset.isOffline}

-
{$t('asset_offline')}
-
+
+ {$t('asset_offline')} +
+

- {$t('asset_offline_description')} + {#if $user?.isAdmin} +

{$t('admin.asset_offline_description')}

+ {:else} + {$t('asset_offline_description')} + {/if}

+
+

{asset.originalPath}

+
{/if} diff --git a/web/src/lib/components/elements/buttons/circle-icon-button.svelte b/web/src/lib/components/elements/buttons/circle-icon-button.svelte index 76f962f107ce1..8af3f75ade296 100644 --- a/web/src/lib/components/elements/buttons/circle-icon-button.svelte +++ b/web/src/lib/components/elements/buttons/circle-icon-button.svelte @@ -1,7 +1,7 @@