Kavita/API/Data/Repositories/LibraryRepository.cs
Joe Milazzo 9cc5953d07
Filtering Overhaul (#2207)
* Implemented the first version of dynamic filtering which is all Extension based.

* Implemented basic generic property filter for expanded metadata filtering.

* Fixed up the code to allow for nested properties and fixed up the Contains to work only for IList's

* Started refactoring for the new approach

* More progress, need to rethink a few filters like read progress to be % based and people needs to be more explicit.

* Refactored most of the existing filtering operations into dedicate extensions for the appropriate comparisons. People still need to be reworked to be more dynamic.

* Fixed a bug with continue point where it fails on chapters or volumes tagged with a range

* Wired up a basic api path to start building groups. No and/or support yet.

* Started on the UI

* Made a bit of progress on the UI as I'm putting the pieces together about how to design it.

* Refactored names to make it more consistent. New thinking is we will have one row that will take a filter statement and manipulate it. It will emit said statement and a builder will turn into the higher level statement.

* Started working on updating code to use new inject() method.

* Fixed the code to switch the comparisons.

* Added dynamic input structure in and moved add/remove to the builder.

* Fixed an enum bug

* Hooked in basic dropdown support that is dynamic to the field. Only language is missing as that needs a DTO change (but don't want to break API)

* Fixed a bug where dropdown options wouldn't re-populate when switching fields that are both dropdowns

* Started adding metadata builder

* Fixed when typing on filter row the focus resetting

* Refactored to add an additional component which handles the compounding of filter rows.

* Started hooking up v2 dto in the UI to send to the backend.

* Started working on building group UI for and/or support.

* Lots of backend code fixes to ensure OR and AND statements combine correctly.

* More trying to figure out how to write the UI code

* Started debugging to remember what I was last doing.

* Lots of progress towards building out the UI recursively

* I got the dto to build and propagate up the chain

* Started hooking up to the actual api to fetch the data.

* Basic wire up to the backend is working.

* HasName is now complete

* Refactored SortOptions code into an extension and streamlined LimitTo to the correct place.

* Fixed a bug where Library Filters from the Group weren't actually being taken into account.

* Refactored a lot of code so builder will now export the full dto.

* Cleaned up the data flow from metadata filter to library detail

* Got the dropdown to load preset values on first load, but now it triggers twice.

* Changed so when you add a new filter, it does it at top and fixed remove

* Fixed the remove button being on the wrong row

* Cleaned up the arrays to make it easier to manage

* Cleaned up some of the backend to ensure it doesn't throw an incorrect exception

* I'm starting to tread water, taking a break

* Fixed a merge issue

* Cleaned up Docker checks.

* Default IpAddresses to empty string.

* Refactored IsDocker to be completely static

* Figured out the issue with the dropdown not working.

* Almost got it, but the event isn't being called.

* I think i might try something else. This doesn't seem to be working.

* On the new implementation, implemented remove group.

* Use enums to reduce copy/paste

* the new system is working pretty well, ill go with it and move on. Can alwasy refactor.

* Code is totally broken, but working the cache resume code with some hiccups.

* I need to take a break

* Stashing my broken code. I have an idea on how to serialize to the URL, but I need to rearchitect a lot.

* Reverted last commit

* remove domain

* Fixed up some hardcoded caching. I'm giving up on this implementation and going to a simpler version

* Refactored the backend to just allow flat filtering.

* Started refactoring the components to make it flat filtering only.

* Finished refactoring so that the base preset case will render.

* Implemented basic query functionality on desktop. Clear needs some work and url code.

* Some cleanup

* Working on filtering url encode/decode

* Interacting with filters now saves to url and can be reloaded from the url. Named filters is not hooked up.

* Fixed a double load on the library detail page.

* Moved the library filtering code out of the FilterBuilder as it needs to be handled differently.

* Fixed up how we handle library statements in the filter.

* Fixed up how links that perform a filter work.

* Refactored a bunch of linking to a search page.

* LimitTo works, my css however does not.

* Switched some code to use localized strings.

* Cleaned up some css

* Hooked up Languages and put some additional code in so that Languages will return invalid Language codes back.

* Removed a duplicate language signature.

* Hooked up ability to preload collection tag.

* Want To Read is converted

* Converted lots of code to new filtering system. Need to do Bookmarks.

* Fixed a potential bug with default filter creation.

* Hooked up the ability to disable certain filter fields from appearing.

* Added mobile drawer code and a hook for Robbie to take a look for some css.

* Converted the APIs for dashboard along with other safety fixes to ensure bad data doesn't break any of the filtering apis

* Added the backend code to handle summary query

* Converted Want to Read api properly now.

* Fixed the HasReadingProgress query

* Hooked back the Reading Progress for legacy APIs

* Fixed some bad localization strings

* Wrote the filtering code for all-bookmarks.

* OPDS is now using the new filter

* Fixed OPDS reading lists and covers not sending their images.

* Fixed up the OPDS feed and fixed a bug where libraries also weren't sending their images over OPDS

* All but dropdown options have been validated and tested.

* Fixed up some default cases for setting up the filter.

* Sorted filter fields and re-keyed to be better suited based on user's needs.

Fixed a bug where OPDS Series (from library view) wasn't showing the summary.

Moved the (Format) from the title to the description to make the UX much better for OPDS.

MOved

* don't send empty summaries in the new summary formatting

* Fixed up some default cases for setting up the filter.

* Fixed the reset button

* Fixed infinite scroller not having correct scope key

* Added localization to the new components and removed old debug code

* Styling fixes

* Fixed deep linking across the app. Made it so you can click Characters from Reading list and open a filtered search.

* A bit of styling for mobile

* Don't show language if it's not properly set

---------

Co-authored-by: Robbie Davis <robbie@therobbiedavis.com>
2023-08-11 14:30:36 -07:00

378 lines
12 KiB
C#

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Threading.Tasks;
using API.DTOs;
using API.DTOs.Filtering;
using API.DTOs.JumpBar;
using API.DTOs.Metadata;
using API.Entities;
using API.Entities.Enums;
using API.Extensions;
using API.Extensions.QueryExtensions;
using API.Services.Tasks.Scanner.Parser;
using AutoMapper;
using AutoMapper.QueryableExtensions;
using Kavita.Common.Extensions;
using Microsoft.EntityFrameworkCore;
namespace API.Data.Repositories;
[Flags]
public enum LibraryIncludes
{
None = 1,
Series = 2,
AppUser = 4,
Folders = 8,
// Ratings = 16
}
public interface ILibraryRepository
{
void Add(Library library);
void Update(Library library);
void Delete(Library? library);
Task<IEnumerable<LibraryDto>> GetLibraryDtosAsync();
Task<bool> LibraryExists(string libraryName);
Task<Library?> GetLibraryForIdAsync(int libraryId, LibraryIncludes includes = LibraryIncludes.None);
IEnumerable<LibraryDto> GetLibraryDtosForUsernameAsync(string userName);
Task<IEnumerable<Library>> GetLibrariesAsync(LibraryIncludes includes = LibraryIncludes.None);
Task<IEnumerable<Library>> GetLibrariesForUserIdAsync(int userId);
IEnumerable<int> GetLibraryIdsForUserIdAsync(int userId, QueryContext queryContext = QueryContext.None);
Task<LibraryType> GetLibraryTypeAsync(int libraryId);
Task<IEnumerable<Library>> GetLibraryForIdsAsync(IEnumerable<int> libraryIds, LibraryIncludes includes = LibraryIncludes.None);
Task<int> GetTotalFiles();
IEnumerable<JumpKeyDto> GetJumpBarAsync(int libraryId);
Task<IList<AgeRatingDto>> GetAllAgeRatingsDtosForLibrariesAsync(List<int> libraryIds);
Task<IList<LanguageDto>> GetAllLanguagesForLibrariesAsync(List<int>? libraryIds);
IEnumerable<PublicationStatusDto> GetAllPublicationStatusesDtosForLibrariesAsync(List<int> libraryIds);
Task<bool> DoAnySeriesFoldersMatch(IEnumerable<string> folders);
Task<string?> GetLibraryCoverImageAsync(int libraryId);
Task<IList<string>> GetAllCoverImagesAsync();
Task<IDictionary<int, LibraryType>> GetLibraryTypesForIdsAsync(IEnumerable<int> libraryIds);
Task<IList<Library>> GetAllWithCoversInDifferentEncoding(EncodeFormat encodeFormat);
}
public class LibraryRepository : ILibraryRepository
{
private readonly DataContext _context;
private readonly IMapper _mapper;
public LibraryRepository(DataContext context, IMapper mapper)
{
_context = context;
_mapper = mapper;
}
public void Add(Library library)
{
_context.Library.Add(library);
}
public void Update(Library library)
{
_context.Entry(library).State = EntityState.Modified;
}
public void Delete(Library? library)
{
if (library == null) return;
_context.Library.Remove(library);
}
public IEnumerable<LibraryDto> GetLibraryDtosForUsernameAsync(string userName)
{
return _context.Library
.Include(l => l.AppUsers)
.Where(library => library.AppUsers.Any(x => x.UserName.Equals(userName)))
.OrderBy(l => l.Name)
.ProjectTo<LibraryDto>(_mapper.ConfigurationProvider)
.AsSplitQuery()
.AsEnumerable();
}
/// <summary>
/// Returns all libraries including their AppUsers + extra includes
/// </summary>
/// <param name="includes"></param>
/// <returns></returns>
public async Task<IEnumerable<Library>> GetLibrariesAsync(LibraryIncludes includes = LibraryIncludes.None)
{
var query = _context.Library
.Include(l => l.AppUsers)
.Select(l => l);
query = AddIncludesToQuery(query, includes);
return await query.ToListAsync();
}
/// <summary>
/// This does not track
/// </summary>
/// <param name="userId"></param>
/// <returns></returns>
public async Task<IEnumerable<Library>> GetLibrariesForUserIdAsync(int userId)
{
return await _context.Library
.Include(l => l.AppUsers)
.Where(l => l.AppUsers.Select(ap => ap.Id).Contains(userId))
.AsNoTracking()
.ToListAsync();
}
public IEnumerable<int> GetLibraryIdsForUserIdAsync(int userId, QueryContext queryContext = QueryContext.None)
{
return _context.Library
.IsRestricted(queryContext)
.Where(l => l.AppUsers.Select(ap => ap.Id).Contains(userId))
.Select(l => l.Id)
.AsEnumerable();
}
public async Task<LibraryType> GetLibraryTypeAsync(int libraryId)
{
return await _context.Library
.Where(l => l.Id == libraryId)
.AsNoTracking()
.Select(l => l.Type)
.FirstAsync();
}
public async Task<IEnumerable<Library>> GetLibraryForIdsAsync(IEnumerable<int> libraryIds, LibraryIncludes includes = LibraryIncludes.None)
{
var query = _context.Library
.Where(x => libraryIds.Contains(x.Id));
AddIncludesToQuery(query, includes);
return await query.ToListAsync();
}
public async Task<int> GetTotalFiles()
{
return await _context.MangaFile.CountAsync();
}
public IEnumerable<JumpKeyDto> GetJumpBarAsync(int libraryId)
{
var seriesSortCharacters = _context.Series.Where(s => s.LibraryId == libraryId)
.Select(s => s.SortName!.ToUpper())
.OrderBy(s => s)
.AsEnumerable()
.Select(s => s[0]);
// Map the title to the number of entities
var firstCharacterMap = new Dictionary<char, int>();
foreach (var sortChar in seriesSortCharacters)
{
var c = sortChar;
var isAlpha = char.IsLetter(sortChar);
if (!isAlpha) c = '#';
firstCharacterMap.TryAdd(c, 0);
firstCharacterMap[c] += 1;
}
return firstCharacterMap.Keys.Select(k => new JumpKeyDto()
{
Key = k + string.Empty,
Size = firstCharacterMap[k],
Title = k + string.Empty
});
}
/// <summary>
/// Returns all Libraries with their Folders
/// </summary>
/// <returns></returns>
public async Task<IEnumerable<LibraryDto>> GetLibraryDtosAsync()
{
return await _context.Library
.Include(f => f.Folders)
.OrderBy(l => l.Name)
.ProjectTo<LibraryDto>(_mapper.ConfigurationProvider)
.AsSplitQuery()
.AsNoTracking()
.ToListAsync();
}
public async Task<Library?> GetLibraryForIdAsync(int libraryId, LibraryIncludes includes = LibraryIncludes.None)
{
var query = _context.Library
.Where(x => x.Id == libraryId);
query = AddIncludesToQuery(query, includes);
return await query.SingleOrDefaultAsync();
}
private static IQueryable<Library> AddIncludesToQuery(IQueryable<Library> query, LibraryIncludes includeFlags)
{
if (includeFlags.HasFlag(LibraryIncludes.Folders))
{
query = query.Include(l => l.Folders);
}
if (includeFlags.HasFlag(LibraryIncludes.Series))
{
query = query.Include(l => l.Series);
}
if (includeFlags.HasFlag(LibraryIncludes.AppUser))
{
query = query.Include(l => l.AppUsers);
}
return query.AsSplitQuery();
}
public async Task<bool> LibraryExists(string libraryName)
{
return await _context.Library
.AsNoTracking()
.AnyAsync(x => x.Name != null && x.Name.Equals(libraryName));
}
public async Task<IEnumerable<LibraryDto>> GetLibrariesForUserAsync(AppUser user)
{
return await _context.Library
.Where(library => library.AppUsers.Contains(user))
.Include(l => l.Folders)
.AsNoTracking()
.AsSplitQuery()
.ProjectTo<LibraryDto>(_mapper.ConfigurationProvider)
.ToListAsync();
}
public async Task<IList<AgeRatingDto>> GetAllAgeRatingsDtosForLibrariesAsync(List<int> libraryIds)
{
return await _context.Series
.Where(s => libraryIds.Contains(s.LibraryId))
.Select(s => s.Metadata.AgeRating)
.Distinct()
.Select(s => new AgeRatingDto()
{
Value = s,
Title = s.ToDescription()
})
.ToListAsync();
}
public async Task<IList<LanguageDto>> GetAllLanguagesForLibrariesAsync(List<int>? libraryIds)
{
var ret = await _context.Series
.WhereIf(libraryIds is {Count: > 0} , s => libraryIds.Contains(s.LibraryId))
.Select(s => s.Metadata.Language)
.AsSplitQuery()
.AsNoTracking()
.Distinct()
.ToListAsync();
return ret
.Where(s => !string.IsNullOrEmpty(s))
.DistinctBy(Parser.Normalize)
.Select(GetCulture)
.Where(s => s != null)
.OrderBy(s => s.Title)
.ToList();
}
private static LanguageDto GetCulture(string s)
{
try
{
return new LanguageDto()
{
Title = CultureInfo.GetCultureInfo(s).DisplayName,
IsoCode = s
};
}
catch (Exception)
{
// ignored
}
return new LanguageDto()
{
Title = s,
IsoCode = s
};;
}
public IEnumerable<PublicationStatusDto> GetAllPublicationStatusesDtosForLibrariesAsync(List<int> libraryIds)
{
return _context.Series
.Where(s => libraryIds.Contains(s.LibraryId))
.AsSplitQuery()
.Select(s => s.Metadata.PublicationStatus)
.Distinct()
.AsEnumerable()
.Select(s => new PublicationStatusDto()
{
Value = s,
Title = s.ToDescription()
})
.OrderBy(s => s.Title);
}
/// <summary>
/// Checks if any series folders match the folders passed in
/// </summary>
/// <param name="folders"></param>
/// <returns></returns>
public async Task<bool> DoAnySeriesFoldersMatch(IEnumerable<string> folders)
{
var normalized = folders.Select(Services.Tasks.Scanner.Parser.Parser.NormalizePath);
return await _context.Series.AnyAsync(s => normalized.Contains(s.FolderPath));
}
public Task<string?> GetLibraryCoverImageAsync(int libraryId)
{
return _context.Library
.Where(l => l.Id == libraryId)
.Select(l => l.CoverImage)
.SingleOrDefaultAsync();
}
public async Task<IList<string>> GetAllCoverImagesAsync()
{
return (await _context.ReadingList
.Select(t => t.CoverImage)
.Where(t => !string.IsNullOrEmpty(t))
.ToListAsync())!;
}
public async Task<IDictionary<int, LibraryType>> GetLibraryTypesForIdsAsync(IEnumerable<int> libraryIds)
{
var types = await _context.Library
.Where(l => libraryIds.Contains(l.Id))
.AsNoTracking()
.Select(l => new
{
LibraryId = l.Id,
LibraryType = l.Type
})
.ToListAsync();
var dict = new Dictionary<int, LibraryType>();
foreach (var type in types)
{
dict.TryAdd(type.LibraryId, type.LibraryType);
}
return dict;
}
public async Task<IList<Library>> GetAllWithCoversInDifferentEncoding(EncodeFormat encodeFormat)
{
var extension = encodeFormat.GetExtension();
return await _context.Library
.Where(c => !string.IsNullOrEmpty(c.CoverImage) && !c.CoverImage.EndsWith(extension))
.ToListAsync();
}
}