25 Commits

Author SHA1 Message Date
Joseph Milazzo
56cf7be799
Scan Chunking (#604)
* Some performance refactoring around getting Library and avoid a byte[] copy for getting cover images for epubs.

* Initial commit. Rewrote the main series scan loop to use chunks of data at a time. Not fully shaken out.

* Hooked in the ability for the UI to react to series being added or removed from the DB.

* Cleaned up the messaging in the scan loop to be more clear.

* Metadata scan and scan work as expected and populate data to the UI. There is a slow down in speed for overall operation.

Scan series and refresh series metadata does not work fully.

* Fixed a bug where MangaFiles were not having LastModified Updated correctly, meaning they were opening archives every scan.

* Modified the code to be more realistic to the underlying file

* Updated ScanService to properly handle deleted files and not result in a higher-level scan.

* Shuffled around volume related repo apis to the volume repo rather than being in series.

* Rewrote scan series to be much cleaner and more concise on the flow. Fixed an issue in UpdateVolumes such that the debug code to log out removed volumes could throw an exception and actually break updating volumes.

* Refactored the code to set MangaFile last modified timestamp into the MangaFile entity.

* Added Series Name to ScanSeries event

* Added additional checks in ScanSeries to ensure we never go outside the library folder.

Added extra debug messages for when a metadata refresh doesn't actually make changes and for when we regen cover images.

* More logging statements saying where they originate from. Fixed a critical bug which caused only 1 chunk to ever be processed.

* Fixed a concurrency issue with natural sorter which could cause issues in ArchiveService.cs.

* Log cleanups

* Fixed an issue with logging out total time of a scan.

* Only show added toastrs for admins. When kicking off a refresh metadata for series, make sure we regenerate all cover images.

* Code smells on benchmark despite it being ignored
2021-09-30 06:08:05 -07:00
Joseph Milazzo
cf7a9aa71e
Reading Lists & More (#564)
* Added continous reading to the book reader. Clicking on the max pages to right of progress bar will now go to last page.

* Forgot a file for continous book reading

* Fixed up some code regarding transitioning between chapters. Arrows now show to represent a chapter transition.

* Laid the foundation for reading lists

* All foundation is laid out. Actions are wired in the UI. Backend repository is setup. Redid the migration to have ReadingList track modification so we can order them for the user.

* Updated add modal to have basic skeleton

* Hooked up ability to fetch reading lists from backend

* Made a huge performance improvement to GetChapterIdsForSeriesAsync() by reducing a JOIN and an iteration loop. Improvement went from 2 seconds -> 200 ms.

* Implemented the ability to add all chapters in a series to a reading list.

* Fixed issue with adding new items to reading list not being in a logical order. Lots of work on getting all the information around the reading list view. Added some foreign keys back to chapter so delete should clean up after itself.

* Added ability to open directly the series

* Reading List Items now have progress attached

* Hooked up list deletion and added a case where if doesn't exist on load, then redirect to library.

* Lots of changes. Introduced a dashboard component for the main app. This will sit on libraries route for now and will have 3 tabs to show different sections.

Moved libraries reel down to bottom as people are more likely to access recently added or in progress than explore their whole library.

Note: Bundles are messed up, they need to be reoptimized and routes need to be updated.

* Added pagination to the reading lists api and implemented a page to show all lists

* Cleaned up old code from all-collections component so now it only handles all collections and doesn't have the old code for an individual collection

* Hooked in actions and navigation on reading lists

* When the user re-arranges items, they are now persisted

* Implemented remove read, but performance is pretty poor. Needs to be optimized.

* Lots of API fixes for adding items to a series, returning items, etc. Committing before fixing incorrect fetches of items for a readingListId.

* Rewrote the joins for GetReadingListItemDtosByIdAsync() to not return extra records.

* Remove bug marker now that it is fixed

* Refactor update-by-series to move more of the code to a re-usable function for update-by-volume/chapter APIs

* Implemented the ability to add via series, volume or chapter.

* Added OPDS support for reading lists. This included adding VolumeId to the ReadingListDto.

* Fixed a bug with deleting items

* After we create a library inform user that a scan has started

* Added some extra help information for users on directory picker, since linux users were getting confused.

* Setup for the reading functionality

* Fixed an issue where opening the edit series modal and pressing save without doing anything would empty collection tags. Would happen often when editing cover images.

* Fixed get-next-chapter for reading list. Refactored all methods to use the new GetUserIdByUsernameAsync(), which is much faster and uses less memory.

* Hooked in prev chapter for continuous reading with reading list

* Hooked up the read code for manga reader and book reader to have list id passed

* Manga reader now functions completely with reading lists

* Implemented reading list and incognito mode into book reader

* Refactored some common reading code into reader service

* Added support for "Series -  - Vol. 03 Ch. 023.5 - Volume 3 Extras.cbz" format that can occur with FMD2.

* Implemented continuous reading with a reading list between different readers. This incurs a 3x performance hit on the book info api.

* style changes. Don't emit an event if position of draggable item hasn't changed

* Styling and added the edit reading list flow.

* Cleaned up some extra spaces when actionables isn't shown. Lots of cleanup for promoted lists.

* Refactored some filter code to a common service

* Added an RBS check in getting Items for a given user.

* Code smells

* More smells
2021-09-08 12:03:27 -05:00
Joseph Milazzo
be2b78fa5a
Manga Redesign (#321)
* Code cleanup, refactored FileRepository into Unit of Work.

* Added AutoCloseMenu and ReaderMode user perferences to match UI

* Added extra information to ChapterInfo

* Build changes

* Updated the readme to have open collective information and thanks to sponsors

* Fixed an issue with UnitOfWork refactor and how stats was bootsrapped. Replaced stats.kavitareader with a temp url to test out redirection bug.
2021-06-24 19:31:42 -05:00
Joseph Milazzo
6e1b227e65
Transaction Support (#309)
* Added transactions to UnitOfWork and refactored code to use it.

* This included blank UI fix from Kavita-webui
2021-06-18 07:37:48 -05:00
Joseph Milazzo
8b1fbc0096
Implemented Dark Mode (#203) 2021-05-09 08:44:12 -05:00
Joseph Milazzo
2be1630af3
Tap to Paginate User Pref (#197)
* Fixed In Progress and removed comments

* Tap to Paginate user setting is implemented. Fixes #193
2021-05-04 17:01:27 -05:00
Joseph Milazzo
a01613f80f
EPUB Support (#178)
* Added book filetype detection and reorganized tests due to size of file

* Added ability to get basic Parse Info from Book and Pages.

* We can now scan books and get them in a library with cover images.

* Take the first image in the epub if the cover isn't set.

* Implemented the ability to unzip the ebup to cache. Implemented a test api to load html files.

* Just some test code to figure out how to approach this.

* Fixed some merge conflicts

* Removed some dead code from merge

* Snapshot: I can now load everything properly into the UI by rewriting the urls before I send them back. I don't notice any lag from this method. It can be optimized further.

* Implemented a way to load the content in the browser not via an iframe.

* Added a note

* Anchor mappings is complete. New anchors are updated so references now resolve to javascript:void() for UI to take care of internally loading and the appropriate page is mapped to it. Anchors that are external have target="_blank" added so they don't force you out of the app and styles are of course inlined.

* Oops i need this

* Table of contents api implemented (rough) and some small enhancements to codebase for books.

* GetBookPageResources now only loads files from within the book. Nested chapter list support and images now use html parsing instead of string parsing.

* Fonts now are remapped to load from endpoint.

* book-resources now uses a key, ensuring the file is in proper format for lookup. Changed chapter list based on structure with one HEADER and nested chapters.

* Properly handle svg resource requests and when there are part anchors that are clickable, make sure we handle them in the UI by adding a kavita-page handler.

* Add Chapter group page even if one isn't set by using first page (without part) from nestedChildren.

* Added extra debug code for issue #163.

* Added new user preferences for books and updated the css so we scope it to our reading section.

* Cleaned up style code

* Implemented ability to save book preferences and some cleanup on existing apis.

* Added an api for checking if a user has read something in a library type before.

* Forgot to make sure the has reading progress is against a user lol.

* Remove cacheservice code for books, sine we use an in-memory method

* Handle svg images as well

* Enhanced cover image extraction to check for a "cover" image if the cover image wasn't set in OPF before falling back to the first image.

* Fixed an issue with special books not properly generating metadata due to not having filename set.

* Cleanup, removed warmup task code from statup/program and changed taskscheduler to schedule tasks on startup only (or if tasks are changed from UI).

* Code cleanup

* Code cleanup

* So much code. Lots of refactors to try to test scanner service. Moved a lot of the queries into Extensions to allow to easier test, even though it's hacky. Support @font-face src:url swaps with ' and ". Source summary information from epubs.

* Well...baseURL needs to come from BE and not from UI lol.

* Adjusted migrations so default values match Entity

* Removed comment

* I think I finally fixed #163! The issue was that when i checked if it had a parserInfo, i wasn't considering that the chapter range might have a - in it (0-6) and so when the code to check if range could parse out a number failed, it treated it like a special and checked range against info's filename.

* Some bugfixes

* Lots of testing, extracting code to make it easier to test. This code is buggy, but fixed a bug where 1) If we changed the normalization code, we would remove the whole db during a scan and 2) We weren't actually removing series properly.

Other than that, code is being extracted to remove duplication and centralize logic.

* More code cleanup and test cleanup to ensure scan loop is working as expected and matches expectaions from tests.

* Cleaned up the code and made it so if I change normalization, which I do in this branch, it wont break existing DBs.

* Some comic parser changes for partial chapter support.

* Added some code for directory service and scanner service along with python code to generate test files (not used yet). Fixed up all the tests.

* Code smells
2021-04-28 16:16:22 -05:00
Joseph Milazzo
bd5a1338c4 Fixed a deployment bug where we weren't listening on port properly. New way will force firewall exception dialog on Windows and work across board. Implemented user preferences and ability to update them. 2021-02-06 13:08:48 -06:00
Joseph Milazzo
14e8c3b820 Fixed some APIs that worked mins ago....something strange happening with EF relationships. 2021-01-19 10:45:37 -06:00
Joseph Milazzo
26660a9bb3 Further cleanup. Moved BackgroundJob Task enqueues into TaskScheduler, so I can have complete control via one interface. 2021-01-18 13:53:24 -06:00
Joseph Milazzo
825afd83a2 Removed some dead code on the interfaces. Introduced UnitOfWork to simplify repo injection. 2021-01-18 13:07:48 -06:00
Joseph Milazzo
effdf07cef Very messy code that implements read status tracking. Needs major cleanup. 2021-01-17 15:05:27 -06:00
Joseph Milazzo
7b1714349d Refactored ScanLibrary to accept and library id rather than DTO. Refactored ScanLibrary to use Task.Run() rather than having synchronous repo methods. 2021-01-02 12:48:48 -06:00
Joseph Milazzo
9168e12483 Refactored Volume to have Name and Number (int) so that we can properly sort and still handle possible split volumes.
Refactored ScanLibrary into Library controller and updated it so it adds the new library to all admins.
2021-01-02 12:21:36 -06:00
Joseph Milazzo
55a44000fc Changed scan to POST and added new API for route gurads on UI. 2021-01-02 09:25:45 -06:00
Joseph Milazzo
fa71a40990 Fixed warnings in code 2021-01-01 14:18:29 -06:00
Joseph Milazzo
c429c50ba2 Fixed a bug in ScanLibrary that caused duplicated Volumes. Implemented APIs for navigating down to Volume for webui.
This is rough code and needs to be polished and refactored.
2021-01-01 14:04:31 -06:00
Joseph Milazzo
8c80ed090d Added basic parsing and unit tests. Setting up an intermediate object to hold information from parser. Removed hangfire db from git. 2020-12-27 11:14:45 -06:00
Joseph Milazzo
4fd9943b91 Added Hangfire with LiteDB for a task running system. At the most basic, this allows us to monitor tasks running on the system (during dev only) and run tasks on a reoccuring or ad-hoc basis. 2020-12-26 14:03:35 -06:00
Andrew Song
2d066ea36c Fixing sonar code quality recommendations 2020-12-25 15:08:40 -06:00
Joseph Milazzo
a40bc9e9f7 Added Policy for getUsers and moved some APIs. 2020-12-24 10:13:22 -06:00
Joseph Milazzo
8156aeb495 Some api work 2020-12-20 17:47:18 -06:00
Joseph Milazzo
b6e0e05205 Added new many to many migration for libraries and users. Add Library now works and you can get Library (entity) objects from a user. LibraryDto project is upcoming. 2020-12-18 12:01:45 -06:00
Joseph Milazzo
d5eed4e85d Lots of changes to get code ready to add library. 2020-12-17 11:27:19 -06:00
Joseph Milazzo
13ed323949 Added new API for getting Member (aka Users but for use in FE). User is just used for login/registering. 2020-12-14 14:33:09 -06:00