merge from trunk

Changelog.yaml

@@ -19,6 +19,66 @@
 #  new recipes:
 #    - title: 
 
+- version: 0.8.7
+  date: 2011-06-24
+
+  new features:
+    - title: "Connect to iTunes: You now need to tell iTunes to keep its own copy of every ebook. Do this in iTunes by going to Preferences->Advanced and setting the 'Copy files to iTunes Media folder when adding to library' option. To learn about why this is necessary, see: http://www.mobileread.com/forums/showthread.php?t=140260"
+      type: major
+
+    - title: "Add a couple of date related functions to the calibre template langauge to get 'todays' date and create text based on the value of a date type field"
+
+    - title: "Improved reading of metadata from FB2 files, with support for reading isbns, tags, published date, etc."
+
+    - title: "Driver for the Imagine IMEB5"
+      tickets: [800642]
+
+    - title: "Show the currently used network proxies in Preferences->Miscellaneous"
+
+    - title: "Kobo Touch driver: Show Favorites as a device collection. Various other minor fixes."
+
+    - title: "Content server now sends the Content-Disposition header when sending ebook files."
+
+    - title: "Allow search and replace on comments custom columns."
+
+    - title: "Add a new action 'Quick View' to show the books in your library by the author/tags/series/etc. of the currently selected book, in a separate window. You can add it to your toolbar or right click menu by going to Preferences->Toolbars."
+
+    - title: "Get Books: Add libri.de as a book source. Fix a bug that caused some books downloads to fail. Fixes to the Legimi and  beam-ebooks.de stores"
+      tickets: [799367]
+  
+  bug fixes:
+    - title: "Fix a memory leak that could result in the leaking of several MB of memory with large libraries"
+      tickets: [800952]
+
+    - title: "Fix the read metadata from format button in the edit metadata dialog using the wrong timezone when setting published date"
+      tickets: [799777]
+
+    - title: "Generating catalog: Fix occassional file in use errors when generating catalogs on windows"
+
+    - title: "Fix clicking on News in Tag Browser not working in non English locales."
+      tickets: [799471]
+
+    - title: "HTML Input: Fix a regression in 0.8.6 that caused CSS stylesheets to be ignored"
+      tickets: [799171]
+
+    - title: "Fix a regression that caused restore database to stop working on some windows sytems"
+
+    - title: "EPUB Output: Convert <br> tags with text in them into <divs> as ADE cannot handle them."
+      tickets: [794427]
+
+  improved recipes:
+    - Le Temps
+    - Perfil
+    - Financial Times UK
+
+  new recipes:
+    - title: "Daytona Beach Journal"
+      author: BRGriff
+
+    - title: "El club del ebook and Frontline"
+      author: Darko Miletic
+
+
 - version: 0.8.6
   date: 2011-06-17
 

recipes/daytona_beach.recipe

@@ -0,0 +1,78 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class DaytonBeachNewsJournal(BasicNewsRecipe):
+    title          ='Daytona Beach News Journal'
+    __author__ = 'BRGriff'
+    pubisher  = 'News-JournalOnline.com'
+    description           = 'Daytona Beach, Florida, Newspaper'
+    category              = 'News, Daytona Beach, Florida'
+    oldest_article = 1
+    max_articles_per_feed = 100
+    remove_javascript = True
+    use_embedded_content   = False
+    no_stylesheets = True
+    language = 'en'
+    filterDuplicates = True
+    remove_attributes = ['style']
+
+    keep_only_tags     = [dict(name='div', attrs={'class':'page-header'}),
+                                    dict(name='div', attrs={'class':'asset-body'})
+                                        ]
+    remove_tags         = [dict(name='div', attrs={'class':['byline-section', 'asset-meta']})
+                                      ]
+
+    feeds            = [
+                             #####NEWS#####
+                            (u"News", u"http://www.news-journalonline.com/rss.xml"),
+                            (u"Breaking News", u"http://www.news-journalonline.com/breakingnews/rss.xml"),
+                            (u"Local - East Volusia", u"http://www.news-journalonline.com/news/local/east-volusia/rss.xml"),
+                            (u"Local - West Volusia", u"http://www.news-journalonline.com/news/local/west-volusia/rss.xml"),
+                            (u"Local - Southeast", u"http://www.news-journalonline.com/news/local/southeast-volusia/rss.xml"),
+                            (u"Local - Flagler", u"http://www.news-journalonline.com/news/local/flagler/rss.xml"),
+                            (u"Florida", u"http://www.news-journalonline.com/news/florida/rss.xml"),
+                            (u"National/World", u"http://www.news-journalonline.com/news/nationworld/rss.xml"),
+                            (u"Politics", u"http://www.news-journalonline.com/news/politics/rss.xml"),
+                            (u"News of Record", u"http://www.news-journalonline.com/news/news-of-record/rss.xml"),
+                             ####BUSINESS####
+                            (u"Business", u"http://www.news-journalonline.com/business/rss.xml"),
+                            #(u"Jobs", u"http://www.news-journalonline.com/business/jobs/rss.xml"),
+                            #(u"Markets", u"http://www.news-journalonline.com/business/markets/rss.xml"),
+                            #(u"Real Estate", u"http://www.news-journalonline.com/business/real-estate/rss.xml"),
+                            #(u"Technology", u"http://www.news-journalonline.com/business/technology/rss.xml"),
+                              ####SPORTS####
+                            (u"Sports", u"http://www.news-journalonline.com/sports/rss.xml"),
+                            (u"Racing", u"http://www.news-journalonline.com/racing/rss.xml"),
+                            (u"Highschool", u"http://www.news-journalonline.com/sports/highschool/rss.xml"),
+                            (u"College", u"http://www.news-journalonline.com/sports/college/rss.xml"),
+                            (u"Basketball", u"http://www.news-journalonline.com/sports/basketball/rss.xml"),
+                            (u"Football", u"http://www.news-journalonline.com/sports/football/rss.xml"),
+                            (u"Golf", u"http://www.news-journalonline.com/sports/golf/rss.xml"),
+                            (u"Other Sports", u"http://www.news-journalonline.com/sports/other/rss.xml"),
+                              ####LIFESTYLE####
+                            (u"Lifestyle", u"http://www.news-journalonline.com/lifestyle/rss.xml"),
+                            #(u"Fashion", u"http://www.news-journalonline.com/lifestyle/fashion/rss.xml"),
+                            (u"Food", u"http://www.news-journalonline.com/lifestyle/food/rss.xml"),
+                            #(u"Health", u"http://www.news-journalonline.com/lifestyle/health/rss.xml"),
+                            (u"Home and Garden", u"http://www.news-journalonline.com/lifestyle/home-and-garden/rss.xml"),
+                            (u"Living", u"http://www.news-journalonline.com/lifestyle/living/rss.xml"),
+                            (u"Religion", u"http://www.news-journalonline.com/lifestyle/religion/rss.xml"),
+                            #(u"Travel", u"http://www.news-journalonline.com/lifestyle/travel/rss.xml"),
+                              ####OPINION####
+                            #(u"Opinion", u"http://www.news-journalonline.com/opinion/rss.xml"),
+                            #(u"Letters to Editor", u"http://www.news-journalonline.com/opinion/letters-to-the-editor/rss.xml"),
+                            #(u"Columns", u"http://www.news-journalonline.com/columns/rss.xml"),
+                            #(u"Podcasts", u"http://www.news-journalonline.com/podcasts/rss.xml"),
+                              ####ENTERTAINMENT####    ##Weekly Feature##
+                            (u"Entertainment", u"http://www.go386.com/rss.xml"),
+                            (u"Go Out", u"http://www.go386.com/go/rss.xml"),
+                            (u"Music", u"http://www.go386.com/music/rss.xml"),
+                            (u"Movies", u"http://www.go386.com/movies/rss.xml"),
+                            #(u"Culture", u"http://www.go386.com/culture/rss.xml"),
+
+                            ]
+
+    extra_css = '''
+                    .page-header{font-family:Arial,Helvetica,sans-serif; font-style:bold;font-size:22pt;}
+                    .asset-body{font-family:Helvetica,Arial,sans-serif; font-size:16pt;}
+
+      '''

recipes/elclubdelebook.recipe

@@ -0,0 +1,61 @@
+
+__license__   = 'GPL v3'
+__copyright__ = '2011, Darko Miletic <darko.miletic at gmail.com>'
+'''
+www.clubdelebook.com
+'''
+
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class ElClubDelEbook(BasicNewsRecipe):
+    title                 = 'El club del ebook'
+    __author__            = 'Darko Miletic'
+    description           = 'El Club del eBook, es la primera fuente de informacion sobre ebooks de Argentina. Aca vas a encontrar noticias, tips, tutoriales, recursos y opiniones sobre el mundo de los libros electronicos.'
+    tags                  = 'ebook, libro electronico, e-book, ebooks, libros electronicos, e-books'
+    oldest_article        = 7
+    max_articles_per_feed = 100
+    language              = 'es_AR'
+    encoding              = 'utf-8'
+    no_stylesheets        = True
+    use_embedded_content  = True
+    publication_type      = 'blog'
+    masthead_url          = 'http://dl.dropbox.com/u/2845131/elclubdelebook.png'
+    extra_css             = """
+                                body{font-family: Arial,Helvetica,sans-serif}
+                                img{ margin-bottom: 0.8em;
+                                     border: 1px solid #333333;
+                                     padding: 4px; display: block
+                                   }
+                            """
+
+    conversion_options = {
+                          'comment'  : description
+                        , 'tags'     : tags
+                        , 'publisher': title
+                        , 'language' : language
+                        }
+
+    remove_tags       = [dict(attrs={'id':'crp_related'})]
+    remove_tags_after = dict(attrs={'id':'crp_related'})
+
+    feeds = [(u'Articulos', u'http://feeds.feedburner.com/ElClubDelEbook')]
+
+    def preprocess_html(self, soup):
+        for item in soup.findAll(style=True):
+            del item['style']
+        for item in soup.findAll('a'):
+            limg = item.find('img')
+            if item.string is not None:
+               str = item.string
+               item.replaceWith(str)
+            else:
+               if limg:
+                  item.name = 'div'
+                  item.attrs = []
+               else:
+                   str = self.tag_to_string(item)
+                   item.replaceWith(str)
+        for item in soup.findAll('img'):
+            if not item.has_key('alt'):
+               item['alt'] = 'image'
+        return soup

recipes/financial_times_uk.recipe

@@ -1,15 +1,17 @@
 __license__   = 'GPL v3'
 __copyright__ = '2010-2011, Darko Miletic <darko.miletic at gmail.com>'
 '''
-ft.com
+www.ft.com/uk-edition
 '''
 from calibre import strftime
 from calibre.web.feeds.news import BasicNewsRecipe
 
 class FinancialTimes(BasicNewsRecipe):
-    title                 = u'Financial Times - UK printed edition'
+    title                 = 'Financial Times - UK printed edition'
     __author__            = 'Darko Miletic'
-    description           = 'Financial world news'
+    description           = "The Financial Times (FT) is one of the world's leading business news and information organisations, recognised internationally for its authority, integrity and accuracy."
+    publisher             = 'The Financial Times Ltd.'
+    category              = 'news, finances, politics, UK, World'
     oldest_article        = 2
     language              = 'en_GB'
     max_articles_per_feed = 250
@@ -17,14 +19,24 @@ class FinancialTimes(BasicNewsRecipe):
     use_embedded_content  = False
     needs_subscription    = True
     encoding              = 'utf8'
-    simultaneous_downloads= 1
+    publication_type      = 'newspaper'
-    delay                 = 1
+    cover_url             = strftime('http://specials.ft.com/vtf_pdf/%d%m%y_FRONT1_LON.pdf')
+    masthead_url          = 'http://im.media.ft.com/m/img/masthead_main.jpg'
     LOGIN                 = 'https://registration.ft.com/registration/barrier/login'
     INDEX                 = 'http://www.ft.com/uk-edition'
     PREFIX                = 'http://www.ft.com'
 
+    conversion_options = {
+                          'comment'          : description
+                        , 'tags'             : category
+                        , 'publisher'        : publisher
+                        , 'language'         : language
+                        , 'linearize_tables' : True
+                        }
+
     def get_browser(self):
         br = BasicNewsRecipe.get_browser()
+        br.open(self.INDEX)
         if self.username is not None and self.password is not None:
             br.open(self.LOGIN)
             br.select_form(name='loginForm')
@@ -33,29 +45,34 @@ class FinancialTimes(BasicNewsRecipe):
             br.submit()
         return br
 
-    keep_only_tags    = [ dict(name='div', attrs={'id':'cont'}) ]
+    keep_only_tags = [dict(name='div', attrs={'class':['fullstory fullstoryHeader','fullstory fullstoryBody','ft-story-header','ft-story-body','index-detail']})]
-    remove_tags_after = dict(name='p', attrs={'class':'copyright'})
     remove_tags = [
                       dict(name='div', attrs={'id':'floating-con'})
                      ,dict(name=['meta','iframe','base','object','embed','link'])
+                     ,dict(attrs={'class':['storyTools','story-package','screen-copy','story-package separator','expandable-image']})
                   ]
     remove_attributes = ['width','height','lang']
 
     extra_css = """
-                body{font-family:Arial,Helvetica,sans-serif;}
+                body{font-family: Georgia,Times,"Times New Roman",serif}
-                h2{font-size:large;}
+                h2{font-size:large}
-                .ft-story-header{font-size:xx-small;}
+                .ft-story-header{font-size: x-small}
-                .ft-story-body{font-size:small;}
-                a{color:#003399;}
                 .container{font-size:x-small;}
                 h3{font-size:x-small;color:#003399;}
                 .copyright{font-size: x-small}
+                img{margin-top: 0.8em; display: block}
+                .lastUpdated{font-family: Arial,Helvetica,sans-serif; font-size: x-small}
+                .byline,.ft-story-body,.ft-story-header{font-family: Arial,Helvetica,sans-serif}
                 """
 
     def get_artlinks(self, elem):
         articles = []
         for item in elem.findAll('a',href=True):
-            url   = self.PREFIX + item['href']
+            rawlink = item['href']
+            if rawlink.startswith('http://'):
+               url = rawlink
+            else:
+               url   = self.PREFIX + rawlink
             title = self.tag_to_string(item)
             date = strftime(self.timefmt)
             articles.append({
@@ -65,7 +82,7 @@ class FinancialTimes(BasicNewsRecipe):
                              ,'description':''
                             })
         return articles
-                
+
     def parse_index(self):
         feeds = []
         soup = self.index_to_soup(self.INDEX)
@@ -80,11 +97,34 @@ class FinancialTimes(BasicNewsRecipe):
            strest.insert(0,st)
         for item in strest:
             ftitle   = self.tag_to_string(item)
-            self.report_progress(0, _('Fetching feed')+' %s...'%(ftitle))            
+            self.report_progress(0, _('Fetching feed')+' %s...'%(ftitle))
             feedarts = self.get_artlinks(item.parent.ul)
             feeds.append((ftitle,feedarts))
         return feeds
 
     def preprocess_html(self, soup):
-        return self.adeify_images(soup)
+        items = ['promo-box','promo-title',
-
+                 'promo-headline','promo-image',
+                 'promo-intro','promo-link','subhead']
+        for item in items:
+            for it in soup.findAll(item):
+                it.name = 'div'
+                it.attrs = []
+        for item in soup.findAll(style=True):
+            del item['style']
+        for item in soup.findAll('a'):
+            limg = item.find('img')
+            if item.string is not None:
+               str = item.string
+               item.replaceWith(str)
+            else:
+               if limg:
+                  item.name = 'div'
+                  item.attrs = []
+               else:
+                   str = self.tag_to_string(item)
+                   item.replaceWith(str)
+        for item in soup.findAll('img'):
+            if not item.has_key('alt'):
+               item['alt'] = 'image'
+        return soup

recipes/frontlineonnet.recipe

@@ -0,0 +1,81 @@
+__license__   = 'GPL v3'
+__copyright__ = '2011, Darko Miletic <darko.miletic at gmail.com>'
+'''
+frontlineonnet.com
+'''
+
+import re
+from calibre import strftime
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class Frontlineonnet(BasicNewsRecipe):
+    title                = 'Frontline'
+    __author__           = 'Darko Miletic'
+    description          = "India's national magazine"
+    publisher            = 'Frontline'
+    category             = 'news, politics, India'
+    no_stylesheets       = True
+    delay                = 1
+    INDEX                = 'http://frontlineonnet.com/'
+    use_embedded_content = False
+    encoding             = 'cp1252'
+    language             = 'en_IN'
+    publication_type     = 'magazine'
+    masthead_url         = 'http://frontlineonnet.com/images/newfline.jpg'
+    extra_css            = """
+                              body{font-family: Verdana,Arial,Helvetica,sans-serif}
+                              img{margin-top:0.5em; margin-bottom: 0.7em; display: block}
+                           """
+
+    conversion_options = {
+                          'comment'          : description
+                        , 'tags'             : category
+                        , 'publisher'        : publisher
+                        , 'language'         : language
+                        , 'linearize_tables' : True
+                        }
+
+    preprocess_regexps = [
+                           (re.compile(r'.*?<base', re.DOTALL|re.IGNORECASE),lambda match: '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html dir="ltr" xml:lang="en-IN"><head><title>title</title><base')
+                          ,(re.compile(r'<base .*?>', re.DOTALL|re.IGNORECASE),lambda match: '</head><body>')
+                          ,(re.compile(r'<byline>', re.DOTALL|re.IGNORECASE),lambda match: '<div class="byline">')
+                          ,(re.compile(r'</byline>', re.DOTALL|re.IGNORECASE),lambda match: '</div>')
+                          ,(re.compile(r'<center>', re.DOTALL|re.IGNORECASE),lambda match: '<div class="ctr">')
+                          ,(re.compile(r'</center>', re.DOTALL|re.IGNORECASE),lambda match: '</div>')
+                         ]
+
+    keep_only_tags= [
+                      dict(name='font', attrs={'class':'storyhead'})
+                     ,dict(attrs={'class':'byline'})
+                    ]
+    remove_attributes=['size','noshade','border']
+
+    def preprocess_html(self, soup):
+        for item in soup.findAll(style=True):
+            del item['style']
+        for item in soup.findAll('img'):
+            if not item.has_key('alt'):
+               item['alt'] = 'image'
+        return soup
+
+    def parse_index(self):
+        articles = []
+        soup = self.index_to_soup(self.INDEX)
+        for feed_link in soup.findAll('a',href=True):
+            if feed_link['href'].startswith('stories/'):
+                url   = self.INDEX + feed_link['href']
+                title = self.tag_to_string(feed_link)
+                date  = strftime(self.timefmt)
+                articles.append({
+                                  'title'      :title
+                                 ,'date'       :date
+                                 ,'url'        :url
+                                 ,'description':''
+                                })
+        return [('Frontline', articles)]
+
+    def print_version(self, url):
+        return "http://www.hinduonnet.com/thehindu/thscrip/print.pl?prd=fline&file=" + url.rpartition('/')[2]
+
+    def image_url_processor(self, baseurl, url):
+        return url.replace('../images/', self.INDEX + 'images/').strip()

recipes/le_temps.recipe

@@ -14,7 +14,7 @@ class LeTemps(BasicNewsRecipe):
      title          = u'Le Temps'
      oldest_article = 7
      max_articles_per_feed = 100
-     __author__ = 'Sujata Raman'
+     __author__ = 'Kovid Goyal'
      description = 'French news. Needs a subscription from http://www.letemps.ch'
      no_stylesheets = True
      remove_javascript = True
@@ -27,6 +27,7 @@ class LeTemps(BasicNewsRecipe):
      def get_browser(self):
          br = BasicNewsRecipe.get_browser(self)
          br.open('http://www.letemps.ch/login')
+         br.select_form(nr=1)
          br['username'] = self.username
          br['password'] = self.password
          raw = br.submit().read()

recipes/perfil.recipe

@@ -26,6 +26,7 @@ class Perfil(BasicNewsRecipe):
                                 .foto1 h1{font-size: x-small}
                                 h1{font-family: Georgia,"Times New Roman",serif}
                                 img{margin-bottom: 0.4em}
+                                .hora{font-size: x-small; color: red}
                             """
 
     conversion_options = {
@@ -60,7 +61,26 @@ class Perfil(BasicNewsRecipe):
              ,(u'Tecnologia'     , u'http://www.perfil.com/rss/tecnologia.xml'   )
             ]
 
+    def get_article_url(self, article):
+        return article.get('guid',  None)
+            
     def preprocess_html(self, soup):
         for item in soup.findAll(style=True):
             del item['style']
+        for item in soup.findAll('a'):
+            limg = item.find('img')
+            if item.string is not None:
+               str = item.string
+               item.replaceWith(str)
+            else:
+               if limg:
+                  item.name = 'div'
+                  item.attrs = []
+               else:
+                   str = self.tag_to_string(item)
+                   item.replaceWith(str)
+        for item in soup.findAll('img'):
+            if not item.has_key('alt'):
+               item['alt'] = 'image'
         return soup
+        

recipes/wsj.recipe

@@ -51,7 +51,7 @@ class WallStreetJournal(BasicNewsRecipe):
             br['password'] = self.password
             res = br.submit()
             raw = res.read()
-            if 'Welcome,' not in raw:
+            if 'Welcome,' not in raw and '>Logout<' not in raw:
                 raise ValueError('Failed to log in to wsj.com, check your '
                         'username and password')
         return br

resources/metadata_sqlite.sql

@@ -13,8 +13,10 @@ CREATE TABLE books ( id      INTEGER PRIMARY KEY AUTOINCREMENT,
                              isbn TEXT DEFAULT "" COLLATE NOCASE,
                              lccn TEXT DEFAULT "" COLLATE NOCASE,
                              path TEXT NOT NULL DEFAULT "",
-                             flags INTEGER NOT NULL DEFAULT 1
+                             flags INTEGER NOT NULL DEFAULT 1,
-                        , uuid TEXT, has_cover BOOL DEFAULT 0, last_modified TIMESTAMP NOT NULL DEFAULT "2000-01-01 00:00:00+00:00");
+                             uuid TEXT,
+                             has_cover BOOL DEFAULT 0,
+                             last_modified TIMESTAMP NOT NULL DEFAULT "2000-01-01 00:00:00+00:00");
 CREATE TABLE books_authors_link ( id INTEGER PRIMARY KEY,
                                           book INTEGER NOT NULL,
                                           author INTEGER NOT NULL,

src/calibre/__init__.py

@@ -578,6 +578,7 @@ def url_slash_cleaner(url):
 def get_download_filename(url, cookie_file=None):
     '''
     Get a local filename for a URL using the content disposition header
+    Returns empty string if no content disposition header present
     '''
     from contextlib import closing
     from urllib2 import unquote as urllib2_unquote
@@ -591,8 +592,10 @@ def get_download_filename(url, cookie_file=None):
         cj.load(cookie_file)
         br.set_cookiejar(cj)
 
+    last_part_name = ''
     try:
         with closing(br.open(url)) as r:
+            last_part_name = r.geturl().split('/')[-1]
             disposition = r.info().get('Content-disposition', '')
             for p in disposition.split(';'):
                 if 'filename' in p:
@@ -612,7 +615,7 @@ def get_download_filename(url, cookie_file=None):
         traceback.print_exc()
 
     if not filename:
-        filename = r.geturl().split('/')[-1]
+        filename = last_part_name
 
     return filename
 

src/calibre/constants.py

@@ -4,7 +4,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = u'calibre'
-numeric_version = (0, 8, 6)
+numeric_version = (0, 8, 7)
 __version__   = u'.'.join(map(unicode, numeric_version))
 __author__    = u"Kovid Goyal <kovid@kovidgoyal.net>"
 

src/calibre/customize/builtins.py

@@ -762,99 +762,132 @@ plugins += input_profiles + output_profiles
 class ActionAdd(InterfaceActionBase):
     name = 'Add Books'
     actual_plugin = 'calibre.gui2.actions.add:AddAction'
+    description = _('Add books to calibre or the connected device')
 
 class ActionFetchAnnotations(InterfaceActionBase):
     name = 'Fetch Annotations'
     actual_plugin = 'calibre.gui2.actions.annotate:FetchAnnotationsAction'
+    description = _('Fetch annotations from a connected Kindle (experimental)')
 
 class ActionGenerateCatalog(InterfaceActionBase):
     name = 'Generate Catalog'
     actual_plugin = 'calibre.gui2.actions.catalog:GenerateCatalogAction'
+    description = _('Generate a catalog of the books in your calibre library')
 
 class ActionConvert(InterfaceActionBase):
     name = 'Convert Books'
     actual_plugin = 'calibre.gui2.actions.convert:ConvertAction'
+    description = _('Convert books to various ebook formats')
 
 class ActionDelete(InterfaceActionBase):
     name = 'Remove Books'
     actual_plugin = 'calibre.gui2.actions.delete:DeleteAction'
+    description = _('Delete books from your calibre library or connected device')
 
 class ActionEditMetadata(InterfaceActionBase):
     name = 'Edit Metadata'
     actual_plugin = 'calibre.gui2.actions.edit_metadata:EditMetadataAction'
+    description = _('Edit the metadata of books in your calibre library')
 
 class ActionView(InterfaceActionBase):
     name = 'View'
     actual_plugin = 'calibre.gui2.actions.view:ViewAction'
+    description = _('Read books in your calibre library')
 
 class ActionFetchNews(InterfaceActionBase):
     name = 'Fetch News'
     actual_plugin = 'calibre.gui2.actions.fetch_news:FetchNewsAction'
+    description = _('Download news from the internet in ebook form')
+
+class ActionQuickview(InterfaceActionBase):
+    name = 'Show Quickview'
+    actual_plugin = 'calibre.gui2.actions.show_quickview:ShowQuickviewAction'
+    description = _('Show a list of related books quickly')
 
 class ActionSaveToDisk(InterfaceActionBase):
     name = 'Save To Disk'
     actual_plugin = 'calibre.gui2.actions.save_to_disk:SaveToDiskAction'
+    description = _('Export books from your calibre library to the hard disk')
 
 class ActionShowBookDetails(InterfaceActionBase):
     name = 'Show Book Details'
     actual_plugin = 'calibre.gui2.actions.show_book_details:ShowBookDetailsAction'
+    description = _('Show book details in a separate popup')
 
 class ActionRestart(InterfaceActionBase):
     name = 'Restart'
     actual_plugin = 'calibre.gui2.actions.restart:RestartAction'
+    description = _('Restart calibre')
 
 class ActionOpenFolder(InterfaceActionBase):
     name = 'Open Folder'
     actual_plugin = 'calibre.gui2.actions.open:OpenFolderAction'
+    description = _('Open the folder that contains the book files in your'
+            ' calibre library')
 
 class ActionSendToDevice(InterfaceActionBase):
     name = 'Send To Device'
     actual_plugin = 'calibre.gui2.actions.device:SendToDeviceAction'
+    description = _('Send books to the connected device')
 
 class ActionConnectShare(InterfaceActionBase):
     name = 'Connect Share'
     actual_plugin = 'calibre.gui2.actions.device:ConnectShareAction'
+    description = _('Send books via email or the web also connect to iTunes or'
+            ' folders on your computer as if they are devices')
 
 class ActionHelp(InterfaceActionBase):
     name = 'Help'
     actual_plugin = 'calibre.gui2.actions.help:HelpAction'
+    description = _('Browse the calibre User Manual')
 
 class ActionPreferences(InterfaceActionBase):
     name = 'Preferences'
     actual_plugin = 'calibre.gui2.actions.preferences:PreferencesAction'
+    description = _('Customize calibre')
 
 class ActionSimilarBooks(InterfaceActionBase):
     name = 'Similar Books'
     actual_plugin = 'calibre.gui2.actions.similar_books:SimilarBooksAction'
+    description = _('Easily find books similar to the currently selected one')
 
 class ActionChooseLibrary(InterfaceActionBase):
     name = 'Choose Library'
     actual_plugin = 'calibre.gui2.actions.choose_library:ChooseLibraryAction'
+    description = _('Switch between different calibre libraries and perform'
+            ' maintenance on them')
 
 class ActionAddToLibrary(InterfaceActionBase):
     name = 'Add To Library'
     actual_plugin = 'calibre.gui2.actions.add_to_library:AddToLibraryAction'
+    description = _('Copy books from the devce to your calibre library')
 
 class ActionEditCollections(InterfaceActionBase):
     name = 'Edit Collections'
     actual_plugin = 'calibre.gui2.actions.edit_collections:EditCollectionsAction'
+    description = _('Edit the collections in which books are placed on your device')
 
 class ActionCopyToLibrary(InterfaceActionBase):
     name = 'Copy To Library'
     actual_plugin = 'calibre.gui2.actions.copy_to_library:CopyToLibraryAction'
+    description = _('Copy a book from one calibre library to another')
 
 class ActionTweakEpub(InterfaceActionBase):
     name = 'Tweak ePub'
     actual_plugin = 'calibre.gui2.actions.tweak_epub:TweakEpubAction'
+    description = _('Make small tweaks to epub files in your calibre library')
 
 class ActionNextMatch(InterfaceActionBase):
     name = 'Next Match'
     actual_plugin = 'calibre.gui2.actions.next_match:NextMatchAction'
+    description = _('Find the next or previous match when searching in '
+            'your calibre library in highlight mode')
 
 class ActionStore(InterfaceActionBase):
     name = 'Store'
     author = 'John Schember'
     actual_plugin = 'calibre.gui2.actions.store:StoreAction'
+    description = _('Search for books from different book sellers')
 
     def customization_help(self, gui=False):
         return 'Customize the behavior of the store search.'
@@ -870,13 +903,13 @@ class ActionStore(InterfaceActionBase):
 class ActionPluginUpdater(InterfaceActionBase):
     name = 'Plugin Updater'
     author = 'Grant Drake'
-    description = 'Queries the MobileRead forums for updates to plugins to install'
+    description = _('Get new calibre plugins or update your existing ones')
     actual_plugin = 'calibre.gui2.actions.plugin_updates:PluginUpdaterAction'
 
 plugins += [ActionAdd, ActionFetchAnnotations, ActionGenerateCatalog,
         ActionConvert, ActionDelete, ActionEditMetadata, ActionView,
-        ActionFetchNews, ActionSaveToDisk, ActionShowBookDetails,
+        ActionFetchNews, ActionSaveToDisk, ActionQuickview,
-        ActionRestart, ActionOpenFolder, ActionConnectShare,
+        ActionShowBookDetails,ActionRestart, ActionOpenFolder, ActionConnectShare,
         ActionSendToDevice, ActionHelp, ActionPreferences, ActionSimilarBooks,
         ActionAddToLibrary, ActionEditCollections, ActionChooseLibrary,
         ActionCopyToLibrary, ActionTweakEpub, ActionNextMatch, ActionStore,
@@ -1307,6 +1340,16 @@ class StoreLegimiStore(StoreBase):
     headquarters = 'PL'
     formats = ['EPUB']
 
+class StoreLibreDEStore(StoreBase):
+    name = 'Libri DE'
+    author = 'Charles Haley'
+    description = u'Sicher Bücher, Hörbücher und Downloads online bestellen.'
+    actual_plugin = 'calibre.gui2.store.libri_de_plugin:LibreDEStore'
+
+    headquarters = 'DE'
+    formats = ['EPUB', 'PDF']
+    affiliate = True
+
 class StoreManyBooksStore(StoreBase):
     name = 'ManyBooks'
     description = u'Public domain and creative commons works from many sources.'
@@ -1335,6 +1378,14 @@ class StoreNextoStore(StoreBase):
     formats = ['EPUB', 'PDF']
     affiliate = True
 
+class StoreOpenBooksStore(StoreBase):
+    name = 'Open Books'
+    description = u'Comprehensive listing of DRM free ebooks from a variety of sources provided by users of calibre.'
+    actual_plugin = 'calibre.gui2.store.open_books_plugin:OpenBooksStore'
+    
+    drm_free_only = True
+    headquarters = 'US'
+
 class StoreOpenLibraryStore(StoreBase):
     name = 'Open Library'
     description = u'One web page for every book ever published. The goal is to be a true online library. Over 20 million records from a variety of large catalogs as well as single contributions, with more on the way.'
@@ -1457,9 +1508,11 @@ plugins += [
     StoreGutenbergStore,
     StoreKoboStore,
     StoreLegimiStore,
+    StoreLibreDEStore,
     StoreManyBooksStore,
     StoreMobileReadStore,
     StoreNextoStore,
+    StoreOpenBooksStore,
     StoreOpenLibraryStore,
     StoreOReillyStore,
     StorePragmaticBookshelfStore,

src/calibre/customize/conversion.py

@@ -259,6 +259,10 @@ class OutputFormatPlugin(Plugin):
     #: (option_name, recommended_value, recommendation_level)
     recommendations = set([])
 
+    @property
+    def description(self):
+        return _('Convert ebooks to the %s format')%self.file_type
+
     def __init__(self, *args):
         Plugin.__init__(self, *args)
         self.report_progress = DummyReporter()

src/calibre/db/__init__.py

@@ -0,0 +1,67 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+
+'''
+Rewrite of the calibre database backend.
+
+Broad Objectives:
+
+    * Use the sqlite db only as a datastore. i.e. do not do
+      sorting/searching/concatenation or anything else in sqlite. Instead
+      mirror the sqlite tables in memory, create caches and lookup maps from
+      them and create a set_* API that updates the memory caches and the sqlite
+      correctly.
+
+    * Move from keeping a list of books in memory as a cache to a per table
+      cache. This allows much faster search and sort operations at the expense
+      of slightly slower lookup operations. That slowdown can be mitigated by
+      keeping lots of maps and updating them in the set_* API. Also
+      get_categories becomes blazingly fast.
+
+    * Separate the database layer from the cache layer more cleanly. Rather
+      than having the db layer refer to the cache layer and vice versa, the
+      cache layer will refer to the db layer only and the new API will be
+      defined on the cache layer.
+
+    * Get rid of index_is_id and other poor design decisions
+
+    * Minimize the API as much as possible and define it cleanly
+
+    * Do not change the on disk format of metadata.db at all (this is for
+      backwards compatibility)
+
+    * Get rid of the need for a separate db access thread by switching to apsw
+      to access sqlite, which is thread safe
+
+    * The new API will have methods to efficiently do bulk operations and will
+      use shared/exclusive/pending locks to serialize access to the in-mem data
+      structures. Use the same locking scheme as sqlite itself does.
+
+How this will proceed:
+
+    1. Create the new API
+    2. Create a test suite for it
+    3. Write a replacement for LibraryDatabase2 that uses the new API
+       internally
+    4. Lots of testing of calibre with the new LibraryDatabase2
+    5. Gradually migrate code to use the (much faster) new api wherever possible (the new api
+       will be exposed via db.new_api)
+
+    I plan to work on this slowly, in parallel to normal calibre development
+    work.
+
+Various things that require other things before they can be migrated:
+    1. From initialize_dynamic(): set_saved_searches,
+                    load_user_template_functions. Also add custom
+                    columns/categories/searches info into
+                    self.field_metadata. Finally, implement metadata dirtied
+                    functionality.
+
+'''

src/calibre/db/backend.py

@@ -0,0 +1,443 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+# Imports {{{
+import os, shutil, uuid, json
+from functools import partial
+
+import apsw
+
+from calibre import isbytestring, force_unicode, prints
+from calibre.constants import (iswindows, filesystem_encoding,
+        preferred_encoding)
+from calibre.ptempfile import PersistentTemporaryFile
+from calibre.library.schema_upgrades import SchemaUpgrade
+from calibre.library.field_metadata import FieldMetadata
+from calibre.ebooks.metadata import title_sort, author_to_author_sort
+from calibre.utils.icu import strcmp
+from calibre.utils.config import to_json, from_json, prefs, tweaks
+from calibre.utils.date import utcfromtimestamp
+from calibre.db.tables import (OneToOneTable, ManyToOneTable, ManyToManyTable,
+        SizeTable, FormatsTable, AuthorsTable, IdentifiersTable)
+# }}}
+
+'''
+Differences in semantics from pysqlite:
+
+    1. execute/executemany/executescript operate in autocommit mode
+
+'''
+
+class DynamicFilter(object): # {{{
+
+    'No longer used, present for legacy compatibility'
+
+    def __init__(self, name):
+        self.name = name
+        self.ids = frozenset([])
+
+    def __call__(self, id_):
+        return int(id_ in self.ids)
+
+    def change(self, ids):
+        self.ids = frozenset(ids)
+# }}}
+
+class DBPrefs(dict): # {{{
+
+    'Store preferences as key:value pairs in the db'
+
+    def __init__(self, db):
+        dict.__init__(self)
+        self.db = db
+        self.defaults = {}
+        self.disable_setting = False
+        for key, val in self.db.conn.get('SELECT key,val FROM preferences'):
+            try:
+                val = self.raw_to_object(val)
+            except:
+                prints('Failed to read value for:', key, 'from db')
+                continue
+            dict.__setitem__(self, key, val)
+
+    def raw_to_object(self, raw):
+        if not isinstance(raw, unicode):
+            raw = raw.decode(preferred_encoding)
+        return json.loads(raw, object_hook=from_json)
+
+    def to_raw(self, val):
+        return json.dumps(val, indent=2, default=to_json)
+
+    def __getitem__(self, key):
+        try:
+            return dict.__getitem__(self, key)
+        except KeyError:
+            return self.defaults[key]
+
+    def __delitem__(self, key):
+        dict.__delitem__(self, key)
+        self.db.conn.execute('DELETE FROM preferences WHERE key=?', (key,))
+
+    def __setitem__(self, key, val):
+        if self.disable_setting:
+            return
+        raw = self.to_raw(val)
+        self.db.conn.execute('INSERT OR REPLACE INTO preferences (key,val) VALUES (?,?)', (key,
+            raw))
+        dict.__setitem__(self, key, val)
+
+    def set(self, key, val):
+        self.__setitem__(key, val)
+
+# }}}
+
+# Extra collators {{{
+def pynocase(one, two, encoding='utf-8'):
+    if isbytestring(one):
+        try:
+            one = one.decode(encoding, 'replace')
+        except:
+            pass
+    if isbytestring(two):
+        try:
+            two = two.decode(encoding, 'replace')
+        except:
+            pass
+    return cmp(one.lower(), two.lower())
+
+def _author_to_author_sort(x):
+    if not x: return ''
+    return author_to_author_sort(x.replace('|', ','))
+
+def icu_collator(s1, s2):
+    return strcmp(force_unicode(s1, 'utf-8'), force_unicode(s2, 'utf-8'))
+# }}}
+
+class Connection(apsw.Connection): # {{{
+
+    BUSY_TIMEOUT = 2000 # milliseconds
+
+    def __init__(self, path):
+        apsw.Connection.__init__(self, path)
+
+        self.setbusytimeout(self.BUSY_TIMEOUT)
+        self.execute('pragma cache_size=5000')
+        self.conn.execute('pragma temp_store=2')
+
+        encoding = self.execute('pragma encoding').fetchone()[0]
+        self.conn.create_collation('PYNOCASE', partial(pynocase,
+            encoding=encoding))
+
+        self.conn.create_function('title_sort', 1, title_sort)
+        self.conn.create_function('author_to_author_sort', 1,
+                _author_to_author_sort)
+
+        self.conn.create_function('uuid4', 0, lambda : str(uuid.uuid4()))
+
+        # Dummy functions for dynamically created filters
+        self.conn.create_function('books_list_filter', 1, lambda x: 1)
+        self.conn.create_collation('icucollate', icu_collator)
+
+    def create_dynamic_filter(self, name):
+        f = DynamicFilter(name)
+        self.conn.create_function(name, 1, f)
+
+    def get(self, *args, **kw):
+        ans = self.cursor().execute(*args)
+        if kw.get('all', True):
+            return ans.fetchall()
+        for row in ans:
+            return ans[0]
+
+    def execute(self, sql, bindings=None):
+        cursor = self.cursor()
+        return cursor.execute(sql, bindings)
+
+    def executemany(self, sql, sequence_of_bindings):
+        return self.cursor().executemany(sql, sequence_of_bindings)
+
+    def executescript(self, sql):
+        with self:
+            # Use an explicit savepoint so that even if this is called
+            # while a transaction is active, it is atomic
+            return self.cursor().execute(sql)
+# }}}
+
+class DB(object, SchemaUpgrade):
+
+    PATH_LIMIT = 40 if iswindows else 100
+    WINDOWS_LIBRARY_PATH_LIMIT = 75
+
+    # Initialize database {{{
+
+    def __init__(self, library_path, default_prefs=None, read_only=False):
+        try:
+            if isbytestring(library_path):
+                library_path = library_path.decode(filesystem_encoding)
+        except:
+            import traceback
+            traceback.print_exc()
+
+        self.field_metadata = FieldMetadata()
+
+        self.library_path = os.path.abspath(library_path)
+        self.dbpath = os.path.join(library_path, 'metadata.db')
+        self.dbpath = os.environ.get('CALIBRE_OVERRIDE_DATABASE_PATH',
+                self.dbpath)
+
+        if iswindows and len(self.library_path) + 4*self.PATH_LIMIT + 10 > 259:
+            raise ValueError(_(
+                'Path to library too long. Must be less than'
+                ' %d characters.')%(259-4*self.PATH_LIMIT-10))
+        exists = self._exists = os.path.exists(self.dbpath)
+        if not exists:
+            # Be more strict when creating new libraries as the old calculation
+            # allowed for max path lengths of 265 chars.
+            if (iswindows and len(self.library_path) >
+                    self.WINDOWS_LIBRARY_PATH_LIMIT):
+                raise ValueError(_(
+                    'Path to library too long. Must be less than'
+                    ' %d characters.')%self.WINDOWS_LIBRARY_PATH_LIMIT)
+
+        if read_only and os.path.exists(self.dbpath):
+            # Work on only a copy of metadata.db to ensure that
+            # metadata.db is not changed
+            pt = PersistentTemporaryFile('_metadata_ro.db')
+            pt.close()
+            shutil.copyfile(self.dbpath, pt.name)
+            self.dbpath = pt.name
+
+        self.is_case_sensitive = (not iswindows and
+            not os.path.exists(self.dbpath.replace('metadata.db',
+                'MeTAdAtA.dB')))
+
+        self._conn = None
+
+        if self.user_version == 0:
+            self.initialize_database()
+
+        with self.conn:
+            SchemaUpgrade.__init__(self)
+
+        # Guarantee that the library_id is set
+        self.library_id
+
+        self.initialize_prefs(default_prefs)
+
+        # Fix legacy triggers and columns
+        self.conn.executescript('''
+        DROP TRIGGER IF EXISTS author_insert_trg;
+        CREATE TEMP TRIGGER author_insert_trg
+            AFTER INSERT ON authors
+            BEGIN
+            UPDATE authors SET sort=author_to_author_sort(NEW.name) WHERE id=NEW.id;
+        END;
+        DROP TRIGGER IF EXISTS author_update_trg;
+        CREATE TEMP TRIGGER author_update_trg
+            BEFORE UPDATE ON authors
+            BEGIN
+            UPDATE authors SET sort=author_to_author_sort(NEW.name)
+            WHERE id=NEW.id AND name <> NEW.name;
+        END;
+        UPDATE authors SET sort=author_to_author_sort(name) WHERE sort IS NULL;
+        ''')
+
+    def initialize_prefs(self, default_prefs):
+        self.prefs = DBPrefs(self)
+
+        if default_prefs is not None and not self._exists:
+            # Only apply default prefs to a new database
+            for key in default_prefs:
+                # be sure that prefs not to be copied are listed below
+                if key not in frozenset(['news_to_be_synced']):
+                    self.prefs[key] = default_prefs[key]
+            if 'field_metadata' in default_prefs:
+                fmvals = [f for f in default_prefs['field_metadata'].values()
+                                if f['is_custom']]
+                for f in fmvals:
+                    self.create_custom_column(f['label'], f['name'],
+                            f['datatype'], f['is_multiple'] is not None,
+                            f['is_editable'], f['display'])
+
+        defs = self.prefs.defaults
+        defs['gui_restriction'] = defs['cs_restriction'] = ''
+        defs['categories_using_hierarchy'] = []
+        defs['column_color_rules'] = []
+
+        # Migrate the bool tristate tweak
+        defs['bools_are_tristate'] = \
+                tweaks.get('bool_custom_columns_are_tristate', 'yes') == 'yes'
+        if self.prefs.get('bools_are_tristate') is None:
+            self.prefs.set('bools_are_tristate', defs['bools_are_tristate'])
+
+        # Migrate column coloring rules
+        if self.prefs.get('column_color_name_1', None) is not None:
+            from calibre.library.coloring import migrate_old_rule
+            old_rules = []
+            for i in range(1, 6):
+                col = self.prefs.get('column_color_name_'+str(i), None)
+                templ = self.prefs.get('column_color_template_'+str(i), None)
+                if col and templ:
+                    try:
+                        del self.prefs['column_color_name_'+str(i)]
+                        rules = migrate_old_rule(self.field_metadata, templ)
+                        for templ in rules:
+                            old_rules.append((col, templ))
+                    except:
+                        pass
+            if old_rules:
+                self.prefs['column_color_rules'] += old_rules
+
+        # Migrate saved search and user categories to db preference scheme
+        def migrate_preference(key, default):
+            oldval = prefs[key]
+            if oldval != default:
+                self.prefs[key] = oldval
+                prefs[key] = default
+            if key not in self.prefs:
+                self.prefs[key] = default
+
+        migrate_preference('user_categories', {})
+        migrate_preference('saved_searches', {})
+
+        # migrate grouped_search_terms
+        if self.prefs.get('grouped_search_terms', None) is None:
+            try:
+                ogst = tweaks.get('grouped_search_terms', {})
+                ngst = {}
+                for t in ogst:
+                    ngst[icu_lower(t)] = ogst[t]
+                self.prefs.set('grouped_search_terms', ngst)
+            except:
+                pass
+
+        # Rename any user categories with names that differ only in case
+        user_cats = self.prefs.get('user_categories', [])
+        catmap = {}
+        for uc in user_cats:
+            ucl = icu_lower(uc)
+            if ucl not in catmap:
+                catmap[ucl] = []
+            catmap[ucl].append(uc)
+        cats_changed = False
+        for uc in catmap:
+            if len(catmap[uc]) > 1:
+                prints('found user category case overlap', catmap[uc])
+                cat = catmap[uc][0]
+                suffix = 1
+                while icu_lower((cat + unicode(suffix))) in catmap:
+                    suffix += 1
+                prints('Renaming user category %s to %s'%(cat, cat+unicode(suffix)))
+                user_cats[cat + unicode(suffix)] = user_cats[cat]
+                del user_cats[cat]
+                cats_changed = True
+        if cats_changed:
+            self.prefs.set('user_categories', user_cats)
+
+    @property
+    def conn(self):
+        if self._conn is None:
+            self._conn = apsw.Connection(self.dbpath)
+            if self._exists and self.user_version == 0:
+                self._conn.close()
+                os.remove(self.dbpath)
+                self._conn = apsw.Connection(self.dbpath)
+        return self._conn
+
+    @dynamic_property
+    def user_version(self):
+        doc = 'The user version of this database'
+
+        def fget(self):
+            return self.conn.get('pragma user_version;', all=False)
+
+        def fset(self, val):
+            self.conn.execute('pragma user_version=%d'%int(val))
+
+        return property(doc=doc, fget=fget, fset=fset)
+
+    def initialize_database(self):
+        metadata_sqlite = P('metadata_sqlite.sql', data=True,
+                allow_user_override=False).decode('utf-8')
+        self.conn.executescript(metadata_sqlite)
+        if self.user_version == 0:
+            self.user_version = 1
+    # }}}
+
+    # Database layer API {{{
+
+    @classmethod
+    def exists_at(cls, path):
+        return path and os.path.exists(os.path.join(path, 'metadata.db'))
+
+    @dynamic_property
+    def library_id(self):
+        doc = ('The UUID for this library. As long as the user only operates'
+                ' on libraries with calibre, it will be unique')
+
+        def fget(self):
+            if getattr(self, '_library_id_', None) is None:
+                ans = self.conn.get('SELECT uuid FROM library_id', all=False)
+                if ans is None:
+                    ans = str(uuid.uuid4())
+                    self.library_id = ans
+                else:
+                    self._library_id_ = ans
+            return self._library_id_
+
+        def fset(self, val):
+            self._library_id_ = unicode(val)
+            self.conn.execute('''
+                    DELETE FROM library_id;
+                    INSERT INTO library_id (uuid) VALUES (?);
+                    ''', self._library_id_)
+
+        return property(doc=doc, fget=fget, fset=fset)
+
+    def last_modified(self):
+        ''' Return last modified time as a UTC datetime object '''
+        return utcfromtimestamp(os.stat(self.dbpath).st_mtime)
+
+    def read_tables(self):
+        tables = {}
+        for col in ('title', 'sort', 'author_sort', 'series_index', 'comments',
+                'timestamp', 'published', 'uuid', 'path', 'cover',
+                'last_modified'):
+            metadata = self.field_metadata[col].copy()
+            if metadata['table'] is None:
+                metadata['table'], metadata['column'] == 'books', ('has_cover'
+                        if col == 'cover' else col)
+            tables[col] = OneToOneTable(col, metadata)
+
+        for col in ('series', 'publisher', 'rating'):
+            tables[col] = ManyToOneTable(col, self.field_metadata[col].copy())
+
+        for col in ('authors', 'tags', 'formats', 'identifiers'):
+            cls = {
+                    'authors':AuthorsTable,
+                    'formats':FormatsTable,
+                    'identifiers':IdentifiersTable,
+                  }.get(col, ManyToManyTable)
+            tables[col] = cls(col, self.field_metadata[col].copy())
+
+        tables['size'] = SizeTable('size', self.field_metadata['size'].copy())
+
+        with self.conn: # Use a single transaction, to ensure nothing modifies
+                        # the db while we are reading
+            for table in tables.itervalues():
+                try:
+                    table.read()
+                except:
+                    prints('Failed to read table:', table.name)
+                    raise
+
+        return tables
+
+   # }}}
+

src/calibre/db/errors.py

@@ -0,0 +1,13 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+
+class NoSuchFormat(ValueError):
+    pass
+

src/calibre/db/tables.py

@@ -0,0 +1,143 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+from datetime import datetime
+
+from dateutil.tz import tzoffset
+
+from calibre.constants import plugins
+from calibre.utils.date import parse_date, local_tz
+from calibre.ebooks.metadata import author_to_author_sort
+
+_c_speedup = plugins['speedup'][0]
+
+def _c_convert_timestamp(val):
+    if not val:
+        return None
+    try:
+        ret = _c_speedup.parse_date(val.strip())
+    except:
+        ret = None
+    if ret is None:
+        return parse_date(val, as_utc=False)
+    year, month, day, hour, minutes, seconds, tzsecs = ret
+    return datetime(year, month, day, hour, minutes, seconds,
+                tzinfo=tzoffset(None, tzsecs)).astimezone(local_tz)
+
+class Table(object):
+
+    def __init__(self, name, metadata):
+        self.name, self.metadata = name, metadata
+
+        # self.adapt() maps values from the db to python objects
+        self.adapt = \
+            {
+                'datetime': _c_convert_timestamp,
+                'bool': bool
+            }.get(
+                metadata['datatype'], lambda x: x)
+        if name == 'authors':
+            # Legacy
+            self.adapt = lambda x: x.replace('|', ',') if x else None
+
+class OneToOneTable(Table):
+
+    def read(self, db):
+        self.book_col_map = {}
+        idcol = 'id' if self.metadata['table'] == 'books' else 'book'
+        for row in db.conn.execute('SELECT {0}, {1} FROM {2}'.format(idcol,
+            self.metadata['column'], self.metadata['table'])):
+            self.book_col_map[row[0]] = self.adapt(row[1])
+
+class SizeTable(OneToOneTable):
+
+    def read(self, db):
+        self.book_col_map = {}
+        for row in db.conn.execute(
+                'SELECT books.id, (SELECT MAX(uncompressed_size) FROM data '
+                'WHERE data.book=books.id) FROM books'):
+            self.book_col_map[row[0]] = self.adapt(row[1])
+
+class ManyToOneTable(Table):
+
+    def read(self, db):
+        self.id_map = {}
+        self.extra_map = {}
+        self.col_book_map = {}
+        self.book_col_map = {}
+        self.read_id_maps(db)
+        self.read_maps(db)
+
+    def read_id_maps(self, db):
+        for row in db.conn.execute('SELECT id, {0} FROM {1}'.format(
+            self.metadata['name'], self.metadata['table'])):
+            if row[1]:
+                self.id_map[row[0]] = self.adapt(row[1])
+
+    def read_maps(self, db):
+        for row in db.conn.execute(
+                'SELECT book, {0} FROM books_{1}_link'.format(
+                    self.metadata['link_column'], self.metadata['table'])):
+            if row[1] not in self.col_book_map:
+                self.col_book_map[row[1]] = []
+            self.col_book_map.append(row[0])
+            self.book_col_map[row[0]] = row[1]
+
+class ManyToManyTable(ManyToOneTable):
+
+    def read_maps(self, db):
+        for row in db.conn.execute(
+                'SELECT book, {0} FROM books_{1}_link'.format(
+                    self.metadata['link_column'], self.metadata['table'])):
+            if row[1] not in self.col_book_map:
+                self.col_book_map[row[1]] = []
+            self.col_book_map.append(row[0])
+            if row[0] not in self.book_col_map:
+                self.book_col_map[row[0]] = []
+            self.book_col_map[row[0]].append(row[1])
+
+class AuthorsTable(ManyToManyTable):
+
+    def read_id_maps(self, db):
+        for row in db.conn.execute(
+                'SELECT id, name, sort FROM authors'):
+            self.id_map[row[0]] = row[1]
+            self.extra_map[row[0]] = (row[2] if row[2] else
+                    author_to_author_sort(row[1]))
+
+class FormatsTable(ManyToManyTable):
+
+    def read_id_maps(self, db):
+        pass
+
+    def read_maps(self, db):
+        for row in db.conn.execute('SELECT book, format, name FROM data'):
+            if row[1] is not None:
+                if row[1] not in self.col_book_map:
+                    self.col_book_map[row[1]] = []
+                self.col_book_map.append(row[0])
+                if row[0] not in self.book_col_map:
+                    self.book_col_map[row[0]] = []
+                self.book_col_map[row[0]].append((row[1], row[2]))
+
+class IdentifiersTable(ManyToManyTable):
+
+    def read_id_maps(self, db):
+        pass
+
+    def read_maps(self, db):
+        for row in db.conn.execute('SELECT book, type, val FROM identifiers'):
+            if row[1] is not None and row[2] is not None:
+                if row[1] not in self.col_book_map:
+                    self.col_book_map[row[1]] = []
+                self.col_book_map.append(row[0])
+                if row[0] not in self.book_col_map:
+                    self.book_col_map[row[0]] = []
+                self.book_col_map[row[0]].append((row[1], row[2]))
+

src/calibre/devices/android/driver.py

@@ -45,8 +45,11 @@ class ANDROID(USBMS):
             0xfce : { 0xd12e : [0x0100]},
 
             # Google
-            0x18d1 : { 0x4e11 : [0x0100, 0x226, 0x227], 0x4e12: [0x0100, 0x226,
+            0x18d1 : {
-                0x227], 0x4e21: [0x0100, 0x226, 0x227], 0xb058: [0x0222]},
+                0x4e11 : [0x0100, 0x226, 0x227],
+                0x4e12: [0x0100, 0x226, 0x227],
+                0x4e21: [0x0100, 0x226, 0x227],
+                0xb058: [0x0222, 0x226, 0x227]},
 
             # Samsung
             0x04e8 : { 0x681d : [0x0222, 0x0223, 0x0224, 0x0400],
@@ -107,7 +110,7 @@ class ANDROID(USBMS):
     VENDOR_NAME      = ['HTC', 'MOTOROLA', 'GOOGLE_', 'ANDROID', 'ACER',
             'GT-I5700', 'SAMSUNG', 'DELL', 'LINUX', 'GOOGLE', 'ARCHOS',
             'TELECHIP', 'HUAWEI', 'T-MOBILE', 'SEMC', 'LGE', 'NVIDIA',
-            'GENERIC-', 'ZTE']
+            'GENERIC-', 'ZTE', 'MID']
     WINDOWS_MAIN_MEM = ['ANDROID_PHONE', 'A855', 'A853', 'INC.NEXUS_ONE',
             '__UMS_COMPOSITE', '_MB200', 'MASS_STORAGE', '_-_CARD', 'SGH-I897',
             'GT-I9000', 'FILE-STOR_GADGET', 'SGH-T959', 'SAMSUNG_ANDROID',
@@ -116,7 +119,7 @@ class ANDROID(USBMS):
             'IDEOS_TABLET', 'MYTOUCH_4G', 'UMS_COMPOSITE', 'SCH-I800_CARD',
             '7', 'A956', 'A955', 'A43', 'ANDROID_PLATFORM', 'TEGRA_2',
             'MB860', 'MULTI-CARD', 'MID7015A', 'INCREDIBLE', 'A7EB', 'STREAK',
-            'MB525']
+            'MB525', 'ANDROID2.3']
     WINDOWS_CARD_A_MEM = ['ANDROID_PHONE', 'GT-I9000_CARD', 'SGH-I897',
             'FILE-STOR_GADGET', 'SGH-T959', 'SAMSUNG_ANDROID', 'GT-P1000_CARD',
             'A70S', 'A101IT', '7', 'INCREDIBLE', 'A7EB', 'SGH-T849_CARD',

src/calibre/devices/apple/driver.py

@@ -5,7 +5,7 @@ __copyright__ = '2010, Gregory Riker'
 __docformat__ = 'restructuredtext en'
 
 
-import cStringIO, ctypes, datetime, os, re, shutil, subprocess, sys, tempfile, time
+import cStringIO, ctypes, datetime, os, re, sys, tempfile, time
 from calibre.constants import __appname__, __version__, DEBUG
 from calibre import fit_image, confirm_config_name
 from calibre.constants import isosx, iswindows
@@ -13,8 +13,7 @@ from calibre.devices.errors import OpenFeedback, UserFeedback
 from calibre.devices.usbms.deviceconfig import DeviceConfig
 from calibre.devices.interface import DevicePlugin
 from calibre.ebooks.BeautifulSoup import BeautifulSoup
-from calibre.ebooks.metadata import authors_to_string, MetaInformation, \
+from calibre.ebooks.metadata import authors_to_string, MetaInformation, title_sort
-    title_sort
 from calibre.ebooks.metadata.book.base import Metadata
 from calibre.ebooks.metadata.epub import set_metadata
 from calibre.library.server.utils import strftime
@@ -107,6 +106,7 @@ class DriverBase(DeviceConfig, DevicePlugin):
     # Needed for config_widget to work
     FORMATS = ['epub', 'pdf']
     USER_CAN_ADD_NEW_FORMATS = False
+    KEEP_TEMP_FILES_AFTER_UPLOAD = True
 
     # Hide the standard customization widgets
     SUPPORTS_SUB_DIRS = False
@@ -164,8 +164,12 @@ class ITUNES(DriverBase):
         settings()
         set_progress_reporter()
         upload_books()
-         _get_fpath()
+         _remove_existing_copy()
-          _update_epub_metadata()
+          _remove_from_device()
+          _remove_from_iTunes()
+         _add_new_copy()
+          _add_library_book()
+         _update_iTunes_metadata()
         add_books_to_metadata()
         use_plugboard_ext()
         set_plugboard()
@@ -182,7 +186,7 @@ class ITUNES(DriverBase):
     supported_platforms = ['osx','windows']
     author = 'GRiker'
     #: The version of this plugin as a 3-tuple (major, minor, revision)
-    version        = (1,0,0)
+    version        = (1,1,0)
 
     DISPLAY_DISABLE_DIALOG = "display_disable_apple_driver_dialog"
 
@@ -277,7 +281,6 @@ class ITUNES(DriverBase):
     description_prefix = "added by calibre"
     ejected = False
     iTunes= None
-    iTunes_media = None
     library_orphans = None
     log = Log()
     manual_sync_mode = False
@@ -413,11 +416,11 @@ class ITUNES(DriverBase):
                             this_book.datetime = parse_date(str(book.date_added())).timetuple()
                         except:
                             this_book.datetime = time.gmtime()
-                        this_book.db_id = None
                         this_book.device_collections = []
                         this_book.library_id = library_books[this_book.path] if this_book.path in library_books else None
                         this_book.size = book.size()
                         this_book.uuid = book.composer()
+                        this_book.cid = None
                         # Hack to discover if we're running in GUI environment
                         if self.report_progress is not None:
                             this_book.thumbnail = self._generate_thumbnail(this_book.path, book)
@@ -452,10 +455,10 @@ class ITUNES(DriverBase):
                                 this_book.datetime = parse_date(str(book.DateAdded)).timetuple()
                             except:
                                 this_book.datetime = time.gmtime()
-                            this_book.db_id = None
                             this_book.device_collections = []
                             this_book.library_id = library_books[this_book.path] if this_book.path in library_books else None
                             this_book.size = book.Size
+                            this_book.cid = None
                             # Hack to discover if we're running in GUI environment
                             if self.report_progress is not None:
                                 this_book.thumbnail = self._generate_thumbnail(this_book.path, book)
@@ -491,7 +494,7 @@ class ITUNES(DriverBase):
 
     def can_handle(self, device_info, debug=False):
         '''
-        Unix version of :method:`can_handle_windows`
+        OSX version of :method:`can_handle_windows`
 
         :param device_info: Is a tupe of (vid, pid, bcd, manufacturer, product,
         serial number)
@@ -1021,17 +1024,14 @@ class ITUNES(DriverBase):
 
         if DEBUG:
             self.log.info("ITUNES.upload_books()")
-            self._dump_files(files, header='upload_books()',indent=2)
-            self._dump_update_list(header='upload_books()',indent=2)
 
         if isosx:
-            for (i,file) in enumerate(files):
+            for (i,fpath) in enumerate(files):
-                format = file.rpartition('.')[2].lower()
+                format = fpath.rpartition('.')[2].lower()
                 path = self.path_template % (metadata[i].title,
                                              authors_to_string(metadata[i].authors),
                                              format)
                 self._remove_existing_copy(path, metadata[i])
-                fpath = self._get_fpath(file, metadata[i], format, update_md=True)
                 db_added, lb_added = self._add_new_copy(fpath, metadata[i])
                 thumb = self._cover_to_thumb(path, metadata[i], db_added, lb_added, format)
                 this_book = self._create_new_book(fpath, metadata[i], path, db_added, lb_added, thumb, format)
@@ -1062,13 +1062,12 @@ class ITUNES(DriverBase):
                 pythoncom.CoInitialize()
                 self.iTunes = win32com.client.Dispatch("iTunes.Application")
 
-                for (i,file) in enumerate(files):
+                for (i,fpath) in enumerate(files):
-                    format = file.rpartition('.')[2].lower()
+                    format = fpath.rpartition('.')[2].lower()
                     path = self.path_template % (metadata[i].title,
                                                  authors_to_string(metadata[i].authors),
                                                  format)
                     self._remove_existing_copy(path, metadata[i])
-                    fpath = self._get_fpath(file, metadata[i],format, update_md=True)
                     db_added, lb_added = self._add_new_copy(fpath, metadata[i])
 
                     if self.manual_sync_mode and not db_added:
@@ -1212,7 +1211,8 @@ class ITUNES(DriverBase):
         '''
         windows assumes pythoncom wrapper
         '''
-        self.log.info(" ITUNES._add_library_book()")
+        if DEBUG:
+            self.log.info(" ITUNES._add_library_book()")
         if isosx:
             added = self.iTunes.add(appscript.mactypes.File(file))
 
@@ -1275,24 +1275,59 @@ class ITUNES(DriverBase):
 
     def _add_new_copy(self, fpath, metadata):
         '''
+        fp = cached_book['lib_book'].location().path
+        fp = cached_book['lib_book'].Location
         '''
         if DEBUG:
             self.log.info(" ITUNES._add_new_copy()")
 
+        def _save_last_known_iTunes_storage(lb_added):
+            if isosx:
+                fp = lb_added.location().path
+                index = fp.rfind('/Books') + len('/Books')
+                last_known_iTunes_storage = fp[:index]
+            elif iswindows:
+                fp = lb_added.Location
+                index = fp.rfind('\Books') + len('\Books')
+                last_known_iTunes_storage = fp[:index]
+            dynamic['last_known_iTunes_storage'] = last_known_iTunes_storage
+            self.log.warning("  last_known_iTunes_storage: %s" % last_known_iTunes_storage)
+
         db_added = None
         lb_added = None
 
         if self.manual_sync_mode:
+            '''
+            This is the unsupported direct-connect mode.
+            In an attempt to avoid resetting the iTunes library Media folder, don't try to
+            add the book to iTunes if the last_known_iTunes_storage path is inaccessible.
+            This means that the path has to be set at least once, probably by using
+            'Connect to iTunes' and doing a transfer.
+            '''
+            self.log.warning("  unsupported direct connect mode")
             db_added = self._add_device_book(fpath, metadata)
-            if not getattr(fpath, 'deleted_after_upload', False):
+            last_known_iTunes_storage = dynamic.get('last_known_iTunes_storage', None)
-                lb_added = self._add_library_book(fpath, metadata)
+            if last_known_iTunes_storage is not None:
-                if lb_added:
+                if os.path.exists(last_known_iTunes_storage):
                     if DEBUG:
-                        self.log.info("  file added to Library|Books for iTunes<->iBooks tracking")
+                        self.log.warning("  iTunes storage online, adding to library")
+                    lb_added = self._add_library_book(fpath, metadata)
+                else:
+                    if DEBUG:
+                        self.log.warning("  iTunes storage not online, can't add to library")
+
+            if lb_added:
+                _save_last_known_iTunes_storage(lb_added)
+            if not lb_added and DEBUG:
+                self.log.warn("  failed to add '%s' to iTunes, iTunes Media folder inaccessible" % metadata.title)
         else:
             lb_added = self._add_library_book(fpath, metadata)
-            if DEBUG:
+            if lb_added:
-                self.log.info("  file added to Library|Books for pending sync")
+                _save_last_known_iTunes_storage(lb_added)
+            else:
+                raise UserFeedback("iTunes Media folder inaccessible",
+                                   details="Failed to add '%s' to iTunes" % metadata.title,
+                                   level=UserFeedback.WARN)
 
         return db_added, lb_added
 
@@ -1301,14 +1336,17 @@ class ITUNES(DriverBase):
         assumes pythoncom wrapper for db_added
         as of iTunes 9.2, iBooks 1.1, can't set artwork for PDF files via automation
         '''
-        self.log.info(" ITUNES._cover_to_thumb()")
+        if DEBUG:
+            self.log.info(" ITUNES._cover_to_thumb()")
 
         thumb = None
         if metadata.cover:
 
             if format == 'epub':
-                # Pre-shrink cover
+                '''
-                # self.MAX_COVER_WIDTH, self.MAX_COVER_HEIGHT
+                Pre-shrink cover
+                self.MAX_COVER_WIDTH, self.MAX_COVER_HEIGHT
+                '''
                 try:
                     img = PILImage.open(metadata.cover)
                     width = img.size[0]
@@ -1316,8 +1354,8 @@ class ITUNES(DriverBase):
                     scaled, nwidth, nheight = fit_image(width, height, self.MAX_COVER_WIDTH, self.MAX_COVER_HEIGHT)
                     if scaled:
                         if DEBUG:
-                            self.log.info("   '%s' scaled from %sx%s to %sx%s" %
+                            self.log.info("   cover scaled from %sx%s to %sx%s" %
-                                          (metadata.cover,width,height,nwidth,nheight))
+                                          (width,height,nwidth,nheight))
                         img = img.resize((nwidth, nheight), PILImage.ANTIALIAS)
                         cd = cStringIO.StringIO()
                         img.convert('RGB').save(cd, 'JPEG')
@@ -1336,9 +1374,11 @@ class ITUNES(DriverBase):
                     return thumb
 
                 if isosx:
-                    # The following commands generate an error, but the artwork does in fact
+                    '''
-                    # get sent to the device.  Seems like a bug in Apple's automation interface?
+                    The following commands generate an error, but the artwork does in fact
-                    # Could also be a problem with the integrity of the cover data?
+                    get sent to the device.  Seems like a bug in Apple's automation interface?
+                    Could also be a problem with the integrity of the cover data?
+                    '''
                     if lb_added:
                         try:
                             lb_added.artworks[1].data_.set(cover_data)
@@ -1361,9 +1401,8 @@ class ITUNES(DriverBase):
                             #ipython(user_ns=locals())
                             pass
 
-
                 elif iswindows:
-                    # Write the data to a real file for Windows iTunes
+                    ''' Write the data to a real file for Windows iTunes '''
                     tc = os.path.join(tempfile.gettempdir(), "cover.jpg")
                     with open(tc,'wb') as tmp_cover:
                         tmp_cover.write(cover_data)
@@ -1422,7 +1461,8 @@ class ITUNES(DriverBase):
 
         this_book = Book(metadata.title, authors_to_string(metadata.authors))
         this_book.datetime = time.gmtime()
-        this_book.db_id = None
+        #this_book.cid = metadata.id
+        this_book.cid = None
         this_book.device_collections = []
         this_book.format = format
         this_book.library_id = lb_added     # ??? GR
@@ -1430,7 +1470,6 @@ class ITUNES(DriverBase):
         this_book.thumbnail = thumb
         this_book.iTunes_id = lb_added      # ??? GR
         this_book.uuid = metadata.uuid
-
         if isosx:
             if lb_added:
                 this_book.size = self._get_device_book_size(fpath, lb_added.size())
@@ -1461,24 +1500,6 @@ class ITUNES(DriverBase):
 
         return this_book
 
-    def _delete_iTunesMetadata_plist(self,fpath):
-        '''
-        Delete the plist file from the file to force recache
-        '''
-        zf = ZipFile(fpath,'a')
-        fnames = zf.namelist()
-        pl_name = 'iTunesMetadata.plist'
-        try:
-            plist = [x for x in fnames if pl_name in x][0]
-        except:
-            plist = None
-        if plist:
-            if DEBUG:
-                self.log.info("  _delete_iTunesMetadata_plist():")
-                self.log.info("    deleting '%s'\n   from '%s'" % (pl_name,fpath))
-                zf.delete(pl_name)
-        zf.close()
-
     def _discover_manual_sync_mode(self, wait=0):
         '''
         Assumes pythoncom for windows
@@ -1663,18 +1684,6 @@ class ITUNES(DriverBase):
         zf.close()
         return (title, author, timestamp)
 
-    def _dump_files(self, files, header=None,indent=0):
-        if header:
-            msg = '\n%sfiles passed to %s:' % (' '*indent,header)
-            self.log.info(msg)
-            self.log.info( "%s%s" % (' '*indent,'-' * len(msg)))
-        for file in files:
-            if getattr(file, 'orig_file_path', None) is not None:
-                self.log.info(" %s%s" % (' '*indent,file.orig_file_path))
-            elif getattr(file, 'name', None) is not None:
-                self.log.info(" %s%s" % (' '*indent,file.name))
-        self.log.info()
-
     def _dump_hex(self, src, length=16):
         '''
         '''
@@ -1698,7 +1707,7 @@ class ITUNES(DriverBase):
         self.log.info()
 
     def _dump_update_list(self,header=None,indent=0):
-        if header:
+        if header and self.update_list:
             msg = '\n%sself.update_list %s' % (' '*indent,header)
             self.log.info(msg)
             self.log.info( "%s%s" % (' '*indent,'-' * len(msg)))
@@ -1717,7 +1726,6 @@ class ITUNES(DriverBase):
                  (' '*indent,
                   ub['title'],
                   ub['author']))
-        self.log.info()
 
     def _find_device_book(self, search):
         '''
@@ -2116,35 +2124,6 @@ class ITUNES(DriverBase):
                         self.log.error("  no iPad|Books playlist found")
                 return pl
 
-    def _get_fpath(self,file, metadata, format, update_md=False):
-        '''
-        If the database copy will be deleted after upload, we have to
-        use file (the PersistentTemporaryFile), which will be around until
-        calibre exits.
-        If we're using the database copy, delete the plist
-        '''
-        if DEBUG:
-            self.log.info(" ITUNES._get_fpath()")
-
-        fpath = file
-        if not getattr(fpath, 'deleted_after_upload', False):
-            if getattr(file, 'orig_file_path', None) is not None:
-                # Database copy
-                fpath = file.orig_file_path
-                self._delete_iTunesMetadata_plist(fpath)
-            elif getattr(file, 'name', None) is not None:
-                # PTF
-                fpath = file.name
-        else:
-            # Recipe - PTF
-            if DEBUG:
-                self.log.info("   file will be deleted after upload")
-
-        if format == 'epub' and update_md:
-            self._update_epub_metadata(fpath, metadata)
-
-        return fpath
-
     def _get_library_books(self):
         '''
         Populate a dict of paths from iTunes Library|Books
@@ -2348,6 +2327,7 @@ class ITUNES(DriverBase):
                 self.iTunes = appscript.app('iTunes')
                 self.initial_status = 'already running'
 
+            '''
             # Read the current storage path for iTunes media
             cmd = "defaults read com.apple.itunes NSNavLastRootDirectory"
             proc = subprocess.Popen( cmd, shell=True, cwd=os.curdir, stdout=subprocess.PIPE)
@@ -2358,12 +2338,13 @@ class ITUNES(DriverBase):
             else:
                 self.log.error("  could not confirm valid iTunes.media_dir from %s" % 'com.apple.itunes')
                 self.log.error("  media_dir: %s" % media_dir)
+            '''
+
             if DEBUG:
                 self.log.info("  %s %s" % (__appname__, __version__))
                 self.log.info("  [OSX %s - %s (%s), driver version %d.%d.%d]" %
                  (self.iTunes.name(), self.iTunes.version(), self.initial_status,
                   self.version[0],self.version[1],self.version[2]))
-                self.log.info("  iTunes_media: %s" % self.iTunes_media)
                 self.log.info("  calibre_library_path: %s" % self.calibre_library_path)
 
         if iswindows:
@@ -2403,6 +2384,7 @@ class ITUNES(DriverBase):
                              ' iTunes automation interface non-responsive, ' +
                              'recommend reinstalling iTunes')
 
+            '''
             # Read the current storage path for iTunes media from the XML file
             media_dir = ''
             string = None
@@ -2421,13 +2403,13 @@ class ITUNES(DriverBase):
                     self.log.error("  '%s' not found" % media_dir)
                 else:
                     self.log.error("  no media dir found: string: %s" % string)
+            '''
 
             if DEBUG:
                 self.log.info("  %s %s" % (__appname__, __version__))
                 self.log.info("  [Windows %s - %s (%s), driver version %d.%d.%d]" %
                  (self.iTunes.Windows[0].name, self.iTunes.Version, self.initial_status,
                   self.version[0],self.version[1],self.version[2]))
-                self.log.info("  iTunes_media: %s" % self.iTunes_media)
                 self.log.info("  calibre_library_path: %s" % self.calibre_library_path)
 
     def _purge_orphans(self,library_books, cached_books):
@@ -2477,13 +2459,14 @@ class ITUNES(DriverBase):
                    (self.cached_books[book]['title'] == metadata.title and \
                    self.cached_books[book]['author'] == authors_to_string(metadata.authors)):
                     self.update_list.append(self.cached_books[book])
-                    self._remove_from_device(self.cached_books[book])
+
                     if DEBUG:
                         self.log.info( "  deleting device book '%s'" % (metadata.title))
-                    if not getattr(file, 'deleted_after_upload', False):
+                    self._remove_from_device(self.cached_books[book])
-                        self._remove_from_iTunes(self.cached_books[book])
+
-                        if DEBUG:
+                    if DEBUG:
-                            self.log.info("  deleting library book '%s'" % metadata.title)
+                        self.log.info("  deleting library book '%s'" % metadata.title)
+                    self._remove_from_iTunes(self.cached_books[book])
                     break
             else:
                 if DEBUG:
@@ -2496,9 +2479,9 @@ class ITUNES(DriverBase):
                    (self.cached_books[book]['title'] == metadata.title and \
                     self.cached_books[book]['author'] == authors_to_string(metadata.authors)):
                     self.update_list.append(self.cached_books[book])
-                    self._remove_from_iTunes(self.cached_books[book])
                     if DEBUG:
                         self.log.info( "  deleting library book '%s'" %  metadata.title)
+                    self._remove_from_iTunes(self.cached_books[book])
                     break
             else:
                 if DEBUG:
@@ -2508,7 +2491,8 @@ class ITUNES(DriverBase):
         '''
         Windows assumes pythoncom wrapper
         '''
-        self.log.info(" ITUNES._remove_from_device()")
+        if DEBUG:
+            self.log.info(" ITUNES._remove_from_device()")
         if isosx:
             if DEBUG:
                 self.log.info("  deleting '%s' from iDevice" % cached_book['title'])
@@ -2529,96 +2513,105 @@ class ITUNES(DriverBase):
 
     def _remove_from_iTunes(self, cached_book):
         '''
-        iTunes does not delete books from storage when removing from database
+        iTunes does not delete books from storage when removing from database via automation
-        We only want to delete stored copies if the file is stored in iTunes
-        We don't want to delete files stored outside of iTunes.
-        Also confirm that storage_path does not point into calibre's storage.
         '''
         if DEBUG:
             self.log.info(" ITUNES._remove_from_iTunes():")
 
         if isosx:
+            ''' Manually remove the book from iTunes storage '''
             try:
-                storage_path = os.path.split(cached_book['lib_book'].location().path)
+                fp = cached_book['lib_book'].location().path
-                if cached_book['lib_book'].location().path.startswith(self.iTunes_media) and \
+                if DEBUG:
-                   not storage_path[0].startswith(prefs['library_path']):
+                    self.log.info("  processing %s" % fp)
-                    title_storage_path = storage_path[0]
+                if fp.startswith(prefs['library_path']):
-                    if DEBUG:
+                    self.log.info("  '%s' stored in calibre database, not removed" % cached_book['title'])
-                        self.log.info("  removing title_storage_path: %s" % title_storage_path)
-                    try:
-                        shutil.rmtree(title_storage_path)
-                    except:
-                        self.log.info("  '%s' not empty" % title_storage_path)
-
-                    # Clean up title/author directories
-                    author_storage_path = os.path.split(title_storage_path)[0]
-                    self.log.info("  author_storage_path: %s" % author_storage_path)
-                    author_files = os.listdir(author_storage_path)
-                    if '.DS_Store' in author_files:
-                        author_files.pop(author_files.index('.DS_Store'))
-                    if not author_files:
-                        shutil.rmtree(author_storage_path)
-                        if DEBUG:
-                            self.log.info("  removing empty author_storage_path")
-                    else:
-                        if DEBUG:
-                            self.log.info("  author_storage_path not empty (%d objects):" % len(author_files))
-                            self.log.info("  %s" % '\n'.join(author_files))
                 else:
-                    self.log.info("  '%s' (stored external to iTunes, no files deleted)" % cached_book['title'])
+                    if os.path.exists(fp):
+                        os.remove(fp)
+                        if DEBUG:
+                            self.log.info("   deleting from iTunes storage")
+                        author_storage_path = os.path.split(fp)[0]
+                        try:
+                            os.rmdir(author_storage_path)
+                            if DEBUG:
+                                self.log.info("   removing empty author directory")
+                        except:
+                            author_files = os.listdir(author_storage_path)
+                            if '.DS_Store' in author_files:
+                                author_files.pop(author_files.index('.DS_Store'))
+                            if not author_files:
+                                os.rmdir(author_storage_path)
+                                if DEBUG:
+                                    self.log.info("   removing empty author directory")
+                            '''
+                            else:
+                                if DEBUG:
+                                    self.log.info("   author_storage_path not empty:")
+                                    self.log.info("   %s" % '\n'.join(author_files))
+                            '''
+                    else:
+                        self.log.info("   '%s' does not exist at storage location" % cached_book['title'])
 
             except:
                 # We get here if there was an error with .location().path
                 if DEBUG:
-                    self.log.info("   '%s' not in iTunes storage" % cached_book['title'])
+                    self.log.info("   '%s' not found in iTunes storage" % cached_book['title'])
 
+            # Delete the book from the iTunes database
             try:
                 self.iTunes.delete(cached_book['lib_book'])
+                if DEBUG:
+                    self.log.info("   removing from iTunes database")
             except:
                 if DEBUG:
-                    self.log.info("   unable to remove '%s' from iTunes" % cached_book['title'])
+                    self.log.info("   unable to remove from iTunes database")
 
         elif iswindows:
             '''
             Assume we're wrapped in a pythoncom
             Windows stores the book under a common author directory, so we just delete the .epub
             '''
+            fp = None
             try:
                 book = cached_book['lib_book']
-                path = book.Location
+                fp = book.Location
             except:
                 book = self._find_library_book(cached_book)
                 if book:
-                    path = book.Location
+                    fp = book.Location
 
             if book:
-                if self.iTunes_media and path.startswith(self.iTunes_media) and \
+                if DEBUG:
-                   not path.startswith(prefs['library_path']):
+                    self.log.info("  processing %s" % fp)
-                    storage_path = os.path.split(path)
+                if fp.startswith(prefs['library_path']):
-                    if DEBUG:
+                    self.log.info("  '%s' stored in calibre database, not removed" % cached_book['title'])
-                        self.log.info("   removing '%s' at %s" %
-                            (cached_book['title'], path))
-                    try:
-                        os.remove(path)
-                    except:
-                        self.log.warning("   '%s' not in iTunes storage" % path)
-                    try:
-                        os.rmdir(storage_path[0])
-                        self.log.info("   removed folder '%s'" % storage_path[0])
-                    except:
-                        self.log.info("   folder '%s' not found or not empty" % storage_path[0])
-
-                    # Delete from iTunes database
                 else:
-                    self.log.info("   '%s' (stored external to iTunes, no files deleted)" % cached_book['title'])
+                    if os.path.exists(fp):
+                        os.remove(fp)
+                        if DEBUG:
+                            self.log.info("   deleting from iTunes storage")
+                        author_storage_path = os.path.split(fp)[0]
+                        try:
+                            os.rmdir(author_storage_path)
+                            if DEBUG:
+                                self.log.info("   removing empty author directory")
+                        except:
+                            pass
+                    else:
+                        self.log.info("   '%s' does not exist at storage location" % cached_book['title'])
             else:
                 if DEBUG:
-                    self.log.info("   '%s' not found in iTunes" % cached_book['title'])
+                    self.log.info("   '%s' not found in iTunes storage" % cached_book['title'])
+
+            # Delete the book from the iTunes database
             try:
                 book.Delete()
+                if DEBUG:
+                    self.log.info("   removing from iTunes database")
             except:
                 if DEBUG:
-                    self.log.info("   unable to remove '%s' from iTunes" % cached_book['title'])
+                    self.log.info("   unable to remove from iTunes database")
 
     def title_sorter(self, title):
         return re.sub('^\s*A\s+|^\s*The\s+|^\s*An\s+', '', title).rstrip()
@@ -2626,7 +2619,8 @@ class ITUNES(DriverBase):
     def _update_epub_metadata(self, fpath, metadata):
         '''
         '''
-        self.log.info(" ITUNES._update_epub_metadata()")
+        if DEBUG:
+            self.log.info(" ITUNES._update_epub_metadata()")
 
         # Fetch plugboard updates
         metadata_x = self._xform_metadata_via_plugboard(metadata, 'epub')
@@ -2797,7 +2791,7 @@ class ITUNES(DriverBase):
             if metadata_x.series and self.settings().extra_customization[self.USE_SERIES_AS_CATEGORY]:
                 if DEBUG:
                     self.log.info(" ITUNES._update_iTunes_metadata()")
-                    self.log.info("   using Series name as Genre")
+                    self.log.info("   using Series name '%s' as Genre" % metadata_x.series)
 
                 # Format the index as a sort key
                 index = metadata_x.series_index
@@ -2977,8 +2971,8 @@ class ITUNES(DriverBase):
             newmi = book.deepcopy_metadata()
             newmi.template_to_attribute(book, pb)
             if pb is not None and DEBUG:
-                self.log.info(" transforming %s using %s:" % (format, pb))
+                #self.log.info(" transforming %s using %s:" % (format, pb))
-                self.log.info("       title: %s %s" % (book.title, ">>> %s" %
+                self.log.info("       title: '%s' %s" % (book.title, ">>> '%s'" %
                                            newmi.title if book.title != newmi.title else ''))
                 self.log.info("  title_sort: %s %s" % (book.title_sort, ">>> %s" %
                                            newmi.title_sort if book.title_sort != newmi.title_sort else ''))
@@ -2993,7 +2987,8 @@ class ITUNES(DriverBase):
                 self.log.info("        tags: %s %s" % (book.tags, ">>> %s" %
                                            newmi.tags if book.tags != newmi.tags else ''))
             else:
-                self.log("  matching plugboard not found")
+                if DEBUG:
+                    self.log("  matching plugboard not found")
 
         else:
             newmi = book
@@ -3082,12 +3077,12 @@ class ITUNES_ASYNC(ITUNES):
                         this_book.datetime = parse_date(str(library_books[book].date_added())).timetuple()
                     except:
                         this_book.datetime = time.gmtime()
-                    this_book.db_id = None
                     this_book.device_collections = []
                     #this_book.library_id = library_books[this_book.path] if this_book.path in library_books else None
                     this_book.library_id = library_books[book]
                     this_book.size = library_books[book].size()
                     this_book.uuid = library_books[book].composer()
+                    this_book.cid = None
                     # Hack to discover if we're running in GUI environment
                     if self.report_progress is not None:
                         this_book.thumbnail = self._generate_thumbnail(this_book.path, library_books[book])
@@ -3123,11 +3118,11 @@ class ITUNES_ASYNC(ITUNES):
                             this_book.datetime = parse_date(str(library_books[book].DateAdded)).timetuple()
                         except:
                             this_book.datetime = time.gmtime()
-                        this_book.db_id = None
                         this_book.device_collections = []
                         this_book.library_id = library_books[book]
                         this_book.size = library_books[book].Size
                         this_book.uuid = library_books[book].Composer
+                        this_book.cid = None
                         # Hack to discover if we're running in GUI environment
                         if self.report_progress is not None:
                             this_book.thumbnail = self._generate_thumbnail(this_book.path, library_books[book])

src/calibre/devices/hanvon/driver.py

@@ -61,7 +61,7 @@ class LIBREAIR(N516):
 
     BCD = [0x399]
     VENDOR_NAME      = 'ALURATEK'
-    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = '_FILE-STOR_GADGET'
+    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = 'FILE-STOR_GADGET'
     EBOOK_DIR_MAIN = 'Books'
 
 class ALEX(N516):

src/calibre/devices/interface.py

@@ -327,12 +327,7 @@ class DevicePlugin(Plugin):
         free space on the device. The text of the FreeSpaceError must contain the
         word "card" if ``on_card`` is not None otherwise it must contain the word "memory".
 
-        :param files: A list of paths and/or file-like objects. If they are paths and
+        :param files: A list of paths
-                      the paths point to temporary files, they may have an additional
-                      attribute, original_file_path pointing to the originals. They may have
-                      another optional attribute, deleted_after_upload which if True means
-                      that the file pointed to by original_file_path will be deleted after
-                      being uploaded to the device.
         :param names: A list of file names that the books should have
                       once uploaded to the device. len(names) == len(files)
         :param metadata: If not None, it is a list of :class:`Metadata` objects.

src/calibre/devices/kobo/driver.py

@@ -100,7 +100,7 @@ class KOBO(USBMS):
         for idx,b in enumerate(bl):
             bl_cache[b.lpath] = idx
 
-        def update_booklist(prefix, path, title, authors, mime, date, ContentType, ImageID, readstatus, MimeType):
+        def update_booklist(prefix, path, title, authors, mime, date, ContentType, ImageID, readstatus, MimeType, expired, favouritesindex):
             changed = False
             try:
                 lpath = path.partition(self.normalize_path(prefix))[2]
@@ -111,12 +111,23 @@ class KOBO(USBMS):
 
                 playlist_map = {}
 
+                if lpath not in playlist_map:
+                    playlist_map[lpath] = []
+
                 if readstatus == 1:
-                    playlist_map[lpath]= "Im_Reading"
+                    playlist_map[lpath].append('Im_Reading')
                 elif readstatus == 2:
-                    playlist_map[lpath]= "Read"
+                    playlist_map[lpath].append('Read')
                 elif readstatus == 3:
-                    playlist_map[lpath]= "Closed"
+                    playlist_map[lpath].append('Closed')
+
+                # Related to a bug in the Kobo firmware that leaves an expired row for deleted books
+                # this shows an expired Collection so the user can decide to delete the book
+                if expired == 3:
+                    playlist_map[lpath].append('Expired')
+                # A SHORTLIST is supported on the touch but the data field is there on most earlier models
+                if favouritesindex == 1:
+                    playlist_map[lpath].append('Shortlist')
 
                 path = self.normalize_path(path)
                 # print "Normalized FileName: " + path
@@ -126,7 +137,13 @@ class KOBO(USBMS):
                     bl_cache[lpath] = None
                     if ImageID is not None:
                         imagename = self.normalize_path(self._main_prefix + '.kobo/images/' + ImageID + ' - NickelBookCover.parsed')
+                        if not os.path.exists(imagename): 
+                            # Try the Touch version if the image does not exist
+                            imagename = self.normalize_path(self._main_prefix + '.kobo/images/' + ImageID + ' - N3_LIBRARY_FULL.parsed')
+
                         #print "Image name Normalized: " + imagename
+                        if not os.path.exists(imagename):
+                            debug_print("Strange - The image name does not exist - title: ", title)
                         if imagename is not None:
                             bl[idx].thumbnail = ImageWrapper(imagename)
                     if (ContentType != '6' and MimeType != 'Shortcover'):
@@ -138,7 +155,7 @@ class KOBO(USBMS):
                              debug_print("    Strange:  The file: ", prefix, lpath, " does mot exist!")
                     if lpath in playlist_map and \
                         playlist_map[lpath] not in bl[idx].device_collections:
-                            bl[idx].device_collections.append(playlist_map[lpath])
+                            bl[idx].device_collections = playlist_map.get(lpath,[])
                 else:
                     if ContentType == '6' and MimeType == 'Shortcover':
                         book =  Book(prefix, lpath, title, authors, mime, date, ContentType, ImageID, size=1048576)
@@ -157,7 +174,7 @@ class KOBO(USBMS):
                             raise
 
                     # print 'Update booklist'
-                    book.device_collections = [playlist_map[lpath]] if lpath in playlist_map else []
+                    book.device_collections = playlist_map.get(lpath,[])# if lpath in playlist_map else []
 
                     if bl.add_book(book, replace_metadata=False):
                         changed = True
@@ -186,24 +203,30 @@ class KOBO(USBMS):
         result = cursor.fetchone()
         self.dbversion = result[0]
 
-        query= 'select Title, Attribution, DateCreated, ContentID, MimeType, ContentType, ' \
+        if self.dbversion >= 14:
-                'ImageID, ReadStatus from content where BookID is Null'
+            query= 'select Title, Attribution, DateCreated, ContentID, MimeType, ContentType, ' \
+                'ImageID, ReadStatus, ___ExpirationStatus, FavouritesIndex from content where BookID is Null'
+        else:
+            query= 'select Title, Attribution, DateCreated, ContentID, MimeType, ContentType, ' \
+                'ImageID, ReadStatus, ___ExpirationStatus, "-1" as FavouritesIndex from content where BookID is Null'
 
         cursor.execute (query)
 
         changed = False
         for i, row in enumerate(cursor):
         #  self.report_progress((i+1) / float(numrows), _('Getting list of books on device...'))
-
+            if row[3].startswith("file:///usr/local/Kobo/help/"):
+                # These are internal to the Kobo device and do not exist
+                continue
             path = self.path_from_contentid(row[3], row[5], row[4], oncard)
             mime = mime_type_ext(path_to_ext(path)) if path.find('kepub') == -1 else 'application/epub+zip'
             # debug_print("mime:", mime)
 
             if oncard != 'carda' and oncard != 'cardb' and not row[3].startswith("file:///mnt/sd/"):
-                changed = update_booklist(self._main_prefix, path, row[0], row[1], mime, row[2], row[5], row[6], row[7], row[4])
+                changed = update_booklist(self._main_prefix, path, row[0], row[1], mime, row[2], row[5], row[6], row[7], row[4], row[8], row[9])
                 # print "shortbook: " + path
             elif oncard == 'carda' and row[3].startswith("file:///mnt/sd/"):
-                changed = update_booklist(self._card_a_prefix, path, row[0], row[1], mime, row[2], row[5], row[6], row[7], row[4])
+                changed = update_booklist(self._card_a_prefix, path, row[0], row[1], mime, row[2], row[5], row[6], row[7], row[4], row[8], row[9])
 
             if changed:
                 need_sync = True
@@ -267,8 +290,12 @@ class KOBO(USBMS):
             cursor.execute('delete from content_keys where volumeid = ?', t)
 
         # Delete the chapters associated with the book next
-        t = (ContentID,ContentID,)
+        t = (ContentID,)
-        cursor.execute('delete from content where BookID  = ? or ContentID = ?', t)
+        # Kobo does not delete the Book row (ie the row where the BookID is Null)
+        # The next server sync should remove the row
+        cursor.execute('delete from content where BookID = ?', t)
+        cursor.execute('update content set ReadStatus=0, FirstTimeReading = \'true\', ___PercentRead=0, ___ExpirationStatus=3 ' \
+                'where BookID is Null and ContentID =?',t)
 
         connection.commit()
 
@@ -286,7 +313,7 @@ class KOBO(USBMS):
             path_prefix = '.kobo/images/'
             path = self._main_prefix + path_prefix + ImageID
 
-            file_endings = (' - iPhoneThumbnail.parsed', ' - bbMediumGridList.parsed', ' - NickelBookCover.parsed',)
+            file_endings = (' - iPhoneThumbnail.parsed', ' - bbMediumGridList.parsed', ' - NickelBookCover.parsed', ' - N3_LIBRARY_FULL.parsed', ' - N3_LIBRARY_GRID.parsed', ' - N3_LIBRARY_LIST.parsed', ' - N3_SOCIAL_CURRENTREAD.parsed',)
 
             for ending in file_endings:
                 fpath = path + ending
@@ -450,7 +477,10 @@ class KOBO(USBMS):
                 path = self._main_prefix + path + '.kobo'
                 # print "Path: " + path
             elif (ContentType == "6" or ContentType == "10") and MimeType == 'application/x-kobo-epub+zip':
-                path = self._main_prefix + '.kobo/kepub/' + path
+                if path.startswith("file:///mnt/onboard/"):
+                    path = self._main_prefix + path.replace("file:///mnt/onboard/", '')
+                else:
+                    path = self._main_prefix + '.kobo/kepub/' + path
                 # print "Internal: " + path
             else:
                 # if path.startswith("file:///mnt/onboard/"):
@@ -527,6 +557,7 @@ class KOBO(USBMS):
         if collections:
             # Process any collections that exist
             for category, books in collections.items():
+                # debug_print (category)
                 if category == 'Im_Reading':
                     # Reset Im_Reading list in the database
                     if oncard == 'carda':
@@ -545,7 +576,8 @@ class KOBO(USBMS):
 
                     for book in books:
 #                        debug_print('Title:', book.title, 'lpath:', book.path)
-                        book.device_collections = ['Im_Reading']
+                        if 'Im_Reading' not in book.device_collections:
+                            book.device_collections.append('Im_Reading') 
 
                         extension =  os.path.splitext(book.path)[1]
                         ContentType = self.get_content_type_from_extension(extension) if extension != '' else self.get_content_type_from_path(book.path)
@@ -588,7 +620,8 @@ class KOBO(USBMS):
 
                     for book in books:
 #                       debug_print('Title:', book.title, 'lpath:', book.path)
-                        book.device_collections = ['Read']
+                        if 'Read' not in book.device_collections:
+                            book.device_collections.append('Read') 
 
                         extension =  os.path.splitext(book.path)[1]
                         ContentType = self.get_content_type_from_extension(extension) if extension != '' else self.get_content_type_from_path(book.path)
@@ -624,7 +657,8 @@ class KOBO(USBMS):
 
                     for book in books:
 #                       debug_print('Title:', book.title, 'lpath:', book.path)
-                        book.device_collections = ['Closed']
+                        if 'Closed' not in book.device_collections:
+                            book.device_collections.append('Closed') 
 
                         extension =  os.path.splitext(book.path)[1]
                         ContentType = self.get_content_type_from_extension(extension) if extension != '' else self.get_content_type_from_path(book.path)
@@ -642,6 +676,44 @@ class KOBO(USBMS):
                         else:
                             connection.commit()
 #                            debug_print('Database: Commit set ReadStatus as Closed')
+                if category == 'Shortlist':
+                    # Reset FavouritesIndex list in the database
+                    if oncard == 'carda':
+                        query= 'update content set FavouritesIndex=-1 where BookID is Null and ContentID like \'file:///mnt/sd/%\''
+                    elif oncard != 'carda' and oncard != 'cardb':
+                        query= 'update content set FavouritesIndex=-1 where BookID is Null and ContentID not like \'file:///mnt/sd/%\''
+
+                    try:
+                        cursor.execute (query)
+                    except:
+                        debug_print('Database Exception:  Unable to reset Shortlist list')
+                        raise
+                    else:
+#                       debug_print('Commit: Reset Shortlist list')
+                        connection.commit()
+
+                    for book in books:
+#                        debug_print('Title:', book.title, 'lpath:', book.path)
+                        if 'Shortlist' not in book.device_collections:
+                            book.device_collections.append('Shortlist') 
+                        # debug_print ("Shortlist found for: ", book.title) 
+                        extension =  os.path.splitext(book.path)[1]
+                        ContentType = self.get_content_type_from_extension(extension) if extension != '' else self.get_content_type_from_path(book.path)
+
+                        ContentID = self.contentid_from_path(book.path, ContentType)
+#                        datelastread = time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime())
+
+                        t = (ContentID,)
+
+                        try:
+                            cursor.execute('update content set FavouritesIndex=1 where BookID is Null and ContentID = ?', t)
+                        except:
+                            debug_print('Database Exception:  Unable set book as Shortlist')
+                            raise
+                        else:
+                            connection.commit()
+#                            debug_print('Database: Commit set Shortlist as Shortlist')
+
         else: # No collections
             # Since no collections exist the ReadStatus needs to be reset to 0 (Unread)
             print "Reseting ReadStatus to 0"

src/calibre/devices/teclast/driver.py

@@ -19,8 +19,9 @@ class TECLAST_K3(USBMS):
     PRODUCT_ID  = [0x3203]
     BCD         = [0x0000, 0x0100]
 
-    VENDOR_NAME      = 'TECLAST'
+    VENDOR_NAME      = ['TECLAST', 'IMAGIN']
-    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = ['DIGITAL_PLAYER', 'TL-K5']
+    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = ['DIGITAL_PLAYER', 'TL-K5',
+            'EREADER']
 
     MAIN_MEMORY_VOLUME_LABEL  = 'K3 Main Memory'
     STORAGE_CARD_VOLUME_LABEL = 'K3 Storage Card'

src/calibre/ebooks/comic/input.py

@@ -351,7 +351,9 @@ class ComicInput(InputFormatPlugin):
         comics = []
         with CurrentDir(tdir):
             if not os.path.exists('comics.txt'):
-                raise ValueError('%s is not a valid comic collection'
+                raise ValueError((
+                    '%s is not a valid comic collection'
+                    ' no comics.txt was found in the file')
                         %stream.name)
             raw = open('comics.txt', 'rb').read()
             if raw.startswith(codecs.BOM_UTF16_BE):

src/calibre/ebooks/conversion/plumber.py

@@ -59,6 +59,8 @@ class CompositeProgressReporter(object):
                 (self.global_max - self.global_min)
         self.global_reporter(global_frac, msg)
 
+ARCHIVE_FMTS = ('zip', 'rar', 'oebzip')
+
 class Plumber(object):
     '''
     The `Plumber` manages the conversion pipeline. An UI should call the methods
@@ -594,7 +596,7 @@ OptionRecommendation(name='sr3_replace',
             raise ValueError('Input file must have an extension')
         input_fmt = input_fmt[1:].lower()
         self.archive_input_tdir = None
-        if input_fmt in ('zip', 'rar', 'oebzip'):
+        if input_fmt in ARCHIVE_FMTS:
             self.log('Processing archive...')
             tdir = PersistentTemporaryDirectory('_plumber_archive')
             self.input, input_fmt = self.unarchive(self.input, tdir)

src/calibre/ebooks/epub/fix/epubcheck.py

@@ -26,6 +26,10 @@ class Epubcheck(ePubFixer):
                 'significant changes to your epub, complain to the epubcheck '
                 'project.')
 
+    @property
+    def description(self):
+        return self.long_description
+
     @property
     def fix_name(self):
         return 'epubcheck'

src/calibre/ebooks/epub/fix/unmanifested.py

@@ -22,6 +22,10 @@ class Unmanifested(ePubFixer):
         'the manifest or delete them as specified by the '
         'delete unmanifested option.')
 
+    @property
+    def description(self):
+        return self.long_description
+
     @property
     def fix_name(self):
         return 'unmanifested'

src/calibre/ebooks/metadata/fb2.py

@@ -1,96 +1,235 @@
 #!/usr/bin/env python
 from __future__ import with_statement
 __license__   = 'GPL v3'
-__copyright__ = '2008, Anatoly Shipitsin <norguhtar at gmail.com>'
+__copyright__ = '2011, Roman Mukhin <ramses_ru at hotmail.com>, '\
-
+                '2008, Anatoly Shipitsin <norguhtar at gmail.com>'
 '''Read meta information from fb2 files'''
 
 import os
+import datetime
+from functools import partial
 from base64 import b64decode
 from lxml import etree
-from calibre.ebooks.metadata import MetaInformation
+from calibre.utils.date import parse_date
+from calibre import guess_all_extensions, prints, force_unicode
+from calibre.ebooks.metadata import MetaInformation, check_isbn
 from calibre.ebooks.chardet import xml_to_unicode
-from calibre import guess_all_extensions
 
-XLINK_NS     = 'http://www.w3.org/1999/xlink'
-def XLINK(name):
-    return '{%s}%s' % (XLINK_NS, name)
 
+NAMESPACES = {
+    'fb2'   :   'http://www.gribuser.ru/xml/fictionbook/2.0',
+    'xlink' :   'http://www.w3.org/1999/xlink'  }
+
+XPath = partial(etree.XPath, namespaces=NAMESPACES)
+tostring = partial(etree.tostring, method='text', encoding=unicode)
 
 def get_metadata(stream):
-    """ Return metadata as a L{MetaInfo} object """
+    """ Return fb2 metadata as a L{MetaInformation} object """
-    XPath = lambda x : etree.XPath(x,
+
-            namespaces={'fb2':'http://www.gribuser.ru/xml/fictionbook/2.0',
+    root = _get_fbroot(stream)
-                'xlink':XLINK_NS})
+
-    tostring = lambda x : etree.tostring(x, method='text',
+    book_title = _parse_book_title(root)
-            encoding=unicode).strip()
+    authors = _parse_authors(root)
+
+    # fallback for book_title
+    if book_title:
+        book_title = unicode(book_title)
+    else:
+        book_title = force_unicode(os.path.splitext(
+            os.path.basename(getattr(stream, 'name',
+                _('Unknown'))))[0])
+    mi = MetaInformation(book_title, authors)
+
+    try:
+        _parse_cover(root, mi)
+    except:
+        pass
+    try:
+        _parse_comments(root, mi)
+    except:
+        pass
+    try:
+        _parse_tags(root, mi)
+    except:
+        pass
+    try:
+        _parse_series(root, mi)
+    except:
+        pass
+    try:
+        _parse_isbn(root, mi)
+    except:
+        pass
+    try:
+        _parse_publisher(root, mi)
+    except:
+        pass
+    try:
+        _parse_pubdate(root, mi)
+    except:
+        pass
+    try:
+        _parse_timestamp(root, mi)
+    except:
+        pass
+
+    try:
+        _parse_language(root, mi)
+    except:
+        pass
+    #_parse_uuid(root, mi)
+
+    #if DEBUG:
+    #   prints(mi)
+    return mi
+
+def _parse_authors(root):
+    authors = []
+    # pick up authors but only from 1 secrion <title-info>; otherwise it is not consistent!
+    # Those are fallbacks: <src-title-info>, <document-info>
+    for author_sec in ['title-info', 'src-title-info', 'document-info']:
+        for au in XPath('//fb2:%s/fb2:author'%author_sec)(root):
+            author = _parse_author(au)
+            if author:
+                authors.append(author)
+        if author:
+            break
+
+    # if no author so far
+    if not authors:
+        authors.append(_('Unknown'))
+
+    return authors
+
+def _parse_author(elm_author):
+    """ Returns a list of display author and sortable author"""
+
+    xp_templ = 'normalize-space(fb2:%s/text())'
+
+    author = XPath(xp_templ % 'first-name')(elm_author)
+    lname = XPath(xp_templ % 'last-name')(elm_author)
+    mname = XPath(xp_templ % 'middle-name')(elm_author)
+
+    if mname:
+        author = (author + ' ' + mname).strip()
+    if lname:
+        author = (author + ' ' + lname).strip()
+
+    # fallback to nickname
+    if not author:
+        nname = XPath(xp_templ % 'nickname')(elm_author)
+        if nname:
+            author = nname
+
+    return author
+
+
+def _parse_book_title(root):
+    # <title-info> has a priority.   (actually <title-info>  is mandatory)
+    # other are backup solution (sequence is important. other then in fb2-doc)
+    xp_ti = '//fb2:title-info/fb2:book-title/text()'
+    xp_pi = '//fb2:publish-info/fb2:book-title/text()'
+    xp_si = '//fb2:src-title-info/fb2:book-title/text()'
+    book_title = XPath('normalize-space(%s|%s|%s)' % (xp_ti, xp_pi, xp_si))(root)
+
+    return book_title
+
+def _parse_cover(root, mi):
+    # pickup from <title-info>, if not exists it fallbacks to <src-title-info>
+    imgid = XPath('substring-after(string(//fb2:coverpage/fb2:image/@xlink:href), "#")')(root)
+    if imgid:
+        try:
+            _parse_cover_data(root, imgid, mi)
+        except:
+            pass
+
+def _parse_cover_data(root, imgid, mi):
+    elm_binary = XPath('//fb2:binary[@id="%s"]'%imgid)(root)
+    if elm_binary:
+        mimetype = elm_binary[0].get('content-type', 'image/jpeg')
+        mime_extensions = guess_all_extensions(mimetype)
+        if mime_extensions:
+            pic_data = elm_binary[0].text
+            if pic_data:
+                mi.cover_data = (mime_extensions[0][1:], b64decode(pic_data))
+        else:
+            prints("WARNING: Unsupported coverpage mime-type '%s' (id=#%s)" % (mimetype, imgid) )
+
+def _parse_tags(root, mi):
+    # pick up genre but only from 1 secrion <title-info>; otherwise it is not consistent!
+    # Those are fallbacks: <src-title-info>
+    for genre_sec in ['title-info', 'src-title-info']:
+        # -- i18n Translations-- ?
+        tags = XPath('//fb2:%s/fb2:genre/text()' % genre_sec)(root)
+        if tags:
+            mi.tags = list(map(unicode, tags))
+            break
+
+def _parse_series(root, mi):
+    #calibri supports only 1 series: use the 1-st one
+    # pick up sequence but only from 1 secrion in prefered order
+    # except <src-title-info>
+    xp_ti = '//fb2:title-info/fb2:sequence[1]'
+    xp_pi = '//fb2:publish-info/fb2:sequence[1]'
+
+    elms_sequence = XPath('%s|%s' % (xp_ti, xp_pi))(root)
+    if elms_sequence:
+        mi.series = elms_sequence[0].get('name', None)
+        if mi.series:
+            mi.series_index = elms_sequence[0].get('number', None)
+
+def _parse_isbn(root, mi):
+    # some people try to put several isbn in this field, but it is not allowed.  try to stick to the 1-st one in this case
+    isbn = XPath('normalize-space(//fb2:publish-info/fb2:isbn/text())')(root)
+    # some people try to put several isbn in this field, but it is not allowed.  try to stick to the 1-st one in this case
+    if ',' in isbn:
+        isbn = isbn[:isbn.index(',')]
+    if check_isbn(isbn):
+        mi.isbn = isbn
+
+def _parse_comments(root, mi):
+    # pick up annotation but only from 1 secrion <title-info>;  fallback: <src-title-info>
+    for annotation_sec in ['title-info', 'src-title-info']:
+        elms_annotation = XPath('//fb2:%s/fb2:annotation' % annotation_sec)(root)
+        if elms_annotation:
+            mi.comments = tostring(elms_annotation[0])
+            # TODO: tags i18n, xslt?
+            break
+
+def _parse_publisher(root, mi):
+    publisher = XPath('string(//fb2:publish-info/fb2:publisher/text())')(root)
+    if publisher:
+        mi.publisher = publisher
+
+def _parse_pubdate(root, mi):
+    year = XPath('number(//fb2:publish-info/fb2:year/text())')(root)
+    if float.is_integer(year):
+        # only year is available, so use 1-st of Jan
+        mi.pubdate = datetime.date(int(year), 1, 1)
+
+def _parse_timestamp(root, mi):
+    #<date value="1996-12-03">03.12.1996</date>
+    xp ='//fb2:document-info/fb2:date/@value|'\
+        '//fb2:document-info/fb2:date/text()'
+    docdate = XPath('string(%s)' % xp)(root)
+    if docdate:
+        mi.timestamp = parse_date(docdate)
+
+def _parse_language(root, mi):
+    language = XPath('string(//fb2:title-info/fb2:lang/text())')(root)
+    if language:
+        mi.language = language
+        mi.languages = [ language ]
+
+def _parse_uuid(root, mi):
+    uuid = XPath('normalize-space(//document-info/fb2:id/text())')(root)
+    if uuid:
+        mi.uuid = uuid
+
+def _get_fbroot(stream):
     parser = etree.XMLParser(recover=True, no_network=True)
     raw = stream.read()
-    raw = xml_to_unicode(raw, strip_encoding_pats=True,
+    raw = xml_to_unicode(raw, strip_encoding_pats=True)[0]
-            assume_utf8=True)[0]
     root = etree.fromstring(raw, parser=parser)
-    authors, author_sort = [], None
+    return root
-    for au in XPath('//fb2:author')(root):
-        fname = lname = author = None
-        fe = XPath('descendant::fb2:first-name')(au)
-        if fe:
-            fname = tostring(fe[0])
-            author = fname
-        le = XPath('descendant::fb2:last-name')(au)
-        if le:
-            lname = tostring(le[0])
-            if author:
-                author += ' '+lname
-            else:
-                author = lname
-        if author:
-            authors.append(author)
-        if len(authors) == 1 and author is not None:
-            if lname:
-                author_sort = lname
-            if fname:
-                if author_sort: author_sort += ', '+fname
-                else: author_sort = fname
-    title = os.path.splitext(os.path.basename(getattr(stream, 'name',
-        _('Unknown'))))[0]
-    for x in XPath('//fb2:book-title')(root):
-        title = tostring(x)
-        break
-    comments = ''
-    for x in XPath('//fb2:annotation')(root):
-        comments += tostring(x)
-    if not comments:
-        comments = None
-    tags = list(map(tostring, XPath('//fb2:genre')(root)))
 
-    cp = XPath('//fb2:coverpage')(root)
-    cdata = None
-    if cp:
-        cimage = XPath('descendant::fb2:image[@xlink:href]')(cp[0])
-        if cimage:
-            id = cimage[0].get(XLINK('href')).replace('#', '')
-            binary = XPath('//fb2:binary[@id="%s"]'%id)(root)
-            if binary:
-                mt = binary[0].get('content-type', 'image/jpeg')
-                exts = guess_all_extensions(mt)
-                if not exts:
-                    exts = ['.jpg']
-                cdata = (exts[0][1:], b64decode(tostring(binary[0])))
-
-    series = None
-    series_index = 1.0
-    for x in XPath('//fb2:sequence')(root):
-        series = x.get('name', None)
-        if series is not None:
-            series_index = x.get('number', 1.0)
-            break
-    mi = MetaInformation(title, authors)
-    mi.comments = comments
-    mi.author_sort = author_sort
-    if tags:
-        mi.tags = tags
-    mi.series = series
-    mi.series_index = series_index
-    if cdata:
-        mi.cover_data = cdata
-    return mi

src/calibre/ebooks/metadata/worker.py

@@ -15,7 +15,7 @@ from calibre.utils.ipc.server import Server
 from calibre.ptempfile import PersistentTemporaryDirectory, TemporaryDirectory
 from calibre import prints, isbytestring
 from calibre.constants import filesystem_encoding
-
+from calibre.db.errors import NoSuchFormat
 
 def debug(*args):
     prints(*args)
@@ -201,27 +201,35 @@ class SaveWorker(Thread):
         self.spare_server = spare_server
         self.start()
 
-    def collect_data(self, ids):
+    def collect_data(self, ids, tdir):
         from calibre.ebooks.metadata.opf2 import metadata_to_opf
         data = {}
         for i in set(ids):
-            mi = self.db.get_metadata(i, index_is_id=True, get_cover=True)
+            mi = self.db.get_metadata(i, index_is_id=True, get_cover=True,
+                    cover_as_data=True)
             opf = metadata_to_opf(mi)
             if isbytestring(opf):
                 opf = opf.decode('utf-8')
             cpath = None
-            if mi.cover:
+            if mi.cover_data and mi.cover_data[1]:
-                cpath = mi.cover
+                cpath = os.path.join(tdir, 'cover_%s.jpg'%i)
+                with lopen(cpath, 'wb') as f:
+                    f.write(mi.cover_data[1])
                 if isbytestring(cpath):
                     cpath = cpath.decode(filesystem_encoding)
             formats = {}
             if mi.formats:
                 for fmt in mi.formats:
-                    fpath = self.db.format_abspath(i, fmt, index_is_id=True)
+                    fpath = os.path.join(tdir, 'fmt_%s.%s'%(i, fmt.lower()))
-                    if fpath is not None:
+                    with lopen(fpath, 'wb') as f:
-                        if isbytestring(fpath):
+                        try:
-                            fpath = fpath.decode(filesystem_encoding)
+                            self.db.copy_format_to(i, fmt, f, index_is_id=True)
-                        formats[fmt.lower()] = fpath
+                        except NoSuchFormat:
+                            continue
+                        else:
+                            if isbytestring(fpath):
+                                fpath = fpath.decode(filesystem_encoding)
+                            formats[fmt.lower()] = fpath
             data[i] = [opf, cpath, formats, mi.last_modified.isoformat()]
         return data
 
@@ -244,7 +252,7 @@ class SaveWorker(Thread):
 
         for i, task in enumerate(tasks):
             tids = [x[-1] for x in task]
-            data = self.collect_data(tids)
+            data = self.collect_data(tids, tdir)
             dpath = os.path.join(tdir, '%d.json'%i)
             with open(dpath, 'wb') as f:
                 f.write(json.dumps(data, ensure_ascii=False).encode('utf-8'))

src/calibre/gui2/__init__.py

@@ -248,10 +248,11 @@ def error_dialog(parent, title, msg, det_msg='', show=False,
         return d.exec_()
     return d
 
-def question_dialog(parent, title, msg, det_msg='', show_copy_button=False):
+def question_dialog(parent, title, msg, det_msg='', show_copy_button=False,
+        default_yes=True):
     from calibre.gui2.dialogs.message_box import MessageBox
     d = MessageBox(MessageBox.QUESTION, title, msg, det_msg, parent=parent,
-                    show_copy_button=show_copy_button)
+                    show_copy_button=show_copy_button, default_yes=default_yes)
     return d.exec_() == d.Accepted
 
 def info_dialog(parent, title, msg, det_msg='', show=False,

src/calibre/gui2/actions/choose_library.py

@@ -252,11 +252,12 @@ class ChooseLibraryAction(InterfaceAction):
 
     def delete_requested(self, name, location):
         loc = location.replace('/', os.sep)
-        if not question_dialog(self.gui, _('Are you sure?'), '<p>'+
+        if not question_dialog(self.gui, _('Are you sure?'),
+                _('<h1 style="color:red">WARNING</h1>')+
                 _('<b style="color: red">All files</b> (not just ebooks) '
                     'from <br><br><b>%s</b><br><br> will be '
                 '<b>permanently deleted</b>. Are you sure?') % loc,
-                show_copy_button=False):
+                show_copy_button=False, default_yes=False):
             return
         exists = self.gui.library_view.model().db.exists_at(loc)
         if exists:

src/calibre/gui2/actions/copy_to_library.py

@@ -53,13 +53,18 @@ class Worker(Thread): # {{{
         from calibre.library.database2 import LibraryDatabase2
         newdb = LibraryDatabase2(self.loc)
         for i, x in enumerate(self.ids):
-            mi = self.db.get_metadata(x, index_is_id=True, get_cover=True)
+            mi = self.db.get_metadata(x, index_is_id=True, get_cover=True,
+                    cover_as_data=True)
             self.progress(i, mi.title)
             fmts = self.db.formats(x, index_is_id=True)
             if not fmts: fmts = []
             else: fmts = fmts.split(',')
-            paths = [self.db.format_abspath(x, fmt, index_is_id=True) for fmt in
+            paths = []
-                    fmts]
+            for fmt in fmts:
+                p = self.db.format(x, fmt, index_is_id=True,
+                    as_path=True)
+                if p:
+                    paths.append(p)
             added = False
             if prefs['add_formats_to_existing']:
                 identical_book_list = newdb.find_identical_books(mi)
@@ -75,6 +80,11 @@ class Worker(Thread): # {{{
                 if co is not None:
                     newdb.set_conversion_options(x, 'PIPE', co)
             self.processed.add(x)
+            for path in paths:
+                try:
+                    os.remove(path)
+                except:
+                    pass
 # }}}
 
 class CopyToLibraryAction(InterfaceAction):

src/calibre/gui2/actions/edit_metadata.py

@@ -17,6 +17,7 @@ from calibre.gui2.dialogs.tag_list_editor import TagListEditor
 from calibre.gui2.actions import InterfaceAction
 from calibre.ebooks.metadata import authors_to_string
 from calibre.utils.icu import sort_key
+from calibre.db.errors import NoSuchFormat
 
 class EditMetadataAction(InterfaceAction):
 
@@ -265,7 +266,7 @@ class EditMetadataAction(InterfaceAction):
                                 +'</p>', 'merge_too_many_books', self.gui):
                 return
 
-        dest_id, src_books, src_ids = self.books_to_merge(rows)
+        dest_id, src_ids = self.books_to_merge(rows)
         title = self.gui.library_view.model().db.title(dest_id, index_is_id=True)
         if safe_merge:
             if not confirm('<p>'+_(
@@ -277,7 +278,7 @@ class EditMetadataAction(InterfaceAction):
                 'Please confirm you want to proceed.')%title
             +'</p>', 'merge_books_safe', self.gui):
                 return
-            self.add_formats(dest_id, src_books)
+            self.add_formats(dest_id, self.formats_for_books(rows))
             self.merge_metadata(dest_id, src_ids)
         elif merge_only_formats:
             if not confirm('<p>'+_(
@@ -293,7 +294,7 @@ class EditMetadataAction(InterfaceAction):
                 'Are you <b>sure</b> you want to proceed?')%title
             +'</p>', 'merge_only_formats', self.gui):
                 return
-            self.add_formats(dest_id, src_books)
+            self.add_formats(dest_id, self.formats_for_books(rows))
             self.delete_books_after_merge(src_ids)
         else:
             if not confirm('<p>'+_(
@@ -308,7 +309,7 @@ class EditMetadataAction(InterfaceAction):
                 'Are you <b>sure</b> you want to proceed?')%title
             +'</p>', 'merge_books', self.gui):
                 return
-            self.add_formats(dest_id, src_books)
+            self.add_formats(dest_id, self.formats_for_books(rows))
             self.merge_metadata(dest_id, src_ids)
             self.delete_books_after_merge(src_ids)
             # leave the selection highlight on first selected book
@@ -329,8 +330,22 @@ class EditMetadataAction(InterfaceAction):
                     self.gui.library_view.model().db.add_format(dest_id, fmt, f, index_is_id=True,
                             notify=False, replace=replace)
 
+    def formats_for_books(self, rows):
+        m = self.gui.library_view.model()
+        ans = []
+        for id_ in map(m.id, rows):
+            dbfmts = m.db.formats(id_, index_is_id=True)
+            if dbfmts:
+                for fmt in dbfmts.split(','):
+                    try:
+                        path = m.db.format(id_, fmt, index_is_id=True,
+                                as_path=True)
+                        ans.append(path)
+                    except NoSuchFormat:
+                        continue
+        return ans
+
     def books_to_merge(self, rows):
-        src_books = []
         src_ids = []
         m = self.gui.library_view.model()
         for i, row in enumerate(rows):
@@ -339,22 +354,19 @@ class EditMetadataAction(InterfaceAction):
                 dest_id = id_
             else:
                 src_ids.append(id_)
-                dbfmts = m.db.formats(id_, index_is_id=True)
+        return [dest_id, src_ids]
-                if dbfmts:
-                    for fmt in dbfmts.split(','):
-                        src_books.append(m.db.format_abspath(id_, fmt,
-                            index_is_id=True))
-        return [dest_id, src_books, src_ids]
 
     def delete_books_after_merge(self, ids_to_delete):
         self.gui.library_view.model().delete_books_by_id(ids_to_delete)
 
     def merge_metadata(self, dest_id, src_ids):
         db = self.gui.library_view.model().db
-        dest_mi = db.get_metadata(dest_id, index_is_id=True, get_cover=True)
+        dest_mi = db.get_metadata(dest_id, index_is_id=True)
         orig_dest_comments = dest_mi.comments
+        dest_cover = db.cover(dest_id, index_is_id=True)
+        had_orig_cover = bool(dest_cover)
         for src_id in src_ids:
-            src_mi = db.get_metadata(src_id, index_is_id=True, get_cover=True)
+            src_mi = db.get_metadata(src_id, index_is_id=True)
             if src_mi.comments and orig_dest_comments != src_mi.comments:
                 if not dest_mi.comments:
                     dest_mi.comments = src_mi.comments
@@ -372,8 +384,10 @@ class EditMetadataAction(InterfaceAction):
                     dest_mi.tags = src_mi.tags
                 else:
                     dest_mi.tags.extend(src_mi.tags)
-            if src_mi.cover and not dest_mi.cover:
+            if not dest_cover:
-                dest_mi.cover = src_mi.cover
+                src_cover = db.cover(src_id, index_is_id=True)
+                if src_cover:
+                    dest_cover = src_cover
             if not dest_mi.publisher:
                 dest_mi.publisher = src_mi.publisher
             if not dest_mi.rating:
@@ -382,6 +396,8 @@ class EditMetadataAction(InterfaceAction):
                 dest_mi.series = src_mi.series
                 dest_mi.series_index = src_mi.series_index
         db.set_metadata(dest_id, dest_mi, ignore_errors=False)
+        if not had_orig_cover and dest_cover:
+            db.set_cover(dest_id, dest_cover)
 
         for key in db.field_metadata: #loop thru all defined fields
           if db.field_metadata[key]['is_custom']:

src/calibre/gui2/actions/show_quickview.py

@@ -0,0 +1,40 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+
+__license__   = 'GPL v3'
+__copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+
+from calibre.gui2.actions import InterfaceAction
+from calibre.gui2.dialogs.quickview import Quickview
+from calibre.gui2 import error_dialog
+
+class ShowQuickviewAction(InterfaceAction):
+
+    name = 'Show quickview'
+    action_spec = (_('Show quickview'), 'search.png', None, _('Q'))
+    dont_add_to = frozenset(['menubar-device', 'toolbar-device', 'context-menu-device'])
+    action_type = 'current'
+
+    current_instance = None
+
+    def genesis(self):
+        self.qaction.triggered.connect(self.show_quickview)
+
+    def show_quickview(self, *args):
+        if self.current_instance:
+            if not self.current_instance.is_closed:
+                return
+            self.current_instance = None
+        if self.gui.current_view() is not self.gui.library_view:
+            error_dialog(self.gui, _('No quickview available'),
+                _('Quickview is not available for books '
+                  'on the device.')).exec_()
+            return
+        index = self.gui.library_view.currentIndex()
+        if index.isValid():
+            self.current_instance = \
+                Quickview(self.gui, self.gui.library_view, index)
+            self.current_instance.show()
+

src/calibre/gui2/actions/tweak_epub.py

@@ -5,6 +5,8 @@ __license__   = 'GPL v3'
 __copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
+import os
+
 from calibre.gui2 import error_dialog
 from calibre.gui2.actions import InterfaceAction
 from calibre.gui2.dialogs.tweak_epub import TweakEpub
@@ -30,8 +32,8 @@ class TweakEpubAction(InterfaceAction):
         # Confirm 'EPUB' in formats
         book_id = self.gui.library_view.model().id(row)
         try:
-            path_to_epub = self.gui.library_view.model().db.format_abspath(
+            path_to_epub = self.gui.library_view.model().db.format(
-                    book_id, 'EPUB', index_is_id=True)
+                    book_id, 'EPUB', index_is_id=True, as_path=True)
         except:
             path_to_epub = None
 
@@ -45,6 +47,7 @@ class TweakEpubAction(InterfaceAction):
         if dlg.exec_() == dlg.Accepted:
             self.update_db(book_id, dlg._output)
         dlg.cleanup()
+        os.remove(path_to_epub)
 
     def update_db(self, book_id, rebuilt):
         '''

src/calibre/gui2/add.py

@@ -445,12 +445,14 @@ class Saver(QObject): # {{{
         self.pd.setModal(True)
         self.pd.show()
         self.pd.set_min(0)
+        self.pd.set_msg(_('Collecting data, please wait...'))
         self._parent = parent
         self.callback = callback
         self.callback_called = False
         self.rq = Queue()
         self.ids = [x for x in map(db.id, [r.row() for r in rows]) if x is not None]
-        self.pd.set_max(len(self.ids))
+        self.pd_max = len(self.ids)
+        self.pd.set_max(0)
         self.pd.value = 0
         self.failures = set([])
 
@@ -509,6 +511,8 @@ class Saver(QObject): # {{{
             id, title, ok, tb = self.rq.get_nowait()
         except Empty:
             return
+        if self.pd.max != self.pd_max:
+            self.pd.max = self.pd_max
         self.pd.value += 1
         self.ids.remove(id)
         if not isinstance(title, unicode):

src/calibre/gui2/convert/regex_builder.py

@@ -4,7 +4,7 @@ __license__ = 'GPL 3'
 __copyright__ = '2009, John Schember <john@nachtimwald.com>'
 __docformat__ = 'restructuredtext en'
 
-import re
+import re, os
 
 from PyQt4.QtCore import SIGNAL, Qt, pyqtSignal
 from PyQt4.QtGui import QDialog, QWidget, QDialogButtonBox, \
@@ -134,7 +134,17 @@ class RegexBuilder(QDialog, Ui_RegexBuilder):
                          _('Cannot build regex using the GUI builder without a book.'),
                          show=True)
             return False
-        self.open_book(db.format_abspath(book_id, format, index_is_id=True))
+        fpath = db.format(book_id, format, index_is_id=True,
+            as_path=True)
+        try:
+            self.open_book(fpath)
+        finally:
+            try:
+                os.remove(fpath)
+            except:
+                # Fails on windows if the input plugin for this format keeps the file open
+                # Happens for LIT files
+                pass
         return True
 
     def open_book(self, pathtoebook):
@@ -143,7 +153,8 @@ class RegexBuilder(QDialog, Ui_RegexBuilder):
         text = [u'']
         preprocessor = HTMLPreProcessor(None, False)
         for path in self.iterator.spine:
-            html = open(path, 'rb').read().decode('utf-8', 'replace')
+            with open(path, 'rb') as f:
+                html = f.read().decode('utf-8', 'replace')
             html = preprocessor(html, get_preprocess_html=True)
             text.append(html)
         self.preview.setPlainText('\n---\n'.join(text))

src/calibre/gui2/convert/single.py

@@ -11,8 +11,8 @@ import sys, cPickle, shutil, importlib
 from PyQt4.Qt import QString, SIGNAL, QAbstractListModel, Qt, QVariant, QFont
 
 from calibre.gui2 import ResizableDialog, NONE
-from calibre.ebooks.conversion.config import GuiRecommendations, save_specifics, \
+from calibre.ebooks.conversion.config import (GuiRecommendations, save_specifics,
-        load_specifics
+        load_specifics)
 from calibre.gui2.convert.single_ui import Ui_Dialog
 from calibre.gui2.convert.metadata import MetadataWidget
 from calibre.gui2.convert.look_and_feel import LookAndFeelWidget
@@ -24,7 +24,8 @@ from calibre.gui2.convert.toc import TOCWidget
 from calibre.gui2.convert.debug import DebugWidget
 
 
-from calibre.ebooks.conversion.plumber import Plumber, supported_input_formats
+from calibre.ebooks.conversion.plumber import (Plumber,
+        supported_input_formats, ARCHIVE_FMTS)
 from calibre.ebooks.conversion.config import delete_specifics
 from calibre.customize.ui import available_output_formats
 from calibre.customize.conversion import OptionRecommendation
@@ -106,7 +107,6 @@ class Config(ResizableDialog, Ui_Dialog):
     Configuration dialog for single book conversion. If accepted, has the
     following important attributes
 
-    input_path - Path to input file
     output_format - Output format (without a leading .)
     input_format  - Input format (without a leading .)
     opf_path - Path to OPF file with user specified metadata
@@ -156,13 +156,13 @@ class Config(ResizableDialog, Ui_Dialog):
         oidx = self.groups.currentIndex().row()
         input_format = self.input_format
         output_format = self.output_format
-        input_path = self.db.format_abspath(self.book_id, input_format,
-                index_is_id=True)
-        self.input_path = input_path
         output_path = 'dummy.'+output_format
         log = Log()
         log.outputs = []
-        self.plumber = Plumber(input_path, output_path, log)
+        input_file = 'dummy.'+input_format
+        if input_format in ARCHIVE_FMTS:
+            input_file = 'dummy.html'
+        self.plumber = Plumber(input_file, output_path, log)
 
         def widget_factory(cls):
             return cls(self.stack, self.plumber.get_option_by_name,

src/calibre/gui2/custom_column_widgets.py

@@ -25,7 +25,7 @@ class Base(object):
     def __init__(self, db, col_id, parent=None):
         self.db, self.col_id = db, col_id
         self.col_metadata = db.custom_column_num_map[col_id]
-        self.initial_val = None
+        self.initial_val = self.widgets = None
         self.setup_ui(parent)
 
     def initialize(self, book_id):
@@ -54,6 +54,9 @@ class Base(object):
     def normalize_ui_val(self, val):
         return val
 
+    def break_cycles(self):
+        self.db = self.widgets = self.initial_val = None
+
 class Bool(Base):
 
     def setup_ui(self, parent):

src/calibre/gui2/device.py

@@ -49,16 +49,26 @@ class DeviceJob(BaseJob): # {{{
         self._aborted = False
 
     def start_work(self):
+        if DEBUG:
+            prints('Job:', self.id, self.description, 'started',
+                safe_encode=True)
         self.start_time = time.time()
         self.job_manager.changed_queue.put(self)
 
     def job_done(self):
         self.duration = time.time() - self.start_time
         self.percent = 1
+        if DEBUG:
+            prints('DeviceJob:', self.id, self.description,
+                    'done, calling callback', safe_encode=True)
+
         try:
             self.callback_on_done(self)
         except:
             pass
+        if DEBUG:
+            prints('DeviceJob:', self.id, self.description,
+                    'callback returned', safe_encode=True)
         self.job_manager.changed_queue.put(self)
 
     def report_progress(self, percent, msg=''):
@@ -386,8 +396,17 @@ class DeviceManager(Thread): # {{{
                             if DEBUG:
                                 prints(traceback.format_exc(), file=sys.__stdout__)
 
-        return self.device.upload_books(files, names, on_card,
+        try:
-                                        metadata=metadata, end_session=False)
+            return self.device.upload_books(files, names, on_card,
+                    metadata=metadata, end_session=False)
+        finally:
+            if metadata:
+                for mi in metadata:
+                    try:
+                        if mi.cover:
+                            os.remove(mi.cover)
+                    except:
+                        pass
 
     def upload_books(self, done, files, names, on_card=None, titles=None,
                      metadata=None, plugboards=None, add_as_step_to_job=None):
@@ -1062,8 +1081,6 @@ class DeviceMixin(object): # {{{
                             'the device?'), autos):
                         self.iactions['Convert Books'].auto_convert_news(auto, format)
             files = [f for f in files if f is not None]
-            for f in files:
-                f.deleted_after_upload = del_on_upload
             if not files:
                 self.news_to_be_synced = set([])
                 return
@@ -1305,8 +1322,17 @@ class DeviceMixin(object): # {{{
             self.card_b_view if on_card == 'cardb' else self.memory_view
         view.model().resort(reset=False)
         view.model().research()
-        for f in files:
+        if files:
-            getattr(f, 'close', lambda : True)()
+            for f in files:
+                # Remove temporary files
+                try:
+                    rem = not getattr(
+                            self.device_manager.device,
+                            'KEEP_TEMP_FILES_AFTER_UPLOAD', False)
+                    if rem and 'caltmpfmt.' in f:
+                        os.remove(f)
+                except:
+                    pass
 
     def book_on_device(self, id, reset=False):
         '''

src/calibre/gui2/dialogs/choose_library.ui

@@ -41,7 +41,7 @@
    <item row="4" column="0" colspan="4">
     <widget class="QRadioButton" name="existing_library">
      <property name="text">
-      <string>Use &amp;existing library at the new location</string>
+      <string>Use the previously &amp;existing library at the new location</string>
      </property>
      <property name="checked">
       <bool>true</bool>

src/calibre/gui2/dialogs/message_box.py

@@ -23,7 +23,7 @@ class MessageBox(QDialog, Ui_Dialog): # {{{
                  det_msg='',
                  q_icon=None,
                  show_copy_button=True,
-                 parent=None):
+                 parent=None, default_yes=True):
         QDialog.__init__(self, parent)
         if q_icon is None:
             icon = {
@@ -65,7 +65,9 @@ class MessageBox(QDialog, Ui_Dialog): # {{{
         self.is_question = type_ == self.QUESTION
         if self.is_question:
             self.bb.setStandardButtons(self.bb.Yes|self.bb.No)
-            self.bb.button(self.bb.Yes).setDefault(True)
+            self.bb.button(self.bb.Yes if default_yes else self.bb.No
+                    ).setDefault(True)
+            self.default_yes = default_yes
         else:
             self.bb.button(self.bb.Ok).setDefault(True)
 
@@ -101,7 +103,8 @@ class MessageBox(QDialog, Ui_Dialog): # {{{
         ret = QDialog.showEvent(self, ev)
         if self.is_question:
             try:
-                self.bb.button(self.bb.Yes).setFocus(Qt.OtherFocusReason)
+                self.bb.button(self.bb.Yes if self.default_yes else self.bb.No
+                        ).setFocus(Qt.OtherFocusReason)
             except:
                 pass# Buttons were changed
         else:

src/calibre/gui2/dialogs/metadata_bulk.py

@@ -24,7 +24,7 @@ from calibre.utils.config import prefs, tweaks
 from calibre.utils.magick.draw import identify_data
 from calibre.utils.date import qt_to_dt
 
-def get_cover_data(path): # {{{
+def get_cover_data(stream, ext): # {{{
     from calibre.ebooks.metadata.meta import get_metadata
     old = prefs['read_file_metadata']
     if not old:
@@ -32,8 +32,8 @@ def get_cover_data(path): # {{{
     cdata = area = None
 
     try:
-        mi = get_metadata(open(path, 'rb'),
+        with stream:
-                os.path.splitext(path)[1][1:].lower())
+            mi = get_metadata(stream, ext)
         if mi.cover and os.access(mi.cover, os.R_OK):
             cdata = open(mi.cover).read()
         elif mi.cover_data[1] is not None:
@@ -186,9 +186,10 @@ class MyBlockingBusy(QDialog): # {{{
                 if fmts:
                     covers = []
                     for fmt in fmts.split(','):
-                        fmt = self.db.format_abspath(id, fmt, index_is_id=True)
+                        fmtf = self.db.format(id, fmt, index_is_id=True,
-                        if not fmt: continue
+                                as_file=True)
-                        cdata, area = get_cover_data(fmt)
+                        if fmtf is None: continue
+                        cdata, area = get_cover_data(fmtf, fmt)
                         if cdata:
                             covers.append((cdata, area))
                     covers.sort(key=lambda x: x[1])
@@ -361,7 +362,7 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog):
         fm = self.db.field_metadata
         for f in fm:
             if (f in ['author_sort'] or
-                    (fm[f]['datatype'] in ['text', 'series', 'enumeration']
+                    (fm[f]['datatype'] in ['text', 'series', 'enumeration', 'comments']
                      and fm[f].get('search_terms', None)
                      and f not in ['formats', 'ondevice']) or
                     (fm[f]['datatype'] in ['int', 'float', 'bool'] and

src/calibre/gui2/dialogs/progress.py

@@ -53,6 +53,13 @@ class ProgressDialog(QDialog, Ui_Dialog):
     def set_max(self, max):
         self.bar.setMaximum(max)
 
+    @dynamic_property
+    def max(self):
+        def fget(self): return self.bar.maximum()
+        def fset(self, val): self.bar.setMaximum(val)
+        return property(fget=fget, fset=fset)
+
+
     def _canceled(self, *args):
         self.canceled = True
         self.button_box.setDisabled(True)

src/calibre/gui2/dialogs/quickview.py

@@ -0,0 +1,254 @@
+#!/usr/bin/env  python
+__license__   = 'GPL v3'
+__copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
+__docformat__ = 'restructuredtext en'
+
+
+from PyQt4.Qt import (Qt, QDialog, QAbstractItemView, QTableWidgetItem,
+                      QListWidgetItem, QByteArray, QCoreApplication,
+                      QApplication)
+
+from calibre.customize.ui import find_plugin
+from calibre.gui2 import gprefs
+from calibre.gui2.dialogs.quickview_ui import Ui_Quickview
+from calibre.utils.icu import sort_key
+
+class TableItem(QTableWidgetItem):
+    '''
+    A QTableWidgetItem that sorts on a separate string and uses ICU rules
+    '''
+
+    def __init__(self, val, sort):
+        self.sort = sort
+        QTableWidgetItem.__init__(self, val)
+        self.setFlags(Qt.ItemIsEnabled|Qt.ItemIsSelectable)
+
+    def __ge__(self, other):
+        return sort_key(self.sort) >= sort_key(other.sort)
+
+    def __lt__(self, other):
+        return sort_key(self.sort) < sort_key(other.sort)
+
+class Quickview(QDialog, Ui_Quickview):
+
+    def __init__(self, gui, view, row):
+        QDialog.__init__(self, gui, flags=Qt.Window)
+        Ui_Quickview.__init__(self)
+        self.setupUi(self)
+        self.isClosed = False
+
+        self.books_table_column_widths = None
+        try:
+            self.books_table_column_widths = \
+                        gprefs.get('quickview_dialog_books_table_widths', None)
+            geom = gprefs.get('quickview_dialog_geometry', bytearray(''))
+            self.restoreGeometry(QByteArray(geom))
+        except:
+            pass
+
+        # Remove the help button from the window title bar
+        icon = self.windowIcon()
+        self.setWindowFlags(self.windowFlags()&(~Qt.WindowContextHelpButtonHint))
+        self.setWindowIcon(icon)
+
+        self.db = view.model().db
+        self.view = view
+        self.gui = gui
+        self.is_closed = False
+        self.current_book_id = None
+        self.current_key = None
+        self.last_search = None
+        self.current_column = None
+        self.current_item = None
+
+        self.items.setSelectionMode(QAbstractItemView.SingleSelection)
+        self.items.currentTextChanged.connect(self.item_selected)
+
+        # Set up the books table columns
+        self.books_table.setSelectionBehavior(QAbstractItemView.SelectRows)
+        self.books_table.setSelectionMode(QAbstractItemView.SingleSelection)
+        self.books_table.setColumnCount(3)
+        t = QTableWidgetItem(_('Title'))
+        self.books_table.setHorizontalHeaderItem(0, t)
+        t = QTableWidgetItem(_('Authors'))
+        self.books_table.setHorizontalHeaderItem(1, t)
+        t = QTableWidgetItem(_('Series'))
+        self.books_table.setHorizontalHeaderItem(2, t)
+        self.books_table_header_height = self.books_table.height()
+        self.books_table.cellDoubleClicked.connect(self.book_doubleclicked)
+        self.books_table.sortByColumn(0, Qt.AscendingOrder)
+
+        # get the standard table row height. Do this here because calling
+        # resizeRowsToContents can word wrap long cell contents, creating
+        # double-high rows
+        self.books_table.setRowCount(1)
+        self.books_table.setItem(0, 0, TableItem('A', ''))
+        self.books_table.resizeRowsToContents()
+        self.books_table_row_height = self.books_table.rowHeight(0)
+        self.books_table.setRowCount(0)
+
+        # Add the data
+        self.refresh(row)
+
+        self.view.clicked.connect(self.slave)
+        QCoreApplication.instance().aboutToQuit.connect(self.save_state)
+        self.search_button.clicked.connect(self.do_search)
+        view.model().new_bookdisplay_data.connect(self.book_was_changed)
+
+    # search button
+    def do_search(self):
+        if self.last_search is not None:
+            self.gui.search.set_search_string(self.last_search)
+
+    # Called when book information is changed in the library view. Make that
+    # book current. This means that prev and next in edit metadata will move
+    # the current book.
+    def book_was_changed(self, mi):
+        if self.is_closed or self.current_column is None:
+            return
+        self.refresh(self.view.model().index(self.db.row(mi.id), self.current_column))
+
+    # clicks on the items listWidget
+    def item_selected(self, txt):
+        self.fill_in_books_box(unicode(txt))
+
+    # Given a cell in the library view, display the information
+    def refresh(self, idx):
+        bv_row = idx.row()
+        self.current_column = idx.column()
+        key = self.view.model().column_map[self.current_column]
+        book_id = self.view.model().id(bv_row)
+
+        # Only show items for categories
+        if not self.db.field_metadata[key]['is_category']:
+            if self.current_key is None:
+                return
+            key = self.current_key
+        self.items_label.setText('{0} ({1})'.format(
+                                    self.db.field_metadata[key]['name'], key))
+
+        self.items.blockSignals(True)
+        self.items.clear()
+        self.books_table.setRowCount(0)
+
+        mi = self.db.get_metadata(book_id, index_is_id=True, get_user_categories=False)
+        vals = mi.get(key, None)
+
+        if vals:
+            if not isinstance(vals, list):
+                vals = [vals]
+            vals.sort(key=sort_key)
+
+            for v in vals:
+                a = QListWidgetItem(v)
+                self.items.addItem(a)
+            self.items.setCurrentRow(0)
+
+            self.current_book_id = book_id
+            self.current_key = key
+
+            self.fill_in_books_box(vals[0])
+        self.items.blockSignals(False)
+
+    def fill_in_books_box(self, selected_item):
+        self.current_item = selected_item
+        # Do a bit of fix-up on the items so that the search works.
+        if selected_item.startswith('.'):
+            sv = '.' + selected_item
+        else:
+            sv = selected_item
+        sv = sv.replace('"', r'\"')
+        self.last_search = self.current_key+':"=' + sv + '"'
+        books = self.db.search_getting_ids(self.last_search,
+                                           self.db.data.search_restriction)
+
+        self.books_table.setRowCount(len(books))
+        self.books_label.setText(_('Books with selected item: {0}').format(len(books)))
+
+        select_item = None
+        self.books_table.setSortingEnabled(False)
+        tt = ('<p>' +
+            _('Double-click on a book to change the selection in the library view. '
+               'Shift- or control-double-click to edit the metadata of a book')
+              + '</p>')
+        for row, b in enumerate(books):
+            mi = self.db.get_metadata(b, index_is_id=True, get_user_categories=False)
+            a = TableItem(mi.title, mi.title_sort)
+            a.setData(Qt.UserRole, b)
+            a.setToolTip(tt)
+            self.books_table.setItem(row, 0, a)
+            if b == self.current_book_id:
+                select_item = a
+            a = TableItem(' & '.join(mi.authors), mi.author_sort)
+            a.setToolTip(tt)
+            self.books_table.setItem(row, 1, a)
+            series = mi.format_field('series')[1]
+            if series is None:
+                series = ''
+            a = TableItem(series, series)
+            a.setToolTip(tt)
+            self.books_table.setItem(row, 2, a)
+            self.books_table.setRowHeight(row, self.books_table_row_height)
+
+        self.books_table.setSortingEnabled(True)
+        if select_item is not None:
+            self.books_table.setCurrentItem(select_item)
+            self.books_table.scrollToItem(select_item, QAbstractItemView.PositionAtCenter)
+
+    # Deal with sizing the table columns. Done here because the numbers are not
+    # correct until the first paint.
+    def resizeEvent(self, *args):
+        QDialog.resizeEvent(self, *args)
+        if self.books_table_column_widths is not None:
+            for c,w in enumerate(self.books_table_column_widths):
+                self.books_table.setColumnWidth(c, w)
+        else:
+            # the vertical scroll bar might not be rendered, so might not yet
+            # have a width. Assume 25. Not a problem because user-changed column
+            # widths will be remembered
+            w = self.books_table.width() - 25 - self.books_table.verticalHeader().width()
+            w /= self.books_table.columnCount()
+            for c in range(0, self.books_table.columnCount()):
+                self.books_table.setColumnWidth(c, w)
+        self.save_state()
+
+    def book_doubleclicked(self, row, column):
+        book_id = self.books_table.item(row, 0).data(Qt.UserRole).toInt()[0]
+        self.view.select_rows([book_id])
+        modifiers = int(QApplication.keyboardModifiers())
+        if modifiers in (Qt.CTRL, Qt.SHIFT):
+            em = find_plugin('Edit Metadata')
+            if em is not None:
+                em.actual_plugin_.edit_metadata(None)
+
+    # called when a book is clicked on the library view
+    def slave(self, current):
+        if self.is_closed:
+            return
+        self.refresh(current)
+        self.view.activateWindow()
+
+    def save_state(self):
+        if self.is_closed:
+            return
+        self.books_table_column_widths = []
+        for c in range(0, self.books_table.columnCount()):
+            self.books_table_column_widths.append(self.books_table.columnWidth(c))
+        gprefs['quickview_dialog_books_table_widths'] = self.books_table_column_widths
+        gprefs['quickview_dialog_geometry'] = bytearray(self.saveGeometry())
+
+    def close(self):
+        self.save_state()
+        # clean up to prevent memory leaks
+        self.db = self.view = self.gui = None
+        self.is_closed = True
+
+    # called by the window system
+    def closeEvent(self, *args):
+        self.close()
+        QDialog.closeEvent(self, *args)
+
+    # called by the close button
+    def reject(self):
+        self.close()
+        QDialog.reject(self)

src/calibre/gui2/dialogs/quickview.ui

@@ -0,0 +1,131 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<ui version="4.0">
+ <class>Quickview</class>
+ <widget class="QDialog" name="Quickview">
+  <property name="geometry">
+   <rect>
+    <x>0</x>
+    <y>0</y>
+    <width>768</width>
+    <height>342</height>
+   </rect>
+  </property>
+  <property name="sizePolicy">
+   <sizepolicy hsizetype="Expanding" vsizetype="MinimumExpanding">
+    <horstretch>0</horstretch>
+    <verstretch>0</verstretch>
+   </sizepolicy>
+  </property>
+  <property name="windowTitle">
+   <string>Quickview</string>
+  </property>
+  <layout class="QGridLayout">
+   <item row="0" column="0">
+    <widget class="QLabel" name="items_label">
+     <property name="text">
+      <string>Items</string>
+     </property>
+    </widget>
+   </item>
+   <item row="1" column="0">
+    <widget class="QListWidget" name="items">
+     <property name="sizePolicy">
+      <sizepolicy hsizetype="Expanding" vsizetype="Expanding" >
+       <horstretch>1</horstretch>
+       <verstretch>0</verstretch>
+      </sizepolicy>
+     </property>
+    </widget>
+   </item>
+   <item row="0" column="1">
+    <widget class="QLabel" name="books_label">
+    </widget>
+   </item>
+   <item row="1" column="1">
+    <widget class="QTableWidget" name="books_table">
+     <property name="sizePolicy">
+      <sizepolicy hsizetype="Expanding" vsizetype="Expanding" >
+       <horstretch>4</horstretch>
+       <verstretch>0</verstretch>
+      </sizepolicy>
+     </property>
+     <property name="columnCount">
+      <number>0</number>
+     </property>
+     <property name="rowCount">
+      <number>0</number>
+     </property>
+    </widget>
+   </item>
+   <item row="2" column="1">
+    <spacer>
+     <property name="orientation">
+      <enum>Qt::Vertical</enum>
+     </property>
+     <property name="sizeHint" stdset="0">
+      <size>
+       <width>0</width>
+       <height>0</height>
+      </size>
+     </property>
+    </spacer>
+   </item>
+   <item row="3" column="0" colspan="2">
+    <layout class="QHBoxLayout">
+     <item>
+      <widget class="QPushButton" name="search_button">
+       <property name="text">
+        <string>Search</string>
+       </property>
+       <property name="toolTip">
+        <string>Search in the library view for the selected item</string>
+       </property>
+      </widget>
+     </item>
+     <item>
+      <spacer>
+       <property name="orientation">
+        <enum>Qt::Horizontal</enum>
+       </property>
+       <property name="sizeHint" stdset="0">
+        <size>
+         <width>0</width>
+         <height>0</height>
+        </size>
+       </property>
+      </spacer>
+     </item>
+     <item>
+      <widget class="QDialogButtonBox" name="buttonBox">
+       <property name="standardButtons">
+        <set>QDialogButtonBox::Close</set>
+       </property>
+       <property name="centerButtons">
+        <bool>false</bool>
+       </property>
+      </widget>
+     </item>
+    </layout>
+   </item>
+  </layout>
+ </widget>
+ <resources/>
+ <connections>
+  <connection>
+   <sender>buttonBox</sender>
+   <signal>rejected()</signal>
+   <receiver>Quickview</receiver>
+   <slot>reject()</slot>
+   <hints>
+    <hint type="sourcelabel">
+     <x>297</x>
+     <y>217</y>
+    </hint>
+    <hint type="destinationlabel">
+     <x>286</x>
+     <y>234</y>
+    </hint>
+   </hints>
+  </connection>
+ </connections>
+</ui>

src/calibre/gui2/email.py

@@ -174,7 +174,8 @@ class EmailMixin(object): # {{{
         else:
             _auto_ids = []
 
-        full_metadata = self.library_view.model().metadata_for(ids)
+        full_metadata = self.library_view.model().metadata_for(ids,
+                get_cover=False)
 
         bad, remove_ids, jobnames = [], [], []
         texts, subjects, attachments, attachment_names = [], [], [], []

src/calibre/gui2/jobs.py

@@ -432,6 +432,10 @@ class JobsDialog(QDialog, Ui_JobsDialog):
             self.jobs_view.horizontalHeader().restoreState(QByteArray(state))
         except:
             pass
+        idx = self.jobs_view.model().index(0, 0)
+        if idx.isValid():
+            sm = self.jobs_view.selectionModel()
+            sm.select(idx, sm.ClearAndSelect|sm.Rows)
 
     def save_state(self):
         try:

src/calibre/gui2/library/models.py

@@ -5,8 +5,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
-import shutil, functools, re, os, traceback
+import functools, re, os, traceback
-from contextlib import closing
 from collections import defaultdict
 
 from PyQt4.Qt import (QAbstractTableModel, Qt, pyqtSignal, QIcon, QImage,
@@ -36,14 +35,6 @@ TIME_FMT = '%d %b %Y'
 ALIGNMENT_MAP = {'left': Qt.AlignLeft, 'right': Qt.AlignRight, 'center':
         Qt.AlignHCenter}
 
-class FormatPath(unicode):
-
-    def __new__(cls, path, orig_file_path):
-        ans = unicode.__new__(cls, path)
-        ans.orig_file_path = orig_file_path
-        ans.deleted_after_upload = False
-        return ans
-
 _default_image = None
 
 def default_image():
@@ -391,10 +382,14 @@ class BooksModel(QAbstractTableModel): # {{{
         data = self.current_changed(index, None, False)
         return data
 
-    def metadata_for(self, ids):
+    def metadata_for(self, ids, get_cover=True):
+        '''
+        WARNING: if get_cover=True temp files are created for mi.cover.
+        Remember to delete them once you are done with them.
+        '''
         ans = []
         for id in ids:
-            mi = self.db.get_metadata(id, index_is_id=True, get_cover=True)
+            mi = self.db.get_metadata(id, index_is_id=True, get_cover=get_cover)
             ans.append(mi)
         return ans
 
@@ -449,18 +444,14 @@ class BooksModel(QAbstractTableModel): # {{{
                     format = f
                     break
             if format is not None:
-                pt = PersistentTemporaryFile(suffix='.'+format)
+                pt = PersistentTemporaryFile(suffix='caltmpfmt.'+format)
-                with closing(self.db.format(id, format, index_is_id=True,
+                self.db.copy_format_to(id, format, pt, index_is_id=True)
-                    as_file=True)) as src:
-                    shutil.copyfileobj(src, pt)
-                    pt.flush()
-                    if getattr(src, 'name', None):
-                        pt.orig_file_path = os.path.abspath(src.name)
                 pt.seek(0)
                 if set_metadata:
                     try:
-                        _set_metadata(pt, self.db.get_metadata(id, get_cover=True, index_is_id=True),
+                        _set_metadata(pt, self.db.get_metadata(
-                                  format)
+                            id, get_cover=True, index_is_id=True,
+                            cover_as_data=True), format)
                     except:
                         traceback.print_exc()
                 pt.close()
@@ -468,9 +459,7 @@ class BooksModel(QAbstractTableModel): # {{{
                     if isbytestring(x):
                         x = x.decode(filesystem_encoding)
                     return x
-                name, op = map(to_uni, map(os.path.abspath, (pt.name,
+                ans.append(to_uni(os.path.abspath(pt.name)))
-                    pt.orig_file_path)))
-                ans.append(FormatPath(name, op))
             else:
                 need_auto.append(id)
                 if not exclude_auto:
@@ -499,13 +488,11 @@ class BooksModel(QAbstractTableModel): # {{{
                     break
             if format is not None:
                 pt = PersistentTemporaryFile(suffix='.'+format)
-                with closing(self.db.format(row, format, as_file=True)) as src:
+                self.db.copy_format_to(id, format, pt, index_is_id=True)
-                    shutil.copyfileobj(src, pt)
-                    pt.flush()
                 pt.seek(0)
                 if set_metadata:
-                    _set_metadata(pt, self.db.get_metadata(row, get_cover=True),
+                    _set_metadata(pt, self.db.get_metadata(row, get_cover=True,
-                                  format)
+                        cover_as_data=True), format)
                 pt.close() if paths else pt.seek(0)
                 ans.append(pt)
             else:

src/calibre/gui2/library/views.py

@@ -584,14 +584,17 @@ class BooksView(QTableView): # {{{
         m = self.model()
         db = m.db
         rows = self.selectionModel().selectedRows()
-        selected = map(m.id, rows)
+        selected = list(map(m.id, rows))
         ids = ' '.join(map(str, selected))
         md = QMimeData()
         md.setData('application/calibre+from_library', ids)
         fmt = prefs['output_format']
 
         def url_for_id(i):
-            ans = db.format_abspath(i, fmt, index_is_id=True)
+            try:
+                ans = db.format_path(i, fmt, index_is_id=True)
+            except:
+                ans = None
             if ans is None:
                 fmts = db.formats(i, index_is_id=True)
                 if fmts:
@@ -599,9 +602,10 @@ class BooksView(QTableView): # {{{
                 else:
                     fmts = []
                 for f in fmts:
-                    ans = db.format_abspath(i, f, index_is_id=True)
+                    try:
-                    if ans is not None:
+                        ans = db.format_path(i, f, index_is_id=True)
-                        break
+                    except:
+                        ans = None
             if ans is None:
                 ans = db.abspath(i, index_is_id=True)
             return QUrl.fromLocalFile(ans)

src/calibre/gui2/metadata/basic_widgets.py

@@ -21,9 +21,10 @@ from calibre.utils.config import tweaks, prefs
 from calibre.ebooks.metadata import (title_sort, authors_to_string,
         string_to_authors, check_isbn, authors_to_sort_string)
 from calibre.ebooks.metadata.meta import get_metadata
-from calibre.gui2 import (file_icon_provider, UNDEFINED_QDATE, UNDEFINED_DATE,
+from calibre.gui2 import (file_icon_provider, UNDEFINED_QDATE,
         choose_files, error_dialog, choose_images)
-from calibre.utils.date import local_tz, qt_to_dt
+from calibre.utils.date import (local_tz, qt_to_dt, as_local_time,
+        UNDEFINED_DATE)
 from calibre import strftime
 from calibre.ebooks import BOOK_EXTENSIONS
 from calibre.customize.ui import run_plugins_on_import
@@ -125,6 +126,9 @@ class TitleEdit(EnLineEdit):
 
         return property(fget=fget, fset=fset)
 
+    def break_cycles(self):
+        self.dialog = None
+
 class TitleSortEdit(TitleEdit):
 
     TITLE_ATTR = 'title_sort'
@@ -150,6 +154,7 @@ class TitleSortEdit(TitleEdit):
         self.title_edit.textChanged.connect(self.update_state)
         self.textChanged.connect(self.update_state)
 
+        self.autogen_button = autogen_button
         autogen_button.clicked.connect(self.auto_generate)
         self.update_state()
 
@@ -169,6 +174,20 @@ class TitleSortEdit(TitleEdit):
     def auto_generate(self, *args):
         self.current_val = title_sort(self.title_edit.current_val)
 
+    def break_cycles(self):
+        try:
+            self.title_edit.textChanged.disconnect()
+        except:
+            pass
+        try:
+            self.textChanged.disconnect()
+        except:
+            pass
+        try:
+            self.autogen_button.clicked.disconnect()
+        except:
+            pass
+
 # }}}
 
 # Authors {{{
@@ -185,6 +204,7 @@ class AuthorsEdit(MultiCompleteComboBox):
         self.setWhatsThis(self.TOOLTIP)
         self.setEditable(True)
         self.setSizeAdjustPolicy(self.AdjustToMinimumContentsLengthWithIcon)
+        self.manage_authors_signal = manage_authors
         manage_authors.triggered.connect(self.manage_authors)
 
     def manage_authors(self):
@@ -269,6 +289,13 @@ class AuthorsEdit(MultiCompleteComboBox):
 
         return property(fget=fget, fset=fset)
 
+    def break_cycles(self):
+        self.db = self.dialog = None
+        try:
+            self.manage_authors_signal.triggered.disconnect()
+        except:
+            pass
+
 class AuthorSortEdit(EnLineEdit):
 
     TOOLTIP = _('Specify how the author(s) of this book should be sorted. '
@@ -297,6 +324,10 @@ class AuthorSortEdit(EnLineEdit):
         self.authors_edit.editTextChanged.connect(self.update_state_and_val)
         self.textChanged.connect(self.update_state)
 
+        self.autogen_button = autogen_button
+        self.copy_a_to_as_action = copy_a_to_as_action
+        self.copy_as_to_a_action = copy_as_to_a_action
+
         autogen_button.clicked.connect(self.auto_generate)
         copy_a_to_as_action.triggered.connect(self.auto_generate)
         copy_as_to_a_action.triggered.connect(self.copy_to_authors)
@@ -368,6 +399,30 @@ class AuthorSortEdit(EnLineEdit):
         db.set_author_sort(id_, aus, notify=False, commit=False)
         return True
 
+    def break_cycles(self):
+        self.db = None
+        try:
+            self.authors_edit.editTextChanged.disconnect()
+        except:
+            pass
+        try:
+            self.textChanged.disconnect()
+        except:
+            pass
+        try:
+            self.autogen_button.clicked.disconnect()
+        except:
+            pass
+        try:
+            self.copy_a_to_as_action.triggered.disconnect()
+        except:
+            pass
+        try:
+            self.copy_as_to_a_action.triggered.disconnect()
+        except:
+            pass
+        self.authors_edit = None
+
 # }}}
 
 # Series {{{
@@ -427,6 +482,10 @@ class SeriesEdit(MultiCompleteComboBox):
                                             commit=True, allow_case_change=True)
         return True
 
+    def break_cycles(self):
+        self.dialog = None
+
+
 class SeriesIndexEdit(QDoubleSpinBox):
 
     TOOLTIP = ''
@@ -488,6 +547,20 @@ class SeriesIndexEdit(QDoubleSpinBox):
                 import traceback
                 traceback.print_exc()
 
+    def break_cycles(self):
+        try:
+            self.series_edit.currentIndexChanged.disconnect()
+        except:
+            pass
+        try:
+            self.series_edit.editTextChanged.disconnect()
+        except:
+            pass
+        try:
+            self.series_edit.lineEdit().editingFinished.disconnect()
+        except:
+            pass
+        self.db = self.series_edit = self.dialog = None
 
 # }}}
 
@@ -688,7 +761,8 @@ class FormatsManager(QWidget): # {{{
             else:
                 stream = open(fmt.path, 'r+b')
             try:
-                mi = get_metadata(stream, ext)
+                with stream:
+                    mi = get_metadata(stream, ext)
                 return mi, ext
             except:
                 error_dialog(self, _('Could not read metadata'),
@@ -698,6 +772,8 @@ class FormatsManager(QWidget): # {{{
             if old != prefs['read_file_metadata']:
                 prefs['read_file_metadata'] = old
 
+    def break_cycles(self):
+        self.dialog = None
 # }}}
 
 class Cover(ImageView): # {{{
@@ -859,6 +935,13 @@ class Cover(ImageView): # {{{
                 db.remove_cover(id_, notify=False, commit=False)
         return True
 
+    def break_cycles(self):
+        try:
+            self.cover_changed.disconnect()
+        except:
+            pass
+        self.dialog = self._cdata = self.current_val = self.original_val = None
+
 # }}}
 
 class CommentsEdit(Editor): # {{{
@@ -1210,6 +1293,7 @@ class DateEdit(QDateEdit): # {{{
         def fset(self, val):
             if val is None:
                 val = UNDEFINED_DATE
+            val = as_local_time(val)
             self.setDate(QDate(val.year, val.month, val.day))
         return property(fget=fget, fset=fset)
 

src/calibre/gui2/metadata/single.py

@@ -481,6 +481,13 @@ class MetadataSingleDialogBase(ResizableDialog):
             x = getattr(self, b, None)
             if x is not None:
                 disconnect(x.clicked)
+        for widget in self.basic_metadata_widgets:
+            bc = getattr(widget, 'break_cycles', None)
+            if bc is not None and callable(bc):
+                bc()
+        for widget in getattr(self, 'custom_metadata_widgets', []):
+            widget.break_cycles()
+
     # }}}
 
 class Splitter(QSplitter):

src/calibre/gui2/preferences/coloring.py

@@ -72,19 +72,27 @@ class ConditionEditor(QWidget): # {{{
         self.l = l = QGridLayout(self)
         self.setLayout(l)
 
-        self.l1 = l1 = QLabel(_('If the '))
+        texts = _('If the ___ column ___ values')
+        try:
+            one, two, three = texts.split('___')
+        except:
+            one, two, three = 'If the ', ' column ', ' value '
+
+        self.l1 = l1 = QLabel(one)
         l.addWidget(l1, 0, 0)
 
         self.column_box = QComboBox(self)
         l.addWidget(self.column_box, 0, 1)
 
-        self.l2 = l2 = QLabel(_(' column '))
+
+
+        self.l2 = l2 = QLabel(two)
         l.addWidget(l2, 0, 2)
 
         self.action_box = QComboBox(self)
         l.addWidget(self.action_box, 0, 3)
 
-        self.l3 = l3 = QLabel(_(' value '))
+        self.l3 = l3 = QLabel(three)
         l.addWidget(l3, 0, 4)
 
         self.value_box = QLineEdit(self)

src/calibre/gui2/preferences/misc.py

@@ -10,6 +10,7 @@ from calibre.gui2.preferences import ConfigWidgetBase, test_widget, Setting
 from calibre.gui2.preferences.misc_ui import Ui_Form
 from calibre.gui2 import error_dialog, config, open_local_file, info_dialog
 from calibre.constants import isosx
+from calibre import get_proxies
 
 class WorkersSetting(Setting):
 
@@ -33,6 +34,13 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         self.user_defined_device_button.clicked.connect(self.user_defined_device)
         self.button_osx_symlinks.clicked.connect(self.create_symlinks)
         self.button_osx_symlinks.setVisible(isosx)
+        proxies = get_proxies(debug=False)
+        txt = _('No proxies used')
+        if proxies:
+            lines = ['<br><code>%s: %s</code>'%(t, p) for t, p in
+                    proxies.iteritems()]
+            txt = _('<b>Using proxies:</b>') + ''.join(lines)
+        self.proxies.setText(txt)
 
     def debug_device_detection(self, *args):
         from calibre.gui2.preferences.device_debug import DebugDevice

src/calibre/gui2/preferences/misc.ui

@@ -118,7 +118,7 @@
      </property>
     </widget>
    </item>
-   <item row="20" column="0">
+   <item row="21" column="0">
     <spacer name="verticalSpacer_9">
      <property name="orientation">
       <enum>Qt::Vertical</enum>
@@ -131,6 +131,13 @@
      </property>
     </spacer>
    </item>
+   <item row="10" column="0" colspan="2">
+    <widget class="QLabel" name="proxies">
+     <property name="text">
+      <string/>
+     </property>
+    </widget>
+   </item>
   </layout>
  </widget>
  <resources/>

src/calibre/gui2/store/beam_ebooks_de_plugin.py

@@ -51,19 +51,19 @@ class BeamEBooksDEStore(BasicStoreConfig, StorePlugin):
                 if counter <= 0:
                     break
 
-                id = ''.join(data.xpath('./tr/td/div[@class="stil2"]/a/@href')).strip()
+                id = ''.join(data.xpath('./tr/td[1]/a/@href')).strip()
                 if not id:
                     continue
                 id = id[7:]
                 cover_url = ''.join(data.xpath('./tr/td[1]/a/img/@src'))
                 if cover_url:
                     cover_url = 'http://www.beam-ebooks.de' + cover_url
-                title = ''.join(data.xpath('./tr/td/div[@class="stil2"]/a/b/text()'))
+                temp = ''.join(data.xpath('./tr/td[1]/a/img/@alt'))
-                author = ' '.join(data.xpath('./tr/td/div[@class="stil2"]/'
+                colon = temp.find(':')
-                                             'child::b/text()'
+                if not temp.startswith('eBook') or colon < 0:
-                                             '|'
+                    continue
-                                             './tr/td/div[@class="stil2"]/'
+                author = temp[5:colon]
-                                             'child::strong/text()'))
+                title = temp[colon+1:]
                 price = ''.join(data.xpath('./tr/td[3]/text()'))
                 pdf = data.xpath(
                         'boolean(./tr/td[3]/a/img[contains(@alt, "PDF")]/@alt)')

src/calibre/gui2/store/config/chooser/models.py

@@ -133,7 +133,7 @@ class Matches(QAbstractItemModel):
                 return QVariant('<p>%s</p>' % result.description)
             elif col == 2:
                 if result.drm_free_only:
-                    return QVariant('<p>' + _('This store only distributes ebooks with DRM.') + '</p>')
+                    return QVariant('<p>' + _('This store only distributes ebooks without DRM.') + '</p>')
                 else:
                     return QVariant('<p>' + _('This store distributes ebooks with DRM. It may have some titles without DRM, but you will need to check on a per title basis.') + '</p>')
             elif col == 3:

src/calibre/gui2/store/declined.txt

@@ -1,9 +1,6 @@
 This is a list of stores that objected, declined
 or asked not to be included in the store integration.
 
-* Borders (http://www.borders.com/)
+* Borders (http://www.borders.com/).
-* Indigo (http://www.chapters.indigo.ca/)
+* Indigo (http://www.chapters.indigo.ca/).
 * Libraria Rizzoli (http://libreriarizzoli.corriere.it/).
-  No reply with two attempts over 2 weeks
-* WH Smith (http://www.whsmith.co.uk/)
-  Refused to permit signing up for the affiliate program

src/calibre/gui2/store/legimi_plugin.py

@@ -58,6 +58,8 @@ class LegimiStore(BasicStoreConfig, StorePlugin):
                 cover_url = ''.join(data.xpath('.//div[@class="item_cover_container"]/a/img/@src'))
                 title = ''.join(data.xpath('.//div[@class="item_entries"]/h2/a/text()'))
                 author = ''.join(data.xpath('.//div[@class="item_entries"]/span[1]/a/text()'))
+                author = re.sub(',','',author)
+                author = re.sub(';',',',author)
                 price = ''.join(data.xpath('.//div[@class="item_entries"]/span[3]/text()'))
                 price = re.sub(r'[^0-9,]*','',price) + ' zł'
 

src/calibre/gui2/store/libri_de_plugin.py

@@ -0,0 +1,90 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import (unicode_literals, division, absolute_import, print_function)
+
+__license__ = 'GPL 3'
+__copyright__ = '2011, John Schember <john@nachtimwald.com>'
+__docformat__ = 'restructuredtext en'
+
+import urllib2
+from contextlib import closing
+
+from lxml import html
+
+from PyQt4.Qt import QUrl
+
+from calibre import browser
+from calibre.gui2 import open_url
+from calibre.gui2.store import StorePlugin
+from calibre.gui2.store.basic_config import BasicStoreConfig
+from calibre.gui2.store.search_result import SearchResult
+from calibre.gui2.store.web_store_dialog import WebStoreDialog
+
+class LibreDEStore(BasicStoreConfig, StorePlugin):
+
+    def open(self, parent=None, detail_item=None, external=False):
+        url = 'http://ad.zanox.com/ppc/?18817073C15644254T'
+        url_details = ('http://ad.zanox.com/ppc/?18845780C1371495675T&ULP=[['
+                       'http://www.libri.de/shop/action/productDetails?artiId={0}]]')
+
+        if external or self.config.get('open_external', False):
+            if detail_item:
+                url = url_details.format(detail_item)
+            open_url(QUrl(url))
+        else:
+            detail_url = None
+            if detail_item:
+                detail_url = url_details.format(detail_item)
+            d = WebStoreDialog(self.gui, url, parent, detail_url)
+            d.setWindowTitle(self.name)
+            d.set_tags(self.config.get('tags', ''))
+            d.exec_()
+
+    def search(self, query, max_results=10, timeout=60):
+        url = ('http://www.libri.de/shop/action/quickSearch?facetNodeId=6'
+               '&mainsearchSubmit=Los!&searchString=' + urllib2.quote(query))
+        br = browser()
+
+        counter = max_results
+        with closing(br.open(url, timeout=timeout)) as f:
+            doc = html.fromstring(f.read())
+            for data in doc.xpath('//div[contains(@class, "item")]'):
+                if counter <= 0:
+                    break
+
+                details = data.xpath('./div[@class="beschreibungContainer"]')
+                if not details:
+                    continue
+                details = details[0]
+                id = ''.join(details.xpath('./div[@class="text"]/a/@name')).strip()
+                if not id:
+                    continue
+                cover_url = ''.join(details.xpath('./div[@class="bild"]/a/img/@src'))
+                title = ''.join(details.xpath('./div[@class="text"]/span[@class="titel"]/a/text()')).strip()
+                author = ''.join(details.xpath('./div[@class="text"]/span[@class="author"]/text()')).strip()
+                pdf = details.xpath(
+                        'boolean(.//span[@class="format" and contains(text(), "pdf")]/text())')
+                epub = details.xpath(
+                        'boolean(.//span[@class="format" and contains(text(), "epub")]/text())')
+                mobi = details.xpath(
+                        'boolean(.//span[@class="format" and contains(text(), "mobipocket")]/text())')
+                price = (''.join(data.xpath('.//span[@class="preis"]/text()'))).replace('*', '')
+                counter -= 1
+
+                s = SearchResult()
+                s.cover_url = cover_url
+                s.title = title.strip()
+                s.author = author.strip()
+                s.price = price
+                s.drm = SearchResult.DRM_UNKNOWN
+                s.detail_item = id
+                formats = []
+                if epub:
+                    formats.append('ePub')
+                if pdf:
+                    formats.append('PDF')
+                if mobi:
+                    formats.append('MOBI')
+                s.formats = ', '.join(formats)
+
+                yield s

src/calibre/gui2/store/open_books_plugin.py

@@ -0,0 +1,75 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import (unicode_literals, division, absolute_import, print_function)
+
+__license__ = 'GPL 3'
+__copyright__ = '2011, John Schember <john@nachtimwald.com>'
+__docformat__ = 'restructuredtext en'
+
+import urllib
+from contextlib import closing
+
+from lxml import html
+
+from PyQt4.Qt import QUrl
+
+from calibre import browser, url_slash_cleaner
+from calibre.gui2 import open_url
+from calibre.gui2.store import StorePlugin
+from calibre.gui2.store.basic_config import BasicStoreConfig
+from calibre.gui2.store.search_result import SearchResult
+from calibre.gui2.store.web_store_dialog import WebStoreDialog
+
+class OpenBooksStore(BasicStoreConfig, StorePlugin):
+
+    def open(self, parent=None, detail_item=None, external=False):
+        url = 'http://drmfree.calibre-ebook.com/'
+
+        if external or self.config.get('open_external', False):
+            open_url(QUrl(url_slash_cleaner(detail_item if detail_item else url)))
+        else:
+            d = WebStoreDialog(self.gui, self.url, parent, detail_item)
+            d.setWindowTitle(self.name)
+            d.set_tags(self.config.get('tags', ''))
+            d.exec_()
+
+    def search(self, query, max_results=10, timeout=60):
+        url = 'http://drmfree.calibre-ebook.com/search/?q=' + urllib.quote_plus(query)
+
+        br = browser()
+
+        counter = max_results
+        with closing(br.open(url, timeout=timeout)) as f:
+            doc = html.fromstring(f.read())
+            for data in doc.xpath('//ul[@id="object_list"]//li'):
+                if counter <= 0:
+                    break
+    
+                id = ''.join(data.xpath('.//div[@class="links"]/a[1]/@href'))
+                id = id.strip()
+                if not id:
+                    continue
+
+                cover_url = ''.join(data.xpath('.//div[@class="cover"]/img/@src'))
+
+                price = ''.join(data.xpath('.//div[@class="price"]/text()'))
+                a, b, price = price.partition('Price:')
+                price = price.strip()
+                if not price:
+                    continue
+
+                title = ''.join(data.xpath('.//div/strong/text()'))
+                author = ''.join(data.xpath('.//div[@class="author"]//text()'))
+                author = author.partition('by')[-1]
+
+                counter -= 1
+
+                s = SearchResult()
+                s.cover_url = cover_url
+                s.title = title.strip()
+                s.author = author.strip()
+                s.price = price.strip()
+                s.detail_item = id.strip()
+                s.drm = SearchResult.DRM_UNLOCKED
+
+                yield s

src/calibre/gui2/store/woblink_plugin.py

@@ -57,7 +57,7 @@ class WoblinkStore(BasicStoreConfig, StorePlugin):
 
                 cover_url = ''.join(data.xpath('.//td[@class="w10 va-t"]/a[1]/img/@src'))
                 title = ''.join(data.xpath('.//h3[@class="title"]/a[1]/text()'))
-                author = ''.join(data.xpath('.//p[@class="author"]/a[1]/text()'))
+                author = ', '.join(data.xpath('.//p[@class="author"]/a/text()'))
                 price = ''.join(data.xpath('.//div[@class="prices"]/p[1]/span/text()'))
                 price = re.sub('PLN', ' zł', price)
                 price = re.sub('\.', ',', price)

src/calibre/gui2/store/zixo_plugin.py

@@ -53,7 +53,7 @@ class ZixoStore(BasicStoreConfig, StorePlugin):
 
                 cover_url = ''.join(data.xpath('.//a[@class="productThumb"]/img/@src'))
                 title = ''.join(data.xpath('.//a[@class="title"]/text()'))
-                author = ''.join(data.xpath('.//div[@class="productDescription"]/span[1]/a/text()'))
+                author = ','.join(data.xpath('.//div[@class="productDescription"]/span[1]/a/text()'))
                 price = ''.join(data.xpath('.//div[@class="priceList"]/span/text()'))
                 price = re.sub('\.', ',', price)
 

src/calibre/gui2/tag_view.py

@@ -149,7 +149,8 @@ class TagsView(QTreeView): # {{{
                                 hidden_categories=self.hidden_categories,
                                 search_restriction=None,
                                 drag_drop_finished=self.drag_drop_finished,
-                                collapse_model=self.collapse_model)
+                                collapse_model=self.collapse_model,
+                                state_map={})
         self.pane_is_visible = True # because TagsModel.init did a recount
         self.sort_by = sort_by
         self.tag_match = tag_match
@@ -173,6 +174,7 @@ class TagsView(QTreeView): # {{{
             self.made_connections = True
         self.refresh_signal_processed = True
         db.add_listener(self.database_changed)
+        self.expanded.connect(self.item_expanded)
 
     def database_changed(self, event, ids):
         if self.refresh_signal_processed:
@@ -541,6 +543,10 @@ class TagsView(QTreeView): # {{{
         return self.isExpanded(idx)
 
     def recount(self, *args):
+        '''
+        Rebuild the category tree, expand any categories that were expanded,
+        reset the search states, and reselect the current node.
+        '''
         if self.disable_recounting or not self.pane_is_visible:
             return
         self.refresh_signal_processed = True
@@ -548,18 +554,23 @@ class TagsView(QTreeView): # {{{
         if not ci.isValid():
             ci = self.indexAt(QPoint(10, 10))
         path = self.model().path_for_index(ci) if self.is_visible(ci) else None
-        try:
+        expanded_categories, state_map = self.model().get_state()
-            if not self.model().refresh(): # categories changed!
+        self.set_new_model(state_map=state_map)
-                self.set_new_model()
+        for category in expanded_categories:
-                path = None
+            self.expand(self.model().index_for_category(category))
-        except: #Database connection could be closed if an integrity check is happening
-            pass
         self._model.show_item_at_path(path)
 
-    # If the number of user categories changed,  if custom columns have come or
+    def item_expanded(self, idx):
-    # gone, or if columns have been hidden or restored, we must rebuild the
+        '''
-    # model. Reason: it is much easier than reconstructing the browser tree.
+        Called by the expanded signal
-    def set_new_model(self, filter_categories_by=None):
+        '''
+        self.setCurrentIndex(idx)
+
+    def set_new_model(self, filter_categories_by=None, state_map={}):
+        '''
+        There are cases where we need to rebuild the category tree without
+        attempting to reposition the current node.
+        '''
         try:
             old = getattr(self, '_model', None)
             if old is not None:
@@ -569,7 +580,8 @@ class TagsView(QTreeView): # {{{
                                     search_restriction=self.search_restriction,
                                     drag_drop_finished=self.drag_drop_finished,
                                     filter_categories_by=filter_categories_by,
-                                    collapse_model=self.collapse_model)
+                                    collapse_model=self.collapse_model,
+                                    state_map=state_map)
             self.setModel(self._model)
         except:
             # The DB must be gone. Set the model to None and hope that someone
@@ -627,7 +639,8 @@ class TagTreeItem(object): # {{{
             except:
                 pass
         self.parent = self.icon_state_map = self.bold_font = self.tag = \
-                self.icon = self.children = None
+                      self.icon = self.children = self.tooltip = \
+                      self.py_name = self.id_set = self.category_key = None
 
     def __str__(self):
         if self.type == self.ROOT:
@@ -751,7 +764,8 @@ class TagsModel(QAbstractItemModel): # {{{
 
     def __init__(self, db, parent, hidden_categories=None,
             search_restriction=None, drag_drop_finished=None,
-            filter_categories_by=None, collapse_model='disable'):
+            filter_categories_by=None, collapse_model='disable',
+            state_map={}):
         QAbstractItemModel.__init__(self, parent)
 
         # must do this here because 'QPixmap: Must construct a QApplication
@@ -775,10 +789,10 @@ class TagsModel(QAbstractItemModel): # {{{
         self.filter_categories_by = filter_categories_by
         self.collapse_model = collapse_model
 
-        # get_node_tree cannot return None here, because row_map is empty. Note
+        # Note that _get_category_nodes can indirectly change the
-        # that get_node_tree can indirectly change the user_categories dict.
+        # user_categories dict.
 
-        data = self.get_node_tree(config['sort_tags_by'])
+        data = self._get_category_nodes(config['sort_tags_by'])
         gst = db.prefs.get('grouped_search_terms', {})
         self.root_item = TagTreeItem(icon_map=self.icon_state_map)
         self.category_nodes = []
@@ -843,7 +857,7 @@ class TagsModel(QAbstractItemModel): # {{{
                 category_node_map[key] = node
                 last_category_node = node
                 self.category_nodes.append(node)
-        self.refresh(data=data)
+        self._create_node_tree(data, state_map)
 
     def break_cycles(self):
         self.root_item.break_cycles()
@@ -1120,8 +1134,10 @@ class TagsModel(QAbstractItemModel): # {{{
     def set_search_restriction(self, s):
         self.search_restriction = s
 
-    def get_node_tree(self, sort):
+    def _get_category_nodes(self, sort):
-        old_row_map = self.row_map[:]
+        '''
+        Called by __init__. Do not directly call this method.
+        '''
         self.row_map = []
         self.categories = {}
 
@@ -1175,19 +1191,27 @@ class TagsModel(QAbstractItemModel): # {{{
             if category in data: # The search category can come and go
                 self.row_map.append(category)
                 self.categories[category] = tb_categories[category]['name']
-
-        if len(old_row_map) != 0 and len(old_row_map) != len(self.row_map):
-            # A category has been added or removed. We must force a rebuild of
-            # the model
-            return None
         return data
 
     def refresh(self, data=None):
+        '''
+        Here to trap usages of refresh in the old architecture. Can eventually
+        be removed.
+        '''
+        print 'TagsModel: refresh called!'
+        traceback.print_stack()
+        return False
+
+    def _create_node_tree(self, data, state_map):
+        '''
+        Called by __init__. Do not directly call this method.
+        '''
         sort_by = config['sort_tags_by']
+
         if data is None:
-            data = self.get_node_tree(sort_by) # get category data
+            print '_create_node_tree: no data!'
-        if data is None:
+            traceback.print_stack()
-            return False
+            return
 
         collapse = gprefs['tags_browser_collapse_at']
         collapse_model = self.collapse_model
@@ -1353,26 +1377,23 @@ class TagsModel(QAbstractItemModel): # {{{
         # }}}
 
         for category in self.category_nodes:
-            if len(category.children) > 0:
+            process_one_node(category, state_map.get(category.py_name, {}))
-                child_map = category.children
-                states = [c.tag.state for c in category.child_tags()]
-                names = [(c.tag.name, c.tag.category) for c in category.child_tags()]
-                state_map = dict(izip(names, states))
-                # temporary sub-categories (the partitioning ones) must follow
-                # the permanent sub-categories. This will happen naturally if
-                # the temp ones are added by process_node
-                ctags = [c for c in child_map if
-                         c.type == TagTreeItem.CATEGORY and not c.temporary]
-                start = len(ctags)
-                self.beginRemoveRows(self.createIndex(category.row(), 0, category),
-                                     start, len(child_map)-1)
-                category.children = ctags
-                self.endRemoveRows()
-            else:
-                state_map = {}
 
-            process_one_node(category, state_map)
+    def get_state(self):
-        return True
+        state_map = {}
+        expanded_categories = []
+        for row, category in enumerate(self.category_nodes):
+            if self.tags_view.isExpanded(self.index(row, 0, QModelIndex())):
+                expanded_categories.append(category.py_name)
+            states = [c.tag.state for c in category.child_tags()]
+            names = [(c.tag.name, c.tag.category) for c in category.child_tags()]
+            state_map[category.py_name] = dict(izip(names, states))
+        return expanded_categories, state_map
+
+    def index_for_category(self, name):
+        for row, category in enumerate(self.category_nodes):
+            if category.py_name == name:
+                return self.index(row, 0, QModelIndex())
 
     def columnCount(self, parent):
         return 1
@@ -1472,7 +1493,6 @@ class TagsModel(QAbstractItemModel): # {{{
             self.tags_view.tag_item_renamed.emit()
             item.tag.name = val
             self.rename_item_in_all_user_categories(name, key, val)
-            self.refresh() # Should work, because no categories can have disappeared
         self.show_item_at_path(path)
         return True
 
@@ -1654,9 +1674,9 @@ class TagsModel(QAbstractItemModel): # {{{
             if node.tag.state:
                 if node.category_key == "news":
                     if node_searches[node.tag.state] == 'true':
-                        ans.append('tags:=news')
+                        ans.append('tags:"=' + _('News') + '"')
                     else:
-                        ans.append('( not tags:=news )')
+                        ans.append('( not tags:"=' + _('News') + '")')
                 else:
                     ans.append('%s:%s'%(node.category_key, node_searches[node.tag.state]))
 
@@ -1785,19 +1805,22 @@ class TagsModel(QAbstractItemModel): # {{{
                         return v
         return None
 
-    def show_item_at_path(self, path, box=False):
+    def show_item_at_path(self, path, box=False,
+                          position=QTreeView.PositionAtCenter):
         '''
         Scroll the browser and open categories to show the item referenced by
         path. If possible, the item is placed in the center. If box=True, a
         box is drawn around the item.
         '''
         if path:
-            self.show_item_at_index(self.index_for_path(path), box)
+            self.show_item_at_index(self.index_for_path(path), box=box,
+                                    position=position)
 
-    def show_item_at_index(self, idx, box=False):
+    def show_item_at_index(self, idx, box=False,
+                           position=QTreeView.PositionAtCenter):
         if idx.isValid():
             self.tags_view.setCurrentIndex(idx)
-            self.tags_view.scrollTo(idx, QTreeView.PositionAtCenter)
+            self.tags_view.scrollTo(idx, position)
             if box:
                 tag_item = idx.internalPointer()
                 tag_item.boxed = True

src/calibre/gui2/tools.py

@@ -51,12 +51,15 @@ def convert_single_ebook(parent, db, book_ids, auto_conversion=False, # {{{
                 #    continue
 
                 mi = db.get_metadata(book_id, True)
-                in_file = db.format_abspath(book_id, d.input_format, True)
+                in_file = PersistentTemporaryFile('.'+d.input_format)
+                with in_file:
+                    db.copy_format_to(book_id, d.input_format, in_file,
+                            index_is_id=True)
 
                 out_file = PersistentTemporaryFile('.' + d.output_format)
                 out_file.write(d.output_format)
                 out_file.close()
-                temp_files = []
+                temp_files = [in_file]
 
                 try:
                     dtitle = unicode(mi.title)
@@ -74,7 +77,7 @@ def convert_single_ebook(parent, db, book_ids, auto_conversion=False, # {{{
                     recs.append(('cover', d.cover_file.name,
                         OptionRecommendation.HIGH))
                     temp_files.append(d.cover_file)
-                args = [in_file, out_file.name, recs]
+                args = [in_file.name, out_file.name, recs]
                 temp_files.append(out_file)
                 jobs.append(('gui_convert_override', args, desc, d.output_format.upper(), book_id, temp_files))
 
@@ -142,12 +145,15 @@ class QueueBulk(QProgressDialog):
         try:
             input_format = get_input_format_for_book(self.db, book_id, None)[0]
             mi, opf_file = create_opf_file(self.db, book_id)
-            in_file = self.db.format_abspath(book_id, input_format, True)
+            in_file = PersistentTemporaryFile('.'+input_format)
+            with in_file:
+                self.db.copy_format_to(book_id, input_format, in_file,
+                        index_is_id=True)
 
             out_file = PersistentTemporaryFile('.' + self.output_format)
             out_file.write(self.output_format)
             out_file.close()
-            temp_files = []
+            temp_files = [in_file]
 
             combined_recs = GuiRecommendations()
             default_recs = bulk_defaults_for_input_format(input_format)
@@ -183,7 +189,7 @@ class QueueBulk(QProgressDialog):
             self.setLabelText(_('Queueing ')+dtitle)
             desc = _('Convert book %d of %d (%s)') % (self.i, len(self.book_ids), dtitle)
 
-            args = [in_file, out_file.name, lrecs]
+            args = [in_file.name, out_file.name, lrecs]
             temp_files.append(out_file)
             self.jobs.append(('gui_convert_override', args, desc, self.output_format.upper(), book_id, temp_files))
 

src/calibre/gui2/update.py

@@ -74,7 +74,7 @@ class UpdateNotification(QDialog):
             'See the <a href="http://calibre-ebook.com/whats-new'
             '">new features</a>.') + '<p>'+_('Update <b>only</b> if one of the '
             'new features or bug fixes is important to you. '
-            'If the current version works well for you, do not update.'))%(
+            'If the current version works well for you, there is no need to update.'))%(
                 __appname__, calibre_version))
         self.label.setOpenExternalLinks(True)
         self.label.setWordWrap(True)

src/calibre/library/__init__.py

@@ -61,22 +61,4 @@ def generate_test_db(library_path, # {{{
     print 'Time per record:', t/float(num_of_records)
 # }}}
 
-def cover_load_timing(path=None):
-    from PyQt4.Qt import QApplication, QImage
-    import os, time
-    app = QApplication([])
-    app
-    d = db(path)
-    paths = [d.cover(i, index_is_id=True, as_path=True) for i in
-            d.data.iterallids()]
-    paths = [p for p in paths if (p and os.path.exists(p) and os.path.isfile(p))]
-
-    start = time.time()
-
-    for p in paths:
-        with open(p, 'rb') as f:
-            img = QImage()
-            img.loadFromData(f.read())
-
-    print 'Average load time:', (time.time() - start)/len(paths), 'seconds'
 

src/calibre/library/catalog.py

@@ -3,7 +3,7 @@
 __license__   = 'GPL v3'
 __copyright__ = '2010, Greg Riker'
 
-import codecs, datetime, htmlentitydefs, os, re, shutil, time, zlib
+import codecs, datetime, htmlentitydefs, os, re, shutil, zlib
 from collections import namedtuple
 from copy import deepcopy
 from xml.sax.saxutils import escape
@@ -25,7 +25,7 @@ from calibre.utils.html2text import html2text
 from calibre.utils.icu import capitalize
 from calibre.utils.logging import default_log as log
 from calibre.utils.magick.draw import thumbnail
-from calibre.utils.zipfile import ZipFile, ZipInfo
+from calibre.utils.zipfile import ZipFile
 
 FIELDS = ['all', 'title', 'title_sort', 'author_sort', 'authors', 'comments',
           'cover', 'formats','id', 'isbn', 'ondevice', 'pubdate', 'publisher',
@@ -4704,24 +4704,33 @@ Author '{0}':
             to be replaced.
             '''
 
+            def open_archive(mode='r'):
+                try:
+                    return ZipFile(self.__archive_path, mode=mode)
+                except:
+                    # Happens on windows if the file is opened by another
+                    # process
+                    pass
+
             # Generate crc for current cover
             #self.opts.log.info(" generateThumbnail():")
-            data = open(title['cover'], 'rb').read()
+            with open(title['cover'], 'rb') as f:
+                data = f.read()
             cover_crc = hex(zlib.crc32(data))
 
             # Test cache for uuid
-            with ZipFile(self.__archive_path, mode='r') as zfr:
+            zf = open_archive()
-                try:
+            if zf is not None:
-                    t_info = zfr.getinfo(title['uuid'])
+                with zf:
-                except:
+                    try:
-                    pass
+                        zf.getinfo(title['uuid']+cover_crc)
-                else:
+                    except:
-                    if t_info.comment == cover_crc:
+                        pass
+                    else:
                         # uuid found in cache with matching crc
-                        thumb_data = zfr.read(title['uuid'])
+                        thumb_data = zf.read(title['uuid']+cover_crc)
-                        zfr.extract(title['uuid'],image_dir)
+                        with open(os.path.join(image_dir, thumb_file), 'wb') as f:
-                        os.rename(os.path.join(image_dir,title['uuid']),
+                            f.write(thumb_data)
-                                    os.path.join(image_dir,thumb_file))
                         return
 
 
@@ -4732,10 +4741,13 @@ Author '{0}':
                 f.write(thumb_data)
 
             # Save thumb to archive
-            t_info = ZipInfo(title['uuid'],time.localtime()[0:6])
+            if zf is not None: # Ensure that the read succeeded
-            t_info.comment = cover_crc
+                # If we failed to open the zip file for reading,
-            with ZipFile(self.__archive_path, mode='a') as zfw:
+                # we dont know if it contained the thumb or not
-                zfw.writestr(t_info, thumb_data)
+                zf = open_archive('a')
+                if zf is not None:
+                    with zf:
+                        zf.writestr(title['uuid']+cover_crc, thumb_data)
 
         def getFriendlyGenreTag(self, genre):
             # Find the first instance of friendly_tag matching genre
@@ -5125,6 +5137,7 @@ Author '{0}':
                     OptionRecommendation.HIGH))
 
             # If cover exists, use it
+            cpath = None
             try:
                 search_text = 'title:"%s" author:%s' % (
                         opts.catalog_title.replace('"', '\\"'), 'calibre')
@@ -5145,5 +5158,10 @@ Author '{0}':
             plumber.merge_ui_recommendations(recommendations)
             plumber.run()
 
+            try:
+                os.remove(cpath)
+            except:
+                pass
+
         # returns to gui2.actions.catalog:catalog_generated()
         return catalog.error

src/calibre/library/database2.py

@@ -7,13 +7,11 @@ __docformat__ = 'restructuredtext en'
 The database used to store ebook metadata
 '''
 import os, sys, shutil, cStringIO, glob, time, functools, traceback, re, \
-        json, uuid, tempfile
+        json, uuid, tempfile, hashlib
 import threading, random
 from itertools import repeat
 from math import ceil
 
-from PyQt4.QtGui import QImage
-
 from calibre import prints
 from calibre.ebooks.metadata import (title_sort, author_to_author_sort,
         string_to_authors, authors_to_string)
@@ -27,7 +25,7 @@ from calibre.library.sqlite import connect, IntegrityError
 from calibre.library.prefs import DBPrefs
 from calibre.ebooks.metadata.book.base import Metadata
 from calibre.constants import preferred_encoding, iswindows, filesystem_encoding
-from calibre.ptempfile import PersistentTemporaryFile
+from calibre.ptempfile import PersistentTemporaryFile, base_dir
 from calibre.customize.ui import run_plugins_on_import
 from calibre import isbytestring
 from calibre.utils.filenames import ascii_filename
@@ -39,8 +37,10 @@ from calibre.ebooks import BOOK_EXTENSIONS, check_ebook_format
 from calibre.utils.magick.draw import save_cover_data_to
 from calibre.utils.recycle_bin import delete_file, delete_tree
 from calibre.utils.formatter_functions import load_user_template_functions
+from calibre.db.errors import NoSuchFormat
 
 copyfile = os.link if hasattr(os, 'link') else shutil.copyfile
+SPOOL_SIZE = 30*1024*1024
 
 class Tag(object):
 
@@ -201,16 +201,14 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
             dbprefs = DBPrefs(self)
             for key in default_prefs:
                 # be sure that prefs not to be copied are listed below
-                if key in ['news_to_be_synced']:
+                if key in frozenset(['news_to_be_synced']):
                     continue
-                try:
+                dbprefs[key] = default_prefs[key]
-                    dbprefs[key] = default_prefs[key]
+            if 'field_metadata' in default_prefs:
-                except:
+                fmvals = [f for f in default_prefs['field_metadata'].values() if f['is_custom']]
-                    pass # ignore options that don't exist anymore
+                for f in fmvals:
-            fmvals = [f for f in default_prefs['field_metadata'].values() if f['is_custom']]
+                    self.create_custom_column(f['label'], f['name'], f['datatype'],
-            for f in fmvals:
+                            f['is_multiple'] is not None, f['is_editable'], f['display'])
-                self.create_custom_column(f['label'], f['name'], f['datatype'],
-                        f['is_multiple'] is not None, f['is_editable'], f['display'])
         self.initialize_dynamic()
 
     def get_property(self, idx, index_is_id=False, loc=-1):
@@ -603,14 +601,12 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                 with lopen(os.path.join(tpath, 'cover.jpg'), 'wb') as f:
                     f.write(cdata)
             for format in formats:
-                # Get data as string (can't use file as source and target files may be the same)
+                with tempfile.SpooledTemporaryFile(max_size=SPOOL_SIZE) as stream:
-                f = self.format(id, format, index_is_id=True, as_file=True)
+                    try:
-                if f is None:
+                        self.copy_format_to(id, format, stream, index_is_id=True)
-                    continue
+                        stream.seek(0)
-                with tempfile.SpooledTemporaryFile(max_size=30*(1024**2)) as stream:
+                    except NoSuchFormat:
-                    with f:
+                        continue
-                        shutil.copyfileobj(f, stream)
-                    stream.seek(0)
                     self.add_format(id, format, stream, index_is_id=True,
                         path=tpath, notify=False)
         self.conn.execute('UPDATE books SET path=? WHERE id=?', (path, id))
@@ -663,32 +659,53 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                 continue
 
     def cover(self, index, index_is_id=False, as_file=False, as_image=False,
-              as_path=False):
+            as_path=False):
         '''
         Return the cover image as a bytestring (in JPEG format) or None.
 
-        `as_file` : If True return the image as an open file object
+        WARNING: Using as_path will copy the cover to a temp file and return
-        `as_image`: If True return the image as a QImage object
+        the path to the temp file. You should delete the temp file when you are
+        done with it.
+
+        :param as_file: If True return the image as an open file object (a SpooledTemporaryFile)
+        :param as_image: If True return the image as a QImage object
         '''
-        id = index if  index_is_id else self.id(index)
+        id = index if index_is_id else self.id(index)
         path = os.path.join(self.library_path, self.path(id, index_is_id=True), 'cover.jpg')
         if os.access(path, os.R_OK):
-            if as_path:
-                return path
             try:
                 f = lopen(path, 'rb')
             except (IOError, OSError):
                 time.sleep(0.2)
                 f = lopen(path, 'rb')
-            if as_image:
+            with f:
-                img = QImage()
+                if as_path:
-                img.loadFromData(f.read())
+                    pt = PersistentTemporaryFile('_dbcover.jpg')
-                f.close()
+                    with pt:
-                return img
+                        shutil.copyfileobj(f, pt)
-            ans = f if as_file else f.read()
+                    return pt.name
-            if ans is not f:
+                if as_file:
-                f.close()
+                    ret = tempfile.SpooledTemporaryFile(SPOOL_SIZE)
-            return ans
+                    shutil.copyfileobj(f, ret)
+                    ret.seek(0)
+                else:
+                    ret = f.read()
+                    if as_image:
+                        from PyQt4.Qt import QImage
+                        i = QImage()
+                        i.loadFromData(ret)
+                        ret = i
+            return ret
+
+    def cover_last_modified(self, index, index_is_id=False):
+        id = index if  index_is_id else self.id(index)
+        path = os.path.join(self.library_path, self.path(id, index_is_id=True), 'cover.jpg')
+        try:
+            return utcfromtimestamp(os.stat(path).st_mtime)
+        except:
+            # Cover doesn't exist
+            pass
+        return self.last_modified()
 
     ### The field-style interface. These use field keys.
 
@@ -861,7 +878,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         return (path, mi, sequence)
 
     def get_metadata(self, idx, index_is_id=False, get_cover=False,
-                     get_user_categories=True):
+                     get_user_categories=True, cover_as_data=False):
         '''
         Convenience method to return metadata as a :class:`Metadata` object.
         Note that the list of formats is not verified.
@@ -936,7 +953,12 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         mi.user_categories = user_cat_vals
 
         if get_cover:
-            mi.cover = self.cover(id, index_is_id=True, as_path=True)
+            if cover_as_data:
+                cdata = self.cover(id, index_is_id=True)
+                if cdata:
+                    mi.cover_data = ('jpeg', cdata)
+            else:
+                mi.cover = self.cover(id, index_is_id=True, as_path=True)
         return mi
 
     def has_book(self, mi):
@@ -1096,12 +1118,59 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         return self.format_abspath(index, format, index_is_id) is not None
 
     def format_last_modified(self, id_, fmt):
+        m = self.format_metadata(id_, fmt)
+        if m:
+            return m['mtime']
+
+    def format_metadata(self, id_, fmt):
         path = self.format_abspath(id_, fmt, index_is_id=True)
+        ans = {}
         if path is not None:
-            return utcfromtimestamp(os.stat(path).st_mtime)
+            stat = os.stat(path)
+            ans['size'] = stat.st_size
+            ans['mtime'] = utcfromtimestamp(stat.st_mtime)
+        return ans
+
+    def format_hash(self, id_, fmt):
+        path = self.format_abspath(id_, fmt, index_is_id=True)
+        if path is None:
+            raise NoSuchFormat('Record %d has no fmt: %s'%(id_, fmt))
+        sha = hashlib.sha256()
+        with lopen(path, 'rb') as f:
+            while True:
+                raw = f.read(SPOOL_SIZE)
+                sha.update(raw)
+                if len(raw) < SPOOL_SIZE:
+                    break
+        return sha.hexdigest()
+
+    def format_path(self, index, fmt, index_is_id=False):
+        '''
+        This method is intended to be used only in those rare situations, like
+        Drag'n Drop, when you absolutely need the path to the original file.
+        Otherwise, use format(..., as_path=True).
+
+        Note that a networked backend will always return None.
+        '''
+        path = self.format_abspath(index, fmt, index_is_id=index_is_id)
+        if path is None:
+            id_ = index if index_is_id else self.id(index)
+            raise NoSuchFormat('Record %d has no format: %s'%(id_, fmt))
+        return path
 
     def format_abspath(self, index, format, index_is_id=False):
-        'Return absolute path to the ebook file of format `format`'
+        '''
+        Return absolute path to the ebook file of format `format`
+
+        WARNING: This method will return a dummy path for a network backend DB,
+        so do not rely on it, use format(..., as_path=True) instead.
+
+        Currently used only in calibredb list, the viewer and the catalogs (via
+        get_data_as_dict()).
+
+        Apart from the viewer, I don't believe any of the others do any file
+        I/O with the results of this call.
+        '''
         id = index if index_is_id else self.id(index)
         try:
             name = self.conn.get('SELECT name FROM data WHERE book=? AND format=?', (id, format), all=False)
@@ -1121,25 +1190,63 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                 shutil.copyfile(candidates[0], fmt_path)
                 return fmt_path
 
-    def format(self, index, format, index_is_id=False, as_file=False, mode='r+b'):
+    def copy_format_to(self, index, fmt, dest, index_is_id=False):
+        '''
+        Copy the format ``fmt`` to the file like object ``dest``. If the
+        specified format does not exist, raises :class:`NoSuchFormat` error.
+        '''
+        path = self.format_abspath(index, fmt, index_is_id=index_is_id)
+        if path is None:
+            id_ = index if index_is_id else self.id(index)
+            raise NoSuchFormat('Record %d has no %s file'%(id_, fmt))
+        with lopen(path, 'rb') as f:
+            shutil.copyfileobj(f, dest)
+        if hasattr(dest, 'flush'):
+            dest.flush()
+
+    def format(self, index, format, index_is_id=False, as_file=False,
+            mode='r+b', as_path=False, preserve_filename=False):
         '''
         Return the ebook format as a bytestring or `None` if the format doesn't exist,
         or we don't have permission to write to the ebook file.
 
-        `as_file`: If True the ebook format is returned as a file object opened in `mode`
+        :param as_file: If True the ebook format is returned as a file object. Note
+                        that the file object is a SpooledTemporaryFile, so if what you want to
+                        do is copy the format to another file, use :method:`copy_format_to`
+                        instead for performance.
+        :param as_path: Copies the format file to a temp file and returns the
+                        path to the temp file
+        :param preserve_filename: If True and returning a path the filename is
+                                  the same as that used in the library. Note that using
+                                  this means that repeated calls yield the same
+                                  temp file (which is re-created each time)
+        :param mode: This is ignored (present for legacy compatibility)
         '''
         path = self.format_abspath(index, format, index_is_id=index_is_id)
         if path is not None:
-            f = lopen(path, mode)
+            with lopen(path, mode) as f:
-            try:
+                if as_path:
-                ret = f if as_file else f.read()
+                    if preserve_filename:
-            except IOError:
+                        bd = base_dir()
-                f.seek(0)
+                        d = os.path.join(bd, 'format_abspath')
-                out = cStringIO.StringIO()
+                        try:
-                shutil.copyfileobj(f, out)
+                            os.makedirs(d)
-                ret = out.getvalue()
+                        except:
-            if not as_file:
+                            pass
-                f.close()
+                        fname = os.path.basename(path)
+                        ret = os.path.join(d, fname)
+                        with lopen(ret, 'wb') as f2:
+                            shutil.copyfileobj(f, f2)
+                    else:
+                        with PersistentTemporaryFile('.'+format.lower()) as pt:
+                            shutil.copyfileobj(f, pt)
+                            ret = pt.name
+                elif as_file:
+                    ret = tempfile.SpooledTemporaryFile(max_size=SPOOL_SIZE)
+                    shutil.copyfileobj(f, ret)
+                    ret.seek(0)
+                else:
+                    ret = f.read()
             return ret
 
     def add_format_with_hooks(self, index, format, fpath, index_is_id=False,
@@ -1931,13 +2038,13 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
     def authors_with_sort_strings(self, id, index_is_id=False):
         id = id if index_is_id else self.id(id)
         aut_strings = self.conn.get('''
-                        SELECT authors.name, authors.sort
+                        SELECT authors.id, authors.name, authors.sort
                         FROM authors, books_authors_link as bl
                         WHERE bl.book=? and authors.id=bl.author
                         ORDER BY bl.id''', (id,))
         result = []
-        for (author, sort,) in aut_strings:
+        for (id_, author, sort,) in aut_strings:
-            result.append((author.replace('|', ','), sort))
+            result.append((id_, author.replace('|', ','), sort))
         return result
 
     # Given a book, return the author_sort string for authors of the book
@@ -1945,6 +2052,15 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         auts = self.authors_sort_strings(id, index_is_id)
         return ' & '.join(auts).replace('|', ',')
 
+    # Given an author, return a list of books with that author
+    def books_for_author(self, id_, index_is_id=False):
+        id_ = id_ if index_is_id else self.id(id_)
+        books = self.conn.get('''
+                        SELECT bl.book
+                        FROM books_authors_link as bl
+                        WHERE bl.author=?''', (id_,))
+        return [b[0] for b in books]
+
     # Given a list of authors, return the author_sort string for the authors,
     # preferring the author sort associated with the author over the computed
     # string
@@ -1968,7 +2084,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
 
         aum = self.authors_with_sort_strings(id_, index_is_id=True)
         self.data.set(id_, self.FIELD_MAP['au_map'],
-            ':#:'.join([':::'.join((au.replace(',', '|'), aus)) for (au, aus) in aum]),
+            ':#:'.join([':::'.join((au.replace(',', '|'), aus)) for (_, au, aus) in aum]),
             row_is_id=True)
 
     def _set_authors(self, id, authors, allow_case_change=False):

src/calibre/library/save_to_disk.py

@@ -6,7 +6,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
-import os, traceback, cStringIO, re, shutil
+import os, traceback, cStringIO, re
 
 from calibre.constants import DEBUG
 from calibre.utils.config import Config, StringConfig, tweaks
@@ -238,7 +238,7 @@ def get_components(template, mi, id, timefmt='%b %Y', length=250,
 
 def save_book_to_disk(id_, db, root, opts, length):
     mi = db.get_metadata(id_, index_is_id=True)
-    cover = db.cover(id_, index_is_id=True, as_path=True)
+    cover = db.cover(id_, index_is_id=True)
     plugboards = db.prefs.get('plugboards', {})
 
     available_formats = db.formats(id_, index_is_id=True)
@@ -252,12 +252,20 @@ def save_book_to_disk(id_, db, root, opts, length):
     if fmts:
         fmts = fmts.split(',')
         for fmt in fmts:
-            fpath = db.format_abspath(id_, fmt, index_is_id=True)
+            fpath = db.format(id_, fmt, index_is_id=True, as_path=True)
             if fpath is not None:
                 formats[fmt.lower()] = fpath
 
-    return do_save_book_to_disk(id_, mi, cover, plugboards,
+    try:
+        return do_save_book_to_disk(id_, mi, cover, plugboards,
             formats, root, opts, length)
+    finally:
+        for temp in formats.itervalues():
+            try:
+                os.remove(temp)
+            except:
+                pass
+
 
 
 def do_save_book_to_disk(id_, mi, cover, plugboards,
@@ -289,10 +297,9 @@ def do_save_book_to_disk(id_, mi, cover, plugboards,
             raise
 
     ocover = mi.cover
-    if opts.save_cover and cover and os.access(cover, os.R_OK):
+    if opts.save_cover and cover:
         with open(base_path+'.jpg', 'wb') as f:
-            with open(cover, 'rb') as s:
+            f.write(cover)
-                shutil.copyfileobj(s, f)
         mi.cover = base_name+'.jpg'
     else:
         mi.cover = None
@@ -395,8 +402,13 @@ def save_serialized_to_disk(ids, data, plugboards, root, opts, callback):
             pass
         tb = ''
         try:
-            failed, id, title = do_save_book_to_disk(x, mi, cover, plugboards,
+            with open(cover, 'rb') as f:
-                    format_map, root, opts, length)
+                cover = f.read()
+        except:
+            cover = None
+        try:
+            failed, id, title = do_save_book_to_disk(x, mi, cover,
+                plugboards, format_map, root, opts, length)
             tb = _('Requested formats not available')
         except:
             failed, id, title = True, x, mi.title

src/calibre/library/server/content.py

@@ -10,12 +10,13 @@ import re, os, posixpath
 import cherrypy
 
 from calibre import fit_image, guess_type
-from calibre.utils.date import fromtimestamp
+from calibre.utils.date import fromtimestamp, utcnow
 from calibre.library.caches import SortKeyGenerator
 from calibre.library.save_to_disk import find_plugboard
-
+from calibre.ebooks.metadata import authors_to_string
-from calibre.utils.magick.draw import save_cover_data_to, Image, \
+from calibre.utils.magick.draw import (save_cover_data_to, Image,
-        thumbnail as generate_thumbnail
+        thumbnail as generate_thumbnail)
+from calibre.utils.filenames import ascii_filename
 
 plugboard_content_server_value = 'content_server'
 plugboard_content_server_formats = ['epub']
@@ -46,7 +47,7 @@ class ContentServer(object):
     # Utility methods {{{
     def last_modified(self, updated):
         '''
-        Generates a local independent, english timestamp from a datetime
+        Generates a locale independent, english timestamp from a datetime
         object
         '''
         lm = updated.strftime('day, %d month %Y %H:%M:%S GMT')
@@ -151,14 +152,12 @@ class ContentServer(object):
         try:
             cherrypy.response.headers['Content-Type'] = 'image/jpeg'
             cherrypy.response.timeout = 3600
-            cover = self.db.cover(id, index_is_id=True, as_file=True)
+            cover = self.db.cover(id, index_is_id=True)
             if cover is None:
                 cover = self.default_cover
                 updated = self.build_time
             else:
-                with cover as f:
+                updated = self.db.cover_last_modified(id, index_is_id=True)
-                    updated = fromtimestamp(os.fstat(f.fileno()).st_mtime)
-                    cover = f.read()
             cherrypy.response.headers['Last-Modified'] = self.last_modified(updated)
 
             if thumbnail:
@@ -187,9 +186,9 @@ class ContentServer(object):
                 mode='rb')
         if fmt is None:
             raise cherrypy.HTTPError(404, 'book: %d does not have format: %s'%(id, format))
+        mi = self.db.get_metadata(id, index_is_id=True)
         if format == 'EPUB':
             # Get the original metadata
-            mi = self.db.get_metadata(id, index_is_id=True)
 
             # Get any EPUB plugboards for the content server
             plugboards = self.db.prefs.get('plugboards', {})
@@ -203,24 +202,22 @@ class ContentServer(object):
                 newmi = mi
 
             # Write the updated file
-            from tempfile import TemporaryFile
             from calibre.ebooks.metadata.meta import set_metadata
-            raw = fmt.read()
-            fmt = TemporaryFile()
-            fmt.write(raw)
-            fmt.seek(0)
             set_metadata(fmt, newmi, 'epub')
             fmt.seek(0)
 
         mt = guess_type('dummy.'+format.lower())[0]
         if mt is None:
             mt = 'application/octet-stream'
+        au = authors_to_string(mi.authors if mi.authors else [_('Unknown')])
+        title = mi.title if mi.title else _('Unknown')
+        fname = u'%s - %s_%s.%s'%(title[:30], au[:30], id, format.lower())
+        fname = ascii_filename(fname).replace('"', '_')
         cherrypy.response.headers['Content-Type'] = mt
+        cherrypy.response.headers['Content-Disposition'] = \
+                b'attachment; filename="%s"'%fname
         cherrypy.response.timeout = 3600
-        path = getattr(fmt, 'name', None)
+        cherrypy.response.headers['Last-Modified'] = self.last_modified(utcnow())
-        if path and os.path.exists(path):
-            updated = fromtimestamp(os.stat(path).st_mtime)
-            cherrypy.response.headers['Last-Modified'] = self.last_modified(updated)
         return fmt
     # }}}
 

src/calibre/manual/conversion.rst

@@ -633,6 +633,7 @@ TXT input supports a number of options to differentiate how paragraphs are detec
     :guilabel:`Formatting Style: None`
         Applies no special formatting to the text, the document is converted to html with no other changes.
 
+.. _pdfconversion:
 
 Convert PDF documents
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~

src/calibre/manual/faq.rst

@@ -35,29 +35,11 @@ What are the best source formats to convert?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 In order of decreasing preference: LIT, MOBI, EPUB, FB2, HTML, PRC, RTF, PDB, TXT, PDF
 
-Why does the PDF conversion lose some images/tables?
+I converted a PDF file, but the result has various problems?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The PDF conversion tries to extract the text and images from the PDF file and convert them to and HTML based ebook. Some PDF files have images in a format that cannot be extracted (vector images). All tables
-are also represented as vector diagrams, thus they cannot be extracted.
 
-How do I convert a collection of HTML files in a specific order?
+PDF is a terrible format to convert from. For a list of the various issues you will encounter when converting PDF, see: :ref:`pdfconversion`.
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-In order to convert a collection of HTML files in a specific oder, you have to create a table of contents file. That is, another HTML file that contains links to all the other files in the desired order. Such a file looks like::
 
-   <html>
-      <body>
-        <h1>Table of Contents</h1>
-        <p style="text-indent:0pt">
-           <a href="file1.html">First File</a><br/>
-           <a href="file2.html">Second File</a><br/>
-           .
-           .
-           .
-        </p>
-      </body>
-   </html>
-
-Then just add this HTML file to the GUI and use the convert button to create your ebook.
 
 .. _char-encoding-faq:
 
@@ -85,6 +67,26 @@ If you have a hand edited TOC in the input document, you can use the TOC detecti
 
 Finally, I encourage you to ditch the content TOC and only have a metadata TOC in your ebooks. Metadata TOCs will give the people reading your ebooks a much superior navigation experience (except on the Kindle, where they are essentially the same as a content TOC).
 
+How do I convert a collection of HTML files in a specific order?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+In order to convert a collection of HTML files in a specific oder, you have to create a table of contents file. That is, another HTML file that contains links to all the other files in the desired order. Such a file looks like::
+
+   <html>
+      <body>
+        <h1>Table of Contents</h1>
+        <p style="text-indent:0pt">
+           <a href="file1.html">First File</a><br/>
+           <a href="file2.html">Second File</a><br/>
+           .
+           .
+           .
+        </p>
+      </body>
+   </html>
+
+Then just add this HTML file to the GUI and use the convert button to create your ebook.
+
+
 How do I use some of the advanced features of the conversion tools?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  You can get help on any individual feature of the converters by mousing over it in the GUI or running ``ebook-convert dummy.html .epub -h`` at a terminal. A good place to start is to look at the following demo files that demonstrate some of the advanced features: