Sync to trunk.

Changelog.yaml

@@ -19,6 +19,78 @@
 #  new recipes:
 #    - title: 
 
+- version: 0.7.58
+  date: 2011-04-29
+
+  new features:
+    - title: "Support for converting and reading metadata from Plucker format PDB files"
+      type: major
+
+    - title: "The metadata that is displayed in the book details panel on the right is now completely configurable via Preferences->Look & Feel"
+
+    - title: "Add a column that shows the date when the metadata of a book record was last modified in calibre. To see the column, right click on the column headers in calibre and select Show column->Modified. Note that the dates may be incorrect for books added with older versions of calibre."
+
+    - title: "Add command line option to shutdown running calibre"
+
+    - title: "CHM Input: Store extracted files in the input/ sub dir for easy debugging when --debug-pipeline is specified"
+
+    - title: "Add a popup menu to the 'Create saved search button' to allow easy deleting of saved searches"
+  
+  bug fixes:
+    - title: "Fix regression that broke converting to LIT in 0.7.57"
+      tickets: [769334]
+
+    - title: "Conversion pipeline: Remove encoding declarations from input HTML documents to guarantee that there is only a single encoding declaration in the output HTML."
+      tickets: [773337]
+
+    - title: "Correctly parenthesize searches that are used to make search restrictions"
+
+    - title: "Fix ratings in save to disk templates not being divided by 2"
+
+    - title: "TXT to EPUB: Underlined words (following quotes?) fail to become italics"
+      tickets: [772267]
+
+    - title: "Fix template function source code unavailable when not running calibre from source"
+
+    - title: "Fix adding html books from the top of a deep folder hierarchy very slow"
+
+    - title: "Only set language in MOBI metadata if it is not null"
+
+    - title: "Fix 'count-of' searches (e.g., tags:#>3)."
+      tickets: [771175]
+
+    - title: "Fix regression that broke connection to iTunes in some cases"
+      tickets: [771164]
+
+    - title: "Fix buggy regex that made converting PDFs with the string ****************** very slow"
+      tickets: [770534]
+
+    - title: "Fix Ctrl+L shortcut to lookup word not working in ebook viewer"
+      tickets: [769492]
+
+    - title: "Fix regression that broke searching on boolean columns"
+
+  improved recipes:
+    - HBR Blogs
+    - The Marker
+    - Financial Times
+    - Clarin
+    - Honolulu Star Advertiser
+
+  new recipes:
+    - title: Novi Standard
+      author: Darko Miletic
+
+    - title: Autobild.ro and Social Diva
+      author: Silviu Cotoara
+
+    - title: Novinky
+      author: Tomas Latal
+
+    - title: "De Volksrant (subscriber version)"
+      author: Selcal
+
+
 - version: 0.7.57
   date: 2011-04-22
 

recipes/autobild.recipe

@@ -0,0 +1,55 @@
+# -*- coding: utf-8 -*-
+#!/usr/bin/env  python
+
+__license__   = 'GPL v3'
+__copyright__ = u'2011, Silviu Cotoar\u0103'
+'''
+auto-bild.ro
+'''
+
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class AutoBild(BasicNewsRecipe):
+    title                       = u'Auto Bild'
+    __author__                   = u'Silviu Cotoar\u0103'
+    description                   = 'Auto'
+    publisher                   = 'Auto Bild'
+    oldest_article              = 50
+    language                    = 'ro'
+    max_articles_per_feed = 100
+    no_stylesheets          = True
+    use_embedded_content  = False
+    category                  = 'Ziare,Reviste,Auto'
+    encoding                = 'utf-8'
+    cover_url     = 'http://www.auto-bild.ro/images/autobild.gif'
+
+    conversion_options = {
+                             'comments'  : description
+                            ,'tags'           : category
+                            ,'language'    : language
+                            ,'publisher'   : publisher
+                         }
+
+
+    keep_only_tags = [
+                dict(name='div', attrs={'class':'box_2 articol clearfix'})
+                 ]
+
+    remove_tags = [
+             dict(name='div', attrs={'class':['detail']})
+           , dict(name='a', attrs={'id':['zoom_link']})
+           , dict(name='div', attrs={'class':['icons clearfix']})
+           , dict(name='div', attrs={'class':['pub_articol clearfix']})
+
+                          ]
+
+    remove_tags_after = [
+              dict(name='div', attrs={'class':['pub_articol clearfix']})
+           ]
+
+    feeds          = [
+        (u'Feeds', u'http://www.auto-bild.ro/rss/toate')
+                     ]
+
+    def preprocess_html(self, soup):
+        return self.adeify_images(soup)

recipes/diario_ibiza.recipe

@@ -0,0 +1,55 @@
+__license__   = 'GPL v3'
+__author__    = 'Joan Tur, based on El Pais version by Jordi Balcells & elargentino.com version by Darko Miletic'
+description   = 'Principal periodico de las islas Pitiusas, Ibiza y Formentera (Espanya) - v1.06 (29/04/2011)'
+__docformat__ = 'restructuredtext en'
+
+'''
+diariodeibiza.es
+'''
+
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class DiarioDeIbiza(BasicNewsRecipe):
+    __author__        = 'Joan Tur, cullet'
+    description   = 'Principal periodico de las islas Pitiusas, Ibiza y Formentera (Espanya) - v1.06 (29/04/2011)'
+
+    cover_url      = 'http://estaticos01.diariodeibiza.es//elementosWeb/mediaweb/images/logo.jpg'
+    title          = u'Diario de Ibiza digital'
+    publisher      = u'Editorial Prensa Iberica'
+    category       = 'News, politics, culture, economy, general interest'
+    language = 'es'
+
+    encoding              = 'iso-8859-1'
+
+    timefmt        = '[%a, %d %b, %Y]'
+
+    oldest_article = 2
+    max_articles_per_feed = 20
+
+    use_embedded_content  = False
+    recursion             = 5
+
+    remove_javascript = True
+    no_stylesheets = True
+
+    keep_only_tags = [
+                        dict(name='div', attrs={'class':['noticia_titular','epigrafe','subtitulo','actualizada','noticia_fecha','noticia_texto']}),
+                        dict(name='font', attrs={'class':['actualizada']})
+                     ]
+
+    feeds          = [
+                        (u'Portada de Ibiza', u'http://www.diariodeibiza.es/elementosInt/rss/1'),
+                        (u'Pitiuses i Balears', u'http://www.diariodeibiza.es/elementosInt/rss/2'),
+                        (u'Opini\xf3n', u'http://www.diariodeibiza.es/elementosInt/rss/3'),
+                        (u'Nacional', u'http://www.diariodeibiza.es/elementosInt/rss/4'),
+                        (u'Internacional', u'http://www.diariodeibiza.es/elementosInt/rss/5'),
+                        (u'Econom\xeda', u'http://www.diariodeibiza.es/elementosInt/rss/6'),
+                        (u'Deportes', u'http://www.diariodeibiza.es/elementosInt/rss/7'),
+                        (u'Sociedad', u'http://www.diariodeibiza.es/elementosInt/rss/8'),
+                        (u'Ciencia', u'http://www.diariodeibiza.es/elementosInt/rss/11'),
+                        (u'Tecnolog\xeda', u'http://www.diariodeibiza.es/elementosInt/rss/12'),
+                        (u'Gente', u'http://www.diariodeibiza.es/elementosInt/rss/13'),
+                        (u'Sucesos', u'http://www.diariodeibiza.es/elementosInt/rss/15'),
+                        (u'Cultura', u'http://www.diariodeibiza.es/elementosInt/rss/16Piti')
+                    ]
+

recipes/hbr_blogs.recipe

@@ -1,9 +1,6 @@
 from calibre.web.feeds.news import BasicNewsRecipe
 import re
 
-# Needed for BLOGs
-from calibre.web.feeds import Feed
-
 class HBR(BasicNewsRecipe):
 
     title = 'Harvard Business Review Blogs'
@@ -32,6 +29,7 @@ class HBR(BasicNewsRecipe):
         feeds = [('Blog','http://feeds.harvardbusiness.org/harvardbusiness')]
         oldest_article = 30
         max_articles_per_feed = 100
+        use_embedded_content = False
     else:
         timefmt                = ' [%B %Y]'
 
@@ -59,9 +57,9 @@ class HBR(BasicNewsRecipe):
     def get_browser(self):
         br = BasicNewsRecipe.get_browser(self)
         br.open(self.LOGIN_URL)
-        br.select_form(name='signInForm')
-        br['signInForm:username'] = self.username
-        br['signInForm:password'] = self.password
+        br.select_form(name='signin-form')
+        br['signin-form:username'] = self.username
+        br['signin-form:password'] = self.password
         raw = br.submit().read()
         if 'My Account' not in raw:
             raise Exception('Failed to login, are you sure your username and password are correct?')
@@ -161,27 +159,13 @@ class HBR(BasicNewsRecipe):
         return startDate, endDate
 
 #-------------------------------------------------------------------------------------------------
-    def hbr_parse_blogs(self, feeds):
-        # Do the "official" parse_feeds first
-        rssFeeds = Feed()
 
-        # Use the PARSE_FEEDS method to get a Feeds object of the articles
-        rssFeeds = BasicNewsRecipe.parse_feeds(self)
-
-        # Create a new feed of the right configuration and append to existing afeeds
-        self.feed_to_index_append(rssFeeds[:], feeds)
-
-#-------------------------------------------------------------------------------------------------
     def parse_index(self):
         if self.INCLUDE_ARTICLES == True:
             soup = self.hbr_get_toc()
             feeds = self.hbr_parse_toc(soup)
         else:
-            feeds = []
-
-        # blog stuff
-        if self.INCLUDE_BLOGS == True:
-            self.hbr_parse_blogs(feeds)
+            return BasicNewsRecipe.parse_index(self)
 
         return feeds
 #-------------------------------------------------------------------------------------------------

recipes/le_monde.recipe

@@ -48,7 +48,7 @@ class LeMonde(BasicNewsRecipe):
             if alink.string is not None:
                tstr = alink.string
                alink.replaceWith(tstr)
-        return soup
+        return self.adeify_images(soup)
 
     preprocess_regexps = [
         (re.compile(r'([0-9])%'), lambda m: m.group(1) + ' %'),

recipes/novinky.recipe

@@ -0,0 +1,42 @@
+__license__   = 'GPL v3'
+__copyright__ = '2011, Tomas Latal<latal.tomas at gmail.com>'
+
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class NovinkyCZ(BasicNewsRecipe):
+    title                   = 'Novinky'
+    __author__              = 'Tomas Latal'
+    __version__             = '1.0'
+    __date__                = '24 April 2011'
+    description             = 'News from server Novinky.cz'
+    oldest_article          = 1
+    max_articles_per_feed   = 10
+    encoding                = 'utf8'
+    publisher               = 'Novinky'
+    category                = 'news, CZ'
+    language                = 'cs'
+    publication_type        = 'newsportal'
+    no_stylesheets          = True
+    remove_javascript       = True
+    extra_css               = 'p.acmDescription{font-style:italic;} p.acmAuthor{font-size:0.8em; color:#707070}'
+
+    feeds          = [
+                        (u'Dom\xe1c\xed', u'http://www.novinky.cz/rss/domaci/'),
+                        (u'Zahrani\u010d\xed', u'http://www.novinky.cz/rss/zahranicni/'),
+                        (u'Krimi', u'http://www.novinky.cz/rss/krimi/'),
+                        (u'Ekonomika', u'http://www.novinky.cz/rss/ekonomika/'),
+                        (u'Finance', u'http://www.novinky.cz/rss/finance/'),
+                        (u'Kultura', u'http://www.novinky.cz/rss/kultura/'),
+                        (u'Koktejl', u'http://www.novinky.cz/rss/koktejl/'),
+                        (u'Internet a PC', u'http://www.novinky.cz/rss/internet-a-pc/'),
+                        (u'Auto-moto', u'http://www.novinky.cz/rss/auto/'),
+                    ]
+
+    remove_tags_before = dict(id='articleContent')
+
+    remove_tags_after = [dict(id='movedArticleAuthors')]
+
+    remove_tags = [
+                      dict(name='div', attrs={'id':['articleColumnInfo','pictureInnerBox']}),
+                      dict(name='p', attrs={'id':['articleDate']})
+                  ]

recipes/novistandard.recipe

@@ -0,0 +1,100 @@
+
+__license__   = 'GPL v3'
+__copyright__ = '2011, Darko Miletic <darko.miletic at gmail.com>'
+'''
+www.standard.rs
+'''
+
+import re
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class NoviStandard(BasicNewsRecipe):
+    title                  = 'Novi Standard'
+    __author__             = 'Darko Miletic'
+    description            = 'NoviStandard - energija je neunistiva!'
+    publisher              = 'Novi Standard'
+    category               = 'news, politics, Serbia'
+    no_stylesheets         = True
+    delay                  = 1
+    oldest_article         = 15
+    encoding               = 'utf-8'
+    publication_type       = 'magazine'
+    needs_subscription     = 'optional'
+    remove_empty_feeds     = True
+    INDEX                  = 'http://www.standard.rs/'
+    use_embedded_content   = False
+    language               = 'sr'
+    publication_type       = 'magazine'
+    masthead_url           = 'http://www.standard.rs/templates/ja_opal/images/red/logo.png'
+    extra_css              = """
+                                 @font-face {font-family: "serif1";src:url(res:///opt/sony/ebook/FONT/tt0011m_.ttf)}
+                                 @font-face {font-family: "sans1";src:url(res:///opt/sony/ebook/FONT/tt0003m_.ttf)}
+                                 body{font-family: Arial,"Segoe UI","Trebuchet MS",Helvetica,sans1,sans-serif}
+                                 .dropcap{font-family: Georgia,Times,serif1,serif; display:inline}
+                                 .dropcap:first-letter{display: inline; font-size: xx-large; font-weight: bold}
+                                 .contentheading{color: gray; font-size: x-large}
+                                 .article-meta, .createdby{color: red}
+                                 img{margin-top:0.5em; margin-bottom: 0.7em; display: block}
+                             """
+
+    conversion_options = {
+                          'comment'   : description
+                        , 'tags'      : category
+                        , 'publisher' : publisher
+                        , 'language'  : language
+                        }
+
+    preprocess_regexps = [(re.compile(u'\u0110'), lambda match: u'\u00D0')]
+
+    def get_browser(self):
+        br = BasicNewsRecipe.get_browser()
+        br.open(self.INDEX)
+        if self.username is not None and self.password is not None:
+            br.select_form(name='login')
+            br['username'] = self.username
+            br['passwd'  ] = self.password
+            br.submit()
+        return br
+
+    keep_only_tags    =[dict(attrs={'class':['contentheading','article-meta','article-content']})]
+    remove_tags_after =dict(attrs={'class':'extravote-container'})
+    remove_tags       = [
+                           dict(name=['object','link','iframe','meta','base'])
+                          ,dict(attrs={'class':'extravote-container'})
+                        ]
+    remove_attributes =['border','background','height','width','align','valign','lang']
+    feeds             = [
+                            (u'Naslovna', u'http://www.standard.rs/index.php?format=feed&type=rss')
+                           ,(u'Politika', u'http://www.standard.rs/vesti/36-politika.html?format=feed&type=rss')
+                           ,(u'Cvijanovic preporucuje', u'http://www.standard.rs/-cvijanovi-vam-preporuuje.html?format=feed&type=rss')
+                           ,(u'Kolumne', u'http://www.standard.rs/vesti/49-kolumne.html?format=feed&type=rss')
+                           ,(u'Kultura', u'http://www.standard.rs/vesti/40-kultura.html?format=feed&type=rss')
+                           ,(u'Lifestyle', u'http://www.standard.rs/vesti/39-lifestyle.html?format=feed&type=rss')
+                           ,(u'Svet', u'http://www.standard.rs/vesti/41-svet.html?format=feed&type=rss')
+                           ,(u'Ekonomija', u'http://www.standard.rs/vesti/37-ekonomija.html?format=feed&type=rss')
+                           ,(u'Sport', u'http://www.standard.rs/vesti/38-sport.html?format=feed&type=rss')
+                        ]
+
+
+    def preprocess_html(self, soup):
+        for item in soup.findAll(style=True):
+            del item['style']
+        for item in soup.findAll('div'):
+            if len(item.contents) == 0:
+               item.extract()
+        for item in soup.findAll('a'):
+            limg = item.find('img')
+            if item.string is not None:
+               str = item.string
+               item.replaceWith(str)
+            else:
+               if limg:
+                  item.name = 'div'
+                  item.attrs = []
+               else:
+                   str = self.tag_to_string(item)
+                   item.replaceWith(str)
+        for item in soup.findAll('img'):
+            if not item.has_key('alt'):
+               item['alt'] = 'image'
+        return soup

recipes/socialdiva.recipe

@@ -0,0 +1,54 @@
+# -*- coding: utf-8 -*-
+#!/usr/bin/env  python
+
+__license__   = 'GPL v3'
+__copyright__ = u'2011'
+'''
+socialdiva.ro
+'''
+
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class SocialDiva(BasicNewsRecipe):
+    title                 = u'Social Diva'
+    __author__            = u'Silviu Cotoara'
+    description           = u'When in doubt, wear red'
+    publisher             = 'Social Diva'
+    oldest_article        = 5
+    language              = 'ro'
+    max_articles_per_feed = 100
+    no_stylesheets        = True
+    use_embedded_content  = False
+    category              = 'Ziare,Reviste,Femei'
+    encoding              = 'utf-8'
+    cover_url             = 'http://www.socialdiva.ro/images/logo.png'
+
+    conversion_options = {
+                             'comments'   : description
+                            ,'tags'       : category
+                            ,'language'   : language
+                            ,'publisher'  : publisher
+                         }
+
+    keep_only_tags = [
+            dict(name='div', attrs={'class':'col-alpha mt5 content_articol'}),
+            dict(name='div', attrs={'class':'mt5'})
+                     ]
+
+    remove_tags = [
+             dict(name='a', attrs={'class':['comments float-left scroll mt5']}),
+             dict(name='a', attrs={'class':['comments float-left scroll']}),
+             dict(name='div', attrs={'class':['rating-container relative float-left']}),
+             dict(name='div', attrs={'class':['float-right social_articol']})
+                  ]
+
+    remove_tags_after = [
+            dict(name='a', attrs={'class':['comments float-left scroll mt5']})
+            ]
+
+    feeds          = [
+            (u'Feeds', u'http://www.socialdiva.ro/rss.html')
+                 ]
+
+    def preprocess_html(self, soup):
+        return self.adeify_images(soup)

recipes/the_marker.recipe

@@ -3,7 +3,7 @@ from calibre.web.feeds.news import BasicNewsRecipe
 
 class AdvancedUserRecipe1283848012(BasicNewsRecipe):
     description   = 'TheMarker Financial News in Hebrew'
-    __author__            = 'TonyTheBookworm, Marbs'
+    __author__            = 'Marbs'
     cover_url      = 'http://static.ispot.co.il/wp-content/upload/2009/09/themarker.jpg'
     title          = u'TheMarker'
     language              = 'he'
@@ -11,42 +11,38 @@ class AdvancedUserRecipe1283848012(BasicNewsRecipe):
     remove_javascript     = True
     timefmt        = '[%a, %d %b, %Y]'
     oldest_article = 1
-    remove_tags = [dict(name='tr', attrs={'bgcolor':['#738A94']})          ]
-    max_articles_per_feed = 10
+    keep_only_tags =dict(name='div', attrs={'id':'content'})
+    remove_attributes = ['width','float','margin-left']
+    no_stylesheets        = True
+    remove_tags = [dict(name='div', attrs={'class':['social-nav article-social-nav','prsnlArticleEnvelope','cb']}) ,
+                            dict(name='a', attrs={'href':['/misc/mobile']})  ,
+                            dict(name='span', attrs={'class':['post-summ']}) ]
+    max_articles_per_feed = 100
     extra_css='body{direction: rtl;} .article_description{direction: rtl; } a.article{direction: rtl; } .calibre_feed_description{direction: rtl; }'
-    feeds          = [(u'Head Lines', u'http://www.themarker.com/tmc/content/xml/rss/hpfeed.xml'),
-                      (u'TA Market', u'http://www.themarker.com/tmc/content/xml/rss/sections/marketfeed.xml'),
-                      (u'Real Estate', u'http://www.themarker.com/tmc/content/xml/rss/sections/realEstaterfeed.xml'),
-                      (u'Wall Street & Global', u'http://www.themarker.com/tmc/content/xml/rss/sections/wallsfeed.xml'),
-                      (u'Law', u'http://www.themarker.com/tmc/content/xml/rss/sections/lawfeed.xml'),
-                      (u'Media', u'http://www.themarker.com/tmc/content/xml/rss/sections/mediafeed.xml'),
-                      (u'Consumer', u'http://www.themarker.com/tmc/content/xml/rss/sections/consumerfeed.xml'),
-                      (u'Career', u'http://www.themarker.com/tmc/content/xml/rss/sections/careerfeed.xml'),
-                      (u'Car', u'http://www.themarker.com/tmc/content/xml/rss/sections/carfeed.xml'),
-                      (u'High Tech', u'http://www.themarker.com/tmc/content/xml/rss/sections/hightechfeed.xml'),
-                      (u'Investor Guide', u'http://www.themarker.com/tmc/content/xml/rss/sections/investorGuidefeed.xml')]
+    feeds          = [(u'Head Lines', u'http://www.themarker.com/cmlink/1.144'),
+                      (u'TA Market', u'http://www.themarker.com/cmlink/1.243'),
+                      (u'Real Estate', u'http://www.themarker.com/cmlink/1.605656'),
+                      (u'Global', u'http://www.themarker.com/cmlink/1.605658'),
+                      (u'Wall Street', u'http://www.themarker.com/cmlink/1.613713'),
+                      (u'SmartPhone', u'http://www.themarker.com/cmlink/1.605661'),
+                      (u'Law', u'http://www.themarker.com/cmlink/1.605664'),
+                      (u'Media', u'http://www.themarker.com/cmlink/1.605660'),
+                      (u'Consumer', u'http://www.themarker.com/cmlink/1.605662'),
+                      (u'Career', u'http://www.themarker.com/cmlink/1.605665'),
+                      (u'Car', u'http://www.themarker.com/cmlink/1.605663'),
+                      (u'High Tech', u'http://www.themarker.com/cmlink/1.605659'),
+                      (u'Small Business', u'http://www.themarker.com/cmlink/1.605666')]
 
     def print_version(self, url):
-        split1 = url.split("=")
-        weblinks = url
+        #split1 = url.split("/")
+        #print_url='http://www.themarker.com/misc/article-print-page/'+split1[-1]
+        txt=url
 
-        if weblinks is not None:
-            for link in weblinks:
-                #---------------------------------------------------------
-                #here we need some help with some regexpressions
-                #we are trying to find it.themarker.com in a url
-                #-----------------------------------------------------------
-                re1='.*?'   # Non-greedy match on filler
-                re2='(it\\.themarker\\.com)'    # Fully Qualified Domain Name 1
-                rg = re.compile(re1+re2,re.IGNORECASE|re.DOTALL)
-                m = rg.search(url)
+        re1='.*?'	# Non-greedy match on filler
+        re2='(tv)'	# Word 1
 
-
-                if m:
-                 split2 = url.split("article/")
-                 print_url = 'http://it.themarker.com/tmit/PrintArticle/' + split2[1]
-
-                else:
-                    print_url = 'http://www.themarker.com/ibo/misc/printFriendly.jhtml?ElementId=%2Fibo%2Frepositories%2Fstories%2Fm1_2000%2F' + split1[1]+'.xml'
-
-        return print_url
+        rg = re.compile(re1+re2,re.IGNORECASE|re.DOTALL)
+        m = rg.search(txt)
+        if m:
+            #print 'bad link'
+            return 1

recipes/volksrant_sub.recipe

@@ -0,0 +1,115 @@
+from calibre import strftime
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class Volkskrant_full(BasicNewsRecipe):
+    # This recipe will download the Volkskrant newspaper,
+    # from the subscribers site. It requires a password.
+    # Known issues are: articles that are spread out over
+    # multiple pages will appear multiple times. Pages
+    # that contain only adverts will appear, but empty.
+    # The supplement 'Volkskrant Magazine' on saturday
+    # is currently not downloaded.
+    # You can set a manual date, to download an archived
+    # newspaper. Volkskrant stores over a month at the
+    # moment of writing. To do so I suggest you unmark
+    # the date on the line below, and insert it in the title. Then
+    # follow the instructions marked further below.
+
+    title = 'De Volkskrant (subscription)' # [za, 13 nov 2010]'
+    __author__ = u'Selcal'
+    description = u"Volkskrant"
+    oldest_article = 30
+    max_articles_per_feed = 100
+    no_stylesheets = True
+    language = 'nl'
+    use_embedded_content = False
+    simultaneous_downloads = 1
+    delay = 1
+    needs_subscription = True
+    # Set RETRIEVEDATE to 'yyyymmdd' to load an older
+    # edition. Otherwise keep '%Y%m%d'
+    # When setting a manual date, unmark and add the date
+    # to the title above, and unmark the timefmt line to stop
+    # Calibre from adding today's date in addition.
+
+    # timefmt = ''
+    RETRIEVEDATE = strftime('%Y%m%d')
+    INDEX_MAIN = 'http://www.volkskrant.nl/vk-online/VK/' + RETRIEVEDATE + '___/VKN01_001/#text'
+    INDEX_ARTICLE = 'http://www.volkskrant.nl/vk-online/VK/' + RETRIEVEDATE + '___/VKN01_001/'
+    LOGIN = 'http://www.volkskrant.nl/vk/user/loggedIn.do'
+    remove_tags = [dict(name='address')]
+    cover_url = 'http://www.volkskrant.nl/vk-online/VK/' + RETRIEVEDATE + '___/VKN01_001/page.jpg'
+
+    def get_browser(self):
+        br = BasicNewsRecipe.get_browser()
+
+        if self.username is not None and self.password is not None:
+           br.open(self.LOGIN)
+           br.select_form(nr = 0)
+           br['username'] = self.username
+           br['password'] = self.password
+           br.submit()
+        return br
+
+    def parse_index(self):
+        krant = []
+        def strip_title(_title):
+            i = 0
+            while ((_title[i] <> ":") and (i <= len(_title))):
+               i = i + 1
+            return(_title[0:i])
+        for temp in range (5):
+              try:
+                soup = self.index_to_soup(self.INDEX_MAIN)
+                break
+              except:
+                #print '(Retrying main index load)'
+                continue
+        mainsoup = soup.find('td', attrs={'id': 'select_page_top'})
+        for option in mainsoup.findAll('option'):
+           articles = []
+           _INDEX = 'http://www.volkskrant.nl/vk-online/VK/' + self.RETRIEVEDATE + '___/' + option['value'] + '/#text'
+           _INDEX_ARTICLE = 'http://www.volkskrant.nl/vk-online/VK/' + self.RETRIEVEDATE + '___/' + option['value'] + '/'
+           #print ''
+           #print '<-------    Processing section: ' + _INDEX + ' ------------------------->'
+           for temp in range (5):
+              try:
+                soup = self.index_to_soup(_INDEX)
+                break
+              except:
+                #print '(Retrying index load)'
+                continue
+           for item in soup.findAll('area'):
+              art_nr = item['class']
+              attrname = art_nr[0:12] + '_section' + option['value'][0:5] + '_' + art_nr[26:len(art_nr)]
+              #print '==> Found: ' + attrname;
+              index_title = soup.find('div', attrs={'class': attrname})
+              get_title = index_title['title'];
+              _ARTICLE   = _INDEX_ARTICLE + attrname + '.html#text'
+              title = get_title;
+              #print '--> Title: ' + title;
+              #print '--> URL: ' + _ARTICLE;
+              for temp in range (5):
+                 try:
+                   souparticle =  self.index_to_soup(_ARTICLE);
+                   break
+                 except:
+                   print '(Retrying URL load)'
+                   continue
+              headerurl = souparticle.findAll('frame')[0]['src'];
+              #print '--> Read frame name for header: ' + headerurl;
+              url = _INDEX_ARTICLE + headerurl[0:len(headerurl)-12] + '_text.html';
+              #print '--> Corrected URL: ' + url;
+              if (get_title <> ''):
+                 title = strip_title(get_title)
+                 date  = strftime(' %B %Y')
+              if (title <> ''):
+                 articles.append({
+                                         'title'      :title
+                                        ,'date'       :date
+                                        ,'url'        :url
+                                        ,'description':''
+                                        })
+           krant.append( (option.string, articles))
+        return krant
+

resources/default_tweaks.py

@@ -118,6 +118,7 @@ sort_columns_at_startup = None
 # timestamp default if not set: dd MMM yyyy
 gui_pubdate_display_format = 'MMM yyyy'
 gui_timestamp_display_format = 'dd MMM yyyy'
+gui_last_modified_display_format = 'dd MMM yyyy'
 
 #: Control sorting of titles and series in the library display
 # Control title and series sorting in the library view. If set to
@@ -266,26 +267,6 @@ max_content_server_tags_shown=5
 content_server_will_display = ['*']
 content_server_wont_display = []
 
-#: Set custom metadata fields that the book details panel will or will not display.
-# book_details_will_display is a list of custom fields to be displayed.
-# book_details_wont_display is a list of custom fields not to be displayed.
-# wont_display has priority over will_display.
-# The special value '*' means all custom fields. The value [] means no entries.
-# Defaults:
-#    book_details_will_display = ['*']
-#    book_details_wont_display = []
-# Examples:
-# To display only the custom fields #mytags and #genre:
-#   book_details_will_display = ['#mytags', '#genre']
-#   book_details_wont_display = []
-# To display all fields except #mycomments:
-#   book_details_will_display = ['*']
-#   book_details_wont_display['#mycomments']
-# As above, this tweak affects only display of custom fields. The standard
-# fields are not affected
-book_details_will_display = ['*']
-book_details_wont_display = []
-
 #: Set the maximum number of sort 'levels'
 # Set the maximum number of sort 'levels' that calibre will use to resort the
 # library after certain operations such as searches or device insertion. Each

resources/metadata_sqlite.sql

@@ -7,17 +7,30 @@ CREATE TABLE books ( id      INTEGER PRIMARY KEY AUTOINCREMENT,
                              title     TEXT NOT NULL DEFAULT 'Unknown' COLLATE NOCASE,
                              sort      TEXT COLLATE NOCASE,
                              timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-                             uri       TEXT,
-                             series_index INTEGER NOT NULL DEFAULT 1,
+                             pubdate   TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                             series_index REAL NOT NULL DEFAULT 1.0,
                              author_sort TEXT COLLATE NOCASE,
                              isbn TEXT DEFAULT "" COLLATE NOCASE,
-                             path TEXT NOT NULL DEFAULT ""
-                        );
+                             lccn TEXT DEFAULT "" COLLATE NOCASE,
+                             path TEXT NOT NULL DEFAULT "",
+                             flags INTEGER NOT NULL DEFAULT 1
+                        , uuid TEXT, has_cover BOOL DEFAULT 0, last_modified TIMESTAMP NOT NULL DEFAULT "2000-01-01 00:00:00+00:00");
 CREATE TABLE books_authors_link ( id INTEGER PRIMARY KEY,
                                           book INTEGER NOT NULL,
                                           author INTEGER NOT NULL,
                                           UNIQUE(book, author)
                                         );
+CREATE TABLE books_languages_link ( id INTEGER PRIMARY KEY,
+                                            book INTEGER NOT NULL,
+                                            lang_code INTEGER NOT NULL,
+                                            item_order INTEGER NOT NULL DEFAULT 0,
+                                            UNIQUE(book, lang_code)
+        );
+CREATE TABLE books_plugin_data(id INTEGER PRIMARY KEY,
+                                     book INTEGER NON NULL,
+                                     name TEXT NON NULL,
+                                     val TEXT NON NULL,
+                                     UNIQUE(book,name));
 CREATE TABLE books_publishers_link ( id INTEGER PRIMARY KEY,
                                           book INTEGER NOT NULL,
                                           publisher INTEGER NOT NULL,
@@ -49,11 +62,51 @@ CREATE TABLE conversion_options ( id INTEGER PRIMARY KEY,
                                           data BLOB NOT NULL,
                                           UNIQUE(format,book)
                                         );
+CREATE TABLE custom_columns (
+                    id       INTEGER PRIMARY KEY AUTOINCREMENT,
+                    label    TEXT NOT NULL,
+                    name     TEXT NOT NULL,
+                    datatype TEXT NOT NULL,
+                    mark_for_delete   BOOL DEFAULT 0 NOT NULL,
+                    editable BOOL DEFAULT 1 NOT NULL,
+                    display  TEXT DEFAULT "{}" NOT NULL,
+                    is_multiple BOOL DEFAULT 0 NOT NULL,
+                    normalized BOOL NOT NULL,
+                    UNIQUE(label)
+                );
+CREATE TABLE data ( id     INTEGER PRIMARY KEY,
+                            book   INTEGER NON NULL,
+                            format TEXT NON NULL COLLATE NOCASE,
+                            uncompressed_size INTEGER NON NULL,
+                            name TEXT NON NULL,
+                            UNIQUE(book, format)
+);
 CREATE TABLE feeds ( id   INTEGER PRIMARY KEY,
                               title TEXT NOT NULL,
                               script TEXT NOT NULL,
                               UNIQUE(title)
                              );
+CREATE TABLE identifiers  ( id     INTEGER PRIMARY KEY,
+                                    book   INTEGER NON NULL,
+                                    type   TEXT NON NULL DEFAULT "isbn" COLLATE NOCASE,
+                                    val    TEXT NON NULL COLLATE NOCASE,
+                                    UNIQUE(book, type)
+        );
+CREATE TABLE languages    ( id        INTEGER PRIMARY KEY,
+                                    lang_code TEXT NON NULL COLLATE NOCASE,
+                                    UNIQUE(lang_code)
+        );
+CREATE TABLE library_id ( id   INTEGER PRIMARY KEY,
+                                  uuid TEXT NOT NULL,
+                                  UNIQUE(uuid)
+        );
+CREATE TABLE metadata_dirtied(id INTEGER PRIMARY KEY,
+                             book INTEGER NOT NULL,
+                             UNIQUE(book));
+CREATE TABLE preferences(id INTEGER PRIMARY KEY,
+                                 key TEXT NON NULL,
+                                 val TEXT NON NULL,
+                                 UNIQUE(key));
 CREATE TABLE publishers ( id   INTEGER PRIMARY KEY,
                                   name TEXT NOT NULL COLLATE NOCASE,
                                   sort TEXT COLLATE NOCASE,
@@ -72,34 +125,143 @@ CREATE TABLE tags ( id   INTEGER PRIMARY KEY,
                             name TEXT NOT NULL COLLATE NOCASE,
                             UNIQUE (name)
                              );
-CREATE TABLE data ( id     INTEGER PRIMARY KEY,
-                            book   INTEGER NON NULL,
-                            format TEXT NON NULL COLLATE NOCASE,
-                            uncompressed_size INTEGER NON NULL,
-                            name TEXT NON NULL,
-                            UNIQUE(book, format)
-);
-
 CREATE VIEW meta AS
-    SELECT id, title,
-           (SELECT concat(name) FROM authors WHERE authors.id IN (SELECT author from books_authors_link WHERE book=books.id)) authors,
-           (SELECT name FROM publishers WHERE publishers.id IN (SELECT publisher from books_publishers_link WHERE book=books.id)) publisher,
-           (SELECT rating FROM ratings WHERE ratings.id IN (SELECT rating from books_ratings_link WHERE book=books.id)) rating,
-           timestamp,
-           (SELECT MAX(uncompressed_size) FROM data WHERE book=books.id) size,
-           (SELECT concat(name) FROM tags WHERE tags.id IN (SELECT tag from books_tags_link WHERE book=books.id)) tags,
-           (SELECT text FROM comments WHERE book=books.id) comments,
-           (SELECT name FROM series WHERE series.id IN (SELECT series FROM books_series_link WHERE book=books.id)) series,
-           series_index,
-           sort,
-           author_sort,
-           (SELECT concat(format) FROM data WHERE data.book=books.id) formats,
-           isbn
-    FROM books;
+        SELECT id, title,
+               (SELECT sortconcat(bal.id, name) FROM books_authors_link AS bal JOIN authors ON(author = authors.id) WHERE book = books.id) authors,
+               (SELECT name FROM publishers WHERE publishers.id IN (SELECT publisher from books_publishers_link WHERE book=books.id)) publisher,
+               (SELECT rating FROM ratings WHERE ratings.id IN (SELECT rating from books_ratings_link WHERE book=books.id)) rating,
+               timestamp,
+               (SELECT MAX(uncompressed_size) FROM data WHERE book=books.id) size,
+               (SELECT concat(name) FROM tags WHERE tags.id IN (SELECT tag from books_tags_link WHERE book=books.id)) tags,
+               (SELECT text FROM comments WHERE book=books.id) comments,
+               (SELECT name FROM series WHERE series.id IN (SELECT series FROM books_series_link WHERE book=books.id)) series,
+               series_index,
+               sort,
+               author_sort,
+               (SELECT concat(format) FROM data WHERE data.book=books.id) formats,
+               isbn,
+               path,
+               lccn,
+               pubdate,
+               flags,
+               uuid
+        FROM books;
+CREATE VIEW tag_browser_authors AS SELECT
+                    id,
+                    name,
+                    (SELECT COUNT(id) FROM books_authors_link WHERE author=authors.id) count,
+                    (SELECT AVG(ratings.rating)
+                     FROM books_authors_link AS tl, books_ratings_link AS bl, ratings
+                     WHERE tl.author=authors.id AND bl.book=tl.book AND
+                     ratings.id = bl.rating AND ratings.rating <> 0) avg_rating,
+                     sort AS sort
+                FROM authors;
+CREATE VIEW tag_browser_filtered_authors AS SELECT
+                    id,
+                    name,
+                    (SELECT COUNT(books_authors_link.id) FROM books_authors_link WHERE
+                        author=authors.id AND books_list_filter(book)) count,
+                    (SELECT AVG(ratings.rating)
+                     FROM books_authors_link AS tl, books_ratings_link AS bl, ratings
+                     WHERE tl.author=authors.id AND bl.book=tl.book AND
+                     ratings.id = bl.rating AND ratings.rating <> 0 AND
+                     books_list_filter(bl.book)) avg_rating,
+                     sort AS sort
+                FROM authors;
+CREATE VIEW tag_browser_filtered_publishers AS SELECT
+                    id,
+                    name,
+                    (SELECT COUNT(books_publishers_link.id) FROM books_publishers_link WHERE
+                        publisher=publishers.id AND books_list_filter(book)) count,
+                    (SELECT AVG(ratings.rating)
+                     FROM books_publishers_link AS tl, books_ratings_link AS bl, ratings
+                     WHERE tl.publisher=publishers.id AND bl.book=tl.book AND
+                     ratings.id = bl.rating AND ratings.rating <> 0 AND
+                     books_list_filter(bl.book)) avg_rating,
+                     name AS sort
+                FROM publishers;
+CREATE VIEW tag_browser_filtered_ratings AS SELECT
+                    id,
+                    rating,
+                    (SELECT COUNT(books_ratings_link.id) FROM books_ratings_link WHERE
+                        rating=ratings.id AND books_list_filter(book)) count,
+                    (SELECT AVG(ratings.rating)
+                     FROM books_ratings_link AS tl, books_ratings_link AS bl, ratings
+                     WHERE tl.rating=ratings.id AND bl.book=tl.book AND
+                     ratings.id = bl.rating AND ratings.rating <> 0 AND
+                     books_list_filter(bl.book)) avg_rating,
+                     rating AS sort
+                FROM ratings;
+CREATE VIEW tag_browser_filtered_series AS SELECT
+                    id,
+                    name,
+                    (SELECT COUNT(books_series_link.id) FROM books_series_link WHERE
+                        series=series.id AND books_list_filter(book)) count,
+                    (SELECT AVG(ratings.rating)
+                     FROM books_series_link AS tl, books_ratings_link AS bl, ratings
+                     WHERE tl.series=series.id AND bl.book=tl.book AND
+                     ratings.id = bl.rating AND ratings.rating <> 0 AND
+                     books_list_filter(bl.book)) avg_rating,
+                     (title_sort(name)) AS sort
+                FROM series;
+CREATE VIEW tag_browser_filtered_tags AS SELECT
+                    id,
+                    name,
+                    (SELECT COUNT(books_tags_link.id) FROM books_tags_link WHERE
+                        tag=tags.id AND books_list_filter(book)) count,
+                    (SELECT AVG(ratings.rating)
+                     FROM books_tags_link AS tl, books_ratings_link AS bl, ratings
+                     WHERE tl.tag=tags.id AND bl.book=tl.book AND
+                     ratings.id = bl.rating AND ratings.rating <> 0 AND
+                     books_list_filter(bl.book)) avg_rating,
+                     name AS sort
+                FROM tags;
+CREATE VIEW tag_browser_publishers AS SELECT
+                    id,
+                    name,
+                    (SELECT COUNT(id) FROM books_publishers_link WHERE publisher=publishers.id) count,
+                    (SELECT AVG(ratings.rating)
+                     FROM books_publishers_link AS tl, books_ratings_link AS bl, ratings
+                     WHERE tl.publisher=publishers.id AND bl.book=tl.book AND
+                     ratings.id = bl.rating AND ratings.rating <> 0) avg_rating,
+                     name AS sort
+                FROM publishers;
+CREATE VIEW tag_browser_ratings AS SELECT
+                    id,
+                    rating,
+                    (SELECT COUNT(id) FROM books_ratings_link WHERE rating=ratings.id) count,
+                    (SELECT AVG(ratings.rating)
+                     FROM books_ratings_link AS tl, books_ratings_link AS bl, ratings
+                     WHERE tl.rating=ratings.id AND bl.book=tl.book AND
+                     ratings.id = bl.rating AND ratings.rating <> 0) avg_rating,
+                     rating AS sort
+                FROM ratings;
+CREATE VIEW tag_browser_series AS SELECT
+                    id,
+                    name,
+                    (SELECT COUNT(id) FROM books_series_link WHERE series=series.id) count,
+                    (SELECT AVG(ratings.rating)
+                     FROM books_series_link AS tl, books_ratings_link AS bl, ratings
+                     WHERE tl.series=series.id AND bl.book=tl.book AND
+                     ratings.id = bl.rating AND ratings.rating <> 0) avg_rating,
+                     (title_sort(name)) AS sort
+                FROM series;
+CREATE VIEW tag_browser_tags AS SELECT
+                    id,
+                    name,
+                    (SELECT COUNT(id) FROM books_tags_link WHERE tag=tags.id) count,
+                    (SELECT AVG(ratings.rating)
+                     FROM books_tags_link AS tl, books_ratings_link AS bl, ratings
+                     WHERE tl.tag=tags.id AND bl.book=tl.book AND
+                     ratings.id = bl.rating AND ratings.rating <> 0) avg_rating,
+                     name AS sort
+                FROM tags;
 CREATE INDEX authors_idx ON books (author_sort COLLATE NOCASE);
 CREATE INDEX books_authors_link_aidx ON books_authors_link (author);
 CREATE INDEX books_authors_link_bidx ON books_authors_link (book);
 CREATE INDEX books_idx ON books (sort COLLATE NOCASE);
+CREATE INDEX books_languages_link_aidx ON books_languages_link (lang_code);
+CREATE INDEX books_languages_link_bidx ON books_languages_link (book);
 CREATE INDEX books_publishers_link_aidx ON books_publishers_link (publisher);
 CREATE INDEX books_publishers_link_bidx ON books_publishers_link (book);
 CREATE INDEX books_ratings_link_aidx ON books_ratings_link (rating);
@@ -111,32 +273,38 @@ CREATE INDEX books_tags_link_bidx ON books_tags_link (book);
 CREATE INDEX comments_idx ON comments (book);
 CREATE INDEX conversion_options_idx_a ON conversion_options (format COLLATE NOCASE);
 CREATE INDEX conversion_options_idx_b ON conversion_options (book);
+CREATE INDEX custom_columns_idx ON custom_columns (label);
 CREATE INDEX data_idx ON data (book);
+CREATE INDEX formats_idx ON data (format);
+CREATE INDEX languages_idx ON languages (lang_code COLLATE NOCASE);
 CREATE INDEX publishers_idx ON publishers (name COLLATE NOCASE);
-CREATE INDEX series_idx ON series (sort COLLATE NOCASE);
+CREATE INDEX series_idx ON series (name COLLATE NOCASE);
 CREATE INDEX tags_idx ON tags (name COLLATE NOCASE);
 CREATE TRIGGER books_delete_trg
-        AFTER DELETE ON books
-        BEGIN
-            DELETE FROM books_authors_link WHERE book=OLD.id;
-            DELETE FROM books_publishers_link WHERE book=OLD.id;
-            DELETE FROM books_ratings_link WHERE book=OLD.id;
-            DELETE FROM books_series_link WHERE book=OLD.id;
-            DELETE FROM books_tags_link WHERE book=OLD.id;
-            DELETE FROM data WHERE book=OLD.id;
-            DELETE FROM comments WHERE book=OLD.id;
-            DELETE FROM conversion_options WHERE book=OLD.id;
+            AFTER DELETE ON books
+            BEGIN
+                DELETE FROM books_authors_link WHERE book=OLD.id;
+                DELETE FROM books_publishers_link WHERE book=OLD.id;
+                DELETE FROM books_ratings_link WHERE book=OLD.id;
+                DELETE FROM books_series_link WHERE book=OLD.id;
+                DELETE FROM books_tags_link WHERE book=OLD.id;
+                DELETE FROM books_languages_link WHERE book=OLD.id;
+                DELETE FROM data WHERE book=OLD.id;
+                DELETE FROM comments WHERE book=OLD.id;
+                DELETE FROM conversion_options WHERE book=OLD.id;
+                DELETE FROM books_plugin_data WHERE book=OLD.id;
+                DELETE FROM identifiers WHERE book=OLD.id;
         END;
-CREATE TRIGGER books_insert_trg
-        AFTER INSERT ON books
+CREATE TRIGGER books_insert_trg AFTER INSERT ON books
         BEGIN
-          UPDATE books SET sort=title_sort(NEW.title) WHERE id=NEW.id;
+            UPDATE books SET sort=title_sort(NEW.title),uuid=uuid4() WHERE id=NEW.id;
         END;
 CREATE TRIGGER books_update_trg
-        AFTER UPDATE ON books
-        BEGIN
-          UPDATE books SET sort=title_sort(NEW.title) WHERE id=NEW.id;
-        END;
+            AFTER UPDATE ON books
+            BEGIN
+            UPDATE books SET sort=title_sort(NEW.title)
+                         WHERE id=NEW.id AND OLD.title <> NEW.title;
+            END;
 CREATE TRIGGER fkc_comments_insert
         BEFORE INSERT ON comments
         BEGIN
@@ -169,23 +337,41 @@ CREATE TRIGGER fkc_data_update
                 THEN RAISE(ABORT, 'Foreign key violation: book not in books')
             END;
         END;
-CREATE TRIGGER fkc_delete_books_authors_link
+CREATE TRIGGER fkc_delete_on_authors
         BEFORE DELETE ON authors
         BEGIN
             SELECT CASE
-                WHEN (SELECT COUNT(id) FROM books_authors_link WHERE book=OLD.book) > 0
-                THEN RAISE(ABORT, 'Foreign key violation: author is still referenced')
+                WHEN (SELECT COUNT(id) FROM books_authors_link WHERE author=OLD.id) > 0
+                THEN RAISE(ABORT, 'Foreign key violation: authors is still referenced')
             END;
         END;
-CREATE TRIGGER fkc_delete_books_publishers_link
+CREATE TRIGGER fkc_delete_on_languages
+        BEFORE DELETE ON languages
+        BEGIN
+            SELECT CASE
+                WHEN (SELECT COUNT(id) FROM books_languages_link WHERE lang_code=OLD.id) > 0
+                THEN RAISE(ABORT, 'Foreign key violation: language is still referenced')
+            END;
+        END;
+CREATE TRIGGER fkc_delete_on_languages_link
+        BEFORE INSERT ON books_languages_link
+        BEGIN
+          SELECT CASE
+              WHEN (SELECT id from books WHERE id=NEW.book) IS NULL
+              THEN RAISE(ABORT, 'Foreign key violation: book not in books')
+              WHEN (SELECT id from languages WHERE id=NEW.lang_code) IS NULL
+              THEN RAISE(ABORT, 'Foreign key violation: lang_code not in languages')
+          END;
+        END;
+CREATE TRIGGER fkc_delete_on_publishers
         BEFORE DELETE ON publishers
         BEGIN
             SELECT CASE
-                WHEN (SELECT COUNT(id) FROM books_publishers_link WHERE book=OLD.book) > 0
-                THEN RAISE(ABORT, 'Foreign key violation: publisher is still referenced')
+                WHEN (SELECT COUNT(id) FROM books_publishers_link WHERE publisher=OLD.id) > 0
+                THEN RAISE(ABORT, 'Foreign key violation: publishers is still referenced')
             END;
         END;
-CREATE TRIGGER fkc_delete_books_series_link
+CREATE TRIGGER fkc_delete_on_series
         BEFORE DELETE ON series
         BEGIN
             SELECT CASE
@@ -193,12 +379,12 @@ CREATE TRIGGER fkc_delete_books_series_link
                 THEN RAISE(ABORT, 'Foreign key violation: series is still referenced')
             END;
         END;
-CREATE TRIGGER fkc_delete_books_tags_link
+CREATE TRIGGER fkc_delete_on_tags
         BEFORE DELETE ON tags
         BEGIN
             SELECT CASE
                 WHEN (SELECT COUNT(id) FROM books_tags_link WHERE tag=OLD.id) > 0
-                THEN RAISE(ABORT, 'Foreign key violation: tag is still referenced')
+                THEN RAISE(ABORT, 'Foreign key violation: tags is still referenced')
             END;
         END;
 CREATE TRIGGER fkc_insert_books_authors_link
@@ -267,6 +453,22 @@ CREATE TRIGGER fkc_update_books_authors_link_b
                 THEN RAISE(ABORT, 'Foreign key violation: author not in authors')
             END;
         END;
+CREATE TRIGGER fkc_update_books_languages_link_a
+        BEFORE UPDATE OF book ON books_languages_link
+        BEGIN
+            SELECT CASE
+                WHEN (SELECT id from books WHERE id=NEW.book) IS NULL
+                THEN RAISE(ABORT, 'Foreign key violation: book not in books')
+            END;
+        END;
+CREATE TRIGGER fkc_update_books_languages_link_b
+        BEFORE UPDATE OF lang_code ON books_languages_link
+        BEGIN
+            SELECT CASE
+                WHEN (SELECT id from languages WHERE id=NEW.lang_code) IS NULL
+                THEN RAISE(ABORT, 'Foreign key violation: lang_code not in languages')
+            END;
+        END;
 CREATE TRIGGER fkc_update_books_publishers_link_a
         BEFORE UPDATE OF book ON books_publishers_link
         BEGIN
@@ -341,3 +543,4 @@ CREATE TRIGGER series_update_trg
         BEGIN
           UPDATE series SET sort=NEW.name WHERE id=NEW.id;
         END;
+pragma user_version=20;

resources/templates/book_details.css

@@ -0,0 +1,46 @@
+a { 
+    text-decoration: none;
+    color: blue
+}
+
+a:hover {
+    color: red
+}
+
+.comments { 
+    margin-top: 0;
+    padding-top: 0;
+    text-indent: 0
+}
+
+table.fields { 
+    margin-bottom: 0;
+    padding-bottom: 0;
+}
+
+table.fields td { 
+    vertical-align: top
+}
+
+table.fields td.title {
+    font-weight: bold
+}
+
+.series_name {
+    font-style: italic
+}
+
+/*
+The HTML that this styleshhet applies to looks like this:
+
+<table class="fields">
+    <tr id="formats" class="datatype_text"><td class="title">Formats:</td><td><a href="format:572:EPUB">EPUB</a>, <a href="format:572:LIT">LIT</a></td></tr>
+    <tr id="series" class="datatype_series"><td class="title">Series:</td><td>Book II of <span class="series_name">The Sea Beggars</span></td></tr>
+    <tr id="tags" class="datatype_text"><td class="title">Tags:</td><td>Fantasy, Fiction</td></tr>
+    <tr id="path" class="datatype_text"><td class="title">Path:</td><td><a href="path:572" title="/home/kovid/test library/Paul Kearney/This Forsaken Earth (572)">Click to open</a></td></tr>
+</table>
+
+<div id="comments" class="comments"><h3>From Publishers Weekly</h3><p>At the start of Kearney's rousing sequel to <em>The Mark of Ran</em> (2005), Rol Cortishane, the youthful captain of the privateer <em>Revenant</em>, captures a slaver and frees its chained slaves. Back in the harbor of Ganesh Ka in the land of Umer, Rol encounters an untrustworthy acquaintance he hasn't seen in years, Canker, a former king of thieves, who urges Rol to join in the fight to save Rowen, a darkly beautiful queen, whose throne is at risk in mountainous Bionar. That Rowen is Rol's half-sister for whom he has lusted in the past doesn't make Rol's decision to help an easy one. If as in <em>The Mark of Ran</em> the action is more lively at sea than on land, Kearney's solid storytelling and nautical detail worthy of C.S. Forester or Patrick O'Brian will keep readers turning the pages. <em>(Dec.)</em> <br />Copyright © Reed Business Information, a division of Reed Elsevier Inc. All rights reserved. </p><h3>From</h3><p>The sequel to <em>The Mark of Ran</em> (2005) finds heroic young Rol Cortishane grown to be a much-feared sea captain. Deciding to ignore his mysterious past, he spends his energy on ship and crew. He is still an outlaw, however, and the only port he can call home is Ganesh Ka, the endangered city of exiles. When word comes from Rowan, his half-sister, asking him to fight on her behalf, he must weigh the safety of Ganesh Ka against Rowan's treachery in the past. Finally persuaded to aid Rowan, he learns more of betrayal and his heritage in the ensuing battles than he had wanted to know. Kearney's characters are much better developed here than they were in <em>The Mark of Ran</em>, and since the book tells a single story, the plot is tighter. Moreover, because almost all the action transpires in the here and now, the sequel can be read without reference to the predecessor. Since it ends hanging on a particularly bloody cliff, expect to see more of Kearney's excellent maritime fantasy. <em>Frieda Murray</em><br /><em>Copyright © American Library Association. All rights reserved</em></p>
+</div>
+*/
+

setup/installer/windows/notes.rst

@@ -69,7 +69,24 @@ nmake -f ms\ntdll.mak install
 Qt
 --------
 
-Extract Qt sourcecode to C:\Qt\4.x.x. Run configure and make::
+Extract Qt sourcecode to C:\Qt\4.x.x. 
+
+Qt uses its own routine to locate and load "system libraries" including the openssl libraries needed for "Get Books". This means that we have to apply the following patch to have Qt load the openssl libraries bundled with calibre:
+
+
+--- src/corelib/plugin/qsystemlibrary.cpp	2011-02-22 05:04:00.000000000 -0700
++++ src/corelib/plugin/qsystemlibrary.cpp	2011-04-25 20:53:13.635247466 -0600
+@@ -110,7 +110,7 @@ HINSTANCE QSystemLibrary::load(const wch
+ 
+ #if !defined(QT_BOOTSTRAPPED)
+     if (!onlySystemDirectory)
+-        searchOrder << QFileInfo(qAppFileName()).path();
++        searchOrder << (QFileInfo(qAppFileName()).path().replace(QLatin1Char('/'), QLatin1Char('\\')) + QString::fromLatin1("\\DLLs\\"));
+ #endif
+     searchOrder << qSystemDirectory();
+ 
+
+Now, run configure and make::
 
     configure -opensource -release -qt-zlib -qt-gif -qt-libmng -qt-libpng -qt-libtiff -qt-libjpeg -release -platform win32-msvc2008 -no-qt3support -webkit -xmlpatterns -no-phonon -no-style-plastique -no-style-cleanlooks -no-style-motif -no-style-cde -no-declarative -no-scripttools -no-audio-backend -no-multimedia -no-dbus -no-openvg -no-opengl -no-qt3support -confirm-license -nomake examples -nomake demos -nomake docs -openssl -I Q:\openssl\include -L Q:\openssl\lib && nmake
 

setup/installer/windows/wix-template.xml

@@ -11,7 +11,10 @@
 						SummaryCodepage='1252' />
 
                 <Media Id="1" Cabinet="{app}.cab" CompressionLevel="{compression}" EmbedCab="yes" />
-                
+                <!-- The following line is needed because of the patch to QtCore4.dll. You can remove this line
+                     after you update Qt beyond 4.7.2. 'emus' means re-install even if version is the same not just if it is older. -->
+                <Property Id='REINSTALLMODE' Value='emus'/>
+
                 <Upgrade Id="{upgrade_code}">
                     <UpgradeVersion Maximum="{version}"
                        IncludeMaximum="yes"

setup/upload.py

@@ -347,9 +347,10 @@ class UploadUserManual(Command): # {{{
         with NamedTemporaryFile(suffix='.zip') as f:
             os.fchmod(f.fileno(),
                 stat.S_IRUSR|stat.S_IRGRP|stat.S_IROTH|stat.S_IWRITE)
-            with CurrentDir(self.d(path)):
+            with CurrentDir(path):
                 with ZipFile(f, 'w') as zf:
                     for x in os.listdir('.'):
+                        if x.endswith('.swp'): continue
                         zf.write(x)
                         if os.path.isdir(x):
                             for y in os.listdir(x):

src/calibre/__init__.py

@@ -388,7 +388,11 @@ class CurrentDir(object):
         return self.cwd
 
     def __exit__(self, *args):
-        os.chdir(self.cwd)
+        try:
+            os.chdir(self.cwd)
+        except:
+            # The previous CWD no longer exists
+            pass
 
 
 class StreamReadWrapper(object):

src/calibre/constants.py

@@ -4,7 +4,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = u'calibre'
-numeric_version = (0, 7, 57)
+numeric_version = (0, 7, 58)
 __version__   = u'.'.join(map(unicode, numeric_version))
 __author__    = u"Kovid Goyal <kovid@kovidgoyal.net>"
 

src/calibre/customize/builtins.py

@@ -1109,6 +1109,11 @@ class StoreAmazonKindleStore(StoreBase):
     description = _('Kindle books from Amazon')
     actual_plugin = 'calibre.gui2.store.amazon_plugin:AmazonKindleStore'
 
+class StoreAmazonUKKindleStore(StoreBase):
+    name = 'Amazon UK Kindle'
+    description = _('Kindle books from Amazon.uk')
+    actual_plugin = 'calibre.gui2.store.amazon_uk_plugin:AmazonUKKindleStore'
+
 class StoreBaenWebScriptionStore(StoreBase):
     name = 'Baen WebScription'
     description = _('Ebooks for readers.')
@@ -1174,10 +1179,27 @@ class StoreSmashwordsStore(StoreBase):
     description = _('Your ebook. Your way.')
     actual_plugin = 'calibre.gui2.store.smashwords_plugin:SmashwordsStore'
 
-plugins += [StoreAmazonKindleStore, StoreBaenWebScriptionStore, StoreBNStore,
+class StoreWaterstonesUKStore(StoreBase):
+    name = 'Waterstones UK'
+    description = _('Feel every word')
+    actual_plugin = 'calibre.gui2.store.waterstones_uk_plugin:WaterstonesUKStore'
+
+class StoreFoylesUKStore(StoreBase):
+    name = 'Foyles UK'
+    description = _('Foyles of London, online')
+    actual_plugin = 'calibre.gui2.store.foyles_uk_plugin:FoylesUKStore'
+
+class AmazonDEKindleStore(StoreBase):
+    name = 'Amazon DE Kindle'
+    description = _('Kindle eBooks')
+    actual_plugin = 'calibre.gui2.store.amazon_de_plugin:AmazonDEKindleStore'
+
+plugins += [StoreAmazonKindleStore, AmazonDEKindleStore, StoreAmazonUKKindleStore,
+    StoreBaenWebScriptionStore, StoreBNStore,
     StoreBeWriteStore, StoreDieselEbooksStore, StoreEbookscomStore,
-    StoreEHarlequinStoretore,
-    StoreFeedbooksStore, StoreGutenbergStore, StoreKoboStore, StoreManyBooksStore,
-    StoreMobileReadStore, StoreOpenLibraryStore, StoreSmashwordsStore]
+    StoreEHarlequinStoretore, StoreFeedbooksStore,
+    StoreFoylesUKStore, StoreGutenbergStore, StoreKoboStore, StoreManyBooksStore,
+    StoreMobileReadStore, StoreOpenLibraryStore, StoreSmashwordsStore,
+    StoreWaterstonesUKStore]
 
 # }}}

src/calibre/devices/android/driver.py

@@ -109,10 +109,10 @@ class ANDROID(USBMS):
             'SGH-T849', '_MB300', 'A70S', 'S_ANDROID', 'A101IT', 'A70H',
             'IDEOS_TABLET', 'MYTOUCH_4G', 'UMS_COMPOSITE', 'SCH-I800_CARD',
             '7', 'A956', 'A955', 'A43', 'ANDROID_PLATFORM', 'TEGRA_2',
-            'MB860', 'MULTI-CARD', 'MID7015A', 'INCREDIBLE']
+            'MB860', 'MULTI-CARD', 'MID7015A', 'INCREDIBLE', 'A7EB']
     WINDOWS_CARD_A_MEM = ['ANDROID_PHONE', 'GT-I9000_CARD', 'SGH-I897',
             'FILE-STOR_GADGET', 'SGH-T959', 'SAMSUNG_ANDROID', 'GT-P1000_CARD',
-            'A70S', 'A101IT', '7', 'INCREDIBLE']
+            'A70S', 'A101IT', '7', 'INCREDIBLE', 'A7EB']
 
     OSX_MAIN_MEM = 'Android Device Main Memory'
 

src/calibre/devices/apple/driver.py

@@ -460,7 +460,7 @@ class ITUNES(DriverBase):
 
                             cached_books[this_book.path] = {
                              'title':book.Name,
-                             'author':book.artist().split(' & '),
+                             'author':book.Artist.split(' & '),
                              'lib_book':library_books[this_book.path] if this_book.path in library_books else None,
                              'uuid': book.Composer,
                              'format': 'pdf' if book.KindAsString.startswith('PDF') else 'epub'

src/calibre/ebooks/chm/input.py

@@ -19,12 +19,12 @@ class CHMInput(InputFormatPlugin):
     description = 'Convert CHM files to OEB'
     file_types  = set(['chm'])
 
-    def _chmtohtml(self, output_dir, chm_path, no_images, log):
+    def _chmtohtml(self, output_dir, chm_path, no_images, log, debug_dump=False):
         from calibre.ebooks.chm.reader import CHMReader
         log.debug('Opening CHM file')
         rdr = CHMReader(chm_path, log, self.opts)
         log.debug('Extracting CHM to %s' % output_dir)
-        rdr.extract_content(output_dir)
+        rdr.extract_content(output_dir, debug_dump=debug_dump)
         self._chm_reader = rdr
         return rdr.hhc_path
 
@@ -47,7 +47,12 @@ class CHMInput(InputFormatPlugin):
             stream.close()
             log.debug('tdir=%s' % tdir)
             log.debug('stream.name=%s' % stream.name)
-            mainname = self._chmtohtml(tdir, chm_name, no_images, log)
+            debug_dump = False
+            odi = options.debug_pipeline
+            if odi:
+                debug_dump = os.path.join(odi, 'input')
+            mainname = self._chmtohtml(tdir, chm_name, no_images, log,
+                    debug_dump=debug_dump)
             mainpath = os.path.join(tdir, mainname)
 
             metadata = get_metadata_from_reader(self._chm_reader)
@@ -56,7 +61,6 @@ class CHMInput(InputFormatPlugin):
             #from calibre import ipython
             #ipython()
 
-            odi = options.debug_pipeline
             options.debug_pipeline = None
             options.input_encoding = 'utf-8'
             # try a custom conversion:

src/calibre/ebooks/chm/reader.py

@@ -97,7 +97,7 @@ class CHMReader(CHMFile):
             raise CHMError("'%s' is zero bytes in length!"%(path,))
         return data
 
-    def ExtractFiles(self, output_dir=os.getcwdu()):
+    def ExtractFiles(self, output_dir=os.getcwdu(), debug_dump=False):
         html_files = set([])
         for path in self.Contents():
             lpath = os.path.join(output_dir, path)
@@ -123,6 +123,9 @@ class CHMReader(CHMFile):
                     self.log.warn('%r filename too long, skipping'%path)
                     continue
                 raise
+        if debug_dump:
+            import shutil
+            shutil.copytree(output_dir, os.path.join(debug_dump, 'debug_dump'))
         for lpath in html_files:
             with open(lpath, 'r+b') as f:
                 data = f.read()
@@ -249,8 +252,8 @@ class CHMReader(CHMFile):
         if not os.path.isdir(dir):
             os.makedirs(dir)
 
-    def extract_content(self, output_dir=os.getcwdu()):
-        self.ExtractFiles(output_dir=output_dir)
+    def extract_content(self, output_dir=os.getcwdu(), debug_dump=False):
+        self.ExtractFiles(output_dir=output_dir, debug_dump=debug_dump)
 
 
 

src/calibre/ebooks/conversion/plumber.py

@@ -854,7 +854,8 @@ OptionRecommendation(name='sr3_replace',
         if isinstance(ret, basestring):
             shutil.copytree(output_dir, out_dir)
         else:
-            os.makedirs(out_dir)
+            if not os.path.exists(out_dir):
+                os.makedirs(out_dir)
             self.dump_oeb(ret, out_dir)
         if self.input_fmt == 'recipe':
             zf = ZipFile(os.path.join(self.opts.debug_pipeline,

src/calibre/ebooks/html/input.py

@@ -309,9 +309,9 @@ class HTMLInput(InputFormatPlugin):
 
     def create_oebbook(self, htmlpath, basedir, opts, log, mi):
         from calibre.ebooks.conversion.plumber import create_oebbook
-        from calibre.ebooks.oeb.base import DirContainer, \
-            rewrite_links, urlnormalize, urldefrag, BINARY_MIME, OEB_STYLES, \
-            xpath
+        from calibre.ebooks.oeb.base import (DirContainer,
+            rewrite_links, urlnormalize, urldefrag, BINARY_MIME, OEB_STYLES,
+            xpath)
         from calibre import guess_type
         from calibre.ebooks.oeb.transforms.metadata import \
             meta_info_to_oeb_metadata
@@ -345,7 +345,8 @@ class HTMLInput(InputFormatPlugin):
         htmlfile_map = {}
         for f in filelist:
             path = f.path
-            oeb.container = DirContainer(os.path.dirname(path), log)
+            oeb.container = DirContainer(os.path.dirname(path), log,
+                    ignore_opf=True)
             bname = os.path.basename(path)
             id, href = oeb.manifest.generate(id='html',
                     href=ascii_filename(bname))
@@ -369,7 +370,7 @@ class HTMLInput(InputFormatPlugin):
         for f in filelist:
             path = f.path
             dpath = os.path.dirname(path)
-            oeb.container = DirContainer(dpath, log)
+            oeb.container = DirContainer(dpath, log, ignore_opf=True)
             item = oeb.manifest.hrefs[htmlfile_map[path]]
             rewrite_links(item.data, partial(self.resource_adder, base=dpath))
 
@@ -409,7 +410,7 @@ class HTMLInput(InputFormatPlugin):
             if not item.linear: continue
             toc.add(title, item.href)
 
-        oeb.container = DirContainer(os.getcwdu(), oeb.log)
+        oeb.container = DirContainer(os.getcwdu(), oeb.log, ignore_opf=True)
         return oeb
 
     def link_to_local_path(self, link_, base=None):
@@ -456,7 +457,7 @@ class HTMLInput(InputFormatPlugin):
                     href=bhref)
             self.oeb.log.debug('Added', link)
             self.oeb.container = self.DirContainer(os.path.dirname(link),
-                    self.oeb.log)
+                    self.oeb.log, ignore_opf=True)
             # Load into memory
             guessed = self.guess_type(href)[0]
             media_type = guessed or self.BINARY_MIME

src/calibre/ebooks/metadata/book/base.py

@@ -19,6 +19,9 @@ from calibre.utils.date import isoformat, format_date
 from calibre.utils.icu import sort_key
 from calibre.utils.formatter import TemplateFormatter
 
+def human_readable(size, precision=2):
+    """ Convert a size in bytes into megabytes """
+    return ('%.'+str(precision)+'f'+ 'MB') % ((size/(1024.*1024.)),)
 
 NULL_VALUES = {
                 'user_metadata': {},
@@ -65,7 +68,19 @@ composite_formatter = SafeFormat()
 class Metadata(object):
 
     '''
-    A class representing all the metadata for a book.
+    A class representing all the metadata for a book. The various standard metadata
+    fields are available as attributes of this object. You can also stick
+    arbitrary attributes onto this object.
+
+    Metadata from custom columns should be accessed via the get() method,
+    passing in the lookup name for the column, for example: "#mytags".
+
+    Use the :meth:`is_null` method to test if a filed is null.
+
+    This object also has functions to format fields into strings.
+
+    The list of standard metadata fields grows with time is in
+    :data:`STANDARD_METADATA_FIELDS`.
 
     Please keep the method based API of this class to a minimum. Every method
     becomes a reserved field name.
@@ -85,11 +100,19 @@ class Metadata(object):
             if title:
                 self.title = title
             if authors:
-                #: List of strings or []
+                # List of strings or []
                 self.author = list(authors) if authors else []# Needed for backward compatibility
                 self.authors = list(authors) if authors else []
 
     def is_null(self, field):
+        '''
+        Return True if the value of filed is null in this object.
+        'null' means it is unknown or evaluates to False. So a title of
+        _('Unknown') is null or a language of 'und' is null.
+
+        Be careful with numeric fields since this will return True for zero as
+        well as None.
+        '''
         null_val = NULL_VALUES.get(field, None)
         val = getattr(self, field, None)
         return not val or val == null_val
@@ -117,7 +140,11 @@ class Metadata(object):
                                             _('TEMPLATE ERROR'),
                                             self).strip()
             return val
-
+        if field.startswith('#') and field.endswith('_index'):
+            try:
+                return self.get_extra(field[:-6])
+            except:
+                pass
         raise AttributeError(
                 'Metadata object has no attribute named: '+ repr(field))
 
@@ -167,11 +194,6 @@ class Metadata(object):
         try:
             return self.__getattribute__(field)
         except AttributeError:
-            if field.startswith('#') and field.endswith('_index'):
-                try:
-                    return self.get_extra(field[:-6])
-                except:
-                    pass
             return default
 
     def get_extra(self, field, default=None):
@@ -541,17 +563,25 @@ class Metadata(object):
     def format_tags(self):
         return u', '.join([unicode(t) for t in sorted(self.tags, key=sort_key)])
 
-    def format_rating(self):
-        return unicode(self.rating)
+    def format_rating(self, v=None, divide_by=1.0):
+        if v is None:
+            if self.rating is not None:
+                return unicode(self.rating/divide_by)
+            return u'None'
+        return unicode(v/divide_by)
 
     def format_field(self, key, series_with_index=True):
+        '''
+        Returns the tuple (display_name, formatted_value)
+        '''
         name, val, ign, ign = self.format_field_extended(key, series_with_index)
         return (name, val)
 
     def format_field_extended(self, key, series_with_index=True):
         from calibre.ebooks.metadata import authors_to_string
         '''
-        returns the tuple (field_name, formatted_value)
+        returns the tuple (display_name, formatted_value, original_value,
+        field_metadata)
         '''
 
         # Handle custom series index
@@ -627,11 +657,17 @@ class Metadata(object):
                 res = format_date(res, fmeta['display'].get('date_format','dd MMM yyyy'))
             elif datatype == 'rating':
                 res = res/2.0
+            elif key == 'size':
+                res = human_readable(res)
             return (name, unicode(res), orig_res, fmeta)
 
         return (None, None, None, None)
 
     def __unicode__(self):
+        '''
+        A string representation of this object, suitable for printing to
+        console
+        '''
         from calibre.ebooks.metadata import authors_to_string
         ans = []
         def fmt(x, y):
@@ -675,6 +711,9 @@ class Metadata(object):
         return u'\n'.join(ans)
 
     def to_html(self):
+        '''
+        A HTML representation of this object.
+        '''
         from calibre.ebooks.metadata import authors_to_string
         ans = [(_('Title'), unicode(self.title))]
         ans += [(_('Author(s)'), (authors_to_string(self.authors) if self.authors else _('Unknown')))]

src/calibre/ebooks/metadata/mobi.py

@@ -400,7 +400,8 @@ class MetadataUpdater(object):
         if getattr(self, 'exth', None) is None:
             raise MobiError('No existing EXTH record. Cannot update metadata.')
 
-        self.record0[92:96] = iana2mobi(mi.language)
+        if not mi.is_null('language'):
+            self.record0[92:96] = iana2mobi(mi.language)
         self.create_exth(exth=exth, new_title=mi.title)
 
         # Fetch updated timestamp, cover_record, thumbnail_record

src/calibre/ebooks/metadata/sources/amazon.py

@@ -301,7 +301,7 @@ class Amazon(Source):
         if asin is None:
             asin = identifiers.get('asin', None)
         if asin:
-            return 'http://amzn.com/%s'%asin
+            return ('amazon', asin, 'http://amzn.com/%s'%asin)
     # }}}
 
     def create_query(self, log, title=None, authors=None, identifiers={}): # {{{

src/calibre/ebooks/metadata/sources/base.py

@@ -56,7 +56,8 @@ class InternalMetadataCompareKeyGen(object):
 
     '''
     Generate a sort key for comparison of the relevance of Metadata objects,
-    given a search query.
+    given a search query. This is used only to compare results from the same
+    metadata source, not across different sources.
 
     The sort key ensures that an ascending order sort is a sort by order of
     decreasing relevance.
@@ -374,7 +375,11 @@ class Source(Plugin):
 
     def get_book_url(self, identifiers):
         '''
-        Return the URL for the book identified by identifiers at this source.
+        Return a 3-tuple or None. The 3-tuple is of the form:
+        (identifier_type, identifier_value, URL).
+        The URL is the URL for the book identified by identifiers at this
+        source. identifier_type, identifier_value specify the identifier
+        corresponding to the URL.
         This URL must be browseable to by a human using a browser. It is meant
         to provide a clickable link for the user to easily visit the books page
         at this source.

src/calibre/ebooks/metadata/sources/google.py

@@ -173,7 +173,7 @@ class GoogleBooks(Source):
     def get_book_url(self, identifiers): # {{{
         goog = identifiers.get('google', None)
         if goog is not None:
-            return 'http://books.google.com/books?id=%s'%goog
+            return ('google', goog, 'http://books.google.com/books?id=%s'%goog)
     # }}}
 
     def create_query(self, log, title=None, authors=None, identifiers={}): # {{{

src/calibre/ebooks/metadata/sources/identify.py

@@ -435,18 +435,30 @@ def identify(log, abort, # {{{
 # }}}
 
 def urls_from_identifiers(identifiers): # {{{
+    identifiers = dict([(k.lower(), v) for k, v in identifiers.iteritems()])
     ans = []
     for plugin in all_metadata_plugins():
         try:
-            url = plugin.get_book_url(identifiers)
-            if url is not None:
-                ans.append((plugin.name, url))
+            id_type, id_val, url = plugin.get_book_url(identifiers)
+            ans.append((plugin.name, id_type, id_val, url))
         except:
             pass
     isbn = identifiers.get('isbn', None)
     if isbn:
-        ans.append((isbn,
-            'http://www.worldcat.org/search?q=bn%%3A%s&qt=advanced'%isbn))
+        ans.append((isbn, 'isbn', isbn,
+            'http://www.worldcat.org/isbn/'+isbn))
+    doi = identifiers.get('doi', None)
+    if doi:
+        ans.append(('DOI', 'doi', doi,
+            'http://dx.doi.org/'+doi))
+    arxiv = identifiers.get('arxiv', None)
+    if arxiv:
+        ans.append(('arXiv', 'arxiv', arxiv,
+            'http://arxiv.org/abs/'+arxiv))
+    oclc = identifiers.get('oclc', None)
+    if oclc:
+        ans.append(('OCLC', 'oclc', oclc,
+            'http://www.worldcat.org/oclc/'+oclc))
     return ans
 # }}}
 

src/calibre/ebooks/metadata/sources/isbndb.py

@@ -81,7 +81,7 @@ class ISBNDB(Source):
             author_tokens = self.get_author_tokens(authors,
                     only_first_author=True)
             tokens += author_tokens
-            tokens = [quote(t) for t in tokens]
+            tokens = [quote(t.encode('utf-8') if isinstance(t, unicode) else t) for t in tokens]
             q = '+'.join(tokens)
             q = 'index1=combined&value1='+q
 

src/calibre/ebooks/metadata/sources/overdrive.py

@@ -265,7 +265,7 @@ class OverDrive(Source):
                     if creators:
                         creators = creators.split(', ')
                     # if an exact match in a preferred format occurs
-                    if ((author and creators[0] == author[0]) or (not author and not creators)) and od_title.lower() == title.lower() and int(formatid) in [1, 50, 410, 900] and thumbimage:
+                    if ((author and creators and creators[0] == author[0]) or (not author and not creators)) and od_title.lower() == title.lower() and int(formatid) in [1, 50, 410, 900] and thumbimage:
                         return self.format_results(reserveid, od_title, subtitle, series, publisher,
                                 creators, thumbimage, worldcatlink, formatid)
                     else:
@@ -291,7 +291,7 @@ class OverDrive(Source):
                                 close_matches.insert(0, self.format_results(reserveid, od_title, subtitle, series, publisher, creators, thumbimage, worldcatlink, formatid))
                             else:
                                 close_matches.append(self.format_results(reserveid, od_title, subtitle, series, publisher, creators, thumbimage, worldcatlink, formatid))
-                                
+
                         elif close_title_match and close_author_match and int(formatid) in [1, 50, 410, 900]:
                             close_matches.append(self.format_results(reserveid, od_title, subtitle, series, publisher, creators, thumbimage, worldcatlink, formatid))
 

src/calibre/ebooks/metadata/worker.py

@@ -222,7 +222,7 @@ class SaveWorker(Thread):
                         if isbytestring(fpath):
                             fpath = fpath.decode(filesystem_encoding)
                         formats[fmt.lower()] = fpath
-            data[i] = [opf, cpath, formats]
+            data[i] = [opf, cpath, formats, mi.last_modified.isoformat()]
         return data
 
     def run(self):

src/calibre/ebooks/oeb/base.py

@@ -16,7 +16,7 @@ from urllib import unquote as urlunquote
 from lxml import etree, html
 from calibre.constants import filesystem_encoding, __version__
 from calibre.translations.dynamic import translate
-from calibre.ebooks.chardet import xml_to_unicode
+from calibre.ebooks.chardet import xml_to_unicode, strip_encoding_declarations
 from calibre.ebooks.oeb.entitydefs import ENTITYDEFS
 from calibre.ebooks.conversion.preprocess import CSSPreProcessor
 from calibre import isbytestring, as_unicode, get_types_map
@@ -446,22 +446,23 @@ class NullContainer(object):
 class DirContainer(object):
     """Filesystem directory container."""
 
-    def __init__(self, path, log):
+    def __init__(self, path, log, ignore_opf=False):
         self.log = log
         if isbytestring(path):
             path = path.decode(filesystem_encoding)
+        self.opfname = None
         ext = os.path.splitext(path)[1].lower()
         if ext == '.opf':
             self.opfname = os.path.basename(path)
             self.rootdir = os.path.dirname(path)
             return
         self.rootdir = path
-        for path in self.namelist():
-            ext = os.path.splitext(path)[1].lower()
-            if ext == '.opf':
-                self.opfname = path
-                return
-        self.opfname = None
+        if not ignore_opf:
+            for path in self.namelist():
+                ext = os.path.splitext(path)[1].lower()
+                if ext == '.opf':
+                    self.opfname = path
+                    return
 
     def read(self, path):
         if path is None:
@@ -852,6 +853,7 @@ class Manifest(object):
             self.oeb.log.debug('Parsing', self.href, '...')
             # Convert to Unicode and normalize line endings
             data = self.oeb.decode(data)
+            data = strip_encoding_declarations(data)
             data = self.oeb.html_preprocessor(data)
             # There could be null bytes in data if it had � entities in it
             data = data.replace('\0', '')

src/calibre/ebooks/pdb/plucker/reader.py

@@ -16,6 +16,7 @@ from calibre import CurrentDir
 from calibre.ebooks.pdb.formatreader import FormatReader
 from calibre.ptempfile import TemporaryFile
 from calibre.utils.magick import Image, create_canvas
+from calibre.ebooks.compression.palmdoc import decompress_doc
 
 DATATYPE_PHTML = 0
 DATATYPE_PHTML_COMPRESSED = 1
@@ -359,7 +360,7 @@ class Reader(FormatReader):
         # plugin assemble the order based on hyperlinks.
         with CurrentDir(output_dir):
             for uid, num in self.uid_text_secion_number.items():
-                self.log.debug(_('Writing record with uid: %s as %s.html' % (uid, uid)))
+                self.log.debug('Writing record with uid: %s as %s.html' % (uid, uid))
                 with open('%s.html' % uid, 'wb') as htmlf:
                     html = u'<html><body>'
                     section_header, section_data = self.sections[num]
@@ -465,7 +466,7 @@ class Reader(FormatReader):
             if not home_html:
                 home_html = self.uid_text_secion_number.items()[0][0]
         except:
-            raise Exception(_('Could not determine home.html'))
+            raise Exception('Could not determine home.html')
         # Generate oeb from html conversion.
         oeb = html_input.convert(open('%s.html' % home_html, 'rb'), self.options, 'html', self.log, {})
         self.options.debug_pipeline = odi

src/calibre/gui2/__init__.py

@@ -80,6 +80,14 @@ gprefs.defaults['font'] = None
 gprefs.defaults['tags_browser_partition_method'] = 'first letter'
 gprefs.defaults['tags_browser_collapse_at'] = 100
 gprefs.defaults['edit_metadata_single_layout'] = 'default'
+gprefs.defaults['book_display_fields'] = [
+        ('title', False), ('authors', False), ('formats', True),
+        ('series', True), ('identifiers', True), ('tags', True),
+        ('path', True), ('publisher', False), ('rating', False),
+        ('author_sort', False), ('sort', False), ('timestamp', False),
+        ('uuid', False), ('comments', True), ('id', False), ('pubdate', False),
+        ('last_modified', False), ('size', False),
+        ]
 
 # }}}
 
@@ -89,7 +97,7 @@ UNDEFINED_QDATE = QDate(UNDEFINED_DATE)
 ALL_COLUMNS = ['title', 'ondevice', 'authors', 'size', 'timestamp', 'rating', 'publisher',
         'tags', 'series', 'pubdate']
 
-def _config():
+def _config(): # {{{
     c = Config('gui', 'preferences for the calibre GUI')
     c.add_opt('send_to_storage_card_by_default', default=False,
               help=_('Send file to storage card instead of main memory by default'))
@@ -181,6 +189,8 @@ def _config():
     return ConfigProxy(c)
 
 config = _config()
+# }}}
+
 # Turn off DeprecationWarnings in windows GUI
 if iswindows:
     import warnings
@@ -729,12 +739,6 @@ def build_forms(srcdir, info=None):
             dat = dat.replace('from QtWebKit.QWebView import QWebView',
                     'from PyQt4 import QtWebKit\nfrom PyQt4.QtWebKit import QWebView')
 
-            if form.endswith('viewer%smain.ui'%os.sep):
-                info('\t\tPromoting WebView')
-                dat = dat.replace('self.view = QtWebKit.QWebView(', 'self.view = DocumentView(')
-                dat = dat.replace('self.view = QWebView(', 'self.view = DocumentView(')
-                dat += '\n\nfrom calibre.gui2.viewer.documentview import DocumentView'
-
             open(compiled_form, 'wb').write(dat)
 
 _df = os.environ.get('CALIBRE_DEVELOP_FROM', None)

src/calibre/gui2/actions/show_book_details.py

@@ -30,5 +30,5 @@ class ShowBookDetailsAction(InterfaceAction):
         index = self.gui.library_view.currentIndex()
         if index.isValid():
             BookInfo(self.gui, self.gui.library_view, index,
-                    self.gui.iactions['View'].view_format_by_id).show()
+                    self.gui.book_details.handle_click).show()
 

src/calibre/gui2/book_details.py

@@ -5,67 +5,155 @@ __license__   = 'GPL v3'
 __copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
-import collections, sys
-from Queue import Queue
 
-from PyQt4.Qt import QPixmap, QSize, QWidget, Qt, pyqtSignal, QUrl, \
-    QPropertyAnimation, QEasingCurve, QThread, QApplication, QFontInfo, \
-    QSizePolicy, QPainter, QRect, pyqtProperty, QLayout, QPalette, QMenu
+from PyQt4.Qt import (QPixmap, QSize, QWidget, Qt, pyqtSignal, QUrl,
+    QPropertyAnimation, QEasingCurve, QApplication, QFontInfo,
+    QSizePolicy, QPainter, QRect, pyqtProperty, QLayout, QPalette, QMenu)
 from PyQt4.QtWebKit import QWebView
 
-from calibre import fit_image, prepare_string_for_xml
-from calibre.gui2.dnd import dnd_has_image, dnd_get_image, dnd_get_files, \
-    IMAGE_EXTENSIONS, dnd_has_extension
+from calibre import fit_image, force_unicode, prepare_string_for_xml
+from calibre.gui2.dnd import (dnd_has_image, dnd_get_image, dnd_get_files,
+    IMAGE_EXTENSIONS, dnd_has_extension)
 from calibre.ebooks import BOOK_EXTENSIONS
-from calibre.constants import preferred_encoding
+from calibre.ebooks.metadata.book.base import (field_metadata, Metadata)
+from calibre.ebooks.metadata import fmt_sidx
+from calibre.ebooks.metadata.sources.identify import urls_from_identifiers
+from calibre.constants import filesystem_encoding
 from calibre.library.comments import comments_to_html
-from calibre.gui2 import config, open_local_file, open_url, pixmap_to_data
+from calibre.gui2 import (config, open_local_file, open_url, pixmap_to_data,
+        gprefs)
 from calibre.utils.icu import sort_key
 
-# render_rows(data) {{{
-WEIGHTS = collections.defaultdict(lambda : 100)
-WEIGHTS[_('Path')] = 5
-WEIGHTS[_('Formats')] = 1
-WEIGHTS[_('Collections')] = 2
-WEIGHTS[_('Series')] = 3
-WEIGHTS[_('Tags')] = 4
+def render_html(mi, css, vertical, widget, all_fields=False): # {{{
+    table = render_data(mi, all_fields=all_fields,
+            use_roman_numbers=config['use_roman_numerals_for_series_number'])
 
-def render_rows(data):
-    keys = data.keys()
-    # First sort by name. The WEIGHTS sort will preserve this sub-order
-    keys.sort(key=sort_key)
-    keys.sort(key=lambda x: WEIGHTS[x])
-    rows = []
-    for key in keys:
-        txt = data[key]
-        if key in ('id', _('Comments')) or not hasattr(txt, 'strip') or not txt.strip() or \
-                txt == 'None':
+    def color_to_string(col):
+        ans = '#000000'
+        if col.isValid():
+            col = col.toRgb()
+            if col.isValid():
+                ans = unicode(col.name())
+        return ans
+
+    f = QFontInfo(QApplication.font(widget)).pixelSize()
+    c = color_to_string(QApplication.palette().color(QPalette.Normal,
+                    QPalette.WindowText))
+    templ = u'''\
+    <html>
+        <head>
+        <style type="text/css">
+            body, td {background-color: transparent; font-size: %dpx; color: %s }
+        </style>
+        <style type="text/css">
+            %s
+        </style>
+        </head>
+        <body>
+        %%s
+        </body>
+    <html>
+    '''%(f, c, css)
+    comments = u''
+    if mi.comments:
+        comments = comments_to_html(force_unicode(mi.comments))
+    right_pane = u'<div id="comments" class="comments">%s</div>'%comments
+
+    if vertical:
+        ans = templ%(table+right_pane)
+    else:
+        ans = templ%(u'<table><tr><td valign="top" '
+            'style="padding-right:2em; width:40%%">%s</td><td valign="top">%s</td></tr></table>'
+                % (table, right_pane))
+    return ans
+
+def get_field_list(fm, use_defaults=False):
+    src = gprefs.defaults if use_defaults else gprefs
+    fieldlist = list(src['book_display_fields'])
+    names = frozenset([x[0] for x in fieldlist])
+    for field in fm.displayable_field_keys():
+        if field not in names:
+            fieldlist.append((field, True))
+    return fieldlist
+
+def render_data(mi, use_roman_numbers=True, all_fields=False):
+    ans = []
+    isdevice = not hasattr(mi, 'id')
+    fm = getattr(mi, 'field_metadata', field_metadata)
+
+    for field, display in get_field_list(fm):
+        metadata = fm.get(field, None)
+        if all_fields:
+            display = True
+        if (not display or not metadata or mi.is_null(field) or
+                field == 'comments'):
             continue
-        if isinstance(key, str):
-            key = key.decode(preferred_encoding, 'replace')
-        if isinstance(txt, str):
-            txt = txt.decode(preferred_encoding, 'replace')
-        if key.endswith(u':html'):
-            key = key[:-5]
-            txt = comments_to_html(txt)
-        elif '</font>' not in txt:
-            txt = prepare_string_for_xml(txt)
-        if 'id' in data:
-            if key == _('Path'):
-                txt = u'<a href="path:%s" title="%s">%s</a>'%(data['id'],
-                        txt, _('Click to open'))
-            if key == _('Formats') and txt and txt != _('None'):
-                fmts = [x.strip() for x in txt.split(',')]
-                fmts = [u'<a href="format:%s:%s">%s</a>' % (data['id'], x, x) for x
-                        in fmts]
-                txt = ', '.join(fmts)
+        name = metadata['name']
+        if not name:
+            name = field
+        name += ':'
+        if metadata['datatype'] == 'comments':
+            val = getattr(mi, field)
+            if val:
+                val = force_unicode(val)
+                ans.append((field,
+                    u'<td class="comments" colspan="2">%s</td>'%comments_to_html(val)))
+        elif field == 'path':
+            if mi.path:
+                path = force_unicode(mi.path, filesystem_encoding)
+                scheme = u'devpath' if isdevice else u'path'
+                url = prepare_string_for_xml(path if isdevice else
+                        unicode(mi.id), True)
+                link = u'<a href="%s:%s" title="%s">%s</a>' % (scheme, url,
+                        prepare_string_for_xml(path, True), _('Click to open'))
+                ans.append((field, u'<td class="title">%s</td><td>%s</td>'%(name, link)))
+        elif field == 'formats':
+            if isdevice: continue
+            fmts = [u'<a href="format:%s:%s">%s</a>' % (mi.id, x, x) for x
+                        in mi.formats]
+            ans.append((field, u'<td class="title">%s</td><td>%s</td>'%(name,
+                u', '.join(fmts))))
+        elif field == 'identifiers':
+            urls = urls_from_identifiers(mi.identifiers)
+            links = [u'<a href="%s" title="%s:%s">%s</a>' % (url, id_typ, id_val, name)
+                    for name, id_typ, id_val, url in urls]
+            links = u', '.join(links)
+            if links:
+                ans.append((field, u'<td class="title">%s</td><td>%s</td>'%(
+                    _('Ids')+':', links)))
         else:
-            if key == _('Path'):
-                txt = u'<a href="devpath:%s">%s</a>'%(txt,
-                        _('Click to open'))
+            val = mi.format_field(field)[-1]
+            if val is None:
+                continue
+            val = prepare_string_for_xml(val)
+            if metadata['datatype'] == 'series':
+                sidx = mi.get(field+'_index')
+                if sidx is None:
+                    sidx = 1.0
+                val = _('Book %s of <span class="series_name">%s</span>')%(fmt_sidx(sidx,
+                    use_roman=use_roman_numbers),
+                    prepare_string_for_xml(getattr(mi, field)))
 
-        rows.append((key, txt))
-    return rows
+            ans.append((field, u'<td class="title">%s</td><td>%s</td>'%(name, val)))
+
+    dc = getattr(mi, 'device_collections', [])
+    if dc:
+        dc = u', '.join(sorted(dc, key=sort_key))
+        ans.append(('device_collections',
+            u'<td class="title">%s</td><td>%s</td>'%(
+                _('Collections')+':', dc)))
+
+    def classname(field):
+        try:
+            dt = fm[field]['datatype']
+        except:
+            dt = 'text'
+        return 'datatype_%s'%dt
+
+    ans = [u'<tr id="%s" class="%s">%s</tr>'%(field.replace('#', '_'),
+        classname(field), html) for field, html in ans]
+    # print '\n'.join(ans)
+    return u'<table class="fields">%s</table>'%(u'\n'.join(ans))
 
 # }}}
 
@@ -117,10 +205,10 @@ class CoverView(QWidget): # {{{
 
     def show_data(self, data):
         self.animation.stop()
-        same_item = data.get('id', True) == self.data.get('id', False)
+        same_item = getattr(data, 'id', True) == self.data.get('id', False)
         self.data = {'id':data.get('id', None)}
-        if data.has_key('cover'):
-            self.pixmap = QPixmap.fromImage(data.pop('cover'))
+        if data.cover_data[1]:
+            self.pixmap = QPixmap.fromImage(data.cover_data[1])
             if self.pixmap.isNull() or self.pixmap.width() < 5 or \
                     self.pixmap.height() < 5:
                 self.pixmap = self.default_pixmap
@@ -188,32 +276,6 @@ class CoverView(QWidget): # {{{
 
 # Book Info {{{
 
-class RenderComments(QThread):
-
-    rdone = pyqtSignal(object, object)
-
-    def __init__(self, parent):
-        QThread.__init__(self, parent)
-        self.queue = Queue()
-        self.start()
-
-    def run(self):
-        while True:
-            try:
-                rows, comments = self.queue.get()
-            except:
-                break
-            import time
-            time.sleep(0.001)
-            oint = sys.getcheckinterval()
-            sys.setcheckinterval(5)
-            try:
-                self.rdone.emit(rows, comments_to_html(comments))
-            except:
-                pass
-            sys.setcheckinterval(oint)
-
-
 class BookInfo(QWebView):
 
     link_clicked = pyqtSignal(object)
@@ -221,8 +283,6 @@ class BookInfo(QWebView):
     def __init__(self, vertical, parent=None):
         QWebView.__init__(self, parent)
         self.vertical = vertical
-        self.renderer = RenderComments(self)
-        self.renderer.rdone.connect(self._show_data, type=Qt.QueuedConnection)
         self.page().setLinkDelegationPolicy(self.page().DelegateAllLinks)
         self.linkClicked.connect(self.link_activated)
         self._link_clicked = False
@@ -231,65 +291,21 @@ class BookInfo(QWebView):
         self.setAcceptDrops(False)
         palette.setBrush(QPalette.Base, Qt.transparent)
         self.page().setPalette(palette)
+        self.css = P('templates/book_details.css', data=True).decode('utf-8')
 
     def link_activated(self, link):
         self._link_clicked = True
+        if unicode(link.scheme()) in ('http', 'https'):
+            return open_url(link)
         link = unicode(link.toString())
         self.link_clicked.emit(link)
 
     def turnoff_scrollbar(self, *args):
         self.page().mainFrame().setScrollBarPolicy(Qt.Horizontal, Qt.ScrollBarAlwaysOff)
 
-    def show_data(self, data):
-        rows = render_rows(data)
-        rows = u'\n'.join([u'<tr><td valign="top"><b>%s:</b></td><td valign="top">%s</td></tr>'%(k,t) for
-            k, t in rows])
-        comments = data.get(_('Comments'), '')
-        if not comments or comments == u'None':
-            comments = ''
-        self.renderer.queue.put((rows, comments))
-        self._show_data(rows, '')
-
-
-    def _show_data(self, rows, comments):
-
-        def color_to_string(col):
-            ans = '#000000'
-            if col.isValid():
-                col = col.toRgb()
-                if col.isValid():
-                    ans = unicode(col.name())
-            return ans
-
-        f = QFontInfo(QApplication.font(self.parent())).pixelSize()
-        c = color_to_string(QApplication.palette().color(QPalette.Normal,
-                        QPalette.WindowText))
-        templ = u'''\
-        <html>
-            <head>
-            <style type="text/css">
-                body, td {background-color: transparent; font-size: %dpx; color: %s }
-                a { text-decoration: none; color: blue }
-                div.description { margin-top: 0; padding-top: 0; text-indent: 0 }
-                table { margin-bottom: 0; padding-bottom: 0; }
-            </style>
-            </head>
-            <body>
-            %%s
-            </body>
-        <html>
-        '''%(f, c)
-        if self.vertical:
-            extra = ''
-            if comments:
-                extra = u'<div class="description">%s</div>'%comments
-            self.setHtml(templ%(u'<table>%s</table>%s'%(rows, extra)))
-        else:
-            left_pane = u'<table>%s</table>'%rows
-            right_pane = u'<div>%s</div>'%comments
-            self.setHtml(templ%(u'<table><tr><td valign="top" '
-                'style="padding-right:2em; width:40%%">%s</td><td valign="top">%s</td></tr></table>'
-                    % (left_pane, right_pane)))
+    def show_data(self, mi):
+        html = render_html(mi, self.css, self.vertical, self.parent())
+        self.setHtml(html)
 
     def mouseDoubleClickEvent(self, ev):
         swidth = self.page().mainFrame().scrollBarGeometry(Qt.Vertical).width()
@@ -457,10 +473,10 @@ class BookDetails(QWidget): # {{{
         self._layout.addWidget(self.cover_view)
         self.book_info = BookInfo(vertical, self)
         self._layout.addWidget(self.book_info)
-        self.book_info.link_clicked.connect(self._link_clicked)
+        self.book_info.link_clicked.connect(self.handle_click)
         self.setCursor(Qt.PointingHandCursor)
 
-    def _link_clicked(self, link):
+    def handle_click(self, link):
         typ, _, val = link.partition(':')
         if typ == 'path':
             self.open_containing_folder.emit(int(val))
@@ -484,7 +500,7 @@ class BookDetails(QWidget): # {{{
     def show_data(self, data):
         self.book_info.show_data(data)
         self.cover_view.show_data(data)
-        self.current_path = data.get(_('Path'), '')
+        self.current_path = getattr(data, u'path', u'')
         self.update_layout()
 
     def update_layout(self):
@@ -500,7 +516,7 @@ class BookDetails(QWidget): # {{{
         )
 
     def reset_info(self):
-        self.show_data({})
+        self.show_data(Metadata(_('Unknown')))
 
 # }}}
 

src/calibre/gui2/dialogs/book_info.py

@@ -3,30 +3,33 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 
-import textwrap, os, re
 
-from PyQt4.Qt import QCoreApplication, SIGNAL, QModelIndex, QTimer, Qt, \
-    QDialog, QPixmap, QIcon, QSize
+from PyQt4.Qt import (QCoreApplication, SIGNAL, QModelIndex, QTimer, Qt,
+    QDialog, QPixmap, QIcon, QSize, QPalette)
 
 from calibre.gui2.dialogs.book_info_ui import Ui_BookInfo
-from calibre.gui2 import dynamic, open_local_file, open_url
+from calibre.gui2 import dynamic
 from calibre import fit_image
-from calibre.library.comments import comments_to_html
-from calibre.utils.icu import sort_key
-
+from calibre.gui2.book_details import render_html
 
 class BookInfo(QDialog, Ui_BookInfo):
 
-    def __init__(self, parent, view, row, view_func):
+    def __init__(self, parent, view, row, link_delegate):
         QDialog.__init__(self, parent)
         Ui_BookInfo.__init__(self)
         self.setupUi(self)
         self.gui = parent
         self.cover_pixmap = None
-        self.comments.sizeHint = self.comments_size_hint
-        self.comments.page().setLinkDelegationPolicy(self.comments.page().DelegateAllLinks)
-        self.comments.linkClicked.connect(self.link_clicked)
-        self.view_func = view_func
+        self.details.sizeHint = self.details_size_hint
+        self.details.page().setLinkDelegationPolicy(self.details.page().DelegateAllLinks)
+        self.details.linkClicked.connect(self.link_clicked)
+        self.css = P('templates/book_details.css', data=True).decode('utf-8')
+        self.link_delegate = link_delegate
+        self.details.setAttribute(Qt.WA_OpaquePaintEvent, False)
+        palette = self.details.palette()
+        self.details.setAcceptDrops(False)
+        palette.setBrush(QPalette.Base, Qt.transparent)
+        self.details.page().setPalette(palette)
 
 
         self.view = view
@@ -37,7 +40,6 @@ class BookInfo(QDialog, Ui_BookInfo):
         self.connect(self.view.selectionModel(), SIGNAL('currentChanged(QModelIndex,QModelIndex)'), self.slave)
         self.connect(self.next_button, SIGNAL('clicked()'), self.next)
         self.connect(self.previous_button, SIGNAL('clicked()'), self.previous)
-        self.connect(self.text, SIGNAL('linkActivated(QString)'), self.open_book_path)
         self.fit_cover.stateChanged.connect(self.toggle_cover_fit)
         self.cover.resizeEvent = self.cover_view_resized
         self.cover.cover_changed.connect(self.cover_changed)
@@ -46,6 +48,10 @@ class BookInfo(QDialog, Ui_BookInfo):
         screen_height = desktop.availableGeometry().height() - 100
         self.resize(self.size().width(), screen_height)
 
+    def link_clicked(self, qurl):
+        link = unicode(qurl.toString())
+        self.link_delegate(link)
+
     def cover_changed(self, data):
         if self.current_row is not None:
             id_ = self.view.model().id(self.current_row)
@@ -60,11 +66,8 @@ class BookInfo(QDialog, Ui_BookInfo):
         if self.fit_cover.isChecked():
             self.resize_cover()
 
-    def link_clicked(self, url):
-        open_url(url)
-
-    def comments_size_hint(self):
-        return QSize(350, 250)
+    def details_size_hint(self):
+        return QSize(350, 550)
 
     def toggle_cover_fit(self, state):
         dynamic.set('book_info_dialog_fit_cover', self.fit_cover.isChecked())
@@ -77,13 +80,6 @@ class BookInfo(QDialog, Ui_BookInfo):
         row = current.row()
         self.refresh(row)
 
-    def open_book_path(self, path):
-        path = unicode(path)
-        if os.sep in path:
-            open_local_file(path)
-        else:
-            self.view_func(self.view.model().id(self.current_row), path)
-
     def next(self):
         row = self.view.currentIndex().row()
         ni = self.view.model().index(row+1, 0)
@@ -117,8 +113,8 @@ class BookInfo(QDialog, Ui_BookInfo):
             row = row.row()
         if row == self.current_row:
             return
-        info = self.view.model().get_book_info(row)
-        if info is None:
+        mi = self.view.model().get_book_display_info(row)
+        if mi is None:
             # Indicates books was deleted from library, or row numbers have
             # changed
             return
@@ -126,40 +122,11 @@ class BookInfo(QDialog, Ui_BookInfo):
         self.previous_button.setEnabled(False if row == 0 else True)
         self.next_button.setEnabled(False if row == self.view.model().rowCount(QModelIndex())-1 else True)
         self.current_row = row
-        self.setWindowTitle(info[_('Title')])
-        self.title.setText('<b>'+info.pop(_('Title')))
-        comments = info.pop(_('Comments'), '')
-        if comments:
-            comments = comments_to_html(comments)
-        if re.search(r'<[a-zA-Z]+>', comments) is None:
-            lines = comments.splitlines()
-            lines = [x if x.strip() else '<br><br>' for x in lines]
-            comments = '\n'.join(lines)
-        self.comments.setHtml('<div>%s</div>' % comments)
-        self.comments.page().setLinkDelegationPolicy(self.comments.page().DelegateAllLinks)
-        cdata = info.pop('cover', '')
-        self.cover_pixmap = QPixmap.fromImage(cdata)
+        self.setWindowTitle(mi.title)
+        self.title.setText('<b>'+mi.title)
+        mi.title = _('Unknown')
+        self.cover_pixmap = QPixmap.fromImage(mi.cover_data[1])
         self.resize_cover()
+        html = render_html(mi, self.css, True, self, all_fields=True)
+        self.details.setHtml(html)
 
-        rows = u''
-        self.text.setText('')
-        self.data = info
-        if _('Path') in info.keys():
-            p = info[_('Path')]
-            info[_('Path')] = '<a href="%s">%s</a>'%(p, p)
-        if _('Formats') in info.keys():
-            formats = info[_('Formats')].split(',')
-            info[_('Formats')] = ''
-            for f in formats:
-                f = f.strip()
-                info[_('Formats')] += '<a href="%s">%s</a>, '%(f,f)
-        for key in sorted(info.keys(), key=sort_key):
-            if key == 'id': continue
-            txt  = info[key]
-            if key.endswith(':html'):
-                key = key[:-5]
-                txt = comments_to_html(txt)
-            if key != _('Path'):
-                txt  = u'<br />\n'.join(textwrap.wrap(txt, 120))
-            rows += u'<tr><td><b>%s:</b></td><td>%s</td></tr>'%(key, txt)
-        self.text.setText(u'<table>'+rows+'</table>')

src/calibre/gui2/dialogs/book_info.ui

@@ -20,6 +20,12 @@
   <layout class="QGridLayout" name="gridLayout">
    <item row="0" column="0" colspan="2">
     <widget class="QLabel" name="title">
+     <property name="sizePolicy">
+      <sizepolicy hsizetype="Preferred" vsizetype="Maximum">
+       <horstretch>0</horstretch>
+       <verstretch>0</verstretch>
+      </sizepolicy>
+     </property>
      <property name="font">
       <font>
        <weight>75</weight>
@@ -34,82 +40,17 @@
      </property>
     </widget>
    </item>
-   <item row="1" column="0" rowspan="3">
+   <item row="2" column="0" rowspan="3">
     <widget class="CoverView" name="cover"/>
    </item>
-   <item row="1" column="1">
-    <widget class="QScrollArea" name="scrollArea">
-     <property name="frameShape">
-      <enum>QFrame::NoFrame</enum>
-     </property>
-     <property name="widgetResizable">
-      <bool>true</bool>
-     </property>
-     <widget class="QWidget" name="scrollAreaWidgetContents">
-      <property name="geometry">
-       <rect>
-        <x>0</x>
-        <y>0</y>
-        <width>435</width>
-        <height>670</height>
-       </rect>
-      </property>
-      <layout class="QVBoxLayout" name="verticalLayout">
-       <item>
-        <widget class="QLabel" name="text">
-         <property name="text">
-          <string>TextLabel</string>
-         </property>
-         <property name="alignment">
-          <set>Qt::AlignLeading|Qt::AlignLeft|Qt::AlignTop</set>
-         </property>
-         <property name="wordWrap">
-          <bool>true</bool>
-         </property>
-        </widget>
-       </item>
-       <item>
-        <widget class="QGroupBox" name="groupBox">
-         <property name="title">
-          <string>Comments</string>
-         </property>
-         <layout class="QVBoxLayout" name="verticalLayout_2">
-          <item>
-           <widget class="QWebView" name="comments">
-            <property name="sizePolicy">
-             <sizepolicy hsizetype="Preferred" vsizetype="Expanding">
-              <horstretch>0</horstretch>
-              <verstretch>0</verstretch>
-             </sizepolicy>
-            </property>
-            <property name="maximumSize">
-             <size>
-              <width>350</width>
-              <height>16777215</height>
-             </size>
-            </property>
-            <property name="url">
-             <url>
-              <string>about:blank</string>
-             </url>
-            </property>
-           </widget>
-          </item>
-         </layout>
-        </widget>
-       </item>
-      </layout>
-     </widget>
-    </widget>
-   </item>
-   <item row="2" column="1">
+   <item row="3" column="1">
     <widget class="QCheckBox" name="fit_cover">
      <property name="text">
       <string>Fit &amp;cover within view</string>
      </property>
     </widget>
    </item>
-   <item row="3" column="1">
+   <item row="4" column="1">
     <layout class="QHBoxLayout" name="horizontalLayout">
      <item>
       <widget class="QPushButton" name="previous_button">
@@ -135,6 +76,15 @@
      </item>
     </layout>
    </item>
+   <item row="2" column="1">
+    <widget class="QWebView" name="details">
+     <property name="url">
+      <url>
+       <string>about:blank</string>
+      </url>
+     </property>
+    </widget>
+   </item>
   </layout>
  </widget>
  <customwidgets>

src/calibre/gui2/dialogs/confirm_delete_location.ui

@@ -22,6 +22,12 @@
     <layout class="QHBoxLayout" name="horizontalLayout">
      <item>
       <widget class="QLabel" name="label">
+       <property name="sizePolicy">
+        <sizepolicy hsizetype="Maximum" vsizetype="Preferred">
+         <horstretch>0</horstretch>
+         <verstretch>0</verstretch>
+        </sizepolicy>
+       </property>
        <property name="pixmap">
         <pixmap resource="../../../../resources/images.qrc">:/images/dialog_warning.png</pixmap>
        </property>
@@ -46,6 +52,10 @@
        <property name="text">
         <string>Library</string>
        </property>
+       <property name="icon">
+        <iconset resource="../../../../resources/images.qrc">
+         <normaloff>:/images/library.png</normaloff>:/images/library.png</iconset>
+       </property>
       </widget>
      </item>
      <item>
@@ -53,6 +63,10 @@
        <property name="text">
         <string>Device</string>
        </property>
+       <property name="icon">
+        <iconset resource="../../../../resources/images.qrc">
+         <normaloff>:/images/reader.png</normaloff>:/images/reader.png</iconset>
+       </property>
       </widget>
      </item>
      <item>
@@ -60,6 +74,10 @@
        <property name="text">
         <string>Library and Device</string>
        </property>
+       <property name="icon">
+        <iconset resource="../../../../resources/images.qrc">
+         <normaloff>:/images/trash.png</normaloff>:/images/trash.png</iconset>
+       </property>
       </widget>
      </item>
      <item>

src/calibre/gui2/layout.py

@@ -214,7 +214,6 @@ class SearchBar(QWidget): # {{{
         x.setIcon(QIcon(I("search_add_saved.png")))
         x.setObjectName("save_search_button")
         l.addWidget(x)
-        x.setToolTip(_("Save current search under the name shown in the box"))
 
 # }}}
 

src/calibre/gui2/library/delegates.py

@@ -97,18 +97,25 @@ class RatingDelegate(QStyledItemDelegate): # {{{
 
 class DateDelegate(QStyledItemDelegate): # {{{
 
+    def __init__(self, parent, tweak_name='gui_timestamp_display_format',
+            default_format='dd MMM yyyy', editor_format='dd MMM yyyy'):
+        QStyledItemDelegate.__init__(self, parent)
+        self.tweak_name = tweak_name
+        self.default_format = default_format
+        self.editor_format = editor_format
+
     def displayText(self, val, locale):
         d = val.toDate()
         if d <= UNDEFINED_QDATE:
             return ''
-        format = tweaks['gui_timestamp_display_format']
+        format = tweaks[self.tweak_name]
         if format is None:
-            format = 'dd MMM yyyy'
+            format = self.default_format
         return format_date(d.toPyDate(), format)
 
     def createEditor(self, parent, option, index):
         qde = QStyledItemDelegate.createEditor(self, parent, option, index)
-        qde.setDisplayFormat('dd MMM yyyy')
+        qde.setDisplayFormat(self.editor_format)
         qde.setMinimumDate(UNDEFINED_QDATE)
         qde.setSpecialValueText(_('Undefined'))
         qde.setCalendarPopup(True)

src/calibre/gui2/library/models.py

@@ -8,20 +8,20 @@ __docformat__ = 'restructuredtext en'
 import shutil, functools, re, os, traceback
 from contextlib import closing
 
-from PyQt4.Qt import QAbstractTableModel, Qt, pyqtSignal, QIcon, QImage, \
-        QModelIndex, QVariant, QDate, QColor
+from PyQt4.Qt import (QAbstractTableModel, Qt, pyqtSignal, QIcon, QImage,
+        QModelIndex, QVariant, QDate, QColor)
 
-from calibre.gui2 import NONE, config, UNDEFINED_QDATE
+from calibre.gui2 import NONE, UNDEFINED_QDATE
 from calibre.utils.pyparsing import ParseException
 from calibre.ebooks.metadata import fmt_sidx, authors_to_string, string_to_authors
 from calibre.ptempfile import PersistentTemporaryFile
 from calibre.utils.config import tweaks, prefs
-from calibre.utils.date import dt_factory, qt_to_dt, isoformat
+from calibre.utils.date import dt_factory, qt_to_dt
 from calibre.utils.icu import sort_key
 from calibre.utils.search_query_parser import SearchQueryParser
-from calibre.library.caches import _match, CONTAINS_MATCH, EQUALS_MATCH, \
-    REGEXP_MATCH, MetadataBackup, force_to_bool
-from calibre import strftime, isbytestring, prepare_string_for_xml
+from calibre.library.caches import (_match, CONTAINS_MATCH, EQUALS_MATCH,
+    REGEXP_MATCH, MetadataBackup, force_to_bool)
+from calibre import strftime, isbytestring
 from calibre.constants import filesystem_encoding, DEBUG
 from calibre.gui2.library import DEFAULT_SORT
 
@@ -70,6 +70,7 @@ class BooksModel(QAbstractTableModel): # {{{
                         'publisher' : _("Publisher"),
                         'tags'      : _("Tags"),
                         'series'    : _("Series"),
+                        'last_modified' : _('Modified'),
     }
 
     def __init__(self, parent=None, buffer=40):
@@ -114,7 +115,7 @@ class BooksModel(QAbstractTableModel): # {{{
         return cc_label in self.custom_columns
 
     def read_config(self):
-        self.use_roman_numbers = config['use_roman_numerals_for_series_number']
+        pass
 
     def set_device_connected(self, is_connected):
         self.device_connected = is_connected
@@ -355,63 +356,13 @@ class BooksModel(QAbstractTableModel): # {{{
         return self.rowCount(None)
 
     def get_book_display_info(self, idx):
-        def custom_keys_to_display():
-            ans = getattr(self, '_custom_fields_in_book_info', None)
-            if ans is None:
-                cfkeys = set(self.db.custom_field_keys())
-                yes_fields = set(tweaks['book_details_will_display'])
-                no_fields = set(tweaks['book_details_wont_display'])
-                if '*' in yes_fields:
-                    yes_fields = cfkeys
-                if '*' in no_fields:
-                    no_fields = cfkeys
-                ans = frozenset(yes_fields - no_fields)
-                setattr(self, '_custom_fields_in_book_info', ans)
-            return ans
-
-        data = {}
-        cdata = self.cover(idx)
-        if cdata:
-            data['cover'] = cdata
-        tags = list(self.db.get_tags(self.db.id(idx)))
-        if tags:
-            tags.sort(key=sort_key)
-            tags = ', '.join(tags)
-        else:
-            tags = _('None')
-        data[_('Tags')] = tags
-        formats = self.db.formats(idx)
-        if formats:
-            formats = formats.replace(',', ', ')
-        else:
-            formats = _('None')
-        data[_('Formats')] = formats
-        data[_('Path')] = self.db.abspath(idx)
-        data['id'] = self.id(idx)
-        comments = self.db.comments(idx)
-        if not comments:
-            comments = _('None')
-        data[_('Comments')] = comments
-        series = self.db.series(idx)
-        if series:
-            sidx = self.db.series_index(idx)
-            sidx = fmt_sidx(sidx, use_roman = self.use_roman_numbers)
-            data[_('Series')] = \
-                _('Book %s of %s.')%\
-                    (sidx, prepare_string_for_xml(series))
         mi = self.db.get_metadata(idx)
-        cf_to_display = custom_keys_to_display()
-        for key in mi.custom_field_keys():
-            if key not in cf_to_display:
-                continue
-            name, val = mi.format_field(key)
-            if mi.metadata_for_field(key)['datatype'] == 'comments':
-                name += ':html'
-            if val and name not in data:
-                data[name] = val
-
-        return data
-
+        mi.size = mi.book_size
+        mi.cover_data = ('jpg', self.cover(idx))
+        mi.id = self.db.id(idx)
+        mi.field_metadata = self.db.field_metadata
+        mi.path = self.db.abspath(idx, create_dirs=False)
+        return mi
 
     def current_changed(self, current, previous, emit_signal=True):
         if current.isValid():
@@ -425,16 +376,8 @@ class BooksModel(QAbstractTableModel): # {{{
     def get_book_info(self, index):
         if isinstance(index, int):
             index = self.index(index, 0)
+        # If index is not valid returns None
         data = self.current_changed(index, None, False)
-        if data is None:
-            return data
-        row = index.row()
-        data[_('Title')] = self.db.title(row)
-        au = self.db.authors(row)
-        if not au:
-            au = _('Unknown')
-        au = authors_to_string([a.strip().replace('|', ',') for a in au.split(',')])
-        data[_('Author(s)')] = au
         return data
 
     def metadata_for(self, ids):
@@ -678,6 +621,8 @@ class BooksModel(QAbstractTableModel): # {{{
                                 idx=self.db.field_metadata['timestamp']['rec_index']),
                    'pubdate'  : functools.partial(datetime_type,
                                 idx=self.db.field_metadata['pubdate']['rec_index']),
+                   'last_modified': functools.partial(datetime_type,
+                                idx=self.db.field_metadata['last_modified']['rec_index']),
                    'rating'   : functools.partial(rating_type,
                                 idx=self.db.field_metadata['rating']['rec_index']),
                    'publisher': functools.partial(text_type,
@@ -1189,39 +1134,46 @@ class DeviceBooksModel(BooksModel): # {{{
             img = self.default_image
         return img
 
-    def current_changed(self, current, previous):
-        data = {}
-        item = self.db[self.map[current.row()]]
-        cover = self.cover(current.row())
-        if cover is not self.default_image:
-            data['cover'] = cover
-        type = _('Unknown')
+    def get_book_display_info(self, idx):
+        from calibre.ebooks.metadata.book.base import Metadata
+        item = self.db[self.map[idx]]
+        cover = self.cover(idx)
+        if cover is self.default_image:
+            cover = None
+        title = item.title
+        if not title:
+            title = _('Unknown')
+        au = item.authors
+        if not au:
+            au = [_('Unknown')]
+        mi = Metadata(title, au)
+        mi.cover_data = ('jpg', cover)
+        fmt = _('Unknown')
         ext = os.path.splitext(item.path)[1]
         if ext:
-            type = ext[1:].lower()
-        data[_('Format')] = type
-        data[_('Path')] = item.path
+            fmt = ext[1:].lower()
+        mi.formats = [fmt]
+        mi.path = (item.path if item.path else None)
         dt = dt_factory(item.datetime, assume_utc=True)
-        data[_('Timestamp')] = isoformat(dt, sep=' ', as_utc=False)
-        data[_('Collections')] = ', '.join(item.device_collections)
-
-        tags = getattr(item, 'tags', None)
-        if tags:
-            tags = u', '.join(tags)
-        else:
-            tags = _('None')
-        data[_('Tags')] = tags
-        comments = getattr(item, 'comments', None)
-        if not comments:
-            comments = _('None')
-        data[_('Comments')] = comments
+        mi.timestamp = dt
+        mi.device_collections = list(item.device_collections)
+        mi.tags = list(getattr(item, 'tags', []))
+        mi.comments = getattr(item, 'comments', None)
         series = getattr(item, 'series', None)
         if series:
             sidx = getattr(item, 'series_index', 0)
-            sidx = fmt_sidx(sidx, use_roman = self.use_roman_numbers)
-            data[_('Series')] = _('Book <font face="serif">%s</font> of %s.')%(sidx, series)
+            mi.series = series
+            mi.series_index = sidx
+        return mi
 
-        self.new_bookdisplay_data.emit(data)
+    def current_changed(self, current, previous, emit_signal=True):
+        if current.isValid():
+            idx = current.row()
+            data = self.get_book_display_info(idx)
+            if emit_signal:
+                self.new_bookdisplay_data.emit(data)
+            else:
+                return data
 
     def paths(self, rows):
         return [self.db[self.map[r.row()]].path for r in rows ]
@@ -1281,7 +1233,7 @@ class DeviceBooksModel(BooksModel): # {{{
             elif cname == 'authors':
                 au = self.db[self.map[row]].authors
                 if not au:
-                    au = self.unknown
+                    au = [_('Unknown')]
                 return QVariant(authors_to_string(au))
             elif cname == 'size':
                 size = self.db[self.map[row]].size

src/calibre/gui2/library/views.py

@@ -76,6 +76,8 @@ class BooksView(QTableView): # {{{
         self.rating_delegate = RatingDelegate(self)
         self.timestamp_delegate = DateDelegate(self)
         self.pubdate_delegate = PubDateDelegate(self)
+        self.last_modified_delegate = DateDelegate(self,
+                tweak_name='gui_last_modified_display_format')
         self.tags_delegate = CompleteDelegate(self, ',', 'all_tags')
         self.authors_delegate = CompleteDelegate(self, '&', 'all_author_names', True)
         self.cc_names_delegate = CompleteDelegate(self, '&', 'all_custom', True)
@@ -296,6 +298,7 @@ class BooksView(QTableView): # {{{
         state = {}
         state['hidden_columns'] = [cm[i] for i in  range(h.count())
                 if h.isSectionHidden(i) and cm[i] != 'ondevice']
+        state['last_modified_injected'] = True
         state['sort_history'] = \
             self.cleanup_sort_history(self.model().sort_history)
         state['column_positions'] = {}
@@ -380,7 +383,7 @@ class BooksView(QTableView): # {{{
 
     def get_default_state(self):
         old_state = {
-                'hidden_columns': [],
+                'hidden_columns': ['last_modified'],
                 'sort_history':[DEFAULT_SORT],
                 'column_positions': {},
                 'column_sizes': {},
@@ -388,6 +391,7 @@ class BooksView(QTableView): # {{{
                     'size':'center',
                     'timestamp':'center',
                     'pubdate':'center'},
+                'last_modified_injected': True,
                 }
         h = self.column_header
         cm = self.column_map
@@ -398,7 +402,7 @@ class BooksView(QTableView): # {{{
                 old_state['column_sizes'][name] = \
                     min(350, max(self.sizeHintForColumn(i),
                         h.sectionSizeHint(i)))
-                if name == 'timestamp':
+                if name in ('timestamp', 'last_modified'):
                     old_state['column_sizes'][name] += 12
         return old_state
 
@@ -418,6 +422,13 @@ class BooksView(QTableView): # {{{
                         pass
                     if ans is not None:
                         db.prefs[name] = ans
+                else:
+                    if not ans.get('last_modified_injected', False):
+                        ans['last_modified_injected'] = True
+                        hc = ans.get('hidden_columns', [])
+                        if 'last_modified' not in hc:
+                            hc.append('last_modified')
+                        db.prefs[name] = ans
         return ans
 
 
@@ -459,7 +470,8 @@ class BooksView(QTableView): # {{{
     def database_changed(self, db):
         for i in range(self.model().columnCount(None)):
             if self.itemDelegateForColumn(i) in (self.rating_delegate,
-                    self.timestamp_delegate, self.pubdate_delegate):
+                    self.timestamp_delegate, self.pubdate_delegate,
+                    self.last_modified_delegate):
                 self.setItemDelegateForColumn(i, self.itemDelegate())
 
         cm = self.column_map
@@ -650,6 +662,11 @@ class BooksView(QTableView): # {{{
     def column_map(self):
         return self._model.column_map
 
+    def refresh_book_details(self):
+        idx = self.currentIndex()
+        if idx.isValid():
+            self._model.current_changed(idx, idx)
+
     def scrollContentsBy(self, dx, dy):
         # Needed as Qt bug causes headerview to not always update when scrolling
         QTableView.scrollContentsBy(self, dx, dy)

src/calibre/gui2/main.py

@@ -40,6 +40,11 @@ path_to_ebook to the database.
     parser.add_option('--ignore-plugins', default=False, action='store_true',
             help=_('Ignore custom plugins, useful if you installed a plugin'
                 ' that is preventing calibre from starting'))
+    parser.add_option('-s', '--shutdown-running-calibre', default=False,
+            action='store_true',
+            help=_('Cause a running calibre instance, if any, to be'
+                ' shutdown. Note that if there are running jobs, they '
+                'will be silently aborted, so use with care.'))
     return parser
 
 def init_qt(args):
@@ -339,7 +344,7 @@ def cant_start(msg=_('If you are sure it is not running')+', ',
 
     raise SystemExit(1)
 
-def communicate(args):
+def communicate(opts, args):
     t = RC()
     t.start()
     time.sleep(3)
@@ -348,9 +353,12 @@ def communicate(args):
         cant_start(what=_('try deleting the file')+': '+f)
         raise SystemExit(1)
 
-    if len(args) > 1:
-        args[1] = os.path.abspath(args[1])
-    t.conn.send('launched:'+repr(args))
+    if opts.shutdown_running_calibre:
+        t.conn.send('shutdown:')
+    else:
+        if len(args) > 1:
+            args[1] = os.path.abspath(args[1])
+        t.conn.send('launched:'+repr(args))
     t.conn.close()
     raise SystemExit(0)
 
@@ -365,6 +373,8 @@ def main(args=sys.argv):
     from calibre.utils.lock import singleinstance
     from multiprocessing.connection import Listener
     si = singleinstance('calibre GUI')
+    if si and opts.shutdown_running_calibre:
+        return 0
     if si:
         try:
             listener = Listener(address=ADDRESS)
@@ -390,10 +400,10 @@ def main(args=sys.argv):
     else:
         # On windows only singleinstance can be trusted
         otherinstance = True if iswindows else False
-    if not otherinstance:
+    if not otherinstance and not opts.shutdown_running_calibre:
         return run_gui(opts, args, actions, listener, app, gui_debug=gui_debug)
 
-    communicate(args)
+    communicate(opts, args)
 
     return 0
 

src/calibre/gui2/metadata/basic_widgets.py

@@ -18,11 +18,11 @@ from calibre.gui2.widgets import EnLineEdit, FormatList, ImageView
 from calibre.gui2.complete import MultiCompleteLineEdit, MultiCompleteComboBox
 from calibre.utils.icu import sort_key
 from calibre.utils.config import tweaks, prefs
-from calibre.ebooks.metadata import title_sort, authors_to_string, \
-        string_to_authors, check_isbn
+from calibre.ebooks.metadata import (title_sort, authors_to_string,
+        string_to_authors, check_isbn)
 from calibre.ebooks.metadata.meta import get_metadata
-from calibre.gui2 import file_icon_provider, UNDEFINED_QDATE, UNDEFINED_DATE, \
-        choose_files, error_dialog, choose_images, question_dialog
+from calibre.gui2 import (file_icon_provider, UNDEFINED_QDATE, UNDEFINED_DATE,
+        choose_files, error_dialog, choose_images, question_dialog)
 from calibre.utils.date import local_tz, qt_to_dt
 from calibre import strftime
 from calibre.ebooks import BOOK_EXTENSIONS
@@ -280,11 +280,16 @@ class AuthorSortEdit(EnLineEdit):
         aus = self.current_val
         meth = tweaks['author_sort_copy_method']
         if aus:
-            ln, _, rest = aus.partition(',')
-            if rest:
-                if meth in ('invert', 'nocomma', 'comma'):
-                    aus = rest.strip() + ' ' + ln.strip()
-            self.authors_edit.current_val = [aus]
+            ans = []
+            for one in [a.strip() for a in aus.split('&')]:
+                if not one:
+                    continue
+                ln, _, rest = one.partition(',')
+                if rest:
+                    if meth in ('invert', 'nocomma', 'comma'):
+                        one = rest.strip() + ' ' + ln.strip()
+                ans.append(one)
+            self.authors_edit.current_val = ans
 
     def auto_generate(self, *args):
         au = unicode(self.authors_edit.text())
@@ -805,6 +810,7 @@ class CommentsEdit(Editor): # {{{
             else:
                 val = comments_to_html(val)
             self.html = val
+            self.wyswyg_dirtied()
         return property(fget=fget, fset=fset)
 
     def initialize(self, db, id_):
@@ -936,7 +942,11 @@ class IdentifiersEdit(QLineEdit): # {{{
             ans = {}
             for x in parts:
                 c = x.split(':')
-                if len(c) == 2:
+                if len(c) > 1:
+                    if c[0] == 'isbn':
+                        v = check_isbn(c[1])
+                        if v is not None:
+                            c[1] = v
                     ans[c[0]] = c[1]
             return ans
         def fset(self, val):
@@ -947,6 +957,11 @@ class IdentifiersEdit(QLineEdit): # {{{
                 if x == 'isbn':
                     x = '00isbn'
                 return x
+            for k in list(val):
+                if k == 'isbn':
+                    v = check_isbn(k)
+                    if v is not None:
+                        val[k] = v
             ids = sorted(val.iteritems(), key=keygen)
             txt = ', '.join(['%s:%s'%(k, v) for k, v in ids])
             self.setText(txt.strip())
@@ -954,8 +969,8 @@ class IdentifiersEdit(QLineEdit): # {{{
         return property(fget=fget, fset=fset)
 
     def initialize(self, db, id_):
-        self.current_val = db.get_identifiers(id_, index_is_id=True)
-        self.original_val = self.current_val
+        self.original_val = db.get_identifiers(id_, index_is_id=True)
+        self.current_val = self.original_val
 
     def commit(self, db, id_):
         if self.original_val != self.current_val:

src/calibre/gui2/metadata/config.py

@@ -41,8 +41,11 @@ class FieldsModel(FM): # {{{
         self.reset()
 
     def commit(self):
-        val = [k for k, v in self.overrides.iteritems() if v == Qt.Unchecked]
-        self.prefs['ignore_fields'] = val
+        ignored_fields = set([x for x in self.prefs['ignore_fields'] if x not in
+            self.overrides])
+        changed = set([k for k, v in self.overrides.iteritems() if v ==
+            Qt.Unchecked])
+        self.prefs['ignore_fields'] = list(ignored_fields.union(changed))
 
 # }}}
 

src/calibre/gui2/metadata/single_download.py

@@ -253,7 +253,7 @@ class ResultsView(QTableView): # {{{
         parts.append('</center>')
         if book.identifiers:
             urls = urls_from_identifiers(book.identifiers)
-            ids = ['<a href="%s">%s</a>'%(url, name) for name, url in urls]
+            ids = ['<a href="%s">%s</a>'%(url, name) for name, ign, ign, url in urls]
             if ids:
                 parts.append('<div><b>%s:</b> %s</div><br>'%(_('See at'), ', '.join(ids)))
         if book.tags:

src/calibre/gui2/preferences/create_custom_column.py

@@ -63,7 +63,7 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn):
         self.shortcuts.linkActivated.connect(self.shortcut_activated)
         text = '<p>'+_('Quick create:')
         for col, name in [('isbn', _('ISBN')), ('formats', _('Formats')),
-                ('last_modified', _('Modified Date')), ('yesno', _('Yes/No')),
+                ('yesno', _('Yes/No')),
                 ('tags', _('Tags')), ('series', _('Series')), ('rating',
                     _('Rating')), ('people', _("People's names"))]:
             text += ' <a href="col:%s">%s</a>,'%(col, name)
@@ -150,7 +150,6 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn):
             'tags': _('My Tags'),
             'series': _('My Series'),
             'rating': _('My Rating'),
-            'last_modified':_('Modified Date'),
             'people': _('People')}[which])
         self.is_names.setChecked(which == 'people')
         if self.composite_box.isVisible():
@@ -158,9 +157,8 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn):
                 {
                     'isbn': '{identifiers:select(isbn)}',
                     'formats': '{formats}',
-                    'last_modified':'''{last_modified:'format_date($, "dd MMM yyyy")'}'''
                     }[which])
-            self.composite_sort_by.setCurrentIndex(2 if which == 'last_modified' else 0)
+            self.composite_sort_by.setCurrentIndex(0)
 
     def datatype_changed(self, *args):
         try:

src/calibre/gui2/preferences/look_feel.py

@@ -5,15 +5,91 @@ __license__   = 'GPL v3'
 __copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
-from PyQt4.Qt import QApplication, QFont, QFontInfo, QFontDialog
+from PyQt4.Qt import (QApplication, QFont, QFontInfo, QFontDialog,
+        QAbstractListModel, Qt)
 
 from calibre.gui2.preferences import ConfigWidgetBase, test_widget, CommaSeparatedList
 from calibre.gui2.preferences.look_feel_ui import Ui_Form
 from calibre.gui2 import config, gprefs, qt_app
-from calibre.utils.localization import available_translations, \
-    get_language, get_lang
+from calibre.utils.localization import (available_translations,
+    get_language, get_lang)
 from calibre.utils.config import prefs
 from calibre.utils.icu import sort_key
+from calibre.gui2 import NONE
+from calibre.gui2.book_details import get_field_list
+
+class DisplayedFields(QAbstractListModel): # {{{
+
+    def __init__(self, db, parent=None):
+        QAbstractListModel.__init__(self, parent)
+
+        self.fields = []
+        self.db = db
+        self.changed = False
+
+    def initialize(self, use_defaults=False):
+        self.fields = [[x[0], x[1]] for x in
+                get_field_list(self.db.field_metadata,
+                    use_defaults=use_defaults)]
+        self.reset()
+        self.changed = True
+
+    def rowCount(self, *args):
+        return len(self.fields)
+
+    def data(self, index, role):
+        try:
+            field, visible = self.fields[index.row()]
+        except:
+            return NONE
+        if role == Qt.DisplayRole:
+            name = field
+            try:
+                name = self.db.field_metadata[field]['name']
+            except:
+                pass
+            if not name:
+                name = field
+            return name
+        if role == Qt.CheckStateRole:
+            return Qt.Checked if visible else Qt.Unchecked
+        return NONE
+
+    def flags(self, index):
+        ans = QAbstractListModel.flags(self, index)
+        return ans | Qt.ItemIsUserCheckable
+
+    def setData(self, index, val, role):
+        ret = False
+        if role == Qt.CheckStateRole:
+            val, ok = val.toInt()
+            if ok:
+                self.fields[index.row()][1] = bool(val)
+                self.changed = True
+                ret = True
+                self.dataChanged.emit(index, index)
+        return ret
+
+    def restore_defaults(self):
+        self.initialize(use_defaults=True)
+
+    def commit(self):
+        if self.changed:
+            gprefs['book_display_fields'] = self.fields
+
+    def move(self, idx, delta):
+        row = idx.row() + delta
+        if row >= 0 and row < len(self.fields):
+            t = self.fields[row]
+            self.fields[row] = self.fields[row-delta]
+            self.fields[row-delta] = t
+            self.dataChanged.emit(idx, idx)
+            idx = self.index(row)
+            self.dataChanged.emit(idx, idx)
+            self.changed = True
+            return idx
+
+# }}}
 
 class ConfigWidget(ConfigWidgetBase, Ui_Form):
 
@@ -76,11 +152,18 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         self.current_font = self.initial_font = None
         self.change_font_button.clicked.connect(self.change_font)
 
+        self.display_model = DisplayedFields(self.gui.current_db,
+                self.field_display_order)
+        self.display_model.dataChanged.connect(self.changed_signal)
+        self.field_display_order.setModel(self.display_model)
+        self.df_up_button.clicked.connect(self.move_df_up)
+        self.df_down_button.clicked.connect(self.move_df_down)
 
     def initialize(self):
         ConfigWidgetBase.initialize(self)
         self.current_font = self.initial_font = gprefs['font']
         self.update_font_display()
+        self.display_model.initialize()
 
     def restore_defaults(self):
         ConfigWidgetBase.restore_defaults(self)
@@ -89,6 +172,8 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         if ofont is not None:
             self.changed_signal.emit()
             self.update_font_display()
+        self.display_model.restore_defaults()
+        self.changed_signal.emit()
 
     def build_font_obj(self):
         font_info = self.current_font
@@ -107,6 +192,24 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         self.font_display.setText(name +
                 ' [%dpt]'%fi.pointSize())
 
+    def move_df_up(self):
+        idx = self.field_display_order.currentIndex()
+        if idx.isValid():
+            idx = self.display_model.move(idx, -1)
+            if idx is not None:
+                sm = self.field_display_order.selectionModel()
+                sm.select(idx, sm.ClearAndSelect)
+                self.field_display_order.setCurrentIndex(idx)
+
+    def move_df_down(self):
+        idx = self.field_display_order.currentIndex()
+        if idx.isValid():
+            idx = self.display_model.move(idx, 1)
+            if idx is not None:
+                sm = self.field_display_order.selectionModel()
+                sm.select(idx, sm.ClearAndSelect)
+                self.field_display_order.setCurrentIndex(idx)
+
     def change_font(self, *args):
         fd = QFontDialog(self.build_font_obj(), self)
         if fd.exec_() == fd.Accepted:
@@ -123,14 +226,16 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
             gprefs['font'] = self.current_font
             QApplication.setFont(self.font_display.font())
             rr = True
+        self.display_model.commit()
         return rr
 
-
     def refresh_gui(self, gui):
         self.update_font_display()
         gui.tags_view.reread_collapse_parameters()
+        gui.library_view.refresh_book_details()
 
 if __name__ == '__main__':
-    app = QApplication([])
+    from calibre.gui2 import Application
+    app = Application([])
     test_widget('Interface', 'Look & Feel')
 

src/calibre/gui2/preferences/look_feel.ui

@@ -7,7 +7,7 @@
     <x>0</x>
     <y>0</y>
     <width>717</width>
-    <height>444</height>
+    <height>390</height>
    </rect>
   </property>
   <property name="windowTitle">
@@ -15,279 +15,400 @@
   </property>
   <layout class="QGridLayout" name="gridLayout_2">
    <item row="0" column="0">
-    <widget class="QLabel" name="label_17">
-     <property name="text">
-      <string>User Interface &amp;layout (needs restart):</string>
+    <widget class="QTabWidget" name="tabWidget">
+     <property name="currentIndex">
+      <number>0</number>
      </property>
-     <property name="buddy">
-      <cstring>opt_gui_layout</cstring>
-     </property>
-    </widget>
-   </item>
-   <item row="0" column="1">
-    <widget class="QComboBox" name="opt_gui_layout">
-     <property name="maximumSize">
-      <size>
-       <width>250</width>
-       <height>16777215</height>
-      </size>
-     </property>
-     <property name="sizeAdjustPolicy">
-      <enum>QComboBox::AdjustToMinimumContentsLengthWithIcon</enum>
-     </property>
-     <property name="minimumContentsLength">
-      <number>20</number>
-     </property>
-    </widget>
-   </item>
-   <item row="1" column="0">
-    <widget class="QLabel" name="label_6">
-     <property name="text">
-      <string>&amp;Number of covers to show in browse mode (needs restart):</string>
-     </property>
-     <property name="buddy">
-      <cstring>opt_cover_flow_queue_length</cstring>
-     </property>
-    </widget>
-   </item>
-   <item row="1" column="1">
-    <widget class="QSpinBox" name="opt_cover_flow_queue_length"/>
-   </item>
-   <item row="2" column="0">
-    <widget class="QLabel" name="label_7">
-     <property name="text">
-      <string>Choose &amp;language (requires restart):</string>
-     </property>
-     <property name="buddy">
-      <cstring>opt_language</cstring>
-     </property>
-    </widget>
-   </item>
-   <item row="2" column="1">
-    <widget class="QComboBox" name="opt_language">
-     <property name="sizeAdjustPolicy">
-      <enum>QComboBox::AdjustToMinimumContentsLengthWithIcon</enum>
-     </property>
-     <property name="minimumContentsLength">
-      <number>20</number>
-     </property>
-    </widget>
-   </item>
-   <item row="3" column="0">
-    <widget class="QCheckBox" name="opt_show_avg_rating">
-     <property name="text">
-      <string>Show &amp;average ratings in the tags browser</string>
-     </property>
-     <property name="checked">
-      <bool>true</bool>
-     </property>
-    </widget>
-   </item>
-   <item row="3" column="1">
-    <widget class="QCheckBox" name="opt_disable_animations">
-     <property name="toolTip">
-      <string>Disable all animations. Useful if you have a slow/old computer.</string>
-     </property>
-     <property name="text">
-      <string>Disable &amp;animations</string>
-     </property>
-    </widget>
-   </item>
-   <item row="4" column="0">
-    <widget class="QCheckBox" name="opt_systray_icon">
-     <property name="text">
-      <string>Enable system &amp;tray icon (needs restart)</string>
-     </property>
-    </widget>
-   </item>
-   <item row="4" column="1">
-    <widget class="QCheckBox" name="opt_show_splash_screen">
-     <property name="text">
-      <string>Show &amp;splash screen at startup</string>
-     </property>
-    </widget>
-   </item>
-   <item row="5" column="0">
-    <widget class="QCheckBox" name="opt_disable_tray_notification">
-     <property name="text">
-      <string>Disable &amp;notifications in system tray</string>
-     </property>
-    </widget>
-   </item>
-   <item row="5" column="1">
-    <widget class="QCheckBox" name="opt_use_roman_numerals_for_series_number">
-     <property name="text">
-      <string>Use &amp;Roman numerals for series</string>
-     </property>
-     <property name="checked">
-      <bool>true</bool>
-     </property>
-    </widget>
-   </item>
-   <item row="6" column="0" colspan="2">
-    <widget class="QCheckBox" name="opt_separate_cover_flow">
-     <property name="text">
-      <string>Show cover &amp;browser in a separate window (needs restart)</string>
-     </property>
-    </widget>
-   </item>
-   <item row="7" column="0" colspan="2">
-    <layout class="QHBoxLayout">
-     <item>
-      <widget class="QLabel" name="label_6">
-       <property name="text">
-        <string>Tags browser category &amp;partitioning method:</string>
-       </property>
-       <property name="buddy">
-        <cstring>opt_tags_browser_partition_method</cstring>
-       </property>
-      </widget>
-     </item>
-     <item>
-      <widget class="QComboBox" name="opt_tags_browser_partition_method">
-       <property name="toolTip">
-        <string>Choose how tag browser subcategories are displayed when
+     <widget class="QWidget" name="tab">
+      <attribute name="icon">
+       <iconset resource="../../../../resources/images.qrc">
+        <normaloff>:/images/lt.png</normaloff>:/images/lt.png</iconset>
+      </attribute>
+      <attribute name="title">
+       <string>Main Interface</string>
+      </attribute>
+      <layout class="QGridLayout" name="gridLayout_9">
+       <item row="0" column="0">
+        <widget class="QLabel" name="label_17">
+         <property name="text">
+          <string>User Interface &amp;layout (needs restart):</string>
+         </property>
+         <property name="buddy">
+          <cstring>opt_gui_layout</cstring>
+         </property>
+        </widget>
+       </item>
+       <item row="0" column="1">
+        <widget class="QComboBox" name="opt_gui_layout">
+         <property name="maximumSize">
+          <size>
+           <width>250</width>
+           <height>16777215</height>
+          </size>
+         </property>
+         <property name="sizeAdjustPolicy">
+          <enum>QComboBox::AdjustToMinimumContentsLengthWithIcon</enum>
+         </property>
+         <property name="minimumContentsLength">
+          <number>20</number>
+         </property>
+        </widget>
+       </item>
+       <item row="1" column="0">
+        <widget class="QLabel" name="label_7">
+         <property name="text">
+          <string>Choose &amp;language (requires restart):</string>
+         </property>
+         <property name="buddy">
+          <cstring>opt_language</cstring>
+         </property>
+        </widget>
+       </item>
+       <item row="1" column="1">
+        <widget class="QComboBox" name="opt_language">
+         <property name="sizeAdjustPolicy">
+          <enum>QComboBox::AdjustToMinimumContentsLengthWithIcon</enum>
+         </property>
+         <property name="minimumContentsLength">
+          <number>20</number>
+         </property>
+        </widget>
+       </item>
+       <item row="2" column="0">
+        <widget class="QCheckBox" name="opt_systray_icon">
+         <property name="text">
+          <string>Enable system &amp;tray icon (needs restart)</string>
+         </property>
+        </widget>
+       </item>
+       <item row="2" column="1">
+        <widget class="QCheckBox" name="opt_disable_animations">
+         <property name="toolTip">
+          <string>Disable all animations. Useful if you have a slow/old computer.</string>
+         </property>
+         <property name="text">
+          <string>Disable &amp;animations</string>
+         </property>
+        </widget>
+       </item>
+       <item row="3" column="0">
+        <widget class="QCheckBox" name="opt_disable_tray_notification">
+         <property name="text">
+          <string>Disable &amp;notifications in system tray</string>
+         </property>
+        </widget>
+       </item>
+       <item row="3" column="1">
+        <widget class="QCheckBox" name="opt_show_splash_screen">
+         <property name="text">
+          <string>Show &amp;splash screen at startup</string>
+         </property>
+        </widget>
+       </item>
+       <item row="5" column="0" colspan="2">
+        <widget class="QGroupBox" name="groupBox_2">
+         <property name="title">
+          <string>&amp;Toolbar</string>
+         </property>
+         <layout class="QGridLayout" name="gridLayout_8">
+          <item row="0" column="1">
+           <widget class="QComboBox" name="opt_toolbar_icon_size"/>
+          </item>
+          <item row="0" column="0">
+           <widget class="QLabel" name="label_5">
+            <property name="text">
+             <string>&amp;Icon size:</string>
+            </property>
+            <property name="buddy">
+             <cstring>opt_toolbar_icon_size</cstring>
+            </property>
+           </widget>
+          </item>
+          <item row="1" column="1">
+           <widget class="QComboBox" name="opt_toolbar_text"/>
+          </item>
+          <item row="1" column="0">
+           <widget class="QLabel" name="label_8">
+            <property name="text">
+             <string>Show &amp;text under icons:</string>
+            </property>
+            <property name="buddy">
+             <cstring>opt_toolbar_text</cstring>
+            </property>
+           </widget>
+          </item>
+         </layout>
+        </widget>
+       </item>
+       <item row="4" column="0">
+        <layout class="QHBoxLayout" name="horizontalLayout">
+         <item>
+          <widget class="QLabel" name="label_2">
+           <property name="text">
+            <string>Interface font:</string>
+           </property>
+           <property name="buddy">
+            <cstring>font_display</cstring>
+           </property>
+          </widget>
+         </item>
+         <item>
+          <widget class="QLineEdit" name="font_display">
+           <property name="readOnly">
+            <bool>true</bool>
+           </property>
+          </widget>
+         </item>
+        </layout>
+       </item>
+       <item row="4" column="1">
+        <widget class="QPushButton" name="change_font_button">
+         <property name="text">
+          <string>Change &amp;font (needs restart)</string>
+         </property>
+        </widget>
+       </item>
+       <item row="6" column="0">
+        <spacer name="verticalSpacer_3">
+         <property name="orientation">
+          <enum>Qt::Vertical</enum>
+         </property>
+         <property name="sizeHint" stdset="0">
+          <size>
+           <width>20</width>
+           <height>40</height>
+          </size>
+         </property>
+        </spacer>
+       </item>
+      </layout>
+     </widget>
+     <widget class="QWidget" name="tab_4">
+      <attribute name="icon">
+       <iconset resource="../../../../resources/images.qrc">
+        <normaloff>:/images/book.png</normaloff>:/images/book.png</iconset>
+      </attribute>
+      <attribute name="title">
+       <string>Book Details</string>
+      </attribute>
+      <layout class="QGridLayout" name="gridLayout_12">
+       <item row="0" column="0" rowspan="2">
+        <widget class="QGroupBox" name="groupBox">
+         <property name="title">
+          <string>Select displayed metadata</string>
+         </property>
+         <layout class="QGridLayout" name="gridLayout_3">
+          <item row="0" column="0" rowspan="3">
+           <widget class="QListView" name="field_display_order">
+            <property name="alternatingRowColors">
+             <bool>true</bool>
+            </property>
+           </widget>
+          </item>
+          <item row="0" column="1">
+           <widget class="QToolButton" name="df_up_button">
+            <property name="toolTip">
+             <string>Move up</string>
+            </property>
+            <property name="icon">
+             <iconset resource="../../../../resources/images.qrc">
+              <normaloff>:/images/arrow-up.png</normaloff>:/images/arrow-up.png</iconset>
+            </property>
+           </widget>
+          </item>
+          <item row="2" column="1">
+           <widget class="QToolButton" name="df_down_button">
+            <property name="toolTip">
+             <string>Move down</string>
+            </property>
+            <property name="icon">
+             <iconset resource="../../../../resources/images.qrc">
+              <normaloff>:/images/arrow-down.png</normaloff>:/images/arrow-down.png</iconset>
+            </property>
+           </widget>
+          </item>
+          <item row="1" column="1">
+           <spacer name="verticalSpacer_5">
+            <property name="orientation">
+             <enum>Qt::Vertical</enum>
+            </property>
+            <property name="sizeHint" stdset="0">
+             <size>
+              <width>20</width>
+              <height>40</height>
+             </size>
+            </property>
+           </spacer>
+          </item>
+         </layout>
+        </widget>
+       </item>
+       <item row="0" column="1">
+        <widget class="QCheckBox" name="opt_use_roman_numerals_for_series_number">
+         <property name="text">
+          <string>Use &amp;Roman numerals for series</string>
+         </property>
+         <property name="checked">
+          <bool>true</bool>
+         </property>
+        </widget>
+       </item>
+       <item row="1" column="1">
+        <widget class="QLabel" name="label_3">
+         <property name="text">
+          <string>Note that &lt;b&gt;comments&lt;/b&gt; will always be displayed at the end, regardless of the position you assign here.</string>
+         </property>
+         <property name="wordWrap">
+          <bool>true</bool>
+         </property>
+        </widget>
+       </item>
+      </layout>
+     </widget>
+     <widget class="QWidget" name="tab_2">
+      <attribute name="icon">
+       <iconset resource="../../../../resources/images.qrc">
+        <normaloff>:/images/tags.png</normaloff>:/images/tags.png</iconset>
+      </attribute>
+      <attribute name="title">
+       <string>Tag Browser</string>
+      </attribute>
+      <layout class="QGridLayout" name="gridLayout_10">
+       <item row="0" column="0" colspan="2">
+        <widget class="QLabel" name="label_9">
+         <property name="text">
+          <string>Tags browser category &amp;partitioning method:</string>
+         </property>
+         <property name="buddy">
+          <cstring>opt_tags_browser_partition_method</cstring>
+         </property>
+        </widget>
+       </item>
+       <item row="0" column="2">
+        <widget class="QComboBox" name="opt_tags_browser_partition_method">
+         <property name="toolTip">
+          <string>Choose how tag browser subcategories are displayed when
 there are more items than the limit. Select by first
 letter to see an A, B, C list. Choose partitioned to
 have a list of fixed-sized groups. Set to disabled
 if you never want subcategories</string>
-       </property>
-      </widget>
-     </item>
-     <item>
-      <widget class="QLabel" name="label_6">
-       <property name="text">
-        <string>&amp;Collapse when more items than:</string>
-       </property>
-       <property name="buddy">
-        <cstring>opt_tags_browser_collapse_at</cstring>
-       </property>
-      </widget>
-     </item>
-     <item>
-      <widget class="QSpinBox" name="opt_tags_browser_collapse_at">
-       <property name="toolTip">
-        <string>If a Tag Browser category has more than this number of items, it is divided
+         </property>
+        </widget>
+       </item>
+       <item row="0" column="3">
+        <widget class="QLabel" name="label_10">
+         <property name="text">
+          <string>&amp;Collapse when more items than:</string>
+         </property>
+         <property name="buddy">
+          <cstring>opt_tags_browser_collapse_at</cstring>
+         </property>
+        </widget>
+       </item>
+       <item row="0" column="4">
+        <widget class="QSpinBox" name="opt_tags_browser_collapse_at">
+         <property name="toolTip">
+          <string>If a Tag Browser category has more than this number of items, it is divided
 up into sub-categories. If the partition method is set to disable, this value is ignored.</string>
-       </property>
-       <property name="maximum">
-        <number>10000</number>
-       </property>
-      </widget>
-     </item>
-     <item>
-      <spacer name="verticalSpacer">
-       <property name="orientation">
-        <enum>Qt::Horizontal</enum>
-       </property>
-       <property name="sizeHint" stdset="0">
-        <size>
-         <width>20</width>
-         <height>5</height>
-        </size>
-       </property>
-      </spacer>
-     </item>
-    </layout>
-   </item>
-   <item row="8" column="0">
-    <widget class="QLabel" name="label_81">
-     <property name="text">
-      <string>Categories with &amp;hierarchical items:</string>
-     </property>
-     <property name="buddy">
-      <cstring>opt_categories_using_hierarchy</cstring>
-     </property>
-    </widget>
-   </item>
-   <item row="8" column="1">
-    <widget class="MultiCompleteLineEdit" name="opt_categories_using_hierarchy">
-     <property name="toolTip">
-      <string>A comma-separated list of columns in which items containing
+         </property>
+         <property name="maximum">
+          <number>10000</number>
+         </property>
+        </widget>
+       </item>
+       <item row="1" column="0" colspan="5">
+        <widget class="QCheckBox" name="opt_show_avg_rating">
+         <property name="text">
+          <string>Show &amp;average ratings in the tags browser</string>
+         </property>
+         <property name="checked">
+          <bool>true</bool>
+         </property>
+        </widget>
+       </item>
+       <item row="2" column="0">
+        <widget class="QLabel" name="label_81">
+         <property name="text">
+          <string>Categories with &amp;hierarchical items:</string>
+         </property>
+         <property name="buddy">
+          <cstring>opt_categories_using_hierarchy</cstring>
+         </property>
+        </widget>
+       </item>
+       <item row="3" column="0" colspan="5">
+        <spacer name="verticalSpacer_2">
+         <property name="orientation">
+          <enum>Qt::Vertical</enum>
+         </property>
+         <property name="sizeHint" stdset="0">
+          <size>
+           <width>690</width>
+           <height>252</height>
+          </size>
+         </property>
+        </spacer>
+       </item>
+       <item row="2" column="2" colspan="3">
+        <widget class="MultiCompleteLineEdit" name="opt_categories_using_hierarchy">
+         <property name="toolTip">
+          <string>A comma-separated list of columns in which items containing
 periods are displayed in the tag browser trees. For example, if
 this box contains 'tags' then tags of the form 'Mystery.English'
 and 'Mystery.Thriller' will be displayed with English and Thriller
 both under 'Mystery'. If 'tags' is not in this box,
 then the tags will be displayed each on their own line.</string>
-     </property>
+         </property>
+        </widget>
+       </item>
+      </layout>
+     </widget>
+     <widget class="QWidget" name="tab_3">
+      <attribute name="icon">
+       <iconset resource="../../../../resources/images.qrc">
+        <normaloff>:/images/cover_flow.png</normaloff>:/images/cover_flow.png</iconset>
+      </attribute>
+      <attribute name="title">
+       <string>Cover Browser</string>
+      </attribute>
+      <layout class="QGridLayout" name="gridLayout_11">
+       <item row="0" column="0" colspan="2">
+        <widget class="QCheckBox" name="opt_separate_cover_flow">
+         <property name="text">
+          <string>Show cover &amp;browser in a separate window (needs restart)</string>
+         </property>
+        </widget>
+       </item>
+       <item row="1" column="0">
+        <widget class="QLabel" name="label_6">
+         <property name="text">
+          <string>&amp;Number of covers to show in browse mode (needs restart):</string>
+         </property>
+         <property name="buddy">
+          <cstring>opt_cover_flow_queue_length</cstring>
+         </property>
+        </widget>
+       </item>
+       <item row="1" column="1">
+        <widget class="QSpinBox" name="opt_cover_flow_queue_length"/>
+       </item>
+       <item row="2" column="0" colspan="2">
+        <spacer name="verticalSpacer_4">
+         <property name="orientation">
+          <enum>Qt::Vertical</enum>
+         </property>
+         <property name="sizeHint" stdset="0">
+          <size>
+           <width>690</width>
+           <height>283</height>
+          </size>
+         </property>
+        </spacer>
+       </item>
+      </layout>
+     </widget>
     </widget>
    </item>
-   <item row="15" column="0" colspan="2">
-    <widget class="QGroupBox" name="groupBox_2">
-     <property name="title">
-      <string>&amp;Toolbar</string>
-     </property>
-     <layout class="QGridLayout" name="gridLayout">
-      <item row="0" column="1">
-       <widget class="QComboBox" name="opt_toolbar_icon_size"/>
-      </item>
-      <item row="0" column="0">
-       <widget class="QLabel" name="label">
-        <property name="text">
-         <string>&amp;Icon size:</string>
-        </property>
-        <property name="buddy">
-         <cstring>opt_toolbar_icon_size</cstring>
-        </property>
-       </widget>
-      </item>
-      <item row="1" column="1">
-       <widget class="QComboBox" name="opt_toolbar_text"/>
-      </item>
-      <item row="1" column="0">
-       <widget class="QLabel" name="label_4">
-        <property name="text">
-         <string>Show &amp;text under icons:</string>
-        </property>
-        <property name="buddy">
-         <cstring>opt_toolbar_text</cstring>
-        </property>
-       </widget>
-      </item>
-     </layout>
-    </widget>
-   </item>
-   <item row="16" column="0">
-    <layout class="QHBoxLayout" name="horizontalLayout">
-     <item>
-      <widget class="QLabel" name="label_2">
-       <property name="text">
-        <string>Interface font:</string>
-       </property>
-       <property name="buddy">
-        <cstring>font_display</cstring>
-       </property>
-      </widget>
-     </item>
-     <item>
-      <widget class="QLineEdit" name="font_display">
-       <property name="readOnly">
-        <bool>true</bool>
-       </property>
-      </widget>
-     </item>
-    </layout>
-   </item>
-   <item row="16" column="1">
-    <widget class="QPushButton" name="change_font_button">
-     <property name="text">
-      <string>Change &amp;font (needs restart)</string>
-     </property>
-    </widget>
-   </item>
-   <item row="17" column="0" colspan="2">
-    <spacer name="verticalSpacer">
-     <property name="orientation">
-      <enum>Qt::Vertical</enum>
-     </property>
-     <property name="sizeHint" stdset="0">
-      <size>
-       <width>20</width>
-       <height>40</height>
-      </size>
-     </property>
-    </spacer>
-   </item>
   </layout>
  </widget>
  <customwidgets>
@@ -297,6 +418,8 @@ then the tags will be displayed each on their own line.</string>
    <header>calibre/gui2/complete.h</header>
   </customwidget>
  </customwidgets>
- <resources/>
+ <resources>
+  <include location="../../../../resources/images.qrc"/>
+ </resources>
  <connections/>
 </ui>

src/calibre/gui2/preferences/main.py

@@ -89,7 +89,7 @@ class Category(QWidget): # {{{
         self.bar = QToolBar(self)
         self.bar.setStyleSheet(
                 'QToolBar { border: none; background: none }')
-        self.bar.setIconSize(QSize(48, 48))
+        self.bar.setIconSize(QSize(32, 32))
         self.bar.setMovable(False)
         self.bar.setFloatable(False)
         self.bar.setToolButtonStyle(Qt.ToolButtonTextUnderIcon)

src/calibre/gui2/preferences/metadata_sources.py

@@ -209,8 +209,11 @@ class FieldsModel(QAbstractListModel): # {{{
         return ret
 
     def commit(self):
-        val = [k for k, v in self.overrides.iteritems() if v == Qt.Unchecked]
-        msprefs['ignore_fields'] = val
+        ignored_fields = set([x for x in msprefs['ignore_fields'] if x not in
+            self.overrides])
+        changed = set([k for k, v in self.overrides.iteritems() if v ==
+            Qt.Unchecked])
+        msprefs['ignore_fields'] = list(ignored_fields.union(changed))
 
 
 # }}}

src/calibre/gui2/preferences/template_functions.py

@@ -74,9 +74,10 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
 
     def initialize(self):
         try:
-            with open(P('template-functions.json'), 'rb') as f:
-                self.builtin_source_dict = json.load(f, encoding='utf-8')
+            self.builtin_source_dict = json.loads(P('template-functions.json', data=True,
+                allow_user_override=False).decode('utf-8'))
         except:
+            traceback.print_exc()
             self.builtin_source_dict = {}
 
         self.funcs = formatter_functions.get_functions()
@@ -186,7 +187,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         self.argument_count.setValue(func.arg_count)
         self.documentation.setText(func.doc)
         if txt in self.builtins:
-            if hasattr(func, 'program_text'):
+            if hasattr(func, 'program_text') and func.program_text:
                 self.program.setPlainText(func.program_text)
             elif txt in self.builtin_source_dict:
                 self.program.setPlainText(self.builtin_source_dict[txt])

src/calibre/gui2/search_box.py

@@ -7,12 +7,15 @@ __copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
 import re
+from functools import partial
+
 
 from PyQt4.Qt import QComboBox, Qt, QLineEdit, QStringList, pyqtSlot, QDialog, \
                      pyqtSignal, QCompleter, QAction, QKeySequence, QTimer, \
-                     QString, QIcon
+                     QString, QIcon, QMenu
 
-from calibre.gui2 import config
+from calibre.gui2 import config, error_dialog
+from calibre.gui2.dialogs.confirm_delete import confirm
 from calibre.gui2.dialogs.saved_search_editor import SavedSearchEditor
 from calibre.gui2.dialogs.search import SearchDialog
 from calibre.utils.search_query_parser import saved_searches
@@ -330,6 +333,24 @@ class SavedSearchBox(QComboBox): # {{{
         self.saved_search_selected (name)
         self.changed.emit()
 
+    def delete_current_search(self):
+        idx = self.currentIndex()
+        if idx <= 0:
+            error_dialog(self, _('Delete current search'),
+                         _('No search is selected'), show=True)
+            return
+        if not confirm('<p>'+_('The selected search will be '
+                       '<b>permanently deleted</b>. Are you sure?')
+                    +'</p>', 'saved_search_delete', self):
+            return
+        ss = saved_searches().lookup(unicode(self.currentText()))
+        if ss is None:
+            return
+        saved_searches().delete(unicode(self.currentText()))
+        self.clear()
+        self.search_box.clear()
+        self.changed.emit()
+
     # SIGNALed from the main UI
     def copy_search_button_clicked (self):
         idx = self.currentIndex();
@@ -428,6 +449,22 @@ class SavedSearchBoxMixin(object): # {{{
         for x in ('copy', 'save'):
             b = getattr(self, x+'_search_button')
             b.setStatusTip(b.toolTip())
+        self.save_search_button.setToolTip('<p>' +
+         _("Save current search under the name shown in the box. "
+           "Press and hold for a pop-up options menu.") + '</p>')
+        self.save_search_button.setMenu(QMenu())
+        self.save_search_button.menu().addAction(
+                            QIcon(I('plus.png')),
+                            _('Create saved search'),
+                            self.saved_search.save_search_button_clicked)
+        self.save_search_button.menu().addAction(
+                             QIcon(I('trash.png')),
+                             _('Delete saved search'),
+                            self.saved_search.delete_current_search)
+        self.save_search_button.menu().addAction(
+                             QIcon(I('search.png')),
+                            _('Manage saved searches'),
+                            partial(self.do_saved_search_edit, None))
 
     def saved_searches_changed(self, set_restriction=None, recount=True):
         p = sorted(saved_searches().names(), key=sort_key)

src/calibre/gui2/store/amazon_de_plugin.py

@@ -0,0 +1,34 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import (unicode_literals, division, absolute_import, print_function)
+
+__license__ = 'GPL 3'
+__copyright__ = '2011, John Schember <john@nachtimwald.com>'
+__docformat__ = 'restructuredtext en'
+
+from PyQt4.Qt import QUrl
+
+from calibre.gui2 import open_url
+from calibre.gui2.store.amazon_plugin import AmazonKindleStore
+
+class AmazonDEKindleStore(AmazonKindleStore):
+    '''
+    For comments on the implementation, please see amazon_plugin.py
+    '''
+
+    search_url = 'http://www.amazon.de/s/url=search-alias%3Ddigital-text&field-keywords='
+    details_url = 'http://amazon.de/dp/'
+    drm_search_text = u'Gleichzeitige Verwendung von Geräten'
+    drm_free_text = u'Keine Einschränkung'
+
+    def open(self, parent=None, detail_item=None, external=False):
+        aff_id = {'tag': 'charhale0a-21'}
+        store_link = ('http://www.amazon.de/gp/redirect.html?ie=UTF8&site-redirect=de'
+                     '&tag=%(tag)s&linkCode=ur2&camp=1638&creative=19454'
+                     '&location=http://www.amazon.de/ebooks-kindle/b?node=530886031') % aff_id
+        if detail_item:
+            aff_id['asin'] = detail_item
+            store_link = ('http://www.amazon.de/gp/redirect.html?ie=UTF8'
+                          '&location=http://www.amazon.de/dp/%(asin)s&site-redirect=de'
+                          '&tag=%(tag)s&linkCode=ur2&camp=1638&creative=6742') % aff_id
+        open_url(QUrl(store_link))

src/calibre/gui2/store/amazon_plugin.py

@@ -22,6 +22,11 @@ from calibre.gui2.store.search_result import SearchResult
 
 class AmazonKindleStore(StorePlugin):
 
+    search_url = 'http://www.amazon.com/s/url=search-alias%3Ddigital-text&field-keywords='
+    details_url = 'http://amazon.com/dp/'
+    drm_search_text = u'Simultaneous Device Usage'
+    drm_free_text = u'Unlimited'
+
     def open(self, parent=None, detail_item=None, external=False):
         '''
         Amazon comes with a number of difficulties.
@@ -117,7 +122,7 @@ class AmazonKindleStore(StorePlugin):
         open_url(QUrl(store_link))
 
     def search(self, query, max_results=10, timeout=60):
-        url = 'http://www.amazon.com/s/url=search-alias%3Ddigital-text&field-keywords=' + urllib2.quote(query)
+        url =  self.search_url + urllib2.quote(query)
         br = browser()
 
         counter = max_results
@@ -180,18 +185,19 @@ class AmazonKindleStore(StorePlugin):
                 yield s
 
     def get_details(self, search_result, timeout):
-        url = 'http://amazon.com/dp/'
+        url = self.details_url
 
         br = browser()
         with closing(br.open(url + search_result.detail_item, timeout=timeout)) as nf:
             idata = html.fromstring(nf.read())
-            if idata.xpath('boolean(//div[@class="content"]//li/b[contains(text(), "Simultaneous Device Usage")])'):
-                if idata.xpath('boolean(//div[@class="content"]//li[contains(., "Unlimited") and contains(b, "Simultaneous Device Usage")])'):
+            if idata.xpath('boolean(//div[@class="content"]//li/b[contains(text(), "' +
+                           self.drm_search_text + '")])'):
+                if idata.xpath('boolean(//div[@class="content"]//li[contains(., "' +
+                               self.drm_free_text + '") and contains(b, "' +
+                               self.drm_search_text + '")])'):
                     search_result.drm = SearchResult.DRM_UNLOCKED
                 else:
                     search_result.drm = SearchResult.DRM_UNKNOWN
             else:
                 search_result.drm = SearchResult.DRM_LOCKED
         return True
-
-

src/calibre/gui2/store/amazon_uk_plugin.py

@@ -0,0 +1,30 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import (unicode_literals, division, absolute_import, print_function)
+
+__license__ = 'GPL 3'
+__copyright__ = '2011, John Schember <john@nachtimwald.com>'
+__docformat__ = 'restructuredtext en'
+
+
+from PyQt4.Qt import QUrl
+
+from calibre.gui2 import open_url
+from calibre.gui2.store.amazon_plugin import AmazonKindleStore
+
+class AmazonUKKindleStore(AmazonKindleStore):
+    '''
+    For comments on the implementation, please see amazon_plugin.py
+    '''
+
+    search_url = 'http://www.amazon.co.uk/s/url=search-alias%3Ddigital-text&field-keywords='
+    details_url = 'http://amazon.co.uk/dp/'
+
+    def open(self, parent=None, detail_item=None, external=False):
+        aff_id = {'tag': 'calcharles-21'}
+        store_link = 'http://www.amazon.co.uk/gp/redirect.html?ie=UTF8&location=http://www.amazon.co.uk/Kindle-eBooks/b?ie=UTF8&node=341689031&ref_=sa_menu_kbo2&tag=%(tag)s&linkCode=ur2&camp=1634&creative=19450' % aff_id
+
+        if detail_item:
+            aff_id['asin'] = detail_item
+            store_link = 'http://www.amazon.co.uk/gp/redirect.html?ie=UTF8&location=http://www.amazon.co.uk/dp/%(asin)s&tag=%(tag)s&linkCode=ur2&camp=1634&creative=6738' % aff_id
+        open_url(QUrl(store_link))

src/calibre/gui2/store/foyles_uk_plugin.py

@@ -0,0 +1,78 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import (unicode_literals, division, absolute_import, print_function)
+
+__license__ = 'GPL 3'
+__copyright__ = '2011, John Schember <john@nachtimwald.com>'
+__docformat__ = 'restructuredtext en'
+
+import urllib2
+from contextlib import closing
+
+from lxml import html
+
+from PyQt4.Qt import QUrl
+
+from calibre import browser, url_slash_cleaner
+from calibre.gui2 import open_url
+from calibre.gui2.store import StorePlugin
+from calibre.gui2.store.basic_config import BasicStoreConfig
+from calibre.gui2.store.search_result import SearchResult
+from calibre.gui2.store.web_store_dialog import WebStoreDialog
+
+class FoylesUKStore(BasicStoreConfig, StorePlugin):
+
+    def open(self, parent=None, detail_item=None, external=False):
+        url = 'http://www.awin1.com/cread.php?awinmid=1414&awinaffid=120917&clickref=&p='
+        url_redirect = 'http://www.foyles.co.uk'
+
+        if external or self.config.get('open_external', False):
+            if detail_item:
+                url = url + url_redirect + detail_item
+            open_url(QUrl(url_slash_cleaner(url)))
+        else:
+            detail_url = None
+            if detail_item:
+                detail_url = url + url_redirect + detail_item
+            d = WebStoreDialog(self.gui, url, parent, detail_url)
+            d.setWindowTitle(self.name)
+            d.set_tags(self.config.get('tags', ''))
+            d.exec_()
+
+    def search(self, query, max_results=10, timeout=60):
+        url = 'http://www.foyles.co.uk/Public/Shop/Search.aspx?fFacetId=1015&searchBy=1&quick=true&term=' + urllib2.quote(query)
+
+        br = browser()
+
+        counter = max_results
+        with closing(br.open(url, timeout=timeout)) as f:
+            doc = html.fromstring(f.read())
+            for data in doc.xpath('//table[contains(@id, "MainContent")]/tr/td/div[contains(@class, "Item")]'):
+                if counter <= 0:
+                    break
+                id = ''.join(data.xpath('.//a[@class="Title"]/@href')).strip()
+                if not id:
+                    continue
+
+                cover_url = ''.join(data.xpath('.//a[@class="Jacket"]/img/@src'))
+                if cover_url:
+                    cover_url = 'http://www.foyles.co.uk' + cover_url
+                #print(cover_url)
+
+                title = ''.join(data.xpath('.//a[@class="Title"]/text()'))
+                author = ', '.join(data.xpath('.//span[@class="Author"]/text()'))
+                price = ''.join(data.xpath('./ul/li[@class="Strong"]/text()'))
+                price = price[price.rfind(' '):]
+
+                counter -= 1
+
+                s = SearchResult()
+                s.cover_url = cover_url
+                s.title = title.strip()
+                s.author = author.strip()
+                s.price = price
+                s.detail_item = id
+                s.drm = SearchResult.DRM_LOCKED
+                s.formats = 'EPUB'
+
+                yield s

src/calibre/gui2/store/waterstones_uk_plugin.py

@@ -0,0 +1,84 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import (unicode_literals, division, absolute_import, print_function)
+
+__license__ = 'GPL 3'
+__copyright__ = '2011, John Schember <john@nachtimwald.com>'
+__docformat__ = 'restructuredtext en'
+
+import urllib2
+from contextlib import closing
+
+from lxml import html
+
+from PyQt4.Qt import QUrl
+
+from calibre import browser
+from calibre.gui2 import open_url
+from calibre.gui2.store import StorePlugin
+from calibre.gui2.store.basic_config import BasicStoreConfig
+from calibre.gui2.store.search_result import SearchResult
+from calibre.gui2.store.web_store_dialog import WebStoreDialog
+
+class WaterstonesUKStore(BasicStoreConfig, StorePlugin):
+
+    def open(self, parent=None, detail_item=None, external=False):
+        url = 'http://clkuk.tradedoubler.com/click?p=51196&a=1951604&g=19333484'
+        url_details = 'http://clkuk.tradedoubler.com/click?p(51196)a(1951604)g(16460516)url({0})'
+
+        if external or self.config.get('open_external', False):
+            if detail_item:
+                url = url_details.format(detail_item)
+            open_url(QUrl(url))
+        else:
+            detail_url = None
+            if detail_item:
+                detail_url = url_details.format(detail_item)
+            d = WebStoreDialog(self.gui, url, parent, detail_url)
+            d.setWindowTitle(self.name)
+            d.set_tags(self.config.get('tags', ''))
+            d.exec_()
+
+    def search(self, query, max_results=10, timeout=60):
+        url = 'http://www.waterstones.com/waterstonesweb/advancedSearch.do?buttonClicked=1&format=3757&bookkeywords=' + urllib2.quote(query)
+
+        br = browser()
+
+        counter = max_results
+        with closing(br.open(url, timeout=timeout)) as f:
+            doc = html.fromstring(f.read())
+            for data in doc.xpath('//div[contains(@class, "results-pane")]'):
+                if counter <= 0:
+                    break
+
+                id = ''.join(data.xpath('./div/div/h2/a/@href')).strip()
+                if not id:
+                    continue
+                cover_url = ''.join(data.xpath('.//div[@class="image"]/a/img/@src'))
+                title = ''.join(data.xpath('./div/div/h2/a/text()'))
+                author = ', '.join(data.xpath('.//p[@class="byAuthor"]/a/text()'))
+                price = ''.join(data.xpath('.//p[@class="price"]/span[@class="priceStandard"]/text()'))
+                drm = data.xpath('boolean(.//td[@headers="productFormat" and contains(., "DRM")])')
+                pdf = data.xpath('boolean(.//td[@headers="productFormat" and contains(., "PDF")])')
+                epub = data.xpath('boolean(.//td[@headers="productFormat" and contains(., "EPUB")])')
+
+                counter -= 1
+
+                s = SearchResult()
+                s.cover_url = cover_url
+                s.title = title.strip()
+                s.author = author.strip()
+                s.price = price
+                if drm:
+                    s.drm = SearchResult.DRM_LOCKED
+                else:
+                    s.drm = SearchResult.DRM_UNKNOWN
+                s.detail_item = id
+                formats = []
+                if epub:
+                    formats.append('EPUB')
+                if pdf:
+                    formats.append('PDF')
+                s.formats = ', '.join(formats)
+
+                yield s

src/calibre/gui2/threaded_jobs.py

@@ -100,7 +100,8 @@ class ThreadedJob(BaseJob):
         try:
             self.consolidate_log()
         except:
-            self.log.exception('Log consolidation failed')
+            if self.log is not None:
+                self.log.exception('Log consolidation failed')
 
         # No need to keep references to these around anymore
         self.func = self.args = self.kwargs = self.notifications = None
@@ -112,7 +113,7 @@ class ThreadedJob(BaseJob):
             self.start_time = time.time()
             self.duration = 0.0001
         else:
-            self.duration = time.time() - self.start_time()
+            self.duration = time.time() - self.start_time
             self.abort.set()
 
         self.log('Aborted job:', self.description)

src/calibre/gui2/ui.py

@@ -278,11 +278,8 @@ class Main(MainWindow, MainWindowMixin, DeviceMixin, EmailMixin, # {{{
         self.library_view.model().count_changed_signal.connect(
                 self.iactions['Choose Library'].count_changed)
         if not gprefs.get('quick_start_guide_added', False):
-            from calibre.ebooks.metadata import MetaInformation
-            mi = MetaInformation(_('Calibre Quick Start Guide'), ['John Schember'])
-            mi.author_sort = 'Schember, John'
-            mi.comments = "A guide to get you up and running with calibre"
-            mi.publisher = 'calibre'
+            from calibre.ebooks.metadata.meta import get_metadata
+            mi = get_metadata(open(P('quick_start.epub'), 'rb'), 'epub')
             self.library_view.model().add_books([P('quick_start.epub')], ['epub'],
                     [mi])
             gprefs['quick_start_guide_added'] = True
@@ -449,6 +446,8 @@ class Main(MainWindow, MainWindowMixin, DeviceMixin, EmailMixin, # {{{
             self.library_view.model().refresh()
             self.library_view.model().research()
             self.tags_view.recount()
+        elif msg.startswith('shutdown:'):
+            self.quit(confirm_quit=False)
         else:
             print msg
 
@@ -602,8 +601,9 @@ class Main(MainWindow, MainWindowMixin, DeviceMixin, EmailMixin, # {{{
         dynamic.set('sort_history', self.library_view.model().sort_history)
         self.save_layout_state()
 
-    def quit(self, checked=True, restart=False, debug_on_restart=False):
-        if not self.confirm_quit():
+    def quit(self, checked=True, restart=False, debug_on_restart=False,
+            confirm_quit=True):
+        if confirm_quit and not self.confirm_quit():
             return
         try:
             self.shutdown()

src/calibre/gui2/viewer/main.ui

@@ -33,24 +33,21 @@
         <enum>QFrame::Raised</enum>
        </property>
        <layout class="QGridLayout" name="gridLayout">
-        <item row="0" column="0">
-         <widget class="QWebView" name="view"/>
-        </item>
-        <item row="0" column="1">
+        <item row="1" column="1">
          <widget class="QScrollBar" name="vertical_scrollbar">
           <property name="orientation">
            <enum>Qt::Vertical</enum>
           </property>
          </widget>
         </item>
-        <item row="1" column="0">
+        <item row="2" column="0">
          <widget class="QScrollBar" name="horizontal_scrollbar">
           <property name="orientation">
            <enum>Qt::Horizontal</enum>
           </property>
          </widget>
         </item>
-        <item row="2" column="0" colspan="2">
+        <item row="3" column="0" colspan="2">
          <widget class="QFrame" name="dictionary_box">
           <property name="frameShape">
            <enum>QFrame::StyledPanel</enum>
@@ -91,6 +88,9 @@
           </layout>
          </widget>
         </item>
+        <item row="1" column="0">
+         <widget class="DocumentView" name="view" native="true"/>
+        </item>
        </layout>
       </widget>
      </widget>
@@ -108,7 +108,7 @@
     </size>
    </property>
    <attribute name="toolBarArea">
-    <enum>Qt::LeftToolBarArea</enum>
+    <enum>LeftToolBarArea</enum>
    </attribute>
    <attribute name="toolBarBreak">
     <bool>false</bool>
@@ -136,7 +136,7 @@
   </widget>
   <widget class="QToolBar" name="tool_bar2">
    <attribute name="toolBarArea">
-    <enum>Qt::TopToolBarArea</enum>
+    <enum>TopToolBarArea</enum>
    </attribute>
    <attribute name="toolBarBreak">
     <bool>false</bool>
@@ -316,6 +316,12 @@
    <extends>QWidget</extends>
    <header>QtWebKit/QWebView</header>
   </customwidget>
+  <customwidget>
+   <class>DocumentView</class>
+   <extends>QWidget</extends>
+   <header>calibre/gui2/viewer/documentview.h</header>
+   <container>1</container>
+  </customwidget>
  </customwidgets>
  <resources>
   <include location="../../../../resources/images.qrc"/>

src/calibre/library/caches.py

@@ -406,11 +406,9 @@ class ResultCache(SearchQueryParser): # {{{
         if val_func is None:
             loc = self.field_metadata[location]['rec_index']
             val_func = lambda item, loc=loc: item[loc]
-        dt = self.field_metadata[location]['datatype']
-
         q = ''
-        val_func = lambda item, loc=loc: item[loc]
         cast = adjust = lambda x: x
+        dt = self.field_metadata[location]['datatype']
 
         if query == 'false':
             if dt == 'rating' or location == 'cover':
@@ -785,7 +783,7 @@ class ResultCache(SearchQueryParser): # {{{
         else:
             q = query
             if search_restriction:
-                q = u'%s (%s)' % (search_restriction, query)
+                q = u'(%s) and (%s)' % (search_restriction, query)
         if not q:
             if set_restriction_count:
                 self.search_restriction_book_count = len(self._map)

src/calibre/library/database2.py

@@ -464,9 +464,12 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
             self.refresh_ondevice = None
 
     def initialize_database(self):
-        metadata_sqlite = open(P('metadata_sqlite.sql'), 'rb').read()
+        metadata_sqlite = P('metadata_sqlite.sql', data=True,
+                allow_user_override=False).decode('utf-8')
         self.conn.executescript(metadata_sqlite)
-        self.user_version = 1
+        self.conn.commit()
+        if self.user_version == 0:
+            self.user_version = 1
 
     def last_modified(self):
         ''' Return last modified time as a UTC datetime object'''
@@ -3214,7 +3217,6 @@ books_series_link      feeds
             if callable(callback):
                 if callback(''):
                     break
-
         return duplicates
 
     def add_custom_book_data(self, book_id, name, val):
@@ -3223,12 +3225,19 @@ books_series_link      feeds
             raise ValueError('add_custom_book_data: no such book_id %d'%book_id)
         # Do the json encode first, in case it throws an exception
         s = json.dumps(val, default=to_json)
-        self.conn.execute('DELETE FROM books_plugin_data WHERE book=? AND name=?',
-                          (book_id, name))
-        self.conn.execute('''INSERT INTO books_plugin_data(book, name, val)
+        self.conn.execute('''INSERT OR REPLACE INTO books_plugin_data(book, name, val)
                              VALUES(?, ?, ?)''', (book_id, name, s))
         self.commit()
 
+    def add_multiple_custom_book_data(self, name, vals, delete_first=False):
+        if delete_first:
+            self.conn.execute('DELETE FROM books_plugin_data WHERE name=?', (name, ))
+        self.conn.executemany(
+            'INSERT OR REPLACE INTO books_plugin_data (book, name, val) VALUES (?, ?, ?)',
+            [(book_id, name, json.dumps(val, default=to_json))
+                    for book_id, val in vals.iteritems()])
+        self.commit()
+
     def get_custom_book_data(self, book_id, name, default=None):
         try:
             s = self.conn.get('''select val FROM books_plugin_data
@@ -3240,11 +3249,29 @@ books_series_link      feeds
             pass
         return default
 
+    def get_all_custom_book_data(self, name, default=None):
+        try:
+            s = self.conn.get('''select book, val FROM books_plugin_data
+                    WHERE name=?''', (name,))
+            if s is None:
+                return default
+            res = {}
+            for r in s:
+                res[r[0]] = json.loads(r[1], object_hook=from_json)
+            return res
+        except:
+            pass
+        return default
+
     def delete_custom_book_data(self, book_id, name):
         self.conn.execute('DELETE FROM books_plugin_data WHERE book=? AND name=?',
                           (book_id, name))
         self.commit()
 
+    def delete_all_custom_book_data(self, name):
+        self.conn.execute('DELETE FROM books_plugin_data WHERE name=?', (name, ))
+        self.commit()
+
     def get_ids_for_custom_book_data(self, name):
         s = self.conn.get('''SELECT book FROM books_plugin_data WHERE name=?''', (name,))
         return [x[0] for x in s]

src/calibre/library/db/__init__.py

@@ -1,9 +0,0 @@
-#!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
-
-__license__   = 'GPL v3'
-__copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
-__docformat__ = 'restructuredtext en'
-
-
-

src/calibre/library/db/base.py

@@ -1,37 +0,0 @@
-#!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
-
-__license__   = 'GPL v3'
-__copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
-__docformat__ = 'restructuredtext en'
-
-
-''' Design documentation {{{
-
-    Storage paradigm {{{
-        * Agnostic to storage paradigm (i.e. no book per folder assumptions)
-        * Two separate concepts: A store and collection
-          A store is a backend, like a sqlite database associated with a path on
-          the local filesystem, or a cloud based storage solution.
-          A collection is a user defined group of stores. Most of the logic for
-          data manipulation sorting/searching/restrictions should be in the collection
-          class. The collection class should transparently handle the
-          conversion from store name + id to row number in the collection.
-        * Not sure how feasible it is to allow many-many maps between stores
-          and collections.
-    }}}
-
-    Event system {{{
-        * Comprehensive event system that other components can subscribe to
-        * Subscribers should be able to temporarily block receiving events
-        * Should event dispatch be asynchronous?
-        * Track last modified time for metadata and each format
-    }}}
-}}}'''
-
-# Imports {{{
-# }}}
-
-
-
-

src/calibre/library/field_metadata.py

@@ -188,7 +188,7 @@ class FieldMetadata(dict):
                             'datatype':'text',
                            'is_multiple':None,
                            'kind':'field',
-                           'name':None,
+                           'name':_('Author Sort'),
                            'search_terms':['author_sort'],
                            'is_custom':False,
                            'is_category':False,
@@ -238,7 +238,7 @@ class FieldMetadata(dict):
                            'datatype':'datetime',
                            'is_multiple':None,
                            'kind':'field',
-                           'name':_('Date'),
+                           'name':_('Modified'),
                            'search_terms':['last_modified'],
                            'is_custom':False,
                            'is_category':False,
@@ -258,7 +258,7 @@ class FieldMetadata(dict):
                            'datatype':'text',
                            'is_multiple':None,
                            'kind':'field',
-                           'name':None,
+                           'name':_('Path'),
                            'search_terms':[],
                            'is_custom':False,
                            'is_category':False,
@@ -308,7 +308,7 @@ class FieldMetadata(dict):
                            'datatype':'float',
                            'is_multiple':None,
                            'kind':'field',
-                           'name':_('Size (MB)'),
+                           'name':_('Size'),
                            'search_terms':['size'],
                            'is_custom':False,
                            'is_category':False,
@@ -368,7 +368,8 @@ class FieldMetadata(dict):
                         'date_format': tweaks['gui_timestamp_display_format']}
         self._tb_cats['pubdate']['display'] = {
                         'date_format': tweaks['gui_pubdate_display_format']}
-        self._tb_cats['last_modified']['display'] = {'date_format': 'iso'}
+        self._tb_cats['last_modified']['display'] = {
+                        'date_format': tweaks['gui_last_modified_display_format']}
         self.custom_field_prefix = '#'
         self.get = self._tb_cats.get
 
@@ -399,6 +400,13 @@ class FieldMetadata(dict):
                 if self._tb_cats[k]['kind']=='field' and
                    self._tb_cats[k]['datatype'] is not None]
 
+    def displayable_field_keys(self):
+        return [k for k in self._tb_cats.keys()
+                if self._tb_cats[k]['kind']=='field' and
+                   self._tb_cats[k]['datatype'] is not None and
+                   k not in ('au_map', 'marked', 'ondevice', 'cover') and
+                   not self.is_series_index(k)]
+
     def standard_field_keys(self):
         return [k for k in self._tb_cats.keys()
                 if self._tb_cats[k]['kind']=='field' and
@@ -442,6 +450,11 @@ class FieldMetadata(dict):
     def is_custom_field(self, key):
         return key.startswith(self.custom_field_prefix)
 
+    def is_series_index(self, key):
+        m = self[key]
+        return (m['datatype'] == 'float' and key.endswith('_index') and
+                key[:-6] in self)
+
     def key_to_label(self, key):
         if 'label' not in self._tb_cats[key]:
             return key

src/calibre/library/save_to_disk.py

@@ -17,6 +17,7 @@ from calibre.ebooks.metadata.opf2 import metadata_to_opf
 from calibre.constants import preferred_encoding
 from calibre.ebooks.metadata import fmt_sidx
 from calibre.ebooks.metadata import title_sort
+from calibre.utils.date import parse_date
 from calibre import strftime, prints, sanitize_file_name_unicode
 
 plugboard_any_device_value = 'any device'
@@ -42,6 +43,8 @@ FORMAT_ARG_DESCS = dict(
         publisher=_('The publisher'),
         timestamp=_('The date'),
         pubdate=_('The published date'),
+        last_modified=_('The date when the metadata for this book record'
+            ' was last modified'),
         id=_('The calibre internal id')
         )
 
@@ -186,11 +189,14 @@ def get_components(template, mi, id, timefmt='%b %Y', length=250,
     else:
         template = re.sub(r'\{series_index[^}]*?\}', '', template)
     if mi.rating is not None:
-        format_args['rating'] = mi.format_rating()
+        format_args['rating'] = mi.format_rating(divide_by=2.0)
     if hasattr(mi.timestamp, 'timetuple'):
         format_args['timestamp'] = strftime(timefmt, mi.timestamp.timetuple())
     if hasattr(mi.pubdate, 'timetuple'):
         format_args['pubdate'] = strftime(timefmt, mi.pubdate.timetuple())
+    if hasattr(mi, 'last_modified') and hasattr(mi.last_modified, 'timetuple'):
+        format_args['last_modified'] = strftime(timefmt, mi.last_modified.timetuple())
+
     format_args['id'] = str(id)
     # Now format the custom fields
     custom_metadata = mi.get_all_user_metadata(make_copy=False)
@@ -205,6 +211,9 @@ def get_components(template, mi, id, timefmt='%b %Y', length=250,
                 format_args[key] = strftime(timefmt, format_args[key].timetuple())
             elif cm['datatype'] == 'bool':
                 format_args[key] = _('yes') if format_args[key] else _('no')
+            elif cm['datatype'] == 'rating':
+                format_args[key] = mi.format_rating(format_args[key],
+                        divide_by=2.0)
             elif cm['datatype'] in ['int', 'float']:
                 if format_args[key] != 0:
                     format_args[key] = unicode(format_args[key])
@@ -373,10 +382,14 @@ def save_serialized_to_disk(ids, data, plugboards, root, opts, callback):
     root, opts, length = _sanitize_args(root, opts)
     failures = []
     for x in ids:
-        opf, cover, format_map = data[x]
+        opf, cover, format_map, last_modified = data[x]
         if isinstance(opf, unicode):
             opf = opf.encode('utf-8')
         mi = OPF(cStringIO.StringIO(opf)).to_book_metadata()
+        try:
+            mi.last_modified = parse_date(last_modified)
+        except:
+            pass
         tb = ''
         try:
             failed, id, title = do_save_book_to_disk(x, mi, cover, plugboards,

src/calibre/manual/creating_plugins.rst

@@ -195,9 +195,10 @@ It can get tiresome to keep re-adding a plugin to calibre to test small changes.
 
 Once you've located the zip file of your plugin you can then directly update it with your changes instead of re-adding it each time. To do so from the command line, in the directory that contains your plugin source code, use::
 
-    zip -R /path/to/plugin/zip/file.zip *
+    calibre -s; sleep 4s; zip -R /path/to/plugin/zip/file.zip *; calibre
 
-This will update all changed files. It relies on the freely available zip command line tool. Note that you should quit calibre before running this command.
+This will shutdown a running calibre. Wait for the shutdown to complete, then update your plugin files and relaunch calibre.
+It relies on the freely available zip command line tool.
 
 More plugin examples
 ----------------------

src/calibre/manual/faq.rst

@@ -20,13 +20,14 @@ What formats does |app| support conversion to/from?
 |app| supports the conversion of many input formats to many output formats.
 It can convert every input format in the following list, to every output format.
 
-*Input Formats:* CBZ, CBR, CBC, CHM, EPUB, FB2, HTML, HTMLZ, LIT, LRF, MOBI, ODT, PDF, PRC**, PDB***, PML, RB, RTF, SNB, TCR, TXT, TXTZ
+*Input Formats:* CBZ, CBR, CBC, CHM, EPUB, FB2, HTML, HTMLZ, LIT, LRF, MOBI, ODT, PDF, PRC, PDB, PML, RB, RTF, SNB, TCR, TXT, TXTZ
 
 *Output Formats:* EPUB, FB2, OEB, LIT, LRF, MOBI, HTMLZ, PDB, PML, RB, PDF, SNB, TCR, TXT, TXTZ
 
-** PRC is a generic format, |app| supports PRC files with TextRead and MOBIBook headers
+.. note ::
 
-*** PDB is also a generic format. |app| supports eReder, Plucker, PML and zTxt PDB files.
+    PRC is a generic format, |app| supports PRC files with TextRead and MOBIBook headers.
+    PDB is also a generic format. |app| supports eReder, Plucker, PML and zTxt PDB files.
 
 .. _best-source-formats:
 

src/calibre/manual/gui.rst

@@ -365,6 +365,8 @@ Dates and numeric fields support the relational operators ``=`` (equals), ``>``
 Rating fields are considered to be numeric. For example, the search ``rating:>=3`` will find all books rated 3
 or higher.
 
+You can search for the number of items in multiple-valued fields such as tags). These searches begin with the character ``#``, then use the same syntax as numeric fields. For example, to find all books with more than 4 tags, use ``tags:#>4``. To find all books with exactly 10 tags, use ``tags:#=10``.
+
 Series indices are searchable. For the standard series, the search name is 'series_index'. For
 custom series columns, use the column search name followed by _index. For example, to search the indices for a
 custom series column named ``#my_series``, you would use the search name ``#my_series_index``.

src/calibre/manual/plugins.rst

@@ -65,17 +65,14 @@ Catalog plugins
 Metadata download plugins
 --------------------------
 
-.. module:: calibre.ebooks.metadata.fetch
+.. module:: calibre.ebooks.metadata.sources.base
 
-.. autoclass:: MetadataSource
+.. autoclass:: Source
    :show-inheritance:
    :members:
    :member-order: bysource
 
-.. autoclass:: calibre.ebooks.metadata.covers.CoverDownload
-   :show-inheritance:
-   :members:
-   :member-order: bysource
+.. autoclass:: InternalMetadataCompareKeyGen
 
 Conversion plugins
 --------------------

src/calibre/manual/template_lang.rst

@@ -396,3 +396,19 @@ You might find the following tips useful.
     * In a plugboard, you can set a field to empty (or whatever is equivalent to empty) by using the special template ``{null}``. This template will always evaluate to an empty string.
     * The technique described above to show numbers even if they have a zero value works with the standard field series_index.
     
+API of the Metadata objects
+----------------------------
+
+.. module:: calibre.ebooks.metadata.book.base
+
+.. autoclass:: Metadata
+   :members:
+   :member-order: bysource
+
+.. data:: STANDARD_METADATA_FIELDS
+
+    The set of standard metadata fields.
+
+.. literalinclude:: ../ebooks/metadata/book/__init__.py
+   :lines: 7-
+