0.9.17

Changelog.yaml

@@ -19,6 +19,51 @@
 #   new recipes:
 #     - title: 
 
+- version: 0.9.17
+  date: 2013-02-01
+
+  new features:
+    - title: "Allow adding user specified icons to the main book list for books whose metadata matches specific criteria. Go to Preferences->Look & Feel->Column icons to setup these icons. They work in the same way as the column coloring rules."
+      type: major
+
+    - title: "Allow choosing which page of a PDF to use as the cover."
+      description: "To access this functionality add the PDF to calibre then click the edit metadata button. In the top right area of the edit metadata dialog there is a button to get the cover from the ebook file, this will now allow you to choose which page (from the first ten pages) of the pdf to use as the cover."
+      tickets: [1110019]
+
+    - title: "Add option to turn off reflections in the cover browser (Preferences->Look & Feel->Cover Browser)"
+
+    - title: "PDF Output: Add an option to add page numbers to the bottom of every page in the generated PDF file (look in the PDF Output section of the conversion dialog)"
+
+    - title: "Add the full item name to the tool tip of a leaf item displayed in the tag browser."
+      tickets: [1106231]
+ 
+  bug fixes:
+    - title: "Fix out-of-bounds data causing errors in the Tag Browser"
+      tickets: [1108017]
+
+    - title: "Conversion: Handle input documents that use multiple prefixes referring to the XHTML namespace correctly."
+      tickets: [1107220]
+
+    - title: "PDF Output: Fix regression that caused some svg images to be rendered as black rectangles."
+      tickets: [1105294]
+
+    - title: "Metadata download: Only normalize title case if the result has no language set or its language is English"
+
+  improved recipes:
+    - Baltimore Sun
+    - Harvard Business Review
+    - Victoria Times
+    - South China Morning Post
+    - Volksrant
+    - Seattle Times
+
+  new recipes:
+    - title: Dob NeviNosti 
+      author: Darko Miletic
+
+    - title: La Nacion (CR) 
+      author: Douglas Delgado
+
 - version: 0.9.16
   date: 2013-01-25
 

manual/faq.rst

@@ -158,13 +158,23 @@ My device is not being detected by |app|?
 
 Follow these steps to find the problem:
 
-    * Make sure that you are connecting only a single device to your computer at a time. Do not have another |app| supported device like an iPhone/iPad etc. at the same time.
+    * Make sure that you are connecting only a single device to your computer
-    * If you are connecting an Apple iDevice (iPad, iPod Touch, iPhone), use the 'Connect to iTunes' method in the 'Getting started' instructions in `Calibre + Apple iDevices: Start here <http://www.mobileread.com/forums/showthread.php?t=118559>`_.
+      at a time. Do not have another |app| supported device like an iPhone/iPad
-    * Make sure you are running the latest version of |app|. The latest version can always be downloaded from `the calibre website <http://calibre-ebook.com/download>`_.
+      etc. at the same time.
-    * Ensure your operating system is seeing the device. That is, the device should show up in Windows Explorer (in Windows) or Finder (in OS X).
+    * If you are connecting an Apple iDevice (iPad, iPod Touch, iPhone), use
+      the 'Connect to iTunes' method in the 'Getting started' instructions in
+      `Calibre + Apple iDevices: Start here <http://www.mobileread.com/forums/showthread.php?t=118559>`_.
+    * Make sure you are running the latest version of |app|. The latest version
+      can always be downloaded from `the calibre website <http://calibre-ebook.com/download>`_.
+      You can tell what version of |app| you are currently running by looking
+      at the bottom line of the main |app| window.
+    * Ensure your operating system is seeing the device. That is, the device
+      should show up in Windows Explorer (in Windows) or Finder (in OS X).
     * In |app|, go to Preferences->Ignored Devices and check that your device
       is not being ignored
-    * If all the above steps fail, go to Preferences->Miscellaneous and click debug device detection with your device attached and post the output as a ticket on `the calibre bug tracker <http://bugs.calibre-ebook.com>`_.
+    * If all the above steps fail, go to Preferences->Miscellaneous and click
+      debug device detection with your device attached and post the output as a
+      ticket on `the calibre bug tracker <http://bugs.calibre-ebook.com>`_.
 
 My device is non-standard or unusual. What can I do to connect to it?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -540,9 +550,9 @@ Yes, you can. Follow the instructions in the answer above for adding custom colu
 
 How do I move my |app| library from one computer to another?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Simply copy the |app| library folder from the old to the new computer. You can find out what the library folder is by clicking the calibre icon in the toolbar. The very first item is the path to the library folder. Now on the new computer, start |app| for the first time. It will run the Welcome Wizard asking you for the location of the |app| library. Point it to the previously copied folder. If the computer you are transferring to already has a calibre installation, then the Welcome wizard wont run. In that case, click the calibre icon in the tooolbar and point it to the newly copied directory. You will now have two calibre libraries on your computer and you can switch between them by clicking the calibre icon on the toolbar.
+Simply copy the |app| library folder from the old to the new computer. You can find out what the library folder is by clicking the calibre icon in the toolbar. The very first item is the path to the library folder. Now on the new computer, start |app| for the first time. It will run the Welcome Wizard asking you for the location of the |app| library. Point it to the previously copied folder. If the computer you are transferring to already has a calibre installation, then the Welcome wizard wont run. In that case, right-click the |app| icon in the tooolbar and point it to the newly copied directory. You will now have two calibre libraries on your computer and you can switch between them by clicking the |app| icon on the toolbar. Transferring your library in this manner preserver all your metadata, tags, custom columns, etc.
 
-Note that if you are transferring between different types of computers (for example Windows to OS X) then after doing the above you should also right-click the calibre icon on the tool bar, select Library Maintenance and run the Check Library action. It will warn you about any problems in your library, which you should fix by hand.
+Note that if you are transferring between different types of computers (for example Windows to OS X) then after doing the above you should also right-click the |app| icon on the tool bar, select Library Maintenance and run the Check Library action. It will warn you about any problems in your library, which you should fix by hand.
 
 .. note:: A |app| library is just a folder which contains all the book files and their metadata. All the metadata is stored in a single file called metadata.db, in the top level folder. If this file gets corrupted, you may see an empty list of books in |app|. In this case you can ask |app| to restore your books by doing a right-click on the |app| icon in the toolbar and selecting Library Maintenance->Restore Library.
 
@@ -672,8 +682,11 @@ There are three possible things I know of, that can cause this:
     * The Logitech SetPoint Settings application causes random crashes in
       |app| when it is open. Close it before starting |app|.
 
+    * Constant Guard Protection by Xfinity causes crashes in |app|. You have to
+      manually allow |app| in it or uninstall Constant Guard Protection.
+
 If none of the above apply to you, then there is some other program on your
-computer that is interfering with |app|. First reboot your computer is safe
+computer that is interfering with |app|. First reboot your computer in safe
 mode, to have as few running programs as possible, and see if the crashes still
 happen. If they do not, then you know it is some program causing the problem.
 The most likely such culprit is a program that modifies other programs'
@@ -784,7 +797,7 @@ Why doesn't |app| have an automatic update?
 For many reasons:
 
   * *There is no need to update every week*. If you are happy with how |app| works turn off the update notification and be on your merry way. Check back to see if you want to update once a year or so.
-  * Pre downloading the updates for all users in the background would mean require about 80TB of bandwidth *every week*. That costs thousands of dollars a month. And |app| is currently growing at 300,000 new users every month.
+  * Pre downloading the updates for all users in the background would require about 80TB of bandwidth *every week*. That costs thousands of dollars a month. And |app| is currently growing at 300,000 new users every month.
   * If I implement a dialog that downloads the update and launches it, instead of going to the website as it does now, that would save the most ardent |app| updater, *at most five clicks a week*. There are far higher priority things to do in |app| development.
   * If you really, really hate downloading |app| every week but still want to be up to the latest, I encourage you to run from source, which makes updating trivial. Instructions are :ref:`available here <develop>`.
 

recipes/baltimore_sun.recipe

@@ -19,6 +19,7 @@ class BaltimoreSun(BasicNewsRecipe):
     use_embedded_content = False
     no_stylesheets       = True
     remove_javascript    = True
+    #auto_cleanup = True
     recursions           = 1
 
     ignore_duplicate_articles = {'title'}
@@ -78,6 +79,7 @@ class BaltimoreSun(BasicNewsRecipe):
          #(u'High School', u'http://www.baltimoresun.com/sports/high-school/rss2.0.xml'),
          #(u'Outdoors', u'http://www.baltimoresun.com/sports/outdoors/rss2.0.xml'),
 
+
 ## Entertainment ##
          (u'Celebrity News', u'http://www.baltimoresun.com/entertainment/celebrities/rss2.0.xml'),
          (u'Arts & Theater', u'http://www.baltimoresun.com/entertainment/arts/rss2.0.xml'),
@@ -142,12 +144,12 @@ class BaltimoreSun(BasicNewsRecipe):
          (u'Read Street', u'http://www.baltimoresun.com/features/books/read-street/rss2.0.xml'),
          (u'Z on TV', u'http://www.baltimoresun.com/entertainment/tv/z-on-tv-blog/rss2.0.xml'),
 
-## Life Blogs ##
+### Life Blogs ##
-         (u'BMore Green', u'http://weblogs.baltimoresun.com/features/green/index.xml'),
+         #(u'BMore Green', u'http://weblogs.baltimoresun.com/features/green/index.xml'),
-         (u'Baltimore Insider',u'http://www.baltimoresun.com/features/baltimore-insider-blog/rss2.0.xml'),
+         #(u'Baltimore Insider',u'http://www.baltimoresun.com/features/baltimore-insider-blog/rss2.0.xml'),
-         (u'Homefront', u'http://www.baltimoresun.com/features/parenting/homefront/rss2.0.xml'),
+         #(u'Homefront', u'http://www.baltimoresun.com/features/parenting/homefront/rss2.0.xml'),
-         (u'Picture of Health', u'http://www.baltimoresun.com/health/blog/rss2.0.xml'),
+         #(u'Picture of Health', u'http://www.baltimoresun.com/health/blog/rss2.0.xml'),
-         (u'Unleashed', u'http://weblogs.baltimoresun.com/features/mutts/blog/index.xml'),
+         #(u'Unleashed', u'http://weblogs.baltimoresun.com/features/mutts/blog/index.xml'),
 
 ## b the site blogs ##
          (u'Game Cache', u'http://www.baltimoresun.com/entertainment/bthesite/game-cache/rss2.0.xml'),
@@ -167,6 +169,7 @@ class BaltimoreSun(BasicNewsRecipe):
              ]
 
 
+
     def get_article_url(self, article):
         ans = None
         try:

recipes/dobanevinosti.recipe

@@ -0,0 +1,46 @@
+
+__license__   = 'GPL v3'
+__copyright__ = '2013, Darko Miletic <darko.miletic at gmail.com>'
+'''
+dobanevinosti.blogspot.com
+'''
+import re
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class DobaNevinosti(BasicNewsRecipe):
+    title                 = 'Doba Nevinosti'
+    __author__            = 'Darko Miletic'
+    description           = 'Filmski blog'
+    oldest_article        = 15
+    max_articles_per_feed = 100
+    language              = 'sr'
+    encoding              = 'utf-8'
+    no_stylesheets        = True
+    use_embedded_content  = True
+    publication_type      = 'blog'
+    auto_cleanup          = True
+    extra_css             = """
+                               @font-face {font-family: "sans1";src:url(res:///opt/sony/ebook/FONT/tt0003m_.ttf)}
+                               body{font-family: "Trebuchet MS",Trebuchet,Verdana,sans1,sans-serif}
+                               img{margin-bottom: 0.8em; display:block;}
+                            """
+
+    conversion_options = {
+                          'comment'  : description
+                        , 'tags'     : 'film, blog, srbija, tv'
+                        , 'publisher': 'Dimitrije Vojinov'
+                        , 'language' : language
+                        }
+    remove_attributes = ['lang', 'border']
+    preprocess_regexps = [(re.compile(u'\u0110'), lambda match: u'\u00D0')]
+
+    feeds = [(u'Tekstovi', u'http://dobanevinosti.blogspot.com/feeds/posts/default')]
+
+    def preprocess_html(self, soup):
+        for item in soup.findAll(style=True):
+            del item['style']
+        for item in soup.findAll('img'):
+            if not item.has_key('alt'):
+               item['alt'] = 'image'
+        return soup
+

recipes/globe_and_mail.recipe

@@ -21,6 +21,10 @@ class AdvancedUserRecipe1287083651(BasicNewsRecipe):
     encoding               = 'utf8'
     publisher              = 'Globe & Mail'
     language               = 'en_CA'
+    use_embedded_content = False
+
+    no_stylesheets = True
+    auto_cleanup = True
     extra_css = 'p.meta {font-size:75%}\n .redtext {color: red;}\n .byline {font-size: 70%}'
 
     feeds          = [
@@ -44,12 +48,12 @@ class AdvancedUserRecipe1287083651(BasicNewsRecipe):
         (re.compile(r'<script.*?</script>', re.DOTALL), lambda m: ''),
         ]
 
-    remove_tags_before = dict(name='h1')
+    #remove_tags_before = dict(name='h1')
-    remove_tags = [
+    #remove_tags = [
-            dict(name='div', attrs={'id':['ShareArticles', 'topStories']}),
+            #dict(name='div', attrs={'id':['ShareArticles', 'topStories']}),
-            dict(href=lambda x: x and 'tracking=' in x),
+            #dict(href=lambda x: x and 'tracking=' in x),
-            {'class':['articleTools', 'pagination', 'Ads', 'topad',
+            #{'class':['articleTools', 'pagination', 'Ads', 'topad',
-                'breadcrumbs', 'footerNav', 'footerUtil', 'downloadlinks']}]
+                #'breadcrumbs', 'footerNav', 'footerUtil', 'downloadlinks']}]
 
     def populate_article_metadata(self, article, soup, first):
         if first and hasattr(self, 'add_toc_thumbnail'):

recipes/hbr.recipe

@@ -11,11 +11,11 @@ class HBR(BasicNewsRecipe):
     timefmt                = ' [%B %Y]'
     language = 'en'
     no_stylesheets = True
-    recipe_disabled = ('hbr.org has started requiring the use of javascript'
+    # recipe_disabled = ('hbr.org has started requiring the use of javascript'
-            ' to log into their website. This is unsupported in calibre, so'
+    #         ' to log into their website. This is unsupported in calibre, so'
-            ' this recipe has been disabled. If you would like to see '
+    #         ' this recipe has been disabled. If you would like to see '
-            ' HBR supported in calibre, contact hbr.org and ask them'
+    #         ' HBR supported in calibre, contact hbr.org and ask them'
-            ' to provide a javascript free login method.')
+    #         ' to provide a javascript free login method.')
 
     LOGIN_URL = 'https://hbr.org/login?request_url=/'
     LOGOUT_URL = 'https://hbr.org/logout?request_url=/'
@@ -38,46 +38,37 @@ class HBR(BasicNewsRecipe):
                 #articleAuthors{font-family:Georgia,"Times New Roman",Times,serif; font-style:italic; color:#000000;font-size:x-small;}
                 #summaryText{font-family:Georgia,"Times New Roman",Times,serif; font-weight:bold; font-size:x-small;}
                 '''
+    use_javascript_to_login = True
 
-    def get_browser(self):
+    def javascript_login(self, br, username, password):
-        br = BasicNewsRecipe.get_browser(self)
+        from calibre.web.jsbrowser.browser import Timeout
-        self.logout_url = None
-
-        #'''
-        br.open(self.LOGIN_URL)
-        br.select_form(name='signin-form')
-        br['signin-form:username'] = self.username
-        br['signin-form:password'] = self.password
-        raw = br.submit().read()
-        if '>Sign out<' not in raw:
-            raise Exception('Failed to login, are you sure your username and password are correct?')
         try:
-            link = br.find_link(text='Sign out')
+            br.visit('https://hbr.org/login?request_url=/', timeout=20)
-            if link:
+        except Timeout:
-                self.logout_url = link.absolute_url
+            pass
-        except:
+        br.click('#accordion div[tabindex="0"]', wait_for_load=False)
-            self.logout_url = self.LOGOUT_URL
+        f = br.select_form('#signin-form')
-        #'''
+        f['signin-form:username'] = username
-        return br
+        f['signin-form:password'] = password
-
+        br.submit(wait_for_load=False)
-    def cleanup(self):
+        br.run_for_a_time(30)
-        if self.logout_url is not None:
-            self.browser.open(self.logout_url)
 
     def map_url(self, url):
         if url.endswith('/ar/1'):
             return url[:-1]+'pr'
 
-
     def hbr_get_toc(self):
-        #return self.index_to_soup(open('/t/hbr.html').read())
+        # return self.index_to_soup(open('/t/toc.html').read())
 
         today = date.today()
         future = today + timedelta(days=30)
         for x in [x.strftime('%y%m') for x in (future, today)]:
             url = self.INDEX + x
             soup = self.index_to_soup(url)
-            if not soup.find(text='Issue Not Found'):
+            if (not soup.find(text='Issue Not Found') and not soup.find(
+                text="We're Sorry.  There was an error processing your request")
+                and 'Exception: java.io.FileNotFoundException' not in
+                unicode(soup)):
                 return soup
         raise Exception('Could not find current issue')
 
@@ -85,8 +76,9 @@ class HBR(BasicNewsRecipe):
         feeds = []
         current_section = None
         articles = []
-        for x in soup.find(id='archiveToc').findAll(['h3', 'h4']):
+        for x in soup.find(id='issueFeaturesContent').findAll(['li', 'h4']):
-            if x.name == 'h3':
+            if x.name == 'h4':
+                if x.get('class', None) == 'basic':continue
                 if current_section is not None and articles:
                     feeds.append((current_section, articles))
                 current_section = self.tag_to_string(x).capitalize()
@@ -102,7 +94,7 @@ class HBR(BasicNewsRecipe):
                 if url.startswith('/'):
                     url = 'http://hbr.org' + url
                 url = self.map_url(url)
-                p = x.parent.find('p')
+                p = x.find('p', attrs={'class':'author'})
                 desc = ''
                 if p is not None:
                     desc = self.tag_to_string(p)
@@ -114,10 +106,9 @@ class HBR(BasicNewsRecipe):
                     'date':''})
         return feeds
 
-
     def parse_index(self):
         soup = self.hbr_get_toc()
-        #open('/t/hbr.html', 'wb').write(unicode(soup).encode('utf-8'))
+        # open('/t/hbr.html', 'wb').write(unicode(soup).encode('utf-8'))
         feeds = self.hbr_parse_toc(soup)
         return feeds
 

recipes/la_nacion_cr.recipe

@@ -0,0 +1,40 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class crnews(BasicNewsRecipe):
+    __author__     = 'Douglas Delgado'
+    title          = u'La Nacion'
+    publisher      = 'GRUPO NACION GN, S. A.'
+    description    = 'Diario de circulacion nacional de Costa Rica. Recipe creado por Douglas Delgado (doudelgado@gmail.com) para su uso con Calibre por Kovid Goyal'
+    category       = 'Spanish, Entertainment'
+    masthead_url   = 'http://www.nacion.com/App_Themes/nacioncom/Images/logo_nacioncom.png'
+
+    oldest_article = 7
+    delay          = 1
+    max_articles_per_feed = 100
+    auto_cleanup = True
+    encoding       = 'utf-8'
+    language       = 'es_CR'
+    use_embedded_content  = False
+    remove_empty_feeds    = True
+    remove_javascript = True
+    no_stylesheets = True
+
+
+    feeds          = [(u'Portada', u'http://www.nacion.com/Generales/RSS/EdicionRss.aspx?section=portada'), (u'Ultima Hora', u'http://www.nacion.com/Generales/RSS/UltimaHoraRss.aspx'), (u'Nacionales', u'http://www.nacion.com/Generales/RSS/EdicionRss.aspx?section=elpais'), (u'Entretenimiento', u'http://www.nacion.com/Generales/RSS/EdicionRss.aspx?section=entretenimiento'), (u'Sucesos', u'http://www.nacion.com/Generales/RSS/EdicionRss.aspx?section=sucesos'), (u'Deportes', u'http://www.nacion.com/Generales/RSS/EdicionRss.aspx?section=deportes'), (u'Internacionales', u'http://www.nacion.com/Generales/RSS/EdicionRss.aspx?section=mundo'), (u'Economia', u'http://www.nacion.com/Generales/RSS/EdicionRss.aspx?section=economia'), (u'Aldea Global', u'http://www.nacion.com/Generales/RSS/EdicionRss.aspx?section=aldeaglobal'), (u'Tecnologia', u'http://www.nacion.com/Generales/RSS/EdicionRss.aspx?section=tecnologia'), (u'Opinion', u'http://www.nacion.com/Generales/RSS/EdicionRss.aspx?section=opinion')]
+
+    def get_cover_url(self):
+        index = 'http://kiosko.net/cr/np/cr_nacion.html'
+        soup = self.index_to_soup(index)
+        for image in soup.findAll('img',src=True):
+           if image['src'].endswith('cr_nacion.750.jpg'):
+              return image['src']
+        return None
+
+    def get_article_url(self, article):
+        url = article.get('guid', None)
+        return url
+
+    extra_css = '''
+                    h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:30px;}
+                    h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal; font-style:italic; font-size:18px;}
+                '''

recipes/libertad_digital.recipe

@@ -0,0 +1,65 @@
+__license__   = 'GPL v3'
+__copyright__ = '2013, Darko Miletic <darko.miletic at gmail.com>'
+'''
+www.libertaddigital.com
+'''
+
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class LibertadDigital(BasicNewsRecipe):
+    title                 = 'Libertad Digital'
+    __author__            = 'Darko Miletic'
+    description           = 'En Libertad Digital encontraras noticias y opinion sobre: España, el Mundo, Internet, sociedad, economia y deportes'
+    publisher             = 'Libertad Digital S.A.'
+    category              = 'noticias, ultima hora, españa, internet, mundo, economia, sociedad, Libertad Digital'
+    oldest_article        = 2
+    max_articles_per_feed = 200
+    no_stylesheets        = True
+    encoding              = 'cp1252'
+    use_embedded_content  = False
+    language              = 'es'
+    remove_empty_feeds    = True
+    publication_type      = 'website'
+    masthead_url          = 'http://s.libertaddigital.com/images/logo.gif'
+    extra_css             = """
+                               body{font-family: Verdana,sans-serif }
+                               img{margin-bottom: 0.4em; display:block}
+                            """
+
+    conversion_options = {
+                          'comment'   : description
+                        , 'tags'      : category
+                        , 'publisher' : publisher
+                        , 'language'  : language
+                        }
+
+    remove_tags = [
+                     dict(name=['meta','link','iframe','embed','object'])
+                    ,dict(name='p', attrs={'class':'copyright'})
+                  ]
+    remove_attributes=['lang']
+
+
+    feeds = [
+              (u'Portada'      , u'http://feeds2.feedburner.com/libertaddigital/deportes'     )
+             ,(u'Opinion'      , u'http://feeds2.feedburner.com/libertaddigital/opinion'      )
+             ,(u'España'       , u'http://feeds2.feedburner.com/libertaddigital/nacional'     )
+             ,(u'Internacional', u'http://feeds2.feedburner.com/libertaddigital/internacional')
+             ,(u'Libre Mercado', u'http://feeds2.feedburner.com/libertaddigital/economia'     )
+             ,(u'Chic'         , u'http://feeds2.feedburner.com/libertaddigital/el-candelabro')
+             ,(u'Internet'     , u'http://feeds2.feedburner.com/libertaddigital/internet'     )
+             ,(u'Deportes'     , u'http://feeds2.feedburner.com/libertaddigital/deportes'     )
+            ]
+
+    def get_article_url(self, article):
+        return article.get('guid',  None)
+            
+    def print_version(self, url):
+        art, sep, rest = url.rpartition('/')
+        aart, asep, artid = art.rpartition('-')
+        return 'http://www.libertaddigital.com/c.php?op=imprimir&id=' + artid
+
+    def preprocess_html(self, soup):
+        for item in soup.findAll(style=True):
+            del item['style']
+        return soup

recipes/scmp.recipe

@@ -4,7 +4,6 @@ __copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
 scmp.com
 '''
 
-import re
 from calibre.web.feeds.news import BasicNewsRecipe
 
 class SCMP(BasicNewsRecipe):
@@ -18,10 +17,11 @@ class SCMP(BasicNewsRecipe):
     max_articles_per_feed = 200
     no_stylesheets        = True
     encoding              = 'utf-8'
+    auto_cleanup = True
     use_embedded_content  = False
     language              = 'en_CN'
     remove_empty_feeds    = True
-    needs_subscription    = True
+    needs_subscription    = 'optional'
     publication_type      = 'newspaper'
     masthead_url          = 'http://www.scmp.com/images/logo_scmp_home.gif'
     extra_css             = ' body{font-family: Arial,Helvetica,sans-serif } '
@@ -46,17 +46,17 @@ class SCMP(BasicNewsRecipe):
            br.submit()
         return br
 
-    remove_attributes=['width','height','border']
+    #remove_attributes=['width','height','border']
 
-    keep_only_tags = [
+    #keep_only_tags = [
-                        dict(attrs={'id':['ART','photoBox']})
+                        #dict(attrs={'id':['ART','photoBox']})
-                       ,dict(attrs={'class':['article_label','article_byline','article_body']})
+                       #,dict(attrs={'class':['article_label','article_byline','article_body']})
-                     ]
+                     #]
 
-    preprocess_regexps = [
+    #preprocess_regexps = [
-       (re.compile(r'<P><table((?!<table).)*class="embscreen"((?!</table>).)*</table>', re.DOTALL|re.IGNORECASE),
+       #(re.compile(r'<P><table((?!<table).)*class="embscreen"((?!</table>).)*</table>', re.DOTALL|re.IGNORECASE),
-        lambda match: ''),
+        #lambda match: ''),
-    ]
+    #]
 
     feeds = [
               (u'Business'     , u'http://www.scmp.com/rss/business.xml'       )
@@ -68,13 +68,13 @@ class SCMP(BasicNewsRecipe):
              ,(u'Sport'        , u'http://www.scmp.com/rss/sport.xml'          )
             ]
 
-    def print_version(self, url):
+    #def print_version(self, url):
-        rpart, sep, rest = url.rpartition('&')
+        #rpart, sep, rest = url.rpartition('&')
-        return rpart #+ sep + urllib.quote_plus(rest)
+        #return rpart #+ sep + urllib.quote_plus(rest)
 
-    def preprocess_html(self, soup):
+    #def preprocess_html(self, soup):
-        for item in soup.findAll(style=True):
+        #for item in soup.findAll(style=True):
-            del item['style']
+            #del item['style']
-        items = soup.findAll(src="/images/label_icon.gif")
+        #items = soup.findAll(src="/images/label_icon.gif")
-        [item.extract() for item in items]
+        #[item.extract() for item in items]
-        return self.adeify_images(soup)
+        #return self.adeify_images(soup)

recipes/seattle_times.recipe

@@ -23,6 +23,7 @@ class SeattleTimes(BasicNewsRecipe):
     language = 'en'
     auto_cleanup          = True
     auto_cleanup_keep     = '//div[@id="PhotoContainer"]'
+    cover_url = 'http://seattletimes.com/PDF/frontpage.pdf'
 
     feeds              = [
                           (u'Top Stories',

recipes/vic_times.recipe

@@ -1,105 +1,46 @@
 #!/usr/bin/env  python
 # -*- coding: utf-8 -*-
-
 __license__   = 'GPL v3'
 
 '''
 www.canada.com
 '''
-
 import re
-from calibre.web.feeds.news import BasicNewsRecipe
+from calibre.web.feeds.recipes import BasicNewsRecipe
-from calibre.ebooks.BeautifulSoup import BeautifulStoneSoup
+from calibre.ebooks.BeautifulSoup import Tag, BeautifulStoneSoup
 
 
-class CanWestPaper(BasicNewsRecipe):
+class TimesColonist(BasicNewsRecipe):
 
-    # un-comment the following four lines for the Victoria Times Colonist
     title = u'Victoria Times Colonist'
     url_prefix = 'http://www.timescolonist.com'
     description = u'News from Victoria, BC'
     fp_tag = 'CAN_TC'
 
-    # un-comment the following four lines for the Vancouver Province
+    url_list = []
-##    title = u'Vancouver Province'
-##    url_prefix = 'http://www.theprovince.com'
-##    description = u'News from Vancouver, BC'
-##    fp_tag = 'CAN_VP'
-
-    # un-comment the following four lines for the Vancouver Sun
-##    title = u'Vancouver Sun'
-##    url_prefix = 'http://www.vancouversun.com'
-##    description = u'News from Vancouver, BC'
-##    fp_tag = 'CAN_VS'
-
-    # un-comment the following four lines for the Edmonton Journal
-##    title = u'Edmonton Journal'
-##    url_prefix = 'http://www.edmontonjournal.com'
-##    description = u'News from Edmonton, AB'
-##    fp_tag = 'CAN_EJ'
-
-    # un-comment the following four lines for the Calgary Herald
-##    title = u'Calgary Herald'
-##    url_prefix = 'http://www.calgaryherald.com'
-##    description = u'News from Calgary, AB'
-##    fp_tag = 'CAN_CH'
-
-    # un-comment the following four lines for the Regina Leader-Post
-##    title = u'Regina Leader-Post'
-##    url_prefix = 'http://www.leaderpost.com'
-##    description = u'News from Regina, SK'
-##    fp_tag = ''
-
-    # un-comment the following four lines for the Saskatoon Star-Phoenix
-##    title = u'Saskatoon Star-Phoenix'
-##    url_prefix = 'http://www.thestarphoenix.com'
-##    description = u'News from Saskatoon, SK'
-##    fp_tag = ''
-
-    # un-comment the following four lines for the Windsor Star
-##    title = u'Windsor Star'
-##    url_prefix = 'http://www.windsorstar.com'
-##    description = u'News from Windsor, ON'
-##    fp_tag = 'CAN_'
-
-    # un-comment the following four lines for the Ottawa Citizen
-##    title = u'Ottawa Citizen'
-##    url_prefix = 'http://www.ottawacitizen.com'
-##    description = u'News from Ottawa, ON'
-##    fp_tag = 'CAN_OC'
-
-    # un-comment the following four lines for the Montreal Gazette
-##    title = u'Montreal Gazette'
-##    url_prefix = 'http://www.montrealgazette.com'
-##    description = u'News from Montreal, QC'
-##    fp_tag = 'CAN_MG'
-
-
     language = 'en_CA'
     __author__ = 'Nick Redding'
     no_stylesheets = True
-    timefmt = ' [%b %d]'
+    timefmt =  ' [%b %d]'
+    encoding = 'utf-8'
     extra_css = '''
-                .timestamp {  font-size:xx-small; display: block; }
+                .byline { font-size:xx-small; font-weight: bold;}
-                #storyheader { font-size: medium; }
+                h3 { margin-bottom: 6px; }
-                #storyheader h1 { font-size: x-large; }
+                .caption { font-size: xx-small; font-style: italic; font-weight: normal; }
-                #storyheader h2 { font-size: large;  font-style: italic; }
+                '''
-                .byline { font-size:xx-small; }
+    keep_only_tags = [dict(name='div', attrs={'class':re.compile('main.content')})]
-                #photocaption { font-size: small; font-style: italic }
-                #photocredit { font-size: xx-small; }'''
-    keep_only_tags = [dict(name='div', attrs={'id':'storyheader'}),dict(name='div', attrs={'id':'storycontent'})]
     remove_tags = [{'class':'comments'},
-                   dict(name='div', attrs={'class':'navbar'}),dict(name='div', attrs={'class':'morelinks'}),
+                   {'id':'photocredit'},
-                   dict(name='div', attrs={'class':'viewmore'}),dict(name='li', attrs={'class':'email'}),
+                   dict(name='div', attrs={'class':re.compile('top.controls')}),
-                   dict(name='div', attrs={'class':'story_tool_hr'}),dict(name='div', attrs={'class':'clear'}),
+                   dict(name='div', attrs={'class':re.compile('social')}),
-                   dict(name='div', attrs={'class':'story_tool'}),dict(name='div', attrs={'class':'copyright'}),
+                   dict(name='div', attrs={'class':re.compile('tools')}),
-                   dict(name='div', attrs={'class':'rule_grey_solid'}),
+                   dict(name='div', attrs={'class':re.compile('bottom.tools')}),
-                   dict(name='li', attrs={'class':'print'}),dict(name='li', attrs={'class':'share'}),dict(name='ul', attrs={'class':'bullet'})]
+                   dict(name='div', attrs={'class':re.compile('window')}),
+                   dict(name='div', attrs={'class':re.compile('related.news.element')})]
+
 
     def get_cover_url(self):
         from datetime import timedelta, date
-        if self.fp_tag=='':
-            return None
         cover = 'http://webmedia.newseum.org/newseum-multimedia/dfp/jpg'+str(date.today().day)+'/lg/'+self.fp_tag+'.jpg'
         br = BasicNewsRecipe.get_browser(self)
         daysback=1
@@ -120,6 +61,18 @@ class CanWestPaper(BasicNewsRecipe):
             cover = None
         return cover
 
+    def prepare_masthead_image(self, path_to_image, out_path):
+        if self.Kindle_Fire:
+            from calibre.utils.magick import Image, create_canvas
+            img = Image()
+            img.open(path_to_image)
+            width, height = img.size
+            img2 = create_canvas(width, height)
+            img2.compose(img)
+            img2.save(out_path)
+        else:
+            BasicNewsRecipe.prepare_masthead_image(path_to_image, out_path)
+
     def fixChars(self,string):
         # Replace lsquo (\x91)
         fixed = re.sub("\x91","‘",string)
@@ -166,55 +119,107 @@ class CanWestPaper(BasicNewsRecipe):
                     a.replaceWith(a.renderContents().decode('cp1252','replace'))
         return soup
 
-    def preprocess_html(self, soup):
+    def preprocess_html(self,soup):
+        byline = soup.find('p',attrs={'class':re.compile('ancillary')})
+        if byline is not None:
+            byline.find('a')
+            authstr = self.tag_to_string(byline,False)
+            authstr = re.sub('/ *Times Colonist','/',authstr, flags=re.IGNORECASE)
+            authstr = re.sub('BY */','',authstr, flags=re.IGNORECASE)
+            newdiv = Tag(soup,'div')
+            newdiv.insert(0,authstr)
+            newdiv['class']='byline'
+            byline.replaceWith(newdiv)
+        for caption in soup.findAll('p',attrs={'class':re.compile('caption')}):
+            capstr = self.tag_to_string(caption,False)
+            capstr = re.sub('Photograph by.*$','',capstr, flags=re.IGNORECASE)
+            newdiv = Tag(soup,'div')
+            newdiv.insert(0,capstr)
+            newdiv['class']='caption'
+            caption.replaceWith(newdiv)
+        for ptag in soup.findAll('p'):
+            ptext = self.tag_to_string(ptag,use_alt=False, normalize_whitespace=True)
+            ptext = re.sub(r'\s+','', ptext)
+            if (ptext=='') or (ptext==' '):
+                ptag.extract()
         return self.strip_anchors(soup)
 
+    raeside = False
+    def handle_articles(self,htag,article_list,sectitle):
+        atag = htag.a
+        if atag is not None:
+            url = atag['href']
+            #print("Checking "+url)
+            if atag['href'].startswith('/'):
+                url = self.url_prefix+atag['href']
+            if url in self.url_list:
+                return
+            self.url_list.append(url)
+            title = self.tag_to_string(atag,False)
+            if 'VIDEO' in title.upper():
+                return
+            if 'GALLERY' in title.upper():
+                return
+            if 'PHOTOS' in title.upper():
+                return
+            if 'RAESIDE' in title.upper():
+                if self.raeside:
+                    return
+                self.raeside = True
+            dtag = htag.findNext('p')
+            description=''
+            if dtag is not None:
+                description = self.tag_to_string(dtag,False)
+            article_list.append(dict(title=title,url=url,date='',description=description,author='',content=''))
+            #print(sectitle+title+": description = "+description+" URL="+url)
 
+    def add_section_index(self,ans,securl,sectitle):
+        print("Add section url="+self.url_prefix+'/'+securl)
+        try:
+            soup = self.index_to_soup(self.url_prefix+'/'+securl)
+        except:
+            return ans
+        mainsoup = soup.find('div',attrs={'class':re.compile('main.content')})
+        article_list = []
+        for wdiv in mainsoup.findAll('div',attrs={'id':re.compile('featured.story')}):
+            for htag in wdiv.findAll('h3'):
+                self.handle_articles(htag,article_list,sectitle)
+        for ladiv in mainsoup.findAll(attrs={'class':re.compile('leading.articles')}):
+            for wdiv in mainsoup.findAll('div',attrs={'class':re.compile('article.row')}):
+                for htag in wdiv.findAll('h2'):
+                    self.handle_articles(htag,article_list,sectitle)
+        ans.append((sectitle,article_list))
+        return ans
 
     def parse_index(self):
-        soup = self.index_to_soup(self.url_prefix+'/news/todays-paper/index.html')
+        ans = []
-
+        ans = self.add_section_index(ans,'','Web Front Page')
-        articles = {}
+        ans = self.add_section_index(ans,'news/','News Headlines')
-        key = 'News'
+        ans = self.add_section_index(ans,'news/b-c/','BC News')
-        ans = ['News']
+        ans = self.add_section_index(ans,'news/national/','Natioanl News')
-
+        ans = self.add_section_index(ans,'news/world/','World News')
-        # Find each instance of class="sectiontitle", class="featurecontent"
+        ans = self.add_section_index(ans,'opinion/','Opinion')
-        for divtag in soup.findAll('div',attrs={'class' : ["section_title02","featurecontent"]}):
+        ans = self.add_section_index(ans,'opinion/letters/','Letters')
-                #self.log(" div class = %s" % divtag['class'])
+        ans = self.add_section_index(ans,'business/','Business')
-                if divtag['class'].startswith('section_title'):
+        ans = self.add_section_index(ans,'business/money/','Money')
-                    # div contains section title
+        ans = self.add_section_index(ans,'business/technology/','Technology')
-                    if not divtag.h3:
+        ans = self.add_section_index(ans,'business/working/','Working')
-                        continue
+        ans = self.add_section_index(ans,'sports/','Sports')
-                    key = self.tag_to_string(divtag.h3,False)
+        ans = self.add_section_index(ans,'sports/hockey/','Hockey')
-                    ans.append(key)
+        ans = self.add_section_index(ans,'sports/football/','Football')
-                    self.log("Section name %s" % key)
+        ans = self.add_section_index(ans,'sports/basketball/','Basketball')
-                    continue
+        ans = self.add_section_index(ans,'sports/golf/','Golf')
-                # div contains article data
+        ans = self.add_section_index(ans,'entertainment/','entertainment')
-                h1tag = divtag.find('h1')
+        ans = self.add_section_index(ans,'entertainment/go/','Go!')
-                if not h1tag:
+        ans = self.add_section_index(ans,'entertainment/music/','Music')
-                    continue
+        ans = self.add_section_index(ans,'entertainment/books/','Books')
-                atag = h1tag.find('a',href=True)
+        ans = self.add_section_index(ans,'entertainment/Movies/','movies')
-                if not atag:
+        ans = self.add_section_index(ans,'entertainment/television/','Television')
-                    continue
+        ans = self.add_section_index(ans,'life/','Life')
-                url = self.url_prefix+'/news/todays-paper/'+atag['href']
+        ans = self.add_section_index(ans,'life/health/','Health')
-                #self.log("Section %s" % key)
+        ans = self.add_section_index(ans,'life/travel/','Travel')
-                #self.log("url %s" % url)
+        ans = self.add_section_index(ans,'life/driving/','Driving')
-                title = self.tag_to_string(atag,False)
+        ans = self.add_section_index(ans,'life/homes/','Homes')
-                #self.log("title %s" % title)
+        ans = self.add_section_index(ans,'life/food-drink/','Food & Drink')
-                pubdate = ''
-                description = ''
-                ptag = divtag.find('p');
-                if ptag:
-                    description = self.tag_to_string(ptag,False)
-                    #self.log("description %s" % description)
-                author = ''
-                autag = divtag.find('h4')
-                if autag:
-                    author = self.tag_to_string(autag,False)
-                    #self.log("author %s" % author)
-                if not articles.has_key(key):
-                    articles[key] = []
-                articles[key].append(dict(title=title,url=url,date=pubdate,description=description,author=author,content=''))
-
-        ans = [(key, articles[key]) for key in ans if articles.has_key(key)]
         return ans
+

recipes/volksrant.recipe

@@ -9,7 +9,6 @@ __docformat__ = 'restructuredtext en'
  Modified by Tony Stegall
  on 10/10/10 to include function to grab print version of articles
 '''
-
 from datetime import date
 from calibre.web.feeds.news import BasicNewsRecipe
 '''
@@ -42,9 +41,16 @@ class AdvancedUserRecipe1249039563(BasicNewsRecipe):
    #######################################################################################################
     temp_files = []
     articles_are_obfuscated = True
+    use_javascript_to_login = True
+
+    def javascript_login(self, br, username, password):
+        'Volksrant wants the user to explicitly allow cookies'
+        if not br.visit('http://www.volkskrant.nl'):
+            raise Exception('Failed to connect to volksrant website')
+        br.click('#pop_cookie_text a[onclick]', wait_for_load=True, timeout=120)
 
     def get_obfuscated_article(self, url):
-        br = self.get_browser()
+        br = self.browser.clone_browser()
         print 'THE CURRENT URL IS: ', url
         br.open(url)
         year = date.today().year

src/calibre/constants.py

@@ -4,7 +4,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = u'calibre'
-numeric_version = (0, 9, 16)
+numeric_version = (0, 9, 17)
 __version__   = u'.'.join(map(unicode, numeric_version))
 __author__    = u"Kovid Goyal <kovid@kovidgoyal.net>"
 

src/calibre/customize/__init__.py

@@ -712,7 +712,7 @@ class ViewerPlugin(Plugin): # {{{
 
     def run_javascript(self, evaljs):
         '''
-        This method is called every time a document has finished laoding. Use
+        This method is called every time a document has finished loading. Use
         it in the same way as load_javascript().
         '''
         pass

src/calibre/db/backend.py

@@ -400,6 +400,7 @@ class DB(object):
         defs['gui_restriction'] = defs['cs_restriction'] = ''
         defs['categories_using_hierarchy'] = []
         defs['column_color_rules'] = []
+        defs['column_icon_rules'] = []
         defs['grouped_search_make_user_categories'] = []
         defs['similar_authors_search_key'] = 'authors'
         defs['similar_authors_match_kind'] = 'match_any'

src/calibre/db/categories.py

@@ -35,6 +35,8 @@ class Tag(object):
         self.avg_rating = avg/2.0 if avg is not None else 0
         self.sort = sort
         self.use_sort_as_name = use_sort_as_name
+        if tooltip is None:
+            tooltip = '(%s:%s)'%(category, name)
         if self.avg_rating > 0:
             if tooltip:
                 tooltip = tooltip + ': '
@@ -64,8 +66,8 @@ def find_categories(field_metadata):
 
 def create_tag_class(category, fm, icon_map):
     cat = fm[category]
+    dt = cat['datatype']
     icon = None
-    tooltip = None if category in {'formats', 'identifiers'} else ('(' + category + ')')
     label = fm.key_to_label(category)
     if icon_map:
         if not fm.is_custom_field(category):
@@ -75,20 +77,19 @@ def create_tag_class(category, fm, icon_map):
             icon = icon_map['custom:']
             icon_map[category] = icon
     is_editable = category not in {'news', 'rating', 'languages', 'formats',
-                                   'identifiers'}
+                                   'identifiers'} and dt != 'composite'
 
     if (tweaks['categories_use_field_for_author_name'] == 'author_sort' and
             (category == 'authors' or
                 (cat['display'].get('is_names', False) and
                 cat['is_custom'] and cat['is_multiple'] and
-                cat['datatype'] == 'text'))):
+                dt == 'text'))):
         use_sort_as_name = True
     else:
         use_sort_as_name = False
 
     return partial(Tag, use_sort_as_name=use_sort_as_name, icon=icon,
-                        tooltip=tooltip, is_editable=is_editable,
+                        is_editable=is_editable, category=category)
-                        category=category)
 
 def clean_user_categories(dbcache):
     user_cats = dbcache.pref('user_categories', {})

src/calibre/devices/android/driver.py

@@ -191,7 +191,7 @@ class ANDROID(USBMS):
             # Pantech
             0x10a9 : { 0x6050 : [0x227] },
 
-            # Prestigio
+            # Prestigio and Teclast
             0x2207 : { 0 : [0x222], 0x10 : [0x222] },
 
             }
@@ -215,7 +215,7 @@ class ANDROID(USBMS):
             'POCKET', 'ONDA_MID', 'ZENITHIN', 'INGENIC', 'PMID701C', 'PD',
             'PMP5097C', 'MASS', 'NOVO7', 'ZEKI', 'COBY', 'SXZ', 'USB_2.0',
             'COBY_MID', 'VS', 'AINOL', 'TOPWISE', 'PAD703', 'NEXT8D12',
-            'MEDIATEK', 'KEENHI']
+            'MEDIATEK', 'KEENHI', 'TECLAST']
     WINDOWS_MAIN_MEM = ['ANDROID_PHONE', 'A855', 'A853', 'INC.NEXUS_ONE',
             '__UMS_COMPOSITE', '_MB200', 'MASS_STORAGE', '_-_CARD', 'SGH-I897',
             'GT-I9000', 'FILE-STOR_GADGET', 'SGH-T959_CARD', 'SGH-T959', 'SAMSUNG_ANDROID',

src/calibre/ebooks/conversion/plugins/epub_input.py

@@ -9,6 +9,19 @@ from itertools import cycle
 from calibre.customize.conversion import InputFormatPlugin, OptionRecommendation
 
 ADOBE_OBFUSCATION =  'http://ns.adobe.com/pdf/enc#RC'
+IDPF_OBFUSCATION = 'http://www.idpf.org/2008/embedding'
+
+def decrypt_font(key, path, algorithm):
+    is_adobe = algorithm == ADOBE_OBFUSCATION
+    crypt_len = 1024 if is_adobe else 1040
+    with open(path, 'rb') as f:
+        raw = f.read()
+    crypt = bytearray(raw[:crypt_len])
+    key = cycle(iter(bytearray(key)))
+    decrypt = bytes(bytearray(x^key.next() for x in crypt))
+    with open(path, 'wb') as f:
+        f.write(decrypt)
+        f.write(raw[crypt_len:])
 
 class EPUBInput(InputFormatPlugin):
 
@@ -20,18 +33,6 @@ class EPUBInput(InputFormatPlugin):
 
     recommendations = set([('page_breaks_before', '/', OptionRecommendation.MED)])
 
-    def decrypt_font(self, key, path, algorithm):
-        is_adobe = algorithm == ADOBE_OBFUSCATION
-        crypt_len = 1024 if is_adobe else 1040
-        with open(path, 'rb') as f:
-            raw = f.read()
-        crypt = bytearray(raw[:crypt_len])
-        key = cycle(iter(bytearray(key)))
-        decrypt = bytes(bytearray(x^key.next() for x in crypt))
-        with open(path, 'wb') as f:
-            f.write(decrypt)
-            f.write(raw[crypt_len:])
-
     def process_encryption(self, encfile, opf, log):
         from lxml import etree
         import uuid, hashlib
@@ -58,8 +59,7 @@ class EPUBInput(InputFormatPlugin):
             root = etree.parse(encfile)
             for em in root.xpath('descendant::*[contains(name(), "EncryptionMethod")]'):
                 algorithm = em.get('Algorithm', '')
-                if algorithm not in {ADOBE_OBFUSCATION,
+                if algorithm not in {ADOBE_OBFUSCATION, IDPF_OBFUSCATION}:
-                        'http://www.idpf.org/2008/embedding'}:
                     return False
                 cr = em.getparent().xpath('descendant::*[contains(name(), "CipherReference")]')[0]
                 uri = cr.get('URI')
@@ -67,7 +67,7 @@ class EPUBInput(InputFormatPlugin):
                 tkey = (key if algorithm == ADOBE_OBFUSCATION else idpf_key)
                 if (tkey and os.path.exists(path)):
                     self._encrypted_font_uris.append(uri)
-                    self.decrypt_font(tkey, path, algorithm)
+                    decrypt_font(tkey, path, algorithm)
             return True
         except:
             import traceback

src/calibre/ebooks/conversion/plugins/pdf_output.py

@@ -99,6 +99,18 @@ class PDFOutput(OutputFormatPlugin):
             recommended_value=False, help=_(
                 'Generate an uncompressed PDF, useful for debugging, '
                 'only works with the new PDF engine.')),
+        OptionRecommendation(name='pdf_page_numbers', recommended_value=False,
+            help=_('Add page numbers to the bottom of every page in the generated PDF file. If you '
+                   'specify a footer template, it will take precedence '
+                   'over this option.')),
+        OptionRecommendation(name='pdf_footer_template', recommended_value=None,
+            help=_('An HTML template used to generate footers on every page.'
+                   ' The string _PAGENUM_ will be replaced by the current page'
+                   ' number.')),
+        OptionRecommendation(name='pdf_header_template', recommended_value=None,
+            help=_('An HTML template used to generate headers on every page.'
+                   ' The string _PAGENUM_ will be replaced by the current page'
+                   ' number.')),
         ])
 
     def convert(self, oeb_book, output_path, input_plugin, opts, log):

src/calibre/ebooks/metadata/pdf.py

@@ -15,6 +15,14 @@ from calibre.utils.ipc.simple_worker import fork_job, WorkerError
 
 #_isbn_pat = re.compile(r'ISBN[: ]*([-0-9Xx]+)')
 
+def get_tools():
+    from calibre.ebooks.pdf.pdftohtml import PDFTOHTML
+    base = os.path.dirname(PDFTOHTML)
+    suffix = '.exe' if iswindows else ''
+    pdfinfo = os.path.join(base, 'pdfinfo') + suffix
+    pdftoppm = os.path.join(base, 'pdftoppm') + suffix
+    return pdfinfo, pdftoppm
+
 def read_info(outputdir, get_cover):
     ''' Read info dict and cover from a pdf file named src.pdf in outputdir.
     Note that this function changes the cwd to outputdir and is therefore not
@@ -22,13 +30,8 @@ def read_info(outputdir, get_cover):
     way to pass unicode paths via command line arguments. This also ensures
     that if poppler crashes, no stale file handles are left for the original
     file, only for src.pdf.'''
-
-    from calibre.ebooks.pdf.pdftohtml import PDFTOHTML
     os.chdir(outputdir)
-    base = os.path.dirname(PDFTOHTML)
+    pdfinfo, pdftoppm = get_tools()
-    suffix = '.exe' if iswindows else ''
-    pdfinfo = os.path.join(base, 'pdfinfo') + suffix
-    pdftoppm = os.path.join(base, 'pdftoppm') + suffix
 
     try:
         raw = subprocess.check_output([pdfinfo, '-enc', 'UTF-8', 'src.pdf'])
@@ -58,6 +61,20 @@ def read_info(outputdir, get_cover):
 
     return ans
 
+def page_images(pdfpath, outputdir, first=1, last=1):
+    pdftoppm = get_tools()[1]
+    outputdir = os.path.abspath(outputdir)
+    args = {}
+    if iswindows:
+        import win32process as w
+        args['creationflags'] = w.HIGH_PRIORITY_CLASS | w.CREATE_NO_WINDOW
+    try:
+        subprocess.check_call([pdftoppm, '-jpeg', '-f', unicode(first),
+                               '-l', unicode(last), pdfpath,
+                               os.path.join(outputdir, 'page-images')], **args)
+    except subprocess.CalledProcessError as e:
+        raise ValueError('Failed to render PDF, pdftoppm errorcode: %s'%e.returncode)
+
 def get_metadata(stream, cover=True):
     with TemporaryDirectory('_pdf_metadata_read') as pdfpath:
         stream.seek(0)

src/calibre/ebooks/metadata/sources/amazon.py

@@ -614,10 +614,14 @@ class Amazon(Source):
         return domain
 
     def clean_downloaded_metadata(self, mi):
-        if mi.title and self.domain in ('com', 'uk'):
+        docase = (
+            mi.language == 'eng' or
+            (mi.is_null('language') and self.domain in {'com', 'uk'})
+        )
+        if mi.title and docase:
             mi.title = fixcase(mi.title)
         mi.authors = fixauthors(mi.authors)
-        if self.domain in ('com', 'uk'):
+        if mi.tags and docase:
             mi.tags = list(map(fixcase, mi.tags))
         mi.isbn = check_isbn(mi.isbn)
 

src/calibre/ebooks/metadata/sources/base.py

@@ -418,10 +418,12 @@ class Source(Plugin):
         before putting the Metadata object into result_queue. You can of
         course, use a custom algorithm suited to your metadata source.
         '''
-        if mi.title:
+        docase = mi.language == 'eng' or mi.is_null('language')
+        if docase and mi.title:
             mi.title = fixcase(mi.title)
         mi.authors = fixauthors(mi.authors)
-        mi.tags = list(map(fixcase, mi.tags))
+        if mi.tags and docase:
+            mi.tags = list(map(fixcase, mi.tags))
         mi.isbn = check_isbn(mi.isbn)
 
     # }}}

src/calibre/ebooks/oeb/display/paged.coffee

@@ -29,6 +29,10 @@ class PagedDisplay
         this.current_page_height = null
         this.document_margins = null
         this.use_document_margins = false
+        this.footer_template = null
+        this.header_template = null
+        this.header = null
+        this.footer = null
 
     read_document_margins: () ->
         # Read page margins from the document. First checks for an @page rule.
@@ -102,6 +106,7 @@ class PagedDisplay
             # than max_col_width
             sm += Math.ceil( (col_width - this.max_col_width) / 2*n )
             col_width = Math.max(100, ((ww - adjust)/n) - 2*sm)
+        this.col_width = col_width
         this.page_width = col_width + 2*sm
         this.screen_width = this.page_width * this.cols_per_screen
         this.current_page_height = window.innerHeight - this.margin_top - this.margin_bottom
@@ -171,6 +176,30 @@ class PagedDisplay
         # log('Time to layout:', new Date().getTime() - start_time)
         return sm
 
+    create_header_footer: () ->
+        if this.header_template != null
+            this.header = document.createElement('div')
+            this.header.setAttribute('style', "overflow:hidden; display:block; position:absolute; left:#{ this.side_margin }px; top: 0px; height: #{ this.margin_top }px; width: #{ this.col_width }px; margin: 0; padding: 0")
+            document.body.appendChild(this.header)
+        if this.footer_template != null
+            this.footer = document.createElement('div')
+            this.footer.setAttribute('style', "overflow:hidden; display:block; position:absolute; left:#{ this.side_margin }px; top: #{ window.innerHeight - this.margin_bottom }px; height: #{ this.margin_bottom }px; width: #{ this.col_width }px; margin: 0; padding: 0")
+            document.body.appendChild(this.footer)
+        this.update_header_footer(1)
+
+    position_header_footer: () ->
+        [left, top] = calibre_utils.viewport_to_document(0, 0, document.body.ownerDocument)
+        if this.header != null
+            this.header.style.setProperty('left', left+'px')
+        if this.footer != null
+            this.footer.style.setProperty('left', left+'px')
+
+    update_header_footer: (pagenum) ->
+        if this.header != null
+            this.header.innerHTML = this.header_template.replace(/_PAGENUM_/g, pagenum+"")
+        if this.footer != null
+            this.footer.innerHTML = this.footer_template.replace(/_PAGENUM_/g, pagenum+"")
+
     fit_images: () ->
         # Ensure no images are wider than the available width in a column. Note
         # that this method use getBoundingClientRect() which means it will

src/calibre/ebooks/oeb/parse_utils.py

@@ -340,6 +340,11 @@ def parse_html(data, log=None, decoder=None, preprocessor=None,
             nroot.append(elem)
         data = nroot
 
+    fnsmap = {k:v for k, v in data.nsmap.iteritems() if v != XHTML_NS}
+    fnsmap[None] = XHTML_NS
+    if fnsmap != dict(data.nsmap):
+        # Remove non default prefixes referring to the XHTML namespace
+        data = clone_element(data, nsmap=fnsmap, in_context=False)
 
     data = merge_multiple_html_heads_and_bodies(data, log)
     # Ensure has a <head/>

src/calibre/ebooks/oeb/polish/__init__.py

@@ -0,0 +1,11 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2013, Kovid Goyal <kovid at kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+
+

src/calibre/ebooks/oeb/polish/container.py

@@ -0,0 +1,354 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2013, Kovid Goyal <kovid at kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+import os, posixpath, logging, sys, hashlib, uuid
+from urllib import unquote as urlunquote
+
+from lxml import etree
+
+from calibre import guess_type, CurrentDir
+from calibre.ebooks.chardet import xml_to_unicode
+from calibre.ebooks.conversion.plugins.epub_input import (
+    ADOBE_OBFUSCATION, IDPF_OBFUSCATION, decrypt_font)
+from calibre.ebooks.conversion.preprocess import HTMLPreProcessor, CSSPreProcessor
+from calibre.ebooks.mobi import MobiError
+from calibre.ebooks.mobi.reader.headers import MetadataHeader
+from calibre.ebooks.oeb.base import OEB_DOCS, _css_logger, OEB_STYLES, OPF2_NS
+from calibre.ebooks.oeb.polish.errors import InvalidBook, DRMError
+from calibre.ebooks.oeb.parse_utils import NotHTML, parse_html, RECOVER_PARSER
+from calibre.ptempfile import PersistentTemporaryDirectory
+from calibre.utils.fonts.sfnt.container import Sfnt
+from calibre.utils.ipc.simple_worker  import fork_job, WorkerError
+from calibre.utils.logging import default_log
+from calibre.utils.zipfile import ZipFile
+
+exists, join, relpath = os.path.exists, os.path.join, os.path.relpath
+
+OEB_FONTS = {guess_type('a.ttf')[0], guess_type('b.ttf')[0]}
+
+class Container(object):
+
+    def __init__(self, rootpath, opfpath, log):
+        self.root = os.path.abspath(rootpath)
+        self.log = log
+        self.html_preprocessor = HTMLPreProcessor()
+        self.css_preprocessor = CSSPreProcessor()
+
+        self.parsed_cache = {}
+        self.mime_map = {}
+        self.name_path_map = {}
+
+        # Map of relative paths with '/' separators from root of unzipped ePub
+        # to absolute paths on filesystem with os-specific separators
+        opfpath = os.path.abspath(opfpath)
+        for dirpath, _dirnames, filenames in os.walk(self.root):
+            for f in filenames:
+                path = join(dirpath, f)
+                name = relpath(path, self.root).replace(os.sep, '/')
+                self.name_path_map[name] = path
+                self.mime_map[name] = guess_type(path)[0]
+                # Special case if we have stumbled onto the opf
+                if path == opfpath:
+                    self.opf_name = name
+                    self.opf_dir = posixpath.dirname(path)
+                    self.mime_map[name] = guess_type('a.opf')[0]
+
+        # Update mime map with data from the OPF
+        for item in self.opf.xpath(
+                '//opf:manifest/opf:item[@href and @media-type]',
+                namespaces={'opf':OPF2_NS}):
+            href = item.get('href')
+            self.mime_map[self.href_to_name(href)] = item.get('media-type')
+
+
+    def href_to_name(self, href, base=None):
+        if base is None:
+            base = self.opf_dir
+        href = urlunquote(href.partition('#')[0])
+        fullpath = posixpath.abspath(posixpath.join(base, href))
+        return self.relpath(fullpath)
+
+    def relpath(self, path):
+        return relpath(path, self.root)
+
+    def decode(self, data):
+        """Automatically decode :param:`data` into a `unicode` object."""
+        def fix_data(d):
+            return d.replace('\r\n', '\n').replace('\r', '\n')
+        if isinstance(data, unicode):
+            return fix_data(data)
+        bom_enc = None
+        if data[:4] in {b'\0\0\xfe\xff', b'\xff\xfe\0\0'}:
+            bom_enc = {b'\0\0\xfe\xff':'utf-32-be',
+                       b'\xff\xfe\0\0':'utf-32-le'}[data[:4]]
+            data = data[4:]
+        elif data[:2] in {b'\xff\xfe', b'\xfe\xff'}:
+            bom_enc = {b'\xff\xfe':'utf-16-le', b'\xfe\xff':'utf-16-be'}[data[:2]]
+            data = data[2:]
+        elif data[:3] == b'\xef\xbb\xbf':
+            bom_enc = 'utf-8'
+            data = data[3:]
+        if bom_enc is not None:
+            try:
+                return fix_data(data.decode(bom_enc))
+            except UnicodeDecodeError:
+                pass
+        try:
+            return fix_data(data.decode('utf-8'))
+        except UnicodeDecodeError:
+            pass
+        data, _ = xml_to_unicode(data)
+        return fix_data(data)
+
+    def parse_xml(self, data):
+        data = xml_to_unicode(data, strip_encoding_pats=True, assume_utf8=True,
+                             resolve_entities=True)[0].strip()
+        return etree.fromstring(data, parser=RECOVER_PARSER)
+
+    def parse_xhtml(self, data, fname):
+        try:
+            return parse_html(data, log=self.log,
+                    decoder=self.decode,
+                    preprocessor=self.html_preprocessor,
+                    filename=fname, non_html_file_tags={'ncx'})
+        except NotHTML:
+            return self.parse_xml(data)
+
+    def parse(self, path, mime):
+        with open(path, 'rb') as src:
+            data = src.read()
+        if mime in OEB_DOCS:
+            data = self.parse_xhtml(data, self.relpath(path))
+        elif mime[-4:] in {'+xml', '/xml'}:
+            data = self.parse_xml(data)
+        elif mime in OEB_STYLES:
+            data = self.parse_css(data, self.relpath(path))
+        elif mime in OEB_FONTS or path.rpartition('.')[-1].lower() in {'ttf', 'otf'}:
+            data = Sfnt(data)
+        return data
+
+    def parse_css(self, data, fname):
+        from cssutils import CSSParser, log
+        log.setLevel(logging.WARN)
+        log.raiseExceptions = False
+        data = self.decode(data)
+        data = self.css_preprocessor(data, add_namespace=False)
+        parser = CSSParser(loglevel=logging.WARNING,
+                           # We dont care about @import rules
+                           fetcher=lambda x: (None, None), log=_css_logger)
+        data = parser.parseString(data, href=fname, validate=False)
+        return data
+
+    def parsed(self, name):
+        ans = self.parsed_cache.get(name, None)
+        if ans is None:
+            mime = self.mime_map.get(name, guess_type(name)[0])
+            ans = self.parse(self.name_path_map[name], mime)
+            self.parsed_cache[name] = ans
+        return ans
+
+    @property
+    def opf(self):
+        return self.parsed(self.opf_name)
+
+    @property
+    def spine_items(self):
+        manifest_id_map = {item.get('id'):self.href_to_name(item.get('href'))
+            for item in self.opf.xpath('//opf:manifest/opf:item[@href and @id]',
+                namespaces={'opf':OPF2_NS})}
+
+        linear, non_linear = [], []
+        for item in self.opf.xpath('//opf:spine/opf:itemref[@idref]',
+                                   namespaces={'opf':OPF2_NS}):
+            idref = item.get('idref')
+            name = manifest_id_map.get(idref, None)
+            path = self.name_path_map.get(name, None)
+            if path:
+                if item.get('linear', 'yes') == 'yes':
+                    yield path
+                else:
+                    non_linear.append(path)
+        for path in non_linear:
+            yield path
+
+class InvalidEpub(InvalidBook):
+    pass
+
+OCF_NS = 'urn:oasis:names:tc:opendocument:xmlns:container'
+
+class EpubContainer(Container):
+
+    META_INF = {
+            'container.xml' : True,
+            'manifest.xml' : False,
+            'encryption.xml' : False,
+            'metadata.xml' : False,
+            'signatures.xml' : False,
+            'rights.xml' : False,
+    }
+
+    def __init__(self, pathtoepub, log):
+        self.pathtoepub = pathtoepub
+        tdir = self.root = PersistentTemporaryDirectory('_epub_container')
+        with open(self.pathtoepub, 'rb') as stream:
+            try:
+                zf = ZipFile(stream)
+                zf.extractall(tdir)
+            except:
+                log.exception('EPUB appears to be invalid ZIP file, trying a'
+                        ' more forgiving ZIP parser')
+                from calibre.utils.localunzip import extractall
+                stream.seek(0)
+                extractall(stream)
+        try:
+            os.remove(join(tdir, 'mimetype'))
+        except EnvironmentError:
+            pass
+
+        container_path = join(self.root, 'META-INF', 'container.xml')
+        if not exists(container_path):
+            raise InvalidEpub('No META-INF/container.xml in epub')
+        self.container = etree.fromstring(open(container_path, 'rb').read())
+        opf_files = self.container.xpath((
+            r'child::ocf:rootfiles/ocf:rootfile'
+            '[@media-type="%s" and @full-path]'%guess_type('a.opf')[0]
+            ), namespaces={'ocf':OCF_NS}
+        )
+        if not opf_files:
+            raise InvalidEpub('META-INF/container.xml contains no link to OPF file')
+        opf_path = os.path.join(self.root, *opf_files[0].get('full-path').split('/'))
+        if not exists(opf_path):
+            raise InvalidEpub('OPF file does not exist at location pointed to'
+                    ' by META-INF/container.xml')
+
+        super(EpubContainer, self).__init__(tdir, opf_path, log)
+
+        self.obfuscated_fonts = {}
+        if 'META-INF/encryption.xml' in self.name_path_map:
+            self.process_encryption()
+
+    def process_encryption(self):
+        fonts = {}
+        enc = self.parsed('META-INF/encryption.xml')
+        for em in enc.xpath('//*[local-name()="EncryptionMethod" and @Algorithm]'):
+            alg = em.get('Algorithm')
+            if alg not in {ADOBE_OBFUSCATION, IDPF_OBFUSCATION}:
+                raise DRMError()
+            cr = em.getparent().xpath('descendant::*[local-name()="CipherReference" and @URI]')[0]
+            name = self.href_to_name(cr.get('URI'), self.root)
+            path = self.name_path_map.get(name, None)
+            if path is not None:
+                fonts[name] = alg
+        if not fonts:
+            return
+
+        package_id = unique_identifier = idpf_key = None
+        for attrib, val in self.opf.attrib.iteritems():
+            if attrib.endswith('unique-identifier'):
+                package_id = val
+                break
+        if package_id is not None:
+            for elem in self.opf.xpath('//*[@id=%r]'%package_id):
+                if elem.text:
+                    unique_identifier = elem.text.rpartition(':')[-1]
+                    break
+        if unique_identifier is not None:
+            idpf_key = hashlib.sha1(unique_identifier).digest()
+        key = None
+        for item in self.opf.xpath('//*[local-name()="metadata"]/*'
+                                   '[local-name()="identifier"]'):
+            scheme = None
+            for xkey in item.attrib.keys():
+                if xkey.endswith('scheme'):
+                    scheme = item.get(xkey)
+            if (scheme and scheme.lower() == 'uuid') or \
+                    (item.text and item.text.startswith('urn:uuid:')):
+                try:
+                    key = bytes(item.text).rpartition(':')[-1]
+                    key = uuid.UUID(key).bytes
+                except:
+                    self.log.exception('Failed to parse obfuscation key')
+                    key = None
+
+        for font, alg in fonts.iteritems():
+            path = self.name_path_map[font]
+            tkey = key if alg == ADOBE_OBFUSCATION else idpf_key
+            if not tkey:
+                raise InvalidBook('Failed to find obfuscation key')
+            decrypt_font(tkey, path, alg)
+            self.obfuscated_fonts[name] = (alg, tkey)
+
+class InvalidMobi(InvalidBook):
+    pass
+
+def do_explode(path, dest):
+    from calibre.ebooks.mobi.reader.mobi6 import MobiReader
+    from calibre.ebooks.mobi.reader.mobi8 import Mobi8Reader
+    with open(path, 'rb') as stream:
+        mr = MobiReader(stream, default_log, None, None)
+
+        with CurrentDir(dest):
+            mr = Mobi8Reader(mr, default_log)
+            opf = os.path.abspath(mr())
+            obfuscated_fonts = mr.encrypted_fonts
+            try:
+                os.remove('debug-raw.html')
+            except:
+                pass
+
+    return opf, obfuscated_fonts
+
+class AZW3Container(Container):
+
+    def __init__(self, pathtoazw3, log):
+        self.pathtoazw3 = pathtoazw3
+        tdir = self.root = PersistentTemporaryDirectory('_azw3_container')
+        with open(pathtoazw3, 'rb') as stream:
+            raw = stream.read(3)
+            if raw == b'TPZ':
+                raise InvalidMobi(_('This is not a MOBI file. It is a Topaz file.'))
+
+            try:
+                header = MetadataHeader(stream, default_log)
+            except MobiError:
+                raise InvalidMobi(_('This is not a MOBI file.'))
+
+            if header.encryption_type != 0:
+                raise DRMError()
+
+            kf8_type = header.kf8_type
+
+            if kf8_type is None:
+                raise InvalidMobi(_('This MOBI file does not contain a KF8 format '
+                        'book. KF8 is the new format from Amazon. calibre can '
+                        'only edit MOBI files that contain KF8 books. Older '
+                        'MOBI files without KF8 are not editable.'))
+
+            if kf8_type == 'joint':
+                raise InvalidMobi(_('This MOBI file contains both KF8 and '
+                    'older Mobi6 data. calibre can only edit MOBI files '
+                    'that contain only KF8 data.'))
+
+        try:
+            opf_path, obfuscated_fonts = fork_job(
+            'calibre.ebooks.oeb.polish.container', 'do_explode',
+            args=(pathtoazw3, tdir), no_output=True)['result']
+        except WorkerError as e:
+            log(e.orig_tb)
+            raise InvalidMobi('Failed to explode MOBI')
+        super(AZW3Container, self).__init__(tdir, opf_path, log)
+        self.obfuscated_fonts = {x.replace(os.sep, '/') for x in obfuscated_fonts}
+
+if __name__ == '__main__':
+    f = sys.argv[-1]
+    ebook = (AZW3Container if f.rpartition('.')[-1].lower() in {'azw3', 'mobi'}
+            else EpubContainer)(f, default_log)
+    for s in ebook.spine_items:
+        print (ebook.relpath(s))
+

src/calibre/ebooks/oeb/polish/errors.py

@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2013, Kovid Goyal <kovid at kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+from calibre.ebooks import DRMError as _DRMError
+
+class InvalidBook(ValueError):
+    pass
+
+class DRMError(_DRMError):
+    def __init__(self):
+        super(DRMError, self).__init__(_('This file is locked with DRM. It cannot be edited.'))
+

src/calibre/ebooks/oeb/polish/stats.py

@@ -0,0 +1,99 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2013, Kovid Goyal <kovid at kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+import json
+
+from PyQt4.Qt import (QWebPage, pyqtProperty, QString, QEventLoop, QWebView,
+                      Qt, QSize, QTimer)
+
+from calibre.ebooks.oeb.display.webview import load_html
+from calibre.gui2 import must_use_qt
+
+class Page(QWebPage):
+
+    def __init__(self, log):
+        self.log = log
+        QWebPage.__init__(self)
+
+    def javaScriptConsoleMessage(self, msg, lineno, msgid):
+        self.log(u'JS:', unicode(msg))
+
+    def javaScriptAlert(self, frame, msg):
+        self.log(unicode(msg))
+
+    def shouldInterruptJavaScript(self):
+        return False
+
+    def _pass_json_value_getter(self):
+        val = json.dumps(self.bridge_value)
+        return QString(val)
+
+    def _pass_json_value_setter(self, value):
+        self.bridge_value = json.loads(unicode(value))
+
+    _pass_json_value = pyqtProperty(QString, fget=_pass_json_value_getter,
+            fset=_pass_json_value_setter)
+
+class StatsCollector(object):
+
+    def __init__(self, container):
+        self.container = container
+        self.log = self.logger = container.log
+        must_use_qt()
+
+        self.loop = QEventLoop()
+        self.view = QWebView()
+        self.page = Page(self.log)
+        self.view.setPage(self.page)
+        self.page.setViewportSize(QSize(1200, 1600))
+
+        self.view.loadFinished.connect(self.collect,
+                type=Qt.QueuedConnection)
+
+        self.render_queue = list(container.spine_items)
+        self.font_stats = {}
+
+        QTimer.singleShot(0, self.render_book)
+
+        if self.loop.exec_() == 1:
+            raise Exception('Failed to gather statistics from book, see log for details')
+
+    def render_book(self):
+        try:
+            if not self.render_queue:
+                self.loop.exit()
+            else:
+                self.render_next()
+        except:
+            self.logger.exception('Rendering failed')
+            self.loop.exit(1)
+
+    def render_next(self):
+        item = unicode(self.render_queue.pop(0))
+        self.current_item = item
+        load_html(item, self.view)
+
+    def collect(self, ok):
+        if not ok:
+            self.log.error('Failed to render document: %s'%self.container.relpath(self.current_item))
+            self.loop.exit(1)
+            return
+        try:
+            self.collect_font_stats()
+        except:
+            self.log.exception('Failed to collect font stats from: %s'%self.container.relpath(self.current_item))
+            self.loop.exit(1)
+            return
+
+        self.render_book()
+
+    def collect_font_stats(self):
+        pass
+
+

src/calibre/ebooks/oeb/transforms/subset.py

@@ -280,7 +280,7 @@ class SubsetFonts(object):
         return ans
 
     def find_usage_in(self, elem, inherited_style):
-        style = self.elem_style(elem.get('class', ''), inherited_style)
+        style = self.elem_style(elem.get('class', '') or '', inherited_style)
         for child in elem:
             self.find_usage_in(child, style)
         font = self.used_font(style)

src/calibre/ebooks/pdf/render/from_html.py

@@ -161,6 +161,17 @@ class PDFWriter(QObject):
                              debug=self.log.debug, compress=not
                              opts.uncompressed_pdf,
                              mark_links=opts.pdf_mark_links)
+        self.footer = opts.pdf_footer_template
+        if self.footer is None and opts.pdf_page_numbers:
+            self.footer = '<p style="text-align:center; text-indent: 0">_PAGENUM_</p>'
+        self.header = opts.pdf_header_template
+        min_margin = 36
+        if self.footer and opts.margin_bottom < min_margin:
+            self.log.warn('Bottom margin is too small for footer, increasing it.')
+            opts.margin_bottom = min_margin
+        if self.header and opts.margin_top < min_margin:
+            self.log.warn('Top margin is too small for header, increasing it.')
+            opts.margin_top = min_margin
 
         self.page.setViewportSize(QSize(self.doc.width(), self.doc.height()))
         self.render_queue = items
@@ -225,6 +236,7 @@ class PDFWriter(QObject):
             except:
                 self.log.exception('Rendering failed')
                 self.loop.exit(1)
+                return
         else:
             # The document is so corrupt that we can't render the page.
             self.logger.error('Document cannot be rendered.')
@@ -264,6 +276,15 @@ class PDFWriter(QObject):
         py_bridge.value = book_indexing.all_links_and_anchors();
         '''%(self.margin_top, 0, self.margin_bottom))
 
+        if self.header:
+            self.bridge_value = self.header
+            evaljs('paged_display.header_template = py_bridge.value')
+        if self.footer:
+            self.bridge_value = self.footer
+            evaljs('paged_display.footer_template = py_bridge.value')
+        if self.header or self.footer:
+            evaljs('paged_display.create_header_footer();')
+
         amap = self.bridge_value
         if not isinstance(amap, dict):
             amap = {'links':[], 'anchors':{}} # Some javascript error occurred
@@ -272,6 +293,8 @@ class PDFWriter(QObject):
         mf = self.view.page().mainFrame()
         while True:
             self.doc.init_page()
+            if self.header or self.footer:
+                evaljs('paged_display.update_header_footer(%d)'%self.current_page_num)
             self.painter.save()
             mf.render(self.painter)
             self.painter.restore()
@@ -279,7 +302,7 @@ class PDFWriter(QObject):
             self.doc.end_page()
             if not nsl[1] or nsl[0] <= 0:
                 break
-            evaljs('window.scrollTo(%d, 0)'%nsl[0])
+            evaljs('window.scrollTo(%d, 0); paged_display.position_header_footer();'%nsl[0])
             if self.doc.errors_occurred:
                 break
 

src/calibre/gui2/__init__.py

@@ -22,91 +22,94 @@ from calibre.utils.date import UNDEFINED_DATE
 
 # Setup gprefs {{{
 gprefs = JSONConfig('gui')
+defs = gprefs.defaults
 
 if isosx:
-    gprefs.defaults['action-layout-menubar'] = (
+    defs['action-layout-menubar'] = (
         'Add Books', 'Edit Metadata', 'Convert Books',
         'Choose Library', 'Save To Disk', 'Preferences',
         'Help',
         )
-    gprefs.defaults['action-layout-menubar-device'] = (
+    defs['action-layout-menubar-device'] = (
         'Add Books', 'Edit Metadata', 'Convert Books',
         'Location Manager', 'Send To Device',
         'Save To Disk', 'Preferences', 'Help',
         )
-    gprefs.defaults['action-layout-toolbar'] = (
+    defs['action-layout-toolbar'] = (
         'Add Books', 'Edit Metadata', None, 'Convert Books', 'View', None,
         'Choose Library', 'Donate', None, 'Fetch News', 'Store', 'Save To Disk',
         'Connect Share', None, 'Remove Books',
         )
-    gprefs.defaults['action-layout-toolbar-device'] = (
+    defs['action-layout-toolbar-device'] = (
         'Add Books', 'Edit Metadata', None, 'Convert Books', 'View',
         'Send To Device', None, None, 'Location Manager', None, None,
         'Fetch News', 'Store', 'Save To Disk', 'Connect Share', None,
         'Remove Books',
         )
 else:
-    gprefs.defaults['action-layout-menubar'] = ()
+    defs['action-layout-menubar'] = ()
-    gprefs.defaults['action-layout-menubar-device'] = ()
+    defs['action-layout-menubar-device'] = ()
-    gprefs.defaults['action-layout-toolbar'] = (
+    defs['action-layout-toolbar'] = (
         'Add Books', 'Edit Metadata', None, 'Convert Books', 'View', None,
         'Store', 'Donate', 'Fetch News', 'Help', None,
         'Remove Books', 'Choose Library', 'Save To Disk',
         'Connect Share', 'Preferences',
         )
-    gprefs.defaults['action-layout-toolbar-device'] = (
+    defs['action-layout-toolbar-device'] = (
         'Add Books', 'Edit Metadata', None, 'Convert Books', 'View',
         'Send To Device', None, None, 'Location Manager', None, None,
         'Fetch News', 'Save To Disk', 'Store', 'Connect Share', None,
         'Remove Books', None, 'Help', 'Preferences',
         )
 
-gprefs.defaults['action-layout-toolbar-child'] = ()
+defs['action-layout-toolbar-child'] = ()
 
-gprefs.defaults['action-layout-context-menu'] = (
+defs['action-layout-context-menu'] = (
         'Edit Metadata', 'Send To Device', 'Save To Disk',
         'Connect Share', 'Copy To Library', None,
         'Convert Books', 'View', 'Open Folder', 'Show Book Details',
         'Similar Books', 'Tweak ePub', None, 'Remove Books',
         )
 
-gprefs.defaults['action-layout-context-menu-device'] = (
+defs['action-layout-context-menu-device'] = (
         'View', 'Save To Disk', None, 'Remove Books', None,
         'Add To Library', 'Edit Collections',
         )
 
-gprefs.defaults['action-layout-context-menu-cover-browser'] = (
+defs['action-layout-context-menu-cover-browser'] = (
         'Edit Metadata', 'Send To Device', 'Save To Disk',
         'Connect Share', 'Copy To Library', None,
         'Convert Books', 'View', 'Open Folder', 'Show Book Details',
         'Similar Books', 'Tweak ePub', None, 'Remove Books',
         )
 
-gprefs.defaults['show_splash_screen'] = True
+defs['show_splash_screen'] = True
-gprefs.defaults['toolbar_icon_size'] = 'medium'
+defs['toolbar_icon_size'] = 'medium'
-gprefs.defaults['automerge'] = 'ignore'
+defs['automerge'] = 'ignore'
-gprefs.defaults['toolbar_text'] = 'always'
+defs['toolbar_text'] = 'always'
-gprefs.defaults['font'] = None
+defs['font'] = None
-gprefs.defaults['tags_browser_partition_method'] = 'first letter'
+defs['tags_browser_partition_method'] = 'first letter'
-gprefs.defaults['tags_browser_collapse_at'] = 100
+defs['tags_browser_collapse_at'] = 100
-gprefs.defaults['tag_browser_dont_collapse'] = []
+defs['tag_browser_dont_collapse'] = []
-gprefs.defaults['edit_metadata_single_layout'] = 'default'
+defs['edit_metadata_single_layout'] = 'default'
-gprefs.defaults['default_author_link'] = 'http://en.wikipedia.org/w/index.php?search={author}'
+defs['default_author_link'] = 'http://en.wikipedia.org/w/index.php?search={author}'
-gprefs.defaults['preserve_date_on_ctl'] = True
+defs['preserve_date_on_ctl'] = True
-gprefs.defaults['manual_add_auto_convert'] = False
+defs['manual_add_auto_convert'] = False
-gprefs.defaults['cb_fullscreen'] = False
+defs['cb_fullscreen'] = False
-gprefs.defaults['worker_max_time'] = 0
+defs['worker_max_time'] = 0
-gprefs.defaults['show_files_after_save'] = True
+defs['show_files_after_save'] = True
-gprefs.defaults['auto_add_path'] = None
+defs['auto_add_path'] = None
-gprefs.defaults['auto_add_check_for_duplicates'] = False
+defs['auto_add_check_for_duplicates'] = False
-gprefs.defaults['blocked_auto_formats'] = []
+defs['blocked_auto_formats'] = []
-gprefs.defaults['auto_add_auto_convert'] = True
+defs['auto_add_auto_convert'] = True
-gprefs.defaults['ui_style'] = 'calibre' if iswindows or isosx else 'system'
+defs['ui_style'] = 'calibre' if iswindows or isosx else 'system'
-gprefs.defaults['tag_browser_old_look'] = False
+defs['tag_browser_old_look'] = False
-gprefs.defaults['book_list_tooltips'] = True
+defs['book_list_tooltips'] = True
-gprefs.defaults['bd_show_cover'] = True
+defs['bd_show_cover'] = True
-gprefs.defaults['bd_overlay_cover_size'] = False
+defs['bd_overlay_cover_size'] = False
-gprefs.defaults['tags_browser_category_icons'] = {}
+defs['tags_browser_category_icons'] = {}
+defs['cover_browser_reflections'] = True
+del defs
 # }}}
 
 NONE = QVariant() #: Null value to return from the data function of item models

src/calibre/gui2/convert/pdf_output.py

@@ -22,7 +22,7 @@ class PluginWidget(Widget, Ui_Form):
             'override_profile_size', 'paper_size', 'custom_size',
             'preserve_cover_aspect_ratio', 'pdf_serif_family', 'unit',
             'pdf_sans_family', 'pdf_mono_family', 'pdf_standard_font',
-            'pdf_default_font_size', 'pdf_mono_font_size'])
+            'pdf_default_font_size', 'pdf_mono_font_size', 'pdf_page_numbers'])
         self.db, self.book_id = db, book_id
 
         for x in get_option('paper_size').option.choices:

src/calibre/gui2/convert/pdf_output.ui

@@ -84,7 +84,7 @@
      </property>
     </widget>
    </item>
-   <item row="5" column="0">
+   <item row="6" column="0">
     <widget class="QLabel" name="label_4">
      <property name="text">
       <string>Se&amp;rif family:</string>
@@ -94,10 +94,10 @@
      </property>
     </widget>
    </item>
-   <item row="5" column="1">
+   <item row="6" column="1">
     <widget class="QFontComboBox" name="opt_pdf_serif_family"/>
    </item>
-   <item row="6" column="0">
+   <item row="7" column="0">
     <widget class="QLabel" name="label_5">
      <property name="text">
       <string>&amp;Sans family:</string>
@@ -107,10 +107,10 @@
      </property>
     </widget>
    </item>
-   <item row="6" column="1">
+   <item row="7" column="1">
     <widget class="QFontComboBox" name="opt_pdf_sans_family"/>
    </item>
-   <item row="7" column="0">
+   <item row="8" column="0">
     <widget class="QLabel" name="label_6">
      <property name="text">
       <string>&amp;Monospace family:</string>
@@ -120,10 +120,10 @@
      </property>
     </widget>
    </item>
-   <item row="7" column="1">
+   <item row="8" column="1">
     <widget class="QFontComboBox" name="opt_pdf_mono_family"/>
    </item>
-   <item row="8" column="0">
+   <item row="9" column="0">
     <widget class="QLabel" name="label_7">
      <property name="text">
       <string>S&amp;tandard font:</string>
@@ -133,10 +133,10 @@
      </property>
     </widget>
    </item>
-   <item row="8" column="1">
+   <item row="9" column="1">
     <widget class="QComboBox" name="opt_pdf_standard_font"/>
    </item>
-   <item row="9" column="0">
+   <item row="10" column="0">
     <widget class="QLabel" name="label_8">
      <property name="text">
       <string>Default font si&amp;ze:</string>
@@ -146,14 +146,14 @@
      </property>
     </widget>
    </item>
-   <item row="9" column="1">
+   <item row="10" column="1">
     <widget class="QSpinBox" name="opt_pdf_default_font_size">
      <property name="suffix">
       <string> px</string>
      </property>
     </widget>
    </item>
-   <item row="10" column="0">
+   <item row="11" column="0">
     <widget class="QLabel" name="label_9">
      <property name="text">
       <string>Monospace &amp;font size:</string>
@@ -163,14 +163,14 @@
      </property>
     </widget>
    </item>
-   <item row="10" column="1">
+   <item row="11" column="1">
     <widget class="QSpinBox" name="opt_pdf_mono_font_size">
      <property name="suffix">
       <string> px</string>
      </property>
     </widget>
    </item>
-   <item row="11" column="0">
+   <item row="12" column="0">
     <spacer name="verticalSpacer">
      <property name="orientation">
       <enum>Qt::Vertical</enum>
@@ -183,6 +183,13 @@
      </property>
     </spacer>
    </item>
+   <item row="5" column="0" colspan="2">
+    <widget class="QCheckBox" name="opt_pdf_page_numbers">
+     <property name="text">
+      <string>Add page &amp;numbers to the bottom of every page</string>
+     </property>
+    </widget>
+   </item>
   </layout>
  </widget>
  <resources/>

src/calibre/gui2/cover_flow.py

@@ -106,6 +106,8 @@ if pictureflow is not None:
             self.setContextMenuPolicy(Qt.DefaultContextMenu)
             if hasattr(self, 'setSubtitleFont'):
                 self.setSubtitleFont(QFont(rating_font()))
+            if not gprefs['cover_browser_reflections']:
+                self.setShowReflections(False)
 
         def set_context_menu(self, cm):
             self.context_menu = cm

src/calibre/gui2/dialogs/tag_list_editor.py

@@ -136,7 +136,7 @@ class TagListEditor(QDialog, Ui_TagListEditor):
             item.setFlags (item.flags() & ~Qt.ItemIsSelectable)
             self.table.setItem(row, 1, item)
             item = QTableWidgetItem('')
-            item.setFlags (item.flags() & ~Qt.ItemIsSelectable)
+            item.setFlags (item.flags() & ~(Qt.ItemIsSelectable|Qt.ItemIsEditable))
             self.table.setItem(row, 2, item)
 
         # Scroll to the selected item if there is one

src/calibre/gui2/library/models.py

@@ -24,7 +24,7 @@ from calibre.db.search import _match, CONTAINS_MATCH, EQUALS_MATCH, REGEXP_MATCH
 from calibre.library.caches import (MetadataBackup, force_to_bool)
 from calibre.library.save_to_disk import find_plugboard
 from calibre import strftime, isbytestring
-from calibre.constants import filesystem_encoding, DEBUG
+from calibre.constants import filesystem_encoding, DEBUG, config_dir
 from calibre.gui2.library import DEFAULT_SORT
 from calibre.utils.localization import calibre_langcode_to_name
 from calibre.library.coloring import color_row_key
@@ -48,18 +48,20 @@ def default_image():
 
 class ColumnColor(object):
 
-    def __init__(self):
+    def __init__(self, formatter, colors):
         self.mi = None
+        self.formatter = formatter
+        self.colors = colors
 
-    def __call__(self, id_, key, fmt, db, formatter, color_cache, colors):
+    def __call__(self, id_, key, fmt, db, color_cache):
         if id_ in color_cache and key in color_cache[id_]:
             self.mi = None
             return color_cache[id_][key]
         try:
             if self.mi is None:
                 self.mi = db.get_metadata(id_, index_is_id=True)
-            color = formatter.safe_format(fmt, self.mi, '', self.mi)
+            color = self.formatter.safe_format(fmt, self.mi, '', self.mi)
-            if color in colors:
+            if color in self.colors:
                 color = QColor(color)
                 if color.isValid():
                     color = QVariant(color)
@@ -70,6 +72,36 @@ class ColumnColor(object):
             pass
 
 
+class ColumnIcon(object):
+
+    def __init__(self, formatter):
+        self.mi = None
+        self.formatter = formatter
+
+    def __call__(self, id_, key, fmt, kind, db, icon_cache, icon_bitmap_cache):
+        dex = key+kind
+        if id_ in icon_cache and dex in icon_cache[id_]:
+            self.mi = None
+            return icon_cache[id_][dex]
+        try:
+            if self.mi is None:
+                self.mi = db.get_metadata(id_, index_is_id=True)
+            icon = self.formatter.safe_format(fmt, self.mi, '', self.mi)
+            if icon:
+                if icon in icon_bitmap_cache:
+                    icon_bitmap = icon_bitmap_cache[icon]
+                    icon_cache[id_][dex] = icon_bitmap
+                    return icon_bitmap
+                d = os.path.join(config_dir, 'cc_icons', icon)
+                if (os.path.exists(d)):
+                    icon_bitmap = QIcon(d)
+                    icon_cache[id_][dex] = icon_bitmap
+                    icon_bitmap_cache[icon] = icon_bitmap
+                    self.mi = None
+                    return icon
+        except:
+            pass
+
 class BooksModel(QAbstractTableModel): # {{{
 
     about_to_be_sorted   = pyqtSignal(object, name='aboutToBeSorted')
@@ -97,7 +129,13 @@ class BooksModel(QAbstractTableModel): # {{{
     def __init__(self, parent=None, buffer=40):
         QAbstractTableModel.__init__(self, parent)
         self.db = None
-        self.column_color = ColumnColor()
+
+        self.formatter = SafeFormat()
+        self.colors = frozenset([unicode(c) for c in QColor.colorNames()])
+        self._clear_caches()
+        self.column_color = ColumnColor(self.formatter, self.colors)
+        self.column_icon = ColumnIcon(self.formatter)
+
         self.book_on_device = None
         self.editable_cols = ['title', 'authors', 'rating', 'publisher',
                               'tags', 'series', 'timestamp', 'pubdate',
@@ -109,8 +147,6 @@ class BooksModel(QAbstractTableModel): # {{{
         self.column_map = []
         self.headers = {}
         self.alignment_map = {}
-        self.color_cache = defaultdict(dict)
-        self.color_row_fmt_cache = None
         self.buffer_size = buffer
         self.metadata_backup = None
         self.bool_yes_icon = QIcon(I('ok.png'))
@@ -121,10 +157,14 @@ class BooksModel(QAbstractTableModel): # {{{
         self.ids_to_highlight_set = set()
         self.current_highlighted_idx = None
         self.highlight_only = False
-        self.colors = frozenset([unicode(c) for c in QColor.colorNames()])
-        self.formatter = SafeFormat()
         self.read_config()
 
+    def _clear_caches(self):
+        self.color_cache = defaultdict(dict)
+        self.icon_cache = defaultdict(dict)
+        self.icon_bitmap_cache = {}
+        self.color_row_fmt_cache = None
+
     def change_alignment(self, colname, alignment):
         if colname in self.column_map and alignment in ('left', 'right', 'center'):
             old = self.alignment_map.get(colname, 'left')
@@ -195,15 +235,13 @@ class BooksModel(QAbstractTableModel): # {{{
 
 
     def refresh_ids(self, ids, current_row=-1):
-        self.color_cache = defaultdict(dict)
+        self._clear_caches()
-        self.color_row_fmt_cache = None
         rows = self.db.refresh_ids(ids)
         if rows:
             self.refresh_rows(rows, current_row=current_row)
 
     def refresh_rows(self, rows, current_row=-1):
-        self.color_cache = defaultdict(dict)
+        self._clear_caches()
-        self.color_row_fmt_cache = None
         for row in rows:
             if row == current_row:
                 self.new_bookdisplay_data.emit(
@@ -234,8 +272,7 @@ class BooksModel(QAbstractTableModel): # {{{
         return ret
 
     def count_changed(self, *args):
-        self.color_cache = defaultdict(dict)
+        self._clear_caches()
-        self.color_row_fmt_cache = None
         self.count_changed_signal.emit(self.db.count())
 
     def row_indices(self, index):
@@ -366,8 +403,7 @@ class BooksModel(QAbstractTableModel): # {{{
         self.resort(reset=reset)
 
     def reset(self):
-        self.color_cache = defaultdict(dict)
+        self._clear_caches()
-        self.color_row_fmt_cache = None
         QAbstractTableModel.reset(self)
 
     def resort(self, reset=True):
@@ -750,7 +786,23 @@ class BooksModel(QAbstractTableModel): # {{{
         # we will get asked to display columns we don't know about. Must test for this.
         if col >= len(self.column_to_dc_map):
             return NONE
-        if role in (Qt.DisplayRole, Qt.EditRole, Qt.ToolTipRole):
+        if role == Qt.DisplayRole:
+            rules = self.db.prefs['column_icon_rules']
+            if rules:
+                key = self.column_map[col]
+                id_ = None
+                for kind, k, fmt in rules:
+                    if k == key and kind == 'icon_only':
+                        if id_ is None:
+                            id_ = self.id(index)
+                            self.column_icon.mi = None
+                        ccicon = self.column_icon(id_, key, fmt, 'icon_only', self.db,
+                                              self.icon_cache, self.icon_bitmap_cache)
+                        if ccicon is not None:
+                            return NONE
+                self.icon_cache[id_][key+'icon_only'] = None
+            return self.column_to_dc_map[col](index.row())
+        elif role in (Qt.EditRole, Qt.ToolTipRole):
             return self.column_to_dc_map[col](index.row())
         elif role == Qt.BackgroundRole:
             if self.id(index) in self.ids_to_highlight_set:
@@ -767,7 +819,7 @@ class BooksModel(QAbstractTableModel): # {{{
             for k, fmt in self.db.prefs['column_color_rules']:
                 if k == key:
                     ccol = self.column_color(id_, key, fmt, self.db,
-                                self.formatter, self.color_cache, self.colors)
+                                             self.color_cache)
                     if ccol is not None:
                         return ccol
 
@@ -788,7 +840,7 @@ class BooksModel(QAbstractTableModel): # {{{
 
             for fmt in self.color_row_fmt_cache:
                 ccol = self.column_color(id_, color_row_key, fmt, self.db,
-                            self.formatter, self.color_cache, self.colors)
+                                         self.color_cache)
                 if ccol is not None:
                     return ccol
 
@@ -796,7 +848,30 @@ class BooksModel(QAbstractTableModel): # {{{
             return NONE
         elif role == Qt.DecorationRole:
             if self.column_to_dc_decorator_map[col] is not None:
-                return self.column_to_dc_decorator_map[index.column()](index.row())
+                ccicon = self.column_to_dc_decorator_map[index.column()](index.row())
+                if ccicon != NONE:
+                    return ccicon
+
+            rules = self.db.prefs['column_icon_rules']
+            if rules:
+                key = self.column_map[col]
+                id_ = None
+                need_icon_with_text = False
+                for kind, k, fmt in rules:
+                    if k == key and kind in ('icon', 'icon_only'):
+                        if id_ is None:
+                            id_ = self.id(index)
+                            self.column_icon.mi = None
+                        if kind == 'icon':
+                            need_icon_with_text = True
+                        ccicon = self.column_icon(id_, key, fmt, kind, self.db,
+                                          self.icon_cache, self.icon_bitmap_cache)
+                        if ccicon is not None:
+                            return ccicon
+                if need_icon_with_text:
+                    self.icon_cache[id_][key+'icon'] = self.bool_blank_icon
+                    return self.bool_blank_icon
+                self.icon_cache[id_][key+'icon'] = None
         elif role == Qt.TextAlignmentRole:
             cname = self.column_map[index.column()]
             ans = Qt.AlignVCenter | ALIGNMENT_MAP[self.alignment_map.get(cname,

src/calibre/gui2/metadata/basic_widgets.py

@@ -819,6 +819,27 @@ class FormatsManager(QWidget):
     def show_format(self, item, *args):
         self.dialog.do_view_format(item.path, item.ext)
 
+    def get_selected_format(self):
+        row = self.formats.currentRow()
+        fmt = self.formats.item(row)
+        if fmt is None:
+            if self.formats.count() == 1:
+                fmt = self.formats.item(0)
+            if fmt is None:
+                error_dialog(self, _('No format selected'),
+                    _('No format selected')).exec_()
+                return None
+        return fmt.ext.lower()
+
+    def get_format_path(self, db, id_, fmt):
+        for i in xrange(self.formats.count()):
+            f = self.formats.item(i)
+            ext = f.ext.lower()
+            if ext == fmt:
+                if f.path is None:
+                    return db.format(id_, ext, as_path=True, index_is_id=True)
+                return f.path
+
     def get_selected_format_metadata(self, db, id_):
         old = prefs['read_file_metadata']
         if not old:

src/calibre/gui2/metadata/pdf_covers.py

@@ -0,0 +1,110 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2013, Kovid Goyal <kovid at kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+import sys, shutil, os
+from threading import Thread
+from glob import glob
+
+import sip
+from PyQt4.Qt import (QDialog, QApplication, QLabel, QGridLayout,
+                      QDialogButtonBox, Qt, pyqtSignal, QListWidget,
+                      QListWidgetItem, QSize, QIcon)
+
+from calibre import as_unicode
+from calibre.ebooks.metadata.pdf import page_images
+from calibre.gui2 import error_dialog, file_icon_provider
+from calibre.ptempfile import PersistentTemporaryDirectory
+
+class PDFCovers(QDialog):
+    'Choose a cover from the first few pages of a PDF'
+
+    rendering_done = pyqtSignal()
+
+    def __init__(self, pdfpath, parent=None):
+        QDialog.__init__(self, parent)
+        self.pdfpath = pdfpath
+        self.l = l = QGridLayout()
+        self.setLayout(l)
+
+        self.la = la = QLabel(_('Choose a cover from the list of PDF pages below'))
+        l.addWidget(la)
+        self.loading = la = QLabel('<b>'+_('Rendering PDF pages, please wait...'))
+        l.addWidget(la)
+
+        self.covers = c = QListWidget(self)
+        l.addWidget(c)
+        c.setIconSize(QSize(120, 160))
+        c.setSelectionMode(c.SingleSelection)
+        c.setViewMode(c.IconMode)
+        c.setUniformItemSizes(True)
+        c.setResizeMode(c.Adjust)
+        c.itemDoubleClicked.connect(self.accept)
+
+        self.bb = bb = QDialogButtonBox(QDialogButtonBox.Ok|QDialogButtonBox.Cancel)
+        bb.accepted.connect(self.accept)
+        bb.rejected.connect(self.reject)
+        l.addWidget(bb)
+        self.rendering_done.connect(self.show_pages, type=Qt.QueuedConnection)
+        self.tdir = PersistentTemporaryDirectory('_pdf_covers')
+        self.thread = Thread(target=self.render)
+        self.thread.daemon = True
+        self.thread.start()
+        self.setWindowTitle(_('Choose cover from PDF'))
+        self.setWindowIcon(file_icon_provider().icon_from_ext('pdf'))
+        self.resize(QSize(800, 600))
+
+    @property
+    def cover_path(self):
+        for item in self.covers.selectedItems():
+            return unicode(item.data(Qt.UserRole).toString())
+        if self.covers.count() > 0:
+            return unicode(self.covers.item(0).data(Qt.UserRole).toString())
+
+    def cleanup(self):
+        try:
+            shutil.rmtree(self.tdir)
+        except:
+            pass
+
+    def render(self):
+        self.error = None
+        try:
+            page_images(self.pdfpath, self.tdir, last=10)
+        except Exception as e:
+            self.error = as_unicode(e)
+        if not sip.isdeleted(self) and self.isVisible():
+            self.rendering_done.emit()
+
+    def show_pages(self):
+        self.loading.setVisible(False)
+        if self.error is not None:
+            error_dialog(self, _('Failed to render'),
+                _('Could not render this PDF file'), show=True)
+            self.reject()
+            return
+        files = (glob(os.path.join(self.tdir, '*.jpg')) +
+                 glob(os.path.join(self.tdir, '*.jpeg')))
+        if not files:
+            error_dialog(self, _('Failed to render'),
+                _('This PDF has no pages'), show=True)
+            self.reject()
+            return
+
+        for f in sorted(files):
+            i = QListWidgetItem(QIcon(f), '')
+            i.setData(Qt.UserRole, f)
+            self.covers.addItem(i)
+
+if __name__ == '__main__':
+    app = QApplication([])
+    app
+    d = PDFCovers(sys.argv[-1])
+    d.exec_()
+    print (d.cover_path)
+

src/calibre/gui2/metadata/single.py

@@ -318,7 +318,23 @@ class MetadataSingleDialogBase(ResizableDialog):
         if mi is not None:
             self.update_from_mi(mi)
 
+    def get_pdf_cover(self):
+        pdfpath = self.formats_manager.get_format_path(self.db, self.book_id,
+                                                       'pdf')
+        from calibre.gui2.metadata.pdf_covers import PDFCovers
+        d = PDFCovers(pdfpath, parent=self)
+        if d.exec_() == d.Accepted:
+            cpath = d.cover_path
+            if cpath:
+                with open(cpath, 'rb') as f:
+                    self.update_cover(f.read(), 'PDF')
+        d.cleanup()
+
     def cover_from_format(self, *args):
+        ext = self.formats_manager.get_selected_format()
+        if ext is None: return
+        if ext == 'pdf':
+            return self.get_pdf_cover()
         try:
             mi, ext = self.formats_manager.get_selected_format_metadata(self.db,
                     self.book_id)
@@ -343,12 +359,15 @@ class MetadataSingleDialogBase(ResizableDialog):
             error_dialog(self, _('Could not read cover'),
                          _('Could not read cover from %s format')%ext).exec_()
             return
+        self.update_cover(cdata, ext)
+
+    def update_cover(self, cdata, fmt):
         orig = self.cover.current_val
         self.cover.current_val = cdata
         if self.cover.current_val is None:
             self.cover.current_val = orig
             return error_dialog(self, _('Could not read cover'),
-                         _('The cover in the %s format is invalid')%ext,
+                         _('The cover in the %s format is invalid')%fmt,
                          show=True)
             return
 

src/calibre/gui2/pictureflow/pictureflow.cpp

@@ -340,6 +340,9 @@ public:
   int currentSlide() const;
   void setCurrentSlide(int index);
 
+  bool showReflections() const;
+  void setShowReflections(bool show);
+
   int getTarget() const;
 
   void showPrevious();
@@ -378,6 +381,7 @@ private:
   int slideHeight;
   int fontSize;
   int queueLength;
+  bool doReflections;
 
   int centerIndex;
   SlideInfo centerSlide;
@@ -416,6 +420,7 @@ PictureFlowPrivate::PictureFlowPrivate(PictureFlow* w, int queueLength_)
   slideWidth = 200;
   slideHeight = 200;
   fontSize = 10;
+  doReflections = true;
 
   centerIndex = 0;
   queueLength = queueLength_;
@@ -494,6 +499,15 @@ void PictureFlowPrivate::setCurrentSlide(int index)
   widget->emitcurrentChanged(centerIndex);
 }
 
+bool PictureFlowPrivate::showReflections() const {
+    return doReflections;
+}
+
+void PictureFlowPrivate::setShowReflections(bool show) {
+    doReflections = show;
+    triggerRender();
+}
+
 void PictureFlowPrivate::showPrevious()
 {
   if(step >= 0)
@@ -584,7 +598,7 @@ void PictureFlowPrivate::resetSlides()
   }
 }
 
-static QImage prepareSurface(QImage img, int w, int h)
+static QImage prepareSurface(QImage img, int w, int h, bool doReflections)
 {
   img = img.scaled(w, h, Qt::IgnoreAspectRatio, Qt::SmoothTransformation);
 
@@ -602,19 +616,21 @@ static QImage prepareSurface(QImage img, int w, int h)
     for(int y = 0; y < h; y++)
       result.setPixel(y, x, img.pixel(x, y));
 
-  // create the reflection
+  if (doReflections) {
-  int ht = hs - h;
+    // create the reflection
-  for(int x = 0; x < w; x++)
+    int ht = hs - h;
-    for(int y = 0; y < ht; y++)
+    for(int x = 0; x < w; x++)
-    {
+        for(int y = 0; y < ht; y++)
-      QRgb color = img.pixel(x, img.height()-y-1);
+        {
-      //QRgb565 color = img.scanLine(img.height()-y-1) + x*sizeof(QRgb565); //img.pixel(x, img.height()-y-1);
+        QRgb color = img.pixel(x, img.height()-y-1);
-      int a = qAlpha(color);
+        //QRgb565 color = img.scanLine(img.height()-y-1) + x*sizeof(QRgb565); //img.pixel(x, img.height()-y-1);
-      int r = qRed(color)   * a / 256 * (ht - y) / ht * 3/5;
+        int a = qAlpha(color);
-      int g = qGreen(color) * a / 256 * (ht - y) / ht * 3/5;
+        int r = qRed(color)   * a / 256 * (ht - y) / ht * 3/5;
-      int b = qBlue(color)  * a / 256 * (ht - y) / ht * 3/5;
+        int g = qGreen(color) * a / 256 * (ht - y) / ht * 3/5;
-      result.setPixel(h+y, x, qRgb(r, g, b));
+        int b = qBlue(color)  * a / 256 * (ht - y) / ht * 3/5;
-    }
+        result.setPixel(h+y, x, qRgb(r, g, b));
+        }
+  }
 
   return result;
 }
@@ -652,12 +668,12 @@ QImage* PictureFlowPrivate::surface(int slideIndex)
       painter.setBrush(QBrush());
       painter.drawRect(2, 2, slideWidth-3, slideHeight-3);
       painter.end();
-      blankSurface = prepareSurface(blankSurface, slideWidth, slideHeight);
+      blankSurface = prepareSurface(blankSurface, slideWidth, slideHeight, doReflections);
     }
     return &blankSurface;
   }
 
-  surfaceCache.insert(slideIndex, new QImage(prepareSurface(img, slideWidth, slideHeight)));
+  surfaceCache.insert(slideIndex, new QImage(prepareSurface(img, slideWidth, slideHeight, doReflections)));
   return surfaceCache[slideIndex];
 }
 
@@ -1196,6 +1212,13 @@ QImage PictureFlow::slide(int index) const
   return d->slide(index);
 }
 
+bool PictureFlow::showReflections() const {
+    return d->showReflections();
+}
+
+void PictureFlow::setShowReflections(bool show) {
+    d->setShowReflections(show);
+}
 
 void PictureFlow::setImages(FlowImages *images) 
 {

src/calibre/gui2/pictureflow/pictureflow.h

@@ -121,6 +121,12 @@ public:
   */  
   void setSlideSize(QSize size);
 
+  /*!
+    Turn the reflections on/off.
+  */  
+  void setShowReflections(bool show);
+  bool showReflections() const;
+
   /*!
     Returns the font used to render subtitles
   */  

src/calibre/gui2/pictureflow/pictureflow.sip

@@ -51,6 +51,10 @@ public :
 
   int currentSlide() const;
 
+  bool showReflections() const;
+
+  void setShowReflections(bool show);
+
 public slots:
 
   void setCurrentSlide(int index);

src/calibre/gui2/preferences/coloring.py

@@ -7,15 +7,18 @@ __license__   = 'GPL v3'
 __copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
+import os
+
 from PyQt4.Qt import (QWidget, QDialog, QLabel, QGridLayout, QComboBox, QSize,
         QLineEdit, QIntValidator, QDoubleValidator, QFrame, QColor, Qt, QIcon,
         QScrollArea, QPushButton, QVBoxLayout, QDialogButtonBox, QToolButton,
         QListView, QAbstractListModel, pyqtSignal, QSizePolicy, QSpacerItem,
         QApplication)
 
-from calibre import prepare_string_for_xml
+from calibre import prepare_string_for_xml, sanitize_file_name_unicode
+from calibre.constants import config_dir
 from calibre.utils.icu import sort_key
-from calibre.gui2 import error_dialog
+from calibre.gui2 import error_dialog, choose_files, pixmap_to_data
 from calibre.gui2.dialogs.template_dialog import TemplateDialog
 from calibre.gui2.metadata.single_download import RichTextDelegate
 from calibre.library.coloring import (Rule, conditionable_columns,
@@ -25,6 +28,9 @@ from calibre.utils.icu import lower
 
 all_columns_string = _('All Columns')
 
+icon_rule_kinds = [(_('icon with text'), 'icon'),
+                   (_('icon with no text'), 'icon_only') ]
+
 class ConditionEditor(QWidget): # {{{
 
     ACTION_MAP = {
@@ -207,8 +213,6 @@ class ConditionEditor(QWidget): # {{{
         col = self.current_col
         if not col:
             return
-        m = self.fm[col]
-        dt = m['datatype']
         action = self.current_action
         if not action:
             return
@@ -245,64 +249,103 @@ class ConditionEditor(QWidget): # {{{
 
 class RuleEditor(QDialog): # {{{
 
-    def __init__(self, fm, parent=None):
+    def __init__(self, fm, pref_name, parent=None):
         QDialog.__init__(self, parent)
         self.fm = fm
 
+        if pref_name == 'column_color_rules':
+            self.rule_kind = 'color'
+            rule_text = _('coloring')
+        else:
+            self.rule_kind = 'icon'
+            rule_text = _('icon')
+
         self.setWindowIcon(QIcon(I('format-fill-color.png')))
-        self.setWindowTitle(_('Create/edit a column coloring rule'))
+        self.setWindowTitle(_('Create/edit a column {0} rule').format(rule_text))
 
         self.l = l = QGridLayout(self)
         self.setLayout(l)
 
-        self.l1 = l1 = QLabel(_('Create a coloring rule by'
+        self.l1 = l1 = QLabel(_('Create a column {0} rule by'
-            ' filling in the boxes below'))
+            ' filling in the boxes below'.format(rule_text)))
-        l.addWidget(l1, 0, 0, 1, 5)
+        l.addWidget(l1, 0, 0, 1, 8)
 
         self.f1 = QFrame(self)
         self.f1.setFrameShape(QFrame.HLine)
-        l.addWidget(self.f1, 1, 0, 1, 5)
+        l.addWidget(self.f1, 1, 0, 1, 8)
 
-        self.l2 = l2 = QLabel(_('Set the color of the column:'))
+        self.l2 = l2 = QLabel(_('Set the'))
         l.addWidget(l2, 2, 0)
 
-        self.column_box = QComboBox(self)
+        if self.rule_kind == 'color':
-        l.addWidget(self.column_box, 2, 1)
+            l.addWidget(QLabel(_('color')))
+        else:
+            self.kind_box = QComboBox(self)
+            for tt, t in icon_rule_kinds:
+                self.kind_box.addItem(tt, t)
+            l.addWidget(self.kind_box, 2, 1)
 
-        self.l3 = l3 = QLabel(_('to'))
+        self.l3 = l3 = QLabel(_('of the column:'))
         l.addWidget(l3, 2, 2)
 
-        self.color_box = QComboBox(self)
+        self.column_box = QComboBox(self)
-        self.color_label = QLabel('Sample text Sample text')
+        l.addWidget(self.column_box, 2, 3)
-        self.color_label.setTextFormat(Qt.RichText)
-        l.addWidget(self.color_box, 2, 3)
-        l.addWidget(self.color_label, 2, 4)
-        l.addItem(QSpacerItem(10, 10, QSizePolicy.Expanding), 2, 5)
 
-        self.l4 = l4 = QLabel(
+        self.l4 = l4 = QLabel(_('to'))
+        l.addWidget(l4, 2, 4)
+
+        if self.rule_kind == 'color':
+            self.color_box = QComboBox(self)
+            self.color_label = QLabel('Sample text Sample text')
+            self.color_label.setTextFormat(Qt.RichText)
+            l.addWidget(self.color_box, 2, 5)
+            l.addWidget(self.color_label, 2, 6)
+            l.addItem(QSpacerItem(10, 10, QSizePolicy.Expanding), 2, 7)
+        else:
+            self.filename_box = QComboBox()
+            self.filename_box.setInsertPolicy(self.filename_box.InsertAlphabetically)
+            d = os.path.join(config_dir, 'cc_icons')
+            self.icon_file_names = []
+            if os.path.exists(d):
+                for icon_file in os.listdir(d):
+                    icon_file = lower(icon_file)
+                    if os.path.exists(os.path.join(d, icon_file)):
+                        if icon_file.endswith('.png'):
+                            self.icon_file_names.append(icon_file)
+            self.icon_file_names.sort(key=sort_key)
+            self.update_filename_box()
+
+            l.addWidget(self.filename_box, 2, 5)
+            self.filename_button = QPushButton(QIcon(I('document_open.png')),
+                                               _('&Add icon'))
+            l.addWidget(self.filename_button, 2, 6)
+            l.addWidget(QLabel(_('Icons should be square or landscape')), 2, 7)
+            l.setColumnStretch(7, 10)
+
+        self.l5 = l5 = QLabel(
             _('Only if the following conditions are all satisfied:'))
-        l.addWidget(l4, 3, 0, 1, 6)
+        l.addWidget(l5, 3, 0, 1, 7)
 
         self.scroll_area = sa = QScrollArea(self)
         sa.setMinimumHeight(300)
         sa.setMinimumWidth(950)
         sa.setWidgetResizable(True)
-        l.addWidget(sa, 4, 0, 1, 6)
+        l.addWidget(sa, 4, 0, 1, 8)
 
         self.add_button = b = QPushButton(QIcon(I('plus.png')),
                 _('Add another condition'))
-        l.addWidget(b, 5, 0, 1, 6)
+        l.addWidget(b, 5, 0, 1, 8)
         b.clicked.connect(self.add_blank_condition)
 
-        self.l5 = l5 = QLabel(_('You can disable a condition by'
+        self.l6 = l6 = QLabel(_('You can disable a condition by'
             ' blanking all of its boxes'))
-        l.addWidget(l5, 6, 0, 1, 6)
+        l.addWidget(l6, 6, 0, 1, 8)
 
         self.bb = bb = QDialogButtonBox(
                 QDialogButtonBox.Ok|QDialogButtonBox.Cancel)
         bb.accepted.connect(self.accept)
         bb.rejected.connect(self.reject)
-        l.addWidget(bb, 7, 0, 1, 6)
+        l.addWidget(bb, 7, 0, 1, 8)
 
         self.conditions_widget = QWidget(self)
         sa.setWidget(self.conditions_widget)
@@ -310,24 +353,39 @@ class RuleEditor(QDialog): # {{{
         self.conditions_widget.layout().setAlignment(Qt.AlignTop)
         self.conditions = []
 
-        for b in (self.column_box, self.color_box):
+        if self.rule_kind == 'color':
-            b.setSizeAdjustPolicy(b.AdjustToMinimumContentsLengthWithIcon)
+            for b in (self.column_box, self.color_box):
-            b.setMinimumContentsLength(15)
+                b.setSizeAdjustPolicy(b.AdjustToMinimumContentsLengthWithIcon)
+                b.setMinimumContentsLength(15)
 
         for key in sorted(displayable_columns(fm),
                           key=lambda(k): sort_key(fm[k]['name']) if k != color_row_key else 0):
+            if key == color_row_key and self.rule_kind != 'color':
+                continue
             name = all_columns_string if key == color_row_key else fm[key]['name']
             if name:
                 self.column_box.addItem(name, key)
         self.column_box.setCurrentIndex(0)
 
-        self.color_box.addItems(QColor.colorNames())
+        if self.rule_kind == 'color':
-        self.color_box.setCurrentIndex(0)
+            self.color_box.addItems(QColor.colorNames())
+            self.color_box.setCurrentIndex(0)
+            self.update_color_label()
+            self.color_box.currentIndexChanged.connect(self.update_color_label)
+        else:
+            self.filename_button.clicked.connect(self.filename_button_clicked)
 
-        self.update_color_label()
-        self.color_box.currentIndexChanged.connect(self.update_color_label)
         self.resize(self.sizeHint())
 
+    def update_filename_box(self):
+        self.filename_box.clear()
+        self.icon_file_names.sort(key=sort_key)
+        self.filename_box.addItem('')
+        self.filename_box.addItems(self.icon_file_names)
+        for i,filename in enumerate(self.icon_file_names):
+            icon = QIcon(os.path.join(config_dir, 'cc_icons', filename))
+            self.filename_box.setItemIcon(i+1, icon)
+
     def update_color_label(self):
         pal = QApplication.palette()
         bg1 = unicode(pal.color(pal.Base).name())
@@ -338,22 +396,64 @@ class RuleEditor(QDialog): # {{{
             <span style="color: {c}; background-color: {bg2}">&nbsp;{st}&nbsp;</span>
             '''.format(c=c, bg1=bg1, bg2=bg2, st=_('Sample Text')))
 
+    def filename_button_clicked(self):
+        try:
+            path = choose_files(self, 'choose_category_icon',
+                        _('Select Icon'), filters=[
+                        ('Images', ['png', 'gif', 'jpg', 'jpeg'])],
+                    all_files=False, select_only_single_file=True)
+            if path:
+                icon_path = path[0]
+                icon_name = sanitize_file_name_unicode(
+                             os.path.splitext(
+                                   os.path.basename(icon_path))[0]+'.png')
+                if icon_name not in self.icon_file_names:
+                    self.icon_file_names.append(icon_name)
+                    self.update_filename_box()
+                    try:
+                        p = QIcon(icon_path).pixmap(QSize(128, 128))
+                        d = os.path.join(config_dir, 'cc_icons')
+                        if not os.path.exists(os.path.join(d, icon_name)):
+                            if not os.path.exists(d):
+                                os.makedirs(d)
+                            with open(os.path.join(d, icon_name), 'wb') as f:
+                                f.write(pixmap_to_data(p, format='PNG'))
+                    except:
+                        import traceback
+                        traceback.print_exc()
+                self.filename_box.setCurrentIndex(self.filename_box.findText(icon_name))
+                self.filename_box.adjustSize()
+        except:
+            import traceback
+            traceback.print_exc()
+        return
 
     def add_blank_condition(self):
         c = ConditionEditor(self.fm, parent=self.conditions_widget)
         self.conditions.append(c)
         self.conditions_widget.layout().addWidget(c)
 
-    def apply_rule(self, col, rule):
+    def apply_rule(self, kind, col, rule):
+        if kind == 'color':
+            if rule.color:
+                idx = self.color_box.findText(rule.color)
+                if idx >= 0:
+                    self.color_box.setCurrentIndex(idx)
+        else:
+            self.kind_box.setCurrentIndex(0 if kind == 'icon' else 1)
+            if rule.color:
+                idx = self.filename_box.findText(rule.color)
+                if idx >= 0:
+                    self.filename_box.setCurrentIndex(idx)
+                else:
+                    self.filename_box.setCurrentIndex(0)
+
         for i in range(self.column_box.count()):
             c = unicode(self.column_box.itemData(i).toString())
             if col == c:
                 self.column_box.setCurrentIndex(i)
                 break
-        if rule.color:
+
-            idx = self.color_box.findText(rule.color)
-            if idx >= 0:
-                self.color_box.setCurrentIndex(idx)
         for c in rule.conditions:
             ce = ConditionEditor(self.fm, parent=self.conditions_widget)
             self.conditions.append(ce)
@@ -366,6 +466,12 @@ class RuleEditor(QDialog): # {{{
 
 
     def accept(self):
+        if self.rule_kind != 'color':
+            fname = lower(unicode(self.filename_box.currentText()))
+            if not fname:
+                error_dialog(self, _('No icon selected'),
+                        _('You must choose an icon for this rule'), show=True)
+                return
         if self.validate():
             QDialog.accept(self)
 
@@ -393,32 +499,54 @@ class RuleEditor(QDialog): # {{{
     @property
     def rule(self):
         r = Rule(self.fm)
-        r.color = unicode(self.color_box.currentText())
+        if self.rule_kind != 'color':
+            r.color = unicode(self.filename_box.currentText())
+        else:
+            r.color = unicode(self.color_box.currentText())
         idx = self.column_box.currentIndex()
         col = unicode(self.column_box.itemData(idx).toString())
         for c in self.conditions:
             condition = c.condition
             if condition is not None:
                 r.add_condition(*condition)
+        if self.rule_kind == 'icon':
+            kind = unicode(self.kind_box.itemData(
+                                    self.kind_box.currentIndex()).toString())
+        else:
+            kind = 'color'
 
-        return col, r
+        return kind, col, r
 # }}}
 
 class RulesModel(QAbstractListModel): # {{{
 
-    def __init__(self, prefs, fm, parent=None):
+    def __init__(self, prefs, fm, pref_name, parent=None):
         QAbstractListModel.__init__(self, parent)
 
         self.fm = fm
-        rules = list(prefs['column_color_rules'])
+        self.pref_name = pref_name
-        self.rules = []
+        if pref_name == 'column_color_rules':
-        for col, template in rules:
+            self.rule_kind = 'color'
-            if col not in self.fm: continue
+            rules = list(prefs[pref_name])
-            try:
+            self.rules = []
-                rule = rule_from_template(self.fm, template)
+            for col, template in rules:
-            except:
+                if col not in self.fm and col != color_row_key: continue
-                rule = template
+                try:
-            self.rules.append((col, rule))
+                    rule = rule_from_template(self.fm, template)
+                except:
+                    rule = template
+                self.rules.append(('color', col, rule))
+        else:
+            self.rule_kind = 'icon'
+            rules = list(prefs[pref_name])
+            self.rules = []
+            for kind, col, template in rules:
+                if col not in self.fm and col != color_row_key: continue
+                try:
+                    rule = rule_from_template(self.fm, template)
+                except:
+                    rule = template
+                self.rules.append((kind, col, rule))
 
     def rowCount(self, *args):
         return len(self.rules)
@@ -426,7 +554,7 @@ class RulesModel(QAbstractListModel): # {{{
     def data(self, index, role):
         row = index.row()
         try:
-            col, rule = self.rules[row]
+            kind, col, rule = self.rules[row]
         except:
             return None
         if role == Qt.DisplayRole:
@@ -434,17 +562,17 @@ class RulesModel(QAbstractListModel): # {{{
                 col = all_columns_string
             else:
                 col = self.fm[col]['name']
-            return self.rule_to_html(col, rule)
+            return self.rule_to_html(kind, col, rule)
         if role == Qt.UserRole:
-            return (col, rule)
+            return (kind, col, rule)
 
-    def add_rule(self, col, rule):
+    def add_rule(self, kind, col, rule):
-        self.rules.append((col, rule))
+        self.rules.append((kind, col, rule))
         self.reset()
         return self.index(len(self.rules)-1)
 
-    def replace_rule(self, index, col, r):
+    def replace_rule(self, index, kind, col, r):
-        self.rules[index.row()] = (col, r)
+        self.rules[index.row()] = (kind, col, r)
         self.dataChanged.emit(index, index)
 
     def remove_rule(self, index):
@@ -453,12 +581,15 @@ class RulesModel(QAbstractListModel): # {{{
 
     def commit(self, prefs):
         rules = []
-        for col, r in self.rules:
+        for kind, col, r in self.rules:
             if isinstance(r, Rule):
                 r = r.template
             if r is not None:
-                rules.append((col, r))
+                if kind == 'color':
-        prefs['column_color_rules'] = rules
+                    rules.append((col, r))
+                else:
+                    rules.append((kind, col, r))
+        prefs[self.pref_name] = rules
 
     def move(self, idx, delta):
         row = idx.row() + delta
@@ -475,18 +606,28 @@ class RulesModel(QAbstractListModel): # {{{
         self.rules = []
         self.reset()
 
-    def rule_to_html(self, col, rule):
+    def rule_to_html(self, kind, col, rule):
         if not isinstance(rule, Rule):
             return _('''
             <p>Advanced Rule for column <b>%(col)s</b>:
             <pre>%(rule)s</pre>
             ''')%dict(col=col, rule=prepare_string_for_xml(rule))
         conditions = [self.condition_to_html(c) for c in rule.conditions]
+
+        trans_kind = 'not found'
+        if kind == 'color':
+            trans_kind = _('color')
+        else:
+            for tt, t in icon_rule_kinds:
+                if kind == t:
+                    trans_kind = tt
+                    break
+
         return _('''\
-            <p>Set the color of <b>%(col)s</b> to <b>%(color)s</b> if the following
+            <p>Set the <b>%(kind)s</b> of <b>%(col)s</b> to <b>%(color)s</b> if the following
             conditions are met:</p>
             <ul>%(rule)s</ul>
-            ''') % dict(col=col, color=rule.color, rule=''.join(conditions))
+            ''') % dict(kind=trans_kind, col=col, color=rule.color, rule=''.join(conditions))
 
     def condition_to_html(self, condition):
         c, a, v = condition
@@ -513,12 +654,7 @@ class EditRules(QWidget): # {{{
         self.l = l = QGridLayout(self)
         self.setLayout(l)
 
-        self.l1 = l1 = QLabel('<p>'+_(
+        self.l1 = l1 = QLabel('')
-            'You can control the color of columns in the'
-            ' book list by creating "rules" that tell calibre'
-            ' what color to use. Click the Add Rule button below'
-            ' to get started.<p>You can <b>change an existing rule</b> by double'
-            ' clicking it.'))
         l1.setWordWrap(True)
         l.addWidget(l1, 0, 0, 1, 2)
 
@@ -559,22 +695,38 @@ class EditRules(QWidget): # {{{
         b.clicked.connect(self.add_advanced)
         l.addWidget(b, 3, 0, 1, 2)
 
-    def initialize(self, fm, prefs, mi):
+    def initialize(self, fm, prefs, mi, pref_name):
-        self.model = RulesModel(prefs, fm)
+        self.pref_name = pref_name
+        self.model = RulesModel(prefs, fm, self.pref_name)
         self.rules_view.setModel(self.model)
         self.fm = fm
         self.mi = mi
+        if pref_name == 'column_color_rules':
+            self.l1.setText('<p>'+_(
+                'You can control the color of columns in the'
+                ' book list by creating "rules" that tell calibre'
+                ' what color to use. Click the Add Rule button below'
+                ' to get started.<p>You can <b>change an existing rule</b> by'
+                ' double clicking it.'))
+        else:
+            self.l1.setText('<p>'+_(
+                'You can add icons to columns in the'
+                ' book list by creating "rules" that tell calibre'
+                ' what icon to use. Click the Add Rule button below'
+                ' to get started.<p>You can <b>change an existing rule</b> by'
+                ' double clicking it.'))
+            self.add_advanced_button.setVisible(False)
 
     def _add_rule(self, dlg):
         if dlg.exec_() == dlg.Accepted:
-            col, r = dlg.rule
+            kind, col, r = dlg.rule
-            if r and col:
+            if kind and r and col:
-                idx = self.model.add_rule(col, r)
+                idx = self.model.add_rule(kind, col, r)
                 self.rules_view.scrollTo(idx)
                 self.changed.emit()
 
     def add_rule(self):
-        d = RuleEditor(self.model.fm)
+        d = RuleEditor(self.model.fm, self.pref_name)
         d.add_blank_condition()
         self._add_rule(d)
 
@@ -584,18 +736,18 @@ class EditRules(QWidget): # {{{
 
     def edit_rule(self, index):
         try:
-            col, rule = self.model.data(index, Qt.UserRole)
+            kind, col, rule = self.model.data(index, Qt.UserRole)
         except:
             return
         if isinstance(rule, Rule):
-            d = RuleEditor(self.model.fm)
+            d = RuleEditor(self.model.fm, self.pref_name)
-            d.apply_rule(col, rule)
+            d.apply_rule(kind, col, rule)
         else:
             d = TemplateDialog(self, rule, mi=self.mi, fm=self.fm, color_field=col)
         if d.exec_() == d.Accepted:
-            col, r = d.rule
+            kind, col, r = d.rule
-            if r is not None and col:
+            if kind and r is not None and col:
-                self.model.replace_rule(index, col, r)
+                self.model.replace_rule(index, kind, col, r)
                 self.rules_view.scrollTo(index)
                 self.changed.emit()
 
@@ -651,7 +803,7 @@ if __name__ == '__main__':
     db = db()
 
     if True:
-        d = RuleEditor(db.field_metadata)
+        d = RuleEditor(db.field_metadata, 'column_color_rules')
         d.add_blank_condition()
         d.exec_()
 

src/calibre/gui2/preferences/look_feel.py

@@ -110,6 +110,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         r('bd_overlay_cover_size', gprefs)
 
         r('cover_flow_queue_length', config, restart_required=True)
+        r('cover_browser_reflections', gprefs)
 
         def get_esc_lang(l):
             if l == 'en':
@@ -181,6 +182,12 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         self.edit_rules.changed.connect(self.changed_signal)
         self.tabWidget.addTab(self.edit_rules,
                 QIcon(I('format-fill-color.png')), _('Column coloring'))
+
+        self.icon_rules = EditRules(self.tabWidget)
+        self.icon_rules.changed.connect(self.changed_signal)
+        self.tabWidget.addTab(self.icon_rules,
+                QIcon(I('icon_choose.png')), _('Column icons'))
+
         self.tabWidget.setCurrentIndex(0)
         keys = [QKeySequence('F11', QKeySequence.PortableText), QKeySequence(
             'Ctrl+Shift+F', QKeySequence.PortableText)]
@@ -203,7 +210,8 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
             mi = db.get_metadata(idx, index_is_id=False)
         except:
             mi=None
-        self.edit_rules.initialize(db.field_metadata, db.prefs, mi)
+        self.edit_rules.initialize(db.field_metadata, db.prefs, mi, 'column_color_rules')
+        self.icon_rules.initialize(db.field_metadata, db.prefs, mi, 'column_icon_rules')
 
     def restore_defaults(self):
         ConfigWidgetBase.restore_defaults(self)
@@ -214,6 +222,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
             self.update_font_display()
         self.display_model.restore_defaults()
         self.edit_rules.clear()
+        self.icon_rules.clear()
         self.changed_signal.emit()
 
     def build_font_obj(self):
@@ -273,6 +282,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
             rr = True
         self.display_model.commit()
         self.edit_rules.commit(self.gui.current_db.prefs)
+        self.icon_rules.commit(self.gui.current_db.prefs)
         return rr
 
     def refresh_gui(self, gui):
@@ -280,6 +290,8 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         self.update_font_display()
         gui.tags_view.reread_collapse_parameters()
         gui.library_view.refresh_book_details()
+        if hasattr(gui.cover_flow, 'setShowReflections'):
+            gui.cover_flow.setShowReflections(gprefs['cover_browser_reflections'])
 
 if __name__ == '__main__':
     from calibre.gui2 import Application

src/calibre/gui2/preferences/look_feel.ui

@@ -496,10 +496,7 @@ a few top-level elements.</string>
          </property>
         </widget>
        </item>
-       <item row="1" column="1">
+       <item row="5" column="0" colspan="2">
-        <widget class="QSpinBox" name="opt_cover_flow_queue_length"/>
-       </item>
-       <item row="4" column="0" colspan="2">
         <spacer name="verticalSpacer_4">
          <property name="orientation">
           <enum>Qt::Vertical</enum>
@@ -512,14 +509,17 @@ a few top-level elements.</string>
          </property>
         </spacer>
        </item>
-       <item row="2" column="0" colspan="2">
+       <item row="1" column="1">
+        <widget class="QSpinBox" name="opt_cover_flow_queue_length"/>
+       </item>
+       <item row="3" column="0" colspan="2">
         <widget class="QCheckBox" name="opt_cb_fullscreen">
          <property name="text">
           <string>When showing cover browser in separate window, show it &amp;fullscreen</string>
          </property>
         </widget>
        </item>
-       <item row="3" column="0" colspan="2">
+       <item row="4" column="0" colspan="2">
         <widget class="QLabel" name="fs_help_msg">
          <property name="styleSheet">
           <string notr="true">margin-left: 1.5em</string>
@@ -532,6 +532,13 @@ a few top-level elements.</string>
          </property>
         </widget>
        </item>
+       <item row="2" column="0">
+        <widget class="QCheckBox" name="opt_cover_browser_reflections">
+         <property name="text">
+          <string>Show &amp;reflections in the cover browser</string>
+         </property>
+        </widget>
+       </item>
       </layout>
      </widget>
     </widget>

src/calibre/gui2/tag_browser/model.py

@@ -188,12 +188,14 @@ class TagTreeItem(object): # {{{
 
     def child_tags(self):
         res = []
-        def recurse(nodes, res):
+        def recurse(nodes, res, depth):
+            if depth > 100:
+                return
             for t in nodes:
                 if t.type != TagTreeItem.CATEGORY:
                     res.append(t)
-                recurse(t.children, res)
+                recurse(t.children, res, depth+1)
-        recurse(self.children, res)
+        recurse(self.children, res, 1)
         return res
     # }}}
 

src/calibre/library/database2.py

@@ -211,6 +211,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         defs['gui_restriction'] = defs['cs_restriction'] = ''
         defs['categories_using_hierarchy'] = []
         defs['column_color_rules'] = []
+        defs['column_icon_rules'] = []
         defs['grouped_search_make_user_categories'] = []
         defs['similar_authors_search_key'] = 'authors'
         defs['similar_authors_match_kind'] = 'match_any'
@@ -1881,7 +1882,6 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
             # icon_map is not None if get_categories is to store an icon and
             # possibly a tooltip in the tag structure.
             icon = None
-            tooltip = '(' + category + ')'
             label = tb_cats.key_to_label(category)
             if icon_map:
                 if not tb_cats.is_custom_field(category):
@@ -1932,10 +1932,11 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                 use_sort_as_name = True
             else:
                 use_sort_as_name = False
-            is_editable = category not in ['news', 'rating', 'languages']
+            is_editable = (category not in ['news', 'rating', 'languages'] and
+                                datatype != "composite")
             categories[category] = [tag_class(formatter(r.n), count=r.c, id=r.id,
                                         avg=avgr(r), sort=r.s, icon=icon,
-                                        tooltip=tooltip, category=category,
+                                        category=category,
                                         id_set=r.id_set, is_editable=is_editable,
                                         use_sort_as_name=use_sort_as_name)
                                     for r in items]