Merge from trunk

Changelog.yaml

@@ -19,6 +19,57 @@
 #   new recipes:
 #     - title: 
 
+- version: 0.8.59
+  date: 2012-07-06
+
+  new features:
+    - title: "Drivers for Samsung SGH-T989 and Sony Ericsson Sola"
+      tickets: [1021365]
+
+    - title: "Conversion pipeline: When removing the first image, also remove the html file the image is found in, if that file has no other content. Allows this option to be used to remove covers from EPUB files without leaving behind a blank page."
+
+    - title: "Content server: Add a navigation panel at the bottom of each page."
+      tickets: [1020225]
+
+    - title: "calibredb: Add a backup_metadata command to manually run the backup to opf from the command line"
+
+    - title: "User defined driver: Add option to swap main memory and card a."
+      tickets: [1020056]
+
+    - title: "Add new option to the series_index_auto_increment tweak, no_change, that causes calibre not to change the series_index when the series is changed"
+ 
+  bug fixes:
+    - title: "PDF Output: Resize large images so that they do not get off at the right edge of the page."
+
+    - title: "On linux ensure that WM_CLASS for the main calibre GUI is set to 'calibre-gui' to match the name of the calibre-gui.desktop file. This is apparently required by the GNOME 3 shell."
+      tickets: [1020297]
+
+    - title: "Update ICU in all builds to version 49.1"
+
+    - title: "Tag browser: Fix regression that broke drag and drop between user categories in the tag browser"
+
+    - title: "When copying to library and deleting after copy, do not place deleted files in recycle bin, as this is redundant and slow (they have already been copied into another library)"
+
+    - title: "Fix yes/no fields with value of No not showing up in the book details panel"
+
+    - title: "Catalogs: Better sorting for non English languages"
+      tickets: [930882] 
+
+    - title: "Get Books: Fix Foyles UK, Weightless books, ebooks.com and ozon.ru"
+
+    - title: "CHM Input: Fix handling of chm files that split their html into multiple sub-directories."
+      tickets: [1018792]
+
+  improved recipes:
+    - FHM UK
+    - The Age
+    - weblogs_ssl
+    - Heraldo.es
+
+  new recipes:
+    - title: CATO Institute and Heritage Foundation
+      author: _reader
+
 - version: 0.8.58
   date: 2012-06-29
 

manual/creating_plugins.rst

@@ -15,7 +15,7 @@ Here, we will teach you how to create your own plugins to add new features to |a
   :depth: 2
   :local:
 
-.. note:: This only applies to calibre releases >= 0.7.53
+.. note:: This only applies to calibre releases >= 0.8.60
 
 Anatomy of a |app| plugin
 ---------------------------
@@ -32,11 +32,15 @@ and enter the following Python code into it:
 .. literalinclude:: plugin_examples/helloworld/__init__.py
     :lines: 10-
 
-That's all. To add this code to |app| as a plugin, simply create a zip file with::
+That's all. To add this code to |app| as a plugin, simply run the following in
+the directory in which you created :file:`__init__.py`::
 
-    zip plugin.zip __init__.py
+    calibre-customize -b .
 
-Add this plugin to |app| via :guilabel:`Preferences->Plugins`.
+.. note:: 
+    On OS X you have to first install the |app| command line tools, by
+    going to :guilabel:`Preferences->Miscellaneous` and clicking the
+    :guilabel:`Install command line tools` button.
 
 You can download the Hello World plugin from 
 `helloworld_plugin.zip  <http://calibre-ebook.com/downloads/helloworld_plugin.zip>`_. 
@@ -191,14 +195,12 @@ When running from the command line, debug output will be printed to the console,
 
 You can insert print statements anywhere in your plugin code, they will be output in debug mode. Remember, this is python, you really shouldn't need anything more than print statements to debug ;) I developed all of |app| using just this debugging technique.
 
-It can get tiresome to keep re-adding a plugin to calibre to test small changes. The plugin zip files are stored in the calibre config directory in plugins/ (goto Preferences->Misc and click open config directory to see the config directory). 
+You can quickly test changes to your plugin by using the following command
+line::
 
-Once you've located the zip file of your plugin you can then directly update it with your changes instead of re-adding it each time. To do so from the command line, in the directory that contains your plugin source code, use::
+    calibre-debug -s; calibre-customize -b /path/to/your/plugin/directory; calibre
 
-    calibre -s; zip -r /path/to/plugin/zip/file.zip *; calibre
+This will shutdown a running calibre, wait for the shutdown to complete, then update your plugin in |app| and relaunch |app|.
-
-This will shutdown a running calibre. Wait for the shutdown to complete, then update your plugin files and relaunch calibre.
-It relies on the freely available zip command line tool.
 
 More plugin examples
 ----------------------

recipes/adventuregamers.recipe

@@ -1,5 +1,5 @@
 __license__   = 'GPL v3'
-__copyright__ = '2009-2010, Darko Miletic <darko.miletic at gmail.com>'
+__copyright__ = '2009-2012, Darko Miletic <darko.miletic at gmail.com>'
 '''
 www.adventuregamers.com
 '''
@@ -14,24 +14,24 @@ class AdventureGamers(BasicNewsRecipe):
     publisher             = 'Adventure Gamers'
     category              = 'news, games, adventure, technology'
     oldest_article        = 10
-    delay                 = 10
+    #delay                 = 10
     max_articles_per_feed = 100
     no_stylesheets        = True
-    encoding              = 'cp1252'
+    encoding              = 'utf8'
     remove_javascript     = True
     use_embedded_content  = False
     INDEX                 = u'http://www.adventuregamers.com'
     extra_css             = """
                                 .pageheader_type{font-size: x-large; font-weight: bold; color: #828D74}
-                                .pageheader_title{font-size: xx-large; color: #394128}
+                                .pageheader_title,.page_title{font-size: xx-large; color: #394128}
                                 .pageheader_byline{font-size: small; font-weight: bold; color: #394128}
                                 .score_bg {display: inline; width: 100%; margin-bottom: 2em}
                                 .score_column_1{ padding-left: 10px; font-size: small; width: 50%}
                                 .score_column_2{ padding-left: 10px; font-size: small; width: 50%}
                                 .score_column_3{ padding-left: 10px; font-size: small; width: 50%}
-                                .score_header{font-size: large; color: #50544A}
+                                .score_header{font-size: large; color: #50544A}                                
-                                .bodytext{display: block}
+                                img{margin-bottom: 1em;}
-                                body{font-family: Helvetica,Arial,sans-serif}
+                                body{font-family: 'Open Sans',Helvetica,Arial,sans-serif}                                
                             """
 
     conversion_options = {
@@ -41,35 +41,38 @@ class AdventureGamers(BasicNewsRecipe):
                         , 'language'  : language
                         }
 
-    keep_only_tags = [
+    keep_only_tags = [dict(name='div', attrs={'class':'cleft_inn'})]
-                       dict(name='div', attrs={'class':'content_middle'})
-                     ]
-
     remove_tags = [
-                     dict(name=['object','link','embed','form'])
+                     dict(name=['object','link','embed','form','iframe','meta'])
-                    ,dict(name='div', attrs={'class':['related-stories','article_leadout','prev','next','both']})
+                    ,dict(name='a', attrs={'href':'http://www.adventuregamers.com/about/scoring'})
+                    ,dict(name='a', attrs={'href':'http://www.adventuregamers.com/about/policies'})
                   ]
-
+    remove_tags_after = [dict(name='div', attrs={'class':'bodytext'})]
-    remove_tags_after = [dict(name='div', attrs={'class':'toolbar_fat'})]
     remove_attributes = ['width','height']
 
-    feeds = [(u'Articles', u'http://feeds2.feedburner.com/AdventureGamers')]
+    feeds = [(u'Articles', u'http://www.adventuregamers.com/rss/')]
 
     def get_article_url(self, article):
-        return article.get('guid',  None)
+        url = BasicNewsRecipe.get_article_url(self, article)
+        if '/videos/' in url or '/hypeometer/' in url:
+           return None
+        return url
 
     def append_page(self, soup, appendtag, position):
-        pager = soup.find('div',attrs={'class':'toolbar_fat_next'})
+        pager = soup.find('div', attrs={'class':'pagination_big'})
         if pager:
-           nexturl = self.INDEX + pager.a['href']
+           nextpage = soup.find('a', attrs={'class':'next-page'})
-           soup2 = self.index_to_soup(nexturl)
+           if nextpage:
-           texttag = soup2.find('div', attrs={'class':'bodytext'})
+               nexturl = nextpage['href']
-           for it in texttag.findAll(style=True):
+               soup2 = self.index_to_soup(nexturl)
-               del it['style']
+               texttag = soup2.find('div', attrs={'class':'bodytext'})
-           newpos = len(texttag.contents)
+               for it in texttag.findAll(style=True):
-           self.append_page(soup2,texttag,newpos)
+                   del it['style']
-           texttag.extract()
+               newpos = len(texttag.contents)
-           appendtag.insert(position,texttag)
+               self.append_page(soup2,texttag,newpos)
+               texttag.extract()
+               pager.extract()
+               appendtag.insert(position,texttag)
 
 
     def preprocess_html(self, soup):
@@ -78,7 +81,7 @@ class AdventureGamers(BasicNewsRecipe):
         for item in soup.findAll('div', attrs={'class':'floatright'}):
             item.extract()
         self.append_page(soup, soup.body, 3)
-        pager = soup.find('div',attrs={'class':'toolbar_fat'})
+        pager = soup.find('div',attrs={'class':'pagination_big'})
         if pager:
            pager.extract()
         return self.adeify_images(soup)

recipes/cato.recipe

@@ -0,0 +1,75 @@
+import re
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class CATOInstitute(BasicNewsRecipe):
+    title          		= u'The CATO Institute'
+    description      	= "The Cato Institute is a public policy research organization — a think tank — \
+dedicated to the principles of individual liberty, limited government, free markets and peace.\
+ Its scholars and analysts conduct independent, nonpartisan research on a wide range of policy issues."
+    __author__     	= '_reader'
+    __date__                	= '05 July 2012'
+    __version__            	 = '1.0'
+    cover_url        	= 'http://www.cato.org/images/logo.jpg'
+    masthead_url        	= 'http://www.cato.org/images/logo.jpg'
+    language        	= 'en'
+    oldest_article 	= 30 #days
+    max_articles_per_feed 	= 100
+    needs_subscription 	= False
+    publisher             	= 'CATO Institute'
+    category              	= 'commentary'
+    tags 		= 'commentary'
+    publication_type 	= 'blog'
+    no_stylesheets         	= True
+    use_embedded_content  = False
+    encoding        	= None
+    simultaneous_downloads = 10
+    recursions      	= 0
+    remove_javascript 	= True
+    remove_empty_feeds    	= True
+    auto_cleanup 	= True
+
+    conversion_options = {
+                           'comments'    : description,
+                           'tags'             : tags,
+                           'language'      : language,
+                           'publisher'      : publisher,
+                           'authors'        : publisher,
+                           'smarten_punctuation' : True
+                            }
+
+
+    feeds          = [
+	(u'Cato Recent Op-Eds', u'http://feeds.cato.org/CatoRecentOpeds'),
+	(u'Cato Homepage Headlines', u'http://feeds.cato.org/CatoHomepageHeadlines'),
+	(u'Cato Media Updates', u'http://feeds.cato.org/CatoMediaUpdates'),
+	(u'Cato@Liberty', u'http://feeds.cato.org/Cato-at-liberty'),
+	(u'Cato Unbound', u'http://feeds.feedburner.com/cato-unbound'),
+	(u'Education and Child Policy', u'http://www.cato.org/rss/ra.xml?name=education-child-policy'),
+	(u'Finance, Banking & Monetary Policy', u'http://www.cato.org/rss/ra.xml?name=finance-banking-monetary-policy'),
+	(u'Government and Politics', u'http://www.cato.org/rss/ra.xml?name=government-politics'),
+	(u'International Economics & Development', u'http://www.cato.org/rss/ra.xml?name=international-economics-development'),
+	(u'Political Philosophy', u'http://www.cato.org/rss/ra.xml?name=political-philosophy'),
+	(u'Social Security', u'http://www.cato.org/rss/ra.xml?name=social-security'),
+	(u'Telecom, Internet & Information Policy', u'http://www.cato.org/rss/ra.xml?name=telecom-internet-information-policy'),
+	(u'Energy and Environment', u'http://www.cato.org/rss/ra.xml?name=energy-environment'),
+	(u'Foreign Policy and National Security', u'http://www.cato.org/rss/ra.xml?name=foreign-policy-national-security'),
+	(u'Health Care', u'http://www.cato.org/rss/ra.xml?name=health-care'),
+	(u'Law and Civil Liberties', u'http://www.cato.org/rss/ra.xml?name=law-civil-liberties'),
+	(u'Regulatory Studies', u'http://www.cato.org/rss/ra.xml?name=regulatory-studies'),
+	(u'Tax and Budget Policy', u'http://www.cato.org/rss/ra.xml?name=tax-budget-policy'),
+	(u'Trade and Immigration', u'http://www.cato.org/rss/ra.xml?name=trade-immigration')
+	]
+
+
+
+
+    def print_version(self,url):
+          R_unbound = re.compile(r'(^.*cato-unbound.*)(\/\?utm_source.*$)' , re.DOTALL | re.IGNORECASE )     #CATO Unbound
+          R_pubs       = re.compile(r'(^.*\/publications\/.*$)' , re.DOTALL | re.IGNORECASE )                              #CATO Publications
+          if re.match(R_unbound, url):
+             printURL = r'\g<1>' + '/print/'
+          elif re.match(R_pubs, url):
+             printURL = url + '?print'
+          else:
+             printURL = url + '/print/'
+          return printURL

recipes/cosmopolitan_uk.recipe

@@ -1,13 +1,13 @@
-import re
 from calibre.web.feeds.news import BasicNewsRecipe
-#from calibre import __appname__
+import re
-from calibre.utils.magick import Image
+from calibre import browser
+
 class AdvancedUserRecipe1306097511(BasicNewsRecipe):
     title          = u'Cosmopolitan UK'
-    description = 'Fashion, beauty and Gossip for women from COSMOPOLITAN -UK'
+    description = 'Author : D.Asbury : Womens Fashion, beauty and Gossip for women from COSMOPOLITAN -UK'
 
     __author__ = 'Dave Asbury'
-    #last update 21/12/11
+    #last update 7/7/12 hopefully get current cover from itunes
     # greyscale code by Starson
     cover_url = 'http://www.cosmopolitan.magazine.co.uk/files/4613/2085/8988/Cosmo_Cover3.jpg'
     no_stylesheets = True
@@ -39,14 +39,19 @@ class AdvancedUserRecipe1306097511(BasicNewsRecipe):
     feeds          = [
         (u'Love & Sex', u'http://www.cosmopolitan.co.uk/love-sex/rss/'), (u'Men', u'http://cosmopolitan.co.uk/men/rss/'), (u'Fashion', u'http://cosmopolitan.co.uk/fashion/rss/'), (u'Hair & Beauty', u'http://cosmopolitan.co.uk/beauty-hair/rss/'), (u'LifeStyle', u'http://cosmopolitan.co.uk/lifestyle/rss/'), (u'Cosmo On Campus', u'http://cosmopolitan.co.uk/campus/rss/'), (u'Celebrity Gossip', u'http://cosmopolitan.co.uk/celebrity-gossip/rss/')]
 
-    def postprocess_html(self, soup, first):
+    def get_cover_url(self):
-        #process all the images
+        soup = self.index_to_soup('http://itunes.apple.com/gb/app/cosmopolitan-uk/id461363572?mt=8')
-        for tag in soup.findAll(lambda tag: tag.name.lower()=='img' and tag.has_key('src')):
+        # look for the block containing the sun button and url
-            iurl = tag['src']
+        cov = soup.find(attrs={'alt' : 'iPhone Screenshot 1'})
-            img = Image()
+        cov2 = str(cov['src'])
-            img.open(iurl)
+        br = browser()
-            if img < 0:
+        br.set_handle_redirect(False)
-                raise RuntimeError('Out of memory')
+        try:
-            img.type = "GrayscaleType"
+            br.open_novisit(cov2)
-            img.save(iurl)
+            cover_url = cov2
-        return soup
+        except:
+            cover_url = 'http://www.cosmopolitan.magazine.co.uk/files/4613/2085/8988/Cosmo_Cover3.jpg'
+
+        return cover_url
+
+

recipes/empire_magazine.recipe

@@ -0,0 +1,51 @@
+import re
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class AdvancedUserRecipe1341650280(BasicNewsRecipe):
+
+    title          = u'Empire Magazine'
+    description = 'Author D.Asbury. Film articles from Empire Mag. '
+    __author__ = 'Dave Asbury'
+    # last updated 7/7/12
+    remove_empty_feeds = True
+    remove_javascript     = True
+    no_stylesheets = True
+    #oldest_article = 7
+    max_articles_per_feed = 20
+    cover_url = 'http://www.empireonline.com/images/magazine/cover.jpg'
+    conversion_options = {
+            'linearize_tables' : True,
+	}
+    #auto_cleanup = True
+    preprocess_regexps = [
+        (re.compile(r'<a href="http://twitter.com/share.*?</a>', re.IGNORECASE | re.DOTALL), lambda match: ''),
+        (re.compile(r'<head>.*?<!-- CONTENT: START -->', re.IGNORECASE | re.DOTALL), lambda match: '<head></head><!-- CONTENT: START -->'),
+        (re.compile(r'<!-- LATEST NEWS HEADLINES: START -->.*?<!-- LATEST NEWS HEADLINES: END -->', re.IGNORECASE | re.DOTALL), lambda match: '<!-- LATEST NEWS HEADLINES: START --><!-- LATEST NEWS HEADLINES: END -->'),
+        (re.compile(r'<!-- RELATED FUTURE FILMS: START -->.*?<!-- RELATED FUTURE FILMS: END -->', re.IGNORECASE | re.DOTALL), lambda match: '<!-- RELATED FUTURE FILMS: START --><!-- RELATED FUTURE FILMS: END -->'),
+        (re.compile(r'<!-- CURRENT HIGHLIGHTS: START-->.*?<!-- CURRENT HIGHLIGHTS: END -->', re.IGNORECASE | re.DOTALL), lambda match: '<!-- CURRENT HIGHLIGHTS: START--><!-- CURRENT HIGHLIGHTS: END -->'),
+        (re.compile(r'<!-- RELATED REVIEWS: START -->.*?<!-- RELATED REVIEWS: END -->', re.IGNORECASE | re.DOTALL), lambda match: '<!-- RELATED REVIEWS: START --><!-- RELATED REVIEWS: END -->'),
+        (re.compile(r'<!-- RELATED INTERVIEWS -->.*?<!-- RELATED REVIEWS: END -->', re.IGNORECASE | re.DOTALL), lambda match: '<!-- RELATED INTERVIEWS --><!-- RELATED REVIEWS: END -->'),
+        (re.compile(r'<!-- CONTENT: END -->.*?</body>', re.IGNORECASE | re.DOTALL), lambda match: '<!-- CONTENT: END --></body>'),
+        (re.compile(r'<!-- STORY: END -->.*?</body>', re.IGNORECASE | re.DOTALL), lambda match: '<!-- STORY: END --></body>'),
+        (re.compile(r'<!-- RATINGS GUIDE: START-->.*?<!-- RATINGS GUIDE: END-->', re.IGNORECASE | re.DOTALL), lambda match: '<!-- RATINGS GUIDE: START--><!-- RATINGS GUIDE: END-->'),
+        (re.compile(r'<strong>SUBSCRIBE TO EMPIRE</strong>.*?</tbody>', re.IGNORECASE | re.DOTALL), lambda match: '</tbody>'),
+        (re.compile(r'<!-- USER REVIEWS: START -->.*?<!-- USER REVIEWS: END -->', re.IGNORECASE | re.DOTALL), lambda match: '<!-- USER REVIEWS: START --><!-- USER REVIEWS: END -->'),
+        (re.compile(r'Advertisement', re.IGNORECASE | re.DOTALL), lambda match: ''),
+        (re.compile(r'<a name="haveyoursay".*?now to have your say.', re.IGNORECASE | re.DOTALL), lambda match: ''),
+	]
+    keep_only_tags = [
+                             # dict(name='h1'),
+	         # dict(attrs={'class' : 'mediumblack'}),
+                              ]
+    remove_tags = [dict(name='td', attrs={'width':'200', 'valign' : 'top'}),
+                            dict(name='b'),
+                            dict(name='a',attrs={'name' : 'haveyoursay'}),
+                            dict(attrs={'class' : 'newslink'}),
+		]
+
+
+    feeds          = [(u'News', u'http://feed43.com/7338478755673147.xml'),
+	      (u'Recent Features',u'http://feed43.com/4346347750304760.xml'),
+                          (u'Interviews',u'http://feed43.com/3418350077724081.xml'),
+	      (u'Film Reviews',u'http://feed43.com/2643703076510627.xml'),
+	]

recipes/fhm_uk.recipe

@@ -2,19 +2,19 @@ from calibre.web.feeds.news import BasicNewsRecipe
 
 class AdvancedUserRecipe1325006965(BasicNewsRecipe):
     title          = u'FHM UK'
-    description = 'Good News for Men'
+    description = 'Good News for Men.'
     cover_url = 'http://www.greatmagazines.co.uk/covers/large/w197/current/fhm.jpg'
     #   cover_url = 'http://profile.ak.fbcdn.net/hprofile-ak-snc4/373529_38324934806_64930243_n.jpg'
     masthead_url = 'http://www.fhm.com/App_Resources/Images/Site/re-design/logo.gif'
     __author__ = 'Dave Asbury'
-    # last updated 14/4/12
+    # last updated 1/7/12
     language = 'en_GB'
     oldest_article = 28
-    max_articles_per_feed = 12
+    max_articles_per_feed = 8
     remove_empty_feeds = True
     no_stylesheets = True
     #auto_cleanup = True
-    #articles_are_obfuscated = True
+   # articles_are_obfuscated = True
     keep_only_tags = [
                dict(name='h1'),
                dict(name='img',attrs={'id' : 'ctl00_Body_imgMainImage'}),
@@ -28,11 +28,18 @@ class AdvancedUserRecipe1325006965(BasicNewsRecipe):
 
     #]
     feeds          = [
-    (u'From the Homepage',u'http://feed43.com/0032328550253453.xml'),
+        (u'Homepage 1',u'http://feed43.com/6655867614547036.xml'),
-                           #http://feed43.com/8053226782885416.xml'),
+        (u'Homepage 2',u'http://feed43.com/4167731873103110.xml'),
-    (u'Funny - The Very Best Of The Internet',u'http://feed43.com/4538510106331565.xml'),
+        (u'Homepage 3',u'http://feed43.com/7667138788771570.xml'),
-                           (u'Upgrade',u'http://feed43.com/0877305847443234.xml'),
+        (u'Homepage 4',u'http://feed43.com/6550421522527341.xml'),
-    #(u'The Final Countdown', u'http://feed43.com/3576106158530118.xml'),
+        (u'Funny - The Very Best Of The Internet',u'http://feed43.com/4538510106331565.xml'),
-    #(u'Gaming',u'http://feed43.com/0755006465351035.xml'),
+        (u'Gaming',u'http://feed43.com/6537162612465672.xml'),
-            (u'Gaming',u'http://feed43.com/6537162612465672.xml'),
+        (u'Girls',u'http://feed43.com/3674777224513254.xml'),
                            ]
+
+    extra_css = '''
+                    h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;}
+                    h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;}
+                    p{font-family:Arial,Helvetica,sans-serif;font-size:small;}
+                    body{font-family:Helvetica,Arial,sans-serif;font-size:small;}
+        '''

recipes/heritage_foundation.recipe

@@ -0,0 +1,71 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+class HeritageFoundation(BasicNewsRecipe):
+    title		= u'The Heritage Foundation'
+    description		= 'Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—\
+whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, \
+individual freedom, traditional American values, and a strong national defense.'
+    __author__ 	= '_reader'
+    __date__                	= '05 July 2012'
+    __version__            	 = '1.0'
+    oldest_article	= 30
+    max_articles_per_feed	= 100
+    publisher		= 'The Heritage Foundation'
+    category		= 'commentary'
+    tags 		= 'commentary'
+    language		= 'en'
+    publication_type	= 'blog'
+    cover_url		= 'http://www.heritage.org/static/images/logo.jpg'
+    masthead_url        	= 'http://www.heritage.org/static/images/logo.jpg'
+    encoding        	= None
+    use_embedded_content	= False
+    no_stylesheets	= True
+    remove_javascript	= True
+    recursions		= 0
+    remove_empty_feeds     = True
+    auto_cleanup = True
+
+    conversion_options = {
+                           'comments'    : description,
+                           'tags'             : tags,
+                           'language'      : language,
+                           'publisher'      : publisher,
+                           'authors'        : publisher,
+                           'smarten_punctuation' : True
+                            }
+
+
+    feeds = [
+(u'Agriculture', u'http://www.heritage.org/static/RSS/Agriculture.xml'),
+(u'Alliances', u'http://www.heritage.org/static/RSS/Alliances.xml'),
+(u'Arms Control and Non-Proliferation', u'http://www.heritage.org/static/RSS/Arms-Control-and-Non-Proliferation.xml'),
+(u'Budget and Spending', u'http://www.heritage.org/static/RSS/Budget-and-Spending.xml'),
+(u'Economic Freedom', u'http://www.heritage.org/static/RSS/Economic-Freedom.xml'),
+(u'Economy', u'http://www.heritage.org/static/RSS/Economy.xml'),
+(u'Education', u'http://www.heritage.org/static/RSS/Education.xml'),
+(u'Energy and Environment', u'http://www.heritage.org/static/RSS/Energy-and-Environment.xml'),
+(u'Family and Marriage', u'http://www.heritage.org/static/RSS/Family-And-Marriage.xml'),
+(u'Foreign Aid and Development', u'http://www.heritage.org/static/RSS/Foreign-Aid-and-Development.xml'),
+(u'Health Care', u'http://www.heritage.org/static/RSS/Health-Care.xml'),
+(u'Homeland Security', u'http://www.heritage.org/static/RSS/Homeland-Security.xml'),
+(u'Housing', u'http://www.heritage.org/static/RSS/Housing.xml'),
+(u'Immigration', u'http://www.heritage.org/static/RSS/Immigration.xml'),
+(u'International Conflicts', u'http://www.heritage.org/static/RSS/International-Conflicts.xml'),
+(u'International Law', u'http://www.heritage.org/static/RSS/International-Law.xml'),
+(u'Labor', u'http://www.heritage.org/static/RSS/Labor.xml'),
+(u'Legal Issues', u'http://www.heritage.org/static/RSS/Legal.xml'),
+(u'Missile Defense', u'http://www.heritage.org/static/RSS/Missile-Defense.xml'),
+(u'National Security and Defense', u'http://www.heritage.org/static/RSS/National-Security-and-Defense.xml'),
+(u'Political Thought', u'http://www.heritage.org/static/RSS/Political-Thought.xml'),
+(u'Public Diplomacy', u'http://www.heritage.org/static/RSS/Public-Diplomacy.xml'),
+(u'Regulation', u'http://www.heritage.org/static/RSS/Regulation.xml'),
+(u'Religion and Civil Society', u'http://www.heritage.org/static/RSS/Religion-and-Civil-Society.xml'),
+(u'Retirement Security', u'http://www.heritage.org/static/RSS/Retirement-Security.xml'),
+(u'Space Policy', u'http://www.heritage.org/static/RSS/Space-Policy.xml'),
+(u'Taxes', u'http://www.heritage.org/static/RSS/Taxes.xml'),
+(u'Terrorism', u'http://www.heritage.org/static/RSS/Terrorism.xml'),
+(u'Trade', u'http://www.heritage.org/static/RSS/Trade.xml'),
+(u'Transportation', u'http://www.heritage.org/static/RSS/Transportation.xml'),
+(u'Welfare', u'http://www.heritage.org/static/RSS/Welfare.xml'),
+(u'Worldwide Freedom and Human Rights', u'http://www.heritage.org/static/RSS/Worldwide-Freedom-and-Human-Rights.xml'),
+]
+

recipes/hola.recipe

@@ -1,38 +1,73 @@
 #!/usr/bin/env  python
-__license__   = 'GPL v3'
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
-__copyright__ = '2010, Brendan Sleight <bms.calibre at barwap.com>'
+__license__     = 'GPL v3'
+__copyright__   = '30 June 2012, desUBIKado'
+__author__      = 'desUBIKado'
+__description__ = 'Diario de actualidad, moda y belleza'
+__version__     = 'v0.01'
+__date__        = '30, June 2012'
 '''
-hola.com
+http://www.hola.com/
 '''
 
 from calibre.web.feeds.news import BasicNewsRecipe
 
-class Hackaday(BasicNewsRecipe):
+class hola_es(BasicNewsRecipe):
-    title                 = u'Hola'
+    __author__     = 'desUBIKado'
-    __author__            = 'bmsleight'
+    description    = 'Diario de actualidad, moda y belleza'
-    description           = 'diario de actualidad, moda y belleza.'
+    title          = u'¡Hola!'
-    oldest_article        = 10
+    publisher      = 'Hola S.L.'
+    category       = 'Spanish celebrities, Entertainment News, Royalty, Daily Variety, Hollywood'
+    language       = 'es'
+    masthead_url   = 'http://imagenes.hola.com/comunes/2008/logo-holacom.gif'
+    timefmt        = '[%a, %d %b, %Y]'
+    oldest_article = 7
+    delay          = 1
+    encoding       = 'utf-8'
     max_articles_per_feed = 100
-    no_stylesheets        = True
-    language              = 'es'
-
     use_embedded_content  = False
-
+    remove_empty_feeds    = True
-    keep_only_tags      = [
+    remove_javascript = True
-                           dict(name='div', attrs={'id':'cuerpo'})
+    no_stylesheets = True
-                          ]
 
     feeds               = [
-                         (u'Famosos'        , u'http://www.hola.com/famosos/rss.xml'           ),
+                            (u'Famosos'        , u'http://www.hola.com/famosos/rss.xml'           )
-                         (u'Realeza'        , u'http://www.hola.com/realeza/rss.xml'           ),
+                           ,(u'Realeza'        , u'http://www.hola.com/realeza/rss.xml'           )
-                         (u'Cine'           , u'http://www.hola.com/cine/rss.xml'              ),
+                           ,(u'Cine'           , u'http://www.hola.com/cine/rss.xml'              )
-                         (u'Música'         , u'http://www.hola.com/musica/rss.xml'            ),
+                           ,(u'M\xfasica'      , u'http://www.hola.com/musica/rss.xml'            )
-                         (u'Moda y modelos' , u'http://www.hola.com/moda/portada/rss.xml'      ),
+                           ,(u'Moda y modelos' , u'http://www.hola.com/moda/portada/rss.xml'      )
-                         (u'Belleza y salud', u'http://www.hola.com/belleza/portada/rss.xml'   ),
+                           ,(u'Belleza y salud', u'http://www.hola.com/belleza/portada/rss.xml'   )
-                         (u'Niños'          , u'http://www.hola.com/ninos/rss.xml'             ),
+                           ,(u'Ni\xf1os'       , u'http://www.hola.com/ninos/rss.xml'             )
-                         (u'Todas las noticias', u'http://int2.hola.com/app/feeds/rss_hola.php'),
+                          ]
+
+
+
+
+    keep_only_tags     = [dict(name='div', attrs={'id':['cuerpo','com']})]
+
+    remove_tags        = [dict(name='div', attrs={'id':['relacionadas','slide-enlaces-patrocinados','comentarios']}),
+                          dict(name='div', attrs={'class':['slide-enlaces-patricinados-tit','compartir']})
                          ]
 
+    remove_tags_after  = dict(name='div' , attrs={'id':'comentarios'})
+
+
+    # Recuperamos la portada de papel (la imagen 520 tiene mayor resolucion)
+
+    def get_cover_url(self):
+        index = 'http://www.hola.com/abono/ediciondigital/'
+        soup = self.index_to_soup(index)
+        for image in soup.findAll('img',src=True):
+           if image['src'].endswith('portada-revista-hola-520.jpg'):
+              return 'http://www.hola.com/' + image['src']
+        return None
+
+
     def get_article_url(self, article):
         url = article.get('guid', None)
         return url
+
+    extra_css = '''
+                    h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:30px;}
+                    h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal; font-style:italic; font-size:18px;}
+                '''

recipes/national_geographic_mag.recipe

@@ -8,7 +8,7 @@ class NatGeoMag(BasicNewsRecipe):
     oldest_article         = 31
     max_articles_per_feed  = 50
     category               = 'geography, magazine'
-    language               = 'en_US'
+    language               = 'en'
     publication_type       = 'magazine'
     cover_url              = 'http://www.yourlogoresources.com/wp-content/uploads/2011/09/national-geographic-logo.jpg'
     use_embedded_content   = False

recipes/nzz_folio.recipe

@@ -0,0 +1,61 @@
+
+__license__   = 'GPL v3'
+__copyright__ = '2012 Bernd Leinfelder <skoll1975@gmail.com>'
+
+'''
+www.nzzfolio.ch
+'''
+
+from calibre.web.feeds.recipes import BasicNewsRecipe
+
+class Nzzfolio(BasicNewsRecipe):
+    title                 = 'NZZ Folio'
+    __author__            = 'Bernd Leinfelder'
+    description           = 'Aktuelle Artikel des NZZ Folio'
+    publisher             = 'NZZ AG'
+    category              = 'news, politics, nachrichten, Switzerland'
+    oldest_article        = 35
+    max_articles_per_feed = 100
+    no_stylesheets        = True
+    encoding              = 'utf-8'
+    use_embedded_content  = False
+    language              = 'de'
+    extra_css             = """
+                               body{font-family: Georgia,"Times New Roman",Times,serif }
+                               .artikel h3,.artikel h4,.bildLegende,.question,.autor{font-family: Arial,Verdana,Helvetica,sans-serif}
+                               .bildLegende{font-size: small}
+                               .autor{font-size: 0.9375em; color: #666666}
+                               .quote{font-size: large !important; 
+                                      font-style: italic; 
+                                      font-weight: normal !important; 
+                                      border-bottom: 1px dotted #BFBFBF; 
+                                      border-top: 1px dotted #BFBFBF; 
+                                      line-height: 1.25em}
+                                .quelle{color: #666666; font-style: italic; white-space: nowrap}
+                            """
+
+    conversion_options = {
+                             'comments'  : description
+                            ,'tags'      : category
+                            ,'language'  : language
+                            ,'publisher' : publisher
+                            ,'linearize_tables' : True
+                         }
+
+
+    remove_attributes=['width','height','lang']
+    remove_tags_before = dict(id='content')
+    remove_tags_after = dict(id='content')
+    remove_tags = [
+                     dict(name=['h2','object','link','base','meta','iframe'])
+                    ,dict(id='artikelBar')
+                    ,dict(id='foot')
+                    ,dict(id='bildLegende')
+                    ,dict(name='div',attrs={'class':['box']})
+
+                  ]
+
+    feeds = [
+               (u'NZZ Folio' , u'http://rss.nzzfolio.ch/')
+            ]
+

recipes/onda_rock.recipe

@@ -1,21 +1,27 @@
 __license__   = 'GPL v3'
+__author__    = 'faber1971'
+description   = 'Italian rock webzine - v1.01 (6, July 2012)'
 from calibre.web.feeds.news import BasicNewsRecipe
-
 class AdvancedUserRecipe1328535130(BasicNewsRecipe):
     title          = u'Onda Rock'
     __author__    = 'faber1971'
     description   = 'Italian rock webzine'
     language = 'it'
-
+    oldest_article = 15
-
-    oldest_article = 7
     max_articles_per_feed = 100
     auto_cleanup = False
     remove_tags = [
-                            dict(name='div', attrs={'id':['boxHeader','boxlinks_med','footer','boxinterviste','box_special_med','boxdiscografia_head','path']}),
+                            dict(name='div', attrs={'id':['boxHeader','boxlinks_med','footer','boxinterviste','box_special_med','boxdiscografia_head','path','widget','menuarea','headerarea']}),
                             dict(name='div', attrs={'align':'left'}),
+                            dict(name='div', attrs={'class':['media','boxarticoli']}),
                             dict(name='div', attrs={'style':'text-align: center'}),
+                            dict(name='table', attrs={'cellpadding':'0'}),
+                            dict(name='span', attrs={'class':'liketext'}),
                             ]
     no_stylesheets = True
     feeds          = [(u'Onda Rock', u'http://www.ondarock.it/feed.php')]
-    masthead_url            = 'http://profile.ak.fbcdn.net/hprofile-ak-snc4/71135_45820579767_4993043_n.jpg'
+    masthead_url            = 'http://api.ning.com/files/4ot8ampp*-rYQuwL2NoaHvVqcyu7VMyWyan12a9QMsJUWxk-q5V1-34wnD-Wj9B5qWjc1yPMLGiwQg8hZJxaySeaG2lx8hpV/2009_banner_ondarock.gif'
+
+    extra_css = '''
+        .boxtabscontain_page {border: 1px solid #E0E0E0;clear: both;font-family: "Verdana", "Arial", "Helvetica", sans-serif;font-size: 10px;line-height: 17px;margin: 0px 0px 20px;padding: 10px 10px 10px 40px;position: relative;top: -1px;width: 258px;z-index: 1;}
+                '''

recipes/richmond_times_dispatch.recipe

@@ -1,59 +1,100 @@
+import re
 from calibre.web.feeds.recipes import BasicNewsRecipe
 
-class AdvancedUserRecipe1335532466(BasicNewsRecipe):
+class RichmondTimesDispatch(BasicNewsRecipe):
-    title                = u'Richmond Times-Dispatch'
+    title               = u'Richmond Times-Dispatch'
-    description      = 'News from Richmond, Virginia, USA'
+    description         = "The Richmond Times-Dispatch is the primary daily newspaper in Richmond, \
-    __author__     = 'jde'
+                            the capital of Virginia, United States, as well as the Virginia cities of Petersburg, \
-    cover_url        = 'http://static2.dukecms.com/va_tn/timesdispatch_com/site-media/img/icons/logo252x97.png'
+                            Chester. Hopewell, Colonial Heights, Charlottesville, Lynchburg, Waynesboro, \
-    language        = 'en'
+                            and is also a default paper for rural regions of the state.  \
-    encoding        = 'utf8'
+                            The RTD has published in some form for more than 150 years."
-    oldest_article = 1 #days
+    __author__          = '_reader'
-    max_articles_per_feed = 25
+    __date__            = '05 July 2012'
-    needs_subscription = False
+    __version__         = '1.4'
-    remove_javascript = True
+    cover_url           = 'http://static2.dukecms.com/va_tn/timesdispatch_com/site-media/img/icons/logo252x97.png'
-    recursions      = 0
+    masthead_url            = 'http://static2.dukecms.com/va_tn/timesdispatch_com/site-media/img/icons/logo252x97.png'
-    use_embedded_content = False
+    language            = 'en'
-    no_stylesheets = True
+    oldest_article      = 1.5 #days
-    auto_cleanup = True
+    max_articles_per_feed   = 100
+    needs_subscription  = False
+    publisher           = 'timesdispatch.com'
+    category            = 'news, commentary'
+    tags                = 'news'
+    publication_type    = 'newspaper'
+    no_stylesheets      = True
+    use_embedded_content= False
+    encoding            = None
+    simultaneous_downloads  = 20
+    recursions          = 0
+    remove_javascript   = True
+    remove_empty_feeds  = True
+    auto_cleanup        = False
+
+    conversion_options = {
+                           'comments'    : description,
+                           'tags'             : tags,
+                           'language'      : language,
+                           'publisher'      : publisher,
+                           'authors'        : publisher,
+                           'smarten_punctuation' : True
+                            }
+
+    remove_tags_before = dict(id='hnews hentry item')
+
+    remove_tags_after   = dict(name='hr')
+
+    remove_tags =   [
+                    dict(name='div', attrs={'id':['mg_hd', 'mg_ft', 'sr_b', 'comments_left', 'comments_right']})
+                    ,dict(name='div', attrs={'class':['bottom_social','article_bottom']})
+                    ,dict(name='table', attrs={'class':['ap-mediabox-table', 'ap-htmltable-table', 'ap-photogallery-table', 'ap-htmlfragment-table']})
+                    ]
+
+
+    preprocess_regexps = [
+                   (re.compile(r'<table class="ap-story-table hnews hentry item".*?<td class="ap-story-td">', re.DOTALL|re.IGNORECASE), lambda match: ''),
+                   (re.compile(r'<p>\s*http://www2.timesdispatch.*?</p>', re.DOTALL|re.IGNORECASE), lambda match: ''),
+                   (re.compile(r'<p>\s*<img src="http://static2.dukecms.*?</p>', re.DOTALL|re.IGNORECASE), lambda match: ''),
+                   (re.compile(r'<p>\s*<a href="http://www2.timesdispatch.*?</p>', re.DOTALL|re.IGNORECASE), lambda match: ''),
+                   (re.compile(r'<hr.*?>', re.DOTALL|re.IGNORECASE), lambda match: ''),             #strip <hr /> line break
+                   (re.compile(r'<a\s*rel="item-license.*?Use</a>.', re.DOTALL|re.IGNORECASE), lambda match: ''),           #strip <hr /> line break
+                   (re.compile(r'<small>\s*Richmond Times-Dispatch.*?</small>', re.DOTALL|re.IGNORECASE), lambda match: ''),            #strip <hr /> line break
+                    ]
+
 
     feeds          =   [
+    ('News',        'http://www2.timesdispatch.com/list/feed/rss/news-archive'),
+    ('Breaking News',   'http://www2.timesdispatch.com/list/feed/rss/breaking-news'),
+    ('National News',   'http://www2.timesdispatch.com/list/feed/rss/national-news'),
+    ('Local News',      'http://www2.timesdispatch.com/list/feed/rss/local-news'),
+    ('Business',        'http://www2.timesdispatch.com/list/feed/rss/business'),
+    ('Local Business',  'http://www2.timesdispatch.com/list/feed/rss/local-business'),
+    ('Politics',        'http://www2.timesdispatch.com/list/feed/rss/politics'),
+    ('Virginia Politics',   'http://www2.timesdispatch.com/list/feed/rss/virginia-politics'),
+    ('Sports',      'http://www2.timesdispatch.com/list/feed/rss/sports2'),
+    ('Health',      'http://www2.timesdispatch.com/feed/rss/lifestyles/health_med_fit/'),
+    ('Entertainment/Life',  'http://www2.timesdispatch.com/list/feed/rss/entertainment'),
+    ('Arts/Theatre',    'http://www2.timesdispatch.com/feed/rss/entertainment/arts_theatre/'),
+    ('Movies',      'http://www2.timesdispatch.com/list/feed/rss/movies'),
+    ('Music',       'http://www2.timesdispatch.com/list/feed/rss/music'),
+    ('Dining & Food',   'http://www2.timesdispatch.com/list/feed/rss/dining'),
+    ('Home & Garden',   'http://www2.timesdispatch.com/list/feed/rss/home-and-garden/'),
+    #inactive('Travel',     'http://www2.timesdispatch.com/feed/rss/travel/'),
+    ('Opinion',     'http://www2.timesdispatch.com/feed/rss/news/opinion/'),
+    ('Editorials',      'http://www2.timesdispatch.com/list/feed/rss/editorial-desk'),
+    ('Columnists and Blogs',    'http://www2.timesdispatch.com/list/feed/rss/news-columnists-blogs'),
+    ('Opinion Columnists',  'http://www2.timesdispatch.com/list/feed/rss/opinion-editorial-columnists'),
+    ('Letters to the Editor',   'http://www2.timesdispatch.com/list/feed/rss/opinion-letters'),
+    ('Traffic',     'http://www2.timesdispatch.com/list/feed/rss/traffic'),
+        ]
 
-('News',
+    def print_version(self,url):
-'http://www2.timesdispatch.com/list/feed/rss/news-archive'),
+        article_num = re.sub(r'(^.*)\-([0-9]{4,10})\/$', r'\g<2>', url)
-('Breaking News',
+        ap_pat = re.compile('http')
-'http://www2.timesdispatch.com/list/feed/rss/breaking-news'),
+        #print '\nDEBUG>>>>>>>>: article_num: ', article_num
-('National News',
+        #print 'DEBUG>>>>>>>>: ap_pat.search(article_num): ', ap_pat.search(article_num)
-'http://www2.timesdispatch.com/list/feed/rss/national-news'),
+        if ap_pat.search(article_num):            #AP article, no print url
-('Local News',
+            #print 'DEBUG>>>>>>>>: AP URL: ', url
-'http://www2.timesdispatch.com/list/feed/rss/local-news'),
+            return url
-('Business',
+        else:
-'http://www2.timesdispatch.com/list/feed/rss/business'),
+            printURL = 'http://www2.timesdispatch.com/member-center/share-this/print/?content=ar' + article_num
-('Local Business',
+            return printURL
-'http://www2.timesdispatch.com/list/feed/rss/local-business'),
-('Politics',
-'http://www2.timesdispatch.com/list/feed/rss/politics'),
-('Virginia Politics',
-'http://www2.timesdispatch.com/list/feed/rss/virginia-politics'),
-('Editorials',
-'http://www2.timesdispatch.com/list/feed/rss/editorial-desk'),
-('Columnists and Blogs',
-'http://www2.timesdispatch.com/list/feed/rss/news-columnists-blogs'),
-('Opinion Columnists',
-'http://www2.timesdispatch.com/list/feed/rss/opinion-editorial-columnists'),
-('Letters to the Editor',
-'http://www2.timesdispatch.com/list/feed/rss/opinion-letters'),
-('Traffic',
-'http://www2.timesdispatch.com/list/feed/rss/traffic'),
-('Sports',
-'http://www2.timesdispatch.com/list/feed/rss/sports2'),
-('Entertainment/Life',
-'http://www2.timesdispatch.com/list/feed/rss/entertainment'),
-('Movies',
-'http://www2.timesdispatch.com/list/feed/rss/movies'),
-('Music',
-'http://www2.timesdispatch.com/list/feed/rss/music'),
-('Dining & Food',
-'http://www2.timesdispatch.com/list/feed/rss/dining'),
-
-    ]
-

recipes/warentest.recipe

@@ -0,0 +1,16 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class Warentest(BasicNewsRecipe):
+    title = u'Warentest'
+    language = 'de'
+    description = 'Stiftung Warentest is a German consumer organisation and foundation involved in investigating and comparing goods and services in an unbiased way'
+    __author__ = 'asdfdsfksd'
+    needs_subscription = False
+    max_articles_per_feed = 100
+    auto_cleanup = True
+
+    feeds = [(u'Test', u'http://www.test.de/rss/alles/')]
+
+    def get_cover_url(self):
+        return 'http://www.test.de/img/pp/logo.png'
+

resources/default_tweaks.py

@@ -515,3 +515,13 @@ compile_gpm_templates = True
 #   default_tweak_format = 'remember'
 default_tweak_format = None
 
+#: Enable multi-character first-letters in the tag browser
+# Some languages have letters that can be represented by multiple characters.
+# For example, Czech has a 'character' "ch" that sorts between "h" and "i".
+# If this tweak is True, then the tag browser will take these characters into
+# consideration when partitioning by first letter.
+# Examples:
+#    enable_multicharacters_in_tag_browser = True
+#    enable_multicharacters_in_tag_browser = True
+enable_multicharacters_in_tag_browser = True
+

setup/build_environment.py

@@ -34,6 +34,7 @@ if iswindows:
             MT = os.path.join(os.path.dirname(p), 'bin', 'mt.exe')
     MT = os.path.join(SDK, 'bin', 'mt.exe')
     os.environ['QMAKESPEC'] = 'win32-msvc'
+    ICU = r'Q:\icu'
 
 QMAKE = '/Volumes/sw/qt/bin/qmake' if isosx else 'qmake'
 if find_executable('qmake-qt4'):
@@ -97,8 +98,9 @@ if iswindows:
     prefix  = r'C:\cygwin\home\kovid\sw'
     sw_inc_dir  = os.path.join(prefix, 'include')
     sw_lib_dir  = os.path.join(prefix, 'lib')
-    icu_inc_dirs = [sw_inc_dir]
+    icu_inc_dirs = [os.path.join(ICU, 'source', 'common'), os.path.join(ICU,
-    icu_lib_dirs = [sw_lib_dir]
+        'source', 'i18n')]
+    icu_lib_dirs = [os.path.join(ICU, 'source', 'lib')]
     sqlite_inc_dirs = [sw_inc_dir]
     fc_inc = os.path.join(sw_inc_dir, 'fontconfig')
     fc_lib = sw_lib_dir

setup/installer/__init__.py

@@ -11,7 +11,7 @@ import subprocess, tempfile, os, time
 from setup import Command, installer_name
 from setup.build_environment import HOST, PROJECT
 
-BASE_RSYNC = ['rsync', '-avz', '--delete']
+BASE_RSYNC = ['rsync', '-avz', '--delete', '--force']
 EXCLUDES = []
 for x in [
     'src/calibre/plugins', 'src/calibre/manual', 'src/calibre/trac',

setup/installer/linux/freeze2.py

@@ -12,7 +12,7 @@ import sys, os, shutil, platform, subprocess, stat, py_compile, glob, \
 from setup import Command, modules, basenames, functions, __version__, \
     __appname__
 
-SITE_PACKAGES = ['IPython', 'PIL', 'dateutil', 'dns', 'PyQt4', 'mechanize',
+SITE_PACKAGES = ['PIL', 'dateutil', 'dns', 'PyQt4', 'mechanize',
         'sip.so', 'BeautifulSoup.py', 'cssutils', 'encutils', 'lxml',
         'sipconfig.py', 'xdg', 'dbus', '_dbus_bindings.so', 'dbus_bindings.py',
         '_dbus_glib_bindings.so']
@@ -54,10 +54,10 @@ binary_includes = [
                 '/lib/libreadline.so.6',
                 '/usr/lib/libchm.so.0',
                 '/usr/lib/liblcms2.so.2',
-                '/usr/lib/libicudata.so.46',
+                '/usr/lib/libicudata.so.49',
-                '/usr/lib/libicui18n.so.46',
+                '/usr/lib/libicui18n.so.49',
-                '/usr/lib/libicuuc.so.46',
+                '/usr/lib/libicuuc.so.49',
-                '/usr/lib/libicuio.so.46',
+                '/usr/lib/libicuio.so.49',
                 ]
 binary_includes += [os.path.join(QTDIR, 'lib%s.so.4'%x) for x in QTDLLS]
 

setup/installer/osx/app/main.py

@@ -364,6 +364,7 @@ class Py2App(object):
                 'application. Visit http://calibre-ebook.com for details.'),
                 CFBundleIconFile='library.icns',
                 LSMultipleInstancesProhibited=True,
+                NSHighResolutionCapable=True,
                 LSEnvironment=env
         )
         plistlib.writePlist(pl, join(self.contents_dir, 'Info.plist'))
@@ -385,7 +386,7 @@ class Py2App(object):
     @flush
     def add_poppler(self):
         info('\nAdding poppler')
-        for x in ('libpoppler.25.dylib',):
+        for x in ('libpoppler.26.dylib',):
             self.install_dylib(os.path.join(SW, 'lib', x))
         for x in ('pdftohtml', 'pdftoppm', 'pdfinfo'):
             self.install_dylib(os.path.join(SW, 'bin', x), False)
@@ -482,10 +483,6 @@ class Py2App(object):
                     shutil.rmtree(tdir)
         shutil.rmtree(os.path.join(self.site_packages, 'calibre', 'plugins'))
         self.remove_bytecode(join(self.resources_dir, 'Python', 'site-packages'))
-        # Create dummy IPython README_STARTUP
-        with open(join(self.site_packages,
-            'IPython/config/profile/README_STARTUP'), 'wb') as f:
-            f.write('\n')
 
     @flush
     def add_modules_from_dir(self, src):

setup/installer/windows/freeze.py

@@ -13,6 +13,7 @@ from setup import (Command, modules, functions, basenames, __version__,
 from setup.build_environment import msvc, MT, RC
 from setup.installer.windows.wix import WixMixIn
 
+ICU_DIR = r'Q:\icu'
 OPENSSL_DIR = r'Q:\openssl'
 QT_DIR = 'Q:\\Qt\\4.8.2'
 QT_DLLS = ['Core', 'Gui', 'Network', 'Svg', 'WebKit', 'Xml', 'XmlPatterns']
@@ -22,6 +23,7 @@ SW               = r'C:\cygwin\home\kovid\sw'
 IMAGEMAGICK      = os.path.join(SW, 'build', 'ImageMagick-6.7.6',
         'VisualMagick', 'bin')
 CRT = r'C:\Microsoft.VC90.CRT'
+SIGNTOOL = r'C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin\signtool.exe'
 
 VERSION = re.sub('[a-z]\d+', '', __version__)
 WINVER = VERSION+'.0'
@@ -147,6 +149,8 @@ class Win32Freeze(Command, WixMixIn):
                 ignore=shutil.ignore_patterns('msvc*.dll', 'Microsoft.*'))
         for x in glob.glob(self.j(OPENSSL_DIR, 'bin', '*.dll')):
             shutil.copy2(x, self.dll_dir)
+        for x in glob.glob(self.j(ICU_DIR, 'source', 'lib', '*.dll')):
+            shutil.copy2(x, self.dll_dir)
         for x in QT_DLLS:
             x += '4.dll'
             if not x.startswith('phonon'): x = 'Qt'+x

setup/installer/windows/notes.rst

@@ -30,7 +30,7 @@ If there are no windows binaries already compiled for the version of python you
 
 Run the following command to install python dependencies::
 
-    easy_install --always-unzip -U ipython mechanize pyreadline python-dateutil dnspython cssutils clientform pycrypto
+    easy_install --always-unzip -U mechanize pyreadline python-dateutil dnspython cssutils clientform pycrypto
 
 Install BeautifulSoup 3.0.x manually into site-packages (3.1.x parses broken HTML very poorly)
 
@@ -97,7 +97,7 @@ Now, run configure and make::
 
 -no-plugin-manifests is needed so that loading the plugins does not fail looking for the CRT assembly
 
-    configure -opensource -release -qt-zlib -qt-libmng -qt-libpng -qt-libtiff -qt-libjpeg -release -platform win32-msvc2008 -no-qt3support -webkit -xmlpatterns -no-phonon -no-style-plastique -no-style-cleanlooks -no-style-motif -no-style-cde -no-declarative -no-scripttools -no-audio-backend -no-multimedia -no-dbus -no-openvg -no-opengl -no-qt3support -confirm-license -nomake examples -nomake demos -nomake docs -no-plugin-manifests -openssl -I Q:\openssl\include -L Q:\openssl\lib && nmake
+    configure -ltcg -opensource -release -qt-zlib -qt-libmng -qt-libpng -qt-libtiff -qt-libjpeg -release -platform win32-msvc2008 -no-qt3support -webkit -xmlpatterns -no-phonon -no-style-plastique -no-style-cleanlooks -no-style-motif -no-style-cde -no-declarative -no-scripttools -no-audio-backend -no-multimedia -no-dbus -no-openvg -no-opengl -no-qt3support -confirm-license -nomake examples -nomake demos -nomake docs -no-plugin-manifests -openssl -I Q:\openssl\include -L Q:\openssl\lib && nmake
 
 Add the path to the bin folder inside the Qt dir to your system PATH.
 
@@ -131,11 +131,22 @@ calibre-debug -c "import _imaging, _imagingmath, _imagingft, _imagingcms"
 ICU
 -------
 
-Download the win32 msvc9 binary from http://www.icu-project.org/download/4.4.html
+Download the win32 source .zip from http://www.icu-project.org/download
 
-Note that 4.4 is the last version of ICU that can be compiled (is precompiled) with msvc9
+Extract to q:\icu
 
-Put the dlls into sw/bin and the unicode dir into sw/include and the contents of lib int sw/lib
+Add Q:\icu\bin to PATH and reboot
+
+In a Visual Studio Command Prompt
+cd to <ICU>\source
+Run set PATH=%PATH%;c:\cygwin\bin
+Run dos2unix on configure and runConfigureICU
+
+Run bash ./runConfigureICU Cygwin/MSVC
+
+Run make (note that you must have GNU make installed in cygwin)
+
+Optionally run make check
 
 Libunrar
 ----------

setup/publish.py

@@ -40,7 +40,7 @@ class Stage2(Command):
 class Stage3(Command):
 
    description = 'Stage 3 of the publish process'
-   sub_commands = ['upload_user_manual', 'upload_demo', 'sdist']
+   sub_commands = ['upload_user_manual', 'upload_demo', 'sdist', 'tag_release']
 
 class Stage4(Command):
 
@@ -50,7 +50,7 @@ class Stage4(Command):
 class Stage5(Command):
 
     description = 'Stage 5 of the publish process'
-    sub_commands = ['tag_release', 'upload_to_server']
+    sub_commands = ['upload_to_server']
 
     def run(self, opts):
         subprocess.check_call('rm -rf build/* dist/*', shell=True)

src/calibre/constants.py

@@ -4,7 +4,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = u'calibre'
-numeric_version = (0, 8, 58)
+numeric_version = (0, 8, 59)
 __version__   = u'.'.join(map(unicode, numeric_version))
 __author__    = u"Kovid Goyal <kovid@kovidgoyal.net>"
 

src/calibre/customize/builtins.py

@@ -643,7 +643,7 @@ from calibre.devices.cybook.driver import CYBOOK, ORIZON
 from calibre.devices.eb600.driver import (EB600, COOL_ER, SHINEBOOK,
                 POCKETBOOK360, GER2, ITALICA, ECLICTO, DBOOK, INVESBOOK,
                 BOOQ, ELONEX, POCKETBOOK301, MENTOR, POCKETBOOK602,
-                POCKETBOOK701, POCKETBOOK360P, PI2)
+                POCKETBOOK701, POCKETBOOK360P, PI2, POCKETBOOK622)
 from calibre.devices.iliad.driver import ILIAD
 from calibre.devices.irexdr.driver import IREXDR1000, IREXDR800
 from calibre.devices.jetbook.driver import (JETBOOK, MIBUK, JETBOOK_MINI,
@@ -689,7 +689,7 @@ plugins += [
     JETBOOK, JETBOOK_MINI, MIBUK, JETBOOK_COLOR,
     SHINEBOOK,
     POCKETBOOK360, POCKETBOOK301, POCKETBOOK602, POCKETBOOK701, POCKETBOOK360P,
-    PI2,
+    POCKETBOOK622, PI2,
     KINDLE, KINDLE2, KINDLE_DX, KINDLE_FIRE,
     NOOK, NOOK_COLOR,
     PRS505, PRST1,
@@ -1262,7 +1262,6 @@ class StoreBNStore(StoreBase):
 
     headquarters = 'US'
     formats = ['NOOK']
-    affiliate = True
 
 class StoreBeamEBooksDEStore(StoreBase):
     name = 'Beam EBooks DE'
@@ -1521,15 +1520,6 @@ class StoreOpenBooksStore(StoreBase):
     drm_free_only = True
     headquarters = 'US'
 
-class StoreOReillyStore(StoreBase):
-    name = 'OReilly'
-    description = u'Programming and tech ebooks from OReilly.'
-    actual_plugin = 'calibre.gui2.store.stores.oreilly_plugin:OReillyStore'
-
-    drm_free_only = True
-    headquarters = 'US'
-    formats = ['APK', 'DAISY', 'EPUB', 'MOBI', 'PDF']
-
 class StoreOzonRUStore(StoreBase):
     name = 'OZON.ru'
     description = u'ebooks from OZON.ru'
@@ -1670,7 +1660,6 @@ plugins += [
     StoreMobileReadStore,
     StoreNextoStore,
     StoreOpenBooksStore,
-    StoreOReillyStore,
     StoreOzonRUStore,
     StorePragmaticBookshelfStore,
     StoreRW2010Store,

src/calibre/customize/ui.py

@@ -497,6 +497,7 @@ def initialize_plugin(plugin, path_to_zip_file):
                             %tb) + '\n'+tb)
 
 def has_external_plugins():
+    'True if there are updateable (zip file based) plugins'
     return bool(config['plugins'])
 
 def initialize_plugins(perf=False):
@@ -554,6 +555,23 @@ def initialized_plugins():
 
 # CLI {{{
 
+def build_plugin(path):
+    from calibre import prints
+    from calibre.ptempfile import PersistentTemporaryFile
+    from calibre.utils.zipfile import ZipFile, ZIP_STORED
+    path = type(u'')(path)
+    names = frozenset(os.listdir(path))
+    if u'__init__.py' not in names:
+        prints(path, ' is not a valid plugin')
+        raise SystemExit(1)
+    t = PersistentTemporaryFile(u'.zip')
+    with ZipFile(t, u'w', ZIP_STORED) as zf:
+        zf.add_dir(path)
+    t.close()
+    plugin = add_plugin(t.name)
+    os.remove(t.name)
+    prints(u'Plugin updated:', plugin.name, plugin.version)
+
 def option_parser():
     parser = OptionParser(usage=_('''\
     %prog options
@@ -562,6 +580,10 @@ def option_parser():
     '''))
     parser.add_option('-a', '--add-plugin', default=None,
                       help=_('Add a plugin by specifying the path to the zip file containing it.'))
+    parser.add_option('-b', '--build-plugin', default=None,
+            help=_('For plugin developers: Path to the directory where you are'
+                ' developing the plugin. This command will automatically zip '
+                'up the plugin and update it in calibre.'))
     parser.add_option('-r', '--remove-plugin', default=None,
                       help=_('Remove a custom plugin by name. Has no effect on builtin plugins'))
     parser.add_option('--customize-plugin', default=None,
@@ -583,6 +605,8 @@ def main(args=sys.argv):
     if opts.add_plugin is not None:
         plugin = add_plugin(opts.add_plugin)
         print 'Plugin added:', plugin.name, plugin.version
+    if opts.build_plugin is not None:
+        build_plugin(opts.build_plugin)
     if opts.remove_plugin is not None:
         if remove_plugin(opts.remove_plugin):
             print 'Plugin removed'

src/calibre/db/backend.py

@@ -81,6 +81,9 @@ class DBPrefs(dict): # {{{
     def to_raw(self, val):
         return json.dumps(val, indent=2, default=to_json)
 
+    def has_setting(self, key):
+        return key in self
+
     def __getitem__(self, key):
         try:
             return dict.__getitem__(self, key)
@@ -102,6 +105,53 @@ class DBPrefs(dict): # {{{
     def set(self, key, val):
         self.__setitem__(key, val)
 
+    def get_namespaced(self, namespace, key, default=None):
+        key = u'namespaced:%s:%s'%(namespace, key)
+        try:
+            return dict.__getitem__(self, key)
+        except KeyError:
+            return default
+
+    def set_namespaced(self, namespace, key, val):
+        if u':' in key: raise KeyError('Colons are not allowed in keys')
+        if u':' in namespace: raise KeyError('Colons are not allowed in'
+                ' the namespace')
+        key = u'namespaced:%s:%s'%(namespace, key)
+        self[key] = val
+
+    def write_serialized(self, library_path):
+        try:
+            to_filename = os.path.join(library_path, 'metadata_db_prefs_backup.json')
+            with open(to_filename, "wb") as f:
+                f.write(json.dumps(self, indent=2, default=to_json))
+        except:
+            import traceback
+            traceback.print_exc()
+
+    @classmethod
+    def read_serialized(cls, library_path, recreate_prefs=False):
+        try:
+            from_filename = os.path.join(library_path,
+                    'metadata_db_prefs_backup.json')
+            with open(from_filename, "rb") as f:
+                d = json.load(f, object_hook=from_json)
+                if not recreate_prefs:
+                    return d
+                cls.clear()
+                cls.db.conn.execute('DELETE FROM preferences')
+                for k,v in d.iteritems():
+                    raw = cls.to_raw(v)
+                    cls.db.conn.execute(
+                        'INSERT INTO preferences (key,val) VALUES (?,?)', (k, raw))
+                cls.db.conn.commit()
+                cls.clear()
+                cls.update(d)
+                return d
+        except:
+            import traceback
+            traceback.print_exc()
+            raise
+        return None
 # }}}
 
 # Extra collators {{{
@@ -350,6 +400,23 @@ class DB(object):
         defs['gui_restriction'] = defs['cs_restriction'] = ''
         defs['categories_using_hierarchy'] = []
         defs['column_color_rules'] = []
+        defs['grouped_search_make_user_categories'] = []
+        defs['similar_authors_search_key'] = 'authors'
+        defs['similar_authors_match_kind'] = 'match_any'
+        defs['similar_publisher_search_key'] = 'publisher'
+        defs['similar_publisher_match_kind'] = 'match_any'
+        defs['similar_tags_search_key'] = 'tags'
+        defs['similar_tags_match_kind'] = 'match_all'
+        defs['similar_series_search_key'] = 'series'
+        defs['similar_series_match_kind'] = 'match_any'
+        defs['book_display_fields'] = [
+        ('title', False), ('authors', True), ('formats', True),
+        ('series', True), ('identifiers', True), ('tags', True),
+        ('path', True), ('publisher', False), ('rating', False),
+        ('author_sort', False), ('sort', False), ('timestamp', False),
+        ('uuid', False), ('comments', True), ('id', False), ('pubdate', False),
+        ('last_modified', False), ('size', False), ('languages', False),
+        ]
 
         # Migrate the bool tristate tweak
         defs['bools_are_tristate'] = \

src/calibre/debug.py

@@ -60,6 +60,11 @@ Run an embedded python interpreter.
             'editing tools, and then rebuilds the file from the edited HTML. '
             'Makes no additional changes to the HTML, unlike a full calibre '
             'conversion).')
+    parser.add_option('-s', '--shutdown-running-calibre', default=False,
+            action='store_true',
+            help=_('Cause a running calibre instance, if any, to be'
+                ' shutdown. Note that if there are running jobs, they '
+                'will be silently aborted, so use with care.'))
 
     parser.add_option('--test-build', help='Test binary modules in build',
             action='store_true', default=False)
@@ -258,6 +263,9 @@ def main(args=sys.argv):
     elif opts.test_build:
         from calibre.test_build import test
         test()
+    elif opts.shutdown_running_calibre:
+        from calibre.gui2.main import shutdown_other
+        shutdown_other()
     else:
         from calibre import ipython
         ipython()

src/calibre/devices/android/driver.py

@@ -10,7 +10,7 @@ import cStringIO
 
 from calibre.devices.usbms.driver import USBMS
 
-HTC_BCDS = [0x100, 0x0222, 0x0226, 0x227, 0x228]
+HTC_BCDS = [0x100, 0x0222, 0x0226, 0x227, 0x228, 0x229]
 
 class ANDROID(USBMS):
 
@@ -41,6 +41,7 @@ class ANDROID(USBMS):
                        0xca9  : HTC_BCDS,
                        0xcac  : HTC_BCDS,
                        0xccf  : HTC_BCDS,
+                       0xce5  : HTC_BCDS,
                        0x2910 : HTC_BCDS,
                        0xff9  : HTC_BCDS + [0x9999],
             },
@@ -59,6 +60,7 @@ class ANDROID(USBMS):
                        0x42d6 : [0x216],
                        0x42d7 : [0x216],
                        0x42f7 : [0x216],
+                       0x4365 : [0x216],
                      },
             # Freescale
             0x15a2 : {
@@ -72,6 +74,7 @@ class ANDROID(USBMS):
 
             # Sony Ericsson
             0xfce : {
+                0xa173 : [0x216],
                 0xd12e : [0x0100],
                 0xe156 : [0x226],
                 0xe15d : [0x226],
@@ -99,7 +102,7 @@ class ANDROID(USBMS):
                        0x681c : [0x0222, 0x0223, 0x0224, 0x0400],
                        0x6640 : [0x0100],
                        0x685b : [0x0400, 0x0226],
-                       0x685e : [0x0400],
+                       0x685e : [0x0400, 0x226],
                        0x6860 : [0x0400],
                        0x6863 : [0x226],
                        0x6877 : [0x0400],
@@ -115,7 +118,6 @@ class ANDROID(USBMS):
                     0xc004 : [0x0226],
                     0x8801 : [0x0226, 0x0227],
                     0xe115 : [0x0216], # PocketBook A10
-                    0xe107 : [0x326], # PocketBook 622
             },
 
             # Acer
@@ -209,7 +211,7 @@ class ANDROID(USBMS):
             'XT910', 'BOOK_A10', 'USB_2.0_DRIVER', 'I9100T', 'P999DW',
             'KTABLET_PC', 'INGENIC', 'GT-I9001_CARD', 'USB_2.0_DRIVER',
             'GT-S5830L_CARD', 'UNIVERSE', 'XT875', 'PRO', '.KOBO_VOX',
-            'THINKPAD_TABLET']
+            'THINKPAD_TABLET', 'SGH-T989', 'YP-G70']
     WINDOWS_CARD_A_MEM = ['ANDROID_PHONE', 'GT-I9000_CARD', 'SGH-I897',
             'FILE-STOR_GADGET', 'SGH-T959_CARD', 'SGH-T959', 'SAMSUNG_ANDROID', 'GT-P1000_CARD',
             'A70S', 'A101IT', '7', 'INCREDIBLE', 'A7EB', 'SGH-T849_CARD',
@@ -218,7 +220,7 @@ class ANDROID(USBMS):
             'A1-07___C0541A4F', 'XT912', 'MB855', 'XT910', 'BOOK_A10_CARD',
             'USB_2.0_DRIVER', 'I9100T', 'P999DW_SD_CARD', 'KTABLET_PC',
             'FILE-CD_GADGET', 'GT-I9001_CARD', 'USB_2.0_DRIVER', 'XT875',
-            'UMS_COMPOSITE', 'PRO', '.KOBO_VOX']
+            'UMS_COMPOSITE', 'PRO', '.KOBO_VOX', 'SGH-T989_CARD']
 
     OSX_MAIN_MEM = 'Android Device Main Memory'
 

src/calibre/devices/apple/driver.py

@@ -828,7 +828,15 @@ class ITUNES(DriverBase):
         if DEBUG:
             logger().info("ITUNES.get_file(): exporting '%s'" % path)
 
-        outfile.write(open(self.cached_books[path]['lib_book'].location().path).read())
+        try:
+            outfile.write(open(self.cached_books[path]['lib_book'].location().path).read())
+        except:
+            # Clean up
+            logger().info(" unable to extract books from iDevices")
+            logger().info(" deleting empty ", outfile.name)
+            outfile.close()
+            os.remove(outfile.name)
+            raise UserFeedback("Unable to extract books from iDevices", details=None, level=UserFeedback.WARN)
 
     def open(self, connected_device, library_uuid):
         '''

src/calibre/devices/eb600/driver.py

@@ -251,6 +251,19 @@ class POCKETBOOK602(USBMS):
     WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = ['PB602', 'PB603', 'PB902',
             'PB903', 'PB']
 
+class POCKETBOOK622(POCKETBOOK602):
+
+    name = 'PocketBook 622 Device Interface'
+    description    = _('Communicate with the PocketBook 622 reader.')
+    EBOOK_DIR_MAIN = ''
+
+    VENDOR_ID   = [0x0489]
+    PRODUCT_ID  = [0xe107]
+    BCD         = [0x0326]
+
+    VENDOR_NAME = 'LINUX'
+    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = 'FILE-STOR_GADGET'
+
 class POCKETBOOK360P(POCKETBOOK602):
 
     name = 'PocketBook 360+ Device Interface'

src/calibre/devices/kindle/driver.py

@@ -413,7 +413,8 @@ class KINDLE2(KINDLE):
         if not opts.extra_customization[self.OPT_APNX]:
             return
 
-        if os.path.splitext(filepath.lower())[1] not in ('.azw', '.mobi', '.prc'):
+        if os.path.splitext(filepath.lower())[1] not in ('.azw', '.mobi',
+                '.prc', '.azw3'):
             return
 
         # Create the sidecar folder if necessary

src/calibre/devices/kobo/driver.py

@@ -23,10 +23,11 @@ class KOBO(USBMS):
     gui_name = 'Kobo Reader'
     description = _('Communicate with the Kobo Reader')
     author = 'Timothy Legge'
-    version = (1, 0, 12)
+    version = (1, 0, 13)
 
     dbversion = 0
     fwversion = 0
+    supported_dbversion = 33
     has_kepubs = False
 
     supported_platforms = ['windows', 'osx', 'linux']
@@ -73,6 +74,12 @@ class KOBO(USBMS):
             ':::'+_('Kobo now shows recommendations on the device.  In some case these have '
                 'files but in other cases they are just pointers to the web site to buy. '
                 'Enable if you wish to see/delete them.'),
+            _('Attempt to support newer firmware') +
+            ':::'+_('Kobo routinely updates the firmware and the '
+                'database version.  With this option Calibre will attempt '
+                'to perform full read-write functionality - Here be Dragons!! '
+                'Enable only if you are comfortable with restoring your kobo '
+                'to factory defaults and testing software'),
             ]
 
     EXTRA_CUSTOMIZATION_DEFAULT = [
@@ -81,6 +88,7 @@ class KOBO(USBMS):
             True,
             True,
             False,
+            False,
             False
             ]
 
@@ -90,6 +98,7 @@ class KOBO(USBMS):
     OPT_SHOW_EXPIRED_BOOK_RECORDS = 3
     OPT_SHOW_PREVIEWS = 4
     OPT_SHOW_RECOMMENDATIONS = 5
+    OPT_SUPPORT_NEWER_FIRMWARE = 6
 
     def initialize(self):
         USBMS.initialize(self)
@@ -238,15 +247,6 @@ class KOBO(USBMS):
 
             cursor = connection.cursor()
 
-            #query = 'select count(distinct volumeId) from volume_shortcovers'
-            #cursor.execute(query)
-            #for row in (cursor):
-            #    numrows = row[0]
-            #cursor.close()
-
-            # Determine the database version
-            # 4 - Bluetooth Kobo Rev 2 (1.4)
-            # 8 - WIFI KOBO Rev 1
             cursor.execute('select version from dbversion')
             result = cursor.fetchone()
             self.dbversion = result[0]
@@ -422,6 +422,9 @@ class KOBO(USBMS):
                     os.unlink(fpath)
 
     def delete_books(self, paths, end_session=True):
+        if self.modify_database_check("delete_books") == False:
+            return 
+
         for i, path in enumerate(paths):
             self.report_progress((i+1) / float(len(paths)), _('Removing books from device...'))
             path = self.normalize_path(path)
@@ -458,6 +461,9 @@ class KOBO(USBMS):
         self.report_progress(1.0, _('Removing books from device...'))
 
     def remove_books_from_metadata(self, paths, booklists):
+        if self.modify_datbase_check("remove_books_from_metatata") == False:
+            return 
+
         for i, path in enumerate(paths):
             self.report_progress((i+1) / float(len(paths)), _('Removing books from device metadata listing...'))
             for bl in booklists:
@@ -588,6 +594,34 @@ class KOBO(USBMS):
 
         return path
 
+    def modify_database_check(self, function):
+        # Checks to see whether the database version is supported
+        # and whether the user has chosen to support the firmware version
+        if self.dbversion > self.supported_dbversion:
+            # Unsupported database 
+            opts = self.settings()
+            if not opts.extra_customization[self.OPT_SUPPORT_NEWER_FIRMWARE]:
+                debug_print('The database has been upgraded past supported version')
+                debug_print('The database has been upgraded past supported version')
+                self.report_progress(1.0, _('Removing books from device...'))
+                from calibre.devices.errors import UserFeedback
+                raise UserFeedback(_("Kobo database version unsupported - See details"),
+                    _('Your Kobo is running an updated firmware/database version '
+                     'As Calibre has not been updated, database editing is disabled.  '
+                     'You can enable support for your Kobo in plugin preferences.  '
+                     'Doing so may require you to perform a factory reset.  '
+                     'before selecting the "Attempt to support newer firmware" option '
+                     'you should be familiar with restoring your Kobo to factory defaults.'),
+                     UserFeedback.WARN)
+
+                return False
+            else:
+                # The user chose to edit the database anyway
+                return True
+        else:
+            # Supported database version
+            return True
+
     def get_file(self, path, *args, **kwargs):
         tpath = self.munge_path(path)
         extension =  os.path.splitext(tpath)[1]
@@ -706,6 +740,9 @@ class KOBO(USBMS):
             # debug_print('    Commit: Set FavouritesIndex')
 
     def update_device_database_collections(self, booklists, collections_attributes, oncard):
+        if self.modify_database_check("update_device_database_collections") == False:
+            return 
+
         # Only process categories in this list
         supportedcategories = {
             "Im_Reading":1,

src/calibre/devices/udisks.py

@@ -159,16 +159,29 @@ def get_udisks(ver=None):
         return u
     return UDisks2() if ver == 2 else UDisks()
 
+def get_udisks1():
+    u = None
+    try:
+        u = UDisks()
+    except NoUDisks1:
+        try:
+            u = UDisks2()
+        except NoUDisks2:
+            pass
+    if u is None:
+        raise EnvironmentError('UDisks not available on your system')
+    return u
+
 def mount(node_path):
-    u = UDisks()
+    u = get_udisks1()
     u.mount(node_path)
 
 def eject(node_path):
-    u = UDisks()
+    u = get_udisks1()
     u.eject(node_path)
 
 def umount(node_path):
-    u = UDisks()
+    u = get_udisks1()
     u.unmount(node_path)
 
 def test_udisks(ver=None):

src/calibre/devices/usbms/device.py

@@ -127,12 +127,13 @@ class Device(DeviceConfig, DevicePlugin):
         if not prefix:
             return 0, 0
         prefix = prefix[:-1]
-        import win32file
+        import win32file, winerror
         try:
             sectors_per_cluster, bytes_per_sector, free_clusters, total_clusters = \
                 win32file.GetDiskFreeSpace(prefix)
         except Exception as err:
-            if getattr(err, 'args', [None])[0] == 21: # Disk not ready
+            if getattr(err, 'args', [None])[0] == winerror.ERROR_NOT_READY:
+                # Disk not ready
                 time.sleep(3)
                 sectors_per_cluster, bytes_per_sector, free_clusters, total_clusters = \
                     win32file.GetDiskFreeSpace(prefix)

src/calibre/devices/user_defined/driver.py

@@ -66,6 +66,9 @@ class USER_DEFINED(USBMS):
         _('Card A folder') + ':::<p>' +
             _('Enter the folder where the books are to be stored. This folder '
               'is prepended to any send_to_device template') + '</p>',
+        _('Swap main and card A') + ':::<p>' +
+            _('Check this box if the device\'s main memory is being seen as '
+              'card a and the card is being seen as main memory') + '</p>',
     ]
     EXTRA_CUSTOMIZATION_DEFAULT = [
                 '0xffff',
@@ -78,16 +81,19 @@ class USER_DEFINED(USBMS):
                 '',
                 '',
                 '',
+                False,
     ]
     OPT_USB_VENDOR_ID           = 0
     OPT_USB_PRODUCT_ID          = 1
     OPT_USB_REVISION_ID         = 2
+    # OPT 3 isn't used
     OPT_USB_WINDOWS_MM_VEN_ID   = 4
     OPT_USB_WINDOWS_MM_ID       = 5
     OPT_USB_WINDOWS_CA_VEN_ID   = 6
     OPT_USB_WINDOWS_CA_ID       = 7
     OPT_MAIN_MEM_FOLDER         = 8
     OPT_CARD_A_FOLDER           = 9
+    OPT_SWAP_MAIN_AND_CARD      = 10
 
     def initialize(self):
         self.plugin_needs_delayed_initialization = True
@@ -113,4 +119,41 @@ class USER_DEFINED(USBMS):
             traceback.print_exc()
         self.plugin_needs_delayed_initialization = False
 
+    def windows_sort_drives(self, drives):
+        if len(drives) < 2: return drives
+        e = self.settings().extra_customization
+        if not e[self.OPT_SWAP_MAIN_AND_CARD]:
+            return drives
+        main = drives.get('main', None)
+        carda = drives.get('carda', None)
+        if main and carda:
+            drives['main'] = carda
+            drives['carda'] = main
+        return drives
+
+    def linux_swap_drives(self, drives):
+        if len(drives) < 2 or not drives[1] or not drives[2]: return drives
+        e = self.settings().extra_customization
+        if not e[self.OPT_SWAP_MAIN_AND_CARD]:
+            return drives
+        drives = list(drives)
+        t = drives[0]
+        drives[0] = drives[1]
+        drives[1] = t
+        return tuple(drives)
+
+    def osx_sort_names(self, names):
+        if len(names) < 2: return names
+        e = self.settings().extra_customization
+        if not e[self.OPT_SWAP_MAIN_AND_CARD]:
+            return names
+        main = names.get('main', None)
+        card = names.get('carda', None)
+
+        if main is not None and card is not None:
+            names['main'] = card
+            names['carda'] = main
+
+        return names
+
 

src/calibre/ebooks/chardet.py

@@ -10,10 +10,15 @@ __docformat__ = 'restructuredtext en'
 import re, codecs
 
 ENCODING_PATS = [
+                # XML declaration
                  re.compile(r'<\?[^<>]+encoding\s*=\s*[\'"](.*?)[\'"][^<>]*>',
                             re.IGNORECASE),
+                 # HTML 4 Pragma directive
                  re.compile(r'''<meta\s+?[^<>]*?content\s*=\s*['"][^'"]*?charset=([-_a-z0-9]+)[^'"]*?['"][^<>]*>''',
                             re.IGNORECASE),
+                 # HTML 5 charset
+                 re.compile(r'''<meta\s+charset=['"]([-_a-z0-9]+)['"][^<>]*>''',
+                     re.IGNORECASE),
                  ]
 ENTITY_PATTERN = re.compile(r'&(\S+?);')
 

src/calibre/ebooks/conversion/plugins/htmlz_output.py

@@ -37,6 +37,11 @@ class HTMLZOutput(OutputFormatPlugin):
                    'external: Use an external CSS file that is linked in the document.\n'
                    'inline: Place the CSS in the head section of the document.'
             )),
+        OptionRecommendation(name='htmlz_title_filename',
+            recommended_value=False, level=OptionRecommendation.LOW,
+            help=_('If set this option causes the file name of the html file'
+                ' inside the htmlz archive to be based on the book title.')
+            ),
     ])
 
     def convert(self, oeb_book, output_path, input_plugin, opts, log):
@@ -44,6 +49,7 @@ class HTMLZOutput(OutputFormatPlugin):
         from calibre.ebooks.oeb.base import OEB_IMAGES, SVG_MIME
         from calibre.ebooks.metadata.opf2 import OPF, metadata_to_opf
         from calibre.utils.zipfile import ZipFile
+        from calibre.utils.filenames import ascii_filename
 
         # HTML
         if opts.htmlz_css_type == 'inline':
@@ -59,7 +65,10 @@ class HTMLZOutput(OutputFormatPlugin):
             htmlizer = OEB2HTMLizer(log)
             html = htmlizer.oeb2html(oeb_book, opts)
 
-            with open(os.path.join(tdir, u'index.html'), 'wb') as tf:
+            fname = u'index'
+            if opts.htmlz_title_filename:
+                fname = ascii_filename(unicode(oeb_book.metadata.title[0]))
+            with open(os.path.join(tdir, fname+u'.html'), 'wb') as tf:
                 tf.write(html)
 
             # CSS

src/calibre/ebooks/metadata/sources/ozon.py

@@ -54,30 +54,35 @@ class Ozon(Source):
 
         # for ozon.ru search we have to format ISBN with '-'
         isbn = _format_isbn(log, identifiers.get('isbn', None))
-        # TODO: format isbn!
+        ozonid = identifiers.get('ozon', None)
-        qItems = set([isbn, title])
+        
-        if authors:
+        unk = unicode(_('Unknown')).upper()
-            qItems |= frozenset(authors)
+        if (title and title != unk) or (authors and authors != [unk]) or isbn or not ozonid:
-        qItems.discard(None)
+            qItems = set([isbn, title])
-        qItems.discard('')
+            if authors:
-        qItems = map(_quoteString, qItems)
+                qItems |= frozenset(authors)
-
+            qItems.discard(None)
-        q = u' '.join(qItems).strip()
+            qItems.discard('')
-        log.info(u'search string: ' + q)
+            qItems = map(_quoteString, qItems)
-
+    
-        if isinstance(q, unicode):
+            q = u' '.join(qItems).strip()
-            q = q.encode('utf-8')
+            log.info(u'search string: ' + q)
-        if not q:
+    
-            return None
+            if isinstance(q, unicode):
-
+                q = q.encode('utf-8')
-        search_url += quote_plus(q)
+            if not q:
+                return None
+    
+            search_url += quote_plus(q)
+        else:
+            search_url = self.ozon_url + '/webservices/OzonWebSvc.asmx/ItemDetail?ID=%s' % ozonid
+            
         log.debug(u'search url: %r'%search_url)
-
         return search_url
     # }}}
 
     def identify(self, log, result_queue, abort, title=None, authors=None,
-            identifiers={}, timeout=30): # {{{
+            identifiers={}, timeout=60): # {{{
         from lxml import etree
         from calibre.ebooks.chardet import xml_to_unicode
 
@@ -99,7 +104,7 @@ class Ozon(Source):
         try:
             parser = etree.XMLParser(recover=True, no_network=True)
             feed = etree.fromstring(xml_to_unicode(raw, strip_encoding_pats=True, assume_utf8=True)[0], parser=parser)
-            entries = feed.xpath('//*[local-name() = "SearchItems"]')
+            entries = feed.xpath('//*[local-name()="SearchItems" or local-name()="ItemDetail"]')
             if entries:
                 metadata = self.get_metadata(log, entries, title, authors, identifiers)
                 self.get_all_details(log, metadata, abort, result_queue, identifiers, timeout)
@@ -112,8 +117,8 @@ class Ozon(Source):
     def get_metadata(self, log, entries, title, authors, identifiers): # {{{
         # some book titles have extra characters like this
         # TODO: make a twick
-        reRemoveFromTitle = None
+        #reRemoveFromTitle = None
-        #reRemoveFromTitle = re.compile(r'[?!:.,;+-/&%"\'=]')
+        reRemoveFromTitle = re.compile(r'[?!:.,;+-/&%"\'=]')
 
         title = unicode(title).upper() if title else ''
         if reRemoveFromTitle:
@@ -163,7 +168,7 @@ class Ozon(Source):
                 metadata.append(mi)
                 #log.debug(u'added metadata %s %s.'%(mi.title,  mi.authors))
             else:
-                log.debug(u'skipped metadata %s %s. (does not match the query)'%(mi.title, mi.authors))
+                log.debug(u'skipped metadata %s %s. (does not match the query)'%(unicode(mi.title), mi.authors))
         return metadata
     # }}}
 
@@ -301,7 +306,7 @@ class Ozon(Source):
         if series:
             metadata.series = series
 
-        xpt = u'normalize-space(substring-after(//meta[@name="description"]/@content, "ISBN"))'
+        xpt = u'normalize-space(//*[@class="product-detail"]//text()[starts-with(., "ISBN")])'
         isbn_str = doc.xpath(xpt)
         if isbn_str:
             all_isbns = [check_isbn(isbn) for isbn in self.isbnRegex.findall(isbn_str) if _verifyISBNIntegrity(log, isbn)]
@@ -326,7 +331,7 @@ class Ozon(Source):
 
         # can be set before from xml search responce
         if not metadata.pubdate:
-            xpt = u'normalize-space(//div[@class="product-misc"]//text()[contains(., "г.")])'
+            xpt = u'normalize-space(substring-after(//div[@class="product-detail"]//text()[contains(., "г.")],";"))'
             yearIn = doc.xpath(xpt)
             if yearIn:
                 matcher = re.search(r'\d{4}', yearIn)
@@ -334,17 +339,20 @@ class Ozon(Source):
                     metadata.pubdate = toPubdate(log, matcher.group(0))
 
         # overwrite comments from HTML if any
-        xpt = u'//table[@id="detail_description"]//tr/td'
+        xpt = u'//*[@id="detail_description"]//*[contains(text(), "От производителя")]/../node()[not(self::comment())][not(self::br)][preceding::*[contains(text(), "От производителя")]]'
+        from lxml.etree import ElementBase
         comment_elem = doc.xpath(xpt)
         if comment_elem:
-            comments = unicode(etree.tostring(comment_elem[0], encoding=unicode))
+            comments = u''
-            if comments:
+            for node in comment_elem:
-                # cleanup root tag, TODO: remove tags like object/embeded
+                if isinstance(node, ElementBase):
-                comments = re.sub(ur'\A.*?<td.*?>|</td>.*\Z', u'', comments.strip(), re.MULTILINE).strip()
+                    comments += unicode(etree.tostring(node, encoding=unicode))
-                if comments and (not metadata.comments or len(comments) > len(metadata.comments)):
+                elif isinstance(node, basestring) and node.strip():
-                    metadata.comments = comments
+                    comments += unicode(node) + u'\n'
-                else:
+            if comments and (not metadata.comments or len(comments) > len(metadata.comments)):
-                    log.debug('HTML book description skipped in favour of search service xml responce')
+                metadata.comments = comments
+            else:
+                log.debug('HTML book description skipped in favour of search service xml responce')
         else:
             log.debug('No book description found in HTML')
     # }}}
@@ -430,7 +438,8 @@ def _translageLanguageToCode(displayLang): # {{{
                 u'Китайский': 'zh',
                 u'Японский': 'ja',
                 u'Финский' : 'fi',
-                u'Польский' : 'pl',}
+                u'Польский' : 'pl',
+                u'Украинский' : 'uk',}
     return langTbl.get(displayLang, None)
 # }}}
 
@@ -454,7 +463,7 @@ def toPubdate(log, yearAsString): # {{{
     res = None
     if yearAsString:
         try:
-            res = parse_only_date(yearAsString)
+            res = parse_only_date(u"01.01." + yearAsString)
         except:
             log.error('cannot parse to date %s'%yearAsString)
     return res

src/calibre/ebooks/mobi/reader/markup.py

@@ -9,6 +9,8 @@ __docformat__ = 'restructuredtext en'
 
 import re, os
 
+from calibre.ebooks.chardet import strip_encoding_declarations
+
 def update_internal_links(mobi8_reader):
     # need to update all links that are internal which
     # are based on positions within the xhtml files **BEFORE**
@@ -324,6 +326,8 @@ def expand_mobi8_markup(mobi8_reader, resource_map, log):
     for i, part in enumerate(parts):
         pi = mobi8_reader.partinfo[i]
         with open(os.path.join(pi.type, pi.filename), 'wb') as f:
+            part = strip_encoding_declarations(part)
+            part = part.replace('<head>', '<head><meta charset="UTF-8"/>', 1)
             f.write(part.encode('utf-8'))
             spine.append(f.name)
 

src/calibre/ebooks/oeb/display/paged.coffee

@@ -6,48 +6,7 @@
  Released under the GPLv3 License
 ###
 
-log = (args...) -> # {{{
+log = window.calibre_utils.log
-    if args
-        msg = args.join(' ')
-        if window?.console?.log
-            window.console.log(msg)
-        else if process?.stdout?.write
-            process.stdout.write(msg + '\n')
-# }}}
-
-window_scroll_pos = (win=window) -> # {{{
-    if typeof(win.pageXOffset) == 'number'
-        x = win.pageXOffset
-        y = win.pageYOffset
-    else # IE < 9
-        if document.body and ( document.body.scrollLeft or document.body.scrollTop )
-            x = document.body.scrollLeft
-            y = document.body.scrollTop
-        else if document.documentElement and ( document.documentElement.scrollLeft or document.documentElement.scrollTop)
-            y = document.documentElement.scrollTop
-            x = document.documentElement.scrollLeft
-    return [x, y]
-# }}}
-
-viewport_to_document = (x, y, doc=window?.document) -> # {{{
-    until doc == window.document
-        # We are in a frame
-        frame = doc.defaultView.frameElement
-        rect = frame.getBoundingClientRect()
-        x += rect.left
-        y += rect.top
-        doc = frame.ownerDocument
-    win = doc.defaultView
-    [wx, wy] = window_scroll_pos(win)
-    x += wx
-    y += wy
-    return [x, y]
-# }}}
-
-absleft = (elem) -> # {{{
-    r = elem.getBoundingClientRect()
-    return viewport_to_document(r.left, 0, elem.ownerDocument)[0]
-# }}}
 
 class PagedDisplay
     # This class is a namespace to expose functions via the
@@ -75,6 +34,7 @@ class PagedDisplay
         this.cols_per_screen = cols_per_screen
 
     layout: () ->
+        # start_time = new Date().getTime()
         body_style = window.getComputedStyle(document.body)
         # When laying body out in columns, webkit bleeds the top margin of the
         # first block element out above the columns, leading to an extra top
@@ -160,8 +120,30 @@ class PagedDisplay
 
         this.in_paged_mode = true
         this.current_margin_side = sm
+        # log('Time to layout:', new Date().getTime() - start_time)
         return sm
 
+    fit_images: () ->
+        # Ensure no images are wider than the available width in a column. Note
+        # that this method use getBoundingClientRect() which means it will
+        # force a relayout if the render tree is dirty.
+        images = []
+        for img in document.getElementsByTagName('img')
+            previously_limited = calibre_utils.retrieve(img, 'width-limited', false)
+            br = img.getBoundingClientRect()
+            left = calibre_utils.viewport_to_document(br.left, 0, doc=img.ownerDocument)[0]
+            col = this.column_at(left) * this.page_width
+            rleft = left - col - this.current_margin_side
+            width  = br.right - br.left
+            rright = rleft + width
+            col_width = this.page_width - 2*this.current_margin_side
+            if previously_limited or rright > col_width
+                images.push([img, col_width - rleft])
+
+        for [img, max_width] in images
+            img.style.setProperty('max-width', max_width+'px')
+            calibre_utils.store(img, 'width-limited', true)
+
     scroll_to_pos: (frac) ->
         # Scroll to the position represented by frac (number between 0 and 1)
         xpos = Math.floor(document.body.scrollWidth * frac)
@@ -297,7 +279,7 @@ class PagedDisplay
         elem.scrollIntoView()
         if this.in_paged_mode
             # Ensure we are scrolled to the column containing elem
-            this.scroll_to_xpos(absleft(elem) + 5)
+            this.scroll_to_xpos(calibre_utils.absleft(elem) + 5)
 
     snap_to_selection: () ->
         # Ensure that the viewport is positioned at the start of the column
@@ -306,7 +288,7 @@ class PagedDisplay
             sel = window.getSelection()
             r = sel.getRangeAt(0).getBoundingClientRect()
             node = sel.anchorNode
-            left = viewport_to_document(r.left, r.top, doc=node.ownerDocument)[0]
+            left = calibre_utils.viewport_to_document(r.left, r.top, doc=node.ownerDocument)[0]
 
             # Ensure we are scrolled to the column containing the start of the
             # selection
@@ -365,5 +347,5 @@ if window?
     window.paged_display = new PagedDisplay()
 
 # TODO:
-# Resizing of images
 # Highlight on jump_to_anchor
+# Handle document specified margins and allow them to be overridden

src/calibre/ebooks/oeb/display/utils.coffee

@@ -0,0 +1,96 @@
+#!/usr/bin/env coffee
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+
+###
+ Copyright 2012, Kovid Goyal <kovid@kovidgoyal.net>
+ Released under the GPLv3 License
+###
+
+class CalibreUtils
+    # This class is a namespace to expose functions via the
+    # window.calibre_utils object.
+
+    constructor: () ->
+        if not this instanceof arguments.callee
+            throw new Error('CalibreUtils constructor called as function')
+        this.dom_attr = 'calibre_f3fa75ca98eb4413a4ee413f20f60226'
+        this.dom_data = []
+
+    # Data API {{{
+
+    retrieve: (node, key, def=null) ->
+        # Retrieve data previously stored on node (a DOM node) with key (a
+        # string). If no such data is found then return the value of def.
+        idx = parseInt(node.getAttribute(this.dom_attr))
+        if isNaN(idx)
+            return def
+        data = this.dom_data[idx]
+        if not data.hasOwnProperty(key)
+            return def
+        return data[key]
+
+    store: (node, key, val) ->
+        # Store arbitrary javscript object val on DOM node node with key (a
+        # string). This can be later retrieved by the retrieve method.
+        idx = parseInt(node.getAttribute(this.dom_attr))
+        if isNaN(idx)
+            idx = this.dom_data.length
+            node.setAttribute(this.dom_attr, idx+'')
+            this.dom_data.push({})
+        this.dom_data[idx][key] = val
+    # }}}
+
+    log: (args...) -> # {{{
+        # Output args to the window.console object. args are automatically
+        # coerced to strings
+        if args
+            msg = args.join(' ')
+            if window?.console?.log
+                window.console.log(msg)
+            else if process?.stdout?.write
+                process.stdout.write(msg + '\n')
+    # }}}
+
+    window_scroll_pos: (win=window) -> # {{{
+        # The current scroll position of the browser window
+        if typeof(win.pageXOffset) == 'number'
+            x = win.pageXOffset
+            y = win.pageYOffset
+        else # IE < 9
+            if document.body and ( document.body.scrollLeft or document.body.scrollTop )
+                x = document.body.scrollLeft
+                y = document.body.scrollTop
+            else if document.documentElement and ( document.documentElement.scrollLeft or document.documentElement.scrollTop)
+                y = document.documentElement.scrollTop
+                x = document.documentElement.scrollLeft
+        return [x, y]
+    # }}}
+
+    viewport_to_document: (x, y, doc=window?.document) -> # {{{
+        # Convert x, y from the viewport (window) co-ordinate system to the
+        # document (body) co-ordinate system
+        until doc == window.document
+            # We are in a frame
+            frame = doc.defaultView.frameElement
+            rect = frame.getBoundingClientRect()
+            x += rect.left
+            y += rect.top
+            doc = frame.ownerDocument
+        win = doc.defaultView
+        [wx, wy] = this.window_scroll_pos(win)
+        x += wx
+        y += wy
+        return [x, y]
+    # }}}
+
+    absleft: (elem) -> # {{{
+        # The left edge of elem in document co-ords. Works in all
+        # circumstances, including column layout. Note that this will cause
+        # a relayout if the render tree is dirty.
+        r = elem.getBoundingClientRect()
+        return this.viewport_to_document(r.left, 0, elem.ownerDocument)[0]
+    # }}}
+
+if window?
+    window.calibre_utils = new CalibreUtils()
+

src/calibre/ebooks/pdf/writer.py

@@ -117,7 +117,6 @@ class PDFMetadata(object):
             if len(oeb_metadata.creator) >= 1:
                 self.author = authors_to_string([x.value for x in oeb_metadata.creator])
 
-
 class PDFWriter(QObject): # {{{
 
     def __init__(self, opts, log, cover_data=None):
@@ -185,8 +184,8 @@ class PDFWriter(QObject): # {{{
         from PyQt4.Qt import QSize, QPainter
         if self.paged_js is None:
             from calibre.utils.resources import compiled_coffeescript
-            self.paged_js = compiled_coffeescript('ebooks.oeb.display.paged',
+            self.paged_js = compiled_coffeescript('ebooks.oeb.display.utils')
-                    dynamic=False)
+            self.paged_js += compiled_coffeescript('ebooks.oeb.display.paged')
         printer = get_pdf_printer(self.opts, output_file_name=outpath)
         painter = QPainter(printer)
         zoomx = printer.logicalDpiX()/self.view.logicalDpiX()
@@ -202,6 +201,7 @@ class PDFWriter(QObject): # {{{
         document.body.style.backgroundColor = "white";
         paged_display.set_geometry(1, 0, 0, 0);
         paged_display.layout();
+        paged_display.fit_images();
         ''')
         mf = self.view.page().mainFrame()
         while True:

src/calibre/gui2/__init__.py

@@ -89,14 +89,6 @@ gprefs.defaults['tags_browser_partition_method'] = 'first letter'
 gprefs.defaults['tags_browser_collapse_at'] = 100
 gprefs.defaults['tag_browser_dont_collapse'] = []
 gprefs.defaults['edit_metadata_single_layout'] = 'default'
-gprefs.defaults['book_display_fields'] = [
-        ('title', False), ('authors', True), ('formats', True),
-        ('series', True), ('identifiers', True), ('tags', True),
-        ('path', True), ('publisher', False), ('rating', False),
-        ('author_sort', False), ('sort', False), ('timestamp', False),
-        ('uuid', False), ('comments', True), ('id', False), ('pubdate', False),
-        ('last_modified', False), ('size', False), ('languages', False),
-        ]
 gprefs.defaults['default_author_link'] = 'http://en.wikipedia.org/w/index.php?search={author}'
 gprefs.defaults['preserve_date_on_ctl'] = True
 gprefs.defaults['cb_fullscreen'] = False
@@ -581,30 +573,31 @@ class FileDialog(QObject):
         if not isinstance(initial_dir, basestring):
             initial_dir = os.path.expanduser(default_dir)
         self.selected_files = []
-        if mode == QFileDialog.AnyFile:
+        with SanitizeLibraryPath():
-            f = unicode(QFileDialog.getSaveFileName(parent, title, initial_dir, ftext, ""))
+            if mode == QFileDialog.AnyFile:
-            if f:
+                f = unicode(QFileDialog.getSaveFileName(parent, title, initial_dir, ftext, ""))
-                self.selected_files.append(f)
+                if f:
-        elif mode == QFileDialog.ExistingFile:
+                    self.selected_files.append(f)
-            f = unicode(QFileDialog.getOpenFileName(parent, title, initial_dir, ftext, ""))
+            elif mode == QFileDialog.ExistingFile:
-            if f and os.path.exists(f):
+                f = unicode(QFileDialog.getOpenFileName(parent, title, initial_dir, ftext, ""))
-                self.selected_files.append(f)
-        elif mode == QFileDialog.ExistingFiles:
-            fs = QFileDialog.getOpenFileNames(parent, title, initial_dir, ftext, "")
-            for f in fs:
-                f = unicode(f)
-                if not f: continue
-                if not os.path.exists(f):
-                    # QFileDialog for some reason quotes spaces
-                    # on linux if there is more than one space in a row
-                    f = unquote(f)
                 if f and os.path.exists(f):
                     self.selected_files.append(f)
-        else:
+            elif mode == QFileDialog.ExistingFiles:
-            opts = QFileDialog.ShowDirsOnly if mode == QFileDialog.Directory else QFileDialog.Option()
+                fs = QFileDialog.getOpenFileNames(parent, title, initial_dir, ftext, "")
-            f = unicode(QFileDialog.getExistingDirectory(parent, title, initial_dir, opts))
+                for f in fs:
-            if os.path.exists(f):
+                    f = unicode(f)
-                self.selected_files.append(f)
+                    if not f: continue
+                    if not os.path.exists(f):
+                        # QFileDialog for some reason quotes spaces
+                        # on linux if there is more than one space in a row
+                        f = unquote(f)
+                    if f and os.path.exists(f):
+                        self.selected_files.append(f)
+            else:
+                opts = QFileDialog.ShowDirsOnly if mode == QFileDialog.Directory else QFileDialog.Option()
+                f = unicode(QFileDialog.getExistingDirectory(parent, title, initial_dir, opts))
+                if os.path.exists(f):
+                    self.selected_files.append(f)
         if self.selected_files:
             self.selected_files = [unicode(q) for q in self.selected_files]
             saved_loc = self.selected_files[0]
@@ -734,8 +727,11 @@ gui_thread = None
 qt_app = None
 class Application(QApplication):
 
-    def __init__(self, args, force_calibre_style=False):
+    def __init__(self, args, force_calibre_style=False,
+            override_program_name=None):
         self.file_event_hook = None
+        if override_program_name:
+            args = [override_program_name] + args[1:]
         qargs = [i.encode('utf-8') if isinstance(i, unicode) else i for i in args]
         QApplication.__init__(self, qargs)
         global gui_thread, qt_app
@@ -854,16 +850,26 @@ class Application(QApplication):
 
 _store_app = None
 
+class SanitizeLibraryPath(object):
+    '''Remove the bundled calibre libraries from LD_LIBRARY_PATH on linux. This
+    is needed to prevent library conflicts when launching external utilities.'''
+
+    def __enter__(self):
+        self.orig = os.environ.get('LD_LIBRARY_PATH', '')
+        self.changed = False
+        paths = [x for x in self.orig.split(os.pathsep) if x]
+        if isfrozen and islinux and paths:
+            npaths = [x for x in paths if x != sys.frozen_path+'/lib']
+            os.environ['LD_LIBRARY_PATH'] = os.pathsep.join(npaths)
+            self.changed = True
+
+    def __exit__(self, *args):
+        if self.changed:
+            os.environ['LD_LIBRARY_PATH'] = self.orig
+
 def open_url(qurl):
-    paths = os.environ.get('LD_LIBRARY_PATH',
+    with SanitizeLibraryPath():
-                '').split(os.pathsep)
+        QDesktopServices.openUrl(qurl)
-    paths = [x for x in paths if x]
-    if isfrozen and islinux and paths:
-        npaths = [x for x in paths if x != sys.frozen_path+'/lib']
-        os.environ['LD_LIBRARY_PATH'] = os.pathsep.join(npaths)
-    QDesktopServices.openUrl(qurl)
-    if isfrozen and islinux and paths:
-        os.environ['LD_LIBRARY_PATH'] = os.pathsep.join(paths)
 
 def get_current_db():
     '''

src/calibre/gui2/actions/copy_to_library.py

@@ -216,7 +216,8 @@ class CopyToLibraryAction(InterfaceAction):
                 if ci.isValid():
                     row = ci.row()
 
-                v.model().delete_books_by_id(self.worker.processed)
+                v.model().delete_books_by_id(self.worker.processed,
+                        permanent=True)
                 self.gui.iactions['Remove Books'].library_ids_deleted(
                         self.worker.processed, row)
 

src/calibre/gui2/book_details.py

@@ -84,7 +84,17 @@ def render_html(mi, css, vertical, widget, all_fields=False): # {{{
     return ans
 
 def get_field_list(fm, use_defaults=False):
-    src = gprefs.defaults if use_defaults else gprefs
+    from calibre.gui2.ui import get_gui
+    db = get_gui().current_db
+    if use_defaults:
+        src = db.prefs.defaults
+    else:
+        old_val = gprefs.get('book_display_fields', None)
+        if old_val is not None and not db.prefs.has_setting(
+                'book_display_fields'):
+            src = gprefs
+        else:
+            src = db.prefs
     fieldlist = list(src['book_display_fields'])
     names = frozenset([x[0] for x in fieldlist])
     for field in fm.displayable_field_keys():
@@ -104,8 +114,11 @@ def render_data(mi, use_roman_numbers=True, all_fields=False):
             field = 'title_sort'
         if all_fields:
             display = True
-        if (not display or not metadata or mi.is_null(field) or
+        if metadata['datatype'] == 'bool':
-                field == 'comments'):
+            isnull = mi.get(field) is None
+        else:
+            isnull = mi.is_null(field)
+        if (not display or not metadata or isnull or field == 'comments'):
             continue
         name = metadata['name']
         if not name:

src/calibre/gui2/complete.py

@@ -5,11 +5,15 @@ __license__   = 'GPL v3'
 __copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
+'''
+WARNING: The code in this module is deprecated. Use complete2.py instead. This
+code remains here for legacy plugin support.
+'''
 
 from PyQt4.Qt import (QLineEdit, QAbstractListModel, Qt,
-        QApplication, QCompleter, pyqtSignal)
+        QApplication, QCompleter)
 
-from calibre.utils.icu import sort_key, lower
+from calibre.utils.icu import sort_key
 from calibre.gui2 import NONE
 from calibre.gui2.widgets import EnComboBox, LineEditECM
 from calibre.utils.config_base import tweaks
@@ -24,12 +28,11 @@ class CompleteModel(QAbstractListModel):
     def set_items(self, items):
         items = [unicode(x.strip()) for x in items]
         if len(items) < tweaks['completion_change_to_ascii_sorting']:
-            self.items = sorted(items, key=lambda x: sort_key(x))
+            self.items = sorted(items, key=sort_key)
             self.sorting = QCompleter.UnsortedModel
         else:
             self.items = sorted(items, key=lambda x:x.lower())
             self.sorting = QCompleter.CaseInsensitivelySortedModel
-        self.lowered_items = [lower(x) for x in self.items]
         self.reset()
 
     def rowCount(self, *args):
@@ -56,7 +59,7 @@ class MultiCompleteLineEdit(QLineEdit, LineEditECM):
     to complete non multiple fields as well.
     '''
 
-    def __init__(self, parent=None):
+    def __init__(self, parent=None, completer_widget=None):
         QLineEdit.__init__(self, parent)
 
         self.sep = ','
@@ -66,7 +69,7 @@ class MultiCompleteLineEdit(QLineEdit, LineEditECM):
 
         self._model = CompleteModel(parent=self)
         self._completer = c = QCompleter(self._model, self)
-        c.setWidget(self)
+        c.setWidget(self if completer_widget is None else completer_widget)
         c.setCompletionMode(QCompleter.PopupCompletion)
         c.setCaseSensitivity(Qt.CaseInsensitive)
         c.setModelSorting(self._model.sorting)
@@ -158,21 +161,25 @@ class MultiCompleteLineEdit(QLineEdit, LineEditECM):
 
 class MultiCompleteComboBox(EnComboBox):
 
-    clear_edit_text = pyqtSignal()
-
     def __init__(self, *args):
         EnComboBox.__init__(self, *args)
-        self.setLineEdit(MultiCompleteLineEdit(self))
+        self.le = MultiCompleteLineEdit(self, completer_widget=self)
-        # Needed to allow changing the case of an existing item
+        self.setLineEdit(self.le)
-        # otherwise on focus out, the text is changed to the
+
-        # item that matches case insensitively
+    def showPopup(self):
-        c = self.lineEdit().completer()
+        c = self.le._completer
-        c.setCaseSensitivity(Qt.CaseSensitive)
+        v = c.currentCompletion()
-        self.dummy_model = CompleteModel(self)
+        c.setCompletionPrefix('')
-        c.setModel(self.dummy_model)
+        c.complete()
-        self.lineEdit()._completer.setWidget(self)
+        cs = c.caseSensitivity()
-        self.clear_edit_text.connect(self.clearEditText,
+        i = 0
-                type=Qt.QueuedConnection)
+        while c.setCurrentRow(i):
+            cr = c.currentIndex().data().toString()
+            if cr.startsWith(v, cs):
+                c.popup().setCurrentIndex(c.currentIndex())
+                return
+            i += 1
+        c.setCurrentRow(0)
 
     def update_items_cache(self, complete_items):
         self.lineEdit().update_items_cache(complete_items)
@@ -187,18 +194,10 @@ class MultiCompleteComboBox(EnComboBox):
         self.lineEdit().set_add_separator(what)
 
     def show_initial_value(self, what):
-        '''
+        what = unicode(what) if what else u''
-        Show an initial value. Handle the case of the initial value being blank
-        correctly (on Qt 4.8.0 having a blank value causes the first value from
-        the completer to be shown, when the event loop runs).
-        '''
-        what = unicode(what)
         le = self.lineEdit()
-        if not what.strip():
+        self.setEditText(what)
-            self.clear_edit_text.emit()
+        le.selectAll()
-        else:
-            self.setEditText(what)
-            le.selectAll()
 
 if __name__ == '__main__':
     from PyQt4.Qt import QDialog, QVBoxLayout
@@ -207,5 +206,8 @@ if __name__ == '__main__':
     d.setLayout(QVBoxLayout())
     le = MultiCompleteComboBox(d)
     d.layout().addWidget(le)
-    le.all_items = ['one', 'otwo', 'othree', 'ooone', 'ootwo', 'oothree']
+    items = ['one', 'otwo', 'othree', 'ooone', 'ootwo',
+        'oothree']
+    le.update_items_cache(items)
+    le.show_initial_value('')
     d.exec_()

src/calibre/gui2/complete2.py

@@ -0,0 +1,414 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2012, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+import weakref
+
+import sip
+from PyQt4.Qt import (QLineEdit, QAbstractListModel, Qt, pyqtSignal, QObject,
+        QApplication, QListView, QPoint, QModelIndex)
+
+from calibre.utils.icu import sort_key, primary_startswith
+from calibre.gui2 import NONE
+from calibre.gui2.widgets import EnComboBox, LineEditECM
+
+class CompleteModel(QAbstractListModel): # {{{
+
+    def __init__(self, parent=None):
+        QAbstractListModel.__init__(self, parent)
+        self.all_items = self.current_items = ()
+        self.current_prefix = ''
+
+    def set_items(self, items):
+        items = [unicode(x.strip()) for x in items]
+        items = [x for x in items if x]
+        items = tuple(sorted(items, key=sort_key))
+        self.all_items = self.current_items = items
+        self.reset()
+
+    def set_completion_prefix(self, prefix):
+        old_prefix = self.current_prefix
+        self.current_prefix = prefix
+        if prefix == old_prefix:
+            return
+        if not prefix:
+            self.current_items = self.all_items
+            self.reset()
+            return
+        subset = prefix.startswith(old_prefix)
+        universe = self.current_items if subset else self.all_items
+        self.current_items = tuple(x for x in universe if primary_startswith(x,
+            prefix))
+        self.reset()
+
+    def rowCount(self, *args):
+        return len(self.current_items)
+
+    def data(self, index, role):
+        if role == Qt.DisplayRole:
+            try:
+                return self.current_items[index.row()]
+            except IndexError:
+                pass
+        return NONE
+
+    def index_for_prefix(self, prefix):
+        for i, item in enumerate(self.current_items):
+            if primary_startswith(item, prefix):
+                return self.index(i)
+# }}}
+
+class Completer(QListView): # {{{
+
+    item_selected = pyqtSignal(object)
+    relayout_needed = pyqtSignal()
+
+    def __init__(self, completer_widget, max_visible_items=7):
+        QListView.__init__(self)
+        self.completer_widget = weakref.ref(completer_widget)
+        self.setWindowFlags(Qt.Popup)
+        self.max_visible_items = max_visible_items
+        self.setEditTriggers(self.NoEditTriggers)
+        self.setHorizontalScrollBarPolicy(Qt.ScrollBarAlwaysOff)
+        self.setSelectionBehavior(self.SelectRows)
+        self.setSelectionMode(self.SingleSelection)
+        self.setAlternatingRowColors(True)
+        self.setModel(CompleteModel(self))
+        self.setMouseTracking(True)
+        self.entered.connect(self.item_entered)
+        self.activated.connect(self.item_chosen)
+        self.pressed.connect(self.item_chosen)
+        self.installEventFilter(self)
+
+    def hide(self):
+        self.setCurrentIndex(QModelIndex())
+        QListView.hide(self)
+
+    def item_chosen(self, index):
+        if not self.isVisible(): return
+        self.hide()
+        text = self.model().data(index, Qt.DisplayRole)
+        self.item_selected.emit(unicode(text))
+
+    def set_items(self, items):
+        self.model().set_items(items)
+        if self.isVisible():
+            self.relayout_needed.emit()
+
+    def set_completion_prefix(self, prefix):
+        self.model().set_completion_prefix(prefix)
+        if self.isVisible():
+            self.relayout_needed.emit()
+
+    def item_entered(self, idx):
+        self.setCurrentIndex(idx)
+
+    def next_match(self, previous=False):
+        c = self.currentIndex()
+        if c.isValid():
+            r = c.row()
+        else:
+            r = self.model().rowCount() if previous else -1
+        r = r + (-1 if previous else 1)
+        index = self.model().index(r % self.model().rowCount())
+        self.setCurrentIndex(index)
+
+    def scroll_to(self, orig):
+        if orig:
+            index = self.model().index_for_prefix(orig)
+            if index is not None and index.isValid():
+                self.setCurrentIndex(index)
+
+    def popup(self, select_first=True):
+        p = self
+        m = p.model()
+        widget = self.completer_widget()
+        if widget is None:
+            return
+        screen = QApplication.desktop().availableGeometry(widget)
+        h = (p.sizeHintForRow(0) * min(self.max_visible_items, m.rowCount()) +
+                3) + 3
+        hsb = p.horizontalScrollBar()
+        if hsb and hsb.isVisible():
+            h += hsb.sizeHint().height()
+
+        rh = widget.height()
+        pos = widget.mapToGlobal(QPoint(0, widget.height() - 2))
+        w = min(widget.width(), screen.width())
+
+        if (pos.x() + w) > (screen.x() + screen.width()):
+            pos.setX(screen.x() + screen.width() - w)
+        if pos.x() < screen.x():
+            pos.setX(screen.x())
+
+        top = pos.y() - rh - screen.top() + 2
+        bottom = screen.bottom() - pos.y()
+        h = max(h, p.minimumHeight())
+        if h > bottom:
+            h = min(max(top, bottom), h)
+
+            if top > bottom:
+                pos.setY(pos.y() - h - rh + 2)
+
+        p.setGeometry(pos.x(), pos.y(), w, h)
+
+        if (select_first and not self.currentIndex().isValid() and
+                self.model().rowCount() > 0):
+            self.setCurrentIndex(self.model().index(0))
+
+        if not p.isVisible():
+            p.show()
+
+    def eventFilter(self, obj, e):
+        'Redirect key presses from the popup to the widget'
+        widget = self.completer_widget()
+        if widget is None or sip.isdeleted(widget):
+            return False
+        etype = e.type()
+        if obj is not self:
+            return QObject.eventFilter(self, obj, e)
+
+        if etype == e.KeyPress:
+            key = e.key()
+            if key == Qt.Key_Escape:
+                self.hide()
+                e.accept()
+                return True
+            if key == Qt.Key_F4 and e.modifiers() & Qt.AltModifier:
+                self.hide()
+                e.accept()
+                return True
+            if key in (Qt.Key_Enter, Qt.Key_Return):
+                if not self.currentIndex().isValid():
+                    self.hide()
+                    e.accept()
+                    return True
+                return False
+            if key in (Qt.Key_End, Qt.Key_Home, Qt.Key_Up, Qt.Key_Down,
+                    Qt.Key_PageUp, Qt.Key_PageDown):
+                # Let the list view handle these keys
+                return False
+            if key in (Qt.Key_Tab, Qt.Key_Backtab):
+                self.next_match(previous=key == Qt.Key_Backtab)
+                e.accept()
+                return True
+            # Send to widget
+            widget.eat_focus_out = False
+            widget.keyPressEvent(e)
+            widget.eat_focus_out = True
+            if not widget.hasFocus():
+                # Widget lost focus hide the popup
+                self.hide()
+            if e.isAccepted():
+                return True
+        elif etype == e.MouseButtonPress:
+            if not self.underMouse():
+                self.hide()
+                e.accept()
+                return True
+        return False
+# }}}
+
+class LineEdit(QLineEdit, LineEditECM):
+    '''
+    A line edit that completes on multiple items separated by a
+    separator. Use the :meth:`update_items_cache` to set the list of
+    all possible completions. Separator can be controlled with the
+    :meth:`set_separator` and :meth:`set_space_before_sep` methods.
+
+    A call to self.set_separator(None) will allow this widget to be used
+    to complete non multiple fields as well.
+    '''
+
+    def __init__(self, parent=None, completer_widget=None):
+        QLineEdit.__init__(self, parent)
+
+        self.sep = ','
+        self.space_before_sep = False
+        self.add_separator = True
+        self.original_cursor_pos = None
+        completer_widget = (self if completer_widget is None else
+                completer_widget)
+
+        self.mcompleter = Completer(completer_widget)
+        self.mcompleter.item_selected.connect(self.completion_selected,
+                type=Qt.QueuedConnection)
+        self.mcompleter.relayout_needed.connect(self.relayout)
+        self.mcompleter.setFocusProxy(completer_widget)
+        self.textEdited.connect(self.text_edited)
+        self.no_popup = False
+
+    # Interface {{{
+    def update_items_cache(self, complete_items):
+        self.all_items = complete_items
+
+    def set_separator(self, sep):
+        self.sep = sep
+
+    def set_space_before_sep(self, space_before):
+        self.space_before_sep = space_before
+
+    def set_add_separator(self, what):
+        self.add_separator = bool(what)
+
+    @dynamic_property
+    def all_items(self):
+        def fget(self):
+            return self.mcompleter.model().all_items
+        def fset(self, items):
+            self.mcompleter.model().set_items(items)
+        return property(fget=fget, fset=fset)
+
+    # }}}
+
+    def complete(self, show_all=False, select_first=True):
+        orig = None
+        if show_all:
+            orig = self.mcompleter.model().current_prefix
+            self.mcompleter.set_completion_prefix('')
+        if not self.mcompleter.model().current_items:
+            self.mcompleter.hide()
+            return
+        self.mcompleter.popup(select_first=select_first)
+        self.mcompleter.scroll_to(orig)
+
+    def relayout(self):
+        self.mcompleter.popup()
+
+    def text_edited(self, *args):
+        if self.no_popup: return
+        self.update_completions()
+        self.complete(select_first=len(unicode(self.text()))>0)
+
+    def update_completions(self):
+        ' Update the list of completions '
+        self.original_cursor_pos = cpos = self.cursorPosition()
+        text = unicode(self.text())
+        prefix = text[:cpos]
+        complete_prefix = prefix.lstrip()
+        if self.sep:
+            complete_prefix = prefix.split(self.sep)[-1].lstrip()
+        self.mcompleter.set_completion_prefix(complete_prefix)
+
+    def get_completed_text(self, text):
+        'Get completed text in before and after parts'
+        if self.sep is None:
+            return text, ''
+        else:
+            cursor_pos = self.original_cursor_pos
+            if cursor_pos is None:
+                cursor_pos = self.cursorPosition()
+            self.original_cursor_pos = None
+            # Split text
+            curtext = unicode(self.text())
+            before_text = curtext[:cursor_pos]
+            after_text = curtext[cursor_pos:].rstrip()
+            # Remove the completion prefix from the before text
+            before_text = self.sep.join(before_text.split(self.sep)[:-1]).rstrip()
+            if before_text:
+                # Add the separator to the end of before_text
+                if self.space_before_sep:
+                    before_text += ' '
+                before_text += self.sep + ' '
+            if self.add_separator or after_text:
+                # Add separator to the end of completed text
+                if self.space_before_sep:
+                    text = text.rstrip() + ' '
+                completed_text = text + self.sep + ' '
+            else:
+                completed_text = text
+            return before_text + completed_text, after_text
+
+    def completion_selected(self, text):
+        before_text, after_text = self.get_completed_text(unicode(text))
+        self.setText(before_text + after_text)
+        self.setCursorPosition(len(before_text))
+
+class EditWithComplete(EnComboBox):
+
+    def __init__(self, *args):
+        EnComboBox.__init__(self, *args)
+        self.setLineEdit(LineEdit(self, completer_widget=self))
+        self.setCompleter(None)
+        self.eat_focus_out = True
+        self.installEventFilter(self)
+
+    # Interface {{{
+    def showPopup(self):
+        self.lineEdit().complete(show_all=True)
+
+    def update_items_cache(self, complete_items):
+        self.lineEdit().update_items_cache(complete_items)
+
+    def set_separator(self, sep):
+        self.lineEdit().set_separator(sep)
+
+    def set_space_before_sep(self, space_before):
+        self.lineEdit().set_space_before_sep(space_before)
+
+    def set_add_separator(self, what):
+        self.lineEdit().set_add_separator(what)
+
+    def show_initial_value(self, what):
+        what = unicode(what) if what else u''
+        le = self.lineEdit()
+        self.setEditText(what)
+        le.selectAll()
+
+    @dynamic_property
+    def all_items(self):
+        def fget(self): return self.lineEdit().all_items
+        def fset(self, val): self.lineEdit().all_items = val
+        return property(fget=fget, fset=fset)
+
+
+    # }}}
+
+    def text(self):
+        return unicode(self.lineEdit().text())
+
+    def setText(self, val):
+        le = self.lineEdit()
+        le.no_popup = True
+        le.setText(val)
+        le.no_popup = False
+
+    def setCursorPosition(self, *args):
+        self.lineEdit().setCursorPosition(*args)
+
+    @property
+    def textChanged(self):
+        return self.lineEdit().textChanged
+
+    def clear(self):
+        self.lineEdit().clear()
+        EnComboBox.clear(self)
+
+    def eventFilter(self, obj, e):
+        try:
+            c = self.lineEdit().mcompleter
+        except AttributeError:
+            return False
+        etype = e.type()
+        if self.eat_focus_out and self is obj and etype == e.FocusOut:
+            if c.isVisible():
+                return True
+        return EnComboBox.eventFilter(self, obj, e)
+
+if __name__ == '__main__':
+    from PyQt4.Qt import QDialog, QVBoxLayout
+    app = QApplication([])
+    d = QDialog()
+    d.setLayout(QVBoxLayout())
+    le = EditWithComplete(d)
+    d.layout().addWidget(le)
+    items = ['one', 'otwo', 'othree', 'ooone', 'ootwo',
+        'oothree', 'a1', 'a2',u'Edgas', u'Èdgar', u'Édgaq', u'Edgar', u'Édgar']
+    le.update_items_cache(items)
+    le.show_initial_value('')
+    d.exec_()

src/calibre/gui2/convert/htmlz_output.py

@@ -17,7 +17,8 @@ class PluginWidget(Widget, Ui_Form):
     ICON = I('mimetypes/html.png')
 
     def __init__(self, parent, get_option, get_help, db=None, book_id=None):
-        Widget.__init__(self, parent, ['htmlz_css_type', 'htmlz_class_style'])
+        Widget.__init__(self, parent, ['htmlz_css_type', 'htmlz_class_style',
+            'htmlz_title_filename'])
         self.db, self.book_id = db, book_id
         for x in get_option('htmlz_css_type').option.choices:
             self.opt_htmlz_css_type.addItem(x)

src/calibre/gui2/convert/htmlz_output.ui

@@ -14,7 +14,7 @@
    <string>Form</string>
   </property>
   <layout class="QGridLayout" name="gridLayout">
-   <item row="2" column="0">
+   <item row="3" column="0">
     <spacer name="verticalSpacer">
      <property name="orientation">
       <enum>Qt::Vertical</enum>
@@ -27,6 +27,9 @@
      </property>
     </spacer>
    </item>
+   <item row="1" column="1">
+    <widget class="QComboBox" name="opt_htmlz_class_style"/>
+   </item>
    <item row="0" column="0">
     <widget class="QLabel" name="label">
      <property name="text">
@@ -51,8 +54,12 @@
      </property>
     </widget>
    </item>
-   <item row="1" column="1">
+   <item row="2" column="0" colspan="2">
-    <widget class="QComboBox" name="opt_htmlz_class_style"/>
+    <widget class="QCheckBox" name="opt_htmlz_title_filename">
+     <property name="text">
+      <string>Use book &amp;title as the filename for the HTML file inside the archive</string>
+     </property>
+    </widget>
    </item>
   </layout>
  </widget>

src/calibre/gui2/convert/metadata.py

@@ -12,8 +12,8 @@ from PyQt4.Qt import QPixmap, SIGNAL
 
 from calibre.gui2 import choose_images, error_dialog
 from calibre.gui2.convert.metadata_ui import Ui_Form
-from calibre.ebooks.metadata import (authors_to_string, string_to_authors,
+from calibre.ebooks.metadata import (string_to_authors, MetaInformation,
-        MetaInformation, title_sort)
+        title_sort)
 from calibre.ebooks.metadata.opf2 import metadata_to_opf
 from calibre.ptempfile import PersistentTemporaryFile
 from calibre.gui2.convert import Widget
@@ -74,14 +74,12 @@ class MetadataWidget(Widget, Ui_Form):
 
         mi = self.db.get_metadata(self.book_id, index_is_id=True)
         self.title.setText(mi.title)
-        if mi.publisher:
+        self.publisher.show_initial_value(mi.publisher if mi.publisher else '')
-            self.publisher.setCurrentIndex(self.publisher.findText(mi.publisher))
         self.author_sort.setText(mi.author_sort if mi.author_sort else '')
         self.tags.setText(', '.join(mi.tags if mi.tags else []))
         self.tags.update_items_cache(self.db.all_tags())
         self.comment.html = comments_to_html(mi.comments) if mi.comments else ''
-        if mi.series:
+        self.series.show_initial_value(mi.series if mi.series else '')
-            self.series.setCurrentIndex(self.series.findText(mi.series))
         if mi.series_index is not None:
             try:
                 self.series_index.setValue(mi.series_index)
@@ -99,6 +97,9 @@ class MetadataWidget(Widget, Ui_Form):
         else:
             self.cover.setPixmap(QPixmap(I('default_cover.png')))
             self.cover.setToolTip(_('This book has no cover'))
+        for x in ('author', 'series', 'publisher'):
+            x = getattr(self, x)
+            x.lineEdit().deselect()
 
     def set_cover_tooltip(self, pm):
         tt = _('Cover size: %(width)d x %(height)d pixels') % dict(
@@ -118,16 +119,11 @@ class MetadataWidget(Widget, Ui_Form):
         self.author.set_add_separator(tweaks['authors_completer_append_separator'])
         self.author.update_items_cache(self.db.all_author_names())
 
-        for i in all_authors:
-            id, name = i
-            name = authors_to_string([name.strip().replace('|', ',') for n in name.split(',')])
-            self.author.addItem(name)
-
         au = self.db.authors(self.book_id, True)
         if not au:
             au = _('Unknown')
         au = ' & '.join([a.strip().replace('|', ',') for a in au.split(',')])
-        self.author.setEditText(au)
+        self.author.show_initial_value(au)
 
     def initialize_series(self):
         all_series = self.db.all_series()
@@ -135,22 +131,12 @@ class MetadataWidget(Widget, Ui_Form):
         self.series.set_separator(None)
         self.series.update_items_cache([x[1] for x in all_series])
 
-        for i in all_series:
-            id, name = i
-            self.series.addItem(name)
-        self.series.setCurrentIndex(-1)
-
     def initialize_publisher(self):
         all_publishers = self.db.all_publishers()
         all_publishers.sort(key=lambda x : sort_key(x[1]))
         self.publisher.set_separator(None)
         self.publisher.update_items_cache([x[1] for x in all_publishers])
 
-        for i in all_publishers:
-            id, name = i
-            self.publisher.addItem(name)
-        self.publisher.setCurrentIndex(-1)
-
     def get_title_and_authors(self):
         title = unicode(self.title.text()).strip()
         if not title:

src/calibre/gui2/convert/metadata.ui

@@ -190,7 +190,7 @@
         </widget>
        </item>
        <item row="4" column="1">
-        <widget class="MultiCompleteLineEdit" name="tags">
+        <widget class="EditWithComplete" name="tags">
          <property name="toolTip">
           <string>Tags categorize the book. This is particularly useful while searching. &lt;br&gt;&lt;br&gt;They can be any words or phrases, separated by commas.</string>
          </property>
@@ -213,7 +213,7 @@
         </widget>
        </item>
        <item row="5" column="1">
-        <widget class="MultiCompleteComboBox" name="series">
+        <widget class="EditWithComplete" name="series">
          <property name="sizePolicy">
           <sizepolicy hsizetype="Preferred" vsizetype="Fixed">
            <horstretch>10</horstretch>
@@ -248,14 +248,14 @@
         </widget>
        </item>
        <item row="3" column="1">
-        <widget class="MultiCompleteComboBox" name="publisher">
+        <widget class="EditWithComplete" name="publisher">
          <property name="editable">
           <bool>true</bool>
          </property>
         </widget>
        </item>
        <item row="1" column="1">
-        <widget class="MultiCompleteComboBox" name="author">
+        <widget class="EditWithComplete" name="author">
          <property name="editable">
           <bool>true</bool>
          </property>
@@ -277,14 +277,9 @@
    <header>widgets.h</header>
   </customwidget>
   <customwidget>
-   <class>MultiCompleteComboBox</class>
+   <class>EditWithComplete</class>
    <extends>QComboBox</extends>
-   <header>calibre/gui2/complete.h</header>
+   <header>calibre/gui2/complete2.h</header>
-  </customwidget>
-  <customwidget>
-   <class>MultiCompleteLineEdit</class>
-   <extends>QLineEdit</extends>
-   <header>calibre/gui2/complete.h</header>
   </customwidget>
   <customwidget>
    <class>ImageView</class>

src/calibre/gui2/custom_column_widgets.py

@@ -13,7 +13,7 @@ from PyQt4.Qt import QComboBox, QLabel, QSpinBox, QDoubleSpinBox, QDateTimeEdit,
         QPushButton, QMessageBox, QToolButton
 
 from calibre.utils.date import qt_to_dt, now
-from calibre.gui2.complete import MultiCompleteLineEdit, MultiCompleteComboBox
+from calibre.gui2.complete2 import EditWithComplete
 from calibre.gui2.comments_editor import Editor as CommentsEditor
 from calibre.gui2 import UNDEFINED_QDATETIME, error_dialog
 from calibre.gui2.dialogs.tag_editor import TagEditor
@@ -235,7 +235,7 @@ class MultipleWidget(QWidget):
         layout.setSpacing(5)
         layout.setContentsMargins(0, 0, 0, 0)
 
-        self.tags_box = MultiCompleteLineEdit(parent)
+        self.tags_box = EditWithComplete(parent)
         layout.addWidget(self.tags_box, stretch=1000)
         self.editor_button = QToolButton(self)
         self.editor_button.setToolTip(_('Open Item Editor'))
@@ -293,7 +293,7 @@ class Text(Base):
             w.setSizePolicy(QSizePolicy.Minimum, QSizePolicy.Preferred)
             w.get_editor_button().clicked.connect(self.edit)
         else:
-            w = MultiCompleteComboBox(parent)
+            w = EditWithComplete(parent)
             w.set_separator(None)
             w.setSizeAdjustPolicy(w.AdjustToMinimumContentsLengthWithIcon)
             w.setMinimumContentsLength(25)
@@ -314,14 +314,7 @@ class Text(Base):
         if self.col_metadata['is_multiple']:
             self.setter(val)
         else:
-            idx = None
+            self.widgets[1].show_initial_value(val)
-            for i, c in enumerate(values):
-                if c == val:
-                    idx = i
-                self.widgets[1].addItem(c)
-            self.widgets[1].setEditText('')
-            if idx is not None:
-                self.widgets[1].setCurrentIndex(idx)
 
     def setter(self, val):
         if self.col_metadata['is_multiple']:
@@ -370,7 +363,7 @@ class Text(Base):
 class Series(Base):
 
     def setup_ui(self, parent):
-        w = MultiCompleteComboBox(parent)
+        w = EditWithComplete(parent)
         w.set_separator(None)
         w.setSizeAdjustPolicy(w.AdjustToMinimumContentsLengthWithIcon)
         w.setMinimumContentsLength(25)
@@ -396,16 +389,8 @@ class Series(Base):
         self.initial_index = s_index
         self.initial_val = val
         val = self.normalize_db_val(val)
-        idx = None
-        self.name_widget.clear()
-        for i, c in enumerate(values):
-            if c == val:
-                idx = i
-            self.name_widget.addItem(c)
         self.name_widget.update_items_cache(values)
-        self.name_widget.setEditText('')
+        self.name_widget.show_initial_value(val)
-        if idx is not None:
-            self.widgets[1].setCurrentIndex(idx)
 
     def getter(self):
         n = unicode(self.name_widget.currentText()).strip()
@@ -822,7 +807,7 @@ class BulkDateTime(BulkBase):
 class BulkSeries(BulkBase):
 
     def setup_ui(self, parent):
-        self.make_widgets(parent, MultiCompleteComboBox)
+        self.make_widgets(parent, EditWithComplete)
         values = self.all_values = list(self.db.all_custom(num=self.col_id))
         values.sort(key=sort_key)
         self.main_widget.setSizeAdjustPolicy(self.main_widget.AdjustToMinimumContentsLengthWithIcon)
@@ -860,8 +845,6 @@ class BulkSeries(BulkBase):
         self.idx_widget.setChecked(False)
         self.main_widget.set_separator(None)
         self.main_widget.update_items_cache(self.all_values)
-        for c in self.all_values:
-            self.main_widget.addItem(c)
         self.main_widget.setEditText('')
         self.a_c_checkbox.setChecked(False)
 
@@ -951,7 +934,7 @@ class RemoveTags(QWidget):
         layout.setSpacing(5)
         layout.setContentsMargins(0, 0, 0, 0)
 
-        self.tags_box = MultiCompleteLineEdit(parent)
+        self.tags_box = EditWithComplete(parent)
         self.tags_box.update_items_cache(values)
         layout.addWidget(self.tags_box, stretch=3)
         self.checkbox = QCheckBox(_('Remove all tags'), parent)
@@ -973,7 +956,7 @@ class BulkText(BulkBase):
         values = self.all_values = list(self.db.all_custom(num=self.col_id))
         values.sort(key=sort_key)
         if self.col_metadata['is_multiple']:
-            self.make_widgets(parent, MultiCompleteLineEdit,
+            self.make_widgets(parent, EditWithComplete,
                               extra_label_text=_('tags to add'))
             self.main_widget.setSizePolicy(QSizePolicy.Minimum, QSizePolicy.Preferred)
             self.adding_widget = self.main_widget
@@ -993,7 +976,7 @@ class BulkText(BulkBase):
                 self.main_widget.set_add_separator(
                                 tweaks['authors_completer_append_separator'])
         else:
-            self.make_widgets(parent, MultiCompleteComboBox)
+            self.make_widgets(parent, EditWithComplete)
             self.main_widget.set_separator(None)
             self.main_widget.setSizeAdjustPolicy(
                         self.main_widget.AdjustToMinimumContentsLengthWithIcon)
@@ -1005,15 +988,8 @@ class BulkText(BulkBase):
         if not self.col_metadata['is_multiple']:
             val = self.get_initial_value(book_ids)
             self.initial_val = val = self.normalize_db_val(val)
-            idx = None
             self.main_widget.blockSignals(True)
-            for i, c in enumerate(self.all_values):
+            self.main_widget.show_initial_value(val)
-                if c == val:
-                    idx = i
-                self.main_widget.addItem(c)
-            self.main_widget.setEditText('')
-            if idx is not None:
-                self.main_widget.setCurrentIndex(idx)
             self.main_widget.blockSignals(False)
 
     def commit(self, book_ids, notify=False):

src/calibre/gui2/dialogs/add_empty_book.py

@@ -6,9 +6,8 @@ __license__   = 'GPL v3'
 
 from PyQt4.Qt import QDialog, QGridLayout, QLabel, QDialogButtonBox,  \
             QApplication, QSpinBox, QToolButton, QIcon
-from calibre.ebooks.metadata import authors_to_string, string_to_authors
+from calibre.ebooks.metadata import string_to_authors
-from calibre.utils.icu import sort_key
+from calibre.gui2.complete2 import EditWithComplete
-from calibre.gui2.complete import MultiCompleteComboBox
 from calibre.utils.config import tweaks
 
 class AddEmptyBookDialog(QDialog):
@@ -33,7 +32,7 @@ class AddEmptyBookDialog(QDialog):
         self.author_label = QLabel(_('Set the author of the new books to:'))
         self._layout.addWidget(self.author_label, 2, 0, 1, 2)
 
-        self.authors_combo = MultiCompleteComboBox(self)
+        self.authors_combo = EditWithComplete(self)
         self.authors_combo.setSizeAdjustPolicy(
                 self.authors_combo.AdjustToMinimumContentsLengthWithIcon)
         self.authors_combo.setEditable(True)
@@ -56,17 +55,10 @@ class AddEmptyBookDialog(QDialog):
         self.authors_combo.setEditText(_('Unknown'))
 
     def initialize_authors(self, db, author):
-        all_authors = db.all_authors()
-        all_authors.sort(key=lambda x : sort_key(x[1]))
-        for i in all_authors:
-            id, name = i
-            name = [name.strip().replace('|', ',') for n in name.split(',')]
-            self.authors_combo.addItem(authors_to_string(name))
-
         au = author
         if not au:
             au = _('Unknown')
-        self.authors_combo.setEditText(au.replace('|', ','))
+        self.authors_combo.show_initial_value(au.replace('|', ','))
 
         self.authors_combo.set_separator('&')
         self.authors_combo.set_space_before_sep(True)

src/calibre/gui2/dialogs/metadata_bulk.py

@@ -876,38 +876,25 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog):
         all_authors = self.db.all_authors()
         all_authors.sort(key=lambda x : sort_key(x[1]))
 
-        for i in all_authors:
-            id, name = i
-            name = name.strip().replace('|', ',')
-            self.authors.addItem(name)
-        self.authors.setEditText('')
-
         self.authors.set_separator('&')
         self.authors.set_space_before_sep(True)
         self.authors.set_add_separator(tweaks['authors_completer_append_separator'])
         self.authors.update_items_cache(self.db.all_author_names())
+        self.authors.show_initial_value('')
 
     def initialize_series(self):
         all_series = self.db.all_series()
         all_series.sort(key=lambda x : sort_key(x[1]))
         self.series.set_separator(None)
         self.series.update_items_cache([x[1] for x in all_series])
-
+        self.series.show_initial_value('')
-        for i in all_series:
-            id, name = i
-            self.series.addItem(name)
-        self.series.setEditText('')
 
     def initialize_publisher(self):
         all_publishers = self.db.all_publishers()
         all_publishers.sort(key=lambda x : sort_key(x[1]))
         self.publisher.set_separator(None)
         self.publisher.update_items_cache([x[1] for x in all_publishers])
-
+        self.publisher.show_initial_value('')
-        for i in all_publishers:
-            id, name = i
-            self.publisher.addItem(name)
-        self.publisher.setEditText('')
 
     def tag_editor(self, *args):
         d = TagEditor(self, self.db, None)

src/calibre/gui2/dialogs/metadata_bulk.ui

@@ -76,7 +76,7 @@
             </widget>
            </item>
            <item row="0" column="1">
-            <widget class="MultiCompleteComboBox" name="authors">
+            <widget class="EditWithComplete" name="authors">
              <property name="editable">
               <bool>true</bool>
              </property>
@@ -175,7 +175,7 @@
             </widget>
            </item>
            <item row="4" column="1">
-            <widget class="MultiCompleteComboBox" name="publisher">
+            <widget class="EditWithComplete" name="publisher">
              <property name="editable">
               <bool>true</bool>
              </property>
@@ -195,7 +195,7 @@
             </widget>
            </item>
            <item row="5" column="1">
-            <widget class="MultiCompleteLineEdit" name="tags">
+            <widget class="EditWithComplete" name="tags">
              <property name="toolTip">
               <string>Tags categorize the book. This is particularly useful while searching. &lt;br&gt;&lt;br&gt;They can be any words or phrases, separated by commas.</string>
              </property>
@@ -229,7 +229,7 @@
             </widget>
            </item>
            <item row="6" column="1">
-            <widget class="MultiCompleteLineEdit" name="remove_tags">
+            <widget class="EditWithComplete" name="remove_tags">
              <property name="toolTip">
               <string>Comma separated list of tags to remove from the books. </string>
              </property>
@@ -262,7 +262,7 @@
             </widget>
            </item>
            <item row="7" column="1">
-            <widget class="MultiCompleteComboBox" name="series">
+            <widget class="EditWithComplete" name="series">
              <property name="sizePolicy">
               <sizepolicy hsizetype="Preferred" vsizetype="Fixed">
                <horstretch>0</horstretch>
@@ -1181,14 +1181,9 @@ not multiple and the destination field is multiple</string>
    <header>widgets.h</header>
   </customwidget>
   <customwidget>
-   <class>MultiCompleteComboBox</class>
+   <class>EditWithComplete</class>
    <extends>QComboBox</extends>
-   <header>calibre/gui2/complete.h</header>
+   <header>calibre/gui2/complete2.h</header>
-  </customwidget>
-  <customwidget>
-   <class>MultiCompleteLineEdit</class>
-   <extends>QLineEdit</extends>
-   <header>calibre/gui2/complete.h</header>
   </customwidget>
   <customwidget>
    <class>HistoryLineEdit</class>

src/calibre/gui2/dialogs/search.py

@@ -25,10 +25,6 @@ class SearchDialog(QDialog, Ui_Dialog):
 
         all_authors = db.all_authors()
         all_authors.sort(key=lambda x : sort_key(x[1]))
-        for i in all_authors:
-            id, name = i
-            name = name.strip().replace('|', ',')
-            self.authors_box.addItem(name)
         self.authors_box.setEditText('')
         self.authors_box.set_separator('&')
         self.authors_box.set_space_before_sep(True)
@@ -39,10 +35,7 @@ class SearchDialog(QDialog, Ui_Dialog):
         all_series.sort(key=lambda x : sort_key(x[1]))
         self.series_box.set_separator(None)
         self.series_box.update_items_cache([x[1] for x in all_series])
-        for i in all_series:
+        self.series_box.show_initial_value('')
-            id, name = i
-            self.series_box.addItem(name)
-        self.series_box.setEditText('')
 
         all_tags = db.all_tags()
         self.tags_box.update_items_cache(all_tags)

src/calibre/gui2/dialogs/search.ui

@@ -237,21 +237,21 @@
         </widget>
        </item>
        <item row="2" column="1">
-        <widget class="MultiCompleteComboBox" name="authors_box">
+        <widget class="EditWithComplete" name="authors_box">
          <property name="toolTip">
           <string>Enter an author's name. Only one author can be used.</string>
          </property>
         </widget>
        </item>
        <item row="3" column="1">
-        <widget class="MultiCompleteComboBox" name="series_box">
+        <widget class="EditWithComplete" name="series_box">
          <property name="toolTip">
           <string>Enter a series name, without an index. Only one series name can be used.</string>
          </property>
         </widget>
        </item>
        <item row="4" column="1">
-        <widget class="MultiCompleteLineEdit" name="tags_box">
+        <widget class="EditWithComplete" name="tags_box">
          <property name="toolTip">
           <string>Enter tags separated by spaces</string>
          </property>
@@ -327,14 +327,9 @@
    <header>widgets.h</header>
   </customwidget>
   <customwidget>
-   <class>MultiCompleteLineEdit</class>
+   <class>EditWithComplete</class>
-   <extends>QLineEdit</extends>
-   <header>calibre/gui2/complete.h</header>
-  </customwidget>
-  <customwidget>
-   <class>MultiCompleteComboBox</class>
    <extends>QComboBox</extends>
-   <header>calibre/gui2/complete.h</header>
+   <header>calibre/gui2/complete2.h</header>
   </customwidget>
  </customwidgets>
  <tabstops>

src/calibre/gui2/languages.py

@@ -7,14 +7,14 @@ __license__   = 'GPL v3'
 __copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
-from calibre.gui2.complete import MultiCompleteComboBox
+from calibre.gui2.complete2 import EditWithComplete
 from calibre.utils.localization import lang_map
 from calibre.utils.icu import sort_key, lower
 
-class LanguagesEdit(MultiCompleteComboBox):
+class LanguagesEdit(EditWithComplete):
 
     def __init__(self, parent=None, db=None):
-        MultiCompleteComboBox.__init__(self, parent)
+        EditWithComplete.__init__(self, parent)
 
         self.setSizeAdjustPolicy(self.AdjustToMinimumContentsLengthWithIcon)
         self.setMinimumContentsLength(20)
@@ -32,8 +32,6 @@ class LanguagesEdit(MultiCompleteComboBox):
         all_items = sorted(self._lang_map.itervalues(),
             key=lambda x: (-pmap.get(x, 0), sort_key(x)))
         self.update_items_cache(all_items)
-        for item in all_items:
-            self.addItem(item)
 
     @property
     def vals(self):

src/calibre/gui2/library/delegates.py

@@ -14,7 +14,7 @@ from PyQt4.Qt import (Qt, QApplication, QStyle, QIcon,  QDoubleSpinBox,
 from calibre.gui2 import UNDEFINED_QDATETIME, error_dialog, rating_font
 from calibre.constants import iswindows
 from calibre.gui2.widgets import EnLineEdit
-from calibre.gui2.complete import MultiCompleteLineEdit, MultiCompleteComboBox
+from calibre.gui2.complete2 import EditWithComplete
 from calibre.utils.date import now, format_date, qt_to_dt
 from calibre.utils.config import tweaks
 from calibre.utils.formatter import validation_formatter
@@ -121,12 +121,10 @@ class TextDelegate(QStyledItemDelegate): # {{{
 
     def createEditor(self, parent, option, index):
         if self.auto_complete_function:
-            editor = MultiCompleteComboBox(parent)
+            editor = EditWithComplete(parent)
             editor.set_separator(None)
             complete_items = [i[1] for i in self.auto_complete_function()]
             editor.update_items_cache(complete_items)
-            for item in sorted(complete_items, key=sort_key):
-                editor.addItem(item)
             ct = index.data(Qt.DisplayRole).toString()
             editor.show_initial_value(ct)
         else:
@@ -134,7 +132,7 @@ class TextDelegate(QStyledItemDelegate): # {{{
         return editor
 
     def setModelData(self, editor, model, index):
-        if isinstance(editor, MultiCompleteComboBox):
+        if isinstance(editor, EditWithComplete):
             val = editor.lineEdit().text()
             model.setData(index, QVariant(val), Qt.EditRole)
         else:
@@ -155,7 +153,7 @@ class CompleteDelegate(QStyledItemDelegate): # {{{
     def createEditor(self, parent, option, index):
         if self.db and hasattr(self.db, self.items_func_name):
             col = index.model().column_map[index.column()]
-            editor = MultiCompleteComboBox(parent)
+            editor = EditWithComplete(parent)
             editor.set_separator(self.sep)
             editor.set_space_before_sep(self.space_before_sep)
             if self.sep == '&':
@@ -166,8 +164,6 @@ class CompleteDelegate(QStyledItemDelegate): # {{{
                 all_items = list(self.db.all_custom(
                     label=self.db.field_metadata.key_to_label(col)))
             editor.update_items_cache(all_items)
-            for item in sorted(all_items, key=sort_key):
-                editor.addItem(item)
             ct = index.data(Qt.DisplayRole).toString()
             editor.show_initial_value(ct)
         else:
@@ -175,7 +171,7 @@ class CompleteDelegate(QStyledItemDelegate): # {{{
         return editor
 
     def setModelData(self, editor, model, index):
-        if isinstance(editor, MultiCompleteComboBox):
+        if isinstance(editor, EditWithComplete):
             val = editor.lineEdit().text()
             model.setData(index, QVariant(val), Qt.EditRole)
         else:
@@ -248,7 +244,7 @@ class CcTextDelegate(QStyledItemDelegate): # {{{
     def createEditor(self, parent, option, index):
         m = index.model()
         col = m.column_map[index.column()]
-        editor = MultiCompleteLineEdit(parent)
+        editor = EditWithComplete(parent)
         editor.set_separator(None)
         complete_items = sorted(list(m.db.all_custom(label=m.db.field_metadata.key_to_label(col))),
                                 key=sort_key)

src/calibre/gui2/library/models.py

@@ -220,16 +220,15 @@ class BooksModel(QAbstractTableModel): # {{{
         self.count_changed()
         self.reset()
 
-    def delete_books(self, indices):
+    def delete_books(self, indices, permanent=False):
         ids = map(self.id, indices)
-        for id in ids:
+        self.delete_books_by_id(ids, permanent=permanent)
-            self.db.delete_book(id, notify=False)
-        self.books_deleted()
         return ids
 
-    def delete_books_by_id(self, ids):
+    def delete_books_by_id(self, ids, permanent=False):
         for id in ids:
-            self.db.delete_book(id)
+            self.db.delete_book(id, permanent=permanent, do_clean=False)
+        self.db.clean()
         self.books_deleted()
 
     def books_added(self, num):

src/calibre/gui2/library/views.py

@@ -125,7 +125,7 @@ class BooksView(QTableView): # {{{
         self.last_modified_delegate = DateDelegate(self,
                 tweak_name='gui_last_modified_display_format')
         self.languages_delegate = LanguagesDelegate(self)
-        self.tags_delegate = CompleteDelegate(self, ',', 'all_tags')
+        self.tags_delegate = CompleteDelegate(self, ',', 'all_tag_names')
         self.authors_delegate = CompleteDelegate(self, '&', 'all_author_names', True)
         self.cc_names_delegate = CompleteDelegate(self, '&', 'all_custom', True)
         self.series_delegate = TextDelegate(self)

src/calibre/gui2/lrf_renderer/main.py

@@ -309,7 +309,8 @@ def main(args=sys.argv, logger=None):
         return 1
     pid = os.fork() if (islinux or isbsd) else -1
     if pid <= 0:
-        app = Application(args)
+        override = 'calibre-lrf-viewer' if islinux else None
+        app = Application(args, override_program_name=override)
         app.setWindowIcon(QIcon(I('viewer.png')))
         QCoreApplication.setOrganizationName(ORG_NAME)
         QCoreApplication.setApplicationName(APP_UID)

src/calibre/gui2/main.py

@@ -8,7 +8,7 @@ from PyQt4.Qt import (QCoreApplication, QIcon, QObject, QTimer,
         QPixmap, QSplashScreen, QApplication)
 
 from calibre import prints, plugins, force_unicode
-from calibre.constants import (iswindows, __appname__, isosx, DEBUG,
+from calibre.constants import (iswindows, __appname__, isosx, DEBUG, islinux,
         filesystem_encoding)
 from calibre.utils.ipc import gui_socket_address, RC
 from calibre.gui2 import (ORG_NAME, APP_UID, initialize_file_icon_provider,
@@ -58,7 +58,8 @@ def init_qt(args):
             prints('Using library at', prefs['library_path'])
     QCoreApplication.setOrganizationName(ORG_NAME)
     QCoreApplication.setApplicationName(APP_UID)
-    app = Application(args)
+    override = 'calibre-gui' if islinux else None
+    app = Application(args, override_program_name=override)
     actions = tuple(Main.create_application_menubar())
     app.setWindowIcon(QIcon(I('lt.png')))
     return app, opts, args, actions
@@ -312,21 +313,39 @@ def cant_start(msg=_('If you are sure it is not running')+', ',
 
     raise SystemExit(1)
 
-def communicate(opts, args):
+def build_pipe(print_error=True):
-    t = RC()
+    t = RC(print_error=print_error)
     t.start()
-    time.sleep(3)
+    t.join(3.0)
-    if not t.done:
+    if t.is_alive():
-        f = os.path.expanduser('~/.calibre_calibre GUI.lock')
+        if iswindows():
-        cant_start(what=_('try deleting the file')+': '+f)
+            cant_start()
+        else:
+            f = os.path.expanduser('~/.calibre_calibre GUI.lock')
+            cant_start(what=_('try deleting the file')+': '+f)
         raise SystemExit(1)
+    return t
 
+def shutdown_other(rc=None):
+    if rc is None:
+        rc = build_pipe(print_error=False)
+        if rc.conn is None:
+            prints(_('No running calibre found'))
+            return # No running instance found
+    from calibre.utils.lock import singleinstance
+    rc.conn.send('shutdown:')
+    prints(_('Shutdown command sent, waiting for shutdown...'))
+    for i in xrange(50):
+        if singleinstance('calibre GUI'):
+            return
+        time.sleep(0.1)
+    prints(_('Failed to shutdown running calibre instance'))
+    raise SystemExit(1)
+
+def communicate(opts, args):
+    t = build_pipe()
     if opts.shutdown_running_calibre:
-        t.conn.send('shutdown:')
+        shutdown_other(t)
-        from calibre.utils.lock import singleinstance
-        prints(_('Shutdown command sent, waiting for shutdown...'))
-        while not singleinstance('calibre GUI'):
-            time.sleep(0.1)
     else:
         if len(args) > 1:
             args[1] = os.path.abspath(args[1])
@@ -334,7 +353,6 @@ def communicate(opts, args):
     t.conn.close()
     raise SystemExit(0)
 
-
 def main(args=sys.argv):
     gui_debug = None
     if args[0] == '__CALIBRE_GUI_DEBUG__':

src/calibre/gui2/metadata/basic_widgets.py

@@ -15,7 +15,6 @@ from PyQt4.Qt import (Qt, QDateTimeEdit, pyqtSignal, QMessageBox,
     QPushButton, QSpinBox, QLineEdit, QSizePolicy, QDialogButtonBox, QAction)
 
 from calibre.gui2.widgets import EnLineEdit, FormatList as _FormatList, ImageView
-from calibre.gui2.complete import MultiCompleteLineEdit, MultiCompleteComboBox
 from calibre.utils.icu import sort_key
 from calibre.utils.config import tweaks, prefs
 from calibre.ebooks.metadata import (title_sort, authors_to_string,
@@ -23,6 +22,7 @@ from calibre.ebooks.metadata import (title_sort, authors_to_string,
 from calibre.ebooks.metadata.meta import get_metadata
 from calibre.gui2 import (file_icon_provider, UNDEFINED_QDATETIME,
         choose_files, error_dialog, choose_images)
+from calibre.gui2.complete2 import EditWithComplete
 from calibre.utils.date import (local_tz, qt_to_dt, as_local_time,
         UNDEFINED_DATE)
 from calibre import strftime
@@ -204,7 +204,7 @@ class TitleSortEdit(TitleEdit):
 # }}}
 
 # Authors {{{
-class AuthorsEdit(MultiCompleteComboBox):
+class AuthorsEdit(EditWithComplete):
 
     TOOLTIP = ''
     LABEL = _('&Author(s):')
@@ -212,7 +212,7 @@ class AuthorsEdit(MultiCompleteComboBox):
     def __init__(self, parent, manage_authors):
         self.dialog = parent
         self.books_to_refresh = set([])
-        MultiCompleteComboBox.__init__(self, parent)
+        EditWithComplete.__init__(self, parent)
         self.setToolTip(self.TOOLTIP)
         self.setWhatsThis(self.TOOLTIP)
         self.setEditable(True)
@@ -246,14 +246,6 @@ class AuthorsEdit(MultiCompleteComboBox):
 
     def initialize(self, db, id_):
         self.books_to_refresh = set([])
-        all_authors = db.all_authors()
-        all_authors.sort(key=lambda x : sort_key(x[1]))
-        self.clear()
-        for i in all_authors:
-            id, name = i
-            name = name.strip().replace('|', ',')
-            self.addItem(name)
-
         self.set_separator('&')
         self.set_space_before_sep(True)
         self.set_add_separator(tweaks['authors_completer_append_separator'])
@@ -299,7 +291,6 @@ class AuthorsEdit(MultiCompleteComboBox):
             self.setEditText(' & '.join([x.strip() for x in val]))
             self.lineEdit().setCursorPosition(0)
 
-
         return property(fget=fget, fset=fset)
 
     def break_cycles(self):
@@ -452,13 +443,13 @@ class AuthorSortEdit(EnLineEdit):
 # }}}
 
 # Series {{{
-class SeriesEdit(MultiCompleteComboBox):
+class SeriesEdit(EditWithComplete):
 
     TOOLTIP = _('List of known series. You can add new series.')
     LABEL = _('&Series:')
 
     def __init__(self, parent):
-        MultiCompleteComboBox.__init__(self, parent)
+        EditWithComplete.__init__(self, parent)
         self.set_separator(None)
         self.dialog = parent
         self.setSizeAdjustPolicy(
@@ -488,19 +479,12 @@ class SeriesEdit(MultiCompleteComboBox):
         all_series.sort(key=lambda x : sort_key(x[1]))
         self.update_items_cache([x[1] for x in all_series])
         series_id = db.series_id(id_, index_is_id=True)
-        idx, c = None, 0
+        inval = ''
-        self.clear()
         for i in all_series:
-            id, name = i
+            if i[0] == series_id:
-            if id == series_id:
+                inval = i[1]
-                idx = c
+                break
-            self.addItem(name)
+        self.original_val = self.current_val = inval
-            c += 1
-
-        self.lineEdit().setText('')
-        if idx is not None:
-            self.setCurrentIndex(idx)
-        self.original_val = self.current_val
 
     def commit(self, db, id_):
         series = self.current_val
@@ -1102,14 +1086,14 @@ class RatingEdit(QSpinBox): # {{{
 
 # }}}
 
-class TagsEdit(MultiCompleteLineEdit): # {{{
+class TagsEdit(EditWithComplete): # {{{
     LABEL = _('Ta&gs:')
     TOOLTIP = '<p>'+_('Tags categorize the book. This is particularly '
             'useful while searching. <br><br>They can be any words '
             'or phrases, separated by commas.')
 
     def __init__(self, parent):
-        MultiCompleteLineEdit.__init__(self, parent)
+        EditWithComplete.__init__(self, parent)
         self.books_to_refresh = set([])
         self.setToolTip(self.TOOLTIP)
         self.setWhatsThis(self.TOOLTIP)
@@ -1130,7 +1114,7 @@ class TagsEdit(MultiCompleteLineEdit): # {{{
         tags = db.tags(id_, index_is_id=True)
         tags = tags.split(',') if tags else []
         self.current_val = tags
-        self.all_items = db.all_tags()
+        self.all_items = db.all_tag_names()
         self.original_val = self.current_val
 
     @property
@@ -1343,11 +1327,11 @@ class ISBNDialog(QDialog) : # {{{
 
 # }}}
 
-class PublisherEdit(MultiCompleteComboBox): # {{{
+class PublisherEdit(EditWithComplete): # {{{
     LABEL = _('&Publisher:')
 
     def __init__(self, parent):
-        MultiCompleteComboBox.__init__(self, parent)
+        EditWithComplete.__init__(self, parent)
         self.set_separator(None)
         self.setSizeAdjustPolicy(
                 self.AdjustToMinimumContentsLengthWithIcon)
@@ -1373,17 +1357,12 @@ class PublisherEdit(MultiCompleteComboBox): # {{{
         all_publishers.sort(key=lambda x : sort_key(x[1]))
         self.update_items_cache([x[1] for x in all_publishers])
         publisher_id = db.publisher_id(id_, index_is_id=True)
-        idx = None
+        inval = ''
-        self.clear()
+        for pid, name in all_publishers:
-        for i, x in enumerate(all_publishers):
+            if pid == publisher_id:
-            id_, name = x
+                inval = name
-            if id_ == publisher_id:
+                break
-                idx = i
+        self.original_val = self.current_val = inval
-            self.addItem(name)
-
-        self.setEditText('')
-        if idx is not None:
-            self.setCurrentIndex(idx)
 
     def commit(self, db, id_):
         self.books_to_refresh |= db.set_publisher(id_, self.current_val,

src/calibre/gui2/preferences/__init__.py

@@ -13,6 +13,7 @@ from PyQt4.Qt import (QWidget, pyqtSignal, QCheckBox, QAbstractSpinBox,
 
 from calibre.customize.ui import preferences_plugins
 from calibre.utils.config import ConfigProxy
+from calibre.gui2.complete2 import EditWithComplete
 
 class AbortCommit(Exception):
     pass
@@ -133,11 +134,15 @@ class Setting(object):
     def initialize(self):
         self.gui_obj.blockSignals(True)
         if self.datatype == 'choice':
-            self.gui_obj.clear()
+            choices = self.choices or []
-            for x in self.choices:
+            if isinstance(self.gui_obj, EditWithComplete):
-                if isinstance(x, basestring):
+                self.gui_obj.all_items = choices
-                    x = (x, x)
+            else:
-                self.gui_obj.addItem(x[0], QVariant(x[1]))
+                self.gui_obj.clear()
+                for x in choices:
+                    if isinstance(x, basestring):
+                        x = (x, x)
+                    self.gui_obj.addItem(x[0], QVariant(x[1]))
         self.set_gui_val(self.get_config_val(default=False))
         self.gui_obj.blockSignals(False)
         self.initial_value = self.get_gui_val()
@@ -171,11 +176,14 @@ class Setting(object):
         elif self.datatype == 'string':
             self.gui_obj.setText(val if val else '')
         elif self.datatype == 'choice':
-            idx = self.gui_obj.findData(QVariant(val), role=Qt.UserRole,
+            if isinstance(self.gui_obj, EditWithComplete):
-                    flags=self.CHOICES_SEARCH_FLAGS)
+                self.gui_obj.setText(val)
-            if idx == -1:
+            else:
-                idx = 0
+                idx = self.gui_obj.findData(QVariant(val), role=Qt.UserRole,
-            self.gui_obj.setCurrentIndex(idx)
+                        flags=self.CHOICES_SEARCH_FLAGS)
+                if idx == -1:
+                    idx = 0
+                self.gui_obj.setCurrentIndex(idx)
 
     def get_gui_val(self):
         if self.datatype == 'bool':
@@ -187,9 +195,12 @@ class Setting(object):
             if self.empty_string_is_None and not val:
                 val = None
         elif self.datatype == 'choice':
-            idx = self.gui_obj.currentIndex()
+            if isinstance(self.gui_obj, EditWithComplete):
-            if idx < 0: idx = 0
+                val = unicode(self.gui_obj.text())
-            val = unicode(self.gui_obj.itemData(idx).toString())
+            else:
+                idx = self.gui_obj.currentIndex()
+                if idx < 0: idx = 0
+                val = unicode(self.gui_obj.itemData(idx).toString())
         return val
 
 class CommaSeparatedList(Setting):

src/calibre/gui2/preferences/look_feel.py

@@ -75,7 +75,7 @@ class DisplayedFields(QAbstractListModel): # {{{
 
     def commit(self):
         if self.changed:
-            gprefs['book_display_fields'] = self.fields
+            self.db.prefs['book_display_fields'] = self.fields
 
     def move(self, idx, delta):
         row = idx.row() + delta

src/calibre/gui2/preferences/look_feel.ui

@@ -320,7 +320,7 @@ Manage Authors. You can use the values {author} and
       </attribute>
       <layout class="QGridLayout" name="gridLayout_10">
        <item row="3" column="2" colspan="3">
-        <widget class="MultiCompleteLineEdit" name="opt_categories_using_hierarchy">
+        <widget class="EditWithComplete" name="opt_categories_using_hierarchy">
          <property name="toolTip">
           <string>A comma-separated list of categories in which items containing
 periods are displayed in the tag browser trees. For example, if
@@ -397,7 +397,7 @@ up into subcategories. If the partition method is set to disable, this value is
         </widget>
        </item>
        <item row="1" column="3" colspan="2">
-        <widget class="MultiCompleteLineEdit" name="opt_tag_browser_dont_collapse">
+        <widget class="EditWithComplete" name="opt_tag_browser_dont_collapse">
          <property name="toolTip">
           <string>A comma-separated list of categories that are not to
 be partitioned even if the number of items is larger than
@@ -506,9 +506,9 @@ a few top-level elements.</string>
  </widget>
  <customwidgets>
   <customwidget>
-   <class>MultiCompleteLineEdit</class>
+   <class>EditWithComplete</class>
-   <extends>QLineEdit</extends>
+   <extends>QComboBox</extends>
-   <header>calibre/gui2/complete.h</header>
+   <header>calibre/gui2/complete2.h</header>
   </customwidget>
  </customwidgets>
  <resources>

src/calibre/gui2/preferences/search.py

@@ -13,6 +13,7 @@ from calibre.gui2.preferences.search_ui import Ui_Form
 from calibre.gui2 import config, error_dialog
 from calibre.utils.config import prefs
 from calibre.utils.icu import sort_key
+from calibre.library.caches import set_use_primary_find_in_search
 
 class ConfigWidget(ConfigWidgetBase, Ui_Form):
 
@@ -26,6 +27,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         r('search_as_you_type', config)
         r('highlight_search_matches', config)
         r('limit_search_columns', prefs)
+        r('use_primary_find_in_search', prefs)
         r('limit_search_columns_to', prefs, setting=CommaSeparatedList)
         fl = db.field_metadata.get_search_terms()
         self.opt_limit_search_columns_to.update_items_cache(fl)
@@ -96,7 +98,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
             db.prefs.set('grouped_search_make_user_categories', [])
         r('grouped_search_make_user_categories', db.prefs, setting=CommaSeparatedList)
         self.muc_changed = False
-        self.opt_grouped_search_make_user_categories.editingFinished.connect(
+        self.opt_grouped_search_make_user_categories.lineEdit().editingFinished.connect(
                                                         self.muc_box_changed)
 
     def set_similar_fields(self, initial=False):
@@ -182,6 +184,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         self.opt_grouped_search_make_user_categories.update_items_cache(terms)
         self.gst_names.blockSignals(True)
         self.gst_names.clear()
+        print (1111, self.gst_names)
         self.gst_names.addItem('', '')
         for t in terms:
             self.gst_names.addItem(t, t)
@@ -222,6 +225,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         return ConfigWidgetBase.commit(self)
 
     def refresh_gui(self, gui):
+        set_use_primary_find_in_search(prefs['use_primary_find_in_search'])
         gui.set_highlight_only_button_icon()
         if self.muc_changed:
             gui.tags_view.recount()

src/calibre/gui2/preferences/search.ui

@@ -7,7 +7,7 @@
     <x>0</x>
     <y>0</y>
     <width>670</width>
-    <height>556</height>
+    <height>663</height>
    </rect>
   </property>
   <property name="windowTitle">
@@ -21,14 +21,21 @@
      </property>
     </widget>
    </item>
-   <item row="1" column="0">
+   <item row="0" column="1">
+    <widget class="QCheckBox" name="opt_use_primary_find_in_search">
+     <property name="text">
+      <string>Unaccented characters match accented characters</string>
+     </property>
+    </widget>
+   </item>
+   <item row="1" column="0" colspan="2">
     <widget class="QCheckBox" name="opt_highlight_search_matches">
      <property name="text">
       <string>&amp;Highlight search results instead of restricting the book list to the results</string>
      </property>
     </widget>
    </item>
-   <item row="2" column="0">
+   <item row="4" column="0" colspan="2">
     <widget class="QGroupBox" name="groupBox">
      <property name="title">
       <string>What to search by default</string>
@@ -62,7 +69,7 @@
        </widget>
       </item>
       <item row="2" column="1">
-       <widget class="MultiCompleteLineEdit" name="opt_limit_search_columns_to"/>
+       <widget class="EditWithComplete" name="opt_limit_search_columns_to"/>
       </item>
       <item row="5" column="0" colspan="2">
        <widget class="QLabel" name="label_3">
@@ -77,17 +84,7 @@
      </layout>
     </widget>
    </item>
-   <item row="3" column="0">
+   <item row="6" column="0" colspan="2">
-    <widget class="QPushButton" name="clear_history_button">
-     <property name="toolTip">
-      <string>Clear search histories from all over calibre. Including the book list, e-book viewer, fetch news dialog, etc.</string>
-     </property>
-     <property name="text">
-      <string>Clear search &amp;histories</string>
-     </property>
-    </widget>
-   </item>
-   <item row="4" column="0">
     <widget class="QGroupBox" name="groupBox_2">
      <property name="title">
       <string>Grouped Search Terms</string>
@@ -107,12 +104,6 @@
         </item>
         <item>
          <widget class="QComboBox" name="gst_names">
-          <property name="editable">
-           <bool>true</bool>
-          </property>
-          <property name="minimumContentsLength">
-           <number>10</number>
-          </property>
           <property name="toolTip">
            <string>Contains the names of the currently-defined group search terms.
 Create a new name by entering it into the empty box, then
@@ -120,6 +111,12 @@ pressing Save. Rename a search term by selecting it then
 changing the name and pressing Save. Change the value of
 a search term by changing the value box then pressing Save.</string>
           </property>
+          <property name="editable">
+           <bool>true</bool>
+          </property>
+          <property name="minimumContentsLength">
+           <number>10</number>
+          </property>
          </widget>
         </item>
         <item>
@@ -137,7 +134,7 @@ a search term by changing the value box then pressing Save.</string>
          </widget>
         </item>
         <item>
-         <widget class="MultiCompleteLineEdit" name="gst_value"/>
+         <widget class="EditWithComplete" name="gst_value"/>
         </item>
         <item>
          <widget class="QToolButton" name="gst_save_button">
@@ -176,7 +173,7 @@ of a search term by changing the value box then pressing Save.</string>
          </widget>
         </item>
         <item>
-         <widget class="MultiCompleteLineEdit" name="opt_grouped_search_make_user_categories">
+         <widget class="EditWithComplete" name="opt_grouped_search_make_user_categories">
           <property name="toolTip">
            <string>Enter the names of any grouped search terms you wish
 to be shown as user categories</string>
@@ -201,7 +198,17 @@ to be shown as user categories</string>
      </layout>
     </widget>
    </item>
-   <item row="5" column="0">
+   <item row="5" column="0" colspan="2">
+    <widget class="QPushButton" name="clear_history_button">
+     <property name="toolTip">
+      <string>Clear search histories from all over calibre. Including the book list, e-book viewer, fetch news dialog, etc.</string>
+     </property>
+     <property name="text">
+      <string>Clear search &amp;histories</string>
+     </property>
+    </widget>
+   </item>
+   <item row="7" column="0" colspan="2">
     <widget class="QGroupBox" name="groupBox22">
      <property name="title">
       <string>What to search when searching similar books</string>
@@ -211,7 +218,7 @@ to be shown as user categories</string>
        <widget class="QLabel" name="label">
         <property name="text">
          <string>&lt;p&gt;When you search for similar books by right clicking the
-         book and selecting "Similar books...",
+         book and selecting &quot;Similar books...&quot;,
          calibre constructs a search using the column lookup names specified below.
          By changing the lookup name to a grouped search term you can
          search multiple columns at once.&lt;/p&gt;</string>
@@ -239,8 +246,7 @@ to be shown as user categories</string>
        </widget>
       </item>
       <item row="1" column="2">
-       <widget class="QComboBox" name="opt_similar_authors_match_kind">
+       <widget class="QComboBox" name="opt_similar_authors_match_kind"/>
-       </widget>
       </item>
       <item row="1" column="3">
        <widget class="QLabel" name="label_222">
@@ -260,8 +266,7 @@ to be shown as user categories</string>
        </widget>
       </item>
       <item row="1" column="5">
-       <widget class="QComboBox" name="opt_similar_series_match_kind">
+       <widget class="QComboBox" name="opt_similar_series_match_kind"/>
-       </widget>
       </item>
       <item row="2" column="0">
        <widget class="QLabel" name="label_223">
@@ -271,12 +276,10 @@ to be shown as user categories</string>
        </widget>
       </item>
       <item row="2" column="1">
-       <widget class="QComboBox" name="similar_tags_search_key">
+       <widget class="QComboBox" name="similar_tags_search_key"/>
-       </widget>
       </item>
       <item row="2" column="2">
-       <widget class="QComboBox" name="opt_similar_tags_match_kind">
+       <widget class="QComboBox" name="opt_similar_tags_match_kind"/>
-       </widget>
       </item>
       <item row="2" column="3">
        <widget class="QLabel" name="label_224">
@@ -286,12 +289,10 @@ to be shown as user categories</string>
        </widget>
       </item>
       <item row="2" column="4">
-       <widget class="QComboBox" name="similar_publisher_search_key">
+       <widget class="QComboBox" name="similar_publisher_search_key"/>
-       </widget>
       </item>
       <item row="2" column="5">
-       <widget class="QComboBox" name="opt_similar_publisher_match_kind">
+       <widget class="QComboBox" name="opt_similar_publisher_match_kind"/>
-       </widget>
       </item>
      </layout>
     </widget>
@@ -300,9 +301,9 @@ to be shown as user categories</string>
  </widget>
  <customwidgets>
   <customwidget>
-   <class>MultiCompleteLineEdit</class>
+   <class>EditWithComplete</class>
-   <extends>QLineEdit</extends>
+   <extends>QComboBox</extends>
-   <header>calibre/gui2/complete.h</header>
+   <header>calibre/gui2/complete2.h</header>
   </customwidget>
  </customwidgets>
  <resources>

src/calibre/gui2/preferences/tweaks.py

@@ -5,7 +5,9 @@ __license__   = 'GPL v3'
 __copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
+from functools import partial
 import textwrap
+from collections import OrderedDict
 
 from calibre.gui2.preferences import ConfigWidgetBase, test_widget, AbortCommit
 from calibre.gui2.preferences.tweaks_ui import Ui_Form
@@ -18,8 +20,8 @@ from calibre.utils.search_query_parser import (ParseException,
         SearchQueryParser)
 
 from PyQt4.Qt import (QAbstractListModel, Qt, QStyledItemDelegate, QStyle,
-    QStyleOptionViewItem, QFont, QDialogButtonBox, QDialog,
+    QStyleOptionViewItem, QFont, QDialogButtonBox, QDialog, QApplication,
-    QVBoxLayout, QPlainTextEdit, QLabel, QModelIndex)
+    QVBoxLayout, QPlainTextEdit, QLabel, QModelIndex, QMenu, QIcon)
 
 ROOT = QModelIndex()
 
@@ -46,10 +48,12 @@ class Tweak(object): # {{{
         if self.doc:
             self.doc = translate(self.doc)
         self.var_names = var_names
-        self.default_values = {}
+        if self.var_names:
+            self.doc = u"%s: %s\n\n%s"%(_('ID'), self.var_names[0], self.doc)
+        self.default_values = OrderedDict()
         for x in var_names:
             self.default_values[x] = defaults[x]
-        self.custom_values = {}
+        self.custom_values = OrderedDict()
         for x in var_names:
             if x in custom:
                 self.custom_values[x] = custom[x]
@@ -243,7 +247,8 @@ class Tweaks(QAbstractListModel, SearchQueryParser): # {{{
         query = lower(query)
         for r in candidates:
             dat = self.data(self.index(r), Qt.UserRole)
-            if query in lower(dat.name):# or query in lower(dat.doc):
+            var_names = u' '.join(dat.default_values)
+            if query in lower(dat.name) or query in lower(var_names):
                 ans.add(r)
         return ans
 
@@ -325,7 +330,29 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         self.search.initialize('tweaks_search_history', help_text=
                 _('Search for tweak'))
         self.search.search.connect(self.find)
+        self.view.setContextMenuPolicy(Qt.CustomContextMenu)
+        self.view.customContextMenuRequested.connect(self.show_context_menu)
+        self.copy_icon = QIcon(I('edit-copy.png'))
 
+    def show_context_menu(self, point):
+        idx = self.tweaks_view.currentIndex()
+        if not idx.isValid():
+            return True
+        tweak = self.tweaks.data(idx, Qt.UserRole)
+        self.context_menu = QMenu(self)
+        self.context_menu.addAction(self.copy_icon,
+                            _('Copy to clipboard'),
+                            partial(self.copy_item_to_clipboard,
+                                    val=u"%s (%s: %s)"%(tweak.name,
+                                                        _('ID'),
+                                                        tweak.var_names[0])))
+        self.context_menu.popup(self.mapToGlobal(point))
+        return True
+
+    def copy_item_to_clipboard(self, val):
+        cb = QApplication.clipboard();
+        cb.clear()
+        cb.setText(val)
 
     def plugin_tweaks(self):
         raw = self.tweaks.plugin_tweaks_string
@@ -441,7 +468,6 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
 
 
 if __name__ == '__main__':
-    from PyQt4.Qt import QApplication
     app = QApplication([])
     #Tweaks()
     #test_widget

src/calibre/gui2/store/stores/bn_plugin.py

@@ -24,26 +24,12 @@ from calibre.gui2.store.web_store_dialog import WebStoreDialog
 class BNStore(BasicStoreConfig, StorePlugin):
 
     def open(self, parent=None, detail_item=None, external=False):
-        pub_id = 'sHa5EXvYOwA'
+        url = "http://bn.com"
-        # Use Kovid's affiliate id 30% of the time.
-        if random.randint(1, 10) in (1, 2, 3):
-            pub_id = '0dsO3kDu/AU'
-
-        murl = 'http://click.linksynergy.com/fs-bin/click?id=%s&offerid=239662.13&type=3&subid=0' % pub_id
-
-        if detail_item:
-            purl = 'http://click.linksynergy.com/fs-bin/click?id=%s&subid=&offerid=239662.%s&type=2&subid=0' % (pub_id, detail_item)
-            url = purl
-        else:
-            purl = None
-            url = murl
-
-        #print(url)
 
         if external or self.config.get('open_external', False):
-            open_url(QUrl(url_slash_cleaner(url)))
+            open_url(QUrl(url_slash_cleaner(detail_item if detail_item else url)))
         else:
-            d = WebStoreDialog(self.gui, murl, parent, purl)
+            d = WebStoreDialog(self.gui, url, parent, detail_item)
             d.setWindowTitle(self.name)
             d.set_tags(self.config.get('tags', ''))
             d.exec_()
@@ -60,7 +46,7 @@ class BNStore(BasicStoreConfig, StorePlugin):
                 if counter <= 0:
                     break
 
-                id = ''.join(data.xpath('.//div[contains(@class, "display-tile-item")]/@data-bn-ean'))
+                id = ''.join(data.xpath('.//div[contains(@class, "image-bounding-box")]/a/@href'))
                 if not id:
                     continue
 

src/calibre/gui2/store/stores/ebooks_com_plugin.py

@@ -64,11 +64,11 @@ class EbookscomStore(BasicStoreConfig, StorePlugin):
                     continue
                 id = mo.group()
 
-                cover_url = ''.join(data.xpath('.//div[@class="img"]//img/@src'))
+                cover_url = ''.join(data.xpath('.//div[contains(@class, "img")]//img/@src'))
 
                 title = ''.join(data.xpath(
                     'descendant::span[@class="book-title"]/a/text()')).strip()
-                author = ''.join(data.xpath(
+                author = ', '.join(data.xpath(
                     'descendant::span[@class="author"]/a/text()')).strip()
                 if not title or not author:
                     continue

src/calibre/gui2/store/stores/oreilly_plugin.py

@@ -1,81 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import (unicode_literals, division, absolute_import, print_function)
-
-__license__ = 'GPL 3'
-__copyright__ = '2011, John Schember <john@nachtimwald.com>'
-__docformat__ = 'restructuredtext en'
-
-import urllib
-from contextlib import closing
-
-from lxml import html
-
-from PyQt4.Qt import QUrl
-
-from calibre import browser, url_slash_cleaner
-from calibre.gui2 import open_url
-from calibre.gui2.store import StorePlugin
-from calibre.gui2.store.basic_config import BasicStoreConfig
-from calibre.gui2.store.search_result import SearchResult
-from calibre.gui2.store.web_store_dialog import WebStoreDialog
-
-class OReillyStore(BasicStoreConfig, StorePlugin):
-
-    def open(self, parent=None, detail_item=None, external=False):
-        url = 'http://oreilly.com/ebooks/'
-
-        if external or self.config.get('open_external', False):
-            open_url(QUrl(url_slash_cleaner(detail_item if detail_item else url)))
-        else:
-            d = WebStoreDialog(self.gui, url, parent, detail_item)
-            d.setWindowTitle(self.name)
-            d.set_tags(self.config.get('tags', ''))
-            d.exec_()
-
-    def search(self, query, max_results=10, timeout=60):
-        url = 'http://search.oreilly.com/?t1=Books&t2=Format&t3=Ebook&q=' + urllib.quote_plus(query)
-
-        br = browser()
-
-        counter = max_results
-        with closing(br.open(url, timeout=timeout)) as f:
-            doc = html.fromstring(f.read())
-            for data in doc.xpath('//div[@class="result"]'):
-                if counter <= 0:
-                    break
-
-                ebook = ' '.join(data.xpath('.//p[@class="note"]/text()'))
-                if 'ebook' not in ebook.lower():
-                    continue
-
-                id = ''.join(data.xpath('./div[@class="book_text"]//p[@class="title"]/a/@href'))
-
-                cover_url = ''.join(data.xpath('./a/img[1]/@src'))
-
-                title = ''.join(data.xpath('./div[@class="book_text"]/p[@class="title"]/a/text()'))
-                author = ''.join(data.xpath('./div[@class="book_text"]/p[@class="note"][1]/text()'))
-                author = author.split('By ')[-1].strip()
-
-                # Get the detail here because we need to get the ebook id for the detail_item.
-                with closing(br.open(id, timeout=timeout)) as nf:
-                    idoc = html.fromstring(nf.read())
-
-                    for td in idoc.xpath('//td[@class="optionsTd"]'):
-                        if 'ebook' in ''.join(td.xpath('.//text()')).lower():
-                            price = ''.join(td.xpath('.//span[@class="price"]/text()')).strip()
-                            formats = ''.join(td.xpath('.//a[@id="availableFormats"]/text()')).strip()
-                            break
-
-                counter -= 1
-
-                s = SearchResult()
-                s.cover_url = cover_url.strip()
-                s.title = title.strip()
-                s.author = author.strip()
-                s.detail_item = id.strip()
-                s.price = price.strip()
-                s.drm = SearchResult.DRM_UNLOCKED
-                s.formats = formats.upper()
-
-                yield s

src/calibre/gui2/store/stores/ozon_ru_plugin.py

@@ -46,30 +46,37 @@ class OzonRUStore(BasicStoreConfig, StorePlugin):
             d.set_tags(self.config.get('tags', ''))
             d.exec_()        
         
-
+    def search(self, query, max_results=15, timeout=60):
-    def search(self, query, max_results=10, timeout=60):
         search_url = self.shop_url + '/webservice/webservice.asmx/SearchWebService?'\
                     'searchText=%s&searchContext=ebook' % urllib2.quote(query)
+        search_urls = [ search_url ]
+        
+        ## add this as the fist try if it looks like ozon ID
+        if re.match("^\d{6,9}$", query):
+            ozon_detail = self.shop_url + '/webservices/OzonWebSvc.asmx/ItemDetail?ID=%s' % query
+            search_urls.insert(0, ozon_detail)
+
         xp_template = 'normalize-space(./*[local-name() = "{0}"]/text())'
-                    
         counter = max_results
         br = browser()
-        with closing(br.open(search_url, timeout=timeout)) as f:
+        
-            raw = xml_to_unicode(f.read(), strip_encoding_pats=True, assume_utf8=True)[0]
+        for url in search_urls:
-            doc = etree.fromstring(raw)
+            with closing(br.open(url, timeout=timeout)) as f:
-            for data in doc.xpath('//*[local-name() = "SearchItems"]'):
+                raw = xml_to_unicode(f.read(), strip_encoding_pats=True, assume_utf8=True)[0]
-                if counter <= 0:
+                doc = etree.fromstring(raw)
-                    break
+                for data in doc.xpath('//*[local-name()="SearchItems" or local-name()="ItemDetail"]'):
-                counter -= 1
+                    if counter <= 0:
+                        break
+                    counter -= 1
 
-                s = SearchResult()
+                    s = SearchResult()
-                s.detail_item = data.xpath(xp_template.format('ID'))
+                    s.detail_item = data.xpath(xp_template.format('ID'))
-                s.title = data.xpath(xp_template.format('Name'))
+                    s.title = data.xpath(xp_template.format('Name'))
-                s.author = data.xpath(xp_template.format('Author'))
+                    s.author = data.xpath(xp_template.format('Author'))
-                s.price = data.xpath(xp_template.format('Price'))
+                    s.price = data.xpath(xp_template.format('Price'))
-                s.cover_url = data.xpath(xp_template.format('Picture'))
+                    s.cover_url = data.xpath(xp_template.format('Picture'))
-                s.price = format_price_in_RUR(s.price)
+                    s.price = format_price_in_RUR(s.price)
-                yield s
+                    yield s
 
     def get_details(self, search_result, timeout=60):
         url = self.shop_url + '/context/detail/id/' + urllib2.quote(search_result.detail_item)
@@ -97,6 +104,16 @@ class OzonRUStore(BasicStoreConfig, StorePlugin):
                 search_result.formats = ', '.join(_parse_ebook_formats(formats))
                 # unfortunately no direct links to download books (only buy link)
                 # search_result.downloads['BF2'] = self.shop_url + '/order/digitalorder.aspx?id=' + + urllib2.quote(search_result.detail_item)
+            
+            #<p class="main-cost"><span class="main">215</span><span class="submain">00</span> руб.</p>
+            #<span itemprop="price" class="hidden">215.00</span>
+            #<meta itemprop="priceCurrency" content="RUR " />
+            
+            # if the price not in the search result (the ID search case)
+            if not search_result.price:
+                price = doc.xpath(u'normalize-space(//*[@itemprop="price"]/text())')
+                search_result.price = format_price_in_RUR(price)
+                
         return result
 
 def format_price_in_RUR(price):

src/calibre/gui2/store/stores/weightless_books_plugin.py

@@ -41,7 +41,7 @@ class WeightlessBooksStore(BasicStoreConfig, StorePlugin):
         counter = max_results
         with closing(br.open(url, timeout=timeout)) as f:
             doc = html.fromstring(f.read())
-            for data in doc.xpath('//li[@id="product"]'):
+            for data in doc.xpath('//li[@class="product"]'):
                 if counter <= 0:
                     break
 

src/calibre/gui2/tag_browser/model.py

@@ -17,7 +17,7 @@ from PyQt4.Qt import (QAbstractItemModel, QIcon, QVariant, QFont, Qt,
 from calibre.gui2 import NONE, gprefs, config, error_dialog
 from calibre.library.database2 import Tag
 from calibre.utils.config import tweaks
-from calibre.utils.icu import sort_key, lower, strcmp
+from calibre.utils.icu import sort_key, lower, strcmp, contractions
 from calibre.library.field_metadata import TagsIcons, category_icon_map
 from calibre.gui2.dialogs.confirm_delete import confirm
 from calibre.utils.formatter import EvalFormatter
@@ -258,6 +258,16 @@ class TagsModel(QAbstractItemModel): # {{{
                 self.hidden_categories.add(cat)
         db.prefs.set('tag_browser_hidden_categories', list(self.hidden_categories))
 
+        conts = contractions()
+        if len(conts) == 0 or not tweaks['enable_multicharacters_in_tag_browser']:
+            self.do_contraction = False
+        else:
+            self.do_contraction = True
+            nconts = set()
+            for s in conts:
+                nconts.add(icu_upper(s))
+            self.contraction_set = frozenset(nconts)
+
         self.db = db
         self._run_rebuild()
         self.endResetModel()
@@ -335,7 +345,6 @@ class TagsModel(QAbstractItemModel): # {{{
                         node.is_gst = is_gst
                         if not is_gst:
                             node.tag.is_hierarchical = '5state'
-                        if not is_gst:
                             tree_root[p] = {}
                             tree_root = tree_root[p]
                     else:
@@ -417,7 +426,14 @@ class TagsModel(QAbstractItemModel): # {{{
                     if not tag.sort:
                         c = ' '
                     else:
-                        c = icu_upper(tag.sort[0])
+                        if not self.do_contraction:
+                            c = icu_upper(tag.sort)[0]
+                        else:
+                            v = icu_upper(tag.sort)
+                            c = v[0]
+                            for s in self.contraction_set:
+                                if len(s) > len(c) and v.startswith(s):
+                                    c = s
                     if c not in chardict:
                         chardict[c] = [idx, idx]
                     else:
@@ -519,7 +535,7 @@ class TagsModel(QAbstractItemModel): # {{{
                 # category display order is important here. The following works
                 # only if all the non-user categories are displayed before the
                 # user categories
-                if category_is_hierarchical:
+                if category_is_hierarchical or tag.is_hierarchical:
                     components = get_name_components(tag.original_name)
                 else:
                     components = [tag.original_name]
@@ -581,6 +597,14 @@ class TagsModel(QAbstractItemModel): # {{{
                 return [(t.tag.id, t.tag.original_name, t.tag.count)
                         for t in cat.child_tags() if t.tag.count > 0]
 
+    def is_in_user_category(self, index):
+        if not index.isValid():
+            return False
+        p = self.get_node(index)
+        while p.type != TagTreeItem.CATEGORY:
+            p = p.parent
+        return p.tag.category.startswith('@')
+
     # Drag'n Drop {{{
     def mimeTypes(self):
         return ["application/calibre+from_library",
@@ -646,13 +670,13 @@ class TagsModel(QAbstractItemModel): # {{{
         action is Qt.CopyAction or Qt.MoveAction
         '''
         def process_source_node(user_cats, src_parent, src_parent_is_gst,
-                                is_uc, dest_key, node):
+                                is_uc, dest_key, idx):
             '''
             Copy/move an item and all its children to the destination
             '''
             copied = False
-            src_name = node.tag.original_name
+            src_name = idx.tag.original_name
-            src_cat = node.tag.category
+            src_cat = idx.tag.category
             # delete the item if the source is a user category and action is move
             if is_uc and not src_parent_is_gst and src_parent in user_cats and \
                                     action == Qt.MoveAction:
@@ -675,7 +699,7 @@ class TagsModel(QAbstractItemModel): # {{{
             if add_it:
                 user_cats[dest_key].append([src_name, src_cat, 0])
 
-            for c in node.children:
+            for c in idx.children:
                 copied = process_source_node(user_cats, src_parent, src_parent_is_gst,
                                              is_uc, dest_key, c)
             return copied
@@ -696,11 +720,11 @@ class TagsModel(QAbstractItemModel): # {{{
             if dest_key not in user_cats:
                 continue
 
-            node = self.index_for_path(path)
+            idx = self.index_for_path(path)
-            if node:
+            if idx.isValid():
                 process_source_node(user_cats, src_parent, src_parent_is_gst,
                                              is_uc, dest_key,
-                                             self.get_node(node))
+                                             self.get_node(idx))
 
         self.db.prefs.set('user_categories', user_cats)
         self.refresh_required.emit()
@@ -1139,6 +1163,8 @@ class TagsModel(QAbstractItemModel): # {{{
             return QModelIndex()
 
         ans = self.createIndex(parent_item.row(), 0, parent_item)
+        if not ans.isValid():
+            return QModelIndex()
         return ans
 
     def rowCount(self, parent):

src/calibre/gui2/tag_browser/view.py

@@ -12,7 +12,8 @@ from functools import partial
 from itertools import izip
 
 from PyQt4.Qt import (QStyledItemDelegate, Qt, QTreeView, pyqtSignal, QSize,
-        QIcon, QApplication, QMenu, QPoint, QModelIndex, QToolTip, QCursor)
+        QIcon, QApplication, QMenu, QPoint, QModelIndex, QToolTip, QCursor,
+        QDrag)
 
 from calibre.gui2.tag_browser.model import (TagTreeItem, TAG_SEARCH_STATES,
         TagsModel)
@@ -101,6 +102,7 @@ class TagsView(QTreeView): # {{{
         self.setDragEnabled(True)
         self.setDragDropMode(self.DragDrop)
         self.setDropIndicatorShown(True)
+        self.in_drag_drop = False
         self.setAutoExpandDelay(500)
         self.pane_is_visible = False
         self.search_icon = QIcon(I('search.png'))
@@ -232,10 +234,35 @@ class TagsView(QTreeView): # {{{
         s = s if s else None
         self._model.set_search_restriction(s)
 
+    def mouseMoveEvent(self, event):
+        dex = self.indexAt(event.pos())
+        if self.in_drag_drop or not dex.isValid():
+            QTreeView.mouseMoveEvent(self, event)
+            return
+        # Must deal with odd case where the node being dragged is 'virtual',
+        # created to form a hierarchy. We can't really drag this node, but in
+        # addition we can't allow drag recognition to notice going over some
+        # other node and grabbing that one. So we set in_drag_drop to prevent
+        # this from happening, turning it off when the user lifts the button.
+        self.in_drag_drop = True
+        if not self._model.flags(dex) & Qt.ItemIsDragEnabled:
+            QTreeView.mouseMoveEvent(self, event)
+            return
+        md = self._model.mimeData([dex])
+        pixmap = dex.data(Qt.DecorationRole).toPyObject().pixmap(25, 25)
+        drag = QDrag(self)
+        drag.setPixmap(pixmap)
+        drag.setMimeData(md)
+        if self._model.is_in_user_category(dex):
+            drag.exec_(Qt.CopyAction|Qt.MoveAction, Qt.CopyAction)
+        else:
+            drag.exec_(Qt.CopyAction)
+
     def mouseReleaseEvent(self, event):
         # Swallow everything except leftButton so context menus work correctly
-        if event.button() == Qt.LeftButton:
+        if event.button() == Qt.LeftButton or self.in_drag_drop:
             QTreeView.mouseReleaseEvent(self, event)
+            self.in_drag_drop = False
 
     def mouseDoubleClickEvent(self, event):
         # swallow these to avoid toggling and editing at the same time
@@ -622,11 +649,13 @@ class TagsView(QTreeView): # {{{
         path = self.model().path_for_index(ci) if self.is_visible(ci) else None
         expanded_categories, state_map = self.get_state()
         self._model.rebuild_node_tree(state_map=state_map)
+        self.blockSignals(True)
         for category in expanded_categories:
             idx = self._model.index_for_category(category)
             if idx is not None and idx.isValid():
                 self.expand(idx)
         self.show_item_at_path(path)
+        self.blockSignals(False)
 
     def show_item_at_path(self, path, box=False,
                           position=QTreeView.PositionAtCenter):
@@ -639,12 +668,19 @@ class TagsView(QTreeView): # {{{
             self.show_item_at_index(self._model.index_for_path(path), box=box,
                                     position=position)
 
+    def expand_parent(self, idx):
+        # Needed otherwise Qt sometimes segfaults if the node is buried in a
+        # collapsed, off screen hierarchy. To be safe, we expand from the
+        # outermost in
+        p = self._model.parent(idx)
+        if p.isValid():
+            self.expand_parent(p)
+        self.expand(idx)
+
     def show_item_at_index(self, idx, box=False,
                            position=QTreeView.PositionAtCenter):
         if idx.isValid() and idx.data(Qt.UserRole).toPyObject() is not self._model.root_item:
-            self.expand(self._model.parent(idx)) # Needed otherwise Qt sometimes segfaults if the
+            self.expand_parent(idx)
-                                                 # node is buried in a collapsed, off
-                                                 # screen hierarchy
             self.setCurrentIndex(idx)
             self.scrollTo(idx, position)
             if box:

src/calibre/gui2/tools.py

@@ -11,6 +11,7 @@ import cPickle, os
 
 from PyQt4.Qt import QDialog, QProgressDialog, QString, QTimer
 
+from calibre.constants import DEBUG
 from calibre.ptempfile import PersistentTemporaryFile
 from calibre.gui2 import warning_dialog, question_dialog
 from calibre.gui2.convert.single import NoSupportedInputFormats

src/calibre/gui2/viewer/config.py

@@ -51,6 +51,8 @@ def config(defaults=None):
             help=_('The amount by which to change the font size when clicking'
                 ' the font larger/smaller buttons. Should be a number between '
                 '0 and 1.'))
+    c.add_opt('fullscreen_clock', default=False, action='store_true',
+            help=_('Show a clock in fullscreen mode.'))
 
     fonts = c.add_group('FONTS', _('Font options'))
     fonts('serif_family', default='Times New Roman' if iswindows else 'Liberation Serif',
@@ -117,6 +119,7 @@ class ConfigDialog(QDialog, Ui_Dialog):
             self.hyphenate.setVisible(False)
             self.hyphenate_default_lang.setVisible(False)
             self.hyphenate_label.setVisible(False)
+        self.opt_fullscreen_clock.setChecked(opts.fullscreen_clock)
 
     def accept(self, *args):
         if self.shortcut_config.is_editing:
@@ -148,6 +151,7 @@ class ConfigDialog(QDialog, Ui_Dialog):
                 str(self.hyphenate_default_lang.itemData(idx).toString()))
         c.set('line_scrolling_stops_on_pagebreaks',
                 self.opt_line_scrolling_stops_on_pagebreaks.isChecked())
+        c.set('fullscreen_clock', self.opt_fullscreen_clock.isChecked())
         return QDialog.accept(self, *args)
 
 

src/calibre/gui2/viewer/config.ui

@@ -6,7 +6,7 @@
    <rect>
     <x>0</x>
     <y>0</y>
-    <width>479</width>
+    <width>839</width>
     <height>630</height>
    </rect>
   </property>
@@ -167,20 +167,6 @@
        </item>
        <item>
         <layout class="QGridLayout" name="gridLayout_2">
-         <item row="8" column="0" colspan="2">
-          <widget class="QCheckBox" name="opt_remember_window_size">
-           <property name="text">
-            <string>Remember last used &amp;window size and layout</string>
-           </property>
-          </widget>
-         </item>
-         <item row="9" column="0" colspan="2">
-          <widget class="QCheckBox" name="opt_remember_current_page">
-           <property name="text">
-            <string>Remember the &amp;current page when quitting</string>
-           </property>
-          </widget>
-         </item>
          <item row="5" column="0" colspan="2">
           <widget class="QCheckBox" name="hyphenate">
            <property name="text">
@@ -205,13 +191,6 @@
            </property>
           </widget>
          </item>
-         <item row="7" column="0" colspan="2">
-          <widget class="QCheckBox" name="opt_fit_images">
-           <property name="text">
-            <string>&amp;Resize images larger than the viewer window (needs restart)</string>
-           </property>
-          </widget>
-         </item>
          <item row="4" column="0">
           <widget class="QLabel" name="label_11">
            <property name="text">
@@ -247,13 +226,6 @@
            </property>
           </widget>
          </item>
-         <item row="10" column="0" colspan="2">
-          <widget class="QCheckBox" name="opt_wheel_flips_pages">
-           <property name="text">
-            <string>Mouse &amp;wheel flips pages</string>
-           </property>
-          </widget>
-         </item>
          <item row="3" column="1">
           <widget class="QSpinBox" name="max_fs_width">
            <property name="toolTip">
@@ -301,13 +273,48 @@
            </property>
           </widget>
          </item>
-         <item row="11" column="0" colspan="2">
+         <item row="7" column="0">
+          <widget class="QCheckBox" name="opt_fit_images">
+           <property name="text">
+            <string>&amp;Resize images larger than the viewer window (needs restart)</string>
+           </property>
+          </widget>
+         </item>
+         <item row="8" column="0">
+          <widget class="QCheckBox" name="opt_remember_window_size">
+           <property name="text">
+            <string>Remember last used &amp;window size and layout</string>
+           </property>
+          </widget>
+         </item>
+         <item row="7" column="1">
+          <widget class="QCheckBox" name="opt_wheel_flips_pages">
+           <property name="text">
+            <string>Mouse &amp;wheel flips pages</string>
+           </property>
+          </widget>
+         </item>
+         <item row="8" column="1">
+          <widget class="QCheckBox" name="opt_remember_current_page">
+           <property name="text">
+            <string>Remember the &amp;current page when quitting</string>
+           </property>
+          </widget>
+         </item>
+         <item row="9" column="0">
           <widget class="QCheckBox" name="opt_line_scrolling_stops_on_pagebreaks">
            <property name="text">
             <string>Line &amp;scrolling stops at page breaks</string>
            </property>
           </widget>
          </item>
+         <item row="9" column="1">
+          <widget class="QCheckBox" name="opt_fullscreen_clock">
+           <property name="text">
+            <string>Show &amp;clock in full screen mode</string>
+           </property>
+          </widget>
+         </item>
         </layout>
        </item>
       </layout>

src/calibre/gui2/viewer/documentview.py

@@ -134,9 +134,10 @@ class Document(QWebPage): # {{{
         screen_width = QApplication.desktop().screenGeometry().width()
         # Leave some space for the scrollbar and some border
         self.max_fs_width = min(opts.max_fs_width, screen_width-50)
+        self.fullscreen_clock = opts.fullscreen_clock
 
     def fit_images(self):
-        if self.do_fit_images:
+        if self.do_fit_images and not self.in_paged_mode:
             self.javascript('setup_image_scaling_handlers()')
 
     def add_window_objects(self):
@@ -193,6 +194,14 @@ class Document(QWebPage): # {{{
         self.read_anchor_positions(use_cache=False)
         self.first_load = False
 
+    def colors(self):
+        self.javascript('''
+            bs = getComputedStyle(document.body);
+            py_bridge.value = [bs.backgroundColor, bs.color]
+            ''')
+        ans = self.bridge_value
+        return (ans if isinstance(ans, list) else ['white', 'black'])
+
     def read_anchor_positions(self, use_cache=True):
         self.bridge_value = tuple(self.index_anchors)
         self.javascript(u'''
@@ -219,6 +228,7 @@ class Document(QWebPage): # {{{
         if scroll_width > self.window_width:
             sz.setWidth(scroll_width+side_margin)
             self.setPreferredContentsSize(sz)
+        self.javascript('window.paged_display.fit_images()')
 
     @property
     def column_boundaries(self):

src/calibre/gui2/viewer/javascript.py

@@ -31,10 +31,11 @@ class JavaScriptLoader(object):
             'cfi':'ebooks.oeb.display.cfi',
             'indexing':'ebooks.oeb.display.indexing',
             'paged':'ebooks.oeb.display.paged',
+            'utils':'ebooks.oeb.display.utils',
         }
 
     ORDER = ('jquery', 'jquery_scrollTo', 'bookmarks', 'referencing', 'images',
-            'hyphenation', 'hyphenator', 'cfi', 'indexing', 'paged')
+            'hyphenation', 'hyphenator', 'utils', 'cfi', 'indexing', 'paged')
 
 
     def __init__(self, dynamic_coffeescript=False):

src/calibre/gui2/viewer/main.py

@@ -6,9 +6,10 @@ from functools import partial
 from threading import Thread
 
 from PyQt4.Qt import (QApplication, Qt, QIcon, QTimer, QByteArray, QSize,
-        QDoubleSpinBox, QLabel, QTextBrowser, QPropertyAnimation, QPainter,
+        QTime, QDoubleSpinBox, QLabel, QTextBrowser, QPropertyAnimation,
-        QBrush, QColor, pyqtSignal, QUrl, QRegExpValidator, QRegExp, QLineEdit,
+        QPainter, QBrush, QColor, pyqtSignal, QUrl, QRegExpValidator, QRegExp,
-        QToolButton, QMenu, QInputDialog, QAction, QKeySequence, QModelIndex)
+        QLineEdit, QToolButton, QMenu, QInputDialog, QAction, QKeySequence,
+        QModelIndex)
 
 from calibre.gui2.viewer.main_ui import Ui_EbookViewer
 from calibre.gui2.viewer.printing import Printing
@@ -288,6 +289,23 @@ class EbookViewer(MainWindow, Ui_EbookViewer):
         self.addAction(self.toggle_toolbar_action)
         self.full_screen_label_anim = QPropertyAnimation(
                 self.full_screen_label, 'size')
+        self.clock_label = QLabel('99:99', self)
+        self.clock_label.setVisible(False)
+        self.clock_label.setFocusPolicy(Qt.NoFocus)
+        self.clock_label_style = '''
+            QLabel {
+                text-align: right;
+                border-width: 1px;
+                border-style: solid;
+                border-radius: 8px;
+                background-color: %s;
+                color: %s;
+                font-family: monospace;
+                font-size: larger;
+                padding: 5px;
+        }'''
+        self.clock_timer = QTimer(self)
+        self.clock_timer.timeout.connect(self.update_clock)
         self.esc_full_screen_action = a = QAction(self)
         self.addAction(a)
         a.setShortcut(Qt.Key_Escape)
@@ -454,9 +472,29 @@ class EbookViewer(MainWindow, Ui_EbookViewer):
         a.start()
         QTimer.singleShot(2750, self.full_screen_label.hide)
         self.view.document.switch_to_fullscreen_mode()
+        if self.view.document.fullscreen_clock:
+            self.show_clock()
+
+    def show_clock(self):
+        self.clock_label.setVisible(True)
+        self.clock_label.setText('99:99 AA')
+        self.clock_timer.start(1000)
+        self.clock_label.setStyleSheet(self.clock_label_style%
+                tuple(self.view.document.colors()))
+        self.clock_label.resize(self.clock_label.sizeHint())
+        sw = QApplication.desktop().screenGeometry(self.view)
+        self.clock_label.move(sw.width() - self.vertical_scrollbar.width() - 15
+                - self.clock_label.width(), sw.height() -
+                self.clock_label.height()-10)
+        self.update_clock()
+
+    def update_clock(self):
+        self.clock_label.setText(QTime.currentTime().toString('h:mm a'))
 
     def showNormal(self):
         self.view.document.page_position.save()
+        self.clock_label.setVisible(False)
+        self.clock_timer.stop()
         self.window_mode_changed = 'normal'
         self.esc_full_screen_action.setEnabled(False)
         self.tool_bar.setVisible(True)
@@ -1006,7 +1044,8 @@ def main(args=sys.argv):
     except:
         open_at = None
     if pid <= 0:
-        app = Application(args)
+        override = 'calibre-ebook-viewer' if islinux else None
+        app = Application(args, override_program_name=override)
         app.setWindowIcon(QIcon(I('viewer.png')))
         QApplication.setOrganizationName(ORG_NAME)
         QApplication.setApplicationName(APP_UID)

src/calibre/gui2/viewer/printing.py

@@ -26,8 +26,8 @@ class Printing(QObject):
         for x in (Qt.Horizontal, Qt.Vertical):
             mf.setScrollBarPolicy(x, Qt.ScrollBarAlwaysOff)
         self.view.loadFinished.connect(self.load_finished)
-        self.paged_js = compiled_coffeescript('ebooks.oeb.display.paged',
+        self.paged_js = compiled_coffeescript('ebooks.oeb.display.utils')
-                dynamic=False)
+        self.paged_js += compiled_coffeescript('ebooks.oeb.display.paged')
 
     def load_finished(self, ok):
         self.loaded_ok = ok
@@ -70,6 +70,7 @@ class Printing(QObject):
                 document.body.style.backgroundColor = "white";
                 paged_display.set_geometry(1, 0, 0, 0);
                 paged_display.layout();
+                paged_display.fit_images();
             ''')
 
             while True:

src/calibre/library/caches.py

@@ -6,7 +6,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
-import re, itertools, time, traceback
+import re, itertools, time, traceback, locale
 from itertools import repeat, izip, imap
 from datetime import timedelta
 from threading import Thread
@@ -15,11 +15,11 @@ from calibre.utils.config import tweaks, prefs
 from calibre.utils.date import parse_date, now, UNDEFINED_DATE, clean_date_for_sort
 from calibre.utils.search_query_parser import SearchQueryParser
 from calibre.utils.pyparsing import ParseException
-from calibre.utils.localization import (canonicalize_lang, lang_map, get_udc,
+from calibre.utils.localization import (canonicalize_lang, lang_map, get_udc)
-        get_lang)
 from calibre.ebooks.metadata import title_sort, author_to_author_sort
 from calibre.ebooks.metadata.opf2 import metadata_to_opf
 from calibre import prints
+from calibre.utils.icu import primary_find
 
 class MetadataBackup(Thread): # {{{
     '''
@@ -118,7 +118,15 @@ class MetadataBackup(Thread): # {{{
 
 # }}}
 
+
 ### Global utility function for get_match here and in gui2/library.py
+# This is a global for performance
+pref_use_primary_find_in_search = False
+
+def set_use_primary_find_in_search(toWhat):
+    global pref_use_primary_find_in_search
+    pref_use_primary_find_in_search = toWhat
+
 CONTAINS_MATCH = 0
 EQUALS_MATCH   = 1
 REGEXP_MATCH   = 2
@@ -130,8 +138,8 @@ def _match(query, value, matchkind):
     else:
         internal_match_ok = False
     for t in value:
-        t = icu_lower(t)
         try:     ### ignore regexp exceptions, required because search-ahead tries before typing is finished
+            t = icu_lower(t)
             if (matchkind == EQUALS_MATCH):
                 if internal_match_ok:
                     if query == t:
@@ -147,9 +155,13 @@ def _match(query, value, matchkind):
                             return True
                 elif query == t:
                     return True
-            elif ((matchkind == REGEXP_MATCH and re.search(query, t, re.I|re.UNICODE)) or ### search unanchored
+            elif matchkind == REGEXP_MATCH:
-                  (matchkind == CONTAINS_MATCH and query in t)):
+                return re.search(query, t, re.I|re.UNICODE)
-                    return True
+            elif matchkind == CONTAINS_MATCH:
+                if pref_use_primary_find_in_search:
+                    return primary_find(query, t)[0] != -1
+                else:
+                    return query in t
         except re.error:
             pass
     return False
@@ -226,10 +238,6 @@ class ResultCache(SearchQueryParser): # {{{
     '''
     def __init__(self, FIELD_MAP, field_metadata, db_prefs=None):
         self.FIELD_MAP = FIELD_MAP
-        l = get_lang()
-        asciize_author_names = l and l.lower() in ('en', 'eng')
-        if not asciize_author_names:
-            self.ascii_name = lambda x: False
         self.db_prefs = db_prefs
         self.composites = {}
         self.udc = get_udc()
@@ -249,6 +257,9 @@ class ResultCache(SearchQueryParser): # {{{
         SearchQueryParser.__init__(self, self.all_search_locations, optimize=True)
         self.build_date_relop_dict()
         self.build_numeric_relop_dict()
+        # Do this here so the var get updated when a library changes
+        global pref_use_primary_find_in_search
+        pref_use_primary_find_in_search = prefs['use_primary_find_in_search']
 
     def break_cycles(self):
         self._data = self.field_metadata = self.FIELD_MAP = \
@@ -279,15 +290,6 @@ class ResultCache(SearchQueryParser): # {{{
 
     # Search functions {{{
 
-    def ascii_name(self, name):
-        try:
-            ans = self.udc.decode(name)
-            if ans == name:
-                ans = False
-        except:
-            ans = False
-        return ans
-
     def universal_set(self):
         return set([i[0] for i in self._data if i is not None])
 
@@ -623,6 +625,8 @@ class ResultCache(SearchQueryParser): # {{{
 
     def get_matches(self, location, query, candidates=None,
             allow_recursion=True):
+        # If candidates is not None, it must not be modified. Changing its
+        # value will break query optimization in the search parser
         matches = set([])
         if candidates is None:
             candidates = self.universal_set()
@@ -649,8 +653,13 @@ class ResultCache(SearchQueryParser): # {{{
                     else:
                         invert = False
                     for loc in location:
-                        matches |= self.get_matches(loc, query,
+                        c = candidates.copy()
-                                candidates=candidates, allow_recursion=False)
+                        m = self.get_matches(loc, query,
+                                candidates=c, allow_recursion=False)
+                        matches |= m
+                        c -= m
+                        if len(c) == 0:
+                            break
                     if invert:
                         matches = self.universal_set() - matches
                     return matches
@@ -665,10 +674,15 @@ class ResultCache(SearchQueryParser): # {{{
                     if l and l != 'all' and l in self.all_search_locations:
                         terms.add(l)
                 if terms:
+                    c = candidates.copy()
                     for l in terms:
                         try:
-                            matches |= self.get_matches(l, query,
+                            m = self.get_matches(l, query,
-                                candidates=candidates, allow_recursion=allow_recursion)
+                                candidates=c, allow_recursion=allow_recursion)
+                            matches |= m
+                            c -= m
+                            if len(c) == 0:
+                                break
                         except:
                             pass
                     return matches
@@ -745,6 +759,7 @@ class ResultCache(SearchQueryParser): # {{{
             for i, loc in enumerate(location):
                 location[i] = db_col[loc]
 
+            current_candidates = candidates.copy()
             for loc in location: # location is now an array of field indices
                 if loc == db_col['authors']:
                     ### DB stores authors with commas changed to bars, so change query
@@ -761,9 +776,7 @@ class ResultCache(SearchQueryParser): # {{{
                 else:
                     q = query
 
-                au_loc = self.FIELD_MAP['authors']
+                for id_ in current_candidates:
-
-                for id_ in candidates:
                     item = self._data[id_]
                     if item is None: continue
 
@@ -805,14 +818,12 @@ class ResultCache(SearchQueryParser): # {{{
                     if loc not in exclude_fields: # time for text matching
                         if is_multiple_cols[loc] is not None:
                             vals = [v.strip() for v in item[loc].split(is_multiple_cols[loc])]
-                            if loc == au_loc:
-                                vals += filter(None, map(self.ascii_name,
-                                    vals))
                         else:
                             vals = [item[loc]] ### make into list to make _match happy
                         if _match(q, vals, matchkind):
                             matches.add(item[0])
                             continue
+                current_candidates -= matches
         return matches
 
     def search(self, query, return_matches=False):
@@ -902,7 +913,9 @@ class ResultCache(SearchQueryParser): # {{{
 
     def set(self, row, col, val, row_is_id=False):
         id = row if row_is_id else self._map_filtered[row]
-        self._data[id][col] = val
+        d = self._data[id]
+        d[col] = val
+        d.refresh_composites()
 
     def get(self, row, col, row_is_id=False):
         id = row if row_is_id else self._map_filtered[row]
@@ -1081,15 +1094,14 @@ class SortKeyGenerator(object):
                     dt = 'datetime'
                 elif sb == 'number':
                     try:
-                        val = val.replace(',', '').strip()
                         p = 1
                         for i, candidate in enumerate(
-                                    (' B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB')):
+                                    ('B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB')):
                             if val.endswith(candidate):
                                 p = 1024**(i)
                                 val = val[:-len(candidate)].strip()
                                 break
-                        val = float(val) * p
+                        val = locale.atof(val) * p
                     except:
                         val = 0.0
                     dt = 'float'

src/calibre/library/catalogs/epub_mobi.py

@@ -11,7 +11,6 @@ import os
 from collections import namedtuple
 
 from calibre import strftime
-from calibre.constants import DEBUG
 from calibre.customize import CatalogPlugin
 from calibre.customize.conversion import OptionRecommendation, DummyReporter
 
@@ -277,6 +276,7 @@ class EPUB_MOBI(CatalogPlugin):
             log.error("coercing thumb_width from '%s' to '%s'" % (opts.thumb_width,self.THUMB_SMALLEST))
             opts.thumb_width = "1.0"
 
+
         # Display opts
         keys = opts_dict.keys()
         keys.sort()
@@ -315,11 +315,10 @@ class EPUB_MOBI(CatalogPlugin):
             recommendations = []
             recommendations.append(('remove_fake_margins', False,
                 OptionRecommendation.HIGH))
-            if DEBUG:
+            recommendations.append(('comments', '', OptionRecommendation.HIGH))
-                recommendations.append(('comments', '\n'.join(line for line in build_log),
+
-                    OptionRecommendation.HIGH))
+            # Use to debug generated catalog code before conversion
-            else:
+            #setattr(opts,'debug_pipeline',os.path.expanduser("~/Desktop/Catalog debug"))
-                recommendations.append(('comments', '', OptionRecommendation.HIGH))
 
             dp = getattr(opts, 'debug_pipeline', None)
             if dp is not None:

src/calibre/library/catalogs/epub_mobi_builder.py

@@ -3,7 +3,7 @@
 __license__   = 'GPL v3'
 __copyright__ = '2010, Greg Riker'
 
-import datetime, htmlentitydefs, os, re, shutil, zlib
+import datetime, htmlentitydefs, os, re, shutil, unicodedata, zlib
 from copy import deepcopy
 from xml.sax.saxutils import escape
 
@@ -14,12 +14,12 @@ from calibre.ebooks.chardet import substitute_entites
 from calibre.ptempfile import PersistentTemporaryDirectory
 from calibre.utils.config import config_dir
 from calibre.utils.date import format_date, is_date_undefined, now as nowf
-from calibre.utils.icu import capitalize
+from calibre.utils.filenames import ascii_text
+from calibre.utils.icu import capitalize, sort_key
 from calibre.utils.magick.draw import thumbnail
 from calibre.utils.zipfile import ZipFile
 
 
-
 class CatalogBuilder(object):
     '''
     Generates catalog source files from calibre database
@@ -41,6 +41,9 @@ class CatalogBuilder(object):
     # [] = No date ranges added
     DATE_RANGE=[30]
 
+    # Text used in generated catalog for title section with other-than-ASCII leading letter
+    SYMBOLS = _('Symbols')
+
     # basename              output file basename
     # creator               dc:creator in OPF metadata
     # descriptionClip       limits size of NCX descriptions (Kindle only)
@@ -564,10 +567,9 @@ class CatalogBuilder(object):
 
         self.updateProgressFullStep("Sorting database")
         self.booksByAuthor = list(self.booksByTitle)
-
-        # Test for author_sort mismatches
         self.booksByAuthor = sorted(self.booksByAuthor, key=self.booksByAuthorSorter_author)
-        # Build the unique_authors set from existing data
+
+        # Build the unique_authors set from existing data, test for author_sort mismatches
         authors = [(record['author'], record['author_sort']) for record in self.booksByAuthor]
         current_author = authors[0]
         for (i,author) in enumerate(authors):
@@ -602,7 +604,8 @@ Author '{0}':
 
                 current_author = author
 
-        self.booksByAuthor = sorted(self.booksByAuthor, key=self.booksByAuthorSorter_author_sort)
+        self.booksByAuthor = sorted(self.booksByAuthor,
+                                    key=lambda x: sort_key(self.booksByAuthorSorter_author_sort(x)))
 
         # Build the unique_authors set from existing data
         authors = [(record['author'], capitalize(record['author_sort'])) for record in self.booksByAuthor]
@@ -778,8 +781,11 @@ Author '{0}':
 
         # Re-sort based on title_sort
         if len(titles):
-            self.booksByTitle = sorted(titles,
+            #self.booksByTitle = sorted(titles,
-                                    key=lambda x:(x['title_sort'].upper(), x['title_sort'].upper()))
+            #                        key=lambda x:(x['title_sort'].upper(), x['title_sort'].upper()))
+
+            self.booksByTitle = sorted(titles, key=lambda x: sort_key(x['title_sort'].upper()))
+
             if False and self.verbose:
                 self.opts.log.info("fetchBooksByTitle(): %d books" % len(self.booksByTitle))
                 self.opts.log.info(" %-40s %-40s" % ('title', 'title_sort'))
@@ -921,29 +927,24 @@ Author '{0}':
         body = soup.find('body')
         btc = 0
 
-        # Insert section tag
+        pTag = Tag(soup, "p")
+        pTag['class'] = 'title'
+        ptc = 0
         aTag = Tag(soup,'a')
-        aTag['name'] = 'section_start'
+        aTag['id'] = 'section_start'
-        body.insert(btc, aTag)
+        pTag.insert(ptc, aTag)
-        btc += 1
+        ptc += 1
-
-        # Insert the anchor
-        aTag = Tag(soup, "a")
-        aTag['name'] = "bytitle"
-        body.insert(btc, aTag)
-        btc += 1
 
         if not self.__generateForKindle:
-            # We don't need this because the Kindle shows section titles
+            # Kindle don't need this because it shows section titles in Periodical format
-            #<h2><a name="byalphatitle" id="byalphatitle"></a>By Title</h2>
-            pTag = Tag(soup, "p")
-            pTag['class'] = 'title'
             aTag = Tag(soup, "a")
-            aTag['name'] = "bytitle"
+            aTag['id'] = "bytitle"
-            pTag.insert(0,aTag)
+            pTag.insert(ptc,aTag)
-            pTag.insert(1,NavigableString('Titles'))
+            ptc += 1
-            body.insert(btc,pTag)
+            pTag.insert(ptc,NavigableString('Titles'))
-            btc += 1
+
+        body.insert(btc,pTag)
+        btc += 1
 
         divTag = Tag(soup, "div")
         dtc = 0
@@ -953,7 +954,7 @@ Author '{0}':
         # Incoming title <series> <series_index>: <title>
         if not self.useSeriesPrefixInTitlesSection:
             nspt = deepcopy(self.booksByTitle)
-            nspt = sorted(nspt, key=lambda x:(x['title_sort'].upper(), x['title_sort'].upper()))
+            nspt = sorted(nspt, key=lambda x: sort_key(x['title_sort'].upper()))
             self.booksByTitle_noSeriesPrefix = nspt
 
         # Loop through the books by title
@@ -975,11 +976,14 @@ Author '{0}':
                 if dtc > 0:
                     divRunningTag['class'] = "initial_letter"
                 drtc = 0
-                current_letter = self.letter_or_symbol(book['title_sort'][0])
                 pIndexTag = Tag(soup, "p")
                 pIndexTag['class'] = "author_title_letter_index"
                 aTag = Tag(soup, "a")
-                aTag['name'] = "%s" % self.letter_or_symbol(book['title_sort'][0])
+                current_letter = self.letter_or_symbol(book['title_sort'][0])
+                if current_letter == self.SYMBOLS:
+                    aTag['id'] = self.SYMBOLS
+                else:
+                    aTag['id'] = "%s" % self.generateUnicodeName(current_letter)
                 pIndexTag.insert(0,aTag)
                 pIndexTag.insert(1,NavigableString(self.letter_or_symbol(book['title_sort'][0])))
                 divRunningTag.insert(dtc,pIndexTag)
@@ -1072,19 +1076,6 @@ Author '{0}':
 
         btc = 0
 
-        # Insert section tag
-        aTag = Tag(soup,'a')
-        aTag['name'] = 'section_start'
-        body.insert(btc, aTag)
-        btc += 1
-
-        # Insert the anchor
-        aTag = Tag(soup, "a")
-        anchor_name = friendly_name.lower()
-        aTag['name'] = anchor_name.replace(" ","")
-        body.insert(btc, aTag)
-        btc += 1
-
         divTag = Tag(soup, "div")
         dtc = 0
         divOpeningTag = None
@@ -1115,7 +1106,6 @@ Author '{0}':
                     drtc = 0
                     divRunningTag = None
 
-                current_letter = self.letter_or_symbol(book['author_sort'][0].upper())
                 author_count = 0
                 divOpeningTag = Tag(soup, 'div')
                 if dtc > 0:
@@ -1124,7 +1114,11 @@ Author '{0}':
                 pIndexTag = Tag(soup, "p")
                 pIndexTag['class'] = "author_title_letter_index"
                 aTag = Tag(soup, "a")
-                aTag['name'] = "%sauthors" % self.letter_or_symbol(current_letter)
+                current_letter = self.letter_or_symbol(book['author_sort'][0].upper())
+                if current_letter == self.SYMBOLS:
+                    aTag['id'] = self.SYMBOLS
+                else:
+                    aTag['id'] = "%s_authors" % self.generateUnicodeName(current_letter)
                 pIndexTag.insert(0,aTag)
                 pIndexTag.insert(1,NavigableString(self.letter_or_symbol(book['author_sort'][0].upper())))
                 divOpeningTag.insert(dotc,pIndexTag)
@@ -1156,7 +1150,7 @@ Author '{0}':
                 pAuthorTag = Tag(soup, "p")
                 pAuthorTag['class'] = "author_index"
                 aTag = Tag(soup, "a")
-                aTag['name'] = "%s" % self.generateAuthorAnchor(current_author)
+                aTag['id'] = "%s" % self.generateAuthorAnchor(current_author)
                 aTag.insert(0,NavigableString(current_author))
                 pAuthorTag.insert(0,aTag)
                 if author_count == 1:
@@ -1245,19 +1239,25 @@ Author '{0}':
 
         # Loop ends here
 
+        pTag = Tag(soup, "p")
+        pTag['class'] = 'title'
+        ptc = 0
+        aTag = Tag(soup,'a')
+        aTag['id'] = 'section_start'
+        pTag.insert(ptc, aTag)
+        ptc += 1
+
         if not self.__generateForKindle:
-            # Insert the <h2> tag with book_count at the head
+            # Kindle don't need this because it shows section titles in Periodical format
-            #<h2><a name="byalphaauthor" id="byalphaauthor"></a>By Author</h2>
-            pTag = Tag(soup, "p")
-            pTag['class'] = 'title'
             aTag = Tag(soup, "a")
             anchor_name = friendly_name.lower()
-            aTag['name'] = anchor_name.replace(" ","")
+            aTag['id'] = anchor_name.replace(" ","")
-            pTag.insert(0,aTag)
+            pTag.insert(ptc,aTag)
-            #h2Tag.insert(1,NavigableString('%s (%d)' % (friendly_name, book_count)))
+            ptc += 1
-            pTag.insert(1,NavigableString('%s' % (friendly_name)))
+            pTag.insert(ptc,NavigableString('%s' % (friendly_name)))
-            body.insert(btc,pTag)
+
-            btc += 1
+        body.insert(btc,pTag)
+        btc += 1
 
         if author_count == 1:
             divTag.insert(dtc, divOpeningTag)
@@ -1292,7 +1292,7 @@ Author '{0}':
                 pIndexTag = Tag(soup, "p")
                 pIndexTag['class'] = "date_index"
                 aTag = Tag(soup, "a")
-                aTag['name'] = "bda_%s-%s" % (current_date.year, current_date.month)
+                aTag['id'] = "bda_%s-%s" % (current_date.year, current_date.month)
                 pIndexTag.insert(0,aTag)
                 pIndexTag.insert(1,NavigableString(date_string))
                 divTag.insert(dtc,pIndexTag)
@@ -1310,7 +1310,7 @@ Author '{0}':
                         pAuthorTag['class'] = "author_index"
                         aTag = Tag(soup, "a")
                         if self.opts.generate_authors:
-                            aTag['name'] = "%s" % self.generateAuthorAnchor(current_author)
+                            aTag['id'] = "%s" % self.generateAuthorAnchor(current_author)
                         aTag.insert(0,NavigableString(current_author))
                         pAuthorTag.insert(0,aTag)
                         divTag.insert(dtc,pAuthorTag)
@@ -1384,7 +1384,7 @@ Author '{0}':
                 pIndexTag = Tag(soup, "p")
                 pIndexTag['class'] = "date_index"
                 aTag = Tag(soup, "a")
-                aTag['name'] = "bda_%s" % date_range.replace(' ','')
+                aTag['id'] = "bda_%s" % date_range.replace(' ','')
                 pIndexTag.insert(0,aTag)
                 pIndexTag.insert(1,NavigableString(date_range))
                 divTag.insert(dtc,pIndexTag)
@@ -1455,30 +1455,27 @@ Author '{0}':
 
         btc = 0
 
-        # Insert section tag
+        pTag = Tag(soup, "p")
-        aTag = Tag(soup,'a')
+        pTag['class'] = 'title'
-        aTag['name'] = 'section_start'
+        ptc = 0
-        body.insert(btc, aTag)
-        btc += 1
 
-        # Insert the anchor
+        aTag = Tag(soup,'a')
-        aTag = Tag(soup, "a")
+        aTag['id'] = 'section_start'
-        anchor_name = friendly_name.lower()
+        pTag.insert(ptc, aTag)
-        aTag['name'] = anchor_name.replace(" ","")
+        ptc += 1
-        body.insert(btc, aTag)
-        btc += 1
 
         if not self.__generateForKindle:
-            #<h2><a name="byalphaauthor" id="byalphaauthor"></a>By Author</h2>
+            # Kindle don't need this because it shows section titles in Periodical format
-            pTag = Tag(soup, "p")
-            pTag['class'] = 'title'
             aTag = Tag(soup, "a")
             anchor_name = friendly_name.lower()
-            aTag['name'] = anchor_name.replace(" ","")
+            aTag['id'] = anchor_name.replace(" ","")
-            pTag.insert(0,aTag)
+
-            pTag.insert(1,NavigableString('%s' % friendly_name))
+            pTag.insert(ptc,aTag)
-            body.insert(btc,pTag)
+            ptc += 1
-            btc += 1
+            pTag.insert(ptc, NavigableString('%s' % friendly_name))
+
+        body.insert(btc,pTag)
+        btc += 1
 
         divTag = Tag(soup, "div")
         dtc = 0
@@ -1893,11 +1890,10 @@ Author '{0}':
         self.updateProgressFullStep("'Genres'")
 
         self.genre_tags_dict = self.filterDbTags(self.db.all_tags())
-
         # Extract books matching filtered_tags
         genre_list = []
-        for friendly_tag in sorted(self.genre_tags_dict):
+        for friendly_tag in sorted(self.genre_tags_dict, key=sort_key):
-            #print "\ngenerateHTMLByTags(): looking for books with friendly_tag '%s'" % friendly_tag
+            #print("\ngenerateHTMLByTags(): looking for books with friendly_tag '%s'" % friendly_tag)
             # tag_list => { normalized_genre_tag : [{book},{},{}],
             #               normalized_genre_tag : [{book},{},{}] }
 
@@ -2266,7 +2262,7 @@ Author '{0}':
             navPointTag.insert(1, contentTag)
 
         cmiTag = Tag(soup, '%s' % 'calibre:meta-img')
-        cmiTag['name'] = "mastheadImage"
+        cmiTag['id'] = "mastheadImage"
         cmiTag['src'] = "images/mastheadImage.gif"
         navPointTag.insert(2,cmiTag)
         navMapTag.insert(0,navPointTag)
@@ -2550,7 +2546,10 @@ Author '{0}':
             navLabelTag.insert(0, textTag)
             navPointByLetterTag.insert(0,navLabelTag)
             contentTag = Tag(soup, 'content')
-            contentTag['src'] = "content/%s.html#%s" % (output, title_letters[i])
+            if title_letters[i] == self.SYMBOLS:
+                contentTag['src'] = "content/%s.html#%s" % (output, title_letters[i])
+            else:
+                contentTag['src'] = "content/%s.html#%s" % (output, self.generateUnicodeName(title_letters[i]))
             navPointByLetterTag.insert(1,contentTag)
 
             if self.generateForKindle:
@@ -2638,7 +2637,7 @@ Author '{0}':
             navLabelTag.insert(0, textTag)
             navPointByLetterTag.insert(0,navLabelTag)
             contentTag = Tag(soup, 'content')
-            contentTag['src'] = "%s#%sauthors" % (HTML_file, authors_by_letter[1])
+            contentTag['src'] = "%s#%s_authors" % (HTML_file, self.generateUnicodeName(authors_by_letter[1]))
 
             navPointByLetterTag.insert(1,contentTag)
 
@@ -3211,7 +3210,7 @@ Author '{0}':
             ans = '%s%d %s:\n' %  (' ' * indent, len(tags), header)
             ans += ' ' * (indent + 1)
             out_str = ''
-            sorted_tags = sorted(tags)
+            sorted_tags = sorted(tags, key=sort_key)
             for tag in next_tag(sorted_tags):
                 out_str += tag
                 if len(out_str) >= line_break:
@@ -3232,7 +3231,7 @@ Author '{0}':
             if tag == ' ':
                 continue
 
-            normalized_tags.append(re.sub('\W','',tag).lower())
+            normalized_tags.append(re.sub('\W','',ascii_text(tag)).lower())
             friendly_tags.append(tag)
 
         genre_tags_dict = dict(zip(friendly_tags,normalized_tags))
@@ -3291,18 +3290,24 @@ Author '{0}':
         body = soup.find('body')
 
         btc = 0
+        divTag = Tag(soup, 'div')
+        dtc = 0
+
 
         # Insert section tag if this is the section start - first article only
         if section_head:
             aTag = Tag(soup,'a')
-            aTag['name'] = 'section_start'
+            aTag['id'] = 'section_start'
-            body.insert(btc, aTag)
+            divTag.insert(dtc, aTag)
-            btc += 1
+            dtc += 1
+            #body.insert(btc, aTag)
+            #btc += 1
 
         # Create an anchor from the tag
         aTag = Tag(soup, 'a')
-        aTag['name'] = "Genre_%s" % genre
+        aTag['id'] = "Genre_%s" % genre
-        body.insert(btc,aTag)
+        divTag.insert(dtc, aTag)
+        body.insert(btc,divTag)
         btc += 1
 
         titleTag = body.find(attrs={'class':'title'})
@@ -3475,7 +3480,7 @@ Author '{0}':
             for (i, tag) in enumerate(sorted(book.get('tags', []))):
                 aTag = Tag(_soup,'a')
                 if self.opts.generate_genres:
-                    aTag['href'] = "Genre_%s.html" % re.sub("\W","",tag.lower())
+                    aTag['href'] = "Genre_%s.html" % re.sub("\W","",ascii_text(tag).lower())
                 aTag.insert(0,escape(NavigableString(tag)))
                 genresTag.insert(gtc, aTag)
                 gtc += 1
@@ -3542,8 +3547,10 @@ Author '{0}':
         btc = 0
         # Insert the title anchor for inbound links
         aTag = Tag(soup, "a")
-        aTag['name'] = "book%d" % int(book['id'])
+        aTag['id'] = "book%d" % int(book['id'])
-        body.insert(btc, aTag)
+        divTag = Tag(soup, 'div')
+        divTag.insert(0, aTag)
+        body.insert(btc, divTag)
         btc += 1
 
         # Insert the link to the series or remove <a class="series">
@@ -3822,6 +3829,14 @@ Author '{0}':
                 with zf:
                     zf.writestr(title['uuid']+cover_crc, thumb_data)
 
+    def generateUnicodeName(self, c):
+        '''
+        Generate an anchor name string
+        '''
+        fullname = unicodedata.name(unicode(c))
+        terms = fullname.split()
+        return "_".join(terms)
+
     def getFriendlyGenreTag(self, genre):
         # Find the first instance of friendly_tag matching genre
         for friendly_tag in self.genre_tags_dict:
@@ -3835,8 +3850,8 @@ Author '{0}':
         return markerTags
 
     def letter_or_symbol(self,char):
-        if not re.search('[a-zA-Z]',char):
+        if not re.search('[a-zA-Z]', ascii_text(char)):
-            return 'Symbols'
+            return self.SYMBOLS
         else:
             return char
 

src/calibre/library/cli.py

@@ -999,6 +999,55 @@ def command_saved_searches(args, dbpath):
 
     return 0
 
+def backup_metadata_option_parser():
+    parser = get_parser(_('''\
+%prog backup_metadata [options]
+
+Backup the metadata stored in the database into individual OPF files in each
+books directory. This normally happens automatically, but you can run this
+command to force re-generation of the OPF files, with the --all option.
+
+Note that there is normally no need to do this, as the OPF files are backed up
+automatically, every time metadata is changed.
+'''))
+    parser.add_option('--all', default=False, action='store_true',
+        help=_('Normally, this command only operates on books that have'
+            ' out of date OPF files. This option makes it operate on all'
+            ' books.'))
+    return parser
+
+class BackupProgress(object):
+
+    def __init__(self):
+        self.total = 0
+        self.count = 0
+
+    def __call__(self, book_id, mi, ok):
+        if mi is True:
+            self.total = book_id
+        else:
+            self.count += 1
+            prints(u'%.1f%% %s - %s'%((self.count*100)/float(self.total),
+                book_id, mi.title))
+
+def command_backup_metadata(args, dbpath):
+    parser = backup_metadata_option_parser()
+    opts, args = parser.parse_args(args)
+    if len(args) != 0:
+        parser.print_help()
+        return 1
+
+    if opts.library_path is not None:
+        dbpath = opts.library_path
+    if isbytestring(dbpath):
+        dbpath = dbpath.decode(preferred_encoding)
+    db = LibraryDatabase2(dbpath)
+    book_ids = None
+    if opts.all:
+        book_ids = db.all_ids()
+    db.dump_metadata(book_ids=book_ids, callback=BackupProgress())
+
+
 def check_library_option_parser():
     from calibre.library.check_library import CHECKS
     parser = get_parser(_('''\
@@ -1275,7 +1324,7 @@ COMMANDS = ('list', 'add', 'remove', 'add_format', 'remove_format',
             'show_metadata', 'set_metadata', 'export', 'catalog',
             'saved_searches', 'add_custom_column', 'custom_columns',
             'remove_custom_column', 'set_custom', 'restore_database',
-            'check_library', 'list_categories')
+            'check_library', 'list_categories', 'backup_metadata')
 
 
 def option_parser():

src/calibre/library/database2.py

@@ -251,6 +251,14 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         defs['similar_tags_match_kind'] = 'match_all'
         defs['similar_series_search_key'] = 'series'
         defs['similar_series_match_kind'] = 'match_any'
+        defs['book_display_fields'] = [
+        ('title', False), ('authors', True), ('formats', True),
+        ('series', True), ('identifiers', True), ('tags', True),
+        ('path', True), ('publisher', False), ('rating', False),
+        ('author_sort', False), ('sort', False), ('timestamp', False),
+        ('uuid', False), ('comments', True), ('id', False), ('pubdate', False),
+        ('last_modified', False), ('size', False), ('languages', False),
+        ]
 
         # Migrate the bool tristate tweak
         defs['bools_are_tristate'] = \
@@ -808,18 +816,30 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                 pass
 
     def dump_metadata(self, book_ids=None, remove_from_dirtied=True,
-            commit=True):
+            commit=True, callback=None):
         '''
-        Write metadata for each record to an individual OPF file
+        Write metadata for each record to an individual OPF file. If callback
+        is not None, it is called once at the start with the number of book_ids
+        being processed. And once for every book_id, with arguments (book_id,
+        mi, ok).
         '''
         if book_ids is None:
             book_ids = [x[0] for x in self.conn.get(
                 'SELECT book FROM metadata_dirtied', all=True)]
+
+        if callback is not None:
+            book_ids = tuple(book_ids)
+            callback(len(book_ids), True, False)
+
         for book_id in book_ids:
             if not self.data.has_id(book_id):
+                if callback is not None:
+                    callback(book_id, None, False)
                 continue
             path, mi, sequence = self.get_metadata_for_dump(book_id)
             if path is None:
+                if callback is not None:
+                    callback(book_id, mi, False)
                 continue
             try:
                 raw = metadata_to_opf(mi)
@@ -829,6 +849,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                     self.clear_dirtied(book_id, sequence)
             except:
                 pass
+            if callback is not None:
+                callback(book_id, mi, True)
         if commit:
             self.conn.commit()
 
@@ -1411,7 +1433,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         opath = self.format_abspath(book_id, nfmt, index_is_id=True)
         return fmt if opath is None else nfmt
 
-    def delete_book(self, id, notify=True, commit=True, permanent=False):
+    def delete_book(self, id, notify=True, commit=True, permanent=False,
+            do_clean=True):
         '''
         Removes book from the result cache and the underlying database.
         If you set commit to False, you must call clean() manually afterwards
@@ -1428,7 +1451,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         self.conn.execute('DELETE FROM books WHERE id=?', (id,))
         if commit:
             self.conn.commit()
-            self.clean()
+            if do_clean:
+                self.clean()
         self.data.books_deleted([id])
         if notify:
             self.notify('delete', [id])
@@ -3721,4 +3745,42 @@ books_series_link      feeds
             'SELECT {0}, count(*) FROM books_{1}_link GROUP BY {0}'.format(
                 fm['link_column'], fm['table']))
 
+    def all_author_names(self):
+        ai = self.FIELD_MAP['authors']
+        ans = set()
+        for rec in self.data.iterall():
+            auts = rec[ai]
+            if auts:
+                for x in auts.split(','):
+                    ans.add(x.replace('|', ','))
+        return ans
+
+    def all_tag_names(self):
+        ai = self.FIELD_MAP['tags']
+        ans = set()
+        for rec in self.data.iterall():
+            auts = rec[ai]
+            if auts:
+                for x in auts.split(','):
+                    ans.add(x)
+        return ans
+
+    def all_publisher_names(self):
+        ai = self.FIELD_MAP['publisher']
+        ans = set()
+        for rec in self.data.iterall():
+            auts = rec[ai]
+            if auts:
+                ans.add(auts)
+        return ans
+
+    def all_series_names(self):
+        ai = self.FIELD_MAP['series']
+        ans = set()
+        for rec in self.data.iterall():
+            auts = rec[ai]
+            if auts:
+                ans.add(auts)
+        return ans
+
 

src/calibre/library/prefs.py

@@ -34,6 +34,9 @@ class DBPrefs(dict):
     def to_raw(self, val):
         return json.dumps(val, indent=2, default=to_json)
 
+    def has_setting(self, key):
+        return key in self
+
     def __getitem__(self, key):
         try:
             return dict.__getitem__(self, key)

src/calibre/library/server/mobile.py

@@ -97,6 +97,7 @@ def build_index(books, num, search, sort, order, start, total, url_base, CKEYS,
 
     search_box = build_search_box(num, search, sort, order, prefix)
     navigation = build_navigation(start, num, total, prefix+url_base)
+    navigation2 = build_navigation(start, num, total, prefix+url_base)
     bookt = TABLE(id='listing')
 
     body = BODY(
@@ -104,7 +105,9 @@ def build_index(books, num, search, sort, order, start, total, url_base, CKEYS,
         search_box,
         navigation,
         HR(CLASS('spacer')),
-        bookt
+        bookt,
+        HR(CLASS('spacer')),
+        navigation2
     )
 
     # Book list {{{
@@ -155,7 +158,6 @@ def build_index(books, num, search, sort, order, start, total, url_base, CKEYS,
         bookt.append(TR(thumbnail, data))
     # }}}
 
-    body.append(HR())
     body.append(DIV(
         A(_('Switch to the full interface (non-mobile interface)'),
             href=prefix+"/browse",

src/calibre/linux.py

@@ -5,6 +5,7 @@ __copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
 
 import sys, os, cPickle, textwrap, stat
 from subprocess import check_call
+from functools import partial
 
 from calibre import  __appname__, prints, guess_type
 from calibre.constants import islinux, isnetbsd, isbsd
@@ -346,19 +347,28 @@ class PostInstall:
         try:
             self.info('Setting up desktop integration...')
 
+            env = os.environ.copy()
+            cc = check_call
+            if getattr(sys, 'frozen_path', False) and 'LD_LIBRARY_PATH' in env:
+                paths = env.get('LD_LIBRARY_PATH', '').split(os.pathsep)
+                paths = [x for x in paths if x]
+                npaths = [x for x in paths if x != sys.frozen_path+'/lib']
+                env['LD_LIBRARY_PATH'] = os.pathsep.join(npaths)
+                cc = partial(check_call, env=env)
+
             with TemporaryDirectory() as tdir, CurrentDir(tdir), \
                                 PreserveMIMEDefaults():
                 render_img('mimetypes/lrf.png', 'calibre-lrf.png')
-                check_call('xdg-icon-resource install --noupdate --context mimetypes --size 128 calibre-lrf.png application-lrf', shell=True)
+                cc('xdg-icon-resource install --noupdate --context mimetypes --size 128 calibre-lrf.png application-lrf', shell=True)
                 self.icon_resources.append(('mimetypes', 'application-lrf', '128'))
-                check_call('xdg-icon-resource install --noupdate --context mimetypes --size 128 calibre-lrf.png text-lrs', shell=True)
+                cc('xdg-icon-resource install --noupdate --context mimetypes --size 128 calibre-lrf.png text-lrs', shell=True)
                 self.icon_resources.append(('mimetypes', 'application-lrs',
                 '128'))
-                render_img('lt.png', 'calibre-gui.png')
+                render_img('lt.png', 'calibre-gui.png', width=256, height=256)
-                check_call('xdg-icon-resource install --noupdate --size 128 calibre-gui.png calibre-gui', shell=True)
+                cc('xdg-icon-resource install --noupdate --size 256 calibre-gui.png calibre-gui', shell=True)
                 self.icon_resources.append(('apps', 'calibre-gui', '128'))
                 render_img('viewer.png', 'calibre-viewer.png')
-                check_call('xdg-icon-resource install --size 128 calibre-viewer.png calibre-viewer', shell=True)
+                cc('xdg-icon-resource install --size 128 calibre-viewer.png calibre-viewer', shell=True)
                 self.icon_resources.append(('apps', 'calibre-viewer', '128'))
 
                 mimetypes = set([])
@@ -385,14 +395,14 @@ class PostInstall:
                         'calibre-ebook-viewer.desktop')
                 for x in des:
                     cmd = ['xdg-desktop-menu', 'install', '--noupdate', './'+x]
-                    check_call(' '.join(cmd), shell=True)
+                    cc(' '.join(cmd), shell=True)
                     self.menu_resources.append(x)
-                check_call(['xdg-desktop-menu', 'forceupdate'])
+                cc(['xdg-desktop-menu', 'forceupdate'])
                 f = open('calibre-mimetypes', 'wb')
                 f.write(MIME)
                 f.close()
                 self.mime_resources.append('calibre-mimetypes')
-                check_call('xdg-mime install ./calibre-mimetypes', shell=True)
+                cc('xdg-mime install ./calibre-mimetypes', shell=True)
         except Exception:
             if self.opts.fatal_errors:
                 raise

src/calibre/test_build.py

@@ -76,15 +76,15 @@ def test_qt():
     print ('Qt OK!')
 
 def test_imaging():
-    from calibre.utils.magick.draw import create_canvas, Image
+    from calibre.ebooks import calibre_cover
-    im = create_canvas(20, 20, '#ffffff')
+    data = calibre_cover('test', 'ok')
-    jpg = im.export('jpg')
+    if len(data) > 1000:
-    Image().load(jpg)
+        print ('ImageMagick OK!')
-    im.export('png')
+    else:
-    print ('ImageMagick OK!')
+        raise RuntimeError('ImageMagick choked!')
     from PIL import Image
-    i = Image.open(cStringIO.StringIO(jpg))
+    i = Image.open(cStringIO.StringIO(data))
-    if i.size != (20, 20):
+    if i.size < (20, 20):
         raise RuntimeError('PIL choked!')
     print ('PIL OK!')
 
@@ -94,6 +94,12 @@ def test_unrar():
         raise RuntimeError('Failed to load libunrar')
     print ('Unrar OK!')
 
+def test_icu():
+    from calibre.utils.icu import _icu_not_ok
+    if _icu_not_ok:
+        raise RuntimeError('ICU module not loaded/valid')
+    print ('ICU OK!')
+
 def test():
     test_plugins()
     test_lxml()
@@ -102,6 +108,7 @@ def test():
     test_qt()
     test_imaging()
     test_unrar()
+    test_icu()
     if iswindows:
         test_win32()
         test_winutil()