Sync with Trunk.

Changelog.yaml

@@ -4,6 +4,107 @@
 # for important features/bug fixes.
 # Also, each release can have new and improved recipes.
 
+- version: 0.7.32
+  date: 2010-12-03
+
+  new features:
+    - title: "All new linux binary build. With updated libraries and replacing cx_Freeze with my own C python launcher code."
+
+    - title: "Edit metadata dialog: Add Next and Previous buttons and show cover size in tooltip"
+      tickets: [7706, 7711]
+
+    - title: "A new custom column type: Enumeration. This column can take one of a user defined set of values."
+
+    - title: "PML Output: Add option to reduce image sizes/bit depth to allow PML Output to be used with DropBook"
+
+    - title: "TXT Output: Add option to generate Markdown output. Turn <br> tags into spaces."
+
+    - title: "Add a count function to the template language. Make author_sort searchable."
+
+    - title: "Various consistency and usability enhancements to the search box."
+      tickets: [7726]
+      description: >
+        "Always select first book in result set of a search. Similar books searches added to search history. Search history order is no longer randomized. When focussing the search box with a keyboard shortcut, select all text. If you press enter in the search box, the search is executed and the book list os automatically focussed."
+
+    - title: "Driver for samsung fascinate and PocketBook 902"
+
+    - title: "FB2 Output: Add option to create FB2 sections based on internal file structure of input file (useful for EPUB files that have been split on chapter boundaries). Also add options to mark h1/h2/h3 tags as section titles in the FB2 file."
+      tickets: [7738]
+
+    - title: "Metadata jacket: Add publisher information to jacket."
+
+    - title: "Catalog generation: Allow use of custom columns as well as tags to indicate read books. Note that your previously saved read books setting will be lost."
+
+    - title: "Bulk metadata edit dialog: Add an Apply button to allow you to perform multiple operations in sequence"
+
+    - title: "Allow drag and drop of books onto user categories. If you drag a book from a particular column (say authors) and drop it onto a user category, the column value will be added to the user category. So for authors, the authros will be added to the user category."
+
+    - title: "Check Library can now check and repair the has_cover cache"
+
+    - title: "Allow GUI plugins to be distributed in ZIP files. See http://www.mobileread.com/forums/showthread.php?t=108774"
+
+    - title: "Allow searching by the number of tags/authors/formats/etc. See User Manual for details."
+
+    - title: "Tiny speed up when loading large libraries and make various metadata editing tasks a little faster by reducing the number of times the Tag Browser is refreshed"
+
+  bug fixes:
+    - title: "E-book viewer: Fix broken backwards searching"
+
+    - title: "Fix custom ratings column values being displayed incorrectly in book details area"
+      tickets: [7740]
+
+    - title: "Fix book details dialog not using internal viewer to view ebooks"
+      tickets: [7424]
+
+    - title: "MOBI Output: When the input document does not explicitly specify a size for images, set the size to be the natural size of the image. This works around Amazon's *truly wonderful* MOBI renderer's tendency to expand images that do not have a width and height specified."
+
+    - title: "Conversion pipeline: Fix bug that caused height/width specified in %/em of screen size to be incorrectly calculated by a factor of 72./DPI"
+
+    - title: "Conversion pipeline: Respect max-width and max-height when calculating the effective size of an element"
+
+    - title: "Conversion pipeline: Do not override CSS for images with the value of the img width/height attributes, unless no CSS is specified for the image"
+
+    - title: "E-book viewer: Resize automatically to fit on smaller screens"
+
+    - title: "Use the same MIME database on all platforms that calibre runs on, works around python 2.7's crazy insistence on reading MIME data from the registry"
+
+    - title: "Kobo driver: Allow html, txt and rtf documents to be deleted"
+
+    - title: "Always overwrite title/author metadata when downloading metadata for books added by ISBN"
+
+    - title: "Nook Color profile: Reduce screen height to 900px"
+
+    - title: "Fix regression that broke RTF conversion on some linux systems"
+
+    - title: "Fix bug that could break searching after copying and deleting a book from the current library"
+      tickets: [7459]
+
+  improved recipes:
+    - NZZ
+    - Frankfurter Rundschau
+    - JiJi Press
+    - Revista Muy Intersante
+
+  new recipes:
+    - title: "Global Times"
+      author: "malfi"
+      
+    - title: "The Philosopher's Magazine"
+      author: "Darko Miletic"
+
+    - title: "Poughkeepsie Journal"
+      author: "weebl"
+
+    - title: "Business Spectator and ABC Australia"
+      author: "Dean Cording"
+
+    - title: "La Rijoa and NacionRed"
+      author: "Arturo Martinez Nieves"
+
+    - title: "Animal Politico"
+      author: "leamsi"
+
+
 - version: 0.7.31
   date: 2010-11-27
 

resources/default_tweaks.py

@@ -217,3 +217,15 @@ generate_cover_foot_font = None
 # open_viewer, do_nothing, edit_cell. Default: open_viewer.
 # Example: doubleclick_on_library_view = 'do_nothing'
 doubleclick_on_library_view = 'open_viewer'
+
+
+# Language to use when sorting. Setting this tweak will force sorting to use the
+# collating order for the specified language. This might be useful if you run
+# calibre in English but want sorting to work in the language where you live.
+# Set the tweak to the desired ISO 639-1 language code, in lower case.
+# You can find the list of supported locales at
+# http://publib.boulder.ibm.com/infocenter/iseries/v5r3/topic/nls/rbagsicusortsequencetables.htm
+# Default: locale_for_sorting = '' -- use the language calibre displays in
+# Example: locale_for_sorting = 'fr' -- sort using French rules.
+# Example: locale_for_sorting = 'nb' -- sort using Norwegian rules.
+locale_for_sorting =  ''

resources/jacket/stylesheet.css

@@ -36,22 +36,37 @@
 /*
 **	Title
 */
-.cbj_title {
+table.cbj_header td.cbj_title {
 	font-size: x-large;
+	font-style: italic;
+	text-align: center;
+}
+
+/*
+**	Series
+*/
+table.cbj_header td.cbj_series {
+	font-size: medium;
 	text-align: center;
 }
 
 /*
 **	Author
 */
-.cbj_author {
+table.cbj_header td.cbj_author {
 	font-size: medium;
 	text-align: center;
-    margin-bottom: 1ex;
 }
 
 /*
-**	Table containing Series, Publication Year, Rating and Tags
+**	Publisher/published
+*/
+table.cbj_header td.cbj_pubdata {
+	text-align: center;
+}
+
+/*
+**	Table containing Rating and Tags
 */
 table.cbj_header {
 	width: 100%;
@@ -62,9 +77,8 @@ table.cbj_header {
 */
 table.cbj_header td.cbj_label {
 	font-family: sans-serif;
-	font-weight: bold;
 	text-align: right;
-	width: 40%;
+	width: 33%;
 	}
 
 /*
@@ -73,9 +87,23 @@ table.cbj_header td.cbj_label {
 table.cbj_header td.cbj_content {
 	font-family: sans-serif;
 	text-align: left;
-	width:60%;
+	width:67%;
 	}
 
+/*
+** Metadata divider
+*/
+hr.metadata_divider {
+	width:90%;
+	margin-left:5%;
+	border-top: solid white 0px;
+	border-right: solid white 0px;
+    border-bottom: solid black 1px;
+    border-left: solid white 0px;
+	}
+
+
+
 /*
 **	To skip a banner item (Series|Published|Rating|Tags),
 **	edit the appropriate CSS rule below.

resources/jacket/template.xhtml

@@ -6,17 +6,24 @@
     </head>
     <body>
         <div class="cbj_banner">
-            <div class="cbj_title">{title}</div>
-            <div class="cbj_author">{author}</div>
             <table class="cbj_header">
-                <tr class="cbj_series">
+            	<tr>
-                    <td class="cbj_label">{series_label}:</td>
+            		<td class="cbj_title" colspan="2">{title}</td>
-                    <td class="cbj_content">{series}</td>
+            	</tr>
+                <tr>
+                    <td  class="cbj_series" colspan="2">{series}</td>
                 </tr>
-                <tr class="cbj_pubdate">
+            	<tr>
-                    <td class="cbj_label">{pubdate_label}:</td>
+            		<td class="cbj_author" colspan="2">{author}</td>
-                    <td class="cbj_content">{pubdate}</td>
+            	</tr>
+                <tr>
+                	<td class="cbj_pubdata" colspan="2">{publisher} ({pubdate})</td>
                 </tr>
+
+            	<tr>
+            		<td class="cbj_author" colspan="2"><hr class="metadata_divider" /></td>
+            	</tr>
+
                 <tr class="cbj_rating">
                     <td class="cbj_label">{rating_label}:</td>
                     <td class="cbj_content">{rating}</td>

resources/recipes/abc_au.recipe

@@ -0,0 +1,54 @@
+__license__   = 'GPL v3'
+__copyright__ = '2010, Dean Cording'
+'''
+abc.net.au/news
+'''
+import re
+from calibre.web.feeds.recipes import BasicNewsRecipe
+
+class ABCNews(BasicNewsRecipe):
+    title                  = 'ABC News'
+    __author__             = 'Dean Cording'
+    description            = 'News from Australia'
+    masthead_url           = 'http://www.abc.net.au/news/assets/v5/images/common/logo-news.png'
+    cover_url              = 'http://www.abc.net.au/news/assets/v5/images/common/logo-news.png'
+
+    oldest_article         = 2
+    max_articles_per_feed  = 100
+    no_stylesheets         = False
+    #delay                  = 1
+    use_embedded_content   = False
+    encoding               = 'utf8'
+    publisher              = 'ABC News'
+    category               = 'News, Australia, World'
+    language               = 'en_AU'
+    publication_type       = 'newsportal'
+    preprocess_regexps     = [(re.compile(r'<!--.*?-->', re.DOTALL), lambda m: '')]
+    conversion_options = {
+                             'comments'        : description
+                            ,'tags'            : category
+                            ,'language'        : language
+                            ,'publisher'       : publisher
+                            ,'linearize_tables': False
+                         }
+
+    keep_only_tags    =  dict(id='article')
+
+    remove_tags = [dict(attrs={'class':['related', 'tags']}),
+                     dict(id='statepromo')
+                        ]
+
+    remove_attributes = ['width','height']
+
+    feeds          = [
+                      ('Top Stories', 'http://www.abc.net.au/news/syndicate/topstoriesrss.xml'),
+                      ('Canberra', 'http://www.abc.net.au/news/indexes/idx-act/rss.xml'),
+                      ('Sydney', 'http://www.abc.net.au/news/indexes/sydney/rss.xml'),
+                      ('Melbourne', 'http://www.abc.net.au/news/indexes/melbourne/rss.xml'),
+                      ('Brisbane', 'http://www.abc.net.au/news/indexes/brisbane/rss.xml'),
+                      ('Perth', 'http://www.abc.net.au/news/indexes/perth/rss.xml'),
+                      ('Australia', 'http://www.abc.net.au/news/indexes/idx-australia/rss.xml'),
+                      ('World', 'http://www.abc.net.au/news/indexes/world/rss.xml'),
+                      ('Business', 'http://www.abc.net.au/news/indexes/business/rss.xml'),
+                      ('Science and Technology', 'http://www.abc.net.au/news/tag/science-and-technology/rss.xml'),
+                    ]

resources/recipes/business_spectator.recipe

@@ -0,0 +1,48 @@
+__license__   = 'GPL v3'
+__copyright__ = '2010, Dean Cording'
+'''
+abc.net.au/news
+'''
+import re
+from calibre.web.feeds.recipes import BasicNewsRecipe
+
+class BusinessSpectator(BasicNewsRecipe):
+    title                  = 'Business Spectator'
+    __author__             = 'Dean Cording'
+    description            = 'Australian Business News & commentary delivered the way you want it.'
+    masthead_url           = 'http://www.businessspectator.com.au/bs.nsf/logo-business-spectator.gif'
+    cover_url              = masthead_url
+
+    oldest_article         = 2
+    max_articles_per_feed  = 100
+    no_stylesheets         = True
+    #delay                  = 1
+    use_embedded_content   = False
+    encoding               = 'utf8'
+    publisher              = 'Business Spectator'
+    category               = 'News, Australia, Business'
+    language               = 'en_AU'
+    publication_type       = 'newsportal'
+    preprocess_regexps     = [(re.compile(r'<!--.*?-->', re.DOTALL), lambda m: '')]
+    conversion_options = {
+                             'comments'        : description
+                            ,'tags'            : category
+                            ,'language'        : language
+                            ,'publisher'       : publisher
+                            ,'linearize_tables': False
+                         }
+
+    keep_only_tags    =  [dict(id='storyHeader'), dict(id='body-html')]
+
+    remove_tags = [dict(attrs={'class':'hql'})]
+
+    remove_attributes = ['width','height','style']
+
+    feeds          = [
+                      ('Top Stories', 'http://www.businessspectator.com.au/top-stories.rss'),
+                      ('Alan Kohler', 'http://www.businessspectator.com.au/bs.nsf/RSS?readform&type=spectators&cat=Alan%20Kohler'),
+                      ('Robert Gottliebsen', 'http://www.businessspectator.com.au/bs.nsf/RSS?readform&type=spectators&cat=Robert%20Gottliebsen'),
+                      ('Stephen Bartholomeusz', 'http://www.businessspectator.com.au/bs.nsf/RSS?readform&type=spectators&cat=Stephen%20Bartholomeusz'),
+                      ('Daily Dossier', 'http://www.businessspectator.com.au/bs.nsf/RSS?readform&type=kgb&cat=dossier'),
+                      ('Australia', 'http://www.businessspectator.com.au/bs.nsf/RSS?readform&type=region&cat=australia'),
+                    ]

resources/recipes/esenja.recipe

@@ -0,0 +1,87 @@
+#!/usr/bin/env  python
+
+__license__   = 'GPL v3'
+__copyright__ = '2010, matek09, matek09@gmail.com'
+
+from calibre.web.feeds.news import BasicNewsRecipe
+import re
+
+class Esensja(BasicNewsRecipe):
+
+	title = u'Esensja'
+	__author__ = 'matek09'
+	description = 'Monthly magazine'
+	encoding = 'utf-8'
+	no_stylesheets = True
+	language = 'pl'
+	remove_javascript = True
+	HREF = '0'
+
+	#keep_only_tags =[]
+	#keep_only_tags.append(dict(name = 'div', attrs = {'class' : 'article'})
+	remove_tags_before = dict(dict(name = 'div', attrs = {'class' : 't-title'}))
+	remove_tags_after = dict(dict(name = 'img', attrs = {'src' : '../../../2000/01/img/tab_bot.gif'}))
+
+	remove_tags =[]
+	remove_tags.append(dict(name = 'img', attrs = {'src' : '../../../2000/01/img/tab_top.gif'}))
+	remove_tags.append(dict(name = 'img', attrs = {'src' : '../../../2000/01/img/tab_bot.gif'}))
+	remove_tags.append(dict(name = 'div', attrs = {'class' : 't-title2 nextpage'}))
+
+	extra_css = '''
+					.t-title {font-size: x-large; font-weight: bold; text-align: left}
+					.t-author {font-size: x-small; text-align: left}
+					.t-title2 {font-size: x-small; font-style: italic; text-align: left}
+					.text {font-size: small; text-align: left}
+					.annot-ref {font-style: italic; text-align: left}
+				'''
+
+	preprocess_regexps = [(re.compile(r'alt="[^"]*"'),
+						lambda match: '')]
+
+	def parse_index(self):
+		soup = self.index_to_soup('http://www.esensja.pl/magazyn/')
+		a = soup.find('a', attrs={'href' : re.compile('.*/index.html')})
+		year = a['href'].split('/')[0]
+		month = a['href'].split('/')[1]
+		self.HREF = 'http://www.esensja.pl/magazyn/' + year + '/' + month + '/iso/'
+		soup = self.index_to_soup(self.HREF + '01.html')
+		self.cover_url = 'http://www.esensja.pl/magazyn/' + year + '/' + month + '/img/ilustr/cover_b.jpg'
+		feeds = []
+		intro = soup.find('div', attrs={'class' : 'n-title'})
+		introduction = {'title' : self.tag_to_string(intro.a),
+						'url' : self.HREF + intro.a['href'],
+						'date' : '',
+						'description' : ''}
+		chapter = 'Wprowadzenie'
+		subchapter = ''
+		articles = []
+		articles.append(introduction)
+		for tag in intro.findAllNext(attrs={'class': ['chapter', 'subchapter', 'n-title']}):
+			if tag.name in 'td':
+				if len(articles) > 0:
+					section = chapter
+					if len(subchapter) > 0:
+						section += ' - ' + subchapter
+					feeds.append((section, articles))
+					articles = []
+				if tag['class'] == 'chapter':
+					chapter = self.tag_to_string(tag).capitalize()
+					subchapter = ''
+				else:
+					subchapter = self.tag_to_string(tag)
+					subchapter = self.tag_to_string(tag)
+				continue
+			articles.append({'title' : self.tag_to_string(tag.a), 'url' : self.HREF + tag.a['href'], 'date' : '', 'description' : ''})
+
+			a = self.index_to_soup(self.HREF + tag.a['href'])
+			i = 1
+			while True:
+				div = a.find('div', attrs={'class' : 't-title2 nextpage'})
+				if div is not None:
+					a = self.index_to_soup(self.HREF + div.a['href'])
+					articles.append({'title' : self.tag_to_string(tag.a) + ' c. d. ' + str(i), 'url' : self.HREF + div.a['href'], 'date' : '', 'description' : ''})
+					i = i + 1
+				else:
+					break
+
+		return feeds

resources/recipes/fr_online.recipe

@@ -1,67 +1,61 @@
-__license__   = 'GPL v3'
+#!/usr/bin/env  python
-__copyright__ = '2009, Justus Bisser <justus.bisser at gmail.com>'
+
+__license__            = 'GPL v3'
+__copyright__          = '2010, Christian Schmitt'
+
 '''
 fr-online.de
 '''
-import re
 
-from calibre.web.feeds.news import BasicNewsRecipe
+from calibre.web.feeds.recipes import BasicNewsRecipe
 
-class Spiegel_ger(BasicNewsRecipe):
+class FROnlineRecipe(BasicNewsRecipe):
-    title                 = 'Frankfurter Rundschau'
+  title                  = 'Frankfurter Rundschau'
-    __author__            = 'Justus Bisser'
+  __author__             = 'maccs'
-    description           = "Dies ist die Online-Ausgabe der Frankfurter Rundschau. Um die abgerufenen individuell einzustellen bearbeiten sie die Liste im erweiterten Modus. Die Feeds findet man auf http://www.fr-online.de/verlagsservice/fr_newsreader/?em_cnt=574255"
+  description            = 'Nachrichten aus D und aller Welt'
-    publisher             = 'Druck- und Verlagshaus Frankfurt am Main GmbH'
+  encoding               = 'utf-8'
-    category              = 'FR Online, Frankfurter Rundschau, Nachrichten, News,Dienste, RSS, RSS, Feedreader, Newsfeed, iGoogle, Netvibes, Widget'
+  masthead_url =  'http://www.fr-online.de/image/view/-/1474018/data/823552/-/logo.png'
-    oldest_article        = 7
+  publisher              = 'Druck- und Verlagshaus Frankfurt am Main GmbH'
-    max_articles_per_feed = 100
+  category               = 'news, germany, world'
-    language              = 'de'
+  language               = 'de'
-    lang                  = 'de-DE'
+  publication_type       = 'newspaper'
-    no_stylesheets        = True
+  use_embedded_content   = False
-    use_embedded_content  = False
+  remove_javascript      = True
-    #encoding              = 'cp1252'
+  no_stylesheets         = True
+  oldest_article         = 1   # Increase this number if you're interested in older articles
+  max_articles_per_feed  = 50  # Seems a reasonable number to me
+  extra_css              = '''
+                            body { font-family: "arial", "verdana", "geneva", sans-serif; font-size: 12px; margin: 0px; background-color: #ffffff;}
+                            .imgSubline{background-color: #f4f4f4; font-size: 0.8em;}
+                            .p--heading-1 {font-weight: bold;}
+                            .calibre_navbar {font-size: 0.8em; font-family: "arial", "verdana", "geneva", sans-serif;}
+                            '''
+  remove_tags            = [dict(name='div', attrs={'id':'Logo'})]
+  cover_url              = 'http://www.fr-online.de/image/view/-/1474018/data/823552/-/logo.png'
+  cover_margins          = (100, 150, '#ffffff')
 
-    conversion_options = {
-                          'comment'          : description
-                        , 'tags'             : category
-                        , 'publisher'        : publisher
-                        , 'language'         : lang
-                        }
 
-    recursions = 0
+  feeds = []
-    max_articles_per_feed = 100
+  feeds.append(('Startseite', u'http://www.fr-online.de/home/-/1472778/1472778/-/view/asFeed/-/index.xml'))
-    #keep_only_tags = [dict(name='div', attrs={'class':'text'})]
+  feeds.append(('Politik', u'http://www.fr-online.de/politik/-/1472596/1472596/-/view/asFeed/-/index.xml'))
-    #tags_remove = [dict(name='div', attrs={'style':'text-align: left; margin: 4px 0px 0px 4px; width: 200px; float: right;'})]
+  feeds.append(('Meinung', u'http://www.fr-online.de/politik/meinung/-/1472602/1472602/-/view/asFeed/-/index.xml'))
-    remove_attributes = ['style']
+  feeds.append(('Wirtschaft', u'http://www.fr-online.de/wirtschaft/-/1472780/1472780/-/view/asFeed/-/index.xml'))
-    feeds = []
+  feeds.append(('Sport', u'http://www.fr-online.de/sport/-/1472784/1472784/-/view/asFeed/-/index.xml'))
-    #remove_tags_before = [dict(name='div', attrs={'style':'padding-left: 0px;'})]
+  feeds.append(('Eintracht Frankfurt', u'http://www.fr-online.de/sport/eintracht-frankfurt/-/1473446/1473446/-/view/asFeed/-/index.xml'))
-    #remove_tags_after = [dict(name='div', attrs={'class':'box_head_text'})]
+  feeds.append(('Kultur und Medien', u'http://www.fr-online.de/kultur/-/1472786/1472786/-/view/asFeed/-/index.xml'))
+  feeds.append(('Panorama', u'http://www.fr-online.de/panorama/-/1472782/1472782/-/view/asFeed/-/index.xml'))
+  feeds.append(('Frankfurt', u'http://www.fr-online.de/frankfurt/-/1472798/1472798/-/view/asFeed/-/index.xml'))
+  feeds.append(('Rhein-Main', u'http://www.fr-online.de/rhein-main/-/1472796/1472796/-/view/asFeed/-/index.xml'))
+  feeds.append(('Hanau', u'http://www.fr-online.de/rhein-main/hanau/-/1472866/1472866/-/view/asFeed/-/index.xml'))
+  feeds.append(('Darmstadt', u'http://www.fr-online.de/rhein-main/darmstadt/-/1472858/1472858/-/view/asFeed/-/index.xml'))
+  feeds.append(('Wiesbaden', u'http://www.fr-online.de/rhein-main/wiesbaden/-/1472860/1472860/-/view/asFeed/-/index.xml'))
+  feeds.append(('Offenbach', u'http://www.fr-online.de/rhein-main/offenbach/-/1472856/1472856/-/view/asFeed/-/index.xml'))
+  feeds.append(('Bad Homburg', u'http://www.fr-online.de/rhein-main/bad-homburg/-/1472864/1472864/-/view/asFeed/-/index.xml'))
+  feeds.append(('Digital', u'http://www.fr-online.de/digital/-/1472406/1472406/-/view/asFeed/-/index.xml'))
+  feeds.append(('Wissenschaft', u'http://www.fr-online.de/wissenschaft/-/1472788/1472788/-/view/asFeed/-/index.xml'))
 
-    # enable for all news
-    allNews = 0
-    if allNews:
-        feeds = [(u'Frankfurter Rundschau', u'http://www.fr-online.de/rss/sport/index.xml')]
-    else:
-        #select the feeds you like
-        feeds = [(u'Nachrichten', u'http://www.fr-online.de/rss/politik/index.xml')]
-        feeds.append((u'Kommentare und Analysen', u'http://www.fr-online.de/rss/meinung/index.xml'))
-        feeds.append((u'Dokumentationen', u'http://www.fr-online.de/rss/dokumentation/index.xml'))
-        feeds.append((u'Deutschlandtrend', u'http://www.fr-online.de/rss/deutschlandtrend/index.xml'))
-        feeds.append((u'Wirtschaft', u'http://www.fr-online.de/rss/wirtschaft/index.xml'))
-        feeds.append((u'Sport', u'http://www.fr-online.de/rss/sport/index.xml'))
-        feeds.append((u'Feuilleton', u'http://www.fr-online.de/rss/feuilleton/index.xml'))
-        feeds.append((u'Panorama', u'http://www.fr-online.de/rss/panorama/index.xml'))
-        feeds.append((u'Rhein Main und Hessen', u'http://www.fr-online.de/rss/hessen/index.xml'))
-        feeds.append((u'Fitness und Gesundheit', u'http://www.fr-online.de/rss/fit/index.xml'))
-        feeds.append((u'Multimedia', u'http://www.fr-online.de/rss/multimedia/index.xml'))
-        feeds.append((u'Wissen und Bildung', u'http://www.fr-online.de/rss/wissen/index.xml'))
 
-    def get_article_url(self, article):
+  def print_version(self, url):
-        url = article.link
+    return url.replace('index.html', 'view/printVersion/-/index.html')
-        regex = re.compile("0C[0-9]{6,8}0A?")
 
-        liste = regex.findall(url)
-        string = liste.pop(0)
-        string = string[2:len(string)-1]
-        return "http://www.fr-online.de/_em_cms/_globals/print.php?em_cnt=" + string
 

resources/recipes/globaltimes.recipe

@@ -0,0 +1,46 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+import re
+
+class globaltimes(BasicNewsRecipe):
+    title          = u'Global Times'
+    __author__ = 'malfi'
+    language = 'zh'
+    oldest_article = 7
+    max_articles_per_feed = 100
+    no_stylesheets = True
+    cover_url = 'http://enhimg2.huanqiu.com/images/logo.png'
+    language = 'en'
+    keep_only_tags = []
+    keep_only_tags.append(dict(name = 'div', attrs = {'id': 'content'}))
+    remove_tags = []
+    remove_tags.append(dict(name = 'div', attrs = {'class': 'location'}))
+    remove_tags.append(dict(name = 'div', attrs = {'class': 'contentpage'}))
+    remove_tags.append(dict(name = 'li', attrs = {'id': 'pl'}))
+
+    extra_css = '''
+        h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;}
+        h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;}
+        p{font-family:Arial,Helvetica,sans-serif;font-size:small;}
+        body{font-family:Helvetica,Arial,sans-serif;font-size:small;}
+        '''
+    def parse_index(self):
+        catnames = {}
+        catnames["http://china.globaltimes.cn/chinanews/"] = "China Politics"
+        catnames["http://china.globaltimes.cn/diplomacy/"] = "China Diplomacy"
+        catnames["http://military.globaltimes.cn/china/"] = "China Military"
+        catnames["http://business.globaltimes.cn/china-economy/"] = "China Economy"
+        catnames["http://world.globaltimes.cn/asia-pacific/"] = "Asia Pacific"
+        feeds = []
+
+        for cat in catnames.keys():
+            articles = []
+            soup = self.index_to_soup(cat)
+            for a in soup.findAll('a',attrs={'href' : re.compile(cat+"201[0-9]-[0-1][0-9]/[0-9][0-9][0-9][0-9][0-9][0-9].html")}):
+                url = a['href'].strip()
+                myarticle=({'title':self.tag_to_string(a), 'url':url, 'description':'', 'date':''})
+                self.log("found %s" % url)
+                articles.append(myarticle)
+                self.log("Adding URL %s\n" %url)
+            if articles:
+                feeds.append((catnames[cat], articles))
+        return feeds

resources/recipes/histmag.recipe

@@ -0,0 +1,59 @@
+#!/usr/bin/env  python
+
+__license__   = 'GPL v3'
+__copyright__ = '2010, matek09, matek09@gmail.com'
+
+from calibre.web.feeds.news import BasicNewsRecipe
+import re
+
+class Histmag(BasicNewsRecipe):
+
+	title = u'Histmag'
+	__author__ = 'matek09'
+	description = u"Artykuly historyczne i publicystyczne"
+	encoding = 'utf-8'
+	no_stylesheets = True
+	language = 'pl'
+	remove_javascript = True
+	#max_articles_per_feed = 1
+	remove_tags_before = dict(dict(name = 'div', attrs = {'id' : 'article'}))
+	remove_tags_after = dict(dict(name = 'h2', attrs = {'class' : 'komentarze'}))
+	#keep_only_tags =[]
+	#keep_only_tags.append(dict(name = 'h2'))
+	#keep_only_tags.append(dict(name = 'p'))
+
+	remove_tags =[]
+	remove_tags.append(dict(name = 'p', attrs = {'class' : 'podpis'}))
+	remove_tags.append(dict(name = 'h2', attrs = {'class' : 'komentarze'}))
+	remove_tags.append(dict(name = 'img', attrs = {'src' : 'style/buttons/wesprzyjnas-1.jpg'}))
+
+	preprocess_regexps = [(re.compile(r'</span>'), lambda match: '</span><br><br>'),
+						(re.compile(r'<span>'), lambda match: '<br><br><span>')]
+	extra_css = '''
+					.left {font-size: x-small}
+					.right {font-size: x-small}
+				'''
+
+	def find_articles(self, soup):
+		articles = []
+		for div in soup.findAll('div', attrs={'class' : 'text'}):
+			articles.append({
+				'title' : self.tag_to_string(div.h3.a),
+				'url'   : 'http://www.histmag.org/' + div.h3.a['href'],
+				'date'  : self.tag_to_string(div.next('p')).split('|')[0],
+				'description' : self.tag_to_string(div.next('p', podpis=False)),
+				})
+		return articles
+
+	def parse_index(self):
+		soup = self.index_to_soup('http://histmag.org/?arc=4&dx=0')
+		feeds = []
+		feeds.append((u"Artykuly historyczne", self.find_articles(soup)))
+		soup = self.index_to_soup('http://histmag.org/?arc=5&dx=0')
+		feeds.append((u"Artykuly publicystyczne", self.find_articles(soup)))
+		soup = self.index_to_soup('http://histmag.org/?arc=1&dx=0')
+		feeds.append((u"Wydarzenia", self.find_articles(soup)))
+
+		return feeds
+
+

resources/recipes/newsweek_polska.recipe

@@ -1,19 +1,22 @@
 #!/usr/bin/env  python
 
 __license__   = 'GPL v3'
-__copyright__ = '2010, Mateusz Kielar, matek09@gmail.com'
+__copyright__ = '2010, matek09, matek09@gmail.com'
 
 from calibre.web.feeds.news import BasicNewsRecipe
 
 class Newsweek(BasicNewsRecipe):
-	EDITION = 0
+	FIND_LAST_FULL_ISSUE = True
+	EDITION = '0'
+	EXCLUDE_LOCKED = True
+	LOCKED_ICO = 'http://www.newsweek.pl/bins/media/static/newsweek/img/ico_locked.gif'
 
 	title = u'Newsweek Polska'
-	__author__ = 'Mateusz Kielar'
+	__author__ = 'matek09'
 	description = 'Weekly magazine'
 	encoding = 'utf-8'
 	no_stylesheets = True
-	language = 'en'
+	language = 'pl'
 	remove_javascript = True
 
 	keep_only_tags =[]
@@ -33,34 +36,54 @@ class Newsweek(BasicNewsRecipe):
 	def print_version(self, url):
 		return url.replace("http://www.newsweek.pl/artykuly/wydanie/" + str(self.EDITION), "http://www.newsweek.pl/artykuly") + '/print'
 
+	def is_locked(self, a):
+		if a.findNext('img')['src'] == 'http://www.newsweek.pl/bins/media/static/newsweek/img/ico_locked.gif':
+			return True
+		else:
+			return False
+
+	def is_full(self, issue_soup):
+		if len(issue_soup.findAll('img', attrs={'src' : 'http://www.newsweek.pl/bins/media/static/newsweek/img/ico_locked.gif'})) > 1:
+			return False
+		else:
+			return True
+
 	def find_last_full_issue(self):
-		page = self.index_to_soup('http://www.newsweek.pl/Frames/IssueCover.aspx')
+		frame_url = 'http://www.newsweek.pl/Frames/IssueCover.aspx'
-		issue = 'http://www.newsweek.pl/Frames/' + page.find(lambda tag: tag.name == 'span' and not tag.attrs).a['href']
+		while True:
-		page = self.index_to_soup(issue)
+			frame_soup = self.index_to_soup(frame_url)
-		issue = 'http://www.newsweek.pl/Frames/' + page.find(lambda tag: tag.name == 'span' and not tag.attrs).a['href']
+			self.EDITION = frame_soup.find('a', attrs={'target' : '_parent'})['href'].replace('/wydania/','')
-		page = self.index_to_soup(issue)
+			issue_soup = self.index_to_soup('http://www.newsweek.pl/wydania/' + self.EDITION)
-		self.EDITION = page.find('a', attrs={'target' : '_parent'})['href'].replace('/wydania/','')
+			if self.is_full(issue_soup):
+				break
+			frame_url = 'http://www.newsweek.pl/Frames/' + frame_soup.find(lambda tag: tag.name == 'span' and not tag.attrs).a['href']
+
+
 
 	def parse_index(self):
-		self.find_last_full_issue()
+		if self.FIND_LAST_FULL_ISSUE:
-		soup = self.index_to_soup('http://www.newsweek.pl/wydania/' + str(self.EDITION))
+			self.find_last_full_issue()
+		soup = self.index_to_soup('http://www.newsweek.pl/wydania/' + self.EDITION)
 		img = soup.find('img', id="ctl00_C1_PaperIsssueView_IssueImage", src=True)
 		self.cover_url = img['src']
 		feeds = []
 		parent = soup.find(id='content-left-big')
 		for txt in parent.findAll(attrs={'class':'txt_normal_red strong'}):
-			section = self.tag_to_string(txt).capitalize()
 			articles = list(self.find_articles(txt))
-			feeds.append((section, articles))
+			if len(articles) > 0:
+				section = self.tag_to_string(txt).capitalize()
+				feeds.append((section, articles))
 		return feeds
 
 	def find_articles(self, txt):
 		for a in txt.findAllNext( attrs={'class':['strong','hr']}):
 			if a.name in "div":
 				break
+			if (not self.FIND_LAST_FULL_ISSUE) & self.EXCLUDE_LOCKED & self.is_locked(a):
+				continue
 			yield {
 				'title' : self.tag_to_string(a),
-				'url'   : 'http://www.newsweek.pl'+a['href'],
+				'url'   : 'http://www.newsweek.pl' + a['href'],
 				'date'  : '',
 				'description' : ''
 				}

resources/recipes/nin.recipe

@@ -8,12 +8,15 @@ www.nin.co.rs
 import re
 from calibre import strftime
 from calibre.web.feeds.news import BasicNewsRecipe
+from contextlib import closing
+from calibre.ebooks.BeautifulSoup import BeautifulSoup
+from calibre import entity_to_unicode
 
 class Nin(BasicNewsRecipe):
     title                  = 'NIN online'
     __author__             = 'Darko Miletic'
     description            = 'Nedeljne Informativne Novine'
-    publisher              = 'NIN d.o.o.'
+    publisher              = 'NIN d.o.o. - Ringier d.o.o.'
     category               = 'news, politics, Serbia'
     no_stylesheets         = True
     delay                  = 1
@@ -26,18 +29,29 @@ class Nin(BasicNewsRecipe):
     use_embedded_content   = False
     language               = 'sr'
     publication_type       = 'magazine'
-    extra_css              = ' @font-face {font-family: "sans1";src:url(res:///opt/sony/ebook/FONT/tt0003m_.ttf)} body{font-family: Verdana, Lucida, sans1, sans-serif} .article_description{font-family: Verdana, Lucida, sans1, sans-serif} .artTitle{font-size: x-large; font-weight: bold; color: #900} .izjava{font-size: x-large; font-weight: bold} .columnhead{font-size: small; font-weight: bold;} img{margin-top:0.5em; margin-bottom: 0.7em} b{margin-top: 1em} '
+    extra_css              = """
+                                 @font-face {font-family: "sans1";src:url(res:///opt/sony/ebook/FONT/tt0003m_.ttf)}
+                                 body{font-family: Verdana, Lucida, sans1, sans-serif}
+                                 .article_description{font-family: Verdana, Lucida, sans1, sans-serif}
+                                 .artTitle{font-size: x-large; font-weight: bold; color: #900}
+                                 .izjava{font-size: x-large; font-weight: bold}
+                                 .columnhead{font-size: small; font-weight: bold;}
+                                 img{margin-top:0.5em; margin-bottom: 0.7em; display: block}
+                                 b{margin-top: 1em}
+                             """
 
     conversion_options = {
-                          'comment'          : description
+                          'comment'   : description
-                        , 'tags'             : category
+                        , 'tags'      : category
-                        , 'publisher'        : publisher
+                        , 'publisher' : publisher
-                        , 'language'         : language
+                        , 'language'  : language
-                        , 'linearize_tables' : True
                         }
 
-    preprocess_regexps = [(re.compile(u'\u0110'), lambda match: u'\u00D0')]
+    preprocess_regexps = [
-    remove_attributes = ['height','width']
+                           (re.compile(r'</body>.*?<html>', re.DOTALL|re.IGNORECASE),lambda match: '</body>')
+                          ,(re.compile(r'</html>.*?</html>', re.DOTALL|re.IGNORECASE),lambda match: '</html>')
+                          ,(re.compile(u'\u0110'), lambda match: u'\u00D0')
+                         ]
 
     def get_browser(self):
         br = BasicNewsRecipe.get_browser()
@@ -50,7 +64,10 @@ class Nin(BasicNewsRecipe):
         return br
 
     keep_only_tags    =[dict(name='td', attrs={'width':'520'})]
+    remove_tags_before =dict(name='span', attrs={'class':'izjava'})
     remove_tags_after =dict(name='html')
+    remove_tags = [dict(name=['object','link','iframe','meta','base'])]
+    remove_attributes=['border','background','height','width','align','valign']
 
     def get_cover_url(self):
         cover_url = None
@@ -63,7 +80,7 @@ class Nin(BasicNewsRecipe):
     def parse_index(self):
         articles = []
         count = 0
-        soup = self.index_to_soup(self.PREFIX)
+        soup = self.index_to_soup(self.INDEX)
         for item in soup.findAll('a',attrs={'class':'lmeninavFont'}):
             count = count +1
             if self.test and count > 2:
@@ -90,3 +107,45 @@ class Nin(BasicNewsRecipe):
             articles.append((section,inarts))
         return articles
 
+    def index_to_soup(self, url_or_raw, raw=False):
+        if re.match(r'\w+://', url_or_raw):
+            open_func = getattr(self.browser, 'open_novisit', self.browser.open)
+            with closing(open_func(url_or_raw)) as f:
+                _raw = f.read()
+            if not _raw:
+                raise RuntimeError('Could not fetch index from %s'%url_or_raw)
+        else:
+            _raw = url_or_raw
+        if raw:
+            return _raw
+        if not isinstance(_raw, unicode) and self.encoding:
+            if callable(self.encoding):
+                _raw = self.encoding(_raw)
+            else:
+                _raw = _raw.decode(self.encoding, 'replace')
+        massage = list(BeautifulSoup.MARKUP_MASSAGE)
+        enc = 'cp1252' if callable(self.encoding) or self.encoding is None else self.encoding
+        massage.append((re.compile(r'&(\S+?);'), lambda match:
+            entity_to_unicode(match, encoding=enc)))
+        massage.append((re.compile(r'[\x00-\x08]+'), lambda match:
+            ''))
+        return BeautifulSoup(_raw, markupMassage=massage)
+
+    def preprocess_html(self, soup):
+        for item in soup.findAll(style=True):
+            del item['style']
+        for item in soup.findAll('div'):
+            if len(item.contents) == 0:
+               item.extract()
+        for item in soup.findAll(['td','tr']):
+            item.name='div'
+        for item in soup.findAll('img'):
+            if not item.has_key('alt'):
+               item['alt'] = 'image'
+        for tbl in soup.findAll('table'):
+            img = tbl.find('img')
+            if img:
+               img.extract()
+               tbl.replaceWith(img)
+        return soup
+

resources/recipes/nytimes.recipe

@@ -282,9 +282,9 @@ class NYTimes(BasicNewsRecipe):
         br = BasicNewsRecipe.get_browser()
         if self.username is not None and self.password is not None:
             br.open('http://www.nytimes.com/auth/login')
-            br.select_form(name='login')
+            br.form = br.forms().next()
-            br['USERID']   = self.username
+            br['userid']   = self.username
-            br['PASSWORD'] = self.password
+            br['password'] = self.password
             raw = br.submit().read()
             if 'Please try again' in raw:
                 raise Exception('Your username and password are incorrect')

resources/recipes/nytimes_sub.recipe

@@ -282,9 +282,9 @@ class NYTimes(BasicNewsRecipe):
         br = BasicNewsRecipe.get_browser()
         if self.username is not None and self.password is not None:
             br.open('http://www.nytimes.com/auth/login')
-            br.select_form(name='login')
+            br.form = br.forms().next()
-            br['USERID']   = self.username
+            br['userid']   = self.username
-            br['PASSWORD'] = self.password
+            br['password'] = self.password
             raw = br.submit().read()
             if 'Please try again' in raw:
                 raise Exception('Your username and password are incorrect')

resources/recipes/nzz_ger.recipe

@@ -1,6 +1,6 @@
 
 __license__   = 'GPL v3'
-__copyright__ = '2009, Darko Miletic <darko.miletic at gmail.com>'
+__copyright__ = '2009-2010, Darko Miletic <darko.miletic at gmail.com>'
 
 '''
 www.nzz.ch
@@ -20,6 +20,19 @@ class Nzz(BasicNewsRecipe):
     encoding              = 'utf-8'
     use_embedded_content  = False
     language              = 'de'
+    extra_css             = """
+                               body{font-family: Georgia,"Times New Roman",Times,serif }
+                               .artikel h3,.artikel h4,.bildLegende,.question,.autor{font-family: Arial,Verdana,Helvetica,sans-serif}
+                               .bildLegende{font-size: small}
+                               .autor{font-size: 0.9375em; color: #666666}
+                               .quote{font-size: large !important; 
+                                      font-style: italic; 
+                                      font-weight: normal !important; 
+                                      border-bottom: 1px dotted #BFBFBF; 
+                                      border-top: 1px dotted #BFBFBF; 
+                                      line-height: 1.25em}
+                                .quelle{color: #666666; font-style: italic; white-space: nowrap}
+                            """
 
     conversion_options = {
                              'comments'  : description
@@ -28,12 +41,14 @@ class Nzz(BasicNewsRecipe):
                             ,'publisher' : publisher
                          }
 
-    keep_only_tags = [dict(name='div', attrs={'class':'article'})]
+    keep_only_tags = [dict(name='div', attrs={'class':'zone'})]
-
+    remove_tags_before = dict(name='p', attrs={'class':'dachzeile'})
+    remove_tags_after=dict(name='p', attrs={'class':'fussnote'})
+    remove_attributes=['width','height','lang']
     remove_tags = [
-                     dict(name=['object','link','base'])
+                     dict(name=['object','link','base','meta','iframe'])
-                    ,dict(name='div',attrs={'class':['more','teaser','advXertXoriXals','legal']})
+                    ,dict(attrs={'id':'content_rectangle_1'})
-                    ,dict(name='div',attrs={'id':['popup-src','readercomments','google-ad','advXertXoriXals']})
+                    ,dict(attrs={'class':['weiterfuehrendeLinks','fussnote','video']})                    
                   ]
 
     feeds = [
@@ -50,7 +65,7 @@ class Nzz(BasicNewsRecipe):
               ,(u'Reisen'        , u'http://www.nzz.ch/magazin/reisen?rss=true')
             ]
 
-    def print_version(self, url):
+    def preprocess_html(self, soup):
-        return url + '?printview=true'
+        for item in soup.findAll(style=True):
-
+            del item['style']
-
+        return self.adeify_images(soup)

resources/recipes/polityka.recipe

@@ -1,18 +1,18 @@
 #!/usr/bin/env  python
 
 __license__   = 'GPL v3'
-__copyright__ = '2010, Mateusz Kielar, matek09@gmail.com'
+__copyright__ = '2010, matek09, matek09@gmail.com'
 
 from calibre.web.feeds.news import BasicNewsRecipe
 
 class Polityka(BasicNewsRecipe):
 
 	title = u'Polityka'
-	__author__ = 'Mateusz Kielar'
+	__author__ = 'matek09'
 	description = 'Weekly magazine. Last archive issue'
 	encoding = 'utf-8'
 	no_stylesheets = True
-	language = 'en'
+	language = 'pl'
 	remove_javascript = True
 
 	remove_tags_before = dict(dict(name = 'h2', attrs = {'class' : 'box_nag'}))
@@ -48,7 +48,6 @@ class Polityka(BasicNewsRecipe):
 				for div in box.findAll('div', attrs={'class': 'list_tresc'}):
 					article_page = self.index_to_soup('http://archiwum.polityka.pl' + div.a['href'],)
 					section = self.tag_to_string(article_page.find('h2', attrs = {'class' : 'box_nag'})).split('/')[0].lstrip().rstrip()
-					print section
 					if not articles.has_key(section):
 						articles[section] = []
 					articles[section].append( {

resources/recipes/poughkeepsie_journal.recipe

@@ -0,0 +1,19 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class AdvancedUserRecipe1291143841(BasicNewsRecipe):
+    title          = u'Poughkeepsipe Journal'
+    language = 'en'
+    __author__ = 'weebl'
+    oldest_article = 7
+    max_articles_per_feed = 100
+    no_stylesheets = True
+    timefmt = ' [%a, %d %b, %Y]'
+    feeds          = [(u'Local News', u'http://poughkeepsiejournal.com/apps/pbcs.dll/oversikt?Category=RSS01&mime=xml'),
+            (u'Local Business', u'http://poughkeepsiejournal.com/apps/pbcs.dll/oversikt?Category=RSS02&mime=xml'),
+            (u'Local Sports', u'http://poughkeepsiejournal.com/apps/pbcs.dll/oversikt?Category=RSS03&mime=xml'),
+            (u'Life', u'http://poughkeepsiejournal.com/apps/pbcs.dll/oversikt?Category=RSS04&mime=xml')]
+    remove_tags = [dict(name='img', attrs={'src':'/graphics/mastlogo.gif'})]
+
+    def print_version(self, url):
+          return url.replace('http://www.poughkeepsiejournal.com', 'http://www.poughkeepsiejournal.com/print')
+

resources/recipes/tpm_uk.recipe

@@ -0,0 +1,72 @@
+__license__   = 'GPL v3'
+__copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
+'''
+www.philosophypress.co.uk
+'''
+
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class TPM_uk(BasicNewsRecipe):
+    title                 = "The Philosophers' Magazine"
+    __author__            = 'Darko Miletic'
+    description           = 'Title says it all'
+    publisher             = "The Philosophers' Magazine"
+    category              = 'philosophy, news'
+    oldest_article        = 25
+    max_articles_per_feed = 200
+    no_stylesheets        = True
+    encoding              = 'utf8'
+    use_embedded_content  = False
+    language              = 'en_GB'
+    remove_empty_feeds    = True
+    publication_type      = 'magazine'
+    masthead_url          = 'http://www.philosophypress.co.uk/wp-content/themes/masterplan/tma/images/bg/sitelogo.png'
+    extra_css             = """
+                               body{font-family: Helvetica,Arial,"Lucida Grande",Verdana,sans-serif }
+                               img{margin-bottom: 0.4em; display:block}
+                            """
+
+    conversion_options = {
+                          'comment'   : description
+                        , 'tags'      : category
+                        , 'publisher' : publisher
+                        , 'language'  : language
+                        }
+
+    remove_tags = [
+                     dict(name=['meta','link','base','iframe','embed','object','img'])
+                    ,dict(attrs={'id':['respond','sharethis_0']})
+                    ,dict(attrs={'class':'wp-caption-text'})
+                  ]
+    keep_only_tags=[
+                     dict(attrs={'class':['post_cat','post_name','post_meta','post_text']})
+                    ,dict(attrs={'id':'comments'})
+                   ]
+    remove_attributes=['lang','width','height']
+
+
+    feeds = [
+              (u'Columns'       , u'http://www.philosophypress.co.uk/?feed=rss2&cat=15' )
+             ,(u'Essays'        , u'http://www.philosophypress.co.uk/?feed=rss2&cat=19' )
+             ,(u"21'st Century" , u'http://www.philosophypress.co.uk/?feed=rss2&cat=101')
+             ,(u'Interviews'    , u'http://www.philosophypress.co.uk/?feed=rss2&cat=9'  )
+             ,(u'News'          , u'http://www.philosophypress.co.uk/?feed=rss2&cat=28' )
+             ,(u'Profiles'      , u'http://www.philosophypress.co.uk/?feed=rss2&cat=59' )
+             ,(u'Reviews'       , u'http://www.philosophypress.co.uk/?feed=rss2&cat=12' )
+            ]
+
+    def get_cover_url(self):
+        soup = self.index_to_soup('http://www.philosophypress.co.uk/')
+        for image in soup.findAll('img',title=True):
+           if image['title'].startswith('Click to Subscribe'):
+              return image['src']
+        return None
+
+    def preprocess_html(self, soup):
+        for item in soup.findAll(style=True):
+            del item['style']
+        for alink in soup.findAll('a', rel=True):
+            if alink.string is not None:
+               tstr = alink.string
+               alink.replaceWith(tstr)
+        return soup

resources/recipes/wprost.recipe

@@ -0,0 +1,91 @@
+#!/usr/bin/env  python
+
+__license__   = 'GPL v3'
+__copyright__ = '2010, matek09, matek09@gmail.com'
+
+from calibre.web.feeds.news import BasicNewsRecipe
+import re
+
+class Wprost(BasicNewsRecipe):
+	EDITION = 0
+	FIND_LAST_FULL_ISSUE = True
+	EXCLUDE_LOCKED = True
+	ICO_BLOCKED = 'http://www.wprost.pl/G/icons/ico_blocked.gif'
+
+	title = u'Wprost'
+	__author__ = 'matek09'
+	description = 'Weekly magazine'
+	encoding = 'ISO-8859-2'
+	no_stylesheets = True
+	language = 'pl'
+	remove_javascript = True
+
+	remove_tags_before = dict(dict(name = 'div', attrs = {'id' : 'print-layer'}))
+	remove_tags_after = dict(dict(name = 'div', attrs = {'id' : 'print-layer'}))
+
+	'''keep_only_tags =[]
+	keep_only_tags.append(dict(name = 'table', attrs = {'id' : 'title-table'}))
+	keep_only_tags.append(dict(name = 'div', attrs = {'class' : 'div-header'}))
+	keep_only_tags.append(dict(name = 'div', attrs = {'class' : 'div-content'}))
+	keep_only_tags.append(dict(name = 'div', attrs = {'class' : 'def element-autor'}))'''
+
+	preprocess_regexps = [(re.compile(r'style="display: none;"'), lambda match: ''),
+						(re.compile(r'display: block;'), lambda match: '')]
+
+
+	remove_tags =[]
+	remove_tags.append(dict(name = 'div', attrs = {'class' : 'def element-date'}))
+	remove_tags.append(dict(name = 'div', attrs = {'class' : 'def silver'}))
+	remove_tags.append(dict(name = 'div', attrs = {'id' : 'content-main-column-right'}))
+
+
+	extra_css = '''
+					.div-header {font-size: x-small; font-weight: bold}
+					'''
+#h2 {font-size: x-large; font-weight: bold}
+	def is_blocked(self, a):
+		if a.findNextSibling('img') is None:
+			return False
+		else:
+			return True
+
+
+
+	def find_last_issue(self):
+		soup = self.index_to_soup('http://www.wprost.pl/archiwum/')
+		a = 0
+		if self.FIND_LAST_FULL_ISSUE:
+			ico_blocked = soup.findAll('img', attrs={'src' : self.ICO_BLOCKED})
+			a = ico_blocked[-1].findNext('a', attrs={'title' : re.compile('Zobacz spis tre.ci')})
+		else:
+			a = soup.find('a', attrs={'title' : re.compile('Zobacz spis tre.ci')})
+		self.EDITION = a['href'].replace('/tygodnik/?I=', '')
+		self.cover_url = a.img['src']
+
+
+
+	def parse_index(self):
+		self.find_last_issue()
+		soup = self.index_to_soup('http://www.wprost.pl/tygodnik/?I=' + self.EDITION)
+		feeds = []
+		for main_block in soup.findAll(attrs={'class':'main-block-s3 s3-head head-red3'}):
+			articles = list(self.find_articles(main_block))
+			if len(articles) > 0:
+				section = self.tag_to_string(main_block)
+				feeds.append((section, articles))
+		return feeds
+
+	def find_articles(self, main_block):
+		for a in main_block.findAllNext( attrs={'style':['','padding-top: 15px;']}):
+			if a.name in "td":
+				break
+			if self.EXCLUDE_LOCKED & self.is_blocked(a):
+				continue
+			yield {
+				'title' : self.tag_to_string(a),
+				'url'   : 'http://www.wprost.pl' + a['href'],
+				'date'  : '',
+				'description' : ''
+				}
+
+

setup/build_environment.py

@@ -91,11 +91,15 @@ podofo_inc = '/usr/include/podofo'
 podofo_lib = '/usr/lib'
 chmlib_inc_dirs = chmlib_lib_dirs = []
 sqlite_inc_dirs = []
+icu_inc_dirs = []
+icu_lib_dirs = []
 
 if iswindows:
     prefix  = r'C:\cygwin\home\kovid\sw'
     sw_inc_dir  = os.path.join(prefix, 'include')
     sw_lib_dir  = os.path.join(prefix, 'lib')
+    icu_inc_dirs = [sw_inc_dir]
+    icu_lib_dirs = [sw_lib_dir]
     sqlite_inc_dirs = [sw_inc_dir]
     fc_inc = os.path.join(sw_inc_dir, 'fontconfig')
     fc_lib = sw_lib_dir

setup/extensions.py

@@ -18,7 +18,8 @@ from setup.build_environment import fc_inc, fc_lib, chmlib_inc_dirs, \
         QMAKE, msvc, MT, win_inc, win_lib, png_inc_dirs, win_ddk, \
         magick_inc_dirs, magick_lib_dirs, png_lib_dirs, png_libs, \
         magick_error, magick_libs, ft_lib_dirs, ft_libs, jpg_libs, \
-        jpg_lib_dirs, chmlib_lib_dirs, sqlite_inc_dirs
+        jpg_lib_dirs, chmlib_lib_dirs, sqlite_inc_dirs, icu_inc_dirs, \
+        icu_lib_dirs
 MT
 isunix = islinux or isosx or isfreebsd
 
@@ -56,8 +57,25 @@ pdfreflow_libs = []
 if iswindows:
     pdfreflow_libs = ['advapi32', 'User32', 'Gdi32', 'zlib']
 
+icu_libs = ['icudata', 'icui18n', 'icuuc', 'icuio']
+icu_cflags = []
+if iswindows:
+    icu_libs = ['icudt', 'icuin', 'icuuc', 'icuio']
+if isosx:
+    icu_libs = ['icucore']
+    icu_cflags = ['-DU_DISABLE_RENAMING'] # Needed to use system libicucore.dylib
+
+
 extensions = [
 
+    Extension('icu',
+        ['calibre/utils/icu.c'],
+        libraries=icu_libs,
+        lib_dirs=icu_lib_dirs,
+        inc_dirs=icu_inc_dirs,
+        cflags=icu_cflags
+        ),
+
     Extension('sqlite_custom',
         ['calibre/library/sqlite_custom.c'],
         inc_dirs=sqlite_inc_dirs

setup/installer/linux/freeze2.py

@@ -14,7 +14,8 @@ from setup import Command, modules, basenames, functions, __version__, \
 
 SITE_PACKAGES = ['IPython', 'PIL', 'dateutil', 'dns', 'PyQt4', 'mechanize',
         'sip.so', 'BeautifulSoup.py', 'cssutils', 'encutils', 'lxml',
-        'sipconfig.py', 'xdg']
+        'sipconfig.py', 'xdg', 'dbus', '_dbus_bindings.so', 'dbus_bindings.py',
+        '_dbus_glib_bindings.so']
 
 QTDIR          = '/usr/lib/qt4'
 QTDLLS         = ('QtCore', 'QtGui', 'QtNetwork', 'QtSvg', 'QtXml', 'QtWebKit', 'QtDBus')
@@ -49,6 +50,10 @@ binary_includes = [
                 '/lib/libreadline.so.6',
                 '/usr/lib/libchm.so.0',
                 '/usr/lib/liblcms2.so.2',
+                '/usr/lib/libicudata.so.46',
+                '/usr/lib/libicui18n.so.46',
+                '/usr/lib/libicuuc.so.46',
+                '/usr/lib/libicuio.so.46',
                 ]
 binary_includes += [os.path.join(QTDIR, 'lib%s.so.4'%x) for x in QTDLLS]
 
@@ -340,6 +345,8 @@ class LinuxFreeze(Command):
                 __builtin__.help = _Helper()
 
             def set_qt_plugin_path():
+                import uuid
+                uuid.uuid4() # Workaround for libuuid/PyQt conflict
                 from PyQt4.Qt import QCoreApplication
                 paths = list(map(unicode, QCoreApplication.libraryPaths()))
                 paths.insert(0, sys.frozen_path + '/lib/qt_plugins')

setup/installer/windows/freeze.py

@@ -199,7 +199,7 @@ class Win32Freeze(Command, WixMixIn):
         for pat in ('*.dll',):
             for f in glob.glob(os.path.join(bindir, pat)):
                 ok = True
-                for ex in ('expatw',):
+                for ex in ('expatw', 'testplug'):
                     if ex in f.lower():
                         ok = False
                 if not ok: continue

setup/installer/windows/notes.rst

@@ -77,6 +77,15 @@ Test it on the target system with
 
 calibre-debug -c "import _imaging, _imagingmath, _imagingft, _imagingcms"
 
+ICU
+-------
+
+Download the win32 msvc9 binary from http://www.icu-project.org/download/4.4.html
+
+Note that 4.4 is the last version of ICU that can be compiled (is precompiled) with msvc9
+
+Put the dlls into sw/bin and the unicode dir into sw/include and the contents of lib int sw/lib
+
 Libunrar
 ----------
 

src/calibre/__init__.py

@@ -3,7 +3,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
-import uuid, sys, os, re, logging, time, mimetypes, \
+import uuid, sys, os, re, logging, time, \
        __builtin__, warnings, multiprocessing
 from urllib import getproxies
 __builtin__.__dict__['dynamic_property'] = lambda(func): func(None)
@@ -19,43 +19,18 @@ from calibre.constants import iswindows, isosx, islinux, isfreebsd, isfrozen, \
                               __appname__, __version__, __author__, \
                               win32event, win32api, winerror, fcntl, \
                               filesystem_encoding, plugins, config_dir
-from calibre.startup import winutil, winutilerror
+from calibre.startup import winutil, winutilerror, guess_type
 
-uuid.uuid4() # Imported before PyQt4 to workaround PyQt4 util-linux conflict on gentoo
+if islinux and not getattr(sys, 'frozen', False):
+    # Imported before PyQt4 to workaround PyQt4 util-linux conflict on gentoo
+    uuid.uuid4()
 
 if False:
+    # Prevent pyflakes from complaining
     winutil, winutilerror, __appname__, islinux, __version__
     fcntl, win32event, isfrozen, __author__, terminal_controller
-    winerror, win32api, isfreebsd
+    winerror, win32api, isfreebsd, guess_type
 
-mimetypes.add_type('application/epub+zip',                '.epub')
-mimetypes.add_type('text/x-sony-bbeb+xml',                '.lrs')
-mimetypes.add_type('application/xhtml+xml',               '.xhtml')
-mimetypes.add_type('image/svg+xml',                       '.svg')
-mimetypes.add_type('text/fb2+xml',                        '.fb2')
-mimetypes.add_type('application/x-sony-bbeb',             '.lrf')
-mimetypes.add_type('application/x-sony-bbeb',             '.lrx')
-mimetypes.add_type('application/x-dtbncx+xml',            '.ncx')
-mimetypes.add_type('application/adobe-page-template+xml', '.xpgt')
-mimetypes.add_type('application/x-font-opentype',         '.otf')
-mimetypes.add_type('application/x-font-truetype',         '.ttf')
-mimetypes.add_type('application/oebps-package+xml',       '.opf')
-mimetypes.add_type('application/vnd.palm',                '.pdb')
-mimetypes.add_type('application/x-mobipocket-ebook',      '.mobi')
-mimetypes.add_type('application/x-mobipocket-ebook',      '.prc')
-mimetypes.add_type('application/x-mobipocket-ebook',      '.azw')
-mimetypes.add_type('application/x-cbz',                   '.cbz')
-mimetypes.add_type('application/x-cbr',                   '.cbr')
-mimetypes.add_type('application/x-koboreader-ebook',      '.kobo')
-mimetypes.add_type('image/wmf',                           '.wmf')
-mimetypes.add_type('image/jpeg',                          '.jpg')
-mimetypes.add_type('image/jpeg',                          '.jpeg')
-mimetypes.add_type('image/png',                           '.png')
-mimetypes.add_type('image/gif',                           '.gif')
-mimetypes.add_type('image/bmp',                           '.bmp')
-mimetypes.add_type('image/svg+xml',                       '.svg')
-
-guess_type = mimetypes.guess_type
 import cssutils
 cssutils.log.setLevel(logging.WARN)
 

src/calibre/constants.py

@@ -2,7 +2,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = 'calibre'
-__version__   = '0.7.31'
+__version__   = '0.7.32'
 __author__    = "Kovid Goyal <kovid@kovidgoyal.net>"
 
 import re
@@ -67,7 +67,8 @@ if plugins is None:
                 'pdfreflow',
                 'progress_indicator',
                 'chmlib',
-                'chm_extra'
+                'chm_extra',
+                'icu',
             ] + \
                     (['winutil'] if iswindows else []) + \
                     (['usbobserver'] if isosx else []):

src/calibre/customize/profiles.py

@@ -37,6 +37,8 @@ class Plugin(_Plugin):
             self.fsizes.append((name, num, float(size)))
         self.fnames = dict((name, sz) for name, _, sz in self.fsizes if name)
         self.fnums = dict((num, sz) for _, num, sz in self.fsizes if num)
+        self.width_pts = self.width * 72./self.dpi
+        self.height_pts = self.height * 72./self.dpi
 
 # Input profiles {{{
 class InputProfile(Plugin):

src/calibre/devices/android/driver.py

@@ -19,7 +19,7 @@ class ANDROID(USBMS):
 
     VENDOR_ID   = {
             # HTC
-            0x0bb4 : { 0x0c02 : [0x100, 0x0227], 0x0c01 : [0x100, 0x0227], 0x0ff9
+            0x0bb4 : { 0x0c02 : [0x100, 0x0227, 0x0226], 0x0c01 : [0x100, 0x0227], 0x0ff9
                 : [0x0100, 0x0227, 0x0226], 0x0c87: [0x0100, 0x0227, 0x0226],
                 0xc92 : [0x100]},
 
@@ -38,7 +38,7 @@ class ANDROID(USBMS):
                 0x227]},
 
             # Samsung
-            0x04e8 : { 0x681d : [0x0222, 0x0224, 0x0400],
+            0x04e8 : { 0x681d : [0x0222, 0x0223, 0x0224, 0x0400],
                        0x681c : [0x0222, 0x0224, 0x0400],
                        0x6640 : [0x0100],
                      },
@@ -62,7 +62,8 @@ class ANDROID(USBMS):
             'GT-I5700', 'SAMSUNG', 'DELL', 'LINUX']
     WINDOWS_MAIN_MEM = ['ANDROID_PHONE', 'A855', 'A853', 'INC.NEXUS_ONE',
             '__UMS_COMPOSITE', '_MB200', 'MASS_STORAGE', '_-_CARD', 'SGH-I897',
-            'GT-I9000', 'FILE-STOR_GADGET', 'SGH-T959', 'SAMSUNG_ANDROID']
+            'GT-I9000', 'FILE-STOR_GADGET', 'SGH-T959', 'SAMSUNG_ANDROID',
+            'SCH-I500_CARD']
     WINDOWS_CARD_A_MEM = ['ANDROID_PHONE', 'GT-I9000_CARD', 'SGH-I897',
             'FILE-STOR_GADGET', 'SGH-T959', 'SAMSUNG_ANDROID']
 

src/calibre/devices/cybook/driver.py

@@ -65,8 +65,8 @@ class ORIZON(CYBOOK):
 
     BCD         = [0x319]
 
-    WINDOWS_MAIN_MEM = re.compile(r'CYBOOK_ORIZON__-FD')
+    WINDOWS_MAIN_MEM = re.compile(r'(CYBOOK_ORIZON__-FD)|(FILE-STOR_GADGET)')
-    WINDOWS_CARD_A_MEM = re.compile('CYBOOK_ORIZON__-SD')
+    WINDOWS_CARD_A_MEM = re.compile('(CYBOOK_ORIZON__-SD)|(FILE-STOR_GADGET)')
 
     EBOOK_DIR_MAIN = EBOOK_DIR_CARD_A = 'Digital Editions'
 

src/calibre/devices/eb600/driver.py

@@ -229,7 +229,7 @@ class POCKETBOOK301(USBMS):
 
 class POCKETBOOK602(USBMS):
 
-    name = 'PocketBook Pro 602 Device Interface'
+    name = 'PocketBook Pro 602/902 Device Interface'
     description    = _('Communicate with the PocketBook 602 reader.')
     author         = 'Kovid Goyal'
     supported_platforms = ['windows', 'osx', 'linux']
@@ -244,5 +244,5 @@ class POCKETBOOK602(USBMS):
     BCD         = [0x0324]
 
     VENDOR_NAME = ''
-    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = 'PB602'
+    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = ['PB602', 'PB902']
 

src/calibre/devices/usbms/books.py

@@ -13,6 +13,7 @@ from calibre.devices.interface import BookList as _BookList
 from calibre.constants import preferred_encoding
 from calibre import isbytestring
 from calibre.utils.config import prefs, tweaks
+from calibre.utils.icu import sort_key
 
 class Book(Metadata):
     def __init__(self, prefix, lpath, size=None, other=None):
@@ -215,14 +216,17 @@ class CollectionsBookList(BookList):
                     elif is_series:
                         if doing_dc:
                             collections[cat_name][lpath] = \
-                                (book, book.get('series_index', sys.maxint), '')
+                                (book, book.get('series_index', sys.maxint),
+                                 book.get('title_sort', 'zzzz'))
                         else:
                             collections[cat_name][lpath] = \
-                                (book, book.get(attr+'_index', sys.maxint), '')
+                                (book, book.get(attr+'_index', sys.maxint),
+                                 book.get('title_sort', 'zzzz'))
                     else:
                         if lpath not in collections[cat_name]:
                             collections[cat_name][lpath] = \
-                                (book, book.get('title_sort', 'zzzz'), '')
+                                (book, book.get('title_sort', 'zzzz'),
+                                 book.get('title_sort', 'zzzz'))
         # Sort collections
         result = {}
 
@@ -230,14 +234,19 @@ class CollectionsBookList(BookList):
             x = xx[1]
             y = yy[1]
             if x is None and y is None:
+                # No sort_key needed here, because defaults are ascii
                 return cmp(xx[2], yy[2])
             if x is None:
                 return 1
             if y is None:
                 return -1
-            c = cmp(x, y)
+            if isinstance(x, (unicode, str)):
+                c = cmp(sort_key(x), sort_key(y))
+            else:
+                c = cmp(x, y)
             if c != 0:
                 return c
+            # same as above -- no sort_key needed here
             return cmp(xx[2], yy[2])
 
         for category, lpaths in collections.items():

src/calibre/ebooks/epub/output.py

@@ -142,6 +142,9 @@ class EPUBOutput(OutputFormatPlugin):
     def convert(self, oeb, output_path, input_plugin, opts, log):
         self.log, self.opts, self.oeb = log, opts, oeb
 
+        #from calibre.ebooks.oeb.transforms.filenames import UniqueFilenames
+        #UniqueFilenames()(oeb, opts)
+
         self.workaround_ade_quirks()
         self.workaround_webkit_quirks()
         self.upshift_markup()

src/calibre/ebooks/metadata/book/base.py

@@ -531,6 +531,8 @@ class Metadata(object):
                 res = format_date(res, cmeta['display'].get('date_format','dd MMM yyyy'))
             elif datatype == 'bool':
                 res = _('Yes') if res else _('No')
+            elif datatype == 'rating':
+                res = res/2
             return (name, unicode(res), orig_res, cmeta)
 
         # Translate aliases into the standard field name

src/calibre/ebooks/mobi/mobiml.py

@@ -10,9 +10,10 @@ import copy
 import re
 from lxml import etree
 from calibre.ebooks.oeb.base import namespace, barename
-from calibre.ebooks.oeb.base import XHTML, XHTML_NS, OEB_DOCS
+from calibre.ebooks.oeb.base import XHTML, XHTML_NS, OEB_DOCS, urlnormalize
 from calibre.ebooks.oeb.stylizer import Stylizer
 from calibre.ebooks.oeb.transforms.flatcss import KeyMapper
+from calibre.utils.magick.draw import identify_data
 
 MBP_NS = 'http://mobipocket.com/ns/mbp'
 def MBP(name): return '{%s}%s' % (MBP_NS, name)
@@ -121,6 +122,7 @@ class MobiMLizer(object):
             body = item.data.find(XHTML('body'))
             nroot = etree.Element(XHTML('html'), nsmap=MOBI_NSMAP)
             nbody = etree.SubElement(nroot, XHTML('body'))
+            self.current_spine_item = item
             self.mobimlize_elem(body, stylizer, BlockState(nbody),
                                 [FormatState()])
             item.data = nroot
@@ -357,8 +359,9 @@ class MobiMLizer(object):
         if tag == 'img' and 'src' in elem.attrib:
             istate.attrib['src'] = elem.attrib['src']
             istate.attrib['align'] = 'baseline'
+            cssdict = style.cssdict()
             for prop in ('width', 'height'):
-                if style[prop] != 'auto':
+                if cssdict[prop] != 'auto':
                     value = style[prop]
                     if value == getattr(self.profile, prop):
                         result = '100%'
@@ -371,8 +374,40 @@ class MobiMLizer(object):
                                 (72./self.profile.dpi)))
                         except:
                             continue
-                        result = "%d"%pixs
+                        result = str(pixs)
                     istate.attrib[prop] = result
+            if 'width' not in istate.attrib or 'height' not in istate.attrib:
+                href = self.current_spine_item.abshref(elem.attrib['src'])
+                try:
+                    item = self.oeb.manifest.hrefs[urlnormalize(href)]
+                except:
+                    self.oeb.logger.warn('Failed to find image:',
+                            href)
+                else:
+                    try:
+                        width, height = identify_data(item.data)[:2]
+                    except:
+                        self.oeb.logger.warn('Invalid image:', href)
+                    else:
+                        if 'width' not in istate.attrib and 'height' not in \
+                                    istate.attrib:
+                            istate.attrib['width'] = str(width)
+                            istate.attrib['height'] = str(height)
+                        else:
+                            ar = float(width)/float(height)
+                            if 'width' not in istate.attrib:
+                                try:
+                                    width = int(istate.attrib['height'])*ar
+                                except:
+                                    pass
+                                istate.attrib['width'] = str(int(width))
+                            else:
+                                try:
+                                    height = int(istate.attrib['width'])/ar
+                                except:
+                                    pass
+                                istate.attrib['height'] = str(int(height))
+                        item.unload_data_from_memory()
         elif tag == 'hr' and asfloat(style['width']) > 0:
             prop = style['width'] / self.profile.width
             istate.attrib['width'] = "%d%%" % int(round(prop * 100))

src/calibre/ebooks/mobi/reader.py

@@ -504,6 +504,9 @@ class MobiReader(object):
             'x-large': '5',
             'xx-large': '6',
             }
+        def barename(x):
+            return x.rpartition(':')[-1]
+
         mobi_version = self.book_header.mobi_version
         for x in root.xpath('//ncx'):
             x.getparent().remove(x)
@@ -512,8 +515,9 @@ class MobiReader(object):
             for x in tag.attrib:
                 if ':' in x:
                     del tag.attrib[x]
-            if tag.tag in ('country-region', 'place', 'placetype', 'placename',
+            if tag.tag and barename(tag.tag.lower()) in \
-                'state', 'city', 'street', 'address', 'content', 'form'):
+                ('country-region', 'place', 'placetype', 'placename',
+                    'state', 'city', 'street', 'address', 'content', 'form'):
                 tag.tag = 'div' if tag.tag in ('content', 'form') else 'span'
                 for key in tag.attrib.keys():
                     tag.attrib.pop(key)

src/calibre/ebooks/oeb/base.py

@@ -775,6 +775,7 @@ class Manifest(object):
             return u'Item(id=%r, href=%r, media_type=%r)' \
                 % (self.id, self.href, self.media_type)
 
+        # Parsing {{{
         def _parse_xml(self, data):
             data = xml_to_unicode(data, strip_encoding_pats=True,
                     assume_utf8=True, resolve_entities=True)[0]
@@ -1035,6 +1036,8 @@ class Manifest(object):
             data = item.data.cssText
             return ('utf-8', data)
 
+        # }}}
+
         @dynamic_property
         def data(self):
             doc = """Provides MIME type sensitive access to the manifest

src/calibre/ebooks/oeb/iterator.py

@@ -96,7 +96,10 @@ class EbookIterator(object):
 
     def search(self, text, index, backwards=False):
         text = text.lower()
-        for i, path in enumerate(self.spine):
+        pmap = [(i, path) for i, path in enumerate(self.spine)]
+        if backwards:
+            pmap.reverse()
+        for i, path in pmap:
             if (backwards and i < index) or (not backwards and i > index):
                 if text in open(path, 'rb').read().decode(path.encoding).lower():
                     return i

src/calibre/ebooks/oeb/stylizer.py

@@ -253,7 +253,10 @@ class Stylizer(object):
                 upd = {}
                 for prop in ('width', 'height'):
                     val = elem.get(prop, '').strip()
-                    del elem.attrib[prop]
+                    try:
+                        del elem.attrib[prop]
+                    except:
+                        pass
                     if val:
                         if num_pat.match(val) is not None:
                             val += 'px'
@@ -572,7 +575,7 @@ class Style(object):
             if parent is not None:
                 base = parent.width
             else:
-                base = self._profile.width
+                base = self._profile.width_pts
             if 'width' in self._element.attrib:
                 width = self._element.attrib['width']
             elif 'width' in self._style:
@@ -584,6 +587,13 @@ class Style(object):
             if isinstance(result, (unicode, str, bytes)):
                 result = self._profile.width
             self._width = result
+            if 'max-width' in self._style:
+                result = self._unit_convert(self._style['max-width'], base=base)
+                if isinstance(result, (unicode, str, bytes)):
+                    result = self._width
+                if result < self._width:
+                    self._width = result
+
         return self._width
 
     @property
@@ -595,7 +605,7 @@ class Style(object):
             if parent is not None:
                 base = parent.height
             else:
-                base = self._profile.height
+                base = self._profile.height_pts
             if 'height' in self._element.attrib:
                 height = self._element.attrib['height']
             elif 'height' in self._style:
@@ -607,6 +617,13 @@ class Style(object):
             if isinstance(result, (unicode, str, bytes)):
                 result = self._profile.height
             self._height = result
+            if 'max-height' in self._style:
+                result = self._unit_convert(self._style['max-height'], base=base)
+                if isinstance(result, (unicode, str, bytes)):
+                    result = self._height
+                if result < self._height:
+                    self._height = result
+
         return self._height
 
     @property

src/calibre/ebooks/oeb/transforms/filenames.py

@@ -0,0 +1,130 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+
+__license__   = 'GPL v3'
+__copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+import posixpath
+from urlparse import urldefrag
+
+from lxml import etree
+import cssutils
+
+from calibre.ebooks.oeb.base import rewrite_links, urlnormalize
+
+class RenameFiles(object):
+
+    '''
+    Rename files and adjust all links pointing to them. Note that the spine
+    and manifest are not touched by this transform.
+    '''
+
+    def __init__(self, rename_map):
+        self.rename_map = rename_map
+
+    def __call__(self, oeb, opts):
+        self.log = oeb.logger
+        self.opts = opts
+        self.oeb = oeb
+
+        for item in oeb.manifest.items:
+            self.current_item = item
+            if etree.iselement(item.data):
+                rewrite_links(self.current_item.data, self.url_replacer)
+            elif hasattr(item.data, 'cssText'):
+                cssutils.replaceUrls(item.data, self.url_replacer)
+
+        if self.oeb.guide:
+            for ref in self.oeb.guide.values():
+                href = urlnormalize(ref.href)
+                href, frag = urldefrag(href)
+                replacement = self.rename_map.get(href, None)
+                if replacement is not None:
+                    nhref = replacement
+                    if frag:
+                        nhref += '#' + frag
+                    ref.href = nhref
+
+        if self.oeb.toc:
+            self.fix_toc_entry(self.oeb.toc)
+
+
+    def fix_toc_entry(self, toc):
+        if toc.href:
+            href = urlnormalize(toc.href)
+            href, frag = urldefrag(href)
+            replacement = self.rename_map.get(href, None)
+
+            if replacement is not None:
+                nhref = replacement
+                if frag:
+                    nhref = '#'.join((nhref, frag))
+                toc.href = nhref
+
+        for x in toc:
+            self.fix_toc_entry(x)
+
+    def url_replacer(self, orig_url):
+         url = urlnormalize(orig_url)
+         path, frag = urldefrag(url)
+         href = self.current_item.abshref(path)
+         replacement = self.rename_map.get(href, None)
+         if replacement is None:
+             return orig_url
+         replacement = self.current_item.relhref(replacement)
+         if frag:
+             replacement += '#' + frag
+         return replacement
+
+class UniqueFilenames(object):
+
+    'Ensure that every item in the manifest has a unique filename'
+
+    def __call__(self, oeb, opts):
+        self.log = oeb.logger
+        self.opts = opts
+        self.oeb = oeb
+
+        self.seen_filenames = set([])
+        self.rename_map = {}
+
+        for item in list(oeb.manifest.items):
+            fname = posixpath.basename(item.href)
+            if fname in self.seen_filenames:
+                suffix = self.unique_suffix(fname)
+                data = item.data
+                base, ext = posixpath.splitext(item.href)
+                nhref = base + suffix + ext
+                nhref = oeb.manifest.generate(href=nhref)[1]
+                nitem = oeb.manifest.add(item.id, nhref, item.media_type, data=data,
+                        fallback=item.fallback)
+                self.seen_filenames.add(posixpath.basename(nhref))
+                self.rename_map[item.href] = nhref
+                if item.spine_position is not None:
+                    oeb.spine.insert(item.spine_position, nitem, item.linear)
+                    oeb.spine.remove(item)
+                oeb.manifest.remove(item)
+            else:
+                self.seen_filenames.add(fname)
+
+        if self.rename_map:
+            self.log('Found non-unique filenames, renaming to support broken'
+                    ' EPUB readers like FBReader, Aldiko and Stanza...')
+            from pprint import pformat
+            self.log.debug(pformat(self.rename_map))
+
+            renamer = RenameFiles(self.rename_map)
+            renamer(oeb, opts)
+
+
+    def unique_suffix(self, fname):
+        base, ext = posixpath.splitext(fname)
+        c = 0
+        while True:
+            c += 1
+            suffix = '_u%d'%c
+            candidate = base + suffix + ext
+            if candidate not in self.seen_filenames:
+                return suffix
+

src/calibre/ebooks/oeb/transforms/jacket.py

@@ -93,7 +93,7 @@ class Jacket(object):
 
 # Render Jacket {{{
 
-def get_rating(rating, rchar):
+def get_rating(rating, rchar, e_rchar):
     ans = ''
     try:
         num = float(rating)/2
@@ -104,12 +104,12 @@ def get_rating(rating, rchar):
     if num < 1:
         return ans
 
-    ans = rchar * int(num)
+    ans = ("%s%s") % (rchar * int(num), e_rchar * (5 - int(num)))
     return ans
 
-
 def render_jacket(mi, output_profile,
-        alt_title=_('Unknown'), alt_tags=[], alt_comments=''):
+        alt_title=_('Unknown'), alt_tags=[], alt_comments='',
+        alt_publisher=('Unknown publisher')):
     css = P('jacket/stylesheet.css', data=True).decode('utf-8')
 
     try:
@@ -124,12 +124,17 @@ def render_jacket(mi, output_profile,
     if not mi.series:
         series = ''
 
+    try:
+        publisher = mi.publisher if mi.publisher else alt_publisher
+    except:
+        publisher = _('Unknown publisher')
+
     try:
         pubdate = strftime(u'%Y', mi.pubdate.timetuple())
     except:
         pubdate = ''
 
-    rating = get_rating(mi.rating, output_profile.ratings_char)
+    rating = get_rating(mi.rating, output_profile.ratings_char, output_profile.empty_ratings_char)
 
     tags = mi.tags if mi.tags else alt_tags
     if tags:
@@ -154,6 +159,7 @@ def render_jacket(mi, output_profile,
                     css=css,
                     title=title,
                     author=author,
+                    publisher=publisher,
                     pubdate_label=_('Published'), pubdate=pubdate,
                     series_label=_('Series'), series=series,
                     rating_label=_('Rating'), rating=rating,
@@ -168,16 +174,16 @@ def render_jacket(mi, output_profile,
         # Post-process the generated html to strip out empty header items
         soup = BeautifulSoup(generated_html)
         if not series:
-            series_tag = soup.find('tr', attrs={'class':'cbj_series'})
+            series_tag = soup.find(attrs={'class':'cbj_series'})
             series_tag.extract()
         if not rating:
-            rating_tag = soup.find('tr', attrs={'class':'cbj_rating'})
+            rating_tag = soup.find(attrs={'class':'cbj_rating'})
             rating_tag.extract()
         if not tags:
-            tags_tag = soup.find('tr', attrs={'class':'cbj_tags'})
+            tags_tag = soup.find(attrs={'class':'cbj_tags'})
             tags_tag.extract()
         if not pubdate:
-            pubdate_tag = soup.find('tr', attrs={'class':'cbj_pubdate'})
+            pubdate_tag = soup.find(attrs={'class':'cbj_pubdate'})
             pubdate_tag.extract()
         if output_profile.short_name != 'kindle':
             hr_tag = soup.find('hr', attrs={'class':'cbj_kindle_banner_hr'})

src/calibre/gui2/actions/catalog.py

@@ -37,7 +37,8 @@ class GenerateCatalogAction(InterfaceAction):
 			dbspec[id] = {'ondevice': db.ondevice(id, index_is_id=True)}
 
         # Calling gui2.tools:generate_catalog()
-        ret = generate_catalog(self.gui, dbspec, ids, self.gui.device_manager)
+        ret = generate_catalog(self.gui, dbspec, ids, self.gui.device_manager,
+                db)
         if ret is None:
             return
 

src/calibre/gui2/actions/edit_metadata.py

@@ -8,7 +8,7 @@ __docformat__ = 'restructuredtext en'
 import os
 from functools import partial
 
-from PyQt4.Qt import Qt, QMenu
+from PyQt4.Qt import Qt, QMenu, QModelIndex
 
 from calibre.gui2 import error_dialog, config
 from calibre.gui2.dialogs.metadata_single import MetadataSingleDialog
@@ -16,6 +16,7 @@ from calibre.gui2.dialogs.metadata_bulk import MetadataBulkDialog
 from calibre.gui2.dialogs.confirm_delete import confirm
 from calibre.gui2.dialogs.tag_list_editor import TagListEditor
 from calibre.gui2.actions import InterfaceAction
+from calibre.utils.icu import sort_key
 
 class EditMetadataAction(InterfaceAction):
 
@@ -126,20 +127,35 @@ class EditMetadataAction(InterfaceAction):
         if bulk or (bulk is None and len(rows) > 1):
             return self.edit_bulk_metadata(checked)
 
-        def accepted(id):
+        row_list = [r.row() for r in rows]
-            self.gui.library_view.model().refresh_ids([id])
+        current_row = 0
+        changed = set([])
+        db = self.gui.library_view.model().db
 
-        for row in rows:
+        if len(row_list) == 1:
-            self.gui.iactions['View'].metadata_view_id = self.gui.library_view.model().db.id(row.row())
+            cr = row_list[0]
-            d = MetadataSingleDialog(self.gui, row.row(),
+            row_list = \
-                                    self.gui.library_view.model().db,
+                list(range(self.gui.library_view.model().rowCount(QModelIndex())))
-                                    accepted_callback=accepted,
+            current_row = row_list.index(cr)
-                                    cancel_all=rows.index(row) < len(rows)-1)
+
-            d.view_format.connect(self.gui.iactions['View'].metadata_view_format)
+        while True:
-            d.exec_()
+            prev = next_ = None
-            if d.cancel_all:
+            if current_row > 0:
+                prev = db.title(row_list[current_row-1])
+            if current_row < len(row_list) - 1:
+                next_ = db.title(row_list[current_row+1])
+
+            d = MetadataSingleDialog(self.gui, row_list[current_row], db,
+                    prev=prev, next_=next_)
+            if d.exec_() != d.Accepted:
                 break
-        if rows:
+            changed.add(d.id)
+            if d.row_delta == 0:
+                break
+            current_row += d.row_delta
+
+        if changed:
+            self.gui.library_view.model().refresh_ids(list(changed))
             current = self.gui.library_view.currentIndex()
             m = self.gui.library_view.model()
             if self.gui.cover_flow:
@@ -162,9 +178,17 @@ class EditMetadataAction(InterfaceAction):
             return
         # Prevent the TagView from updating due to signals from the database
         self.gui.tags_view.blockSignals(True)
+        changed = False
         try:
-            changed = MetadataBulkDialog(self.gui, rows,
+            current_tab = 0
-                self.gui.library_view.model()).changed
+            while True:
+                dialog = MetadataBulkDialog(self.gui, rows,
+                                self.gui.library_view.model(), current_tab)
+                if dialog.changed:
+                    changed = True
+                if not dialog.do_again:
+                    break
+                current_tab = dialog.central_widget.currentIndex()
         finally:
             self.gui.tags_view.blockSignals(False)
         if changed:
@@ -340,8 +364,7 @@ class EditMetadataAction(InterfaceAction):
     def edit_device_collections(self, view, oncard=None):
         model = view.model()
         result = model.get_collections_with_ids()
-        compare = (lambda x,y:cmp(x.lower(), y.lower()))
+        d = TagListEditor(self.gui, tag_to_match=None, data=result, key=sort_key)
-        d = TagListEditor(self.gui, tag_to_match=None, data=result, compare=compare)
         d.exec_()
         if d.result() == d.Accepted:
             to_rename = d.to_rename # dict of new text to old ids

src/calibre/gui2/actions/show_book_details.py

@@ -29,5 +29,6 @@ class ShowBookDetailsAction(InterfaceAction):
             return
         index = self.gui.library_view.currentIndex()
         if index.isValid():
-            BookInfo(self.gui, self.gui.library_view, index).show()
+            BookInfo(self.gui, self.gui.library_view, index,
+                    self.gui.iactions['View'].view_format_by_id).show()
 

src/calibre/gui2/actions/similar_books.py

@@ -58,6 +58,7 @@ class SimilarBooksAction(InterfaceAction):
                                 for a in authors.split(',')]
                 join = ' or '
         if search:
-            self.gui.search.set_search_string(join.join(search))
+            self.gui.search.set_search_string(join.join(search),
+                    store_in_history=True)
 
 

src/calibre/gui2/actions/view.py

@@ -12,7 +12,7 @@ from PyQt4.Qt import Qt, QMenu
 
 from calibre.constants import isosx
 from calibre.gui2 import error_dialog, Dispatcher, question_dialog, config, \
-        open_local_file
+        open_local_file, info_dialog
 from calibre.gui2.dialogs.choose_format import ChooseFormatDialog
 from calibre.utils.config import prefs
 from calibre.ptempfile import PersistentTemporaryFile
@@ -89,18 +89,34 @@ class ViewAction(InterfaceAction):
         self._launch_viewer(name, viewer, internal)
 
     def view_specific_format(self, triggered):
-        rows = self.gui.library_view.selectionModel().selectedRows()
+        rows = list(self.gui.library_view.selectionModel().selectedRows())
         if not rows or len(rows) == 0:
             d = error_dialog(self.gui, _('Cannot view'), _('No book selected'))
             d.exec_()
             return
 
-        row = rows[0].row()
+        db = self.gui.library_view.model().db
-        formats = self.gui.library_view.model().db.formats(row).upper().split(',')
+        rows = [r.row() for r in rows]
-        d = ChooseFormatDialog(self.gui, _('Choose the format to view'), formats)
+        formats = [db.formats(row) for row in rows]
+        formats = [list(f.upper().split(',')) if f else None for f in formats]
+        all_fmts = set([])
+        for x in formats:
+            for f in x: all_fmts.add(f)
+        d = ChooseFormatDialog(self.gui, _('Choose the format to view'),
+                list(sorted(all_fmts)))
         if d.exec_() == d.Accepted:
-            format = d.format()
+            fmt = d.format()
-            self.view_format(row, format)
+            orig_num = len(rows)
+            rows = [rows[i] for i in range(len(rows)) if formats[i] and fmt in
+                    formats[i]]
+            if self._view_check(len(rows)):
+                for row in rows:
+                    self.view_format(row, fmt)
+                if len(rows) < orig_num:
+                    info_dialog(self.gui, _('Format unavailable'),
+                            _('Not all the selected books were available in'
+                                ' the %s format. You should convert'
+                                ' them first.')%fmt, show=True)
 
     def _view_check(self, num, max_=3):
         if num <= max_:

src/calibre/gui2/book_details.py

@@ -19,6 +19,7 @@ from calibre.ebooks import BOOK_EXTENSIONS
 from calibre.constants import preferred_encoding
 from calibre.library.comments import comments_to_html
 from calibre.gui2 import config, open_local_file
+from calibre.utils.icu import sort_key
 
 # render_rows(data) {{{
 WEIGHTS = collections.defaultdict(lambda : 100)
@@ -31,8 +32,8 @@ WEIGHTS[_('Tags')] = 4
 def render_rows(data):
     keys = data.keys()
     # First sort by name. The WEIGHTS sort will preserve this sub-order
-    keys.sort(cmp=lambda x, y: cmp(x.lower(), y.lower()))
+    keys.sort(key=sort_key)
-    keys.sort(cmp=lambda x, y: cmp(WEIGHTS[x], WEIGHTS[y]))
+    keys.sort(key=lambda x: WEIGHTS[x])
     rows = []
     for key in keys:
         txt = data[key]
@@ -208,8 +209,9 @@ class BookInfo(QWebView):
         rows = u'\n'.join([u'<tr><td valign="top"><b>%s:</b></td><td valign="top">%s</td></tr>'%(k,t) for
             k, t in rows])
         comments = data.get(_('Comments'), '')
-        if comments and comments != u'None':
+        if not comments or comments == u'None':
-            self.renderer.queue.put((rows, comments))
+            comments = ''
+        self.renderer.queue.put((rows, comments))
         self._show_data(rows, '')
 
 

src/calibre/gui2/catalog/catalog_bibtex.py

@@ -34,7 +34,7 @@ class PluginWidget(QWidget, Ui_Form):
                 self.all_fields.append(x)
                 QListWidgetItem(x, self.db_fields)
 
-    def initialize(self, name): #not working properly to update
+    def initialize(self, name, db): #not working properly to update
         self.name = name
         fields = gprefs.get(name+'_db_fields', self.all_fields)
         # Restore the activated db_fields from last use

src/calibre/gui2/catalog/catalog_csv_xml.py

@@ -28,7 +28,7 @@ class PluginWidget(QWidget, Ui_Form):
                 self.all_fields.append(x)
                 QListWidgetItem(x, self.db_fields)
 
-    def initialize(self, name):
+    def initialize(self, name, db):
         self.name = name
         fields = gprefs.get(name+'_db_fields', self.all_fields)
         # Restore the activated fields from last use

src/calibre/gui2/catalog/catalog_epub_mobi.py

@@ -7,10 +7,11 @@ __copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
 
-from calibre.gui2 import gprefs
-from catalog_epub_mobi_ui import Ui_Form
 from calibre.ebooks.conversion.config import load_defaults
-from PyQt4.Qt import QWidget
+from calibre.gui2 import gprefs
+
+from catalog_epub_mobi_ui import Ui_Form
+from PyQt4.Qt import QWidget, QLineEdit
 
 class PluginWidget(QWidget,Ui_Form):
 
@@ -23,7 +24,8 @@ class PluginWidget(QWidget,Ui_Form):
                      ('generate_recently_added', True),
                      ('note_tag','*'),
                      ('numbers_as_text', False),
-                     ('read_tag','+'),
+                     ('read_pattern','+'),
+                     ('read_source_field_cb','Tag'),
                      ('wishlist_tag','Wishlist'),
                      ]
 
@@ -38,16 +40,54 @@ class PluginWidget(QWidget,Ui_Form):
         QWidget.__init__(self, parent)
         self.setupUi(self)
 
-    def initialize(self, name):
+    def initialize(self, name, db):
         self.name = name
+
+        # Populate the 'Read book' source fields
+        all_custom_fields = db.custom_field_keys()
+        custom_fields = {}
+        custom_fields['Tag'] = {'field':'tag', 'datatype':u'text'}
+        for custom_field in all_custom_fields:
+            field_md = db.metadata_for_field(custom_field)
+            if field_md['datatype'] in ['bool','composite','datetime','text']:
+                custom_fields[field_md['name']] = {'field':custom_field,
+                                                   'datatype':field_md['datatype']}
+
+        # Add the sorted eligible fields to the combo box
+        for cf in sorted(custom_fields):
+            self.read_source_field_cb.addItem(cf)
+
+        self.read_source_fields = custom_fields
+        self.read_source_field_cb.currentIndexChanged.connect(self.read_source_field_changed)
+
         # Update dialog fields from stored options
         for opt in self.OPTION_FIELDS:
             opt_value = gprefs.get(self.name + '_' + opt[0], opt[1])
-            if opt[0] in ['numbers_as_text','generate_titles','generate_series','generate_recently_added']:
+            if opt[0] in [
+                          'generate_recently_added',
+                          'generate_series',
+                          'generate_titles',
+                          'numbers_as_text',
+                          ]:
                 getattr(self, opt[0]).setChecked(opt_value)
+
+            # Combo box
+            elif opt[0] in ['read_source_field_cb']:
+                # Look for last-stored combo box value
+                index = self.read_source_field_cb.findText(opt_value)
+                if index == -1:
+                    index = self.read_source_field_cb.findText('Tag')
+                self.read_source_field_cb.setCurrentIndex(index)
+
+            # Text fields
             else:
                 getattr(self, opt[0]).setText(opt_value)
 
+        # Init self.read_source_field
+        cs = unicode(self.read_source_field_cb.currentText())
+        read_source_spec = self.read_source_fields[cs]
+        self.read_source_field = read_source_spec['field']
+
     def options(self):
         # Save/return the current options
         # exclude_genre stores literally
@@ -55,16 +95,60 @@ class PluginWidget(QWidget,Ui_Form):
         # others store as lists
         opts_dict = {}
         for opt in self.OPTION_FIELDS:
-            if opt[0] in ['numbers_as_text','generate_titles','generate_series','generate_recently_added']:
+            # Save values to gprefs
+            if opt[0] in [
+                          'generate_recently_added',
+                          'generate_series',
+                          'generate_titles',
+                          'numbers_as_text',
+                          ]:
                 opt_value = getattr(self,opt[0]).isChecked()
+
+            # Combo box uses .currentText()
+            elif opt[0] in ['read_source_field_cb']:
+                opt_value = unicode(getattr(self, opt[0]).currentText())
+
+            # text fields use .text()
             else:
                 opt_value = unicode(getattr(self, opt[0]).text())
             gprefs.set(self.name + '_' + opt[0], opt_value)
 
-            if opt[0] in ['exclude_genre','numbers_as_text','generate_titles','generate_series','generate_recently_added']:
+            # Construct opts
+            if opt[0] in [
+                          'exclude_genre',
+                          'generate_recently_added',
+                          'generate_series',
+                          'generate_titles',
+                          'numbers_as_text',
+                          ]:
                 opts_dict[opt[0]] = opt_value
             else:
                 opts_dict[opt[0]] = opt_value.split(',')
-        opts_dict['output_profile'] = [load_defaults('page_setup')['output_profile']]
 
+        # Generate read_book_marker
+        opts_dict['read_book_marker'] = "%s:%s" % (self.read_source_field, self.read_pattern.text())
+
+        # Append the output profile
+        opts_dict['output_profile'] = [load_defaults('page_setup')['output_profile']]
         return opts_dict
+
+    def read_source_field_changed(self,new_index):
+        '''
+        Process changes in the read_source_field combo box
+        Currently using QLineEdit for all field types
+        Possible to modify to switch QWidget type
+        '''
+        new_source = str(self.read_source_field_cb.currentText())
+        read_source_spec = self.read_source_fields[str(new_source)]
+        self.read_source_field = read_source_spec['field']
+
+        # Change pattern input widget to match the source field datatype
+        if read_source_spec['datatype'] in ['bool','composite','datetime','text']:
+            if not isinstance(self.read_pattern, QLineEdit):
+                self.read_spec_hl.removeWidget(self.read_pattern)
+                dw = QLineEdit(self)
+                dw.setObjectName('read_pattern')
+                dw.setToolTip('Pattern for read book')
+                self.read_pattern = dw
+                self.read_spec_hl.addWidget(dw)
+

src/calibre/gui2/catalog/catalog_epub_mobi.ui

@@ -6,8 +6,8 @@
    <rect>
     <x>0</x>
     <y>0</y>
-    <width>579</width>
+    <width>627</width>
-    <height>411</height>
+    <height>549</height>
    </rect>
   </property>
   <property name="windowTitle">
@@ -28,42 +28,28 @@
      </property>
     </widget>
    </item>
-   <item row="1" column="0">
+   <item row="4" column="0">
-    <widget class="QLabel" name="label_3">
-     <property name="text">
-      <string>'Mark this book as read' tag:</string>
-     </property>
-    </widget>
-   </item>
-   <item row="1" column="1">
-    <widget class="QLineEdit" name="read_tag">
-     <property name="toolTip">
-      <string extracomment="Default: +"/>
-     </property>
-    </widget>
-   </item>
-   <item row="3" column="0">
     <widget class="QLabel" name="label_4">
      <property name="text">
       <string>Additional note tag prefix:</string>
      </property>
     </widget>
    </item>
-   <item row="3" column="1">
+   <item row="4" column="1">
     <widget class="QLineEdit" name="note_tag">
      <property name="toolTip">
       <string extracomment="Default: *"/>
      </property>
     </widget>
    </item>
-   <item row="5" column="1">
+   <item row="6" column="1">
     <widget class="QLineEdit" name="exclude_genre">
      <property name="toolTip">
       <string extracomment="Default: \[[\w]*\]"/>
      </property>
     </widget>
    </item>
-   <item row="5" column="0">
+   <item row="6" column="0">
     <widget class="QLabel" name="label">
      <property name="text">
       <string>Regex pattern describing tags to exclude as genres:</string>
@@ -76,7 +62,7 @@
      </property>
     </widget>
    </item>
-   <item row="6" column="1">
+   <item row="7" column="1">
     <widget class="QLabel" name="label_6">
      <property name="text">
       <string>Regex tips:
@@ -88,7 +74,7 @@
      </property>
     </widget>
    </item>
-   <item row="7" column="0">
+   <item row="8" column="0">
     <spacer name="verticalSpacer">
      <property name="orientation">
       <enum>Qt::Vertical</enum>
@@ -101,44 +87,84 @@
      </property>
     </spacer>
    </item>
-   <item row="9" column="0">
+   <item row="10" column="0">
     <widget class="QCheckBox" name="generate_titles">
      <property name="text">
       <string>Include 'Titles' Section</string>
      </property>
     </widget>
    </item>
-   <item row="11" column="0">
+   <item row="12" column="0">
     <widget class="QCheckBox" name="generate_recently_added">
      <property name="text">
       <string>Include 'Recently Added' Section</string>
      </property>
     </widget>
    </item>
-   <item row="12" column="0">
+   <item row="13" column="0">
     <widget class="QCheckBox" name="numbers_as_text">
      <property name="text">
       <string>Sort numbers as text</string>
      </property>
     </widget>
    </item>
-   <item row="10" column="0">
+   <item row="11" column="0">
     <widget class="QCheckBox" name="generate_series">
      <property name="text">
       <string>Include 'Series' Section</string>
      </property>
     </widget>
    </item>
-   <item row="2" column="1">
+   <item row="3" column="1">
     <widget class="QLineEdit" name="wishlist_tag"/>
    </item>
-   <item row="2" column="0">
+   <item row="3" column="0">
     <widget class="QLabel" name="label_5">
      <property name="text">
       <string>Wishlist tag:</string>
      </property>
     </widget>
    </item>
+   <item row="2" column="1">
+    <layout class="QHBoxLayout" name="read_spec_hl">
+     <property name="sizeConstraint">
+      <enum>QLayout::SetMinimumSize</enum>
+     </property>
+     <item>
+      <widget class="QComboBox" name="read_source_field_cb">
+       <property name="sizePolicy">
+        <sizepolicy hsizetype="MinimumExpanding" vsizetype="Fixed">
+         <horstretch>0</horstretch>
+         <verstretch>0</verstretch>
+        </sizepolicy>
+       </property>
+       <property name="toolTip">
+        <string>Source column for read book</string>
+       </property>
+       <property name="statusTip">
+        <string/>
+       </property>
+      </widget>
+     </item>
+     <item>
+      <widget class="QLineEdit" name="read_pattern">
+       <property name="toolTip">
+        <string>Pattern for read book</string>
+       </property>
+       <property name="statusTip">
+        <string/>
+       </property>
+      </widget>
+     </item>
+    </layout>
+   </item>
+   <item row="2" column="0">
+    <widget class="QLabel" name="label_3">
+     <property name="text">
+      <string>Books marked as read:</string>
+     </property>
+    </widget>
+   </item>
   </layout>
  </widget>
  <resources/>

src/calibre/gui2/convert/metadata.py

@@ -17,6 +17,7 @@ from calibre.ebooks.metadata import authors_to_string, string_to_authors, \
 from calibre.ebooks.metadata.opf2 import metadata_to_opf
 from calibre.ptempfile import PersistentTemporaryFile
 from calibre.gui2.convert import Widget
+from calibre.utils.icu import sort_key
 
 def create_opf_file(db, book_id):
     mi = db.get_metadata(book_id, index_is_id=True)
@@ -102,7 +103,7 @@ class MetadataWidget(Widget, Ui_Form):
 
     def initalize_authors(self):
         all_authors = self.db.all_authors()
-        all_authors.sort(cmp=lambda x, y : cmp(x[1], y[1]))
+        all_authors.sort(key=lambda x : sort_key(x[1]))
 
         for i in all_authors:
             id, name = i
@@ -117,7 +118,7 @@ class MetadataWidget(Widget, Ui_Form):
 
     def initialize_series(self):
         all_series = self.db.all_series()
-        all_series.sort(cmp=lambda x, y : cmp(x[1], y[1]))
+        all_series.sort(key=lambda x : sort_key(x[1]))
 
         for i in all_series:
             id, name = i
@@ -126,7 +127,7 @@ class MetadataWidget(Widget, Ui_Form):
 
     def initialize_publisher(self):
         all_publishers = self.db.all_publishers()
-        all_publishers.sort(cmp=lambda x, y : cmp(x[1], y[1]))
+        all_publishers.sort(key=lambda x : sort_key(x[1]))
 
         for i in all_publishers:
             id, name = i

src/calibre/gui2/custom_column_widgets.py

@@ -15,8 +15,9 @@ from PyQt4.Qt import QComboBox, QLabel, QSpinBox, QDoubleSpinBox, QDateEdit, \
 
 from calibre.utils.date import qt_to_dt, now
 from calibre.gui2.widgets import TagsLineEdit, EnComboBox
-from calibre.gui2 import UNDEFINED_QDATE
+from calibre.gui2 import UNDEFINED_QDATE, error_dialog
 from calibre.utils.config import tweaks
+from calibre.utils.icu import sort_key
 
 class Base(object):
 
@@ -207,7 +208,7 @@ class Text(Base):
 
     def setup_ui(self, parent):
         values = self.all_values = list(self.db.all_custom(num=self.col_id))
-        values.sort(cmp = lambda x,y: cmp(x.lower(), y.lower()))
+        values.sort(key=sort_key)
         if self.col_metadata['is_multiple']:
             w = TagsLineEdit(parent, values)
             w.setSizePolicy(QSizePolicy.Minimum, QSizePolicy.Preferred)
@@ -256,7 +257,7 @@ class Series(Base):
 
     def setup_ui(self, parent):
         values = self.all_values = list(self.db.all_custom(num=self.col_id))
-        values.sort(cmp = lambda x,y: cmp(x.lower(), y.lower()))
+        values.sort(key=sort_key)
         w = EnComboBox(parent)
         w.setSizeAdjustPolicy(w.AdjustToMinimumContentsLengthWithIcon)
         w.setMinimumContentsLength(25)
@@ -310,6 +311,49 @@ class Series(Base):
             self.db.set_custom(book_id, val, extra=s_index,
                                num=self.col_id, notify=notify, commit=False)
 
+class Enumeration(Base):
+
+    def setup_ui(self, parent):
+        self.parent = parent
+        self.widgets = [QLabel('&'+self.col_metadata['name']+':', parent),
+                QComboBox(parent)]
+        w = self.widgets[1]
+        vals = self.col_metadata['display']['enum_values']
+        w.addItem('')
+        for v in vals:
+            w.addItem(v)
+
+    def initialize(self, book_id):
+        val = self.db.get_custom(book_id, num=self.col_id, index_is_id=True)
+        val = self.normalize_db_val(val)
+        self.initial_val = val
+        idx = self.widgets[1].findText(val)
+        if idx < 0:
+            error_dialog(self.parent, '',
+                    _('The enumeration "{0}" contains an invalid value '
+                      'that will be set to the default').format(
+                                            self.col_metadata['name']),
+                    show=True, show_copy_button=False)
+
+            idx = 0
+        self.widgets[1].setCurrentIndex(idx)
+
+    def setter(self, val):
+        self.widgets[1].setCurrentIndex(self.widgets[1].findText(val))
+
+    def getter(self):
+        return unicode(self.widgets[1].currentText())
+
+    def normalize_db_val(self, val):
+        if val is None:
+            val = ''
+        return val
+
+    def normalize_ui_val(self, val):
+        if not val:
+            val = None
+        return val
+
 widgets = {
         'bool' : Bool,
         'rating' : Rating,
@@ -319,13 +363,13 @@ widgets = {
         'text' : Text,
         'comments': Comments,
         'series': Series,
+        'enumeration': Enumeration
 }
 
-def field_sort(y, z, x=None):
+def field_sort_key(y, x=None):
-    m1, m2 = x[y], x[z]
+    m1 = x[y]
     n1 = 'zzzzz' if m1['datatype'] == 'comments' else m1['name']
-    n2 = 'zzzzz' if m2['datatype'] == 'comments' else m2['name']
+    return sort_key(n1)
-    return cmp(n1.lower(), n2.lower())
 
 def populate_metadata_page(layout, db, book_id, bulk=False, two_column=False, parent=None):
     def widget_factory(type, col):
@@ -337,7 +381,7 @@ def populate_metadata_page(layout, db, book_id, bulk=False, two_column=False, pa
         return w
     x = db.custom_column_num_map
     cols = list(x)
-    cols.sort(cmp=partial(field_sort, x=x))
+    cols.sort(key=partial(field_sort_key, x=x))
     count_non_comment = len([c for c in cols if x[c]['datatype'] != 'comments'])
 
     layout.setColumnStretch(1, 10)
@@ -482,7 +526,7 @@ class BulkSeries(BulkBase):
 
     def setup_ui(self, parent):
         values = self.all_values = list(self.db.all_custom(num=self.col_id))
-        values.sort(cmp = lambda x,y: cmp(x.lower(), y.lower()))
+        values.sort(key=sort_key)
         w = EnComboBox(parent)
         w.setSizeAdjustPolicy(w.AdjustToMinimumContentsLengthWithIcon)
         w.setMinimumContentsLength(25)
@@ -551,6 +595,61 @@ class BulkSeries(BulkBase):
             self.db.set_custom_bulk(book_ids, val, extras=extras,
                                    num=self.col_id, notify=notify)
 
+class BulkEnumeration(BulkBase, Enumeration):
+
+    def get_initial_value(self, book_ids):
+        value = None
+        ret_value = None
+        dialog_shown = False
+        for book_id in book_ids:
+            val = self.db.get_custom(book_id, num=self.col_id, index_is_id=True)
+            if val and val not in self.col_metadata['display']['enum_values']:
+                if not dialog_shown:
+                    error_dialog(self.parent, '',
+                            _('The enumeration "{0}" contains invalid values '
+                              'that will not appear in the list').format(
+                                                    self.col_metadata['name']),
+                            show=True, show_copy_button=False)
+                    dialog_shown = True
+                ret_value = ' nochange '
+            elif value is not None and value != val:
+                ret_value = ' nochange '
+            value = val
+        if ret_value is None:
+            return value
+        return ret_value
+
+    def setup_ui(self, parent):
+        self.parent = parent
+        self.widgets = [QLabel('&'+self.col_metadata['name']+':', parent),
+                QComboBox(parent)]
+        w = self.widgets[1]
+        vals = self.col_metadata['display']['enum_values']
+        w.addItem('Do Not Change')
+        w.addItem('')
+        for v in vals:
+            w.addItem(v)
+
+    def getter(self):
+        if self.widgets[1].currentIndex() == 0:
+            return ' nochange '
+        return unicode(self.widgets[1].currentText())
+
+    def setter(self, val):
+        if val == ' nochange ':
+            self.widgets[1].setCurrentIndex(0)
+        else:
+            if val is None:
+                self.widgets[1].setCurrentIndex(1)
+            else:
+                self.widgets[1].setCurrentIndex(self.widgets[1].findText(val))
+
+    def commit(self, book_ids, notify=False):
+        val = self.gui_val
+        val = self.normalize_ui_val(val)
+        if val != self.initial_val and val != ' nochange ':
+            self.db.set_custom_bulk(book_ids, val, num=self.col_id, notify=notify)
+
 class RemoveTags(QWidget):
 
     def __init__(self, parent, values):
@@ -579,7 +678,7 @@ class BulkText(BulkBase):
 
     def setup_ui(self, parent):
         values = self.all_values = list(self.db.all_custom(num=self.col_id))
-        values.sort(cmp = lambda x,y: cmp(x.lower(), y.lower()))
+        values.sort(key=sort_key)
         if self.col_metadata['is_multiple']:
             w = TagsLineEdit(parent, values)
             w.setSizePolicy(QSizePolicy.Minimum, QSizePolicy.Preferred)
@@ -656,4 +755,5 @@ bulk_widgets = {
         'datetime': BulkDateTime,
         'text' : BulkText,
         'series': BulkSeries,
+        'enumeration': BulkEnumeration,
 }

src/calibre/gui2/dialogs/book_info.py

@@ -15,12 +15,13 @@ from calibre.library.comments import comments_to_html
 
 class BookInfo(QDialog, Ui_BookInfo):
 
-    def __init__(self, parent, view, row):
+    def __init__(self, parent, view, row, view_func):
         QDialog.__init__(self, parent)
         Ui_BookInfo.__init__(self)
         self.setupUi(self)
         self.cover_pixmap = None
         self.comments.sizeHint = self.comments_size_hint
+        self.view_func = view_func
 
         desktop = QCoreApplication.instance().desktop()
         screen_height = desktop.availableGeometry().height() - 100
@@ -58,10 +59,7 @@ class BookInfo(QDialog, Ui_BookInfo):
         if os.sep in path:
             open_local_file(path)
         else:
-            path = self.view.model().db.format_abspath(self.current_row, path)
+            self.view_func(self.view.model().id(self.current_row), path)
-            if path is not None:
-                open_local_file(path)
-
 
     def next(self):
         row = self.view.currentIndex().row()

src/calibre/gui2/dialogs/catalog.py

@@ -19,7 +19,7 @@ from calibre.customize.ui import catalog_plugins
 class Catalog(QDialog, Ui_Dialog):
     ''' Catalog Dialog builder'''
 
-    def __init__(self, parent, dbspec, ids):
+    def __init__(self, parent, dbspec, ids, db):
         import re, cStringIO
         from calibre import prints as info
         from PyQt4.uic import compileUi
@@ -51,7 +51,7 @@ class Catalog(QDialog, Ui_Dialog):
                     catalog_widget = __import__('calibre.gui2.catalog.'+name,
                             fromlist=[1])
                     pw = catalog_widget.PluginWidget()
-                    pw.initialize(name)
+                    pw.initialize(name, db)
                     pw.ICON = I('forward.png')
                     self.widgets.append(pw)
                     [self.fmts.append([file_type.upper(), pw.sync_enabled,pw]) for file_type in plugin.file_types]

src/calibre/gui2/dialogs/metadata_bulk.py

@@ -6,7 +6,7 @@ __copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
 import re
 
 from PyQt4.Qt import Qt, QDialog, QGridLayout, QVBoxLayout, QFont, QLabel, \
-                     pyqtSignal
+                     pyqtSignal, QDialogButtonBox
 from PyQt4 import QtGui
 
 from calibre.gui2.dialogs.metadata_bulk_ui import Ui_MetadataBulkDialog
@@ -17,6 +17,7 @@ from calibre.gui2 import error_dialog
 from calibre.gui2.progress_indicator import ProgressIndicator
 from calibre.utils.config import dynamic
 from calibre.utils.titlecase import titlecase
+from calibre.utils.icu import sort_key
 
 class MyBlockingBusy(QDialog):
 
@@ -197,7 +198,7 @@ class MetadataBulkDialog(QDialog, Ui_MetadataBulkDialog):
                             _('Append to field'),
                         ]
 
-    def __init__(self, window, rows, model):
+    def __init__(self, window, rows, model, tab):
         QDialog.__init__(self, window)
         Ui_MetadataBulkDialog.__init__(self)
         self.setupUi(self)
@@ -232,8 +233,20 @@ class MetadataBulkDialog(QDialog, Ui_MetadataBulkDialog):
             self.create_custom_column_editors()
 
         self.prepare_search_and_replace()
+
+        self.button_box.clicked.connect(self.button_clicked)
+        self.button_box.button(QDialogButtonBox.Apply).setToolTip(_(
+            'Immediately make all changes without closing the dialog. '
+            'This operation cannot be canceled or undone'))
+        self.do_again = False
+        self.central_widget.setCurrentIndex(tab)
         self.exec_()
 
+    def button_clicked(self, which):
+        if which == self.button_box.button(QDialogButtonBox.Apply):
+            self.do_again = True
+            self.accept()
+
     def prepare_search_and_replace(self):
         self.search_for.initialize('bulk_edit_search_for')
         self.replace_with.initialize('bulk_edit_replace_with')
@@ -243,7 +256,7 @@ class MetadataBulkDialog(QDialog, Ui_MetadataBulkDialog):
         fm = self.db.field_metadata
         for f in fm:
             if (f in ['author_sort'] or
-                    (fm[f]['datatype'] in ['text', 'series']
+                    (fm[f]['datatype'] in ['text', 'series', 'enumeration']
                      and fm[f].get('search_terms', None)
                      and f not in ['formats', 'ondevice', 'sort'])):
                 self.all_fields.append(f)
@@ -582,7 +595,7 @@ class MetadataBulkDialog(QDialog, Ui_MetadataBulkDialog):
 
     def initalize_authors(self):
         all_authors = self.db.all_authors()
-        all_authors.sort(cmp=lambda x, y : cmp(x[1].lower(), y[1].lower()))
+        all_authors.sort(key=lambda x : sort_key(x[1]))
 
         for i in all_authors:
             id, name = i
@@ -592,7 +605,7 @@ class MetadataBulkDialog(QDialog, Ui_MetadataBulkDialog):
 
     def initialize_series(self):
         all_series = self.db.all_series()
-        all_series.sort(cmp=lambda x, y : cmp(x[1], y[1]))
+        all_series.sort(key=lambda x : sort_key(x[1]))
 
         for i in all_series:
             id, name = i
@@ -601,7 +614,7 @@ class MetadataBulkDialog(QDialog, Ui_MetadataBulkDialog):
 
     def initialize_publisher(self):
         all_publishers = self.db.all_publishers()
-        all_publishers.sort(cmp=lambda x, y : cmp(x[1], y[1]))
+        all_publishers.sort(key=lambda x : sort_key(x[1]))
 
         for i in all_publishers:
             id, name = i
@@ -692,7 +705,6 @@ class MetadataBulkDialog(QDialog, Ui_MetadataBulkDialog):
         self.db.clean()
         return QDialog.accept(self)
 
-
     def series_changed(self, *args):
         self.write_series = True
 

src/calibre/gui2/dialogs/metadata_bulk.ui

@@ -710,7 +710,7 @@ nothing should be put between the original text and the inserted text</string>
       <enum>Qt::Horizontal</enum>
      </property>
      <property name="standardButtons">
-      <set>QDialogButtonBox::Cancel|QDialogButtonBox::Ok</set>
+      <set>QDialogButtonBox::Apply|QDialogButtonBox::Cancel|QDialogButtonBox::Ok</set>
      </property>
     </widget>
    </item>

src/calibre/gui2/dialogs/metadata_single.py

@@ -7,9 +7,11 @@ add/remove formats
 '''
 
 import os, re, time, traceback, textwrap
+from functools import partial
 
 from PyQt4.Qt import SIGNAL, QObject, Qt, QTimer, QThread, QDate, \
-    QPixmap, QListWidgetItem, QDialog, pyqtSignal, QMessageBox
+    QPixmap, QListWidgetItem, QDialog, pyqtSignal, QMessageBox, QIcon, \
+    QPushButton
 
 from calibre.gui2 import error_dialog, file_icon_provider, dynamic, \
                            choose_files, choose_images, ResizableDialog, \
@@ -26,12 +28,13 @@ from calibre.ebooks.metadata.meta import get_metadata
 from calibre.ebooks.metadata import MetaInformation
 from calibre.utils.config import prefs, tweaks
 from calibre.utils.date import qt_to_dt, local_tz, utcfromtimestamp
+from calibre.utils.icu import sort_key
 from calibre.customize.ui import run_plugins_on_import, get_isbndb_key
 from calibre.gui2.preferences.social import SocialMetadata
 from calibre.gui2.custom_column_widgets import populate_metadata_page
 from calibre import strftime
 
-class CoverFetcher(QThread):
+class CoverFetcher(QThread): # {{{
 
     def __init__(self, username, password, isbn, timeout, title, author):
         self.username = username.strip() if username else username
@@ -74,9 +77,9 @@ class CoverFetcher(QThread):
             self.traceback = traceback.format_exc()
             print self.traceback
 
+# }}}
 
-
+class Format(QListWidgetItem): # {{{
-class Format(QListWidgetItem):
 
     def __init__(self, parent, ext, size, path=None, timestamp=None):
         self.path = path
@@ -92,15 +95,70 @@ class Format(QListWidgetItem):
             self.setToolTip(text)
             self.setStatusTip(text)
 
+# }}}
+
 
 class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
 
     COVER_FETCH_TIMEOUT = 240 # seconds
     view_format = pyqtSignal(object)
 
+    # Cover processing {{{
+
+    def set_cover(self):
+        mi, ext = self.get_selected_format_metadata()
+        if mi is None:
+            return
+        cdata = None
+        if mi.cover and os.access(mi.cover, os.R_OK):
+            cdata = open(mi.cover).read()
+        elif mi.cover_data[1] is not None:
+            cdata = mi.cover_data[1]
+        if cdata is None:
+            error_dialog(self, _('Could not read cover'),
+                         _('Could not read cover from %s format')%ext).exec_()
+            return
+        pix = QPixmap()
+        pix.loadFromData(cdata)
+        if pix.isNull():
+            error_dialog(self, _('Could not read cover'),
+                         _('The cover in the %s format is invalid')%ext).exec_()
+            return
+        self.cover.setPixmap(pix)
+        self.update_cover_tooltip()
+        self.cover_changed = True
+        self.cpixmap = pix
+        self.cover_data = cdata
+
+    def trim_cover(self, *args):
+        from calibre.utils.magick import Image
+        cdata = self.cover_data
+        if not cdata:
+            return
+        im = Image()
+        im.load(cdata)
+        im.trim(10)
+        cdata = im.export('png')
+        pix = QPixmap()
+        pix.loadFromData(cdata)
+        self.cover.setPixmap(pix)
+        self.update_cover_tooltip()
+        self.cover_changed = True
+        self.cpixmap = pix
+        self.cover_data = cdata
+
+
+
+    def update_cover_tooltip(self):
+        p = self.cover.pixmap()
+        self.cover.setToolTip(_('Cover size: %dx%d pixels') %
+                (p.width(), p.height()))
+
+
     def do_reset_cover(self, *args):
         pix = QPixmap(I('default_cover.png'))
         self.cover.setPixmap(pix)
+        self.update_cover_tooltip()
         self.cover_changed = True
         self.cover_data = None
 
@@ -136,6 +194,7 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
                 else:
                     self.cover_path.setText(_file)
                     self.cover.setPixmap(pix)
+                    self.update_cover_tooltip()
                     self.cover_changed = True
                     self.cpixmap = pix
                     self.cover_data = cover
@@ -161,9 +220,80 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
         pix = QPixmap()
         pix.loadFromData(self.cover_data)
         self.cover.setPixmap(pix)
+        self.update_cover_tooltip()
         self.cover_changed = True
         self.cpixmap = pix
 
+    def cover_dropped(self, cover_data):
+        self.cover_changed = True
+        self.cover_data = cover_data
+        self.update_cover_tooltip()
+
+    def fetch_cover(self):
+        isbn   = re.sub(r'[^0-9a-zA-Z]', '', unicode(self.isbn.text())).strip()
+        self.fetch_cover_button.setEnabled(False)
+        self.setCursor(Qt.WaitCursor)
+        title, author = map(unicode, (self.title.text(), self.authors.text()))
+        self.cover_fetcher = CoverFetcher(None, None, isbn,
+                                            self.timeout, title, author)
+        self.cover_fetcher.start()
+        self._hangcheck = QTimer(self)
+        self.connect(self._hangcheck, SIGNAL('timeout()'), self.hangcheck)
+        self.cf_start_time = time.time()
+        self.pi.start(_('Downloading cover...'))
+        self._hangcheck.start(100)
+
+    def hangcheck(self):
+        if not self.cover_fetcher.isFinished() and \
+            time.time()-self.cf_start_time < self.COVER_FETCH_TIMEOUT:
+            return
+
+        self._hangcheck.stop()
+        try:
+            if self.cover_fetcher.isRunning():
+                self.cover_fetcher.terminate()
+                error_dialog(self, _('Cannot fetch cover'),
+                    _('<b>Could not fetch cover.</b><br/>')+
+                    _('The download timed out.')).exec_()
+                return
+            if self.cover_fetcher.needs_isbn:
+                error_dialog(self, _('Cannot fetch cover'),
+                    _('Could not find cover for this book. Try '
+                      'specifying the ISBN first.')).exec_()
+                return
+            if self.cover_fetcher.exception is not None:
+                err = self.cover_fetcher.exception
+                error_dialog(self, _('Cannot fetch cover'),
+                    _('<b>Could not fetch cover.</b><br/>')+unicode(err)).exec_()
+                return
+            if self.cover_fetcher.errors and self.cover_fetcher.cover_data is None:
+                details = u'\n\n'.join([e[-1] + ': ' + e[1] for e in self.cover_fetcher.errors])
+                error_dialog(self, _('Cannot fetch cover'),
+                    _('<b>Could not fetch cover.</b><br/>') +
+                    _('For the error message from each cover source, '
+                      'click Show details below.'), det_msg=details, show=True)
+                return
+
+            pix = QPixmap()
+            pix.loadFromData(self.cover_fetcher.cover_data)
+            if pix.isNull():
+                error_dialog(self, _('Bad cover'),
+                             _('The cover is not a valid picture')).exec_()
+            else:
+                self.cover.setPixmap(pix)
+                self.update_cover_tooltip()
+                self.cover_changed = True
+                self.cpixmap = pix
+                self.cover_data = self.cover_fetcher.cover_data
+        finally:
+            self.fetch_cover_button.setEnabled(True)
+            self.unsetCursor()
+            self.pi.stop()
+
+
+    # }}}
+
+    # Formats processing {{{
     def add_format(self, x):
         files = choose_files(self, 'add formats dialog',
                              _("Choose formats for ") + unicode((self.title.text())),
@@ -276,48 +406,6 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
             self.comments.setPlainText(mi.comments)
 
 
-    def set_cover(self):
-        mi, ext = self.get_selected_format_metadata()
-        if mi is None:
-            return
-        cdata = None
-        if mi.cover and os.access(mi.cover, os.R_OK):
-            cdata = open(mi.cover).read()
-        elif mi.cover_data[1] is not None:
-            cdata = mi.cover_data[1]
-        if cdata is None:
-            error_dialog(self, _('Could not read cover'),
-                         _('Could not read cover from %s format')%ext).exec_()
-            return
-        pix = QPixmap()
-        pix.loadFromData(cdata)
-        if pix.isNull():
-            error_dialog(self, _('Could not read cover'),
-                         _('The cover in the %s format is invalid')%ext).exec_()
-            return
-        self.cover.setPixmap(pix)
-        self.cover_changed = True
-        self.cpixmap = pix
-        self.cover_data = cdata
-
-    def trim_cover(self, *args):
-        from calibre.utils.magick import Image
-        cdata = self.cover_data
-        if not cdata:
-            return
-        im = Image()
-        im.load(cdata)
-        im.trim(10)
-        cdata = im.export('png')
-        pix = QPixmap()
-        pix.loadFromData(cdata)
-        self.cover.setPixmap(pix)
-        self.cover_changed = True
-        self.cpixmap = pix
-        self.cover_data = cdata
-
-
-
     def sync_formats(self):
         old_extensions, new_extensions, paths = set(), set(), {}
         for row in range(self.formats.count()):
@@ -338,11 +426,14 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
             if ext not in extensions:
                 self.db.remove_format(self.row, ext, notify=False)
 
-    def do_cancel_all(self):
+    def show_format(self, item, *args):
-        self.cancel_all = True
+        fmt = item.ext
-        self.reject()
+        self.view_format.emit(fmt)
 
-    def __init__(self, window, row, db, accepted_callback=None, cancel_all=False):
+    # }}}
+
+    def __init__(self, window, row, db, prev=None,
+            next_=None):
         ResizableDialog.__init__(self, window)
         self.bc_box.layout().setAlignment(self.cover, Qt.AlignCenter|Qt.AlignHCenter)
         self.cancel_all = False
@@ -354,16 +445,27 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
                 _(' The red color indicates that the current '
                     'author sort does not match the current author'))
 
-        if cancel_all:
+        self.row_delta = 0
-            self.__abort_button = self.button_box.addButton(self.button_box.Abort)
+        if prev:
-            self.__abort_button.setToolTip(_('Abort the editing of all remaining books'))
+            self.prev_button = QPushButton(QIcon(I('back.png')), _('Previous'),
-            self.connect(self.__abort_button, SIGNAL('clicked()'),
+                    self)
-                    self.do_cancel_all)
+            self.button_box.addButton(self.prev_button, self.button_box.ActionRole)
+            tip = _('Save changes and edit the metadata of %s')%prev
+            self.prev_button.setToolTip(tip)
+            self.prev_button.clicked.connect(partial(self.next_triggered,
+                -1))
+        if next_:
+            self.next_button = QPushButton(QIcon(I('forward.png')), _('Next'),
+                    self)
+            self.button_box.addButton(self.next_button, self.button_box.ActionRole)
+            tip = _('Save changes and edit the metadata of %s')%next_
+            self.next_button.setToolTip(tip)
+            self.next_button.clicked.connect(partial(self.next_triggered, 1))
+
         self.splitter.setStretchFactor(100, 1)
         self.read_state()
         self.db = db
         self.pi = ProgressIndicator(self)
-        self.accepted_callback = accepted_callback
         self.id = db.id(row)
         self.row = row
         self.cover_data = None
@@ -412,6 +514,8 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
         self.connect(self.reset_cover, SIGNAL('clicked()'), self.do_reset_cover)
         self.connect(self.swap_button, SIGNAL('clicked()'), self.swap_title_author)
         self.timeout = float(prefs['network_timeout'])
+
+
         self.title.setText(db.title(row))
         isbn = db.isbn(self.id, index_is_id=True)
         if not isbn:
@@ -472,6 +576,7 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
         else:
             self.cover_data = cover
         self.cover.setPixmap(pm)
+        self.update_cover_tooltip()
         self.original_series_name = unicode(self.series.text()).strip()
         if len(db.custom_column_label_map) == 0:
             self.central_widget.tabBar().setVisible(False)
@@ -479,6 +584,9 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
             self.create_custom_column_editors()
         self.generate_cover_button.clicked.connect(self.generate_cover)
 
+        self.original_author = unicode(self.authors.text()).strip()
+        self.original_title = unicode(self.title.text()).strip()
+
     def create_custom_column_editors(self):
         w = self.central_widget.widget(1)
         layout = w.layout()
@@ -531,10 +639,6 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
             self.isbn.setStyleSheet('QLineEdit { background-color: rgba(255,0,0,20%) }')
             self.isbn.setToolTip(_('This ISBN number is invalid'))
 
-    def show_format(self, item, *args):
-        fmt = item.ext
-        self.view_format.emit(fmt)
-
     def deduce_author_sort(self):
         au = unicode(self.authors.text())
         au = re.sub(r'\s+et al\.$', '', au)
@@ -547,9 +651,6 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
         self.authors.setText(title)
         self.author_sort.setText('')
 
-    def cover_dropped(self, cover_data):
-        self.cover_changed = True
-        self.cover_data = cover_data
 
     def initialize_combos(self):
         self.initalize_authors()
@@ -560,7 +661,7 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
 
     def initalize_authors(self):
         all_authors = self.db.all_authors()
-        all_authors.sort(cmp=lambda x, y : cmp(x[1], y[1]))
+        all_authors.sort(key=lambda x : sort_key(x[1]))
         for i in all_authors:
             id, name = i
             name = [name.strip().replace('|', ',') for n in name.split(',')]
@@ -575,7 +676,7 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
     def initialize_series(self):
         self.series.setSizeAdjustPolicy(self.series.AdjustToContentsOnFirstShow)
         all_series = self.db.all_series()
-        all_series.sort(cmp=lambda x, y : cmp(x[1], y[1]))
+        all_series.sort(key=lambda x : sort_key(x[1]))
         series_id = self.db.series_id(self.row)
         idx, c = None, 0
         for i in all_series:
@@ -592,7 +693,7 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
 
     def initialize_publisher(self):
         all_publishers = self.db.all_publishers()
-        all_publishers.sort(cmp=lambda x, y : cmp(x[1], y[1]))
+        all_publishers.sort(key=lambda x : sort_key(x[1]))
         publisher_id = self.db.publisher_id(self.row)
         idx, c = None, 0
         for i in all_publishers:
@@ -625,66 +726,6 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
             self.tags.setText(tag_string)
             self.tags.update_tags_cache(self.db.all_tags())
 
-    def fetch_cover(self):
-        isbn   = re.sub(r'[^0-9a-zA-Z]', '', unicode(self.isbn.text())).strip()
-        self.fetch_cover_button.setEnabled(False)
-        self.setCursor(Qt.WaitCursor)
-        title, author = map(unicode, (self.title.text(), self.authors.text()))
-        self.cover_fetcher = CoverFetcher(None, None, isbn,
-                                            self.timeout, title, author)
-        self.cover_fetcher.start()
-        self._hangcheck = QTimer(self)
-        self.connect(self._hangcheck, SIGNAL('timeout()'), self.hangcheck)
-        self.cf_start_time = time.time()
-        self.pi.start(_('Downloading cover...'))
-        self._hangcheck.start(100)
-
-    def hangcheck(self):
-        if not self.cover_fetcher.isFinished() and \
-            time.time()-self.cf_start_time < self.COVER_FETCH_TIMEOUT:
-            return
-
-        self._hangcheck.stop()
-        try:
-            if self.cover_fetcher.isRunning():
-                self.cover_fetcher.terminate()
-                error_dialog(self, _('Cannot fetch cover'),
-                    _('<b>Could not fetch cover.</b><br/>')+
-                    _('The download timed out.')).exec_()
-                return
-            if self.cover_fetcher.needs_isbn:
-                error_dialog(self, _('Cannot fetch cover'),
-                    _('Could not find cover for this book. Try '
-                      'specifying the ISBN first.')).exec_()
-                return
-            if self.cover_fetcher.exception is not None:
-                err = self.cover_fetcher.exception
-                error_dialog(self, _('Cannot fetch cover'),
-                    _('<b>Could not fetch cover.</b><br/>')+unicode(err)).exec_()
-                return
-            if self.cover_fetcher.errors and self.cover_fetcher.cover_data is None:
-                details = u'\n\n'.join([e[-1] + ': ' + e[1] for e in self.cover_fetcher.errors])
-                error_dialog(self, _('Cannot fetch cover'),
-                    _('<b>Could not fetch cover.</b><br/>') +
-                    _('For the error message from each cover source, '
-                      'click Show details below.'), det_msg=details, show=True)
-                return
-
-            pix = QPixmap()
-            pix.loadFromData(self.cover_fetcher.cover_data)
-            if pix.isNull():
-                error_dialog(self, _('Bad cover'),
-                             _('The cover is not a valid picture')).exec_()
-            else:
-                self.cover.setPixmap(pix)
-                self.cover_changed = True
-                self.cpixmap = pix
-                self.cover_data = self.cover_fetcher.cover_data
-        finally:
-            self.fetch_cover_button.setEnabled(True)
-            self.unsetCursor()
-            self.pi.stop()
-
 
     def fetch_metadata(self):
         isbn   = re.sub(r'[^0-9a-zA-Z]', '', unicode(self.isbn.text()))
@@ -776,6 +817,10 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
             unicode(self.tags.text()).split(',')],
                 notify=notify, commit=commit)
 
+    def next_triggered(self, row_delta, *args):
+        self.row_delta = row_delta
+        self.accept()
+
     def accept(self):
         cf = getattr(self, 'cover_fetcher', None)
         if cf is not None and hasattr(cf, 'terminate'):
@@ -785,9 +830,10 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
             if self.formats_changed:
                 self.sync_formats()
             title = unicode(self.title.text()).strip()
-            self.db.set_title(self.id, title, notify=False)
+            if title != self.original_title:
+                self.db.set_title(self.id, title, notify=False)
             au = unicode(self.authors.text()).strip()
-            if au:
+            if au and au != self.original_author:
                 self.db.set_authors(self.id, string_to_authors(au), notify=False)
             aus = unicode(self.author_sort.text()).strip()
             if aus:
@@ -837,8 +883,6 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
             raise
         self.save_state()
         QDialog.accept(self)
-        if callable(self.accepted_callback):
-            self.accepted_callback(self.id)
 
     def reject(self, *args):
         cf = getattr(self, 'cover_fetcher', None)

src/calibre/gui2/dialogs/saved_search_editor.py

@@ -8,6 +8,7 @@ from PyQt4.QtGui import QDialog
 
 from calibre.gui2.dialogs.saved_search_editor_ui import Ui_SavedSearchEditor
 from calibre.utils.search_query_parser import saved_searches
+from calibre.utils.icu import sort_key
 from calibre.gui2.dialogs.confirm_delete import confirm
 
 class SavedSearchEditor(QDialog, Ui_SavedSearchEditor):
@@ -34,7 +35,7 @@ class SavedSearchEditor(QDialog, Ui_SavedSearchEditor):
 
     def populate_search_list(self):
         self.search_name_box.clear()
-        for name in sorted(self.searches.keys()):
+        for name in sorted(self.searches.keys(), key=sort_key):
             self.search_name_box.addItem(name)
 
     def add_search(self):

src/calibre/gui2/dialogs/scheduler.py

@@ -10,7 +10,8 @@ Scheduler for automated recipe downloads
 from datetime import timedelta
 
 from PyQt4.Qt import QDialog, SIGNAL, Qt, QTime, QObject, QMenu, \
-        QAction, QIcon, QMutex, QTimer, pyqtSignal
+        QAction, QIcon, QMutex, QTimer, pyqtSignal, QWidget, QHBoxLayout, \
+        QLabel
 
 from calibre.gui2.dialogs.scheduler_ui import Ui_Dialog
 from calibre.gui2.search_box import SearchBox2
@@ -28,15 +29,21 @@ class SchedulerDialog(QDialog, Ui_Dialog):
         self.recipe_model = recipe_model
         self.recipe_model.do_refresh()
 
+        self._cont = QWidget(self)
+        self._cont.l = QHBoxLayout()
+        self._cont.setLayout(self._cont.l)
+        self._cont.la = QLabel(_('&Search:'))
+        self._cont.l.addWidget(self._cont.la, 1)
         self.search = SearchBox2(self)
+        self._cont.l.addWidget(self.search, 100)
+        self._cont.la.setBuddy(self.search)
         self.search.setMinimumContentsLength(25)
         self.search.initialize('scheduler_search_history')
-        self.recipe_box.layout().insertWidget(0, self.search)
+        self.recipe_box.layout().insertWidget(0, self._cont)
         self.search.search.connect(self.recipe_model.search)
-        self.connect(self.recipe_model,  SIGNAL('searched(PyQt_PyObject)'),
+        self.recipe_model.searched.connect(self.search.search_done,
-                self.search.search_done)
+                type=Qt.QueuedConnection)
-        self.connect(self.recipe_model,  SIGNAL('searched(PyQt_PyObject)'),
+        self.recipe_model.searched.connect(self.search_done)
-                self.search_done)
         self.search.setFocus(Qt.OtherFocusReason)
         self.commit_on_change = True
 

src/calibre/gui2/dialogs/search.py

@@ -8,6 +8,7 @@ from PyQt4.QtGui import QDialog, QDialogButtonBox
 from calibre.gui2.dialogs.search_ui import Ui_Dialog
 from calibre.library.caches import CONTAINS_MATCH, EQUALS_MATCH
 from calibre.gui2 import gprefs
+from calibre.utils.icu import sort_key
 
 box_values = {}
 
@@ -18,8 +19,7 @@ class SearchDialog(QDialog, Ui_Dialog):
         self.setupUi(self)
         self.mc = ''
         searchables = sorted(db.field_metadata.searchable_fields(),
-                             lambda x, y: cmp(x if x[0] != '#' else x[1:],
+                             key=lambda x: sort_key(x if x[0] != '#' else x[1:]))
-                                              y if y[0] != '#' else y[1:]))
         self.general_combo.addItems(searchables)
 
         self.box_last_values = copy.deepcopy(box_values)

src/calibre/gui2/dialogs/tag_categories.py

@@ -9,6 +9,7 @@ from PyQt4.QtGui import QDialog, QIcon, QListWidgetItem
 from calibre.gui2.dialogs.tag_categories_ui import Ui_TagCategories
 from calibre.gui2.dialogs.confirm_delete import confirm
 from calibre.constants import islinux
+from calibre.utils.icu import sort_key
 
 class Item:
     def __init__(self, name, label, index, icon, exists):
@@ -85,7 +86,7 @@ class TagCategories(QDialog, Ui_TagCategories):
                     # remove any references to a category that no longer exists
                     del self.categories[cat][item]
 
-        self.all_items_sorted = sorted(self.all_items, cmp=lambda x,y: cmp(x.name.lower(), y.name.lower()))
+        self.all_items_sorted = sorted(self.all_items, key=lambda x: sort_key(x.name))
         self.display_filtered_categories(0)
 
         for v in category_names:
@@ -135,7 +136,7 @@ class TagCategories(QDialog, Ui_TagCategories):
             index = self.all_items[node.data(Qt.UserRole).toPyObject()].index
             if index not in self.applied_items:
                 self.applied_items.append(index)
-        self.applied_items.sort(cmp=lambda x, y:cmp(self.all_items[x].name.lower(), self.all_items[y].name.lower()))
+        self.applied_items.sort(key=lambda x:sort_key(self.all_items[x]))
         self.display_filtered_categories(None)
 
     def unapply_tags(self, node=None):
@@ -198,5 +199,5 @@ class TagCategories(QDialog, Ui_TagCategories):
             self.categories[self.current_cat_name] = l
 
     def populate_category_list(self):
-        for n in sorted(self.categories.keys(), cmp=lambda x,y: cmp(x.lower(), y.lower())):
+        for n in sorted(self.categories.keys(), key=sort_key):
             self.category_box.addItem(n)

src/calibre/gui2/dialogs/tag_editor.py

@@ -6,12 +6,10 @@ from PyQt4.QtGui import QDialog
 from calibre.gui2.dialogs.tag_editor_ui import Ui_TagEditor
 from calibre.gui2 import question_dialog, error_dialog
 from calibre.constants import islinux
+from calibre.utils.icu import sort_key
 
 class TagEditor(QDialog, Ui_TagEditor):
 
-    def tag_cmp(self, x, y):
-        return cmp(x.lower(), y.lower())
-
     def __init__(self, window, db, index=None):
         QDialog.__init__(self, window)
         Ui_TagEditor.__init__(self)
@@ -25,7 +23,7 @@ class TagEditor(QDialog, Ui_TagEditor):
             tags = []
         if tags:
             tags = [tag.strip() for tag in tags.split(',') if tag.strip()]
-            tags.sort(cmp=self.tag_cmp)
+            tags.sort(key=sort_key)
             for tag in tags:
                 self.applied_tags.addItem(tag)
         else:
@@ -35,7 +33,7 @@ class TagEditor(QDialog, Ui_TagEditor):
 
         all_tags = [tag for tag in self.db.all_tags()]
         all_tags = list(set(all_tags))
-        all_tags.sort(cmp=self.tag_cmp)
+        all_tags.sort(key=sort_key)
         for tag in all_tags:
             if tag not in tags:
                 self.available_tags.addItem(tag)
@@ -82,7 +80,7 @@ class TagEditor(QDialog, Ui_TagEditor):
             self.tags.append(tag)
             self.available_tags.takeItem(self.available_tags.row(item))
 
-        self.tags.sort(cmp=self.tag_cmp)
+        self.tags.sort(key=sort_key)
         self.applied_tags.clear()
         for tag in self.tags:
             self.applied_tags.addItem(tag)
@@ -96,14 +94,14 @@ class TagEditor(QDialog, Ui_TagEditor):
             self.tags.remove(tag)
             self.available_tags.addItem(tag)
 
-        self.tags.sort(cmp=self.tag_cmp)
+        self.tags.sort(key=sort_key)
         self.applied_tags.clear()
         for tag in self.tags:
             self.applied_tags.addItem(tag)
 
         items = [unicode(self.available_tags.item(x).text()) for x in
                 range(self.available_tags.count())]
-        items.sort(cmp=self.tag_cmp)
+        items.sort(key=sort_key)
         self.available_tags.clear()
         for item in items:
             self.available_tags.addItem(item)
@@ -117,7 +115,7 @@ class TagEditor(QDialog, Ui_TagEditor):
             if tag not in self.tags:
                 self.tags.append(tag)
 
-        self.tags.sort(cmp=self.tag_cmp)
+        self.tags.sort(key=sort_key)
         self.applied_tags.clear()
         for tag in self.tags:
             self.applied_tags.addItem(tag)

src/calibre/gui2/dialogs/tag_list_editor.py

@@ -39,7 +39,7 @@ class ListWidgetItem(QListWidgetItem):
 
 class TagListEditor(QDialog, Ui_TagListEditor):
 
-    def __init__(self, window, tag_to_match, data, compare):
+    def __init__(self, window, tag_to_match, data, key):
         QDialog.__init__(self, window)
         Ui_TagListEditor.__init__(self)
         self.setupUi(self)
@@ -54,7 +54,7 @@ class TagListEditor(QDialog, Ui_TagListEditor):
 
         for k,v in data:
             self.all_tags[v] = k
-        for tag in sorted(self.all_tags.keys(), cmp=compare):
+        for tag in sorted(self.all_tags.keys(), key=key):
             item = ListWidgetItem(tag)
             item.setData(Qt.UserRole, self.all_tags[tag])
             self.available_tags.addItem(item)

src/calibre/gui2/dialogs/user_profiles.py

@@ -13,6 +13,7 @@ from calibre.gui2 import error_dialog, question_dialog, open_url, \
                          choose_files, ResizableDialog, NONE
 from calibre.gui2.widgets import PythonHighlighter
 from calibre.ptempfile import PersistentTemporaryFile
+from calibre.utils.icu import sort_key
 
 class CustomRecipeModel(QAbstractListModel):
 
@@ -256,7 +257,7 @@ class %(classname)s(%(base_class)s):
     def add_builtin_recipe(self):
         from calibre.web.feeds.recipes.collection import \
             get_builtin_recipe_by_title, get_builtin_recipe_titles
-        items = sorted(get_builtin_recipe_titles())
+        items = sorted(get_builtin_recipe_titles(), key=sort_key)
 
 
         title, ok = QInputDialog.getItem(self, _('Pick recipe'), _('Pick the recipe to customize'),

src/calibre/gui2/init.py

@@ -86,6 +86,10 @@ class LibraryViewMixin(object): # {{{
         if view is self.current_view():
             self.search.search_done(ok)
             self.set_number_of_books_shown()
+            if ok:
+                v = self.current_view()
+                if hasattr(v, 'set_current_row'):
+                    v.set_current_row(0)
 
     # }}}
 

src/calibre/gui2/layout.py

@@ -182,7 +182,7 @@ class SearchBar(QWidget): # {{{
         l.addWidget(self.search_button)
         self.search_button.setSizePolicy(QSizePolicy.Minimum,
                 QSizePolicy.Minimum)
-        self.search_button.clicked.connect(parent.search.do_search)
+        self.search_button.clicked.connect(parent.do_search_button)
         self.search_button.setToolTip(
             _('Do Quick Search (you can also press the Enter key)'))
 

src/calibre/gui2/library/delegates.py

@@ -20,6 +20,7 @@ from calibre.gui2.widgets import EnLineEdit, TagsLineEdit
 from calibre.utils.date import now, format_date
 from calibre.utils.config import tweaks
 from calibre.utils.formatter import validation_formatter
+from calibre.utils.icu import sort_key
 from calibre.gui2.dialogs.comments_dialog import CommentsDialog
 
 class RatingDelegate(QStyledItemDelegate): # {{{
@@ -173,7 +174,8 @@ class TagsDelegate(QStyledItemDelegate): # {{{
                 editor = TagsLineEdit(parent, self.db.all_tags())
             else:
                 editor = TagsLineEdit(parent,
-                        sorted(list(self.db.all_custom(label=self.db.field_metadata.key_to_label(col)))))
+                        sorted(list(self.db.all_custom(label=self.db.field_metadata.key_to_label(col))),
+                               key=sort_key))
                 return editor
         else:
             editor = EnLineEdit(parent)
@@ -245,7 +247,8 @@ class CcTextDelegate(QStyledItemDelegate): # {{{
             editor.setDecimals(2)
         else:
             editor = EnLineEdit(parent)
-            complete_items = sorted(list(m.db.all_custom(label=m.db.field_metadata.key_to_label(col))))
+            complete_items = sorted(list(m.db.all_custom(label=m.db.field_metadata.key_to_label(col))),
+                                    key=sort_key)
             completer = QCompleter(complete_items, self)
             completer.setCaseSensitivity(Qt.CaseInsensitive)
             completer.setCompletionMode(QCompleter.PopupCompletion)
@@ -254,6 +257,38 @@ class CcTextDelegate(QStyledItemDelegate): # {{{
 
 # }}}
 
+class CcEnumDelegate(QStyledItemDelegate): # {{{
+    '''
+    Delegate for text/int/float data.
+    '''
+
+    def createEditor(self, parent, option, index):
+        m = index.model()
+        col = m.column_map[index.column()]
+        editor = QComboBox(parent)
+        editor.addItem('')
+        for v in m.custom_columns[col]['display']['enum_values']:
+            editor.addItem(v)
+        return editor
+
+    def setModelData(self, editor, model, index):
+        val = unicode(editor.currentText())
+        if not val:
+            val = None
+        model.setData(index, QVariant(val), Qt.EditRole)
+
+    def setEditorData(self, editor, index):
+        m = index.model()
+        val = m.db.data[index.row()][m.custom_columns[m.column_map[index.column()]]['rec_index']]
+        if val is None:
+            val = ''
+        idx = editor.findText(val)
+        if idx < 0:
+            editor.setCurrentIndex(0)
+        else:
+            editor.setCurrentIndex(idx)
+# }}}
+
 class CcCommentsDelegate(QStyledItemDelegate): # {{{
     '''
     Delegate for comments data.

src/calibre/gui2/library/models.py

@@ -18,6 +18,7 @@ from calibre.ebooks.metadata import fmt_sidx, authors_to_string, string_to_autho
 from calibre.ptempfile import PersistentTemporaryFile
 from calibre.utils.config import tweaks, prefs
 from calibre.utils.date import dt_factory, qt_to_dt, isoformat
+from calibre.utils.icu import sort_key
 from calibre.ebooks.metadata.meta import set_metadata as _set_metadata
 from calibre.utils.search_query_parser import SearchQueryParser
 from calibre.library.caches import _match, CONTAINS_MATCH, EQUALS_MATCH, \
@@ -305,9 +306,10 @@ class BooksModel(QAbstractTableModel): # {{{
         cdata = self.cover(idx)
         if cdata:
             data['cover'] = cdata
-        tags = self.db.tags(idx)
+        tags = list(self.db.get_tags(self.db.id(idx)))
         if tags:
-            tags = tags.replace(',', ', ')
+            tags.sort(key=sort_key)
+            tags = ', '.join(tags)
         else:
             tags = _('None')
         data[_('Tags')] = tags
@@ -544,7 +546,7 @@ class BooksModel(QAbstractTableModel): # {{{
         def tags(r, idx=-1):
             tags = self.db.data[r][idx]
             if tags:
-                return QVariant(', '.join(sorted(tags.split(','))))
+                return QVariant(', '.join(sorted(tags.split(','), key=sort_key)))
             return None
 
         def series_type(r, idx=-1, siix=-1):
@@ -595,7 +597,7 @@ class BooksModel(QAbstractTableModel): # {{{
         def text_type(r, mult=False, idx=-1):
             text = self.db.data[r][idx]
             if text and mult:
-                return QVariant(', '.join(sorted(text.split('|'))))
+                return QVariant(', '.join(sorted(text.split('|'),key=sort_key)))
             return QVariant(text)
 
         def number_type(r, idx=-1):
@@ -634,7 +636,7 @@ class BooksModel(QAbstractTableModel): # {{{
         for col in self.custom_columns:
             idx = self.custom_columns[col]['rec_index']
             datatype = self.custom_columns[col]['datatype']
-            if datatype in ('text', 'comments', 'composite'):
+            if datatype in ('text', 'comments', 'composite', 'enumeration'):
                 self.dc[col] = functools.partial(text_type, idx=idx,
                                                  mult=self.custom_columns[col]['is_multiple'])
             elif datatype in ('int', 'float'):
@@ -722,7 +724,11 @@ class BooksModel(QAbstractTableModel): # {{{
         if typ in ('text', 'comments'):
             val = unicode(value.toString()).strip()
             val = val if val else None
-        if typ == 'bool':
+        elif typ == 'enumeration':
+            val = unicode(value.toString()).strip()
+            if not val:
+                val = None
+        elif typ == 'bool':
             val = value.toPyObject()
         elif typ == 'rating':
             val = value.toInt()[0]
@@ -730,7 +736,7 @@ class BooksModel(QAbstractTableModel): # {{{
             val *= 2
         elif typ in ('int', 'float'):
             val = unicode(value.toString()).strip()
-            if val is None or not val:
+            if not val:
                 val = None
         elif typ == 'datetime':
             val = value.toDate()
@@ -1029,8 +1035,8 @@ class DeviceBooksModel(BooksModel): # {{{
             x, y = int(self.db[x].size), int(self.db[y].size)
             return cmp(x, y)
         def tagscmp(x, y):
-            x = ','.join(sorted(getattr(self.db[x], 'device_collections', []))).lower()
+            x = ','.join(sorted(getattr(self.db[x], 'device_collections', []),key=sort_key))
-            y = ','.join(sorted(getattr(self.db[y], 'device_collections', []))).lower()
+            y = ','.join(sorted(getattr(self.db[y], 'device_collections', []),key=sort_key))
             return cmp(x, y)
         def libcmp(x, y):
             x, y = self.db[x].in_library, self.db[y].in_library
@@ -1207,7 +1213,7 @@ class DeviceBooksModel(BooksModel): # {{{
             elif cname == 'collections':
                 tags = self.db[self.map[row]].device_collections
                 if tags:
-                    tags.sort(cmp=lambda x,y: cmp(x.lower(), y.lower()))
+                    tags.sort(key=sort_key)
                     return QVariant(', '.join(tags))
             elif DEBUG and cname == 'inlibrary':
                 return QVariant(self.db[self.map[row]].in_library)

src/calibre/gui2/library/views.py

@@ -14,7 +14,8 @@ from PyQt4.Qt import QTableView, Qt, QAbstractItemView, QMenu, pyqtSignal, \
 
 from calibre.gui2.library.delegates import RatingDelegate, PubDateDelegate, \
     TextDelegate, DateDelegate, TagsDelegate, CcTextDelegate, \
-    CcBoolDelegate, CcCommentsDelegate, CcDateDelegate, CcTemplateDelegate
+    CcBoolDelegate, CcCommentsDelegate, CcDateDelegate, CcTemplateDelegate, \
+    CcEnumDelegate
 from calibre.gui2.library.models import BooksModel, DeviceBooksModel
 from calibre.utils.config import tweaks, prefs
 from calibre.gui2 import error_dialog, gprefs
@@ -76,6 +77,7 @@ class BooksView(QTableView): # {{{
         self.publisher_delegate = TextDelegate(self)
         self.text_delegate = TextDelegate(self)
         self.cc_text_delegate = CcTextDelegate(self)
+        self.cc_enum_delegate = CcEnumDelegate(self)
         self.cc_bool_delegate = CcBoolDelegate(self)
         self.cc_comments_delegate = CcCommentsDelegate(self)
         self.cc_template_delegate = CcTemplateDelegate(self)
@@ -427,6 +429,8 @@ class BooksView(QTableView): # {{{
                     self.setItemDelegateForColumn(cm.index(colhead), self.rating_delegate)
                 elif cc['datatype'] == 'composite':
                     self.setItemDelegateForColumn(cm.index(colhead), self.cc_template_delegate)
+                elif cc['datatype'] == 'enumeration':
+                    self.setItemDelegateForColumn(cm.index(colhead), self.cc_enum_delegate)
             else:
                 dattr = colhead+'_delegate'
                 delegate = colhead if hasattr(self, dattr) else 'text'

src/calibre/gui2/lrf_renderer/main.py

@@ -127,7 +127,7 @@ class Main(MainWindow, Ui_MainWindow):
             self.progress_label.setText('Parsing '+ self.file_name)
             self.renderer = RenderWorker(self, stream, self.logger, self.opts)
             QObject.connect(self.renderer, SIGNAL('finished()'), self.parsed, Qt.QueuedConnection)
-            self.search.clear_to_help()
+            self.search.clear()
             self.last_search = None
         else:
             self.stack.setCurrentIndex(0)

src/calibre/gui2/preferences/behavior.py

@@ -19,6 +19,7 @@ from calibre.utils.search_query_parser import saved_searches
 from calibre.ebooks import BOOK_EXTENSIONS
 from calibre.ebooks.oeb.iterator import is_supported
 from calibre.constants import iswindows
+from calibre.utils.icu import sort_key
 
 class ConfigWidget(ConfigWidgetBase, Ui_Form):
 
@@ -45,8 +46,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         choices = [(x.upper(), x) for x in output_formats]
         r('output_format', prefs, choices=choices)
 
-        restrictions = sorted(saved_searches().names(),
+        restrictions = sorted(saved_searches().names(), key=sort_key)
-                              cmp=lambda x,y: cmp(x.lower(), y.lower()))
         choices = [('', '')] + [(x, x) for x in restrictions]
         r('gui_restriction', db.prefs, choices=choices)
         r('new_book_tags', prefs, setting=CommaSeparatedList)

src/calibre/gui2/preferences/create_custom_column.py

@@ -27,18 +27,20 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn):
                     3:{'datatype':'series',
                         'text':_('Text column for keeping series-like information'),
                         'is_multiple':False},
-                    4:{'datatype':'datetime',
+                    4:{'datatype':'enumeration',
+                        'text':_('Text, but with a fixed set of permitted values'), 'is_multiple':False},
+                    5:{'datatype':'datetime',
                         'text':_('Date'), 'is_multiple':False},
-                    5:{'datatype':'float',
+                    6:{'datatype':'float',
                         'text':_('Floating point numbers'), 'is_multiple':False},
-                    6:{'datatype':'int',
+                    7:{'datatype':'int',
                         'text':_('Integers'), 'is_multiple':False},
-                    7:{'datatype':'rating',
+                    8:{'datatype':'rating',
                         'text':_('Ratings, shown with stars'),
                         'is_multiple':False},
-                    8:{'datatype':'bool',
+                    9:{'datatype':'bool',
                         'text':_('Yes/No'), 'is_multiple':False},
-                    9:{'datatype':'composite',
+                    10:{'datatype':'composite',
                         'text':_('Column built from other columns'), 'is_multiple':False},
                 }
 
@@ -59,6 +61,7 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn):
         self.editing_col = editing
         self.standard_colheads = standard_colheads
         self.standard_colnames = standard_colnames
+        self.column_type_box.setMaxVisibleItems(len(self.column_types))
         for t in self.column_types:
             self.column_type_box.addItem(self.column_types[t]['text'])
         self.column_type_box.currentIndexChanged.connect(self.datatype_changed)
@@ -91,6 +94,8 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn):
                 self.date_format_box.setText(c['display'].get('date_format', ''))
         elif ct == 'composite':
             self.composite_box.setText(c['display'].get('composite_template', ''))
+        elif ct == 'enumeration':
+            self.enum_box.setText(','.join(c['display'].get('enum_values', [])))
         self.datatype_changed()
         self.exec_()
 
@@ -103,7 +108,8 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn):
             getattr(self, 'date_format_'+x).setVisible(col_type == 'datetime')
         for x in ('box', 'default_label', 'label'):
             getattr(self, 'composite_'+x).setVisible(col_type == 'composite')
-
+        for x in ('box', 'default_label', 'label'):
+            getattr(self, 'enum_'+x).setVisible(col_type == 'enumeration')
 
     def accept(self):
         col = unicode(self.column_name_box.text())
@@ -145,17 +151,31 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn):
             return self.simple_error('', _('The heading %s is already used')%col_heading)
 
         display_dict = {}
+
         if col_type == 'datetime':
             if self.date_format_box.text():
                 display_dict = {'date_format':unicode(self.date_format_box.text())}
             else:
                 display_dict = {'date_format': None}
-
+        elif col_type == 'composite':
-        if col_type == 'composite':
             if not self.composite_box.text():
                 return self.simple_error('', _('You must enter a template for'
                     ' composite columns'))
             display_dict = {'composite_template':unicode(self.composite_box.text())}
+        elif col_type == 'enumeration':
+            if not self.enum_box.text():
+                return self.simple_error('', _('You must enter at least one'
+                    ' value for enumeration columns'))
+            l = [v.strip() for v in unicode(self.enum_box.text()).split(',')]
+            for v in l:
+                if not v:
+                    return self.simple_error('', _('You cannot provide the empty '
+                    'value, as it is included by default'))
+            for i in range(0, len(l)-1):
+                if l[i] in l[i+1:]:
+                    return self.simple_error('', _('The value "{0}" is in the '
+                    'list more than once').format(l[i]))
+            display_dict = {'enum_values': l}
 
         db = self.parent.gui.library_view.model().db
         key = db.field_metadata.custom_field_prefix+col

src/calibre/gui2/preferences/create_custom_column.ui

@@ -10,7 +10,7 @@
     <x>0</x>
     <y>0</y>
     <width>528</width>
-    <height>199</height>
+    <height>212</height>
    </rect>
   </property>
   <property name="sizePolicy">
@@ -24,7 +24,7 @@
   </property>
   <layout class="QVBoxLayout" name="verticalLayout">
    <item>
-    <layout class="QGridLayout" name="gridLayout_2" rowstretch="0,0,0,0">
+    <layout class="QGridLayout" name="gridLayout_2" rowstretch="0,0,0,0,0,0,0,0,0,0,0,0">
      <property name="sizeConstraint">
       <enum>QLayout::SetDefaultConstraint</enum>
      </property>
@@ -56,7 +56,7 @@
          </property>
         </widget>
        </item>
-       <item row="0" column="1">
+       <item row="0" column="2">
         <widget class="QLineEdit" name="column_name_box">
          <property name="minimumSize">
           <size>
@@ -69,7 +69,7 @@
          </property>
         </widget>
        </item>
-       <item row="1" column="1">
+       <item row="1" column="2">
         <widget class="QLineEdit" name="column_heading_box">
          <property name="toolTip">
           <string>Column heading in the library view and category name in the tag browser</string>
@@ -86,7 +86,7 @@
          </property>
         </widget>
        </item>
-       <item row="2" column="1">
+       <item row="2" column="2">
         <widget class="QComboBox" name="column_type_box">
          <property name="sizePolicy">
           <sizepolicy hsizetype="Maximum" vsizetype="Fixed">
@@ -105,7 +105,7 @@
          </property>
         </widget>
        </item>
-       <item row="4" column="1">
+       <item row="4" column="2">
         <layout class="QHBoxLayout" name="horizontalLayout_3">
          <item>
           <widget class="QLineEdit" name="date_format_box">
@@ -147,18 +147,18 @@
          </property>
         </widget>
        </item>
-       <item row="5" column="1">
+       <item row="5" column="2">
         <layout class="QHBoxLayout" name="horizontalLayout_4">
          <item>
           <widget class="QLineEdit" name="composite_box">
            <property name="sizePolicy">
-            <sizepolicy hsizetype="Maximum" vsizetype="Fixed">
+            <sizepolicy hsizetype="Expanding" vsizetype="Fixed">
              <horstretch>0</horstretch>
              <verstretch>0</verstretch>
             </sizepolicy>
            </property>
            <property name="toolTip">
-            <string>&lt;p&gt;Field template. Uses the same syntax as save templates.</string>
+            <string>Field template. Uses the same syntax as save templates.</string>
            </property>
           </widget>
          </item>
@@ -184,7 +184,7 @@
          </property>
         </widget>
        </item>
-       <item row="10" column="0" colspan="3">
+       <item row="11" column="0" colspan="4">
         <spacer name="verticalSpacer_2">
          <property name="orientation">
           <enum>Qt::Vertical</enum>
@@ -197,6 +197,45 @@
          </property>
         </spacer>
        </item>
+       <item row="6" column="0">
+        <widget class="QLabel" name="enum_label">
+         <property name="text">
+          <string>Values</string>
+         </property>
+         <property name="buddy">
+          <cstring>enum_box</cstring>
+         </property>
+        </widget>
+       </item>
+       <item row="6" column="2">
+        <layout class="QHBoxLayout" name="horizontalLayout_2">
+         <item>
+          <widget class="QLineEdit" name="enum_box">
+           <property name="sizePolicy">
+            <sizepolicy hsizetype="Expanding" vsizetype="Fixed">
+             <horstretch>0</horstretch>
+             <verstretch>0</verstretch>
+            </sizepolicy>
+           </property>
+           <property name="toolTip">
+            <string>A comma-separated list of permitted values. The empty value is always
+included, and is the default. For example, the list 'one,two,three' has
+four values, the first of them being the empty value.</string>
+           </property>
+          </widget>
+         </item>
+         <item>
+          <widget class="QLabel" name="enum_default_label">
+           <property name="toolTip">
+            <string>The empty string is always the first value</string>
+           </property>
+           <property name="text">
+            <string>Default: (nothing)</string>
+           </property>
+          </widget>
+         </item>
+        </layout>
+       </item>
       </layout>
      </item>
      <item row="11" column="0">

src/calibre/gui2/search_box.py

@@ -8,82 +8,78 @@ __docformat__ = 'restructuredtext en'
 
 import re
 
-from PyQt4.Qt import QComboBox, Qt, QLineEdit, QStringList, pyqtSlot, \
+from PyQt4.Qt import QComboBox, Qt, QLineEdit, QStringList, pyqtSlot, QDialog, \
-                     pyqtSignal, SIGNAL, QObject, QDialog, QCompleter, \
+                     pyqtSignal, QCompleter, QAction, QKeySequence, QTimer, \
-                     QAction, QKeySequence, QTimer
+                     QString
 
 from calibre.gui2 import config
 from calibre.gui2.dialogs.confirm_delete import confirm
 from calibre.gui2.dialogs.saved_search_editor import SavedSearchEditor
 from calibre.gui2.dialogs.search import SearchDialog
 from calibre.utils.search_query_parser import saved_searches
+from calibre.utils.icu import sort_key
 
-class SearchLineEdit(QLineEdit):
+class SearchLineEdit(QLineEdit): # {{{
     key_pressed = pyqtSignal(object)
-    mouse_released = pyqtSignal(object)
-    focus_out = pyqtSignal(object)
 
     def keyPressEvent(self, event):
         self.key_pressed.emit(event)
         QLineEdit.keyPressEvent(self, event)
 
-    def mouseReleaseEvent(self, event):
-        self.mouse_released.emit(event)
-        QLineEdit.mouseReleaseEvent(self, event)
-
-    def focusOutEvent(self, event):
-        self.focus_out.emit(event)
-        QLineEdit.focusOutEvent(self, event)
-
     def dropEvent(self, ev):
-        if self.parent().help_state:
+        self.parent().normalize_state()
-            self.parent().normalize_state()
         return QLineEdit.dropEvent(self, ev)
 
     def contextMenuEvent(self, ev):
-        if self.parent().help_state:
+        self.parent().normalize_state()
-            self.parent().normalize_state()
         return QLineEdit.contextMenuEvent(self, ev)
 
     @pyqtSlot()
     def paste(self, *args):
-        if self.parent().help_state:
+        self.parent().normalize_state()
-            self.parent().normalize_state()
         return QLineEdit.paste(self)
+# }}}
 
-class SearchBox2(QComboBox):
+class SearchBox2(QComboBox): # {{{
 
     '''
     To use this class:
 
         * Call initialize()
         * Connect to the search() and cleared() signals from this widget.
-        * Connect to the cleared() signal to know when the box content changes
+        * Connect to the changed() signal to know when the box content changes
+        * Connect to focus_to_library() signal to be told to manually change focus
         * Call search_done() after every search is complete
-        * Use clear() to clear back to the help message
+        * Call set_search_string() to perform a search programmatically
+        * You can use the current_text property to get the current search text
+          Be aware that if you are using it in a slot connected to the
+          changed() signal, if the connection is not queued it will not be
+          accurate.
     '''
 
     INTERVAL = 1500 #: Time to wait before emitting search signal
     MAX_COUNT = 25
 
-    search = pyqtSignal(object)
+    search  = pyqtSignal(object)
+    cleared = pyqtSignal()
+    changed = pyqtSignal()
+    focus_to_library = pyqtSignal()
 
     def __init__(self, parent=None):
         QComboBox.__init__(self, parent)
         self.normal_background = 'rgb(255, 255, 255, 0%)'
         self.line_edit = SearchLineEdit(self)
         self.setLineEdit(self.line_edit)
+
         c = self.line_edit.completer()
         c.setCompletionMode(c.PopupCompletion)
-        self.line_edit.key_pressed.connect(self.key_pressed,
+        c.highlighted[QString].connect(self.completer_used)
-                type=Qt.DirectConnection)
+        c.activated[QString].connect(self.history_selected)
-        self.line_edit.mouse_released.connect(self.mouse_released,
+
-                type=Qt.DirectConnection)
+        self.line_edit.key_pressed.connect(self.key_pressed, type=Qt.DirectConnection)
         self.activated.connect(self.history_selected)
         self.setEditable(True)
-        self.help_state = False
         self.as_you_type = True
-        self.prev_search = ''
         self.timer = QTimer()
         self.timer.setSingleShot(True)
         self.timer.timeout.connect(self.timer_event, type=Qt.QueuedConnection)
@@ -97,136 +93,130 @@ class SearchBox2(QComboBox):
     def initialize(self, opt_name, colorize=False, help_text=_('Search')):
         self.as_you_type = config['search_as_you_type']
         self.opt_name = opt_name
-        self.addItems(QStringList(list(set(config[opt_name]))))
+        items = []
-        self.help_text = help_text
+        for item in config[opt_name]:
+            if item not in items:
+                items.append(item)
+        self.addItems(QStringList(items))
+        try:
+            self.line_edit.setPlaceholderText(help_text)
+        except:
+            # Using Qt < 4.7
+            pass
         self.colorize = colorize
-        self.clear_to_help()
+        self.clear()
 
     def normalize_state(self):
         self.setToolTip(self.tool_tip_text)
-        if self.help_state:
-            self.setEditText('')
-            self.line_edit.setStyleSheet(
-                'QLineEdit { color: black; background-color: %s; }' %
-                self.normal_background)
-            self.help_state = False
-        else:
-            self.line_edit.setStyleSheet(
-                'QLineEdit { color: black; background-color: %s; }' %
-                    self.normal_background)
-
-    def clear_to_help(self):
-        self.setToolTip(self.tool_tip_text)
-        if self.help_state:
-            return
-        self.help_state = True
-        self.search.emit('')
-        self._in_a_search = False
-        self.setEditText(self.help_text)
-        self.line_edit.home(False)
         self.line_edit.setStyleSheet(
-                'QLineEdit { color: gray; background-color: %s; }' %
+            'QLineEdit{color:black;background-color:%s;}' % self.normal_background)
-                self.normal_background)
-        self.emit(SIGNAL('cleared()'))
 
     def text(self):
         return self.currentText()
 
-    def clear(self):
+    def clear(self, emit_search=True):
-        self.clear_to_help()
+        self.normalize_state()
+        self.setEditText('')
+        if emit_search:
+            self.search.emit('')
+        self._in_a_search = False
+        self.cleared.emit()
+
+    def clear_clicked(self, *args):
+        self.clear()
 
     def search_done(self, ok):
         if isinstance(ok, basestring):
             self.setToolTip(ok)
             ok = False
         if not unicode(self.currentText()).strip():
-            return self.clear_to_help()
+            self.clear(emit_search=False)
+            return
         self._in_a_search = ok
         col = 'rgba(0,255,0,20%)' if ok else 'rgb(255,0,0,20%)'
         if not self.colorize:
             col = self.normal_background
-        self.line_edit.setStyleSheet('QLineEdit { color: black; background-color: %s; }' % col)
+        self.line_edit.setStyleSheet('QLineEdit{color:black;background-color:%s;}' % col)
 
+    # Comes from the lineEdit control
     def key_pressed(self, event):
         k = event.key()
         if k in (Qt.Key_Left, Qt.Key_Right, Qt.Key_Up, Qt.Key_Down,
-                Qt.Key_Home, Qt.Key_End, Qt.Key_PageUp, Qt.Key_PageDown):
+                Qt.Key_Home, Qt.Key_End, Qt.Key_PageUp, Qt.Key_PageDown,
+                Qt.Key_unknown):
             return
         self.normalize_state()
         if self._in_a_search:
-            self.emit(SIGNAL('changed()'))
+            self.changed.emit()
             self._in_a_search = False
         if event.key() in (Qt.Key_Return, Qt.Key_Enter):
             self.do_search()
-        if self.as_you_type:
+            self.focus_to_library.emit()
+        elif self.as_you_type and unicode(event.text()):
             self.timer.start(1500)
 
-    def mouse_released(self, event):
+    # Comes from the combobox itself
+    def keyPressEvent(self, event):
+        k = event.key()
+        if k not in (Qt.Key_Up, Qt.Key_Down):
+            QComboBox.keyPressEvent(self, event)
+        else:
+            self.blockSignals(True)
+            self.normalize_state()
+            QComboBox.keyPressEvent(self, event)
+            self.blockSignals(False)
+
+    def completer_used(self, text):
+        self.timer.stop()
         self.normalize_state()
-        # Dont trigger a search since it make
-        # re-positioning the cursor using the mouse
-        # impossible
-        #if self.as_you_type:
-        #    self.timer.start(1500)
 
     def timer_event(self):
         self.do_search()
 
     def history_selected(self, text):
-        self.emit(SIGNAL('changed()'))
+        self.changed.emit()
         self.do_search()
 
-    @property
+    def _do_search(self, store_in_history=True):
-    def smart_text(self):
         text = unicode(self.currentText()).strip()
-        if not text or text == self.help_text:
+        if not text:
-            return ''
-        return text
-
-    def do_search(self, *args):
-        text = unicode(self.currentText()).strip()
-        if not text or text == self.help_text:
             return self.clear()
-        self.help_state = False
-        self.prev_search = text
         self.search.emit(text)
 
-        idx = self.findText(text, Qt.MatchFixedString)
+        if store_in_history:
-        self.block_signals(True)
+            idx = self.findText(text, Qt.MatchFixedString)
-        if idx < 0:
+            self.block_signals(True)
-            self.insertItem(0, text)
+            if idx < 0:
-        else:
+                self.insertItem(0, text)
-            t = self.itemText(idx)
+            else:
-            self.removeItem(idx)
+                t = self.itemText(idx)
-            self.insertItem(0, t)
+                self.removeItem(idx)
+                self.insertItem(0, t)
             self.setCurrentIndex(0)
-        self.block_signals(False)
+            self.block_signals(False)
-        config[self.opt_name] = [unicode(self.itemText(i)) for i in
+            history = [unicode(self.itemText(i)) for i in
-                range(self.count())]
+                    range(self.count())]
+            config[self.opt_name] = history
+
+    def do_search(self, *args):
+        self._do_search()
 
     def block_signals(self, yes):
         self.blockSignals(yes)
         self.line_edit.blockSignals(yes)
 
-    def search_from_tokens(self, tokens, all):
+    def set_search_string(self, txt, store_in_history=False, emit_changed=True):
-        ans = u' '.join([u'%s:%s'%x for x in tokens])
+        self.setFocus(Qt.OtherFocusReason)
-        if not all:
-            ans = '[' + ans + ']'
-        self.set_search_string(ans)
-
-    def search_from_tags(self, tags, all):
-        joiner = ' and ' if all else ' or '
-        self.set_search_string(joiner.join(tags))
-
-    def set_search_string(self, txt):
         if not txt:
-            self.clear_to_help()
+            self.clear()
-            return
+        else:
-        self.normalize_state()
+            self.normalize_state()
-        self.setEditText(txt)
+            self.setEditText(txt)
-        self.search.emit(txt)
+            self.line_edit.end(False)
-        self.line_edit.end(False)
+            if emit_changed:
-        self.initial_state = False
+                self.changed.emit()
+            self._do_search(store_in_history=store_in_history)
+        self.focus_to_library.emit()
 
     def search_as_you_type(self, enabled):
         self.as_you_type = enabled
@@ -234,7 +224,13 @@ class SearchBox2(QComboBox):
     def in_a_search(self):
         return self._in_a_search
 
-class SavedSearchBox(QComboBox):
+    @property
+    def current_text(self):
+        return unicode(self.lineEdit().text())
+
+    # }}}
+
+class SavedSearchBox(QComboBox): # {{{
 
     '''
     To use this class:
@@ -243,25 +239,23 @@ class SavedSearchBox(QComboBox):
           if you care about changes to the list of saved searches.
     '''
 
+    changed = pyqtSignal()
+
     def __init__(self, parent=None):
         QComboBox.__init__(self, parent)
         self.normal_background = 'rgb(255, 255, 255, 0%)'
 
         self.line_edit = SearchLineEdit(self)
         self.setLineEdit(self.line_edit)
-        self.line_edit.key_pressed.connect(self.key_pressed,
+        self.line_edit.key_pressed.connect(self.key_pressed, type=Qt.DirectConnection)
-                type=Qt.DirectConnection)
-        self.line_edit.mouse_released.connect(self.mouse_released,
-                type=Qt.DirectConnection)
-        self.line_edit.focus_out.connect(self.focus_out,
-                type=Qt.DirectConnection)
         self.activated[str].connect(self.saved_search_selected)
 
-        completer = QCompleter(self) # turn off auto-completion
+        # Turn off auto-completion so that it doesn't interfere with typing
+        # names of new searches.
+        completer = QCompleter(self)
         self.setCompleter(completer)
+
         self.setEditable(True)
-        self.help_state = True
-        self.prev_search = ''
         self.setInsertPolicy(self.NoInsert)
         self.setSizeAdjustPolicy(self.AdjustToMinimumContentsLengthWithIcon)
         self.setMinimumContentsLength(10)
@@ -269,50 +263,42 @@ class SavedSearchBox(QComboBox):
 
     def initialize(self, _search_box, colorize=False, help_text=_('Search')):
         self.search_box = _search_box
-        self.help_text = help_text
+        try:
+           self.line_edit.setPlaceholderText(help_text)
+        except:
+            # Using Qt < 4.7
+            pass
         self.colorize = colorize
-        self.clear_to_help()
+        self.clear()
 
     def normalize_state(self):
-        self.setEditText('')
+        # need this because line_edit will call it in some cases such as paste
-        self.line_edit.setStyleSheet(
+        pass
-            'QLineEdit { color: black; background-color: %s; }' %
-            self.normal_background)
-        self.help_state = False
 
-    def clear_to_help(self):
+    def clear(self):
-        self.setToolTip(self.tool_tip_text)
+        QComboBox.clear(self)
         self.initialize_saved_search_names()
-        self.setEditText(self.help_text)
+        self.setEditText('')
         self.line_edit.home(False)
-        self.help_state = True
-        self.line_edit.setStyleSheet(
-                'QLineEdit { color: gray; background-color: %s; }' %
-                self.normal_background)
-
-    def focus_out(self, event):
-        if self.currentText() == '':
-            self.clear_to_help()
 
     def key_pressed(self, event):
-        if self.help_state:
+        if event.key() in (Qt.Key_Return, Qt.Key_Enter):
-            self.normalize_state()
+            self.saved_search_selected(self.currentText())
-
-    def mouse_released(self, event):
-        if self.help_state:
-            self.normalize_state()
 
     def saved_search_selected(self, qname):
         qname = unicode(qname)
         if qname is None or not qname.strip():
+            self.search_box.clear()
             return
-        self.normalize_state()
+        if not saved_searches().lookup(qname):
-        self.search_box.set_search_string(u'search:"%s"' % qname)
+            self.search_box.clear()
+            self.setEditText(qname)
+            return
+        self.search_box.set_search_string(u'search:"%s"' % qname, emit_changed=False)
         self.setEditText(qname)
         self.setToolTip(saved_searches().lookup(qname))
 
     def initialize_saved_search_names(self):
-        self.clear()
         qnames = saved_searches().names()
         self.addItems(qnames)
         self.setCurrentIndex(-1)
@@ -330,25 +316,24 @@ class SavedSearchBox(QComboBox):
         if ss is None:
             return
         saved_searches().delete(unicode(self.currentText()))
-        self.clear_to_help()
+        self.clear()
-        self.search_box.clear_to_help()
+        self.search_box.clear()
-        self.emit(SIGNAL('changed()'))
+        self.changed.emit()
 
     # SIGNALed from the main UI
     def save_search_button_clicked(self):
         name = unicode(self.currentText())
-        if self.help_state or not name.strip():
+        if not name.strip():
             name = unicode(self.search_box.text()).replace('"', '')
         saved_searches().delete(name)
         saved_searches().add(name, unicode(self.search_box.text()))
         # now go through an initialization cycle to ensure that the combobox has
         # the new search in it, that it is selected, and that the search box
         # references the new search instead of the text in the search.
-        self.clear_to_help()
+        self.clear()
-        self.normalize_state()
         self.setCurrentIndex(self.findText(name))
         self.saved_search_selected (name)
-        self.emit(SIGNAL('changed()'))
+        self.changed.emit()
 
     # SIGNALed from the main UI
     def copy_search_button_clicked (self):
@@ -357,16 +342,20 @@ class SavedSearchBox(QComboBox):
             return
         self.search_box.set_search_string(saved_searches().lookup(unicode(self.currentText())))
 
-class SearchBoxMixin(object):
+    # }}}
+
+class SearchBoxMixin(object): # {{{
 
     def __init__(self):
         self.search.initialize('main_search_history', colorize=True,
                 help_text=_('Search (For Advanced Search click the button to the left)'))
-        self.connect(self.search, SIGNAL('cleared()'), self.search_box_cleared)
+        self.search.cleared.connect(self.search_box_cleared)
-        self.connect(self.search, SIGNAL('changed()'), self.search_box_changed)
+        # Queued so that search.current_text will be correct
-        self.connect(self.clear_button, SIGNAL('clicked()'), self.search.clear)
+        self.search.changed.connect(self.search_box_changed,
-        QObject.connect(self.advanced_search_button, SIGNAL('clicked(bool)'),
+                type=Qt.QueuedConnection)
-                        self.do_advanced_search)
+        self.search.focus_to_library.connect(self.focus_to_library)
+        self.clear_button.clicked.connect(self.search.clear_clicked)
+        self.advanced_search_button.clicked[bool].connect(self.do_advanced_search)
 
         self.search.clear()
         self.search.setMaximumWidth(self.width()-150)
@@ -374,42 +363,54 @@ class SearchBoxMixin(object):
         shortcuts = QKeySequence.keyBindings(QKeySequence.Find)
         shortcuts = list(shortcuts) + [QKeySequence('/'), QKeySequence('Alt+S')]
         self.action_focus_search.setShortcuts(shortcuts)
-        self.action_focus_search.triggered.connect(lambda x:
+        self.action_focus_search.triggered.connect(self.focus_search_box)
-                self.search.setFocus(Qt.OtherFocusReason))
         self.addAction(self.action_focus_search)
         self.search.setStatusTip(re.sub(r'<\w+>', ' ',
             unicode(self.search.toolTip())))
         self.advanced_search_button.setStatusTip(self.advanced_search_button.toolTip())
         self.clear_button.setStatusTip(self.clear_button.toolTip())
 
+    def focus_search_box(self, *args):
+        self.search.setFocus(Qt.OtherFocusReason)
+        self.search.lineEdit().selectAll()
+
     def search_box_cleared(self):
         self.tags_view.clear()
-        self.saved_search.clear_to_help()
+        self.saved_search.clear()
         self.set_number_of_books_shown()
 
     def search_box_changed(self):
-        self.saved_search.clear_to_help()
+        self.saved_search.clear()
-        self.tags_view.clear()
+        self.tags_view.conditional_clear(self.search.current_text)
 
     def do_advanced_search(self, *args):
         d = SearchDialog(self, self.library_view.model().db)
         if d.exec_() == QDialog.Accepted:
-            self.search.set_search_string(d.search_string())
+            self.search.set_search_string(d.search_string(), store_in_history=True)
 
-class SavedSearchBoxMixin(object):
+    def do_search_button(self):
+        self.search.do_search()
+        self.focus_to_library()
+
+    def focus_to_library(self):
+        self.current_view().setFocus(Qt.OtherFocusReason)
+
+    # }}}
+
+class SavedSearchBoxMixin(object): # {{{
 
     def __init__(self):
-        self.connect(self.saved_search, SIGNAL('changed()'), self.saved_searches_changed)
+        self.saved_search.changed.connect(self.saved_searches_changed)
+        self.clear_button.clicked.connect(self.saved_search.clear)
+        self.save_search_button.clicked.connect(
+                                self.saved_search.save_search_button_clicked)
+        self.delete_search_button.clicked.connect(
+                                self.saved_search.delete_search_button_clicked)
+        self.copy_search_button.clicked.connect(
+                                self.saved_search.copy_search_button_clicked)
         self.saved_searches_changed()
-        self.connect(self.clear_button, SIGNAL('clicked()'), self.saved_search.clear_to_help)
         self.saved_search.initialize(self.search, colorize=True,
                 help_text=_('Saved Searches'))
-        self.connect(self.save_search_button, SIGNAL('clicked()'),
-                self.saved_search.save_search_button_clicked)
-        self.connect(self.delete_search_button, SIGNAL('clicked()'),
-                self.saved_search.delete_search_button_clicked)
-        self.connect(self.copy_search_button, SIGNAL('clicked()'),
-                self.saved_search.copy_search_button_clicked)
         self.saved_search.setToolTip(
             _('Choose saved search or enter name for new saved search'))
         self.saved_search.setStatusTip(self.saved_search.toolTip())
@@ -418,9 +419,10 @@ class SavedSearchBoxMixin(object):
             b.setStatusTip(b.toolTip())
 
     def saved_searches_changed(self):
-        p = sorted(saved_searches().names(), cmp=lambda x,y: cmp(x.lower(), y.lower()))
+        p = sorted(saved_searches().names(), key=sort_key)
         t = unicode(self.search_restriction.currentText())
-        self.search_restriction.clear() # rebuild the restrictions combobox using current saved searches
+        # rebuild the restrictions combobox using current saved searches
+        self.search_restriction.clear()
         self.search_restriction.addItem('')
         self.tags_view.recount()
         for s in p:
@@ -433,6 +435,7 @@ class SavedSearchBoxMixin(object):
         d.exec_()
         if d.result() == d.Accepted:
             self.saved_searches_changed()
-            self.saved_search.clear_to_help()
+            self.saved_search.clear()
 
+    # }}}
 

src/calibre/gui2/search_restriction_mixin.py

@@ -4,6 +4,8 @@ Created on 10 Jun 2010
 @author: charles
 '''
 
+from PyQt4.Qt import Qt
+
 class SearchRestrictionMixin(object):
 
     def __init__(self):
@@ -49,10 +51,11 @@ class SearchRestrictionMixin(object):
             restriction = ''
         self.restriction_count_of_books_in_view = \
                     self.library_view.model().set_search_restriction(restriction)
-        self.search.clear_to_help()
+        self.search.clear()
-        self.saved_search.clear_to_help()
+        self.saved_search.clear()
         self.tags_view.set_search_restriction(restriction)
         self.set_number_of_books_shown()
+        self.current_view().setFocus(Qt.OtherFocusReason)
 
     def set_number_of_books_shown(self):
         if self.current_view() == self.library_view and self.restriction_in_effect:

src/calibre/gui2/shortcuts.py

@@ -14,6 +14,7 @@ from PyQt4.Qt import QAbstractListModel, Qt, QKeySequence, QListView, \
 
 from calibre.gui2 import NONE, error_dialog
 from calibre.utils.config import XMLConfig
+from calibre.utils.icu import sort_key
 from calibre.gui2.shortcuts_ui import Ui_Frame
 
 DEFAULTS = Qt.UserRole
@@ -175,8 +176,7 @@ class Shortcuts(QAbstractListModel):
         for k, v in shortcuts.items():
             self.keys[k] = v[0]
         self.order = list(shortcuts)
-        self.order.sort(cmp=lambda x,y : cmp(self.descriptions[x],
+        self.order.sort(key=lambda x : sort_key(self.descriptions[x]))
-            self.descriptions[y]))
         self.sequences = {}
         for k, v in self.keys.items():
             self.sequences[k] = [QKeySequence(x) for x in v]

src/calibre/gui2/tag_view.py

@@ -18,6 +18,7 @@ from PyQt4.Qt import Qt, QTreeView, QApplication, pyqtSignal, \
 from calibre.ebooks.metadata import title_sort
 from calibre.gui2 import config, NONE
 from calibre.library.field_metadata import TagsIcons, category_icon_map
+from calibre.utils.icu import sort_key
 from calibre.utils.search_query_parser import saved_searches
 from calibre.gui2 import error_dialog
 from calibre.gui2.dialogs.confirm_delete import confirm
@@ -60,7 +61,7 @@ class TagDelegate(QItemDelegate): # {{{
 class TagsView(QTreeView): # {{{
 
     refresh_required    = pyqtSignal()
-    tags_marked         = pyqtSignal(object, object)
+    tags_marked         = pyqtSignal(object)
     user_category_edit  = pyqtSignal(object)
     tag_list_edit       = pyqtSignal(object, object)
     saved_search_edit   = pyqtSignal(object)
@@ -106,10 +107,13 @@ class TagsView(QTreeView): # {{{
             self.refresh_required.connect(self.recount, type=Qt.QueuedConnection)
             self.sort_by.currentIndexChanged.connect(self.sort_changed)
             self.made_connections = True
+        self.refresh_signal_processed = True
         db.add_listener(self.database_changed)
 
     def database_changed(self, event, ids):
-        self.refresh_required.emit()
+        if self.refresh_signal_processed:
+            self.refresh_signal_processed = False
+            self.refresh_required.emit()
 
     @property
     def match_all(self):
@@ -135,11 +139,21 @@ class TagsView(QTreeView): # {{{
         # swallow these to avoid toggling and editing at the same time
         pass
 
+    @property
+    def search_string(self):
+        tokens = self._model.tokens()
+        joiner = ' and ' if self.match_all else ' or '
+        return joiner.join(tokens)
+
     def toggle(self, index):
         modifiers = int(QApplication.keyboardModifiers())
         exclusive = modifiers not in (Qt.CTRL, Qt.SHIFT)
         if self._model.toggle(index, exclusive):
-            self.tags_marked.emit(self._model.tokens(), self.match_all)
+            self.tags_marked.emit(self.search_string)
+
+    def conditional_clear(self, search_string):
+        if search_string != self.search_string:
+            self.clear()
 
     def context_menu_handler(self, action=None, category=None,
                              key=None, index=None):
@@ -212,7 +226,7 @@ class TagsView(QTreeView): # {{{
                     partial(self.context_menu_handler, action='hide', category=category))
                 if self.hidden_categories:
                     m = self.context_menu.addMenu(_('Show category'))
-                    for col in sorted(self.hidden_categories, cmp=lambda x,y: cmp(x.lower(), y.lower())):
+                    for col in sorted(self.hidden_categories, key=sort_key):
                         m.addAction(col,
                             partial(self.context_menu_handler, action='show', category=col))
 
@@ -285,6 +299,7 @@ class TagsView(QTreeView): # {{{
         return self.isExpanded(idx)
 
     def recount(self, *args):
+        self.refresh_signal_processed = True
         ci = self.currentIndex()
         if not ci.isValid():
             ci = self.indexAt(QPoint(10, 10))
@@ -585,7 +600,8 @@ class TagsModel(QAbstractItemModel): # {{{
         # Reconstruct the user categories, putting them into metadata
         self.db.field_metadata.remove_dynamic_categories()
         tb_cats = self.db.field_metadata
-        for user_cat in sorted(self.db.prefs.get('user_categories', {}).keys()):
+        for user_cat in sorted(self.db.prefs.get('user_categories', {}).keys(),
+                               key=sort_key):
             cat_name = user_cat+':' # add the ':' to avoid name collision
             tb_cats.add_user_category(label=cat_name, name=user_cat)
         if len(saved_searches().names()):
@@ -842,8 +858,7 @@ class TagBrowserMixin(object): # {{{
         self.library_view.model().count_changed_signal.connect(self.tags_view.recount)
         self.tags_view.set_database(self.library_view.model().db,
                 self.tag_match, self.sort_by)
-        self.tags_view.tags_marked.connect(self.search.search_from_tags)
+        self.tags_view.tags_marked.connect(self.search.set_search_string)
-        self.tags_view.tags_marked.connect(self.saved_search.clear_to_help)
         self.tags_view.tag_list_edit.connect(self.do_tags_list_edit)
         self.tags_view.user_category_edit.connect(self.do_user_categories_edit)
         self.tags_view.saved_search_edit.connect(self.do_saved_search_edit)
@@ -865,13 +880,13 @@ class TagBrowserMixin(object): # {{{
         db=self.library_view.model().db
         if category == 'tags':
             result = db.get_tags_with_ids()
-            compare = (lambda x,y:cmp(x.lower(), y.lower()))
+            key = sort_key
         elif category == 'series':
             result = db.get_series_with_ids()
-            compare = (lambda x,y:cmp(title_sort(x).lower(), title_sort(y).lower()))
+            key = lambda x:sort_key(title_sort(x))
         elif category == 'publisher':
             result = db.get_publishers_with_ids()
-            compare = (lambda x,y:cmp(x.lower(), y.lower()))
+            key = sort_key
         else: # should be a custom field
             cc_label = None
             if category in db.field_metadata:
@@ -879,9 +894,9 @@ class TagBrowserMixin(object): # {{{
                 result = db.get_custom_items_with_ids(label=cc_label)
             else:
                 result = []
-            compare = (lambda x,y:cmp(x.lower(), y.lower()))
+            key = sort_key
 
-        d = TagListEditor(self, tag_to_match=tag, data=result, compare=compare)
+        d = TagListEditor(self, tag_to_match=tag, data=result, key=key)
         d.exec_()
         if d.result() == d.Accepted:
             to_rename = d.to_rename # dict of new text to old id
@@ -910,14 +925,14 @@ class TagBrowserMixin(object): # {{{
             self.library_view.model().refresh()
             self.tags_view.set_new_model()
             self.tags_view.recount()
-            self.saved_search.clear_to_help()
+            self.saved_search.clear()
-            self.search.clear_to_help()
+            self.search.clear()
 
     def do_tag_item_renamed(self):
         # Clean up library view and search
         self.library_view.model().refresh()
-        self.saved_search.clear_to_help()
+        self.saved_search.clear()
-        self.search.clear_to_help()
+        self.search.clear()
 
     def do_author_sort_edit(self, parent, id):
         db = self.library_view.model().db
@@ -928,7 +943,9 @@ class TagBrowserMixin(object): # {{{
                 if old_author != new_author:
                     # The id might change if the new author already exists
                     id = db.rename_author(id, new_author)
-                db.set_sort_field_for_author(id, unicode(new_sort))
+                db.set_sort_field_for_author(id, unicode(new_sort),
+                                             commit=False, notify=False)
+            db.commit()
             self.library_view.model().refresh()
             self.tags_view.recount()
 

src/calibre/gui2/tools.py

@@ -245,11 +245,11 @@ def fetch_scheduled_recipe(arg):
 
     return 'gui_convert', args, _('Fetch news from ')+arg['title'], fmt.upper(), [pt]
 
-def generate_catalog(parent, dbspec, ids, device_manager):
+def generate_catalog(parent, dbspec, ids, device_manager, db):
     from calibre.gui2.dialogs.catalog import Catalog
 
     # Build the Catalog dialog in gui2.dialogs.catalog
-    d = Catalog(parent, dbspec, ids)
+    d = Catalog(parent, dbspec, ids, db)
 
     if d.exec_() != d.Accepted:
         return None

src/calibre/gui2/ui.py

@@ -383,8 +383,8 @@ class Main(MainWindow, MainWindowMixin, DeviceMixin, EmailMixin, # {{{
         self.tags_view.set_database(db, self.tag_match, self.sort_by)
         self.library_view.model().set_book_on_device_func(self.book_on_device)
         self.status_bar.clear_message()
-        self.search.clear_to_help()
+        self.search.clear()
-        self.saved_search.clear_to_help()
+        self.saved_search.clear()
         self.book_details.reset_info()
         self.library_view.model().count_changed()
         prefs['library_path'] = self.library_path

src/calibre/gui2/viewer/documentview.py

@@ -614,7 +614,7 @@ class DocumentView(QWebView):
 
     def search(self, text, backwards=False):
         if backwards:
-            return self.findText(text, self.document.FindBackwards)
+            return self.findText(text, self.document.FindBackward)
         return self.findText(text)
 
     def path(self):

src/calibre/gui2/viewer/main.py

@@ -17,7 +17,7 @@ from calibre.gui2.viewer.bookmarkmanager import BookmarkManager
 from calibre.gui2.widgets import ProgressIndicator
 from calibre.gui2.main_window import MainWindow
 from calibre.gui2 import Application, ORG_NAME, APP_UID, choose_files, \
-                         info_dialog, error_dialog, open_url
+                         info_dialog, error_dialog, open_url, available_height
 from calibre.ebooks.oeb.iterator import EbookIterator
 from calibre.ebooks import DRMError
 from calibre.constants import islinux, isfreebsd, isosx
@@ -172,6 +172,7 @@ class EbookViewer(MainWindow, Ui_EbookViewer):
         self.iterator          = None
         self.current_page      = None
         self.pending_search    = None
+        self.pending_search_dir= None
         self.pending_anchor    = None
         self.pending_reference = None
         self.pending_bookmark  = None
@@ -237,9 +238,9 @@ class EbookViewer(MainWindow, Ui_EbookViewer):
         self.connect(self.action_previous_page, SIGNAL('triggered(bool)'),
                      lambda x:self.view.previous_page())
         self.connect(self.action_find_next, SIGNAL('triggered(bool)'),
-                     lambda x:self.find(self.search.smart_text, repeat=True))
+                     lambda x:self.find(unicode(self.search.text()), repeat=True))
         self.connect(self.action_find_previous, SIGNAL('triggered(bool)'),
-                     lambda x:self.find(self.search.smart_text,
+                     lambda x:self.find(unicode(self.search.text()),
                          repeat=True, backwards=True))
 
         self.connect(self.action_full_screen, SIGNAL('triggered(bool)'),
@@ -253,6 +254,7 @@ class EbookViewer(MainWindow, Ui_EbookViewer):
         self.connect(self.vertical_scrollbar, SIGNAL('valueChanged(int)'),
                      lambda x: self.goto_page(x/100.))
         self.search.search.connect(self.find)
+        self.search.focus_to_library.connect(lambda: self.view.setFocus(Qt.OtherFocusReason))
         self.connect(self.toc, SIGNAL('clicked(QModelIndex)'), self.toc_clicked)
         self.connect(self.reference, SIGNAL('goto(PyQt_PyObject)'), self.goto)
 
@@ -434,7 +436,7 @@ class EbookViewer(MainWindow, Ui_EbookViewer):
         if not text:
             self.view.search('')
             return self.search.search_done(False)
-        if self.view.search(text):
+        if self.view.search(text, backwards=backwards):
             self.scrolled(self.view.scroll_fraction)
             return self.search.search_done(True)
         index = self.iterator.search(text, self.current_index,
@@ -448,11 +450,13 @@ class EbookViewer(MainWindow, Ui_EbookViewer):
                     return self.search.search_done(True)
             return self.search.search_done(True)
         self.pending_search = text
+        self.pending_search_dir = 'backwards' if backwards else 'forwards'
         self.load_path(self.iterator.spine[index])
 
-    def do_search(self, text):
+    def do_search(self, text, backwards):
         self.pending_search = None
-        if self.view.search(text):
+        self.pending_search_dir = None
+        if self.view.search(text, backwards=backwards):
             self.scrolled(self.view.scroll_fraction)
 
     def keyPressEvent(self, event):
@@ -498,8 +502,10 @@ class EbookViewer(MainWindow, Ui_EbookViewer):
         self.current_index = index
         self.set_page_number(self.view.scroll_fraction)
         if self.pending_search is not None:
-            self.do_search(self.pending_search)
+            self.do_search(self.pending_search,
+                    self.pending_search_dir=='backwards')
             self.pending_search = None
+            self.pending_search_dir = None
         if self.pending_anchor is not None:
             self.view.scroll_to(self.pending_anchor)
             self.pending_anchor = None
@@ -693,6 +699,9 @@ class EbookViewer(MainWindow, Ui_EbookViewer):
             if ss is not None:
                 self.splitter.restoreState(ss)
             self.show_toc_on_open = dynamic.get('viewer_toc_isvisible', False)
+        av = available_height() - 30
+        if self.height() > av:
+            self.resize(self.width(), av)
 
 def config(defaults=None):
     desc = _('Options to control the ebook viewer')

src/calibre/library/caches.py

@@ -520,7 +520,7 @@ class ResultCache(SearchQueryParser): # {{{
                 if len(self.field_metadata[x]['search_terms']):
                     db_col[x] = self.field_metadata[x]['rec_index']
                     if self.field_metadata[x]['datatype'] not in \
-                                    ['composite', 'text', 'comments', 'series']:
+                            ['composite', 'text', 'comments', 'series', 'enumeration']:
                         exclude_fields.append(db_col[x])
                     col_datatype[db_col[x]] = self.field_metadata[x]['datatype']
                     is_multiple_cols[db_col[x]] = self.field_metadata[x]['is_multiple']
@@ -796,11 +796,13 @@ class SortKey(object):
 class SortKeyGenerator(object):
 
     def __init__(self, fields, field_metadata, data):
+        from calibre.utils.icu import sort_key
         self.field_metadata = field_metadata
         self.orders = [-1 if x[1] else 1 for x in fields]
         self.entries = [(x[0], field_metadata[x[0]]) for x in fields]
         self.library_order = tweaks['title_series_sorting'] == 'library_order'
         self.data = data
+        self.string_sort_key = sort_key
 
     def __call__(self, record):
         values = tuple(self.itervals(self.data[record]))
@@ -821,17 +823,14 @@ class SortKeyGenerator(object):
                 if val is None:
                     val = ('', 1)
                 else:
-                    val = val.lower()
                     if self.library_order:
                         val = title_sort(val)
                     sidx_fm = self.field_metadata[name + '_index']
                     sidx = record[sidx_fm['rec_index']]
-                    val = (val, sidx)
+                    val = (self.string_sort_key(val), sidx)
 
-            elif dt in ('text', 'comments', 'composite'):
+            elif dt in ('text', 'comments', 'composite', 'enumeration'):
-                if val is None:
+                val = self.string_sort_key(val)
-                    val = ''
-                val = val.lower()
 
             elif dt == 'bool':
                 val = {True: 1, False: 2, None: 3}.get(val, 3)

src/calibre/library/catalog.py

@@ -606,12 +606,12 @@ class EPUB_MOBI(CatalogPlugin):
                           help=_("Specifies the output profile.  In some cases, an output profile is required to optimize the catalog for the device.  For example, 'kindle' or 'kindle_dx' creates a structured Table of Contents with Sections and Articles.\n"
                           "Default: '%default'\n"
                           "Applies to: ePub, MOBI output formats")),
-                   Option('--read-tag',
+                   Option('--read-book-marker',
-                          default='+',
+                          default='tag:+',
-                          dest='read_tag',
+                          dest='read_book_marker',
                           action = None,
-                          help=_("Tag indicating book has been read.\n" "Default: '%default'\n"
+                          help=_("field:pattern indicating book has been read.\n" "Default: '%default'\n"
-                          "Applies to: ePub, MOBI output formats")),
+                          "Applies to ePub, MOBI output formats")),
                    Option('--wishlist-tag',
                           default='Wishlist',
                           dest='wishlist_tag',
@@ -898,6 +898,8 @@ class EPUB_MOBI(CatalogPlugin):
             self.__plugin = plugin
             self.__progressInt = 0.0
             self.__progressString = ''
+            f, _, p = opts.read_book_marker.partition(':')
+            self.__read_book_marker = {'field':f, 'pattern':p}
             self.__reporter = report_progress
             self.__stylesheet = stylesheet
             self.__thumbs = None
@@ -936,7 +938,6 @@ class EPUB_MOBI(CatalogPlugin):
             if self.opts.generate_series:
                 self.__totalSteps += 2
 
-
         # Accessors
         if True:
             '''
@@ -1210,7 +1211,7 @@ class EPUB_MOBI(CatalogPlugin):
             def READING_SYMBOL(self):
                 def fget(self):
                     return '<span style="color:black">&#x25b7;</span>' if self.generateForKindle else \
-                           '<span style="color:white">%s</span>' % self.opts.read_tag
+                           '<span style="color:white">+</span>'
                 return property(fget=fget)
             @dynamic_property
             def READ_SYMBOL(self):
@@ -1401,8 +1402,7 @@ class EPUB_MOBI(CatalogPlugin):
                 if record['cover']:
                     this_title['cover'] = re.sub('&amp;', '&', record['cover'])
 
-                # This may be updated in self.processSpecialTags()
+                this_title['read'] = self.discoverReadStatus(record)
-                this_title['read'] = False
 
                 if record['tags']:
                     this_title['tags'] = self.processSpecialTags(record['tags'],
@@ -2675,13 +2675,7 @@ class EPUB_MOBI(CatalogPlugin):
                 pBookTag = Tag(soup, "p")
                 ptc = 0
 
-                #  book with read/reading/unread symbol
+                book['read'] = self.discoverReadStatus(book)
-                for tag in book['tags']:
-                    if tag == self.opts.read_tag:
-                        book['read'] = True
-                        break
-                else:
-                    book['read'] = False
 
                 #  book with read|reading|unread symbol or wishlist item
                 if self.opts.wishlist_tag in book.get('tags', []):
@@ -2689,7 +2683,7 @@ class EPUB_MOBI(CatalogPlugin):
                     pBookTag.insert(ptc,NavigableString(self.MISSING_SYMBOL))
                     ptc += 1
                 else:
-                    if book['read']:
+                    if book.get('read', False):
                         # check mark
                         pBookTag.insert(ptc,NavigableString(self.READ_SYMBOL))
                         pBookTag['class'] = "read_book"
@@ -4027,6 +4021,34 @@ class EPUB_MOBI(CatalogPlugin):
             if not os.path.isdir(images_path):
                 os.makedirs(images_path)
 
+        def discoverReadStatus(self, record):
+            '''
+            Given a field:pattern spec, discover if this book marked as read
+
+            if field == tag, scan tags for pattern
+            if custom field, try regex match for pattern
+            This allows maximum flexibility with fields of type
+             datatype bool: #field_name:True
+             datatype text: #field_name:<string>
+             datatype datetime: #field_name:.*
+
+            '''
+            # Legacy handling of special 'read' tag
+            field = self.__read_book_marker['field']
+            pat = self.__read_book_marker['pattern']
+            if field == 'tag' and pat in record['tags']:
+                return True
+
+            field_contents = self.__db.get_field(record['id'],
+                                        field,
+                                        index_is_id=True)
+            if field_contents:
+                if re.search(pat, unicode(field_contents),
+                        re.IGNORECASE) is not None:
+                    return True
+
+            return False
+
         def filterDbTags(self, tags):
             # Remove the special marker tags from the database's tag list,
             # return sorted list of normalized genre tags
@@ -4519,7 +4541,6 @@ class EPUB_MOBI(CatalogPlugin):
             markerTags = []
             markerTags.extend(self.opts.exclude_tags.split(','))
             markerTags.extend(self.opts.note_tag.split(','))
-            markerTags.extend(self.opts.read_tag.split(','))
             return markerTags
 
         def letter_or_symbol(self,char):
@@ -4629,6 +4650,7 @@ class EPUB_MOBI(CatalogPlugin):
 
             if open_pTag:
                 result.insert(rtc, pTag)
+                rtc += 1
 
             paras = result.findAll('p')
             for p in paras:
@@ -4647,10 +4669,12 @@ class EPUB_MOBI(CatalogPlugin):
                 tag = self.convertHTMLEntities(tag)
                 if tag.startswith(opts.note_tag):
                     this_title['notes'] = tag[len(self.opts.note_tag):]
-                elif tag == opts.read_tag:
-                    this_title['read'] = True
                 elif re.search(opts.exclude_genre, tag):
                     continue
+                elif self.__read_book_marker['field'] == 'tag' and \
+                     tag == self.__read_book_marker['pattern']:
+                    # remove 'read' tag
+                    continue
                 else:
                     tag_list.append(tag)
             return tag_list
@@ -4759,7 +4783,7 @@ class EPUB_MOBI(CatalogPlugin):
         for key in keys:
             if key in ['catalog_title','authorClip','connected_kindle','descriptionClip',
                        'exclude_genre','exclude_tags','note_tag','numbers_as_text',
-                       'output_profile','read_tag',
+                       'output_profile','read_book_marker',
                        'search_text','sort_by','sort_descriptions_by_author','sync',
                         'wishlist_tag']:
                 build_log.append("  %s: %s" % (key, opts_dict[key]))

src/calibre/library/cli.py

@@ -565,8 +565,9 @@ datatype is one of: {0}
                           'applies if datatype is text.'))
     parser.add_option('--display', default='{}',
             help=_('A dictionary of options to customize how '
-                'the data in this column will be interpreted.'))
+                'the data in this column will be interpreted. This is a JSON '
-
+                ' string. For enumeration columns, use '
+                '--display=\'{"enum_values":["val1", "val2"]}\''))
     return parser
 
 
@@ -640,7 +641,7 @@ def catalog_option_parser(args):
     log = Log()
     parser = get_parser(_(
     '''
-    %prog catalog /path/to/destination.(csv|epub|mobi|xml ...) [options]
+    %prog catalog /path/to/destination.(CSV|EPUB|MOBI|XML ...) [options]
 
     Export a catalog in format specified by path/to/destination extension.
     Options control how entries are displayed in the generated catalog ouput.

src/calibre/library/custom_columns.py

@@ -18,7 +18,7 @@ from calibre.utils.date import parse_date
 class CustomColumns(object):
 
     CUSTOM_DATA_TYPES = frozenset(['rating', 'text', 'comments', 'datetime',
-        'int', 'float', 'bool', 'series', 'composite'])
+        'int', 'float', 'bool', 'series', 'composite', 'enumeration'])
 
     def custom_table_names(self, num):
         return 'custom_column_%d'%num, 'books_custom_column_%d_link'%num
@@ -136,6 +136,12 @@ class CustomColumns(object):
                 x = bool(int(x))
             return x
 
+        def adapt_enum(x, d):
+            v = adapt_text(x, d)
+            if not v:
+                v = None
+            return v
+
         self.custom_data_adapters = {
                 'float': lambda x,d : x if x is None else float(x),
                 'int':   lambda x,d : x if x is None else int(x),
@@ -144,7 +150,8 @@ class CustomColumns(object):
                 'comments': lambda x,d: adapt_text(x, {'is_multiple':False}),
                 'datetime' : adapt_datetime,
                 'text':adapt_text,
-                'series':adapt_text
+                'series':adapt_text,
+                'enumeration': adapt_enum
         }
 
         # Create Tag Browser categories for custom columns
@@ -439,6 +446,9 @@ class CustomColumns(object):
         val = self.custom_data_adapters[data['datatype']](val, data)
 
         if data['normalized']:
+            if data['datatype'] == 'enumeration' and (
+                    val and val not in data['display']['enum_values']):
+                return None
             if not append or not data['is_multiple']:
                 self.conn.execute('DELETE FROM %s WHERE book=?'%lt, (id_,))
                 self.conn.execute(
@@ -558,7 +568,7 @@ class CustomColumns(object):
 
         if datatype in ('rating', 'int'):
             dt = 'INT'
-        elif datatype in ('text', 'comments', 'series', 'composite'):
+        elif datatype in ('text', 'comments', 'series', 'composite', 'enumeration'):
             dt = 'TEXT'
         elif datatype in ('float',):
             dt = 'REAL'

src/calibre/library/database2.py

@@ -14,6 +14,7 @@ from operator import itemgetter
 
 from PyQt4.QtGui import QImage
 
+
 from calibre.ebooks.metadata import title_sort, author_to_author_sort
 from calibre.ebooks.metadata.opf2 import metadata_to_opf
 from calibre.library.database import LibraryDatabase
@@ -33,6 +34,7 @@ from calibre import isbytestring
 from calibre.utils.filenames import ascii_filename
 from calibre.utils.date import utcnow, now as nowf, utcfromtimestamp
 from calibre.utils.config import prefs, tweaks
+from calibre.utils.icu import sort_key
 from calibre.utils.search_query_parser import saved_searches, set_saved_searches
 from calibre.ebooks import BOOK_EXTENSIONS, check_ebook_format
 from calibre.utils.magick.draw import save_cover_data_to
@@ -287,7 +289,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         # Assumption is that someone else will fix them if they change.
         self.field_metadata.remove_dynamic_categories()
         tb_cats = self.field_metadata
-        for user_cat in sorted(self.prefs.get('user_categories', {}).keys()):
+        for user_cat in sorted(self.prefs.get('user_categories', {}).keys(), key=sort_key):
             cat_name = user_cat+':' # add the ':' to avoid name collision
             tb_cats.add_user_category(label=cat_name, name=user_cat)
         if len(saved_searches().names()):
@@ -1065,7 +1067,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
             if sort == 'popularity':
                 query += ' ORDER BY count DESC, sort ASC'
             elif sort == 'name':
-                query += ' ORDER BY sort ASC'
+                query += ' ORDER BY sort COLLATE icucollate'
             else:
                 query += ' ORDER BY avg_rating DESC, sort ASC'
             data = self.conn.get(query)
@@ -1137,6 +1139,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         if sort == 'popularity':
             categories['formats'].sort(key=lambda x: x.count, reverse=True)
         else: # no ratings exist to sort on
+            # No need for ICU here.
             categories['formats'].sort(key = lambda x:x.name)
 
         #### Now do the user-defined categories. ####
@@ -1151,7 +1154,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         for c in categories.keys():
             taglist[c] = dict(map(lambda t:(t.name, t), categories[c]))
 
-        for user_cat in sorted(user_categories.keys()):
+        for user_cat in sorted(user_categories.keys(), key=sort_key):
             items = []
             for (name,label,ign) in user_categories[user_cat]:
                 if label in taglist and name in taglist[label]:
@@ -1167,7 +1170,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                         sorted(items, key=lambda x: x.count, reverse=True)
                 elif sort == 'name':
                     categories[cat_name] = \
-                        sorted(items, key=lambda x: x.sort.lower())
+                        sorted(items, key=lambda x: sort_key(x.sort))
                 else:
                     categories[cat_name] = \
                         sorted(items, key=lambda x:x.avg_rating, reverse=True)
@@ -1639,15 +1642,16 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
             return []
         return result
 
-    def set_sort_field_for_author(self, old_id, new_sort):
+    def set_sort_field_for_author(self, old_id, new_sort, commit=True, notify=False):
         self.conn.execute('UPDATE authors SET sort=? WHERE id=?', \
                               (new_sort.strip(), old_id))
-        self.conn.commit()
+        if commit:
+            self.conn.commit()
         # Now change all the author_sort fields in books by this author
         bks = self.conn.get('SELECT book from books_authors_link WHERE author=?', (old_id,))
         for (book_id,) in bks:
             ss = self.author_sort_from_book(book_id, index_is_id=True)
-            self.set_author_sort(book_id, ss)
+            self.set_author_sort(book_id, ss, notify=notify, commit=commit)
 
     def rename_author(self, old_id, new_name):
         # Make sure that any commas in new_name are changed to '|'!

src/calibre/library/field_metadata.py

@@ -83,7 +83,7 @@ class FieldMetadata(dict):
     '''
 
     VALID_DATA_TYPES = frozenset([None, 'rating', 'text', 'comments', 'datetime',
-                                  'int', 'float', 'bool', 'series', 'composite'])
+                'int', 'float', 'bool', 'series', 'composite', 'enumeration'])
 
     # Builtin metadata {{{
 
@@ -177,7 +177,7 @@ class FieldMetadata(dict):
                            'is_multiple':None,
                            'kind':'field',
                            'name':None,
-                           'search_terms':[],
+                           'search_terms':['author_sort'],
                            'is_custom':False,
                            'is_category':False}),
             ('comments',  {'table':None,

src/calibre/library/server/browse.py

@@ -16,6 +16,7 @@ from calibre import isbytestring, force_unicode, fit_image, \
 from calibre.utils.ordered_dict import OrderedDict
 from calibre.utils.filenames import ascii_filename
 from calibre.utils.config import prefs
+from calibre.utils.icu import sort_key
 from calibre.utils.magick import Image
 from calibre.library.comments import comments_to_html
 from calibre.library.server import custom_fields_to_display
@@ -273,7 +274,7 @@ class BrowseServer(object):
         opts = ['<option %svalue="%s">%s</option>' % (
             'selected="selected" ' if k==sort else '',
             xml(k), xml(n), ) for k, n in
-                sorted(sort_opts, key=operator.itemgetter(1)) if k and n]
+                sorted(sort_opts, key=lambda x: sort_key(operator.itemgetter(1)(x))) if k and n]
         ans = ans.replace('{sort_select_options}', ('\n'+' '*20).join(opts))
         lp = self.db.library_path
         if isbytestring(lp):
@@ -337,8 +338,7 @@ class BrowseServer(object):
             return category_meta[x]['name'].lower()
 
         displayed_custom_fields = custom_fields_to_display(self.db)
-        for category in sorted(categories,
+        for category in sorted(categories, key=lambda x: sort_key(getter(x))):
-                    cmp=lambda x,y: cmp(getter(x), getter(y))):
             if len(categories[category]) == 0:
                 continue
             if category == 'formats':
@@ -375,12 +375,7 @@ class BrowseServer(object):
     def browse_sort_categories(self, items, sort):
         if sort not in ('rating', 'name', 'popularity'):
             sort = 'name'
-        def sorter(x):
+        items.sort(key=lambda x: sort_key(getattr(x, 'sort', x.name)))
-            ans = getattr(x, 'sort', x.name)
-            if hasattr(ans, 'upper'):
-                ans = ans.upper()
-            return ans
-        items.sort(key=sorter)
         if sort == 'popularity':
             items.sort(key=operator.attrgetter('count'), reverse=True)
         elif sort == 'rating':
@@ -703,7 +698,7 @@ class BrowseServer(object):
                                 args[field]
                 fields.append((m['name'], r))
 
-            fields.sort(key=lambda x: x[0].lower())
+            fields.sort(key=lambda x: sort_key(x[0]))
             fields = [u'<div class="field">{0}</div>'.format(f[1]) for f in
                     fields]
             fields = u'<div class="fields">%s</div>'%('\n\n'.join(fields))

src/calibre/library/server/mobile.py

@@ -21,6 +21,7 @@ from calibre.constants import __appname__
 from calibre import human_readable, isbytestring
 from calibre.utils.date import utcfromtimestamp
 from calibre.utils.filenames import ascii_filename
+from calibre.utils.icu import sort_key
 
 def CLASS(*args, **kwargs): # class is a reserved word in Python
     kwargs['class'] = ' '.join(args)
@@ -211,8 +212,7 @@ class MobileServer(object):
 
         CFM = self.db.field_metadata
         CKEYS = [key for key in sorted(custom_fields_to_display(self.db),
-             cmp=lambda x,y: cmp(CFM[x]['name'].lower(),
+                                       key=lambda x:sort_key(CFM[x]['name']))]
-                                 CFM[y]['name'].lower()))]
         # This method uses its own book dict, not the Metadata dict. The loop
         # below could be changed to use db.get_metadata instead of reading
         # info directly from the record made by the view, but it doesn't seem

src/calibre/library/server/opds.py

@@ -20,6 +20,7 @@ from calibre.library.comments import comments_to_html
 from calibre.library.server import custom_fields_to_display
 from calibre.library.server.utils import format_tag_string, Offsets
 from calibre import guess_type
+from calibre.utils.icu import sort_key
 from calibre.utils.ordered_dict import OrderedDict
 
 BASE_HREFS = {
@@ -279,8 +280,7 @@ class AcquisitionFeed(NavFeed):
         NavFeed.__init__(self, id_, updated, version, offsets, page_url, up_url)
         CFM = db.field_metadata
         CKEYS = [key for key in sorted(custom_fields_to_display(db),
-                 cmp=lambda x,y: cmp(CFM[x]['name'].lower(),
+                                       key=lambda x: sort_key(CFM[x]['name']))]
-                                     CFM[y]['name'].lower()))]
         for item in items:
             self.root.append(ACQUISITION_ENTRY(item, version, db, updated,
                                                CFM, CKEYS, prefix))
@@ -492,7 +492,7 @@ class OPDSServer(object):
                     val = 'A'
                 starts.add(val[0].upper())
             category_groups = OrderedDict()
-            for x in sorted(starts, cmp=lambda x,y:cmp(x.lower(), y.lower())):
+            for x in sorted(starts, key=sort_key):
                 category_groups[x] = len([y for y in items if
                     getattr(y, 'sort', y.name).startswith(x)])
             items = [Group(x, y) for x, y in category_groups.items()]
@@ -571,8 +571,7 @@ class OPDSServer(object):
                 ]
         def getter(x):
             return category_meta[x]['name'].lower()
-        for category in sorted(categories,
+        for category in sorted(categories, key=lambda x: sort_key(getter(x))):
-                               cmp=lambda x,y: cmp(getter(x), getter(y))):
             if len(categories[category]) == 0:
                 continue
             if category == 'formats':

src/calibre/library/server/utils.py

@@ -13,6 +13,7 @@ import cherrypy
 from calibre import strftime as _strftime, prints, isbytestring
 from calibre.utils.date import now as nowf
 from calibre.utils.config import tweaks
+from calibre.utils.icu import sort_key
 
 class Offsets(object):
     'Calculate offsets for a paginated view'
@@ -73,7 +74,7 @@ def format_tag_string(tags, sep, ignore_max=False, no_tag_count=False):
         tlist = [t.strip() for t in tags.split(sep)]
     else:
         tlist = []
-    tlist.sort(cmp=lambda x,y:cmp(x.lower(), y.lower()))
+    tlist.sort(key=sort_key)
     if len(tlist) > MAX:
         tlist = tlist[:MAX]+['...']
     if no_tag_count:

src/calibre/library/server/xml.py

@@ -17,6 +17,7 @@ from calibre.ebooks.metadata import fmt_sidx
 from calibre.constants import preferred_encoding
 from calibre import isbytestring
 from calibre.utils.filenames import ascii_filename
+from calibre.utils.icu import sort_key
 
 E = ElementMaker()
 
@@ -101,8 +102,7 @@ class XMLServer(object):
 
             CFM = self.db.field_metadata
             CKEYS = [key for key in sorted(custom_fields_to_display(self.db),
-                 cmp=lambda x,y: cmp(CFM[x]['name'].lower(),
+                                           key=lambda x: sort_key(CFM[x]['name']))]
-                                     CFM[y]['name'].lower()))]
             custcols = []
             for key in CKEYS:
                 def concat(name, val):

src/calibre/library/sqlite.py

@@ -115,6 +115,8 @@ def pynocase(one, two, encoding='utf-8'):
             pass
     return cmp(one.lower(), two.lower())
 
+def icu_collator(s1, s2, func=None):
+    return cmp(func(unicode(s1)), func(unicode(s2)))
 
 def load_c_extensions(conn, debug=DEBUG):
     try:
@@ -167,6 +169,8 @@ class DBThread(Thread):
         self.conn.create_function('uuid4', 0, lambda : str(uuid.uuid4()))
         # Dummy functions for dynamically created filters
         self.conn.create_function('books_list_filter', 1, lambda x: 1)
+        from calibre.utils.icu import sort_key
+        self.conn.create_collation('icucollate', partial(icu_collator, func=sort_key))
 
     def run(self):
         try:

src/calibre/manual/template_lang.rst

@@ -119,10 +119,11 @@ The functions available are:
     * ``ifempty(text)``	-- if the field is not empty, return the value of the field. Otherwise return `text`.
     * ``test(text if not empty, text if empty)`` -- return `text if not empty` if the field is not empty, otherwise return `text if empty`.
     * ``contains(pattern, text if match, text if not match`` -- checks if field contains matches for the regular expression `pattern`. Returns `text if match` if matches are found, otherwise it returns `text if no match`.
-    * ``switch(pattern, value, pattern, value, ..., else_value)`` -- for each ``pattern, value`` pair, checks if the field matches the regular expression ``pattern`` and if so, returns that ``value``. If no ``pattern`` matches, then ``else_value`` is returned. You can have as many ``pattern, value`` pairs as you want.
+    * ``count(separator)`` -- interprets the value as a list of items separated by `separator`, returning the number of items in the list. Most lists use a comma as the separator, but authors uses an ampersand. Examples: `{tags:count(,)}`, `{authors:count(&)}`
+    * ``lookup(pattern, field, pattern, field, ..., else_field)`` -- like switch, except the arguments are field (metadata) names, not text. The value of the appropriate field will be fetched and used. Note that because composite columns are fields, you can use this function in one composite field to use the value of some other composite field. This is extremely useful when constructing variable save paths (more later).
     * ``re(pattern, replacement)`` -- return the field after applying the regular expression. All instances of `pattern` are replaced with `replacement`. As in all of |app|, these are python-compatible regular expressions.
     * ``shorten(left chars, middle text, right chars)`` -- Return a shortened version of the field, consisting of `left chars` characters from the beginning of the field, followed by `middle text`, followed by `right chars` characters from the end of the string. `Left chars` and `right chars` must be integers. For example, assume the title of the book is `Ancient English Laws in the Times of Ivanhoe`, and you want it to fit in a space of at most 15 characters. If you use ``{title:shorten(9,-,5)}``, the result will be `Ancient E-nhoe`. If the field's length is less than ``left chars`` + ``right chars`` + the length of ``middle text``, then the field will be used intact. For example, the title `The Dome` would not be changed.
-    * ``lookup(pattern, field, pattern, field, ..., else_field)`` -- like switch, except the arguments are field (metadata) names, not text. The value of the appropriate field will be fetched and used. Note that because composite columns are fields, you can use this function in one composite field to use the value of some other composite field. This is extremely useful when constructing variable save paths (more later).
+    * ``switch(pattern, value, pattern, value, ..., else_value)`` -- for each ``pattern, value`` pair, checks if the field matches the regular expression ``pattern`` and if so, returns that ``value``. If no ``pattern`` matches, then ``else_value`` is returned. You can have as many ``pattern, value`` pairs as you want.
 
 
 Now, about using functions and formatting in the same field. Suppose you have an integer custom column called ``#myint`` that you want to see with leading zeros, as in ``003``. To do this, you would use a format of ``0>3s``. However, by default, if a number (integer or float) equals zero then the field produces the empty value, so zero values will produce nothing, not ``000``. If you really want to see ``000`` values, then you use both the format string and the ``ifempty`` function to change the empty value back to a zero. The field reference would be::

src/calibre/startup.py

@@ -199,6 +199,11 @@ if not _run_once:
 
     __builtin__.__dict__['lopen'] = local_open
 
+
+    import mimetypes
+    mimetypes.init([P('mime.types')])
+    guess_type = mimetypes.guess_type
+
 def test_lopen():
     from calibre.ptempfile import TemporaryDirectory
     from calibre import CurrentDir