merge from trunk

Changelog.yaml

@@ -4,6 +4,120 @@
 # for important features/bug fixes.
 # Also, each release can have new and improved recipes.
 
+- version: 0.7.40
+  date: 2011-01-14
+
+  new features:
+    - title: "A new 'highlight matches' search mode" 
+      description: >
+        "There is now a checkbox next to the search bar named 'Highlight'. If you check it, searching will highlight
+        all matched books instead of filtering the book list to all matched books."
+
+    - title: "RTF Input: Improved support for conversion of images. The bug where some images were shrunk should no longer happen"
+
+    - title: "Template language: Allow you to create your own formatting functions. Accessible via Preferences->Advanced->Template functions"
+
+    - title: "News download: Convert various HTML 5 tags into <div> to support readers that cannot handle HTML 5 tags"
+
+    - title: "RTF metadata: Add support for publisher and tags."
+      tickets: [6657]
+
+    - title: "BibTeX catalog: Add support for custom columns"
+
+    - title: "TXT Input: Support for textile markup"
+
+    - title: "Various minor tweaks to improve usability of Preferences->Plugins"
+
+    - title: "TXT Output: Convert <hr> to scene break marker."
+
+    - title: "Support for the Archos 70"
+
+    - title: "SONY Driver: Add an option to automatically refresh the covers on every connect. Accessible via: Preferences->Plugins->Device interface plugins"
+
+    - title: "Add access to the larger template editor from plugboards via context menu."
+
+    - title: "Speed improvement when connecting a large library to a device"
+
+    - title: "Speedup when searching on multiple words in a large library"
+
+    - title: "TXT Input: Add a heauristic formatting processor"
+
+
+  bug fixes:
+    - title: "Fix bug that caused automatic news removal to remove any book that has a tag that contains the word 'news' instead of only books that have the tag News"
+
+    - title: "Refactor the downloading social metadata message box to allow canceling."
+      tickets: [8234]
+
+    - title: "Kobo drive does not deal with Null value in DateCreated column"
+      tickets: [8308]
+
+    - title: "MOBI Input: Fix regression that caused images placed inside svg tags to be discarded"
+
+    - title: "Fix selecting Tablet output profile would actually select the Samsung Galaxy S profile"
+
+    - title: "Catalog generation: Fix a condition that could cause TOCs to not be properly generated in MOBI format catalogs"
+      tickets: [8295]
+
+    - title: "Zip file reading: Be more tolerant when a zip file has a damaged file directory"
+
+    - title: "RTF Input: Various code cleanups. Go back to trying to handle unicode mappings without pre-processing. This will mean that some RTF files that used to convert, won't anymore. Please open tickets and attach them."
+      tickets: [8171]
+
+    - title: "ImageMagick: When identifying an image don't read the entire image"
+
+    - title: "FB2 Output: Add cover to FB2 metadata."
+
+    - title: "Fix inability to customize builting recipe when more than one recipe has the same name"
+      tickets: [8281]
+
+    - title: "RTF Input: Fix regression that broke the Preprocess HTML option"
+
+    - title: "Fix XSS vulnerability in content server."
+      tickets: [7980]
+
+    - title: "TXT Output: Clean up and produce consistant output. Spacing around headings. Headings are not indented when using the remove paragraph spacing option."
+
+    - title: "Catalog generation: Handle invalid covers gracefully"
+
+    - title: "Email settings: Before displaying the email test dialog warn the user that it will expose their email password"
+
+    - title: "PDB Output: Fix regression that caused some PDB files to not work with other software"
+      tickets: [8231]
+
+  improved recipes:
+    - Financial Times UK
+    - Globe and Mail
+    - Wired Daily
+    - MIT Technology Review
+    - MSNBC
+    - expansion.com
+    - New York Times
+    - Heraldo de Aragon
+    - Exiled online
+
+  new recipes:
+    - title: "Yakima Herald and Tri-City Herald"
+      author: "Laura Gjovaag"
+
+    - title: "Wichita Eagle"
+      author: "Jason Cameron"
+
+    - title: "Pressthink and Zero Hedge"
+      author: "Darko Miletic"
+
+    - title: "tyzden"
+      author: "zemiak"
+
+    - title: "El Correo"
+      author: "desUBIKado"
+
+    - title: "Cicero"
+      author: "mad"
+
+    - title: "El Publico"
+      author: "Gerardo Diez"
+
 - version: 0.7.38
   date: 2011-01-07
 

resources/recipes/financial_times_uk.recipe

@@ -1,5 +1,5 @@
 __license__   = 'GPL v3'
-__copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
+__copyright__ = '2010-2011, Darko Miletic <darko.miletic at gmail.com>'
 '''
 ft.com
 '''
@@ -52,22 +52,38 @@ class FinancialTimes(BasicNewsRecipe):
                 .copyright{font-size: x-small}
                 """
 
-    def parse_index(self):
+    def get_artlinks(self, elem):
         articles = []
+        for item in elem.findAll('a',href=True):
+            url   = self.PREFIX + item['href']
+            title = self.tag_to_string(item)
+            date = strftime(self.timefmt)
+            articles.append({
+                              'title'      :title
+                             ,'date'       :date
+                             ,'url'        :url
+                             ,'description':''
+                            })
+        return articles
+                
+    def parse_index(self):
+        feeds = []
         soup = self.index_to_soup(self.INDEX)
         wide = soup.find('div',attrs={'class':'wide'})
-        if wide:
+        if not wide:
-            for item in wide.findAll('a',href=True):
+           return feeds
-                url   = self.PREFIX + item['href']
+        strest = wide.findAll('h3', attrs={'class':'section'})
-                title = self.tag_to_string(item)
+        if not strest:
-                date = strftime(self.timefmt)
+           return feeds
-                articles.append({
+        st = wide.find('h4',attrs={'class':'section-no-arrow'})
-                                  'title'      :title
+        if st:
-                                 ,'date'       :date
+           strest.insert(0,st)
-                                 ,'url'        :url
+        for item in strest:
-                                 ,'description':''
+            ftitle   = self.tag_to_string(item)
-                                })
+            self.report_progress(0, _('Fetching feed')+' %s...'%(ftitle))            
-        return [('FT UK edition',articles)]
+            feedarts = self.get_artlinks(item.parent.ul)
+            feeds.append((ftitle,feedarts))
+        return feeds
 
     def preprocess_html(self, soup):
         return self.adeify_images(soup)

resources/recipes/mail_and_guardian.recipe

@@ -0,0 +1,32 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class AdvancedUserRecipe1295081935(BasicNewsRecipe):
+     title          = u'Mail & Guardian ZA News'
+     __author__     = '77ja65'
+     language = 'en'
+     oldest_article = 7
+     max_articles_per_feed = 30
+     no_stylesheets = True
+     masthead_url          =  'http://c1608832.cdn.cloudfiles.rackspacecloud.com/mg_logo.gif'
+     remove_tags_after = [dict(id='content')]
+
+     feeds          = [
+         (u'National News', u'http://www.mg.co.za/rss/national'),
+         (u'Top Stories', u'http://www.mg.co.za/rss'),
+         (u'Africa News', u'http://www.mg.co.za/rss/africa'),
+         (u'Sport', u'http://www.mg.co.za/rss/sport'),
+         (u'Business', u'http://www.mg.co.za/rss/business'),
+         (u'And In Other News', u'http://www.mg.co.za/rss/and-in-other-news'),
+         (u'World News', u'http://www.mg.co.za/rss/world')
+         ]
+
+     def print_version(self, url):
+           return url.replace('http://www.mg.co.za/article/',
+ 'http://www.mg.co.za/printformat/single/')
+
+     extra_css = '''
+                     h1{font-family:Arial,Helvetica,sans-serif; font-
+ weight:bold;font-size:large;}
+                     h2{font-family:Arial,Helvetica,sans-serif; font-
+ weight:normal;font-size:small;}
+                 '''

resources/recipes/pressthink.recipe

@@ -0,0 +1,61 @@
+__license__   = 'GPL v3'
+__copyright__ = '2011, Darko Miletic <darko.miletic at gmail.com>'
+'''
+pressthink.org
+'''
+
+from calibre.web.feeds.recipes import BasicNewsRecipe
+
+class PressThink(BasicNewsRecipe):
+    title                  = 'PressThink'
+    __author__             = 'Darko Miletic'
+    description            = 'Ghost of democracy in the media machine'
+    oldest_article         = 60
+    max_articles_per_feed  = 100
+    no_stylesheets         = True
+    use_embedded_content   = False
+    encoding               = 'utf8'
+    publisher              = 'Arthur L. Carter Journalism Institute'
+    category               = 'news, USA, world, economy, politics, media'
+    language               = 'en'
+    publication_type       = 'blog'
+    extra_css              = """
+                                body{ font-family: Helvetica,Arial,sans-serif }
+                                img{display: block; margin-bottom: 0.5em}
+                                h6{font-size: 1.1em; font-weight: bold}
+                                .post-author{font-family: Georgia,serif}
+                                .post-title{color: #AB0000}
+                                .says{color: gray}
+                                .comment {
+                                    border-bottom: 1px dotted #555555;
+                                    border-top: 1px dotted #DDDDDD;
+                                    margin-left: 10px;
+                                    min-height: 100px;
+                                    padding: 15px 0 20px;
+                                }
+                             """
+
+    conversion_options = {
+                             'comments' : description
+                            ,'tags'     : category
+                            ,'language' : language
+                            ,'publisher': publisher
+                         }
+
+    remove_tags       = [dict(name=['form','iframe','embed','object','link','base','table','meta'])]
+    keep_only_tags    = [dict(attrs={'class':['post-title','post-author','entry','postmetadata alt','commentlist']})]
+
+    feeds = [(u'Articles', u'http://pressthink.org/feed/')]
+
+    def preprocess_html(self, soup):
+        for item in soup.findAll(style=True):
+            del item['style']
+        for item in soup.findAll('img', alt=False):
+            item['alt'] = 'image'
+        for alink in soup.findAll('a'):
+            if alink.string is not None:
+               tstr = alink.string
+               alink.replaceWith(tstr)
+        return soup
+
+

resources/recipes/sportsillustrated.recipe

@@ -1,5 +1,5 @@
 from calibre.web.feeds.recipes import BasicNewsRecipe
-from calibre.ebooks.BeautifulSoup import BeautifulSoup
+#from calibre.ebooks.BeautifulSoup import BeautifulSoup
 from urllib import quote
 
 class SportsIllustratedRecipe(BasicNewsRecipe) :
@@ -91,7 +91,7 @@ class SportsIllustratedRecipe(BasicNewsRecipe) :
         #   expire : no idea what value to use
         # All this comes from the Javascript function that redirects to the print version. It's called PT() and is defined in the file 48.js
 
-    def preprocess_html(self, soup):
+    '''def preprocess_html(self, soup):
         header = soup.find('div', attrs = {'class' : 'siv_artheader'})
         homeMadeSoup = BeautifulSoup('<html><head></head><body></body></html>')
         body = homeMadeSoup.body
@@ -115,4 +115,5 @@ class SportsIllustratedRecipe(BasicNewsRecipe) :
             body.append(para)
 
         return homeMadeSoup
+        '''
 

resources/recipes/tri_city_herald.recipe

@@ -0,0 +1,25 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class TriCityHeraldRecipe(BasicNewsRecipe):
+    title          = u'Tri-City Herald'
+    description    = 'The Tri-City Herald Mid-Columbia.'
+    language = 'en'
+    __author__     = 'Laura Gjovaag'
+    oldest_article        = 1.5
+    max_articles_per_feed = 100
+    no_stylesheets        = True
+    remove_javascript     = True
+    keep_only_tags = [
+                      dict(name='div', attrs={'id':'story_header'}),
+                      dict(name='img', attrs={'class':'imageCycle'}),
+                      dict(name='div', attrs={'id':['cycleImageCaption', 'story_body']})
+                      ]
+    remove_tags    = [
+                      dict(name='div', attrs={'id':'story_mlt'}),
+                      dict(name='a', attrs={'id':'commentCount'}),
+                      dict(name=['script', 'noscript', 'style'])]
+    extra_css      = 'h1{font: bold 140%;} #cycleImageCaption{font: monospace 60%}'
+
+    feeds          = [
+                      (u'Tri-City Herald Mid-Columbia', u'http://www.tri-cityherald.com/901/index.rss')
+                     ]

resources/recipes/wichita_eagle.recipe

@@ -0,0 +1,29 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class AdvancedUserRecipe1294938721(BasicNewsRecipe):
+     title            = u'Wichita Eagle'
+     language = 'en'
+     __author__ = 'Jason Cameron'
+     description  = 'Daily news from the Wichita Eagle'
+     oldest_article = 1
+     max_articles_per_feed = 30
+     keep_only_tags = [dict(name='div', attrs={'id':'wide'})]
+     feeds          = [
+                                 (u'Local News',
+ u'http://www.kansas.com/news/local/index.rss'),
+                                 (u'National News',
+ u'http://www.kansas.com/news/nation-world/index.rss'),
+                                 (u'Sports',
+ u'http://www.kansas.com/sports/index.rss'),
+                                 (u'Opinion',
+ u'http://www.kansas.com/opinion/index.rss'),
+                                 (u'Life',
+ u'http://www.kansas.com/living/index.rss'),
+                                 (u'Entertainment',
+ u'http://www.kansas.com/entertainment/index.rss')
+                           ]
+
+     def print_version(self, url):
+         urlparts = url.split('/')
+         newadd = urlparts[5]+'/v-print'
+         return url.replace(url, newadd.join(url.split(urlparts[5])))

resources/recipes/yakima_herald.recipe

@@ -0,0 +1,21 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class YakimaHeraldRepublicRecipe(BasicNewsRecipe):
+    title          = u'Yakima Herald-Republic'
+    description    = 'The Yakima Herald-Republic.'
+    language = 'en'
+    __author__     = 'Laura Gjovaag'
+    oldest_article        = 1.5
+    max_articles_per_feed = 100
+    no_stylesheets        = True
+    remove_javascript     = True
+    keep_only_tags = [
+                      dict(name='div', attrs={'id':['searchleft', 'headline_credit']}),
+                      dict(name='div', attrs={'class':['photo', 'cauthor', 'photocredit']}),
+                      dict(name='div', attrs={'id':['content_body', 'footerleft']})
+                      ]
+    extra_css = '.cauthor {font: monospace 60%;} .photocredit {font: monospace 60%}'
+
+    feeds          = [
+                      (u'Yakima Herald Online', u'http://feeds.feedburner.com/yhronlinenews'),
+                     ]

resources/template-functions.json

@@ -0,0 +1,28 @@
+{
+    "contains": "def evaluate(self, formatter, kwargs, mi, locals,\n             val, test, value_if_present, value_if_not):\n    if re.search(test, val):\n        return value_if_present\n    else:\n        return value_if_not\n", 
+    "divide": "def evaluate(self, formatter, kwargs, mi, locals, x, y):\n    x = float(x if x else 0)\n    y = float(y if y else 0)\n    return unicode(x / y)\n", 
+    "uppercase": "def evaluate(self, formatter, kwargs, mi, locals, val):\n    return val.upper()\n", 
+    "strcat": "def evaluate(self, formatter, kwargs, mi, locals, *args):\n    i = 0\n    res = ''\n    for i in range(0, len(args)):\n        res += args[i]\n    return res\n", 
+    "substr": "def evaluate(self, formatter, kwargs, mi, locals, str_, start_, end_):\n    return str_[int(start_): len(str_) if int(end_) == 0 else int(end_)]\n", 
+    "ifempty": "def evaluate(self, formatter, kwargs, mi, locals, val, value_if_empty):\n    if val:\n        return val\n    else:\n        return value_if_empty\n", 
+    "field": "def evaluate(self, formatter, kwargs, mi, locals, name):\n    return formatter.get_value(name, [], kwargs)\n", 
+    "capitalize": "def evaluate(self, formatter, kwargs, mi, locals, val):\n    return capitalize(val)\n", 
+    "list_item": "def evaluate(self, formatter, kwargs, mi, locals, val, index, sep):\n    if not val:\n        return ''\n    index = int(index)\n    val = val.split(sep)\n    try:\n        return val[index]\n    except:\n        return ''\n", 
+    "shorten": "def evaluate(self, formatter, kwargs, mi, locals,\n             val, leading, center_string, trailing):\n    l = max(0, int(leading))\n    t = max(0, int(trailing))\n    if len(val) > l + len(center_string) + t:\n        return val[0:l] + center_string + ('' if t == 0 else val[-t:])\n    else:\n        return val\n", 
+    "re": "def evaluate(self, formatter, kwargs, mi, locals, val, pattern, replacement):\n    return re.sub(pattern, replacement, val)\n", 
+    "add": "def evaluate(self, formatter, kwargs, mi, locals, x, y):\n    x = float(x if x else 0)\n    y = float(y if y else 0)\n    return unicode(x + y)\n", 
+    "lookup": "def evaluate(self, formatter, kwargs, mi, locals, val, *args):\n    if len(args) == 2: # here for backwards compatibility\n        if val:\n            return formatter.vformat('{'+args[0].strip()+'}', [], kwargs)\n        else:\n            return formatter.vformat('{'+args[1].strip()+'}', [], kwargs)\n    if (len(args) % 2) != 1:\n        raise ValueError(_('lookup requires either 2 or an odd number of arguments'))\n    i = 0\n    while i < len(args):\n        if i + 1 >= len(args):\n            return formatter.vformat('{' + args[i].strip() + '}', [], kwargs)\n        if re.search(args[i], val):\n            return formatter.vformat('{'+args[i+1].strip() + '}', [], kwargs)\n        i += 2\n", 
+    "template": "def evaluate(self, formatter, kwargs, mi, locals, template):\n    template = template.replace('[[', '{').replace(']]', '}')\n    return formatter.safe_format(template, kwargs, 'TEMPLATE', mi)\n", 
+    "print": "def evaluate(self, formatter, kwargs, mi, locals, *args):\n    print args\n    return None\n", 
+    "titlecase": "def evaluate(self, formatter, kwargs, mi, locals, val):\n    return titlecase(val)\n", 
+    "test": "def evaluate(self, formatter, kwargs, mi, locals, val, value_if_set, value_not_set):\n    if val:\n        return value_if_set\n    else:\n        return value_not_set\n", 
+    "eval": "def evaluate(self, formatter, kwargs, mi, locals, template):\n    from formatter import eval_formatter\n    template = template.replace('[[', '{').replace(']]', '}')\n    return eval_formatter.safe_format(template, locals, 'EVAL', None)\n", 
+    "multiply": "def evaluate(self, formatter, kwargs, mi, locals, x, y):\n    x = float(x if x else 0)\n    y = float(y if y else 0)\n    return unicode(x * y)\n", 
+    "subtract": "def evaluate(self, formatter, kwargs, mi, locals, x, y):\n    x = float(x if x else 0)\n    y = float(y if y else 0)\n    return unicode(x - y)\n", 
+    "count": "def evaluate(self, formatter, kwargs, mi, locals, val, sep):\n    return unicode(len(val.split(sep)))\n", 
+    "lowercase": "def evaluate(self, formatter, kwargs, mi, locals, val):\n    return val.lower()\n", 
+    "assign": "def evaluate(self, formatter, kwargs, mi, locals, target, value):\n    locals[target] = value\n    return value\n", 
+    "switch": "def evaluate(self, formatter, kwargs, mi, locals, val, *args):\n    if (len(args) % 2) != 1:\n        raise ValueError(_('switch requires an odd number of arguments'))\n    i = 0\n    while i < len(args):\n        if i + 1 >= len(args):\n            return args[i]\n        if re.search(args[i], val):\n            return args[i+1]\n        i += 2\n", 
+    "strcmp": "def evaluate(self, formatter, kwargs, mi, locals, x, y, lt, eq, gt):\n    v = strcmp(x, y)\n    if v < 0:\n        return lt\n    if v == 0:\n        return eq\n    return gt\n", 
+    "cmp": "def evaluate(self, formatter, kwargs, mi, locals, x, y, lt, eq, gt):\n    x = float(x if x else 0)\n    y = float(y if y else 0)\n    if x < y:\n        return lt\n    if x == y:\n        return eq\n    return gt\n"
+}

setup/resources.py

@@ -84,6 +84,23 @@ class Resources(Command):
 
             cPickle.dump(complete, open(dest, 'wb'), -1)
 
+        self.info('\tCreating template-functions.json')
+        dest = self.j(self.RESOURCES, 'template-functions.json')
+        function_dict = {}
+        import inspect
+        from calibre.utils.formatter_functions import all_builtin_functions
+        for obj in all_builtin_functions:
+            eval_func = inspect.getmembers(obj,
+                    lambda x: inspect.ismethod(x) and x.__name__ == 'evaluate')
+            try:
+                lines = [l[4:] for l in inspect.getsourcelines(eval_func[0][1])[0]]
+            except:
+                continue
+            lines = ''.join(lines)
+            function_dict[obj.name] = lines
+        import json
+        json.dump(function_dict, open(dest, 'wb'), indent=4)
+
     def clean(self):
         for x in ('scripts', 'recipes', 'ebook-convert-complete'):
             x = self.j(self.RESOURCES, x+'.pickle')

setup/upload.py

@@ -6,7 +6,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
-import os, re, cStringIO, base64, httplib, subprocess, hashlib, shutil
+import os, re, cStringIO, base64, httplib, subprocess, hashlib, shutil, time
 from subprocess import check_call
 from tempfile import NamedTemporaryFile, mkdtemp
 
@@ -160,7 +160,7 @@ class UploadToGoogleCode(Command):
 
         return 'multipart/form-data; boundary=%s' % BOUNDARY, CRLF.join(body)
 
-    def upload(self, fname, desc, labels=[]):
+    def upload(self, fname, desc, labels=[], retry=0):
         form_fields = [('summary', desc)]
         form_fields.extend([('label', l.strip()) for l in labels])
 
@@ -183,6 +183,10 @@ class UploadToGoogleCode(Command):
 
         print 'Failed to upload with code %d and reason: %s'%(resp.status,
                 resp.reason)
+        if retry < 1:
+            print 'Retrying in 5 seconds....'
+            time.sleep(5)
+            return self.upload(fname, desc, labels=labels, retry=retry+1)
         raise Exception('Failed to upload '+fname)
 
 

src/calibre/constants.py

@@ -2,7 +2,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = 'calibre'
-__version__   = '0.7.38'
+__version__   = '0.7.40'
 __author__    = "Kovid Goyal <kovid@kovidgoyal.net>"
 
 import re

src/calibre/customize/builtins.py

@@ -705,13 +705,17 @@ class ActionTweakEpub(InterfaceActionBase):
     name = 'Tweak ePub'
     actual_plugin = 'calibre.gui2.actions.tweak_epub:TweakEpubAction'
 
+class ActionNextMatch(InterfaceActionBase):
+    name = 'Next Match'
+    actual_plugin = 'calibre.gui2.actions.next_match:NextMatchAction'
+
 plugins += [ActionAdd, ActionFetchAnnotations, ActionGenerateCatalog,
         ActionConvert, ActionDelete, ActionEditMetadata, ActionView,
         ActionFetchNews, ActionSaveToDisk, ActionShowBookDetails,
         ActionRestart, ActionOpenFolder, ActionConnectShare,
         ActionSendToDevice, ActionHelp, ActionPreferences, ActionSimilarBooks,
         ActionAddToLibrary, ActionEditCollections, ActionChooseLibrary,
-        ActionCopyToLibrary, ActionTweakEpub]
+        ActionCopyToLibrary, ActionTweakEpub, ActionNextMatch]
 
 # }}}
 
@@ -843,6 +847,17 @@ class Plugboard(PreferencesPlugin):
     config_widget = 'calibre.gui2.preferences.plugboard'
     description = _('Change metadata fields before saving/sending')
 
+class TemplateFunctions(PreferencesPlugin):
+    name = 'TemplateFunctions'
+    icon = I('template_funcs.png')
+    gui_name = _('Template Functions')
+    category = 'Advanced'
+    gui_category = _('Advanced')
+    category_order = 5
+    name_order = 4
+    config_widget = 'calibre.gui2.preferences.template_functions'
+    description = _('Create your own template functions')
+
 class Email(PreferencesPlugin):
     name = 'Email'
     icon = I('mail.png')
@@ -904,6 +919,6 @@ class Misc(PreferencesPlugin):
 
 plugins += [LookAndFeel, Behavior, Columns, Toolbar, InputOptions,
         CommonOptions, OutputOptions, Adding, Saving, Sending, Plugboard,
-        Email, Server, Plugins, Tweaks, Misc]
+        Email, Server, Plugins, Tweaks, Misc, TemplateFunctions]
 
 #}}}

src/calibre/customize/profiles.py

@@ -441,7 +441,7 @@ class TabletOutput(iPadOutput):
 
 class SamsungGalaxy(TabletOutput):
     name = 'Samsung Galaxy'
-    shortname = 'galaxy'
+    short_name = 'galaxy'
     description = _('Intended for the Samsung Galaxy and similar tablet devices with '
             'a resolution of 600x1280')
     screen_size = comic_screen_size = (600, 1280)

src/calibre/devices/kobo/books.py

@@ -27,7 +27,7 @@ class Book(Book_):
 
         self.size = size # will be set later if None
 
-        if ContentType == '6':
+        if ContentType == '6' and date is not None:
             self.datetime = time.strptime(date, "%Y-%m-%dT%H:%M:%S.%f")
         else:
             try:

src/calibre/devices/sne/driver.py

@@ -33,6 +33,6 @@ class SNE(USBMS):
     STORAGE_CARD_VOLUME_LABEL = 'SNE Storage Card'
 
     EBOOK_DIR_MAIN = EBOOK_DIR_CARD_A = 'Books'
-    SUPPORTS_SUB_DIRS = True
+    SUPPORTS_SUB_DIRS = False
 
 

src/calibre/ebooks/mobi/reader.py

@@ -632,9 +632,18 @@ class MobiReader(object):
                 attrib['class'] = cls
 
         for tag in svg_tags:
-            p = tag.getparent()
+            images = tag.xpath('descendant::img[@src]')
-            if hasattr(p, 'remove'):
+            parent = tag.getparent()
-                p.remove(tag)
+
+            if images and hasattr(parent, 'find'):
+                index = parent.index(tag)
+                for img in images:
+                    img.getparent().remove(img)
+                    img.tail = img.text = None
+                    parent.insert(index, img)
+
+            if hasattr(parent, 'remove'):
+                parent.remove(tag)
 
     def create_opf(self, htmlfile, guide=None, root=None):
         mi = getattr(self.book_header.exth, 'mi', self.embedded_mi)

src/calibre/ebooks/rtf/input.py

@@ -286,7 +286,6 @@ class RTFInput(InputFormatPlugin):
         try:
             xml = self.generate_xml(stream.name)
         except RtfInvalidCodeException, e:
-            raise
             raise ValueError(_('This RTF file has a feature calibre does not '
             'support. Convert it to HTML first and then try it.\n%s')%e)
 

src/calibre/ebooks/rtf2xml/ParseRtf.py

@@ -226,10 +226,6 @@ class ParseRtf:
         try:
             return_value = process_tokens_obj.process_tokens()
         except InvalidRtfException, msg:
-            try:
-                os.remove(self.__temp_file)
-            except OSError:
-                pass
             #Check to see if the file is correctly encoded
             encode_obj = default_encoding.DefaultEncoding(
             in_file = self.__temp_file,
@@ -241,14 +237,17 @@ class ParseRtf:
             check_encoding_obj = check_encoding.CheckEncoding(
                     bug_handler = RtfInvalidCodeException,
                         )
-            enc = encode_obj.get_codepage()
+            enc = 'cp' + encode_obj.get_codepage()
-            if enc != 'mac_roman':
+            msg = 'Exception in token processing'
-                enc = 'cp' + enc
             if check_encoding_obj.check_encoding(self.__file, enc):
                 file_name = self.__file if isinstance(self.__file, str) \
                                     else self.__file.encode('utf-8')
                 msg = 'File %s does not appear to be correctly encoded.\n' % file_name
-                raise InvalidRtfException, msg
+            try:
+                os.remove(self.__temp_file)
+            except OSError:
+                pass
+            raise InvalidRtfException, msg
         delete_info_obj = delete_info.DeleteInfo(
             in_file = self.__temp_file,
             copy = self.__copy,

src/calibre/ebooks/rtf2xml/default_encoding.py

@@ -74,9 +74,6 @@ class DefaultEncoding:
         if not self.__datafetched:
             self._encoding()
             self.__datafetched = True
-        if self.__platform == 'Macintosh':
-            code_page = self.__code_page
-        else:
             code_page = 'ansicpg' + self.__code_page
         return self.__platform, code_page, self.__default_num
 
@@ -94,49 +91,60 @@ class DefaultEncoding:
 
     def _encoding(self):
         with open(self.__file, 'r') as read_obj:
+            cpfound = False
             if not self.__fetchraw:
                 for line in read_obj:
                     self.__token_info = line[:16]
                     if self.__token_info == 'mi<mk<rtfhed-end':
                         break
-                    if self.__token_info == 'cw<ri<ansi-codpg':
-                        #cw<ri<ansi-codpg<nu<10000
-                        self.__code_page = line[20:-1] if int(line[20:-1]) \
-                                            else '1252'
                     if self.__token_info == 'cw<ri<macintosh_':
                         self.__platform = 'Macintosh'
-                        self.__code_page = 'mac_roman'
                     elif self.__token_info == 'cw<ri<pc________':
                         self.__platform = 'IBMPC'
-                        self.__code_page = '437'
                     elif self.__token_info == 'cw<ri<pca_______':
                         self.__platform = 'OS/2'
-                        self.__code_page = '850'
+                    if self.__token_info == 'cw<ri<ansi-codpg' \
+                        and int(line[20:-1]):
+                            self.__code_page = line[20:-1]
                     if self.__token_info == 'cw<ri<deflt-font':
                         self.__default_num = line[20:-1]
+                        cpfound = True
                         #cw<ri<deflt-font<nu<0
+                if self.__platform != 'Windows' and \
+                        not cpfound:
+                    if self.__platform == 'Macintosh':
+                       self.__code_page = '10000'
+                    elif self.__platform == 'IBMPC':
+                        self.__code_page = '437'
+                    elif self.__platform == 'OS/2':
+                        self.__code_page = '850'
             else:
                 fenc = re.compile(r'\\(mac|pc|ansi|pca)[\\ \{\}\t\n]+')
                 fenccp = re.compile(r'\\ansicpg(\d+)[\\ \{\}\t\n]+')
+
                 for line in read_obj:
+                    if fenc.search(line):
+                        enc = fenc.search(line).group(1)
                     if fenccp.search(line):
                         cp = fenccp.search(line).group(1)
                         if not int(cp):
                             self.__code_page = cp
+                        cpfound = True
                         break
-                    if fenc.search(line):
+                if self.__platform != 'Windows' and \
-                        enc = fenc.search(line).group(1)
+                        not cpfound:
-                        if enc == 'mac':
+                    if enc == 'mac':
-                            self.__code_page = 'mac_roman'
+                        self.__code_page = '10000'
-                        elif enc == 'pc':
+                    elif enc == 'pc':
-                            self.__code_page = '437'
+                        self.__code_page = '437'
-                        elif enc == 'pca':
+                    elif enc == 'pca':
-                            self.__code_page = '850'
+                        self.__code_page = '850'
 
-# if __name__ == '__main__':
+if __name__ == '__main__':
-    # encode_obj = DefaultEncoding(
+    import sys
-            # in_file = sys.argv[1],
+    encode_obj = DefaultEncoding(
-            # bug_handler = Exception,
+            in_file = sys.argv[1],
-            # check_raw = True,
+            bug_handler = Exception,
-            # )
+            check_raw = True,
-    # print encode_obj.get_codepage()
+            )
+    print encode_obj.get_codepage()

src/calibre/ebooks/rtf2xml/delete_info.py

@@ -20,7 +20,7 @@ import sys, os, tempfile
 from calibre.ebooks.rtf2xml import copy
 
 class DeleteInfo:
-    """Delelet unecessary destination groups"""
+    """Delete unecessary destination groups"""
     def __init__(self,
             in_file ,
             bug_handler,
@@ -31,17 +31,14 @@ class DeleteInfo:
         self.__bug_handler = bug_handler
         self.__copy = copy
         self.__write_to = tempfile.mktemp()
+        self.__run_level = run_level
+        self.__initiate_allow()
         self.__bracket_count= 0
         self.__ob_count = 0
         self.__cb_count = 0
-        # self.__after_asterisk = False
-        # self.__delete = 0
-        self.__initiate_allow()
         self.__ob = 0
         self.__write_cb = False
-        self.__run_level = run_level
         self.__found_delete = False
-        # self.__list = False
 
     def __initiate_allow(self):
         """
@@ -57,6 +54,8 @@ class DeleteInfo:
                             'cw<an<annotation',
                             'cw<cm<comment___',
                             'cw<it<lovr-table',
+                            # info table
+                            'cw<di<company___',
                             # 'cw<ls<list______',
                         )
         self.__not_allowable = (
@@ -116,7 +115,6 @@ class DeleteInfo:
         """
         # Test for {\*}, in which case don't enter
         # delete state
-        # self.__after_asterisk = False # only enter this function once
         self.__found_delete = True
         if self.__token_info == 'cb<nu<clos-brack':
             if self.__delete_count == self.__cb_count:
@@ -128,7 +126,7 @@ class DeleteInfo:
                 # not sure what happens here!
                 # believe I have a '{\*}
                 if self.__run_level > 3:
-                    msg = 'flag problem\n'
+                    msg = 'Flag problem\n'
                     raise self.__bug_handler, msg
                 return True
         elif self.__token_info in self.__allowable :
@@ -173,8 +171,8 @@ class DeleteInfo:
         Return True for all control words.
         Return False otherwise.
         """
-        if self.__delete_count == self.__cb_count and self.__token_info ==\
+        if self.__delete_count == self.__cb_count and \
-            'cb<nu<clos-brack':
+                self.__token_info == 'cb<nu<clos-brack':
             self.__state = 'default'
             if self.__write_cb:
                 self.__write_cb = False
@@ -186,32 +184,24 @@ class DeleteInfo:
             return False
 
     def delete_info(self):
-        """Main method for handling other methods. Read one line in at
+        """Main method for handling other methods. Read one line at
         a time, and determine whether to print the line based on the state."""
         with open(self.__file, 'r') as read_obj:
             with open(self.__write_to, 'w') as self.__write_obj:
                 for line in read_obj:
                     #ob<nu<open-brack<0001
-                    to_print = True
                     self.__token_info = line[:16]
                     if self.__token_info == 'ob<nu<open-brack':
                         self.__ob_count = line[-5:-1]
                     if self.__token_info == 'cb<nu<clos-brack':
                         self.__cb_count = line[-5:-1]
+                    # Get action to perform
                     action = self.__state_dict.get(self.__state)
                     if not action:
-                        sys.stderr.write(_('No action in dictionary state is "%s" \n')
+                        sys.stderr.write('No action in dictionary state is "%s" \n'
                                 % self.__state)
-                    to_print = action(line)
+                    # Print if allowed by action
-                    # if self.__after_asterisk:
+                    if action(line):
-                        # to_print = self.__asterisk_func(line)
-                    # elif self.__list:
-                        # self.__in_list_func(line)
-                    # elif self.__delete:
-                        # to_print = self.__delete_func(line)
-                    # else:
-                        # to_print = self.__default_func(line)
-                    if to_print:
                         self.__write_obj.write(line)
         copy_obj = copy.Copy(bug_handler = self.__bug_handler)
         if self.__copy:

src/calibre/ebooks/rtf2xml/info.py

@@ -15,8 +15,10 @@
 #                                                                       #
 #                                                                       #
 #########################################################################
-import sys, os, tempfile
+import sys, os, tempfile, re
+
 from calibre.ebooks.rtf2xml import copy
+
 class Info:
     """
     Make tags for document-information
@@ -42,12 +44,14 @@ class Info:
         self.__copy = copy
         self.__run_level = run_level
         self.__write_to = tempfile.mktemp()
+
     def __initiate_values(self):
         """
         Initiate all values.
         """
         self.__text_string = ''
         self.__state = 'before_info_table'
+        self.rmspace = re.compile(r'\s+')
         self.__state_dict = {
         'before_info_table': self.__before_info_table_func,
         'after_info_table': self.__after_info_table_func,
@@ -58,27 +62,49 @@ class Info:
         self.__info_table_dict = {
         'cw<di<title_____'  : (self.__found_tag_with_text_func, 'title'),
         'cw<di<author____'  : (self.__found_tag_with_text_func, 'author'),
+        'cw<di<operator__'  : (self.__found_tag_with_text_func, 'operator'),
+        'cw<di<manager___'  : (self.__found_tag_with_text_func, 'manager'),
+        'cw<di<company___'  : (self.__found_tag_with_text_func, 'company'),
         'cw<di<keywords__'  : (self.__found_tag_with_text_func, 'keywords'),
+        'cw<di<category__'  : (self.__found_tag_with_text_func, 'category'),
         'cw<di<doc-notes_'  : (self.__found_tag_with_text_func, 'doc-notes'),
         'cw<di<subject___'  : (self.__found_tag_with_text_func, 'subject'),
-        'cw<di<operator__'  : (self.__found_tag_with_text_func, 'operator'),
+        'cw<di<linkbase__'  : (self.__found_tag_with_text_func, 'hyperlink-base'),
+        
         'cw<di<create-tim'  : (self.__found_tag_with_tokens_func, 'creation-time'),
-        'cw<di<revis-time'  :  (self.__found_tag_with_tokens_func, 'revision-time'),
+        'cw<di<revis-time'  : (self.__found_tag_with_tokens_func, 'revision-time'),
-        'cw<di<edit-time_'  : (self.__single_field_func, 'editing-time'),
+        'cw<di<edit-time_'  : (self.__found_tag_with_tokens_func, 'editing-time'),
+        'cw<di<print-time'  : (self.__found_tag_with_tokens_func, 'printing-time'),
+        'cw<di<backuptime'  : (self.__found_tag_with_tokens_func, 'backup-time'),
+        
         'cw<di<num-of-wor'  : (self.__single_field_func, 'number-of-words'),
         'cw<di<num-of-chr'  : (self.__single_field_func, 'number-of-characters'),
+        'cw<di<numofchrws'  : (self.__single_field_func, 'number-of-characters-without-space'),
         'cw<di<num-of-pag'  : (self.__single_field_func, 'number-of-pages'),
+        'cw<di<version___'  : (self.__single_field_func, 'version'),
+        'cw<di<intern-ver'  : (self.__single_field_func, 'internal-version-number'),
+        'cw<di<internalID'  : (self.__single_field_func, 'internal-id-number'),
         }
         self.__token_dict = {
         'year______'        : 'year',
         'month_____'        : 'month',
         'day_______'        : 'day',
         'minute____'        : 'minute',
+        'second____'        : 'second',
         'revis-time'        : 'revision-time',
+        'create-tim'        : 'creation-time',
+        'edit-time_'        : 'editing-time',
+        'print-time'        : 'printing-time',
+        'backuptime'        : 'backup-time',
         'num-of-wor'        : 'number-of-words',
         'num-of-chr'        : 'number-of-characters',
+        'numofchrws'        : 'number-of-characters-without-space',
         'num-of-pag'        : 'number-of-pages',
+        'version___'        : 'version',
+        'intern-ver'        : 'internal-version-number',
+        'internalID'        : 'internal-id-number',
         }
+
     def __before_info_table_func(self, line):
         """
         Required:
@@ -92,6 +118,7 @@ class Info:
         if self.__token_info == 'mi<mk<doc-in-beg':
             self.__state = 'in_info_table'
         self.__write_obj.write(line)
+
     def __in_info_table_func(self, line):
         """
         Requires:
@@ -112,6 +139,7 @@ class Info:
                 action(line, tag)
             else:
                 self.__write_obj.write(line)
+
     def __found_tag_with_text_func(self, line, tag):
         """
         Requires:
@@ -126,6 +154,7 @@ class Info:
         """
         self.__tag = tag
         self.__state = 'collect_text'
+
     def __collect_text_func(self, line):
         """
         Requires:
@@ -139,14 +168,17 @@ class Info:
         """
         if self.__token_info == 'mi<mk<docinf-end':
             self.__state = 'in_info_table'
-            self.__write_obj.write(
+            #Don't print empty tags
-                'mi<tg<open______<%s\n'
+            if len(self.rmspace.sub('',self.__text_string)):
-                'tx<nu<__________<%s\n'
+                self.__write_obj.write(
-                'mi<tg<close_____<%s\n' % (self.__tag, self.__text_string, self.__tag)
+                    'mi<tg<open______<%s\n'
-            )
+                    'tx<nu<__________<%s\n'
+                    'mi<tg<close_____<%s\n' % (self.__tag, self.__text_string, self.__tag)
+                )
             self.__text_string = ''
         elif line[0:2] == 'tx':
             self.__text_string += line[17:-1]
+
     def __found_tag_with_tokens_func(self, line, tag):
         """
         Requires:
@@ -163,6 +195,7 @@ class Info:
         self.__state = 'collect_tokens'
         self.__text_string = 'mi<tg<empty-att_<%s' % tag
         #mi<tg<empty-att_<page-definition<margin>33\n
+
     def __collect_tokens_func(self, line):
         """
         Requires:
@@ -194,18 +227,19 @@ class Info:
             att = line[6:16]
             value = line[20:-1]
             att_changed = self.__token_dict.get(att)
-            if att_changed == None:
+            if att_changed is None:
                 if self.__run_level > 3:
-                    msg = 'no dictionary match for %s\n' % att
+                    msg = 'No dictionary match for %s\n' % att
                     raise self.__bug_handler, msg
             else:
                 self.__text_string += '<%s>%s' % (att_changed, value)
+
     def __single_field_func(self, line, tag):
         value = line[20:-1]
         self.__write_obj.write(
-        'mi<tg<empty-att_<%s'
+        'mi<tg<empty-att_<%s<%s>%s\n' % (tag, tag, value)
-        '<%s>%s\n' % (tag, tag, value)
         )
+
     def __after_info_table_func(self, line):
         """
         Requires:
@@ -217,6 +251,7 @@ class Info:
             the file.
         """
         self.__write_obj.write(line)
+
     def fix_info(self):
         """
         Requires:
@@ -234,20 +269,15 @@ class Info:
             information table, simply write the line to the output file.
         """
         self.__initiate_values()
-        read_obj = open(self.__file, 'r')
+        with open(self.__file, 'r') as read_obj:
-        self.__write_obj = open(self.__write_to, 'w')
+            with open(self.__write_to, 'wb') as self.__write_obj:
-        line_to_read = 1
+                for line in read_obj:
-        while line_to_read:
+                    self.__token_info = line[:16]
-            line_to_read = read_obj.readline()
+                    action = self.__state_dict.get(self.__state)
-            line = line_to_read
+                    if action is None:
-            self.__token_info = line[:16]
+                        sys.stderr.write('No matching state in module styles.py\n')
-            action = self.__state_dict.get(self.__state)
+                        sys.stderr.write(self.__state + '\n')
-            if action == None:
+                    action(line)
-                sys.stderr.write('no no matching state in module styles.py\n')
-                sys.stderr.write(self.__state + '\n')
-            action(line)
-        read_obj.close()
-        self.__write_obj.close()
         copy_obj = copy.Copy(bug_handler = self.__bug_handler)
         if self.__copy:
             copy_obj.copy_file(self.__write_to, "info.data")

src/calibre/ebooks/rtf2xml/process_tokens.py

@@ -70,7 +70,7 @@ class ProcessTokens:
         ';'                  :	('mc', ';', self.ms_sub_func),
         # this must be wrong
         '-'                  :	('mc', '-', self.ms_sub_func),
-        'line'               :  ('mi', 'hardline-break', self.hardline_func), #calibre
+        'line'               :  ('mi', 'hardline-break', self.direct_conv_func), #calibre
         # misc => ml
         '*'                  :	('ml', 'asterisk__', self.default_func),
         ':'                  :	('ml', 'colon_____', self.default_func),
@@ -78,7 +78,6 @@ class ProcessTokens:
         'backslash'          :	('nu', '\\', self.text_func),
         'ob'                 :	('nu', '{', self.text_func),
         'cb'                 :	('nu', '}', self.text_func),
-        #'line'               :  ('nu', ' ', self.text_func), calibre
         # paragraph formatting => pf
         'page'               :  ('pf', 'page-break', self.default_func),
         'par'                :	('pf', 'par-end___', self.default_func),
@@ -231,11 +230,15 @@ class ProcessTokens:
         'trhdr'              :  ('tb', 'row-header', self.default_func),
         # preamble => pr
         # document information => di
+        # TODO integrate \userprops
         'info'               :	('di', 'doc-info__', self.default_func),
+        'title'              :	('di', 'title_____', self.default_func),
         'author'             :	('di', 'author____', self.default_func),
         'operator'           :	('di', 'operator__', self.default_func),
-        'title'              :	('di', 'title_____', self.default_func),
+        'manager'            :	('di', 'manager___', self.default_func),
+        'company'            :	('di', 'company___', self.default_func),
         'keywords'           :  ('di', 'keywords__', self.default_func),
+        'category'           :  ('di', 'category__', self.default_func),
         'doccomm'            :  ('di', 'doc-notes_', self.default_func),
         'comment'            :  ('di', 'doc-notes_', self.default_func),
         'subject'            :  ('di', 'subject___', self.default_func),
@@ -244,11 +247,19 @@ class ProcessTokens:
         'mo'                 :	('di', 'month_____', self.default_func),
         'dy'                 :	('di', 'day_______', self.default_func),
         'min'                :	('di', 'minute____', self.default_func),
+        'sec'                :	('di', 'second____', self.default_func),
         'revtim'             :	('di', 'revis-time', self.default_func),
+        'edmins'             :	('di', 'edit-time_', self.default_func),
+        'printim'            :	('di', 'print-time', self.default_func),
+        'buptim'             :	('di', 'backuptime', self.default_func),
         'nofwords'           :	('di', 'num-of-wor', self.default_func),
         'nofchars'           :	('di', 'num-of-chr', self.default_func),
+        'nofcharsws'         :	('di', 'numofchrws', self.default_func),
         'nofpages'           :	('di', 'num-of-pag', self.default_func),
-        'edmins'             :	('di', 'edit-time_', self.default_func),
+        'version'            :	('di', 'version___', self.default_func),
+        'vern'               :	('di', 'intern-ver', self.default_func),
+        'hlinkbase'          :	('di', 'linkbase__', self.default_func),
+        'id'                 :	('di', 'internalID', self.default_func),
         # headers and footers => hf
         'headerf'            :	('hf', 'head-first', self.default_func),
         'headerl'            :	('hf', 'head-left_', self.default_func),
@@ -605,7 +616,7 @@ class ProcessTokens:
     def ms_sub_func(self, pre, token, num):
         return 'tx<mc<__________<%s\n' % token
 
-    def hardline_func(self, pre, token, num):
+    def direct_conv_func(self, pre, token, num):
         return 'mi<tg<empty_____<%s\n' % token
 
     def default_func(self, pre, token, num):

src/calibre/ebooks/rtf2xml/tokenize.py

@@ -27,11 +27,13 @@ class Tokenize:
             bug_handler,
             copy = None,
             run_level = 1,
-    ):
+            # out_file = None,
+        ):
         self.__file = in_file
         self.__bug_handler = bug_handler
         self.__copy = copy
         self.__write_to = tempfile.mktemp()
+        # self.__out_file = out_file
         self.__compile_expressions()
         #variables
         self.__uc_char = 0
@@ -113,6 +115,8 @@ class Tokenize:
 
     def __sub_reg_split(self,input_file):
         input_file = self.__replace_spchar.mreplace(input_file)
+        # this is for older RTF
+        input_file = self.__par_exp.sub('\n\\par \n', input_file)
         input_file = self.__ms_hex_exp.sub("\\mshex0\g<1> ", input_file)
         input_file = self.__utf_ud.sub("\\{\\uc0 \g<1>\\}", input_file)
         #remove \n in bin data
@@ -153,8 +157,6 @@ class Tokenize:
             # put a backslash in front of to eliminate special cases and
             # make processing easier
             "}": "\\}",
-            # this is for older RTF
-            r'\\$': '\\par ',
             }
         self.__replace_spchar = MReplace(SIMPLE_RPL)
         #add ;? in case of char following \u
@@ -168,10 +170,12 @@ class Tokenize:
         #why keep backslash whereas \is replaced before?
         #remove \n from endline char
         self.__splitexp = re.compile(r"(\\[{}]|\n|\\[^\s\\{}&]+(?:[ \t\r\f\v])?)")
+        #this is for old RTF
+        self.__par_exp = re.compile(r'\\\n+')
+        # self.__par_exp = re.compile(r'\\$')
         #self.__bin_exp = re.compile(r"\\bin(-?\d{1,8}) {0,1}")
         #self.__utf_exp = re.compile(r"^\\u(-?\d{3,6})")
         #self.__splitexp = re.compile(r"(\\[\\{}]|{|}|\n|\\[^\s\\{}&]+(?:\s)?)")
-        #self.__par_exp = re.compile(r'\\$')
         #self.__remove_line = re.compile(r'\n+')
         #self.__mixed_exp = re.compile(r"(\\[a-zA-Z]+\d+)(\D+)")
         ##self.num_exp = re.compile(r"(\*|:|[a-zA-Z]+)(.*)")
@@ -199,7 +203,24 @@ class Tokenize:
         copy_obj = copy.Copy(bug_handler = self.__bug_handler)
         if self.__copy:
             copy_obj.copy_file(self.__write_to, "tokenize.data")
+        # if self.__out_file:
+            # self.__file = self.__out_file
         copy_obj.rename(self.__write_to, self.__file)
         os.remove(self.__write_to)
         
         #self.__special_tokens = [ '_', '~', "'", '{', '}' ]
+
+# import sys
+# def main(args=sys.argv):
+    # if len(args) < 1:
+        # print 'No file'
+        # return
+    # file = 'data_tokens.txt'
+    # if len(args) == 3:
+        # file = args[2]
+    # to = Tokenize(args[1], Exception, out_file = file)
+    # to.tokenize()
+
+
+# if __name__ == '__main__':
+    # sys.exit(main())

src/calibre/gui2/__init__.py

@@ -505,7 +505,7 @@ class FileDialog(QObject):
         self.selected_files = []
         if mode == QFileDialog.AnyFile:
             f = unicode(QFileDialog.getSaveFileName(parent, title, initial_dir, ftext, ""))
-            if f and os.path.exists(f):
+            if f:
                 self.selected_files.append(f)
         elif mode == QFileDialog.ExistingFile:
             f = unicode(QFileDialog.getOpenFileName(parent, title, initial_dir, ftext, ""))

src/calibre/gui2/actions/__init__.py

@@ -111,7 +111,10 @@ class InterfaceAction(QObject):
         action.setWhatsThis(text)
         action.setAutoRepeat(False)
         if shortcut:
-            action.setShortcut(shortcut)
+            if isinstance(shortcut, list):
+                action.setShortcuts(shortcut)
+            else:
+                action.setShortcut(shortcut)
         setattr(self, attr, action)
         return action
 
@@ -170,6 +173,14 @@ class InterfaceAction(QObject):
         '''
         pass
 
+    def gui_layout_complete(self):
+        '''
+        Called once per action when the layout of the main GUI is
+        completed. If your action needs to make changes to the layout, they
+        should be done here, rather than in :meth:`initialization_complete`.
+        '''
+        pass
+
     def initialization_complete(self):
         '''
         Called once per action when the initialization of the main GUI is

src/calibre/gui2/actions/catalog.py

@@ -28,7 +28,7 @@ class GenerateCatalogAction(InterfaceAction):
 
         if not ids:
             return error_dialog(self.gui, _('No books selected'),
-                    _('No books selected to generate catalog for'),
+                    _('No books selected for catalog generation'),
                     show=True)
 
 		db = self.gui.library_view.model().db
@@ -55,9 +55,9 @@ class GenerateCatalogAction(InterfaceAction):
 
     def catalog_generated(self, job):
         if job.result:
-            # Search terms nulled catalog results
+            # Error during catalog generation
-            return error_dialog(self.gui, _('No books found'),
+            return error_dialog(self.gui, _('Catalog generation terminated'),
-                    _("No books to catalog\nCheck exclusion criteria"),
+                    job.result,
                     show=True)
         if job.failed:
             return self.gui.job_exception(job)

src/calibre/gui2/actions/edit_metadata.py

@@ -94,12 +94,12 @@ class EditMetadataAction(InterfaceAction):
             get_social_metadata = config['get_social_metadata']
         else:
             get_social_metadata = set_social_metadata
-        from calibre.gui2.metadata import DoDownload
+        from calibre.gui2.metadata.bulk_download import DoDownload
         if set_social_metadata is not None and set_social_metadata:
             x = _('social metadata')
         else:
             x = _('covers') if covers and not set_metadata else _('metadata')
-        title = _('Downloading %s for %d book(s)')%(x, len(ids))
+        title = _('Downloading {0} for {1} book(s)').format(x, len(ids))
         self._download_book_metadata = DoDownload(self.gui, title, db, ids,
                 get_covers=covers, set_metadata=set_metadata,
                 get_social_metadata=get_social_metadata)

src/calibre/gui2/actions/next_match.py

@@ -0,0 +1,56 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+
+__license__   = 'GPL v3'
+__copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+from calibre.gui2.actions import InterfaceAction
+
+class NextMatchAction(InterfaceAction):
+    name = 'Move to next highlighted book'
+    action_spec = (_('Move to next match'), 'arrow-down.png',
+            _('Move to next highlighted match'), [_('N'), _('F3')])
+    dont_add_to = frozenset(['toolbar-device', 'context-menu-device'])
+    action_type = 'current'
+
+    def genesis(self):
+        '''
+        Setup this plugin. Only called once during initialization. self.gui is
+        available. The action secified by :attr:`action_spec` is available as
+        ``self.qaction``.
+        '''
+        self.can_move = None
+        self.qaction.triggered.connect(self.move_forward)
+        self.create_action(spec=(_('Move to previous item'), 'arrow-up.png',
+              _('Move to previous highlighted item'), [_('Shift+N'),
+                  _('Shift+F3')]), attr='p_action')
+        self.gui.addAction(self.p_action)
+        self.p_action.triggered.connect(self.move_backward)
+
+    def gui_layout_complete(self):
+        self.gui.search_highlight_only.setVisible(True)
+
+    def location_selected(self, loc):
+        self.can_move = loc == 'library'
+        try:
+            self.gui.search_highlight_only.setVisible(self.can_move)
+        except:
+            import traceback
+            traceback.print_exc()
+
+    def move_forward(self):
+        if self.can_move is None:
+            self.can_move = self.gui.current_view() is self.gui.library_view
+            self.gui.search_highlight_only.setVisible(self.can_move)
+
+        if self.can_move:
+            self.gui.current_view().move_highlighted_row(forward=True)
+
+    def move_backward(self):
+        if self.can_move is None:
+            self.can_move = self.gui.current_view() is self.gui.library_view
+            self.gui.search_highlight_only.setVisible(self.can_move)
+
+        if self.can_move:
+            self.gui.current_view().move_highlighted_row(forward=False)

src/calibre/gui2/actions/view.py

@@ -92,7 +92,12 @@ class ViewAction(InterfaceAction):
         formats = [list(f.upper().split(',')) if f else None for f in formats]
         all_fmts = set([])
         for x in formats:
-            for f in x: all_fmts.add(f)
+            if x:
+                for f in x: all_fmts.add(f)
+        if not all_fmts:
+            error_dialog(self.gui,  _('Format unavailable'),
+                    _('Selected books have no formats'), show=True)
+            return
         d = ChooseFormatDialog(self.gui, _('Choose the format to view'),
                 list(sorted(all_fmts)))
         if d.exec_() == d.Accepted:

src/calibre/gui2/catalog/catalog_epub_mobi.py

@@ -6,67 +6,18 @@ __license__   = 'GPL v3'
 __copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
-
 from calibre.ebooks.conversion.config import load_defaults
 from calibre.gui2 import gprefs
 
 from catalog_epub_mobi_ui import Ui_Form
-from PyQt4.Qt import QWidget, QLineEdit
+from PyQt4.Qt import QCheckBox, QComboBox, QDoubleSpinBox, QLineEdit, \
+                     QRadioButton, QWidget
 
 class PluginWidget(QWidget,Ui_Form):
 
     TITLE = _('E-book options')
     HELP  = _('Options specific to')+' EPUB/MOBI '+_('output')
 
-    CheckBoxControls = [
-                        'generate_titles',
-                        'generate_series',
-                        'generate_genres',
-                        'generate_recently_added',
-                        'generate_descriptions',
-                        'include_hr'
-                        ]
-    ComboBoxControls = [
-                        'read_source_field',
-                        'exclude_source_field',
-                        'header_note_source_field',
-                        'merge_source_field'
-                        ]
-    LineEditControls = [
-                        'exclude_genre',
-                        'exclude_pattern',
-                        'exclude_tags',
-                        'read_pattern',
-                        'wishlist_tag'
-                        ]
-    RadioButtonControls = [
-                        'merge_before',
-                        'merge_after'
-                        ]
-    SpinBoxControls = [
-                        'thumb_width'
-                        ]
-
-    OPTION_FIELDS = zip(CheckBoxControls,
-                        [True for i in CheckBoxControls],
-                        ['check_box' for i in CheckBoxControls])
-    OPTION_FIELDS += zip(ComboBoxControls,
-                        [None for i in ComboBoxControls],
-                        ['combo_box' for i in ComboBoxControls])
-    OPTION_FIELDS += zip(RadioButtonControls,
-                        [None for i in RadioButtonControls],
-                        ['radio_button' for i in RadioButtonControls])
-
-    # LineEditControls
-    OPTION_FIELDS += zip(['exclude_genre'],['\[.+\]'],['line_edit'])
-    OPTION_FIELDS += zip(['exclude_pattern'],[None],['line_edit'])
-    OPTION_FIELDS += zip(['exclude_tags'],['~,'+_('Catalog')],['line_edit'])
-    OPTION_FIELDS += zip(['read_pattern'],['+'],['line_edit'])
-    OPTION_FIELDS += zip(['wishlist_tag'],['Wishlist'],['line_edit'])
-
-    # SpinBoxControls
-    OPTION_FIELDS += zip(['thumb_width'],[1.00],['spin_box'])
-
     # Output synced to the connected device?
     sync_enabled = True
 
@@ -76,8 +27,69 @@ class PluginWidget(QWidget,Ui_Form):
     def __init__(self, parent=None):
         QWidget.__init__(self, parent)
         self.setupUi(self)
+        self._initControlArrays()
+
+    def _initControlArrays(self):
+
+        CheckBoxControls = []
+        ComboBoxControls = []
+        DoubleSpinBoxControls = []
+        LineEditControls = []
+        RadioButtonControls = []
+
+        for item in self.__dict__:
+            if type(self.__dict__[item]) is QCheckBox:
+                CheckBoxControls.append(str(self.__dict__[item].objectName()))
+            elif type(self.__dict__[item]) is QComboBox:
+                ComboBoxControls.append(str(self.__dict__[item].objectName()))
+            elif type(self.__dict__[item]) is QDoubleSpinBox:
+                DoubleSpinBoxControls.append(str(self.__dict__[item].objectName()))
+            elif type(self.__dict__[item]) is QLineEdit:
+                LineEditControls.append(str(self.__dict__[item].objectName()))
+            elif type(self.__dict__[item]) is QRadioButton:
+                RadioButtonControls.append(str(self.__dict__[item].objectName()))
+
+        option_fields = zip(CheckBoxControls,
+                            [True for i in CheckBoxControls],
+                            ['check_box' for i in CheckBoxControls])
+        option_fields += zip(ComboBoxControls,
+                            [None for i in ComboBoxControls],
+                            ['combo_box' for i in ComboBoxControls])
+        option_fields += zip(RadioButtonControls,
+                            [None for i in RadioButtonControls],
+                            ['radio_button' for i in RadioButtonControls])
+
+        # LineEditControls
+        option_fields += zip(['exclude_genre'],['\[.+\]'],['line_edit'])
+        option_fields += zip(['exclude_pattern'],[None],['line_edit'])
+        option_fields += zip(['exclude_tags'],['~,'+_('Catalog')],['line_edit'])
+        option_fields += zip(['read_pattern'],['+'],['line_edit'])
+        option_fields += zip(['wishlist_tag'],['Wishlist'],['line_edit'])
+
+        # SpinBoxControls
+        option_fields += zip(['thumb_width'],[1.00],['spin_box'])
+
+        self.OPTION_FIELDS = option_fields
 
     def initialize(self, name, db):
+        '''
+
+        CheckBoxControls (c_type: check_box):
+            ['generate_titles','generate_series','generate_genres',
+                            'generate_recently_added','generate_descriptions','include_hr']
+        ComboBoxControls (c_type: combo_box):
+            ['read_source_field','exclude_source_field','header_note_source_field',
+                            'merge_source_field']
+        LineEditControls (c_type: line_edit):
+            ['exclude_genre','exclude_pattern','exclude_tags','read_pattern',
+                            'wishlist_tag']
+        RadioButtonControls (c_type: radio_button):
+            ['merge_before','merge_after']
+        SpinBoxControls (c_type: spin_box):
+            ['thumb_width']
+
+        '''
+
         self.name = name
         self.db = db
         self.populateComboBoxes()
@@ -135,7 +147,7 @@ class PluginWidget(QWidget,Ui_Form):
     def options(self):
         # Save/return the current options
         # exclude_genre stores literally
-        # generate_titles, generate_recently_added, numbers_as_text stores as True/False
+        # generate_titles, generate_recently_added store as True/False
         # others store as lists
 
         opts_dict = {}

src/calibre/gui2/catalog/catalog_epub_mobi.ui

@@ -7,7 +7,7 @@
     <x>0</x>
     <y>0</y>
     <width>650</width>
-    <height>582</height>
+    <height>596</height>
    </rect>
   </property>
   <property name="sizePolicy">
@@ -41,41 +41,54 @@
       <string>Included sections</string>
      </property>
      <layout class="QGridLayout" name="gridLayout_2">
-      <item row="0" column="0">
+      <item row="0" column="1">
-       <widget class="QCheckBox" name="generate_titles">
-        <property name="text">
-         <string>Books by &amp;Title</string>
-        </property>
-       </widget>
-      </item>
-      <item row="4" column="0">
-       <widget class="QCheckBox" name="generate_series">
-        <property name="text">
-         <string>Books by &amp;Series</string>
-        </property>
-       </widget>
-      </item>
-      <item row="0" column="2">
-       <widget class="QCheckBox" name="generate_recently_added">
-        <property name="text">
-         <string>Recently &amp;Added</string>
-        </property>
-       </widget>
-      </item>
-      <item row="5" column="0">
        <widget class="QCheckBox" name="generate_genres">
         <property name="text">
          <string>Books by &amp;Genre</string>
         </property>
        </widget>
       </item>
-      <item row="4" column="2">
+      <item row="4" column="1">
+       <widget class="QCheckBox" name="generate_recently_added">
+        <property name="text">
+         <string>Recently &amp;Added</string>
+        </property>
+       </widget>
+      </item>
+      <item row="5" column="1">
        <widget class="QCheckBox" name="generate_descriptions">
         <property name="text">
          <string>&amp;Descriptions</string>
         </property>
        </widget>
       </item>
+      <item row="5" column="0">
+       <widget class="QCheckBox" name="generate_series">
+        <property name="text">
+         <string>Books by &amp;Series</string>
+        </property>
+       </widget>
+      </item>
+      <item row="4" column="0">
+       <widget class="QCheckBox" name="generate_titles">
+        <property name="text">
+         <string>Books by &amp;Title</string>
+        </property>
+       </widget>
+      </item>
+      <item row="0" column="0">
+       <widget class="QCheckBox" name="generate_authors">
+        <property name="enabled">
+         <bool>false</bool>
+        </property>
+        <property name="text">
+         <string>Books by Author</string>
+        </property>
+        <property name="checked">
+         <bool>true</bool>
+        </property>
+       </widget>
+      </item>
      </layout>
     </widget>
    </item>
@@ -94,14 +107,10 @@
       </size>
      </property>
      <property name="toolTip">
-      <string>&lt;!DOCTYPE HTML PUBLIC &quot;-//W3C//DTD HTML 4.0//EN&quot; &quot;http://www.w3.org/TR/REC-html40/strict.dtd&quot;&gt;
+      <string>&lt;p&gt;Default pattern 
-&lt;html&gt;&lt;head&gt;&lt;meta name=&quot;qrichtext&quot; content=&quot;1&quot; /&gt;&lt;style type=&quot;text/css&quot;&gt;
+\[.+\]
-p, li { white-space: pre-wrap; }
+excludes tags of the form [tag], 
-&lt;/style&gt;&lt;/head&gt;&lt;body style=&quot; font-family:'Lucida Grande'; font-size:13pt; font-weight:400; font-style:normal;&quot;&gt;
+e.g., [Project Gutenberg]&lt;/p&gt;</string>
-&lt;p style=&quot; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;Default pattern &lt;/p&gt;
-&lt;p style=&quot; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;&lt;span style=&quot; font-family:'Courier New,courier';&quot;&gt;\[.+\]&lt;/span&gt;&lt;/p&gt;
-&lt;p style=&quot; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;excludes tags of the form [&lt;span style=&quot; font-family:'Courier New,courier';&quot;&gt;tag&lt;/span&gt;], &lt;/p&gt;
-&lt;p style=&quot; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;e.g., [Project Gutenberg]&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</string>
      </property>
      <property name="title">
       <string>Excluded genres</string>
@@ -239,12 +248,8 @@ p, li { white-space: pre-wrap; }
            </size>
           </property>
           <property name="toolTip">
-           <string>&lt;!DOCTYPE HTML PUBLIC &quot;-//W3C//DTD HTML 4.0//EN&quot; &quot;http://www.w3.org/TR/REC-html40/strict.dtd&quot;&gt;
+           <string>&lt;p&gt;Comma-separated list of tags to exclude.
-&lt;html&gt;&lt;head&gt;&lt;meta name=&quot;qrichtext&quot; content=&quot;1&quot; /&gt;&lt;style type=&quot;text/css&quot;&gt;
+Default: ~,Catalog</string>
-p, li { white-space: pre-wrap; }
-&lt;/style&gt;&lt;/head&gt;&lt;body style=&quot; font-family:'Lucida Grande'; font-size:13pt; font-weight:400; font-style:normal;&quot;&gt;
-&lt;p style=&quot; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;&lt;span style=&quot; font-size:12pt;&quot;&gt;Comma-separated list of tags to exclude.&lt;/span&gt;&lt;/p&gt;
-&lt;p style=&quot; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;&lt;span style=&quot; font-size:12pt;&quot;&gt;Default:&lt;/span&gt;&lt;span style=&quot; font-family:'Courier New,courier'; font-size:12pt;&quot;&gt; ~,Catalog&lt;/span&gt;&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</string>
           </property>
          </widget>
         </item>
@@ -332,7 +337,7 @@ p, li { white-space: pre-wrap; }
       </size>
      </property>
      <property name="toolTip">
-      <string>Matching books will be displayed with ✓</string>
+      <string>Matching books will be displayed with a check mark</string>
      </property>
      <property name="title">
       <string>Read books</string>
@@ -471,7 +476,7 @@ p, li { white-space: pre-wrap; }
         <item>
          <widget class="QLineEdit" name="wishlist_tag">
           <property name="toolTip">
-           <string>Books tagged as Wishlist items will be displayed with ✕</string>
+           <string>Books tagged as Wishlist items will be displayed with an X</string>
           </property>
          </widget>
         </item>

src/calibre/gui2/dialogs/drm_error.ui

@@ -13,6 +13,10 @@
   <property name="windowTitle">
    <string>This book is DRMed</string>
   </property>
+  <property name="windowIcon">
+   <iconset resource="../../../../resources/images.qrc">
+    <normaloff>:/images/document-encrypt.png</normaloff>:/images/document-encrypt.png</iconset>
+  </property>
   <layout class="QGridLayout" name="gridLayout">
    <item row="0" column="0">
     <widget class="QLabel" name="label">

src/calibre/gui2/dialogs/metadata_bulk.ui

@@ -75,13 +75,31 @@
              </property>
             </widget>
            </item>
-           <item row="1" column="1">
+           <item row="0" column="1">
-            <widget class="QCheckBox" name="auto_author_sort">
+            <widget class="EnComboBox" name="authors">
-             <property name="text">
+             <property name="editable">
-              <string>A&amp;utomatically set author sort</string>
+              <bool>true</bool>
              </property>
             </widget>
            </item>
+           <item row="1" column="1">
+            <layout class="QHBoxLayout" name="horizontalLayout_2">
+             <item>
+              <widget class="QCheckBox" name="auto_author_sort">
+               <property name="text">
+                <string>A&amp;utomatically set author sort</string>
+               </property>
+              </widget>
+             </item>
+             <item>
+              <widget class="QCheckBox" name="swap_title_and_author">
+               <property name="text">
+                <string>&amp;Swap title and author</string>
+               </property>
+              </widget>
+             </item>
+            </layout>
+           </item>
            <item row="2" column="0">
             <widget class="QLabel" name="label_8">
              <property name="text">
@@ -95,7 +113,7 @@
              </property>
             </widget>
            </item>
-           <item row="2" column="1" colspan="2">
+           <item row="2" column="1">
             <widget class="EnLineEdit" name="author_sort">
              <property name="toolTip">
               <string>Specify how the author(s) of this book should be sorted. For example Charles Dickens should be sorted as Dickens, Charles.</string>
@@ -115,7 +133,7 @@
              </property>
             </widget>
            </item>
-           <item row="3" column="1" colspan="2">
+           <item row="3" column="1">
             <widget class="QSpinBox" name="rating">
              <property name="toolTip">
               <string>Rating of this book. 0-5 stars</string>
@@ -156,7 +174,7 @@
              </property>
             </widget>
            </item>
-           <item row="4" column="1" colspan="2">
+           <item row="4" column="1">
             <widget class="EnComboBox" name="publisher">
              <property name="editable">
               <bool>true</bool>
@@ -220,7 +238,7 @@
               <string>Check this box to remove all tags from the books.</string>
              </property>
              <property name="text">
-              <string>Remove all</string>
+              <string>Remove &amp;all</string>
              </property>
             </widget>
            </item>
@@ -241,52 +259,35 @@
             </widget>
            </item>
            <item row="7" column="1">
-            <layout class="QHBoxLayout" name="HLayout_34">
+            <widget class="EnComboBox" name="series">
-             <item>
+             <property name="toolTip">
-              <widget class="EnComboBox" name="series">
+              <string>List of known series. You can add new series.</string>
-               <property name="toolTip">
+             </property>
-                <string>List of known series. You can add new series.</string>
+             <property name="whatsThis">
-               </property>
+              <string>List of known series. You can add new series.</string>
-               <property name="whatsThis">
+             </property>
-                <string>List of known series. You can add new series.</string>
+             <property name="editable">
-               </property>
+              <bool>true</bool>
-               <property name="editable">
+             </property>
-                <bool>true</bool>
+             <property name="insertPolicy">
-               </property>
+              <enum>QComboBox::InsertAlphabetically</enum>
-               <property name="insertPolicy">
+             </property>
-                <enum>QComboBox::InsertAlphabetically</enum>
+             <property name="sizeAdjustPolicy">
-               </property>
+              <enum>QComboBox::AdjustToContents</enum>
-               <property name="sizeAdjustPolicy">
+             </property>
-                <enum>QComboBox::AdjustToContents</enum>
+            </widget>
-               </property>
-              </widget>
-             </item>
-             <item>
-              <widget class="QCheckBox" name="clear_series">
-               <property name="toolTip">
-                <string>If checked, the series will be cleared</string>
-               </property>
-               <property name="text">
-                <string>Clear series</string>
-               </property>
-              </widget>
-             </item>
-             <item>
-              <spacer name="HSpacer_344">
-               <property name="orientation">
-                <enum>Qt::Horizontal</enum>
-               </property>
-               <property name="sizeHint" stdset="0">
-                <size>
-                 <width>20</width>
-                 <height>0</height>
-                </size>
-               </property>
-              </spacer>
-             </item>
-            </layout>
            </item>
-           <item row="8" column="1" colspan="2">
+           <item row="7" column="2">
+            <widget class="QCheckBox" name="clear_series">
+             <property name="toolTip">
+              <string>If checked, the series will be cleared</string>
+             </property>
+             <property name="text">
+              <string>&amp;Clear series</string>
+             </property>
+            </widget>
+           </item>
+           <item row="8" column="1">
             <layout class="QHBoxLayout" name="HLayout_3">
              <item>
               <widget class="QCheckBox" name="autonumber_series">
@@ -297,7 +298,7 @@ you selected them. So if you selected Book A and then Book B,
 Book A will have series number 1 and Book B series number 2.</string>
                </property>
                <property name="text">
-                <string>Automatically number books in this series</string>
+                <string>&amp;Automatically number books in this series</string>
                </property>
               </widget>
              </item>
@@ -312,7 +313,7 @@ for that series. Checking this box will tell calibre to start numbering
 from the value in the box</string>
                </property>
                <property name="text">
-                <string>Force numbers to start with </string>
+                <string>&amp;Force numbers to start with:</string>
                </property>
               </widget>
              </item>
@@ -332,19 +333,6 @@ from the value in the box</string>
                </property>
               </widget>
              </item>
-             <item>
-              <spacer name="HSpacer_34">
-               <property name="orientation">
-                <enum>Qt::Horizontal</enum>
-               </property>
-               <property name="sizeHint" stdset="0">
-                <size>
-                 <width>20</width>
-                 <height>10</height>
-                </size>
-               </property>
-              </spacer>
-             </item>
             </layout>
            </item>
            <item row="9" column="0">
@@ -358,59 +346,56 @@ from the value in the box</string>
             </widget>
            </item>
            <item row="9" column="1">
-            <widget class="QComboBox" name="remove_format"/>
+            <widget class="QComboBox" name="remove_format">
-           </item>
+             <property name="maximumSize">
-           <item row="0" column="1">
-            <widget class="EnComboBox" name="authors">
-             <property name="editable">
-              <bool>true</bool>
-             </property>
-            </widget>
-           </item>
-           <item row="11" column="0" colspan="2">
-            <widget class="QCheckBox" name="swap_title_and_author">
-             <property name="text">
-              <string>&amp;Swap title and author</string>
-             </property>
-            </widget>
-           </item>
-           <item row="12" column="0" colspan="2">
-            <widget class="QCheckBox" name="change_title_to_title_case">
-             <property name="toolTip">
-              <string>Force the title to be in title case. If both this and swap authors are checked,
-title and author are swapped before the title case is set</string>
-             </property>
-             <property name="text">
-              <string>Change title to title case</string>
-             </property>
-            </widget>
-           </item>
-           <item row="10" column="0" colspan="2">
-            <widget class="QCheckBox" name="remove_conversion_settings">
-             <property name="toolTip">
-              <string>Remove stored conversion settings for the selected books.
-
-Future conversion of these books will use the default settings.</string>
-             </property>
-             <property name="text">
-              <string>Remove &amp;stored conversion settings for the selected books</string>
-             </property>
-            </widget>
-           </item>
-           <item row="14" column="0" colspan="3">
-            <spacer name="verticalSpacer_2">
-             <property name="orientation">
-              <enum>Qt::Vertical</enum>
-             </property>
-             <property name="sizeHint" stdset="0">
               <size>
-               <width>20</width>
+               <width>120</width>
-               <height>40</height>
+               <height>16777215</height>
               </size>
              </property>
-            </spacer>
+            </widget>
            </item>
-           <item row="13" column="0" colspan="3">
+           <item row="10" column="0" colspan="3">
+            <layout class="QHBoxLayout" name="horizontalLayout_3">
+             <item>
+              <widget class="QCheckBox" name="change_title_to_title_case">
+               <property name="toolTip">
+                <string>Force the title to be in title case. If both this and swap authors are checked,
+title and author are swapped before the title case is set</string>
+               </property>
+               <property name="text">
+                <string>Change title to title &amp;case</string>
+               </property>
+              </widget>
+             </item>
+             <item>
+              <spacer name="horizontalSpacer">
+               <property name="orientation">
+                <enum>Qt::Horizontal</enum>
+               </property>
+               <property name="sizeHint" stdset="0">
+                <size>
+                 <width>40</width>
+                 <height>20</height>
+                </size>
+               </property>
+              </spacer>
+             </item>
+             <item>
+              <widget class="QCheckBox" name="remove_conversion_settings">
+               <property name="toolTip">
+                <string>Remove stored conversion settings for the selected books.
+
+Future conversion of these books will use the default settings.</string>
+               </property>
+               <property name="text">
+                <string>Remove &amp;stored conversion settings for the selected books</string>
+               </property>
+              </widget>
+             </item>
+            </layout>
+           </item>
+           <item row="11" column="0" colspan="3">
             <widget class="QGroupBox" name="groupBox">
              <property name="title">
               <string>Change &amp;cover</string>
@@ -440,6 +425,19 @@ Future conversion of these books will use the default settings.</string>
              </layout>
             </widget>
            </item>
+           <item row="12" column="0">
+            <spacer name="verticalSpacer_2">
+             <property name="orientation">
+              <enum>Qt::Vertical</enum>
+             </property>
+             <property name="sizeHint" stdset="0">
+              <size>
+               <width>20</width>
+               <height>40</height>
+              </size>
+             </property>
+            </spacer>
+           </item>
           </layout>
          </widget>
          <widget class="QWidget" name="tab">
@@ -902,14 +900,10 @@ not multiple and the destination field is multiple</string>
   <tabstop>remove_tags</tabstop>
   <tabstop>remove_all_tags</tabstop>
   <tabstop>series</tabstop>
-  <tabstop>clear_series</tabstop>
   <tabstop>autonumber_series</tabstop>
   <tabstop>series_numbering_restarts</tabstop>
   <tabstop>series_start_number</tabstop>
   <tabstop>remove_format</tabstop>
-  <tabstop>remove_conversion_settings</tabstop>
-  <tabstop>swap_title_and_author</tabstop>
-  <tabstop>change_title_to_title_case</tabstop>
   <tabstop>button_box</tabstop>
   <tabstop>search_field</tabstop>
   <tabstop>search_mode</tabstop>

src/calibre/gui2/dialogs/metadata_single.py

@@ -790,7 +790,13 @@ class MetadataSingleDialog(ResizableDialog, Ui_MetadataSingleDialog):
                         if d.opt_get_social_metadata.isChecked():
                             d2 = SocialMetadata(book, self)
                             d2.exec_()
-                            if d2.exceptions:
+                            if d2.timed_out:
+                                warning_dialog(self, _('Timed out'),
+                                    _('The download of social'
+                                    ' metadata timed out, the servers are'
+                                    ' probably busy. Try again later.'),
+                                    show=True)
+                            elif d2.exceptions:
                                 det = '\n'.join([x[0]+'\n\n'+x[-1]+'\n\n\n' for
                                     x in d2.exceptions])
                                 warning_dialog(self, _('There were errors'),

src/calibre/gui2/dialogs/template_dialog.py

@@ -3,8 +3,11 @@ __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __license__   = 'GPL v3'
 
+import json
+
 from PyQt4.Qt import Qt, QDialog, QDialogButtonBox
 from calibre.gui2.dialogs.template_dialog_ui import Ui_TemplateDialog
+from calibre.utils.formatter_functions import formatter_functions
 
 class TemplateDialog(QDialog, Ui_TemplateDialog):
 
@@ -17,9 +20,41 @@ class TemplateDialog(QDialog, Ui_TemplateDialog):
         self.setWindowFlags(self.windowFlags()&(~Qt.WindowContextHelpButtonHint))
         self.setWindowIcon(icon)
 
+        self.textbox.setTabStopWidth(10)
+        self.source_code.setTabStopWidth(10)
+        self.documentation.setReadOnly(True)
+        self.source_code.setReadOnly(True)
+
         if text is not None:
             self.textbox.setPlainText(text)
-        self.textbox.setTabStopWidth(50)
         self.buttonBox.button(QDialogButtonBox.Ok).setText(_('&OK'))
         self.buttonBox.button(QDialogButtonBox.Cancel).setText(_('&Cancel'))
 
+        try:
+            with open(P('template-functions.json'), 'rb') as f:
+                self.builtin_source_dict = json.load(f, encoding='utf-8')
+        except:
+            self.builtin_source_dict = {}
+
+        self.funcs = formatter_functions.get_functions()
+        self.builtins = formatter_functions.get_builtins()
+
+        func_names = sorted(self.funcs)
+        self.function.clear()
+        self.function.addItem('')
+        self.function.addItems(func_names)
+        self.function.setCurrentIndex(0)
+        self.function.currentIndexChanged[str].connect(self.function_changed)
+
+    def function_changed(self, toWhat):
+        name = unicode(toWhat)
+        self.source_code.clear()
+        self.documentation.clear()
+        if name in self.funcs:
+            self.documentation.setPlainText(self.funcs[name].doc)
+            if name in self.builtins:
+                if name in self.builtin_source_dict:
+                    self.source_code.setPlainText(self.builtin_source_dict[name])
+            else:
+                self.source_code.setPlainText(self.funcs[name].program_text)
+

src/calibre/gui2/dialogs/template_dialog.ui

@@ -6,8 +6,8 @@
    <rect>
     <x>0</x>
     <y>0</y>
-    <width>500</width>
+    <width>588</width>
-    <height>235</height>
+    <height>546</height>
    </rect>
   </property>
   <property name="sizePolicy">
@@ -19,21 +19,77 @@
   <property name="windowTitle">
    <string>Edit Comments</string>
   </property>
-   <layout class="QVBoxLayout" name="verticalLayout">
+  <layout class="QVBoxLayout" name="verticalLayout">
-    <item>
+   <item>
-     <widget class="QPlainTextEdit" name="textbox"/>
+    <widget class="QPlainTextEdit" name="textbox"/>
-    </item>
+   </item>
-    <item>
+   <item>
-     <widget class="QDialogButtonBox" name="buttonBox">
+    <widget class="QDialogButtonBox" name="buttonBox">
-      <property name="orientation">
+     <property name="orientation">
-       <enum>Qt::Horizontal</enum>
+      <enum>Qt::Horizontal</enum>
-      </property>
+     </property>
-      <property name="standardButtons">
+     <property name="standardButtons">
-       <set>QDialogButtonBox::Cancel|QDialogButtonBox::Ok</set>
+      <set>QDialogButtonBox::Cancel|QDialogButtonBox::Ok</set>
-      </property>
+     </property>
-     </widget>
+    </widget>
-    </item>
+   </item>
-   </layout>
+   <item>
+    <layout class="QGridLayout" name="gridLayout">
+     <item row="0" column="0">
+      <widget class="QLabel" name="label">
+       <property name="text">
+        <string>Function &amp;name:</string>
+       </property>
+       <property name="buddy">
+        <cstring>function</cstring>
+       </property>
+      </widget>
+     </item>
+     <item row="0" column="1">
+      <widget class="QComboBox" name="function"/>
+     </item>
+     <item row="1" column="0">
+      <widget class="QLabel" name="label_2">
+       <property name="text">
+        <string>&amp;Documentation:</string>
+       </property>
+       <property name="alignment">
+        <set>Qt::AlignLeading|Qt::AlignLeft|Qt::AlignTop</set>
+       </property>
+       <property name="buddy">
+        <cstring>documentation</cstring>
+       </property>
+      </widget>
+     </item>
+     <item row="2" column="0">
+      <widget class="QLabel" name="label_3">
+       <property name="text">
+        <string>Python &amp;code:</string>
+       </property>
+       <property name="alignment">
+        <set>Qt::AlignLeading|Qt::AlignLeft|Qt::AlignTop</set>
+       </property>
+       <property name="buddy">
+        <cstring>source_code</cstring>
+       </property>
+      </widget>
+     </item>
+     <item row="1" column="1">
+      <widget class="QPlainTextEdit" name="documentation">
+       <property name="maximumSize">
+        <size>
+         <width>16777215</width>
+         <height>75</height>
+        </size>
+       </property>
+      </widget>
+     </item>
+     <item row="2" column="1">
+      <widget class="QPlainTextEdit" name="source_code"/>
+     </item>
+    </layout>
+   </item>
+  </layout>
  </widget>
  <resources/>
  <connections>

src/calibre/gui2/layout.py

@@ -196,9 +196,11 @@ class SearchBar(QWidget): # {{{
 
         x = parent.search_highlight_only = QCheckBox()
         x.setText(_('&Highlight'))
-        x.setToolTip(_('Highlight matched books in the book list, instead '
+        x.setToolTip('<p>'+_('When searching, highlight matched books, instead '
-            'of restricting the book list to the matches.'))
+            'of restricting the book list to the matches.<p> You can use the '
+            'N or F3 keys to go to the next match.'))
         l.addWidget(x)
+        x.setVisible(False)
 
         x = parent.saved_search = SavedSearchBox(self)
         x.setMaximumSize(QSize(150, 16777215))

src/calibre/gui2/library/models.py

@@ -93,8 +93,9 @@ class BooksModel(QAbstractTableModel): # {{{
         self.bool_no_icon = QIcon(I('list_remove.png'))
         self.bool_blank_icon = QIcon(I('blank.png'))
         self.device_connected = False
-        self.rows_matching = set()
+        self.ids_to_highlight = []
-        self.lowest_row_matching = None
+        self.ids_to_highlight_set = set()
+        self.current_highlighted_idx = None
         self.highlight_only = False
         self.read_config()
 
@@ -130,6 +131,9 @@ class BooksModel(QAbstractTableModel): # {{{
         self.book_on_device = func
 
     def set_database(self, db):
+        self.ids_to_highlight = []
+        self.ids_to_highlight_set = set()
+        self.current_highlighted_idx = None
         self.db = db
         self.custom_columns = self.db.field_metadata.custom_field_metadata()
         self.column_map = list(self.orig_headers.keys()) + \
@@ -237,21 +241,55 @@ class BooksModel(QAbstractTableModel): # {{{
         if self.last_search:
             self.research()
 
+    def get_current_highlighted_id(self):
+        if len(self.ids_to_highlight) == 0 or self.current_highlighted_idx is None:
+            return None
+        try:
+            return self.ids_to_highlight[self.current_highlighted_idx]
+        except:
+            return None
+
+    def get_next_highlighted_id(self, current_row, forward):
+        if len(self.ids_to_highlight) == 0 or self.current_highlighted_idx is None:
+            return None
+        if current_row is None:
+            row_ = self.current_highlighted_idx
+        else:
+            row_ = current_row
+        while True:
+            row_ += 1 if forward else -1
+            if row_ < 0:
+                row_ = self.count() - 1;
+            elif row_ >= self.count():
+                row_ = 0
+            if self.id(row_) in self.ids_to_highlight_set:
+                break
+        try:
+            self.current_highlighted_idx = self.ids_to_highlight.index(self.id(row_))
+        except:
+            # This shouldn't happen ...
+            return None
+        return self.get_current_highlighted_id()
+
     def search(self, text, reset=True):
         try:
             if self.highlight_only:
                 self.db.search('')
                 if not text:
-                    self.rows_matching = set()
+                    self.ids_to_highlight = []
-                    self.lowest_row_matching = None
+                    self.ids_to_highlight_set = set()
+                    self.current_highlighted_idx = None
                 else:
-                    self.rows_matching = self.db.search(text, return_matches=True)
+                    self.ids_to_highlight = self.db.search(text, return_matches=True)
-                    if self.rows_matching:
+                    self.ids_to_highlight_set = set(self.ids_to_highlight)
-                        self.lowest_row_matching = self.db.row(self.rows_matching[0])
+                    if self.ids_to_highlight:
-                    self.rows_matching = set(self.rows_matching)
+                        self.current_highlighted_idx = 0
+                    else:
+                        self.current_highlighted_idx = None
             else:
-                self.rows_matching = set()
+                self.ids_to_highlight = []
-                self.lowest_row_matching = None
+                self.ids_to_highlight_set = set()
+                self.current_highlighted_idx = None
                 self.db.search(text)
         except ParseException as e:
             self.searched.emit(e.msg)
@@ -674,7 +712,7 @@ class BooksModel(QAbstractTableModel): # {{{
         if role in (Qt.DisplayRole, Qt.EditRole):
             return self.column_to_dc_map[col](index.row())
         elif role == Qt.BackgroundColorRole:
-            if self.id(index) in self.rows_matching:
+            if self.id(index) in self.ids_to_highlight_set:
                 return QColor('lightgreen')
         elif role == Qt.DecorationRole:
             if self.column_to_dc_decorator_map[col] is not None:

src/calibre/gui2/library/views.py

@@ -680,10 +680,21 @@ class BooksView(QTableView): # {{{
     def set_editable(self, editable, supports_backloading):
         self._model.set_editable(editable)
 
+    def move_highlighted_row(self, forward):
+        rows = self.selectionModel().selectedRows()
+        if len(rows) > 0:
+            current_row = rows[0].row()
+        else:
+            current_row = None
+        id_to_select = self._model.get_next_highlighted_id(current_row, forward)
+        if id_to_select is not None:
+            self.select_rows([id_to_select], using_ids=True)
+
     def search_proxy(self, txt):
         self._model.search(txt)
-        if self._model.lowest_row_matching is not None:
+        id_to_select = self._model.get_current_highlighted_id()
-            self.select_rows([self._model.lowest_row_matching], using_ids=False)
+        if id_to_select is not None:
+            self.select_rows([id_to_select], using_ids=True)
         self.setFocus(Qt.OtherFocusReason)
 
     def connect_to_search_box(self, sb, search_done):

src/calibre/gui2/main.py

@@ -34,6 +34,9 @@ path_to_ebook to the database.
                       help=_('Log debugging information to console'))
     parser.add_option('--no-update-check', default=False, action='store_true',
             help=_('Do not check for updates'))
+    parser.add_option('--ignore-plugins', default=False, action='store_true',
+            help=_('Ignore custom plugins, useful if you installed a plugin'
+                ' that is preventing calibre from starting'))
     return parser
 
 def init_qt(args):

src/calibre/gui2/metadata/__init__.py

@@ -0,0 +1,9 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+
+__license__   = 'GPL v3'
+__copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+
+

src/calibre/gui2/preferences/social.py

@@ -6,16 +6,19 @@ __license__   = 'GPL v3'
 __copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
+import time
+from threading import Thread
 
 from PyQt4.Qt import QDialog, QDialogButtonBox, Qt, QLabel, QVBoxLayout, \
-        SIGNAL, QThread
+        QTimer
 
 from calibre.ebooks.metadata import MetaInformation
 
-class Worker(QThread):
+class Worker(Thread):
 
-    def __init__(self, mi, parent):
+    def __init__(self, mi):
-        QThread.__init__(self, parent)
+        Thread.__init__(self)
+        self.daemon = True
         self.mi = MetaInformation(mi)
         self.exceptions = []
 
@@ -25,10 +28,12 @@ class Worker(QThread):
 
 class SocialMetadata(QDialog):
 
+    TIMEOUT = 300 # seconds
+
     def __init__(self, mi, parent):
         QDialog.__init__(self, parent)
 
-        self.bbox = QDialogButtonBox(QDialogButtonBox.Ok, Qt.Horizontal, self)
+        self.bbox = QDialogButtonBox(QDialogButtonBox.Cancel, Qt.Horizontal, self)
         self.mi = mi
         self.layout = QVBoxLayout(self)
         self.label = QLabel(_('Downloading social metadata, please wait...'), self)
@@ -36,15 +41,30 @@ class SocialMetadata(QDialog):
         self.layout.addWidget(self.label)
         self.layout.addWidget(self.bbox)
 
-        self.worker = Worker(mi, self)
+        self.worker = Worker(mi)
-        self.connect(self.worker, SIGNAL('finished()'), self.accept)
+        self.bbox.rejected.connect(self.reject)
-        self.connect(self.bbox, SIGNAL('rejected()'), self.reject)
         self.worker.start()
+        self.start_time = time.time()
+        self.timed_out = False
+        self.rejected = False
+        QTimer.singleShot(50, self.update)
 
     def reject(self):
-        self.disconnect(self.worker, SIGNAL('finished()'), self.accept)
+        self.rejected = True
         QDialog.reject(self)
 
+    def update(self):
+        if self.rejected:
+            return
+        if time.time() - self.start_time > self.TIMEOUT:
+            self.timed_out = True
+            self.reject()
+            return
+        if not self.worker.is_alive():
+            self.accept()
+            return
+        QTimer.singleShot(50, self.update)
+
     def accept(self):
         self.mi.tags = self.worker.mi.tags
         self.mi.rating = self.worker.mi.rating

src/calibre/gui2/preferences/template_functions.py

@@ -0,0 +1,227 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+
+__license__   = 'GPL v3'
+__copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+import json, traceback
+
+from calibre.gui2 import error_dialog
+from calibre.gui2.preferences import ConfigWidgetBase, test_widget
+from calibre.gui2.preferences.template_functions_ui import Ui_Form
+from calibre.gui2.widgets import PythonHighlighter
+from calibre.utils.formatter_functions import formatter_functions, compile_user_function
+
+
+class ConfigWidget(ConfigWidgetBase, Ui_Form):
+
+    def genesis(self, gui):
+        self.gui = gui
+        self.db = gui.library_view.model().db
+        self.current_plugboards = self.db.prefs.get('plugboards',{})
+        help_text = _('''
+        <p>Here you can add and remove functions used in template processing. A
+        template function is written in python. It takes information from the
+        book, processes it in some way, then returns a string result. Functions
+        defined here are usable in templates in the same way that builtin
+        functions are usable. The function must be named <b>evaluate</b>, and
+        must have the signature shown below.</p>
+        <p><code>evaluate(self, formatter, kwargs, mi, locals, your parameters)
+        &rarr; returning a unicode string</code></p>
+        <p>The parameters of the evaluate function are:
+        <ul>
+        <li><b>formatter</b>: the instance of the formatter being used to
+        evaluate the current template. You can use this to do recursive
+        template evaluation.</li>
+        <li><b>kwargs</b>: a dictionary of metadata. Field values are in this
+        dictionary.
+        <li><b>mi</b>: a Metadata instance. Used to get field information.
+        This parameter can be None in some cases, such as when evaluating
+        non-book templates.</li>
+        <li><b>locals</b>: the local variables assigned to by the current
+        template program.</li>
+        <li><b>your parameters</b>: You must supply one or more formal
+        parameters. The number must match the arg count box, unless arg count is
+        -1 (variable number or arguments), in which case the last argument must
+        be *args. At least one argument is required, and is usually the value of
+        the field being operated upon. Note that when writing in basic template
+        mode, the user does not provide this first argument. Instead it is
+        supplied by the formatter.</li>
+        </ul></p>
+        <p>
+        The following example function checks the value of the field. If the
+        field is not empty, the field's value is returned, otherwise the value
+        EMPTY is returned.
+        <pre>
+        name: my_ifempty
+        arg count: 1
+        doc: my_ifempty(val) -- return val if it is not empty, otherwise the string 'EMPTY'
+        program code:
+        def evaluate(self, formatter, kwargs, mi, locals, val):
+            if val:
+                return val
+            else:
+                return 'EMPTY'</pre>
+        This function can be called in any of the three template program modes:
+        <ul>
+        <li>single-function mode: {tags:my_ifempty()}</li>
+        <li>template program mode: {tags:'my_ifempty($)'}</li>
+        <li>general program mode: program: my_ifempty(field('tags'))</li>
+        </p>
+        ''')
+        self.textBrowser.setHtml(help_text)
+
+    def initialize(self):
+        try:
+            with open(P('template-functions.json'), 'rb') as f:
+                self.builtin_source_dict = json.load(f, encoding='utf-8')
+        except:
+            self.builtin_source_dict = {}
+
+        self.funcs = formatter_functions.get_functions()
+        self.builtins = formatter_functions.get_builtins()
+
+        self.build_function_names_box()
+        self.function_name.currentIndexChanged[str].connect(self.function_index_changed)
+        self.function_name.editTextChanged.connect(self.function_name_edited)
+        self.argument_count.valueChanged.connect(self.enable_replace_button)
+        self.documentation.textChanged.connect(self.enable_replace_button)
+        self.program.textChanged.connect(self.enable_replace_button)
+        self.create_button.clicked.connect(self.create_button_clicked)
+        self.delete_button.clicked.connect(self.delete_button_clicked)
+        self.create_button.setEnabled(False)
+        self.delete_button.setEnabled(False)
+        self.replace_button.setEnabled(False)
+        self.clear_button.clicked.connect(self.clear_button_clicked)
+        self.replace_button.clicked.connect(self.replace_button_clicked)
+        self.program.setTabStopWidth(20)
+        self.highlighter = PythonHighlighter(self.program.document())
+
+    def enable_replace_button(self):
+        self.replace_button.setEnabled(self.delete_button.isEnabled())
+
+    def clear_button_clicked(self):
+        self.build_function_names_box()
+        self.program.clear()
+        self.documentation.clear()
+        self.argument_count.clear()
+        self.create_button.setEnabled(False)
+        self.delete_button.setEnabled(False)
+
+    def build_function_names_box(self, scroll_to='', set_to=''):
+        self.function_name.blockSignals(True)
+        func_names = sorted(self.funcs)
+        self.function_name.clear()
+        self.function_name.addItem('')
+        self.function_name.addItems(func_names)
+        self.function_name.setCurrentIndex(0)
+        if set_to:
+            self.function_name.setEditText(set_to)
+            self.create_button.setEnabled(True)
+        self.function_name.blockSignals(False)
+        if scroll_to:
+            idx = self.function_name.findText(scroll_to)
+            if idx >= 0:
+                self.function_name.setCurrentIndex(idx)
+                if scroll_to not in self.builtins:
+                    self.delete_button.setEnabled(True)
+
+    def delete_button_clicked(self):
+        name = unicode(self.function_name.currentText())
+        if name in self.builtins:
+            error_dialog(self.gui, _('Template functions'),
+                         _('You cannot delete a built-in function'), show=True)
+        if name in self.funcs:
+            del self.funcs[name]
+            self.changed_signal.emit()
+            self.create_button.setEnabled(True)
+            self.delete_button.setEnabled(False)
+            self.build_function_names_box(set_to=name)
+            self.program.setReadOnly(False)
+        else:
+            error_dialog(self.gui, _('Template functions'),
+                         _('Function not defined'), show=True)
+
+    def create_button_clicked(self):
+        self.changed_signal.emit()
+        name = unicode(self.function_name.currentText())
+        if name in self.funcs:
+            error_dialog(self.gui, _('Template functions'),
+                         _('Name already used'), show=True)
+            return
+        if self.argument_count.value() == 0:
+            error_dialog(self.gui, _('Template functions'),
+                         _('Argument count must be -1 or greater than zero'),
+                         show=True)
+            return
+        try:
+            prog = unicode(self.program.toPlainText())
+            cls = compile_user_function(name, unicode(self.documentation.toPlainText()),
+                                        self.argument_count.value(), prog)
+            self.funcs[name] = cls
+            self.build_function_names_box(scroll_to=name)
+        except:
+            error_dialog(self.gui, _('Template functions'),
+                         _('Exception while compiling function'), show=True,
+                         det_msg=traceback.format_exc())
+
+    def function_name_edited(self, txt):
+        self.documentation.setReadOnly(False)
+        self.argument_count.setReadOnly(False)
+        self.create_button.setEnabled(True)
+        self.replace_button.setEnabled(False)
+        self.program.setReadOnly(False)
+
+    def function_index_changed(self, txt):
+        txt = unicode(txt)
+        self.create_button.setEnabled(False)
+        if not txt:
+            self.argument_count.clear()
+            self.documentation.clear()
+            self.documentation.setReadOnly(False)
+            self.argument_count.setReadOnly(False)
+            return
+        func = self.funcs[txt]
+        self.argument_count.setValue(func.arg_count)
+        self.documentation.setText(func.doc)
+        if txt in self.builtins:
+            if hasattr(func, 'program_text'):
+                self.program.setPlainText(func.program_text)
+            elif txt in self.builtin_source_dict:
+                self.program.setPlainText(self.builtin_source_dict[txt])
+            else:
+                self.program.setPlainText(_('function source code not available'))
+            self.documentation.setReadOnly(True)
+            self.argument_count.setReadOnly(True)
+            self.program.setReadOnly(True)
+            self.delete_button.setEnabled(False)
+        else:
+            self.program.setPlainText(func.program_text)
+            self.delete_button.setEnabled(True)
+            self.program.setReadOnly(False)
+        self.replace_button.setEnabled(False)
+
+    def replace_button_clicked(self):
+        self.delete_button_clicked()
+        self.create_button_clicked()
+    def refresh_gui(self, gui):
+        pass
+
+    def commit(self):
+        formatter_functions.reset_to_builtins()
+        pref_value = []
+        for f in self.funcs:
+            if f in self.builtins:
+                continue
+            func = self.funcs[f]
+            formatter_functions.register_function(func)
+            pref_value.append((func.name, func.doc, func.arg_count, func.program_text))
+        self.db.prefs.set('user_template_functions', pref_value)
+
+
+if __name__ == '__main__':
+    from PyQt4.Qt import QApplication
+    app = QApplication([])
+    test_widget('Advanced', 'TemplateFunctions')
+

src/calibre/gui2/preferences/template_functions.ui

@@ -0,0 +1,160 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<ui version="4.0">
+ <class>Form</class>
+ <widget class="QWidget" name="Form">
+  <property name="geometry">
+   <rect>
+    <x>0</x>
+    <y>0</y>
+    <width>798</width>
+    <height>672</height>
+   </rect>
+  </property>
+  <property name="windowTitle">
+   <string>Form</string>
+  </property>
+  <layout class="QGridLayout" name="gridLayout">
+   <item row="1" column="0" colspan="2">
+    <widget class="Line" name="line">
+     <property name="orientation">
+      <enum>Qt::Horizontal</enum>
+     </property>
+    </widget>
+   </item>
+   <item row="2" column="0">
+    <layout class="QHBoxLayout" name="horizontalLayout">
+     <item>
+      <layout class="QGridLayout" name="gridLayout_3">
+       <item row="0" column="0">
+        <widget class="QLabel" name="label_2">
+         <property name="text">
+          <string>&amp;Function:</string>
+         </property>
+         <property name="buddy">
+          <cstring>function_name</cstring>
+         </property>
+        </widget>
+       </item>
+       <item row="0" column="1">
+        <widget class="QComboBox" name="function_name">
+         <property name="toolTip">
+          <string>Enter the name of the function to create.</string>
+         </property>
+         <property name="editable">
+          <bool>true</bool>
+         </property>
+        </widget>
+       </item>
+       <item row="1" column="0">
+        <widget class="QLabel" name="label_3">
+         <property name="toolTip">
+          <string/>
+         </property>
+         <property name="text">
+          <string>Arg &amp;count:</string>
+         </property>
+         <property name="buddy">
+          <cstring>argument_count</cstring>
+         </property>
+        </widget>
+       </item>
+       <item row="1" column="1">
+        <widget class="QSpinBox" name="argument_count">
+         <property name="toolTip">
+          <string>Set this to -1 if the function takes a variable number of arguments</string>
+         </property>
+         <property name="minimum">
+          <number>-1</number>
+         </property>
+        </widget>
+       </item>
+       <item row="2" column="1">
+        <widget class="QTextEdit" name="documentation"/>
+       </item>
+       <item row="2" column="0">
+        <widget class="QLabel" name="label_4">
+         <property name="text">
+          <string>&amp;Documentation:</string>
+         </property>
+         <property name="alignment">
+          <set>Qt::AlignLeading|Qt::AlignLeft|Qt::AlignTop</set>
+         </property>
+         <property name="buddy">
+          <cstring>documentation</cstring>
+         </property>
+        </widget>
+       </item>
+       <item row="3" column="1">
+        <layout class="QHBoxLayout" name="horizontalLayout_2">
+         <item>
+          <widget class="QPushButton" name="clear_button">
+           <property name="text">
+            <string>&amp;Clear</string>
+           </property>
+          </widget>
+         </item>
+         <item>
+          <widget class="QPushButton" name="delete_button">
+           <property name="text">
+            <string>&amp;Delete</string>
+           </property>
+          </widget>
+         </item>
+         <item>
+          <widget class="QPushButton" name="replace_button">
+           <property name="text">
+            <string>&amp;Replace</string>
+           </property>
+          </widget>
+         </item>
+         <item>
+          <widget class="QPushButton" name="create_button">
+           <property name="text">
+            <string>C&amp;reate</string>
+           </property>
+          </widget>
+         </item>
+        </layout>
+       </item>
+      </layout>
+     </item>
+     <item>
+      <layout class="QVBoxLayout" name="horizontalLayout1">
+       <item>
+        <widget class="QLabel" name="label_4">
+         <property name="text">
+          <string>&amp;Program Code: (be sure to follow python indenting rules)</string>
+         </property>
+         <property name="buddy">
+          <cstring>program</cstring>
+         </property>
+        </widget>
+       </item>
+       <item>
+        <widget class="QPlainTextEdit" name="program">
+         <property name="minimumSize">
+          <size>
+           <width>400</width>
+           <height>0</height>
+          </size>
+         </property>
+         <property name="documentTitle">
+          <string notr="true"/>
+         </property>
+         <property name="tabStopWidth">
+          <number>30</number>
+         </property>
+        </widget>
+       </item>
+      </layout>
+     </item>
+    </layout>
+   </item>
+   <item row="0" column="0">
+    <widget class="QTextBrowser" name="textBrowser"/>
+   </item>
+  </layout>
+ </widget>
+ <resources/>
+ <connections/>
+</ui>

src/calibre/gui2/tag_view.py

@@ -730,7 +730,7 @@ class TagsModel(QAbstractItemModel): # {{{
             else:
                 collapse_model = 'partition'
                 collapse_template = tweaks['categories_collapsed_popularity_template']
-        collapse_letter = None
+        collapse_letter = collapse_letter_sk = None
 
         for i, r in enumerate(self.row_map):
             if self.hidden_categories and self.categories[i] in self.hidden_categories:
@@ -782,8 +782,17 @@ class TagsModel(QAbstractItemModel): # {{{
                         ts = tag.sort
                         if not ts:
                             ts = ' '
-                        if upper(ts[0]) != collapse_letter:
+                        try:
+                            sk = sort_key(ts)[0]
+                        except:
+                            sk = ts[0]
+
+                        if sk != collapse_letter_sk:
                             collapse_letter = upper(ts[0])
+                            try:
+                                collapse_letter_sk = sort_key(collapse_letter)[0]
+                            except:
+                                collapse_letter_sk = collapse_letter
                             sub_cat = TagTreeItem(parent=category,
                                      data = collapse_letter,
                                      category_icon = category_node.icon,

src/calibre/gui2/ui.py

@@ -103,6 +103,8 @@ class Main(MainWindow, MainWindowMixin, DeviceMixin, EmailMixin, # {{{
         self.gui_debug = gui_debug
         acmap = OrderedDict()
         for action in interface_actions():
+            if opts.ignore_plugins and action.plugin_path is not None:
+                continue
             try:
                 ac = action.load_actual_plugin(self)
             except:
@@ -256,6 +258,14 @@ class Main(MainWindow, MainWindowMixin, DeviceMixin, EmailMixin, # {{{
                 self.height())
         self.resize(self.width(), self._calculated_available_height)
 
+        for ac in self.iactions.values():
+            try:
+                ac.gui_layout_complete()
+            except:
+                import traceback
+                traceback.print_exc()
+                if ac.plugin_path is None:
+                    raise
 
         if config['autolaunch_server']:
             self.start_content_server()
@@ -269,7 +279,13 @@ class Main(MainWindow, MainWindowMixin, DeviceMixin, EmailMixin, # {{{
         self.set_window_title()
 
         for ac in self.iactions.values():
-            ac.initialization_complete()
+            try:
+                ac.initialization_complete()
+            except:
+                import traceback
+                traceback.print_exc()
+                if ac.plugin_path is None:
+                    raise
 
         if show_gui and self.gui_debug is not None:
             info_dialog(self, _('Debug mode'), '<p>' +

src/calibre/gui2/widgets.py

@@ -386,11 +386,13 @@ class LineEditECM(object):
         action_lower_case = case_menu.addAction(_('Lower Case'))
         action_swap_case = case_menu.addAction(_('Swap Case'))
         action_title_case = case_menu.addAction(_('Title Case'))
+        action_capitalize = case_menu.addAction(_('Capitalize'))
 
         self.connect(action_upper_case, SIGNAL('triggered()'), self.upper_case)
         self.connect(action_lower_case, SIGNAL('triggered()'), self.lower_case)
         self.connect(action_swap_case, SIGNAL('triggered()'), self.swap_case)
         self.connect(action_title_case, SIGNAL('triggered()'), self.title_case)
+        self.connect(action_capitalize, SIGNAL('triggered()'), self.capitalize)
 
         menu.addMenu(case_menu)
         menu.exec_(event.globalPos())
@@ -408,6 +410,10 @@ class LineEditECM(object):
         from calibre.utils.titlecase import titlecase
         self.setText(titlecase(unicode(self.text())))
 
+    def capitalize(self):
+        from calibre.utils.icu import capitalize
+        self.setText(capitalize(unicode(self.text())))
+
 
 class EnLineEdit(LineEditECM, QLineEdit):
 

src/calibre/library/catalog.py

@@ -599,8 +599,16 @@ class EPUB_MOBI(CatalogPlugin):
                           default=('~,'+_('Catalog')),
                           dest='exclude_tags',
                           action = None,
-                          help=_("Comma-separated list of tag words indicating book should be excluded from output.  Case-insensitive.\n"
+                          help=_("Comma-separated list of tag words indicating book should be excluded from output."
-                          "--exclude-tags=skip will match 'skip this book' and 'Skip will like this'.\n"
+                              "For example: 'skip' will match 'skip this book' and 'Skip will like this'."
+                              "Default: '%default'\n"
+                              "Applies to: ePub, MOBI output formats")),
+                   Option('--generate-authors',
+                          default=True,
+                          dest='generate_authors',
+                          action = 'store_true',
+                          help=_("Include 'Authors' section in catalog."
+                          "This switch is ignored - Books By Author section is always generated."
                           "Default: '%default'\n"
                           "Applies to: ePub, MOBI output formats")),
                    Option('--generate-descriptions',
@@ -1136,7 +1144,9 @@ class EPUB_MOBI(CatalogPlugin):
             def error(self):
                 def fget(self):
                     return self.__error
-                return property(fget=fget)
+                def fset(self, val):
+                    self.__error = val
+                return property(fget=fget,fset=fset)
             @dynamic_property
             def generateForKindle(self):
                 def fget(self):
@@ -1338,7 +1348,8 @@ class EPUB_MOBI(CatalogPlugin):
             if self.booksByTitle is None:
                 if not self.fetchBooksByTitle():
                     return False
-            self.fetchBooksByAuthor()
+            if not self.fetchBooksByAuthor():
+                return False
             self.fetchBookmarks()
             if self.opts.generate_descriptions:
                 self.generateHTMLDescriptions()
@@ -1402,6 +1413,88 @@ class EPUB_MOBI(CatalogPlugin):
                 except:
                     pass
 
+        def fetchBooksByAuthor(self):
+            '''
+            Generate a list of titles sorted by author from the database
+            return = Success
+            '''
+
+            self.updateProgressFullStep("Sorting database")
+
+            '''
+            # Sort titles case-insensitive, by author
+            self.booksByAuthor = sorted(self.booksByTitle,
+                                 key=lambda x:(x['author_sort'].upper(), x['author_sort'].upper()))
+            '''
+
+            self.booksByAuthor = list(self.booksByTitle)
+            self.booksByAuthor.sort(self.author_compare)
+
+            if False and self.verbose:
+                self.opts.log.info("fetchBooksByAuthor(): %d books" % len(self.booksByAuthor))
+                self.opts.log.info(" %-30s %-20s %s" % ('title', 'series', 'series_index'))
+                for title in self.booksByAuthor:
+                    self.opts.log.info((u" %-30s %-20s%5s " % \
+                                        (title['title'][:30],
+                                         title['series'][:20] if title['series'] else '',
+                                         title['series_index'],
+                                         )).encode('utf-8'))
+                raise SystemExit
+
+            # Build the unique_authors set from existing data
+            authors = [(record['author'], record['author_sort'].capitalize()) for record in self.booksByAuthor]
+
+            # authors[] contains a list of all book authors, with multiple entries for multiple books by author
+            #        authors[]: (([0]:friendly  [1]:sort))
+            # unique_authors[]: (([0]:friendly  [1]:sort  [2]:book_count))
+            books_by_current_author = 0
+            current_author = authors[0]
+            multiple_authors = False
+            unique_authors = []
+            for (i,author) in enumerate(authors):
+                if author != current_author:
+                    # Note that current_author and author are tuples: (friendly, sort)
+                    multiple_authors = True
+
+                if author != current_author and i:
+                    # Warn, exit if friendly matches previous, but sort doesn't
+                    if author[0] == current_author[0]:
+                        error_msg = _('''
+\n*** Metadata error ***
+Inconsistent Author Sort values for Author '{0}', unable to continue building catalog.
+Select all books by '{0}', apply correct Author Sort value in Edit Metadata dialog,
+then rebuild the catalog.\n''').format(author[0])
+
+                        self.opts.log.warn(error_msg)
+                        self.error = error_msg
+                        return False
+
+                    # New author, save the previous author/sort/count
+                    unique_authors.append((current_author[0], icu_title(current_author[1]),
+                                           books_by_current_author))
+                    current_author = author
+                    books_by_current_author = 1
+                elif i==0 and len(authors) == 1:
+                    # Allow for single-book lists
+                    unique_authors.append((current_author[0], icu_title(current_author[1]),
+                                           books_by_current_author))
+                else:
+                    books_by_current_author += 1
+            else:
+                # Add final author to list or single-author dataset
+                if (current_author == author and len(authors) > 1) or not multiple_authors:
+                    unique_authors.append((current_author[0], icu_title(current_author[1]),
+                                           books_by_current_author))
+
+            if False and self.verbose:
+                self.opts.log.info("\nfetchBooksByauthor(): %d unique authors" % len(unique_authors))
+                for author in unique_authors:
+                    self.opts.log.info((u" %-50s %-25s %2d" % (author[0][0:45], author[1][0:20],
+                       author[2])).encode('utf-8'))
+
+            self.authors = unique_authors
+            return True
+
         def fetchBooksByTitle(self):
 
             self.updateProgressFullStep("Fetching database")
@@ -1536,18 +1629,6 @@ class EPUB_MOBI(CatalogPlugin):
                                 notes = ' · '.join(notes)
                         elif field_md['datatype'] == 'datetime':
                             notes = format_date(notes,'dd MMM yyyy')
-                        elif field_md['datatype'] == 'composite':
-                            m = re.match(r'\[(.+)\]$', notes)
-                            if m is not None:
-                                # Sniff for special pseudo-list string "[<item, item>]"
-                                bracketed_content = m.group(1)
-                                if ',' in bracketed_content:
-                                    # Recast the comma-separated items as a list
-                                    items = bracketed_content.split(',')
-                                    items = [i.strip() for i in items]
-                                    notes = ' &middot; '.join(items)
-                                else:
-                                    notes = bracketed_content
                         this_title['notes'] = {'source':field_md['name'],
                                                    'content':notes}
 
@@ -1565,79 +1646,9 @@ class EPUB_MOBI(CatalogPlugin):
                                                                title['title_sort'][0:40])).decode('mac-roman'))
                 return True
             else:
+                self.error = _("No books found to catalog.\nCheck 'Excluded books' criteria in E-book options.")
                 return False
 
-        def fetchBooksByAuthor(self):
-            # Generate a list of titles sorted by author from the database
-
-            self.updateProgressFullStep("Sorting database")
-
-            '''
-            # Sort titles case-insensitive, by author
-            self.booksByAuthor = sorted(self.booksByTitle,
-                                 key=lambda x:(x['author_sort'].upper(), x['author_sort'].upper()))
-            '''
-
-            self.booksByAuthor = list(self.booksByTitle)
-            self.booksByAuthor.sort(self.author_compare)
-
-            if False and self.verbose:
-                self.opts.log.info("fetchBooksByAuthor(): %d books" % len(self.booksByAuthor))
-                self.opts.log.info(" %-30s %-20s %s" % ('title', 'series', 'series_index'))
-                for title in self.booksByAuthor:
-                    self.opts.log.info((u" %-30s %-20s%5s " % \
-                                        (title['title'][:30],
-                                         title['series'][:20] if title['series'] else '',
-                                         title['series_index'],
-                                         )).encode('utf-8'))
-                raise SystemExit
-
-            # Build the unique_authors set from existing data
-            authors = [(record['author'], record['author_sort'].capitalize()) for record in self.booksByAuthor]
-
-            # authors[] contains a list of all book authors, with multiple entries for multiple books by author
-            #        authors[]: (([0]:friendly  [1]:sort))
-            # unique_authors[]: (([0]:friendly  [1]:sort  [2]:book_count))
-            books_by_current_author = 0
-            current_author = authors[0]
-            multiple_authors = False
-            unique_authors = []
-            for (i,author) in enumerate(authors):
-                if author != current_author:
-                    # Note that current_author and author are tuples: (friendly, sort)
-                    multiple_authors = True
-
-                if author != current_author and i:
-                    # Warn if friendly matches previous, but sort doesn't
-                    if author[0] == current_author[0]:
-                        self.opts.log.warn("Warning: multiple entries for Author '%s' with differing Author Sort metadata:" % author[0])
-                        self.opts.log.warn(" '%s' != '%s'" % (author[1], current_author[1]))
-
-                    # New author, save the previous author/sort/count
-                    unique_authors.append((current_author[0], icu_title(current_author[1]),
-                                           books_by_current_author))
-                    current_author = author
-                    books_by_current_author = 1
-                elif i==0 and len(authors) == 1:
-                    # Allow for single-book lists
-                    unique_authors.append((current_author[0], icu_title(current_author[1]),
-                                           books_by_current_author))
-                else:
-                    books_by_current_author += 1
-            else:
-                # Add final author to list or single-author dataset
-                if (current_author == author and len(authors) > 1) or not multiple_authors:
-                    unique_authors.append((current_author[0], icu_title(current_author[1]),
-                                           books_by_current_author))
-
-            if False and self.verbose:
-                self.opts.log.info("\nfetchBooksByauthor(): %d unique authors" % len(unique_authors))
-                for author in unique_authors:
-                    self.opts.log.info((u" %-50s %-25s %2d" % (author[0][0:45], author[1][0:20],
-                       author[2])).encode('utf-8'))
-
-            self.authors = unique_authors
-
         def fetchBookmarks(self):
             '''
             Collect bookmarks for catalog entries
@@ -1751,8 +1762,6 @@ class EPUB_MOBI(CatalogPlugin):
                 # Generate the header from user-customizable template
                 soup = self.generateHTMLDescriptionHeader(title)
 
-
-
                 # Write the book entry to contentdir
                 outfile = open("%s/book_%d.html" % (self.contentDir, int(title['id'])), 'w')
                 outfile.write(soup.prettify())
@@ -3250,7 +3259,7 @@ class EPUB_MOBI(CatalogPlugin):
             # Loop over the series titles, find start of each letter, add description_preview_count books
             # Special switch for using different title list
             title_list = self.booksBySeries
-            current_letter = self.letter_or_symbol(title_list[0]['series'][0])
+            current_letter = self.letter_or_symbol(self.generateSortTitle(title_list[0]['series'])[0])
             title_letters = [current_letter]
             current_series_list = []
             current_series = ""
@@ -4362,7 +4371,7 @@ class EPUB_MOBI(CatalogPlugin):
                 _soup = BeautifulSoup('')
                 genresTag = Tag(_soup,'p')
                 gtc = 0
-                for (i, tag) in enumerate(book.get('tags', [])):
+                for (i, tag) in enumerate(sorted(book.get('tags', []))):
                     aTag = Tag(_soup,'a')
                     if self.opts.generate_genres:
                         aTag['href'] = "Genre_%s.html" % re.sub("\W","",tag.lower())
@@ -4381,6 +4390,7 @@ class EPUB_MOBI(CatalogPlugin):
                     formats.append(format.rpartition('.')[2].upper())
                 formats = ' &middot; '.join(formats)
 
+            # Date of publication
             pubdate = book['date']
             pubmonth, pubyear = pubdate.split(' ')
 
@@ -4973,12 +4983,16 @@ class EPUB_MOBI(CatalogPlugin):
             build_log.append(" book count: %d" % len(opts_dict['ids']))
 
         sections_list = ['Authors']
+        '''
+        if opts.generate_authors:
+            sections_list.append('Authors')
+        '''
         if opts.generate_titles:
             sections_list.append('Titles')
-        if opts.generate_recently_added:
-            sections_list.append('Recently Added')
         if opts.generate_genres:
             sections_list.append('Genres')
+        if opts.generate_recently_added:
+            sections_list.append('Recently Added')
         if opts.generate_descriptions:
             sections_list.append('Descriptions')
 
@@ -5058,6 +5072,8 @@ class EPUB_MOBI(CatalogPlugin):
                             abort_after_input_dump=False)
             plumber.merge_ui_recommendations(recommendations)
             plumber.run()
-            return 0
+            # returns to gui2.actions.catalog:catalog_generated()
+            return None
         else:
-            return 1
+            # returns to gui2.actions.catalog:catalog_generated()
+            return catalog.error

src/calibre/library/cli.py

@@ -693,8 +693,12 @@ def command_catalog(args, dbpath):
                             }
 
     with plugin:
-        plugin.run(args[1], opts, get_db(dbpath, opts))
+        ret = plugin.run(args[1], opts, get_db(dbpath, opts))
-    return 0
+    if ret is None:
+        ret = 0
+    else:
+        ret = 1
+    return ret
 
 # end of GR additions
 
@@ -986,8 +990,8 @@ def command_restore_database(args, dbpath):
         return 1
 
     if not opts.really_do_it:
-        prints(_('You must provide the --really-do-it option to do a'
+        prints(_('You must provide the %s option to do a'
-            ' recovery'), end='\n\n')
+            ' recovery')%'--really-do-it', end='\n\n')
         parser.print_help()
         return 1
 

src/calibre/library/database2.py

@@ -38,6 +38,7 @@ from calibre.utils.search_query_parser import saved_searches, set_saved_searches
 from calibre.ebooks import BOOK_EXTENSIONS, check_ebook_format
 from calibre.utils.magick.draw import save_cover_data_to
 from calibre.utils.recycle_bin import delete_file, delete_tree
+from calibre.utils.formatter_functions import load_user_template_functions
 
 
 copyfile = os.link if hasattr(os, 'link') else shutil.copyfile
@@ -185,6 +186,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         migrate_preference('saved_searches', {})
         set_saved_searches(self, 'saved_searches')
 
+        load_user_template_functions(self.prefs.get('user_template_functions', []))
+
         self.conn.executescript('''
         DROP TRIGGER IF EXISTS author_insert_trg;
         CREATE TEMP TRIGGER author_insert_trg
@@ -687,11 +690,14 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         mi = Metadata(None)
 
         aut_list = row[fm['au_map']]
-        aut_list = [p.split(':::') for p in aut_list.split(':#:')]
+        if aut_list:
+            aut_list = [p.split(':::') for p in aut_list.split(':#:') if p]
+        else:
+            aut_list = []
         aum = []
         aus = {}
         for (author, author_sort) in aut_list:
-            aum.append(author)
+            aum.append(author.replace('|', ','))
             aus[author] = author_sort.replace('|', ',')
         mi.title       = row[fm['title']]
         mi.authors     = aum

src/calibre/library/sqlite.py

@@ -100,7 +100,7 @@ class AumSortedConcatenate(object):
         keys = self.ans.keys()
         l = len(keys)
         if l == 0:
-            return 'Unknown:::Unknown'
+            return None
         if l == 1:
             return self.ans[keys[0]]
         return ':#:'.join([self.ans[v] for v in sorted(keys)])

src/calibre/manual/faq.rst

@@ -437,6 +437,15 @@ My antivirus program claims |app| is a virus/trojan?
 
 Your antivirus program is wrong. |app| is a completely open source product. You can actually browse the source code yourself (or hire someone to do it for you) to verify that it is not a virus. Please report the false identification to whatever company you buy your antivirus software from. If the antivirus program is preventing you from downloading/installing |app|, disable it temporarily, install |app| and then re-enable it.
 
+How do I backup |app|?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The most important thing to backup is the |app| library folder, that contains all your books and metadata. This is the folder you chose for your |app| library when you ran |app| for the first time. You can get the path to the library folder by clicking the |app| icon on the main toolbar. You must backup this complete folder with all its files and sub-folders. 
+
+You can switch |app| to using a backed up library folder by simply clicking the |app| icon on the toolbar and choosing your backup library folder. 
+
+If you want to backup the |app| configuration/plugins, you have to backup the config directory. You can find this config directory via :guilabel:`Preferences->Miscellaneous`. Note that restoring configuration directories is not officially supported, but should work in most cases. Just copy the contents of the backup directory into the current configuration directory to restore.
+
 How do I use purchased EPUB books with |app|?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Most purchased EPUB books have `DRM <http://wiki.mobileread.com/wiki/DRM>`_. This prevents |app| from opening them. You can still use |app| to store and transfer them to your e-book reader. First, you must authorize your reader on a windows machine with Adobe Digital Editions. Once this is done, EPUB books transferred with |app| will work fine on your reader. When you purchase an epub book from a website, you will get an ".acsm" file. This file should be opened with Adobe Digital Editions, which will then download the actual ".epub" e-book. The e-book file will be stored in the folder "My Digital Editions", from where you can add it to |app|.

src/calibre/manual/gui.rst

@@ -478,6 +478,10 @@ Calibre has several keyboard shortcuts to save you time and mouse movement. Thes
       - Focus the search bar
     * - :kbd:`Shift+Ctrl+F`
       - Open the advanced search dialog
+    * - :kbd:`N or F3`
+      - Find the next book that matches the current search (only works if the highlight checkbox next to the search bar is checked)
+    * - :kbd:`Shift+N or Shift+F3`
+      - Find the next book that matches the current search (only works if the highlight checkbox next to the search bar is checked)
     * - :kbd:`Ctrl+D` 
       - Download metadata and shortcuts
     * - :kbd:`Ctrl+R` 

src/calibre/manual/template_lang.rst

@@ -308,6 +308,12 @@ The following program produces the same results as the original recipe, using on
 
 It would be possible to do the above with no custom columns by putting the program into the template box of the plugboard. However, to do so, all comments must be removed because the plugboard text box does not support multi-line editing. It is debatable whether the gain of not having the custom column is worth the vast increase in difficulty caused by the program being one giant line.
 
+
+User-defined Template Functions
+-------------------------------
+
+You can add your own functions to the template processor. Such functions are written in python, and can be used in any of the three template programming modes. The functions are added by going to Preferences -> Advanced -> Template Functions. Instructions are shown in that dialog.
+
 Special notes for save/send templates
 -------------------------------------