[Sync] Sync with trunk. Revision 7453.

Changelog.yaml

@@ -4,6 +4,134 @@
 # for important features/bug fixes.
 # Also, each release can have new and improved recipes.
 
+- version: 0.7.37
+  date: 2011-01-02
+
+  new features:
+    - title: "This realease is mostly a bug fix release to fix various things that got broken by all the changes in 0.7.36"
+
+    - title: "Tag browser: Move the configuration of the sub-category grouping from tweaks to the Preferences dialog"
+
+    - title: "Tag browser: Allow changing the sub-categorization scheme from the right click menu"
+
+  bug fixes:
+    - title: "Fix regression in 0.7.36 that caused the Tag Browser to break if you have items in it with empty sort values"
+
+    - title: "Catalog generation: Fix various regressions introduced in 0.7.36 on windows"
+      description: >
+        "Database integrity check not working after catalog generation. Catalog generation failing with a file in use error. Spurious question marks appearing in the catalog"
+
+    - title: "Catalog generation: Work on a copy of the library database so as not to lock it"
+
+    - title: "Catalog generation: Handle merge of comments + custom field when custom filed is None"
+
+    - title: "Fix regression that broke sort_columns_at_startup tweak in 0.7.36"
+
+    - title: "Tag Browser: Fix the Manage X items in the right click menu, which broke in 0.7.36"
+
+    - title: "Tag Browser: Fix grouping by name for authors"
+
+    - title: "Nook color: Fix main memory and SD card swapped in calibre"
+      tickets: [8159]
+
+    - title: "Fix regression in 0.7.36 that broke PDF Output when specifying a cover"
+
+    - title: "Catalog generation: Fix regression in MOBI catalog that caused it to not appear as periodical on Kindle"
+
+    - title: "Fix regression in 0.7.36 that broke opening the book details dialog by double clicking on the book details panel"
+
+
+- version: 0.7.36
+  date: 2011-01-01
+
+  new features:
+    - title: "Tag browser: Add subcategories and search"
+      description: "When a category has many items, it will be automatically split up. Also add a search to quickly find an item in the Tag Browser. The sub categories can be controlled via preferences->Tweaks. Also add a button to collapse all categories"
+      type: major
+
+    - title: "Device drivers for the Google Nexus S, Motorola Backflip, Samsung Galaxy Tablet, PocketBook 603/903, EEEReader DR900 and the NextBook"
+
+    - title: "Tag editor dialog now remebers its last used size"
+      tickets: [8063]
+
+    - title: "OS X dmg: Add a symlink pointing to the Applications folder for easy installation"
+      tickets: [8052]
+
+    - title: "Catalog generation: CSV/XML catalogs now support custom fields. Also write UTF-8 BOM to CSV output file."
+      tickets: [8014]
+
+    - title: "EPUB/MOBI catalogs: Various new features"
+      description: "Added a custom field/value for excluding books, OR'd with existing tag list. Added a thumbnail width hint, from 1.0 - 2.0 inches. Deprecated support for special note tag '*', added support for custom column containing note to be inserted in Description header. Added 'Merge with comments' feature, which non-destructively combines Comments with a custom field when generating Descriptions. Moved Description header into a user-editable template file.  All fields except thumb and comments accessible to template."
+      tickets: [7820, 5297, 6765]
+
+    - title: "SONY driver: Allow the creation of an All by Something category via the tweaks."
+
+    - title: "Add a tweak to control the delay when sending mails using gmail or hotmail."
+      tickets: [8064]
+
+    - title: "Add output encoding option for TXT/PDB/PMLX output plugins to the GUI"
+
+    - title: "Add an environment variable to control the temporary directory calibre uses"
+
+    - title: "Use the new HTML editor widget for comments custom columns as well"
+
+    - title: "Content server: Fix regression that broke saved searches"
+      tickets: [8047]
+
+    - title: "E-book viewer: Fix regression that broke previous page button"
+
+    - title: "Add a tweak to allow double clicking on the book list to open the edit metadata dialog"
+      tickets: [8032]
+
+    - title: "Add a tweak to use a template for formatting SONY collection names"
+      tickets: [8033]
+
+    - title: "Bulk edit metadata, search and replace: Show all values for multiple fields in the text region, separated by :::"
+      tickets: [8030]
+
+    - title: "Update user agent used by calibre when connecting to websites"
+
+  bug fixes:
+    - title: "FB2 Output: Fix regression that broke images in generated FB2 files"
+      tickets: [8142]
+
+    - title: "When unzipping zip files that contain filenames with unknown character encoding, sanitize the filenames correctly"
+      tickets: [8050]
+
+    - title: "TCR Output: Fix TCR compression adding junk to the end of the text. Remove compression level option."
+
+    - title: "PDF Output: Fix regression that broke the margin options."
+
+    - title: "FB2 Input: Handle non UTF-8 encodings on OS X"
+      tickets: [8115]
+
+    - title: "SNB Input: Better error handling if some metadata is missing in the SNB file. Add Wi-Fi connection support for the Bambook"
+
+    - title: "Allow hyperlinks to be clicked in comments metadata in the book details panel"
+      tickets: [8054]
+
+  improved recipes:
+    - Brand Eins
+    - Volksrant
+    - Smithsonian
+    - Business World
+    - El Universal
+    - Salon
+    - The Week
+    - EL Pais
+    - Wired Magazine
+    - Heraldo de Aragon
+
+  new recipes:
+    - title: "Karlsruhe News"
+      author: "tfeld"
+      
+    - title: "El Periodico and Red Aragon"
+      author: "desUBIKado"
+
+    - title: "Business Insider"
+      author: "Darko Miletic"
+
 - version: 0.7.35
   date: 2010-12-23
 

resources/catalog/stylesheet.css

@@ -2,19 +2,29 @@ body { background-color: white; }
 
 p.title  {
 	margin-top:0em;
-	margin-bottom:1em;
+	margin-bottom:0em;
 	text-align:center;
 	font-style:italic;
 	font-size:xx-large;
-    border-bottom: solid black 2px;
+	}
+
+p.series_id {
+	margin-top:0em;
+	margin-bottom:0em;
+	text-align:center;
+	}
+
+a.series_id {
+	font-style:normal;
+	font-size:large;
 	}
 
 p.author {
+	font-size:large;
 	margin-top:0em;
 	margin-bottom:0em;
 	text-align: center;
 	text-indent: 0em;
-	font-size:large;
   	}
 
 p.author_index {
@@ -26,7 +36,8 @@ p.author_index {
 	text-indent: 0em;
 	}
 
-p.tags {
+p.genres {
+	font-style:normal;
 	margin-top:0.5em;
 	margin-bottom:0em;
 	text-align: left;
@@ -108,6 +119,13 @@ p.date_read {
 	text-indent:-6em;
 	}
 
+hr.annotations_divider {
+	width:50%;
+	margin-left:1em;
+	margin-top:0em;
+	margin-bottom:0em;
+	}
+
 hr.description_divider {
 	width:90%;
 	margin-left:5%;
@@ -117,20 +135,37 @@ hr.description_divider {
     border-left: solid white 0px;
 	}
 
-hr.annotations_divider {
+hr.header_divider {
-	width:50%;
+	width:100%;
-	margin-left:1em;
+	border-top: solid white 1px;
-	margin-top:0em;
+	border-right: solid white 0px;
-	margin-bottom:0em;
+    border-bottom: solid black 2px;
+    border-left: solid white 0px;
+	}
+
+hr.merged_comments_divider {
+	width:80%;
+	margin-left:10%;
+	border-top: solid white 0px;
+	border-right: solid white 0px;
+    border-bottom: dashed gray 2px;
+    border-left: solid white 0px;
 	}
 
 td.publisher, td.date {
 	font-weight:bold;
 	text-align:center;
 	}
-td.rating {
+
-	text-align: center;
+td.rating{
+	text-align:center;
 	}
+
+td.notes {
+	font-size: 100%;
+	text-align:center;
+	}
+
 td.thumbnail img {
 	-webkit-box-shadow: 4px 4px 12px #999;
 	}

resources/catalog/template.xhtml

@@ -0,0 +1,41 @@
+<html xmlns="{xmlns}">
+    <head>
+        <title>{title_str}</title>
+        <meta name="catalog description header" http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+		<link rel="stylesheet" type="text/css" href="stylesheet.css" media="screen" />
+    </head>
+    <body>
+		<p class="title">{title}</p>
+		<p class="series_id"><a class="series_id">{series} [{series_index}]</a></p>
+		<hr class="header_divider" />
+		<p class="author">{author_prefix}<a class="author">{author}</a></p>
+		<p class="genres">{genres}</p>
+		<p class="formats">{formats}</p>
+		<table width="100%" border="0">
+		  <tr>
+			<td class="thumbnail" rowspan="7">{thumb}</td>
+			<td class="empty"></td>
+		  </tr>
+		  <tr>
+			<td class="empty"></td>
+		  </tr>
+		  <tr>
+			<td class="publisher">{publisher}</td>
+		  </tr>
+		  <tr>
+			<td class="date">{pubyear}</td>
+		  </tr>
+		  <tr>
+			<td class="rating">{rating}</td>
+		  </tr>
+		  <tr>
+			<td class="notes">{note_source}: {note_content}</td>
+		  </tr>
+		  <tr>
+			<td></td>
+		  </tr>
+		</table>
+		<hr class="description_divider" />
+		<div class="description">{comments}</div>
+	</body>
+</html>

resources/default_tweaks.py

@@ -55,6 +55,24 @@ author_sort_copy_method = 'invert'
 #   categories_use_field_for_author_name = 'author_sort'
 categories_use_field_for_author_name = 'author'
 
+# When partitioning the tags browser, the format of the subcategory label is
+# controlled by a template: categories_collapsed_name_template if sorting by
+# name, categories_collapsed_rating_template if sorting by average rating, and
+# categories_collapsed_popularity_template if sorting by popularity. There are
+# two variables available to the template: first and last. The variable 'first'
+# is the initial item in the subcategory, and the variable 'last' is the final
+# item in the subcategory. Both variables are 'objects'; they each have multiple
+# values that are obtained by using a suffix. For example, first.name for an
+# author category will be the name of the author. The sub-values available are:
+#  name: the printable name of the item
+#  count: the number of books that references this item
+#  avg_rating: the averate rating of all the books referencing this item
+#  sort: the sort value. For authors, this is the author_sort for that author
+#  category: the category (e.g., authors, series) that the item is in.
+categories_collapsed_name_template = '{first.sort:shorten(4,'',0)} - {last.sort:shorten(4,'',0)}'
+categories_collapsed_rating_template = '{first.avg_rating:4.2f:ifempty(0)} - {last.avg_rating:4.2f:ifempty(0)}'
+categories_collapsed_popularity_template = '{first.count:d} - {last.count:d}'
+
 
 # Set whether boolean custom columns are two- or three-valued.
 #  Two-values for true booleans
@@ -289,3 +307,11 @@ locale_for_sorting =  ''
 # metadata  one book at a time. If True, then the fields are laid out using two
 # columns. If False, one column is used.
 metadata_single_use_2_cols_for_custom_fields = True
+
+# The number of seconds to wait before sending emails when using a
+# public email server like gmail or hotmail. Default is: 5 minutes
+# Setting it to lower may cause the server's SPAM controls to kick in,
+# making email sending fail. Changes will take effect only after a restart of
+# calibre.
+public_smtp_relay_delay = 301
+

resources/recipes/brand_eins.recipe

@@ -1,19 +1,16 @@
 #!/usr/bin/env  python
 # -*- coding: utf-8 mode: python -*-
 
-# Find the newest version of this recipe here:
-# https://github.com/consti/BrandEins-Recipe/raw/master/brandeins.recipe
-
 __license__   = 'GPL v3'
 __copyright__ = '2010, Constantin Hofstetter <consti at consti.de>, Steffen Siebert <calibre at steffensiebert.de>'
-__version__   = '0.96'
+__version__   = '0.97'
 
 ''' http://brandeins.de - Wirtschaftsmagazin '''
 import re
 import string
+from calibre.ebooks.BeautifulSoup import Tag
 from calibre.web.feeds.recipes import BasicNewsRecipe
 
-
 class BrandEins(BasicNewsRecipe):
 
   title = u'brand eins'
@@ -28,6 +25,8 @@ class BrandEins(BasicNewsRecipe):
   language = 'de'
   publication_type = 'magazine'
   needs_subscription = 'optional'
+  # Prevent that conversion date is appended to title
+  timefmt = ''
 
   # 2 is the last full magazine (default)
   # 1 is the newest (but not full)
@@ -66,6 +65,13 @@ class BrandEins(BasicNewsRecipe):
         new_p = "<p><i>"+ content +"</i></p>"
         p.replaceWith(new_p)
 
+    # Change <h3> to <h1>
+    header = soup.find("h3")
+    if header:
+        tag = Tag(soup, "h1")
+        tag.insert(0, header.contents[0])
+        header.replaceWith(tag)
+
     return soup
 
   def get_cover(self, soup):
@@ -77,6 +83,7 @@ class BrandEins(BasicNewsRecipe):
 
   def parse_index(self):
     feeds = []
+    issue_map = {}
 
     archive = "http://www.brandeins.de/archiv.html"
 
@@ -88,21 +95,31 @@ class BrandEins(BasicNewsRecipe):
         pass
 
     soup = self.index_to_soup(archive)
-    latest_jahrgang = soup.findAll('div', attrs={'class': re.compile(r'\bjahrgang-latest\b') })[0].findAll('ul')[0]
+    issue_list = soup.findAll('div', attrs={'class': 'tx-brandeinsmagazine-pi1'})[0].findAll('a')
-    pre_latest_issue = latest_jahrgang.findAll('a')[len(latest_jahrgang.findAll('a'))-issue]
+    issue_list = [i for i in issue_list if i.get('onmouseover', False)]
-    url = pre_latest_issue.get('href', False)
+    for i in issue_list:
-    # Get month and year of the magazine issue - build it out of the title of the cover
+        issue_number_string = i.get('onmouseover', False)
-    self.timefmt = " " + re.search(r"(?P<date>\d\d\/\d\d\d\d)", pre_latest_issue.find('img').get('title', False)).group('date')
+        if issue_number_string:
+            match = re.match("^switch_magazine\(([0-9]+), ([0-9]+)\)$", issue_number_string)
+            issue_number = "%04i%02i" % (int(match.group(1)), int(match.group(2)))
+            issue_map[issue_number] = i
+    keys = issue_map.keys()
+    keys.sort()
+    keys.reverse()
+    selected_issue = issue_map[keys[issue-1]]
+    url = selected_issue.get('href', False)
+    # Get the title for the magazin - build it out of the title of the cover - take the issue and year;
+    self.title = "brand eins "+ re.search(r"(?P<date>\d\d\/\d\d\d\d)", selected_issue.find('img').get('title', False)).group('date')
     url = 'http://brandeins.de/'+url
 
     # url = "http://www.brandeins.de/archiv/magazin/tierisch.html"
-    titles_and_articles = self.brand_eins_parse_latest_issue(url)
+    titles_and_articles = self.brand_eins_parse_issue(url)
     if titles_and_articles:
       for title, articles in titles_and_articles:
         feeds.append((title, articles))
     return feeds
 
-  def brand_eins_parse_latest_issue(self, url):
+  def brand_eins_parse_issue(self, url):
     soup = self.index_to_soup(url)
     self.cover_url = self.get_cover(soup)
     article_lists = [soup.find('div', attrs={'class':'subColumnLeft articleList'}), soup.find('div', attrs={'class':'subColumnRight articleList'})]
@@ -145,4 +162,3 @@ class BrandEins(BasicNewsRecipe):
           current_articles.append({'title': title, 'url': url, 'description': description, 'date':''})
     titles_and_articles.append([chapter_title, current_articles])
     return titles_and_articles
-

resources/recipes/business_insider.recipe

@@ -0,0 +1,69 @@
+__license__   = 'GPL v3'
+__copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
+'''
+www.businessinsider.com
+'''
+
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class Business_insider(BasicNewsRecipe):
+    title                 = 'Business Insider'
+    __author__            = 'Darko Miletic'
+    description           = 'Noticias de Argentina y el resto del mundo'
+    publisher             = 'Business Insider, Inc.'
+    category              = 'news, politics, finances, world'
+    oldest_article        = 2
+    max_articles_per_feed = 200
+    no_stylesheets        = True
+    encoding              = 'utf8'
+    use_embedded_content  = True
+    language              = 'en'
+    remove_empty_feeds    = True
+    publication_type      = 'newsportal'
+    masthead_url          = 'http://static.businessinsider.com/assets/images/logos/tbi_print.jpg'
+    extra_css             = """
+                               body{font-family: Arial,Helvetica,sans-serif }
+                               img{margin-bottom: 0.4em; display:block}
+                            """
+
+    conversion_options = {
+                          'comment'   : description
+                        , 'tags'      : category
+                        , 'publisher' : publisher
+                        , 'language'  : language
+                        }
+
+    remove_tags = [
+                      dict(name=['meta','link'])
+                     ,dict(attrs={'class':'feedflare'})
+                  ]
+    remove_attributes=['lang','border']
+
+
+    feeds = [
+              (u'Latest'      , u'http://feeds2.feedburner.com/businessinsider'                          )
+             ,(u'Markets'     , u'http://feeds.feedburner.com/TheMoneyGame'                              )
+             ,(u'Wall Street' , u'http://feeds.feedburner.com/clusterstock'                              )
+             ,(u'Tech'        , u'http://feeds.feedburner.com/typepad/alleyinsider/silicon_alley_insider')
+             ,(u'The Wire'    , u'http://feeds.feedburner.com/businessinsider/thewire'                   )
+             ,(u'War Room'    , u'http://feeds.feedburner.com/businessinsider/warroom'                   )
+             ,(u'Sports'      , u'http://feeds.feedburner.com/businessinsider/sportspage'                )
+             ,(u'Tools'       , u'http://feeds.feedburner.com/businessinsider/tools'                     )
+             ,(u'Travel'      , u'http://feeds.feedburner.com/businessinsider/travel'                    )
+            ]
+
+
+    def preprocess_html(self, soup):
+        for item in soup.findAll(style=True):
+            del item['style']
+        for item in soup.findAll('a'):
+            if item['href'].startswith('http://feedads'):
+               item.extract()
+            else:
+                if item.string is not None:
+                   tstr = item.string
+                   item.replaceWith(tstr)
+        for item in soup.findAll('img'):
+            if not item.has_key('alt'):
+               item['alt'] = 'image'
+        return soup

resources/recipes/businessworldin.recipe

@@ -1,7 +1,5 @@
-#!/usr/bin/env  python
-
 __license__   = 'GPL v3'
-__copyright__ = '2009, Darko Miletic <darko.miletic at gmail.com>'
+__copyright__ = '2009-2010, Darko Miletic <darko.miletic at gmail.com>'
 '''
 www.businessworld.in
 '''
@@ -22,7 +20,11 @@ class BusinessWorldMagazine(BasicNewsRecipe):
     use_embedded_content = False
     encoding             = 'utf-8'
     language             = 'en_IN'
-
+    extra_css            = """
+                              img{display: block; margin-bottom: 0.5em}
+                              body{font-family: Arial,Helvetica,sans-serif}
+                              h2{color: gray; display: block}
+                           """
 
     conversion_options = {
                           'comment'          : description
@@ -42,7 +44,26 @@ class BusinessWorldMagazine(BasicNewsRecipe):
         articles = []
         linklist = []
         soup = self.index_to_soup(self.INDEX)
-
+        
+        tough = soup.find('div', attrs={'id':'tough'})
+        if tough:
+           for item in tough.findAll('h1'):
+                description = ''
+                title_prefix = ''
+                feed_link = item.find('a')
+                if feed_link and feed_link.has_key('href'):
+                    url   = self.ROOT + feed_link['href']
+                    if not self.is_in_list(linklist,url):
+                        title = title_prefix + self.tag_to_string(feed_link)
+                        date  = strftime(self.timefmt)
+                        articles.append({
+                                          'title'      :title
+                                         ,'date'       :date
+                                         ,'url'        :url
+                                         ,'description':description
+                                        })
+                        linklist.append(url)
+        
         for item in soup.findAll('div', attrs={'class':'nametitle'}):
             description = ''
             title_prefix = ''
@@ -62,8 +83,8 @@ class BusinessWorldMagazine(BasicNewsRecipe):
         return [(soup.head.title.string, articles)]
 
     
-    keep_only_tags = [dict(name='div', attrs={'id':['register-panel','printwrapper']})]
+    keep_only_tags = [dict(name='div', attrs={'id':'printwrapper'})]
-    remove_tags = [dict(name=['object','link'])]
+    remove_tags = [dict(name=['object','link','meta','base','iframe','link','table'])]
 
     def print_version(self, url):
         return url.replace('/bw/','/bw/storyContent/')

resources/recipes/el_universal.recipe

@@ -1,7 +1,5 @@
-#!/usr/bin/env  python
-
 __license__   = 'GPL v3'
-__copyright__ = '2009, Darko Miletic <darko.miletic at gmail.com>'
+__copyright__ = '2009-2010, Darko Miletic <darko.miletic at gmail.com>'
 '''
 eluniversal.com.mx
 '''
@@ -18,75 +16,25 @@ class ElUniversal(BasicNewsRecipe):
     category              = 'news, politics, Mexico'
     no_stylesheets        = True
     use_embedded_content  = False
-    encoding              = 'cp1252'
+    encoding              = 'utf8'
     remove_javascript     = True
-    language = 'es'
+    remove_empty_feeds    = True
+    publication_type      = 'newspaper'
+    language              = 'es'
 
     extra_css = '''
-                    body{font-family:Arial,Helvetica,sans-serif; font-size:x-small;}
+                    body{font-family:Arial,Helvetica,sans-serif}
-                    .geoGris30{font-family:Georgia,"Times New Roman",Times,serif; font-size:large; color:#003366; font-weight:bold;}
+                    .noteTitle{font-family: Georgia,"Times New Roman",Times,serif; color: #336699; font-size: xx-large; font-weight: bold}
-                    .arnegro16{font-family:Georgia,"Times New Roman",Times,serif; font-weight:bold; font-size:small;}
+                    .noteInfo{display: block; color: gray}
-                    .tbazull2{font-family:"trebuchet ms",Arial,Helvetica,sans-serif; color:#336699; font-size:xx-small;}
-                    .tbgrisf11{font-family:"trebuchet ms",Arial,Helvetica,sans-serif; color: #666666; font-size:xx-small;}
-                    .verrojo13{font-family:"trebuchet ms",Arial,Helvetica,sans-serif; color: #CC0033; font-size:xx-small;}
-                    .trnegro13{font-family:"trebuchet ms",Arial,Helvetica,sans-serif; font-size:xx-small;}
-                    .txt-fotogaleria{font-family:"trebuchet ms",Arial,Helvetica,sans-serif; font-size:xx-small;}
                 '''
-    keep_only_tags = [ dict(name='table', attrs={'width':"633"}),dict(name='table', attrs={'width':"629"}),]
+    keep_only_tags = [ dict(name='div', attrs={'id':'noteContent'})]
-
+    remove_tags_after = dict(attrs={'class':'noteText'})
     remove_tags  = [
-                    dict(name='table', attrs={'bgcolor':"#f5f5f5"}),
+                     dict(attrs={'class':'noteExtras'}),
-                     dict(name='td', attrs={'bgcolor':"#f7f8f9"}),
+                     dict(name=['meta','iframe','base','embed','object']),
-                    dict(name='td', attrs={'bgcolor':"#f5f5f5"}),
+                     dict(attrs={'id':'tm_box'})
-                    dict(name='table', attrs={'width':"302"}),
+                   ]
-                    dict(name='table', attrs={'width':"214"}),
+    remove_attributes=['lang','onclick']
-                    dict(name='table', attrs={'width':"112"}),
-                    dict(name='table', attrs={'width':"980"}),
-                    dict(name='td', attrs={'height':"1"}),
-                    dict(name='td', attrs={'height':"4"}),
-                     dict(name='td', attrs={'height':"20"}),
-                     dict(name='td', attrs={'height':"10"}),
-                    dict(name='td', attrs={'class':["trrojo11","trbris11","trrojo12","arrojo12s","tbazul13"]}),
-                    dict(name='div', attrs={'id':["mapg","ver_off_todosloscom","todosloscom"]}),
-                    dict(name='span', attrs={'class':["trazul18b","trrojo11","trnaranja11","trbris11","georojo18b","geogris18"]}),
-                    dict(name='span', attrs={'class':["detalles-opinion"]}),
-                    dict(name='a', attrs={'class':["arnaranja12b","trbris11","arazul12rel","trrojo10"]}),
-                    dict(name='img', src = "/img/icono_imprimir.gif"),
-                    dict(name='img', src = "/img/icono_enviar_mail.gif"),
-                    dict(name='img', src = "/img/icono_fuente_g.gif"),
-                    dict(name='img', src = "/img/icono_fuente_m.gif"),
-                    dict(name='img', src = "/img/icono_fuente_c.gif"),
-                    dict(name='img', src = "/img/icono_compartir.gif"),
-                    dict(name='img', src = "/img/icono_enviar_coment.gif"),
-                     dict(name='img', src = "http://www.eluniversal.com.mx/n_img/bot-notasrel.gif"),
-                    dict(name='img', src = "http://www.eluniversal.com.mx/n_img/fr.gif"),
-                    dict(name='img', src = "/img/espiral2.gif"),
-                    dict(name='img', src = "http://www.eluniversal.com.mx/n_img/b"),
-                    dict(name='img', src = "/img/icono_enviar_coment.gifot-notasrel.gif"),
-                    dict(name='img', src = "/n_img/icono_tipo3.gif"),
-                    dict(name='img', src = "/n_img/icono_tipo2.gif"),
-                    dict(name='img', src = "/n_img/icono_print.gif"),
-                    dict(name='img', src = "/n_img/icono_mail2.gif"),
-                    dict(name='img', src = "/n_img/im-comentarios-2a.gif"),
-                    dict(name='img', src = "/n_img/im-comentarios-1a.gif"),
-                    dict(name='img', src = "/img/icono_coment.gif"),
-                     dict(name='img', src = "http://www.eluniversal.com.mx/n_img/bot-sitiosrel.gif"),
-                    dict(name='img', src = "/n_img/icono_tipomenos.gif"),
-                    dict(name='img', src = "/img/futbol/19.jpg"),
-                    dict(name='img', alt = "Facebook"),
-                    dict(name='img', alt = "Twitter"),
-                    dict(name='img', alt = "Google"),
-                    dict(name='img', alt = "LinkedIn"),
-                    dict(name='img', alt = "Viadeo"),
-                    dict(name='img', alt = "Digg"),
-                    dict(name='img', alt = "Delicious"),
-                    dict(name='img', alt = "Meneame"),
-                    dict(name='img', alt = "Yahoo"),
-                    dict(name='img', alt = "Technorati"),
-                    dict(name='a',text =["Compartir","Facebook","Twitter","Google","LinkedIn","Viadeo","Digg","Delicious","Meneame","Yahoo","Technorati"]),
-                    dict(name='select'),
-                    dict(name='a', attrs={'class':"tbgriscompartir"}),
-                    ]
 
     feeds = [
               (u'Minuto por Minuto', u'http://www.eluniversal.com.mx/rss/universalmxm.xml' )
@@ -101,25 +49,3 @@ class ElUniversal(BasicNewsRecipe):
              ,(u'Computacion'      , u'http://www.eluniversal.com.mx/rss/computo.xml'      )
              ,(u'Sociedad'         , u'http://www.eluniversal.com.mx/rss/sociedad.xml'     )
             ]
-
-   # def print_version(self, url):
-   #     return url.replace('/notas/','/notas/vi_')
-
-    def preprocess_html(self, soup):
-       mtag = '<meta http-equiv="Content-Language" content="es-MX"/><meta http-equiv="Content-Type" content="text/html; charset=utf-8">'
-       soup.head.insert(0,mtag)
-       for tag in soup.findAll(name='td',attrs={'class': 'arazul50'}):
-                 tag.insert(0,"<h1>")
-                 tag.insert(2,"</h1>")
-
-       return soup
-
-    def postprocess_html(self, soup,first):
-
-         for tag in soup.findAll(name=['table', 'span','i']):
-                tag.name = 'div'
-         for item in soup.findAll(align = "right"):
-                del item['align']
-
-         return soup
-

resources/recipes/elpais_impreso.recipe

@@ -1,86 +1,95 @@
-# -*- coding: utf-8 -*-
 __license__   = 'GPL v3'
 __copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
 '''
-www.elpais.com/diario/
+www.elpais.com
 '''
 
-from calibre import strftime
 from calibre.web.feeds.news import BasicNewsRecipe
 
-class ElPaisImpresa(BasicNewsRecipe):
+class ElPais_RSS(BasicNewsRecipe):
-    title                 = u'El Pa\xeds - edicion impresa'
+    title                 = 'El Pais'
     __author__            = 'Darko Miletic'
-    description           = u'el periodico global en Espa\xf1ol'
+    description           = 'el periodico global en Castellano'
     publisher             = 'EDICIONES EL PAIS, S.L.'
-    category              = 'news, politics,Spain,actualidad,noticias,informacion,videos,fotografias,audios,graficos,nacional,internacional,deportes,economia,tecnologia,cultura,gente,television,sociedad,opinion,blogs,foros,chats,encuestas,entrevistas,participacion'
+    category              = 'news, politics, finances, world, spain'
+    oldest_article        = 2
+    max_articles_per_feed = 200
     no_stylesheets        = True
-    encoding              = 'latin1'
+    encoding              = 'cp1252'
     use_embedded_content  = False
-    language              = 'es'
+    language              = 'es_ES'
+    remove_empty_feeds    = True
     publication_type      = 'newspaper'
-    masthead_url          = 'http://www.elpais.com/im/tit_logo_global.gif'
+    masthead_url          = 'http://www.elpais.com/im/tit_logo.gif'
-    index                 = 'http://www.elpais.com/diario/'
+    extra_css             = """
-    extra_css             = ' p{text-align: justify} body{ text-align: left; font-family: Georgia,"Times New Roman",Times,serif } h2{font-family: Arial,Helvetica,sans-serif} img{margin-bottom: 0.4em} '
+                               body{font-family: Georgia,"Times New Roman",Times,serif }
+                               h3{font-family: Arial,Helvetica,sans-serif}
+                               img{margin-bottom: 0.4em; display:block}
+                            """
 
     conversion_options = {
-                          'comment'      : description
+                          'comment'   : description
-                        , 'tags'         : category
+                        , 'tags'      : category
-                        , 'publisher'    : publisher
+                        , 'publisher' : publisher
-                        , 'language'     : language
+                        , 'language'  : language
                         }
 
-    feeds          = [
+    keep_only_tags    = [dict(attrs={'class':['cabecera_noticia estirar','cabecera_noticia','','contenido_noticia']})]
-                        (u'Internacional'     , index + u'internacional/'     )
+    remove_tags       = [
-                       ,(u'Espa\xf1a'            , index + u'espana/'            )
+                           dict(name=['meta','link','base','iframe','embed','object'])
-                       ,(u'Economia'          , index + u'economia/'          )
+                          ,dict(attrs={'class':['info_complementa','estructura_2col_der','votos estirar','votos']})
-                       ,(u'Opinion'           , index + u'opinion/'           )
+                          ,dict(attrs={'id':'utilidades'})
-                       ,(u'Vi\xf1etas'           , index + u'vineta/'            )
+                        ]
-                       ,(u'Sociedad'          , index + u'sociedad/'          )
+    remove_tags_after = dict(attrs={'id':'utilidades'})
-                       ,(u'Cultura'           , index + u'cultura/'           )
+    remove_attributes = ['lang','border','width','height']
-                       ,(u'Tendencias'        , index + u'tendencias/'        )
-                       ,(u'Gente'             , index + u'gente/'             )
-                       ,(u'Obituarios'        , index + u'obituarios/'        )
-                       ,(u'Deportes'          , index + u'deportes/'          )
-                       ,(u'Pantallas'         , index + u'radioytv/'          )
-                       ,(u'Ultima'            , index + u'ultima/'            )
-                       ,(u'Educacion'         , index + u'educacion/'         )
-                       ,(u'Saludo'            , index + u'salud/'             )
-                       ,(u'Ciberpais'         , index + u'ciberpais/'         )
-                       ,(u'EP3'               , index + u'ep3/'               )
-                       ,(u'Cine'              , index + u'cine/'              )
-                       ,(u'Babelia'           , index + u'babelia/'           )
-                       ,(u'El viajero'        , index + u'viajero/'           )
-                       ,(u'Negocios'          , index + u'negocios/'          )
-                       ,(u'Domingo'           , index + u'domingo/'           )
-                       ,(u'El Pais semanal'   , index + u'eps/'               )
-                       ,(u'Quadern Catalunya' , index + u'quadern-catalunya/' )
-                     ]
 
-    keep_only_tags=[dict(attrs={'class':['cabecera_noticia','contenido_noticia']})]
+    feeds = [
-    remove_attributes=['width','height']
+              (u'Lo ultimo'            , u'http://www.elpais.com/rss/feed.html?feedId=17046')
-    remove_tags=[dict(name='link')]
+             ,(u'America Latina'       , u'http://www.elpais.com/rss/feed.html?feedId=17041')
-
+             ,(u'Mexico'               , u'http://www.elpais.com/rss/feed.html?feedId=17042')
-    def parse_index(self):
+             ,(u'Europa'               , u'http://www.elpais.com/rss/feed.html?feedId=17043')
-        totalfeeds = []
+             ,(u'Estados Unidos'       , u'http://www.elpais.com/rss/feed.html?feedId=17044')
-        lfeeds = self.get_feeds()
+             ,(u'Oriente proximo'      , u'http://www.elpais.com/rss/feed.html?feedId=17045')
-        for feedobj in lfeeds:
+             ,(u'Espana'               , u'http://www.elpais.com/rss/feed.html?feedId=1002' )
-            feedtitle, feedurl = feedobj
+             ,(u'Andalucia'            , u'http://www.elpais.com/rss/feed.html?feedId=17057')
-            self.report_progress(0, _('Fetching feed')+' %s...'%(feedtitle if feedtitle else feedurl))
+             ,(u'Catalunia'            , u'http://www.elpais.com/rss/feed.html?feedId=17059')
-            articles = []
+             ,(u'Comunidad Valenciana' , u'http://www.elpais.com/rss/feed.html?feedId=17061')
-            soup = self.index_to_soup(feedurl)
+             ,(u'Madrid'               , u'http://www.elpais.com/rss/feed.html?feedId=1016' )
-            for item in soup.findAll('a',attrs={'class':['g19r003','g19i003','g17r003','g17i003']}):
+             ,(u'Pais Vasco'           , u'http://www.elpais.com/rss/feed.html?feedId=17062')
-                url   = 'http://www.elpais.com' + item['href'].rpartition('/')[0]
+             ,(u'Galicia'              , u'http://www.elpais.com/rss/feed.html?feedId=17063')
-                title = self.tag_to_string(item)
+             ,(u'Opinion'              , u'http://www.elpais.com/rss/feed.html?feedId=1003' )             
-                date  = strftime(self.timefmt)
+             ,(u'Sociedad'             , u'http://www.elpais.com/rss/feed.html?feedId=1004' )             
-                articles.append({
+             ,(u'Deportes'             , u'http://www.elpais.com/rss/feed.html?feedId=1007' )
-                                      'title'      :title
+             ,(u'Cultura'              , u'http://www.elpais.com/rss/feed.html?feedId=1008' )
-                                     ,'date'       :date
+             ,(u'Cine'                 , u'http://www.elpais.com/rss/feed.html?feedId=17052')
-                                     ,'url'        :url
+             ,(u'Literatura'           , u'http://www.elpais.com/rss/feed.html?feedId=17053')
-                                     ,'description':''
+             ,(u'Musica'               , u'http://www.elpais.com/rss/feed.html?feedId=17051')
-                                    })
+             ,(u'Arte'                 , u'http://www.elpais.com/rss/feed.html?feedId=17060')             
-            totalfeeds.append((feedtitle, articles))
+             ,(u'Tecnologia'           , u'http://www.elpais.com/rss/feed.html?feedId=1005' )
-        return totalfeeds
+             ,(u'Economia'             , u'http://www.elpais.com/rss/feed.html?feedId=1006' )
+             ,(u'Ciencia'              , u'http://www.elpais.com/rss/feed.html?feedId=17068')
+             ,(u'Salud'                , u'http://www.elpais.com/rss/feed.html?feedId=17074')
+             ,(u'Ocio'                 , u'http://www.elpais.com/rss/feed.html?feedId=17075')
+             ,(u'Justicia y Leyes'     , u'http://www.elpais.com/rss/feed.html?feedId=17069')
+             ,(u'Guerras y conflictos' , u'http://www.elpais.com/rss/feed.html?feedId=17070')
+             ,(u'Politica'             , u'http://www.elpais.com/rss/feed.html?feedId=17073')
+            ]
 
     def print_version(self, url):
         return url + '?print=1'
+
+    def preprocess_html(self, soup):
+        for item in soup.findAll(style=True):
+            del item['style']
+        for item in soup.findAll('a'):
+            if item.string is not None:
+               tstr = item.string
+               item.replaceWith(tstr)
+            else:
+               item.name='span'
+               for atrs in ['href','target','alt','title']:
+                   if item.has_key(atrs):
+                      del item[atrs]
+        for item in soup.findAll('img',alt=False):
+            item['alt'] = 'image'
+        return soup

resources/recipes/karlsruhe.recipe

@@ -0,0 +1,52 @@
+import re
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class KANewsRecipe(BasicNewsRecipe):
+    title = u'KA-News.de'
+    description = u'Nachrichten aus Karlsruhe, Deutschland und der Welt.'
+    __author__ = 'tfeld'
+    lang='de'
+    no_stylesheets = True
+
+    oldest_article = 7
+    max_articles_per_feed = 100
+
+    feeds = [
+                (u'News aus Karlsruhe', 'http://www.ka-news.de/storage/rss/rss/karlsruhe.xml'),
+                (u'Kulturnachrichten aus Karlsruhe', 'http://www.ka-news.de/storage/rss/rss/kultur.xml'),
+                (u'Durlach: News aus Durlach', 'http://www.ka-news.de/storage/rss/rss/durlach.xml'),
+                (u'Stutensee: News aus Stutensee Blankenloch, Büchig, Friedrichstal, Staffort, Spöck', 'http://www.ka-news.de/storage/rss/rss/stutensee.xml'),
+                (u'Bruchsal: News aus Bruchsal', 'http://www.ka-news.de/storage/rss/rss/bruchsal.xml'),
+                (u'Wirtschaftsnews aus Karlsruhe', 'http://www.ka-news.de/storage/rss/rss/wirtschaft.xml'),
+                (u'ka-news.de - Sport', 'http://www.ka-news.de/storage/rss/rss/sport.xml'),
+                (u'KSC-News - News rund um den KSC', 'http://www.ka-news.de/storage/rss/rss/ksc.xml'),
+                (u'ka-news.de - BG Karlsruhe', 'http://www.ka-news.de/storage/rss/rss/basketball.xml')
+            ]
+
+    preprocess_regexps = [
+        (re.compile(r'width:[0-9]*?px', re.DOTALL|re.IGNORECASE), lambda match: ''),
+    ]
+
+    remove_tags_before = dict(id='artdetail_ueberschrift')
+    remove_tags_after = dict(id='artdetail_unterzeile')
+    remove_tags = [dict(name=['div'], attrs={'class': 'lbx_table'}),
+         dict(name=['div'], attrs={'class': 'lk_zumthema'}),
+         dict(name=['div'], attrs={'class': 'lk_thumb'}),
+         dict(name=['div'], attrs={'class': 'lk_trenner'}),
+         dict(name=['div'], attrs={'class': 'lupen_container'}),
+         dict(name=['script']),
+         dict(name=['span'], attrs={'style': 'display:none;'}),
+         dict(name=['span'], attrs={'class': 'comm_info'}),
+         dict(name=['h3'], attrs={'id': 'artdetail_unterzeile'})]
+
+    # removing style attribute _after_ removing specifig tags above
+    remove_attributes = ['width','height','style']
+
+    extra_css = '''
+         h1{ font-size:large; font-weight:bold;  }
+         h2{ font-size:medium; font-weight:bold; }
+    '''
+
+    def get_cover_url(self):
+        return 'http://www.ka-news.de/storage/scl/techkanews/logos/434447_m1t1w250q75s1v29681_ka-news-Logo_mit_Schatten_transparent.png'
+

resources/recipes/la_republica.recipe

@@ -28,6 +28,8 @@ class LaRepubblica(BasicNewsRecipe):
     recursion             = 10
 
     remove_javascript = True
+    no_stylesheets    = True
+
     def get_article_url(self, article):
         link = article.get('id', article.get('guid', None))
         if link is None:

resources/recipes/salon.recipe

@@ -25,22 +25,20 @@ class Salon_com(BasicNewsRecipe):
 
     feeds = [
         ('News & Politics', 'http://feeds.salon.com/salon/news'),
-        ('War Room', 'http://feeds.salon.com/salon/war_room'),
+        ('War Room', 'http://feeds.feedburner.com/salon/war_room'),
-        ('Arts & Entertainment', 'http://feeds.salon.com/salon/ent'),
+        ('Joan Walsh', 'http://feeds.feedburner.com/Salon_Joan_Walsh'),
-        ('I Like to Watch', 'http://feeds.salon.com/salon/iltw'),
+        ('Glenn Greenwald', 'http://feeds.feedburner.com/salon/greenwald'),
-        ('Beyond Multiplex', 'http://feeds.salon.com/salon/btm'),
-        ('Book Reviews', 'http://feeds.salon.com/salon/books'),
-        ('All Life', 'http://feeds.salon.com/salon/mwt'),
-        ('All Opinion', 'http://feeds.salon.com/salon/opinion'),
-        ('Glenn Greenwald', 'http://feeds.salon.com/salon/greenwald'),
-        ('Garrison Keillor', 'http://dir.salon.com/topics/garrison_keillor/index.rss'),
-        ('Joan Walsh', 'http://www.salon.com/rss/walsh.rss'),
-        ('All Sports', 'http://feeds.salon.com/salon/sports'),
         ('Tech & Business', 'http://feeds.salon.com/salon/tech'),
-        ('How World Works', 'http://feeds.salon.com/salon/htww')
+        ('Ask the Pilot', 'http://feeds.feedburner.com/salon/ask_the_pilot'),
+        ('How World Works', 'http://feeds.feedburner.com/salon/htww'),
+        ('Life', 'http://feeds.feedburner.com/salon/mwt'),
+        ('Broadsheet', 'http://feeds.feedburner.com/salon/broadsheet'),
+        ('Movie Reviews', 'http://feeds.feedburner.com/salon/movie_reviews'),
+        ('Film Salon', 'http://feeds.feedburner.com/Salon/Film_Salon'),
+        ('TV', 'http://feeds.feedburner.com/salon/tv'),
+        ('Books', 'http://feeds.feedburner.com/salon/books')
             ]
 
     def print_version(self, url):
         return url.replace('/index.html', '/print.html')
 
-

resources/recipes/smith.recipe

@@ -17,8 +17,8 @@ class SmithsonianMagazine(BasicNewsRecipe):
     remove_tags = [
        dict(name='iframe'),
        dict(name='div', attrs={'class':'article_sidebar_border'}),
-       dict(name='div', attrs={'id':['article_sidebar_border', 'most-popular_large']}),
+       dict(name='div', attrs={'id':['article_sidebar_border', 'most-popular_large', 'most-popular-body_large']}),
-       #dict(name='ul', attrs={'class':'article-tools'}),
+       ##dict(name='ul', attrs={'class':'article-tools'}),
        dict(name='ul', attrs={'class':'cat-breadcrumb col three last'}),
     ]
 
@@ -37,16 +37,16 @@ class SmithsonianMagazine(BasicNewsRecipe):
 ]
 
     def preprocess_html(self, soup):
-        story = soup.find(name='div', attrs={'id':'article-left'})
+        story = soup.find(name='div', attrs={'id':'article-body'})
-        #td = heading.findParent(name='td')
+        ##td = heading.findParent(name='td')
-        #td.extract()
+        ##td.extract()
         soup = BeautifulSoup('<html><head><title>t</title></head><body></body></html>')
         body = soup.find(name='body')
         body.insert(0, story)
         return soup
 
-    def postprocess_html(self, soup, first):
+    #def postprocess_html(self, soup, first):
-        for p in soup.findAll(id='articlePaginationWrapper'): p.extract()
+        #for p in soup.findAll(id='articlePaginationWrapper'): p.extract()
-        if not first:
+        #if not first:
-             for div in soup.findAll(id='article-head'): div.extract()
+             #for div in soup.findAll(id='article-head'): div.extract()
-        return soup
+        #return soup

resources/recipes/the_week_magazine_free.recipe

@@ -1,17 +1,19 @@
-
 __license__   = 'GPL v3'
-__copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
+__copyright__ = '2010, JOlo'
 '''
 www.theweek.com
 '''
 
 from calibre.web.feeds.news import BasicNewsRecipe
+import re
 
-class TheWeekFree(BasicNewsRecipe):
+class TheWeek(BasicNewsRecipe):
-    title                 = 'The Week Magazine - Free content'
+    title                 = 'The Week Magazine'
-    __author__            = 'Darko Miletic'
+    __author__            = 'Jim Olo'
     description           = "The best of the US and international media.  Daily coverage of commentary and analysis of the day's events, as well as arts, entertainment, people and gossip, and political cartoons."
     publisher             = 'The Week Publications, Inc.'
+    masthead_url           = 'http://test.theweek.com/images/logo_theweek.gif'
+    cover_url              = masthead_url
     category              = 'news, politics, USA'
     oldest_article        = 7
     max_articles_per_feed = 100
@@ -19,31 +21,27 @@ class TheWeekFree(BasicNewsRecipe):
     encoding              = 'utf-8'
     use_embedded_content  = False
     language              = 'en'
+    preprocess_regexps      = [(re.compile(r'<h3><a href=.*</body>', re.DOTALL), lambda match: '</body>')]
+    remove_tags_before    = dict(name='h1')
+    remove_tags_after     = dict(name='div', attrs={'class':'articleSubscribe4free'})
+    remove_tags           = [
+                              dict(name='div', attrs={'class':['floatLeft','imageCaption','slideshowImageAttribution','postDate','utilities','cartoonInfo','left','middle','col300','articleSubscribe4free',' articleFlyout','articleFlyout floatRight','fourFreeBar']})
+                             ,dict(name='div', attrs={'id':['cartoonThumbs','rightColumn','header','partners']})
+                             ,dict(name='ul', attrs={'class':['slideshowNav','hotTopicsList topicList']})
+                              ]
+    remove_attributes     = ['width','height', 'style', 'font', 'color']
+    extra_css = '''
+                h1{font-family:Geneva, Arial, Helvetica, sans-serif;color:#154B7A;}
+                h3{font-size: 14px;color:#999999; font-family:Geneva, Arial, Helvetica, sans-serif;font-weight: bold;}
+                h2{color:#666666; font-family:Geneva, Arial, Helvetica, sans-serif;font-size:small;}
+                p {font-family:Arial,Helvetica,sans-serif;}
+                '''
+    filter_regexps = [r'www\.palmcoastdata\.com']
 
-    conversion_options = {
+    feeds          =  [
-                          'comment'   : description
+               (u'News-Opinion', u'http://theweek.com/section/index/news_opinion.rss'),
-                        , 'tags'      : category
+               (u'Business', u'http://theweek.com/section/index/business.rss'),
-                        , 'publisher' : publisher
+               (u'Arts-Life', u'http://theweek.com/section/index/arts_life.rss'),
-                        , 'language'  : language
+               (u'Cartoons', u'http://theweek.com/section/index/cartoon_wit/0/all-cartoons.rss')
-                        }
+                      ]
-
-    keep_only_tags = [
-                        dict(name=['h1','h2'])
-                      , dict(name='div', attrs={'class':'basefont'})
-                      , dict(name='div', attrs={'id':'slideshowLoader'})
-                     ]
-
-    remove_tags = [
-                     dict(name='div', attrs={'id':['digg_dugg','articleRight','dateHeader']})
-                    ,dict(name=['object','embed','iframe'])
-                  ]
-
-
-    feeds = [
-              (u'News & Opinions'       , u'http://www.theweek.com/section/index/news_opinion.rss')
-             ,(u'Arts & Leisure'        , u'http://www.theweek.com/section/index/arts_leisure.rss')
-             ,(u'Business'              , u'http://www.theweek.com/section/index/business.rss'    )
-             ,(u'Cartoon & Short takes' , u'http://www.theweek.com/section/index/cartoons_wit.rss')
-            ]
-
 

resources/recipes/volksrant.recipe

@@ -1,4 +1,3 @@
-#!/usr/bin/env python
 # vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
 from __future__ import with_statement
 
@@ -29,13 +28,16 @@ class AdvancedUserRecipe1249039563(BasicNewsRecipe):
     language = 'nl'
 
     extra_css      = '''
-                        body{font-family:Arial,Helvetica,sans-serif; font-size:small;}
+                        body{font-family:Arial,Helvetica,sans-serif;font-size:small;}
                         h1{font-size:large;}
                      '''
     '''
       Change Log:
         Date:       10/10/10  - Modified code to include obfuscated to get the print version
         Author:   Tony Stegall
+
+        Date:       01/01/11  - Modified for better results around December/January.
+        Author:   Martin Tarenskeen
     '''
    #######################################################################################################
     temp_files = []
@@ -48,11 +50,17 @@ class AdvancedUserRecipe1249039563(BasicNewsRecipe):
         year = date.today().year
 
         try:
-          response = br.follow_link(url_regex='.*?(%d)(\\/)(article)(\\/)(print)(\\/)'%year, nr = 0)
+            response = br.follow_link(url_regex='.*?(%d)(\\/)(article)(\\/)(print)(\\/)'%year, nr = 0)
-          html = response.read()
+            html = response.read()
         except:
-          response = br.open(url)
+            year = year-1
-          html = response.read()
+            try:
+                response = br.follow_link(url_regex='.*?(%d)(\\/)(article)(\\/)(print)(\\/)'%year, nr = 0)
+                html = response.read()
+            except:
+                response = br.open(url)
+                html = response.read()
+
 
         self.temp_files.append(PersistentTemporaryFile('_fa.html'))
         self.temp_files[-1].write(html)
@@ -76,10 +84,3 @@ class AdvancedUserRecipe1249039563(BasicNewsRecipe):
                       (u'Cultuur', u'http://www.volkskrant.nl/rss/kunst.rss'),
                       (u'Gezondheid & Wetenschap', u'http://www.volkskrant.nl/rss/wetenschap.rss'),
                       (u'Internet & Media', u'http://www.volkskrant.nl/rss/media.rss') ]
-
-
-'''
-example for formating
-'''
-# original url: http://www.volkskrant.nl/vk/nl/2668/Buitenland/article/detail/1031493/2010/10/10/Noord-Korea-ziet-nieuwe-leider.dhtml
-# print url :   http://www.volkskrant.nl/vk/nl/2668/2010/article/print/detail/1031493/Noord-Korea-ziet-nieuwe-leider.dhtml

resources/recipes/wired.recipe

@@ -38,12 +38,12 @@ class Wired(BasicNewsRecipe):
     keep_only_tags = [dict(name='div', attrs={'class':'post'})]
     remove_tags_after = dict(name='div', attrs={'class':'tweetmeme_button'})
     remove_tags = [
-                     dict(name=['object','embed','iframe','link'])
+                     dict(name=['object','embed','iframe','link','meta','base'])
                     ,dict(name='div', attrs={'class':['podcast_storyboard','tweetmeme_button']})
                     ,dict(attrs={'id':'ff_bottom_nav'})
                     ,dict(name='a',attrs={'href':'http://www.wired.com/app'})
                   ]
-    remove_attributes = ['height','width']
+    remove_attributes = ['height','width','lang','border','clear']
 
 
     def parse_index(self):
@@ -78,7 +78,9 @@ class Wired(BasicNewsRecipe):
                divurl = item.find('div',attrs={'class':'feature-header'})
                if divurl:
                    divdesc = item.find('div',attrs={'class':'feature-text'})
-                   url   = 'http://www.wired.com' + divurl.a['href']
+                   url = divurl.a['href']
+                   if not divurl.a['href'].startswith('http://www.wired.com'):
+                      url   = 'http://www.wired.com' + divurl.a['href']
                    title = self.tag_to_string(divurl.a)
                    description = self.tag_to_string(divdesc)
                    date  = strftime(self.timefmt)
@@ -127,5 +129,17 @@ class Wired(BasicNewsRecipe):
     def preprocess_html(self, soup):
         for item in soup.findAll(style=True):
             del item['style']
+        for item in soup.findAll('a'):
+            if item.string is not None:
+               tstr = item.string
+               item.replaceWith(tstr)
+            else:
+               item.name='span'
+               for atrs in ['href','target','alt','title','name','id']:
+                   if item.has_key(atrs):
+                      del item[atrs]
+        for item in soup.findAll('img'):
+            if not item.has_key('alt'):
+               item['alt'] = 'image'            
         return soup
 

setup/installer/osx/app/main.py

@@ -612,8 +612,13 @@ class Py2App(object):
         dmg = os.path.join(destdir, volname+'.dmg')
         if os.path.exists(dmg):
             os.unlink(dmg)
-        subprocess.check_call(['/usr/bin/hdiutil', 'create', '-srcfolder', os.path.abspath(d),
+        tdir = tempfile.mkdtemp()
+        shutil.copytree(d, os.path.join(tdir, os.path.basename(d)),
+                symlinks=True)
+        os.symlink('/Applications', os.path.join(tdir, 'Applications'))
+        subprocess.check_call(['/usr/bin/hdiutil', 'create', '-srcfolder', tdir,
                                '-volname', volname, '-format', format, dmg])
+        shutil.rmtree(tdir)
         if internet_enable:
            subprocess.check_call(['/usr/bin/hdiutil', 'internet-enable', '-yes', dmg])
         size = os.stat(dmg).st_size/(1024*1024.)

setup/publish.py

@@ -43,8 +43,8 @@ class Stage3(Command):
 
    description = 'Stage 3 of the publish process'
    sub_commands = ['upload_user_manual', 'upload_demo', 'sdist',
-            'upload_to_google_code', 'tag_release', 'upload_to_server',
+           'upload_to_mobileread', 'upload_to_google_code',
-            'upload_to_sourceforge', 'upload_to_mobileread',
+           'tag_release', 'upload_to_server', 'upload_to_sourceforge',
    ]
 
 class Stage4(Command):

src/calibre/__init__.py

@@ -254,7 +254,7 @@ def browser(honor_time=True, max_time=2, mobile_browser=False):
     opener.set_handle_refresh(True, max_time=max_time, honor_time=honor_time)
     opener.set_handle_robots(False)
     opener.addheaders = [('User-agent', ' Mozilla/5.0 (Windows; U; Windows CE 5.1; rv:1.8.1a3) Gecko/20060610 Minimo/0.016' if mobile_browser else \
-                          'Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4')]
+                          'Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101210 Gentoo Firefox/3.6.13')]
     http_proxy = get_proxies().get('http', None)
     if http_proxy:
         opener.set_proxies({'http':http_proxy})

src/calibre/constants.py

@@ -2,7 +2,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = 'calibre'
-__version__   = '0.7.35'
+__version__   = '0.7.37'
 __author__    = "Kovid Goyal <kovid@kovidgoyal.net>"
 
 import re

src/calibre/customize/__init__.py

@@ -307,6 +307,14 @@ class CatalogPlugin(Plugin): # {{{
     #:  cli_options parsed in library.cli:catalog_option_parser()
     cli_options = []
 
+    def _field_sorter(self, key):
+        '''
+        Custom fields sort after standard fields
+        '''
+        if key.startswith('#'):
+            return '~%s' % key[1:]
+        else:
+            return key
 
     def search_sort_db(self, db, opts):
 
@@ -315,18 +323,18 @@ class CatalogPlugin(Plugin): # {{{
         if opts.sort_by:
             # 2nd arg = ascending
             db.sort(opts.sort_by, True)
-
         return db.get_data_as_dict(ids=opts.ids)
 
-    def get_output_fields(self, opts):
+    def get_output_fields(self, db, opts):
         # Return a list of requested fields, with opts.sort_by first
-        all_fields = set(
+        all_std_fields = set(
                           ['author_sort','authors','comments','cover','formats',
                            'id','isbn','ondevice','pubdate','publisher','rating',
                            'series_index','series','size','tags','timestamp',
                            'title','uuid'])
+        all_custom_fields = set(db.custom_field_keys())
+        all_fields = all_std_fields.union(all_custom_fields)
 
-        fields = all_fields
         if opts.fields != 'all':
             # Make a list from opts.fields
             requested_fields = set(opts.fields.split(','))
@@ -337,7 +345,7 @@ class CatalogPlugin(Plugin): # {{{
         if not opts.connected_device['is_device_connected'] and 'ondevice' in fields:
             fields.pop(int(fields.index('ondevice')))
 
-        fields.sort()
+        fields = sorted(fields, key=self._field_sorter)
         if opts.sort_by and opts.sort_by in fields:
             fields.insert(0,fields.pop(int(fields.index(opts.sort_by))))
         return fields

src/calibre/customize/builtins.py

@@ -478,7 +478,7 @@ from calibre.devices.teclast.driver import TECLAST_K3, NEWSMY, IPAPYRUS, \
 from calibre.devices.sne.driver import SNE
 from calibre.devices.misc import PALMPRE, AVANT, SWEEX, PDNOVEL, KOGAN, \
         GEMEI, VELOCITYMICRO, PDNOVEL_KOBO, Q600, LUMIREAD, ALURATEK_COLOR, \
-        TREKSTOR, EEEREADER
+        TREKSTOR, EEEREADER, NEXTBOOK
 from calibre.devices.folder_device.driver import FOLDER_DEVICE_FOR_CONFIG
 from calibre.devices.kobo.driver import KOBO
 from calibre.devices.bambook.driver import BAMBOOK
@@ -606,6 +606,7 @@ plugins += [
     BAMBOOK,
     TREKSTOR,
     EEEREADER,
+    NEXTBOOK,
     ITUNES,
 ]
 plugins += [x for x in list(locals().values()) if isinstance(x, type) and \

src/calibre/customize/profiles.py

@@ -439,6 +439,13 @@ class TabletOutput(iPadOutput):
     screen_size = (sys.maxint, sys.maxint)
     comic_screen_size = (sys.maxint, sys.maxint)
 
+class SamsungGalaxy(TabletOutput):
+    name = 'Samsung Galaxy'
+    shortname = 'galaxy'
+    description = _('Intended for the Samsung Galaxy and similar tablet devices with '
+            'a resolution of 600x1280')
+    screen_size = comic_screen_size = (600, 1280)
+
 class SonyReaderOutput(OutputProfile):
 
     name        = 'Sony Reader'
@@ -617,6 +624,8 @@ class KindleDXOutput(OutputProfile):
     #comic_screen_size         = (741, 1022)
     supports_mobi_indexing = True
     periodical_date_in_title = False
+	missing_char = u'x\u2009'
+	empty_ratings_char = u'\u2606'
     ratings_char = u'\u2605'
     read_char = u'\u2713'
     mobi_ems_per_blockquote = 2.0
@@ -707,7 +716,7 @@ class BambookOutput(OutputProfile):
 output_profiles = [OutputProfile, SonyReaderOutput, SonyReader300Output,
         SonyReader900Output, MSReaderOutput, MobipocketOutput, HanlinV3Output,
         HanlinV5Output, CybookG3Output, CybookOpusOutput, KindleOutput,
-        iPadOutput, KoboReaderOutput, TabletOutput,
+        iPadOutput, KoboReaderOutput, TabletOutput, SamsungGalaxy,
         SonyReaderLandscapeOutput, KindleDXOutput, IlliadOutput,
         IRexDR1000Output, IRexDR800Output, JetBook5Output, NookOutput,
         BambookOutput, NookColorOutput]

src/calibre/devices/android/driver.py

@@ -27,15 +27,15 @@ class ANDROID(USBMS):
             0x040d : { 0x8510 : [0x0001], 0x0851 : [0x1] },
 
             # Motorola
-            0x22b8 : { 0x41d9 : [0x216], 0x2d67 : [0x100], 0x41db : [0x216],
+            0x22b8 : { 0x41d9 : [0x216], 0x2d61: [0x100], 0x2d67 : [0x100],
-                0x4285 : [0x216], 0x42a3 : [0x216] },
+                0x41db : [0x216], 0x4285 : [0x216], 0x42a3 : [0x216] },
 
             # Sony Ericsson
             0xfce : { 0xd12e : [0x0100]},
 
             # Google
             0x18d1 : { 0x4e11 : [0x0100, 0x226, 0x227], 0x4e12: [0x0100, 0x226,
-                0x227]},
+                0x227], 0x4e21: [0x0100, 0x226, 0x227]},
 
             # Samsung
             0x04e8 : { 0x681d : [0x0222, 0x0223, 0x0224, 0x0400],
@@ -64,7 +64,8 @@ class ANDROID(USBMS):
     WINDOWS_MAIN_MEM = ['ANDROID_PHONE', 'A855', 'A853', 'INC.NEXUS_ONE',
             '__UMS_COMPOSITE', '_MB200', 'MASS_STORAGE', '_-_CARD', 'SGH-I897',
             'GT-I9000', 'FILE-STOR_GADGET', 'SGH-T959', 'SAMSUNG_ANDROID',
-            'SCH-I500_CARD', 'SPH-D700_CARD', 'MB810', 'GT-P1000']
+            'SCH-I500_CARD', 'SPH-D700_CARD', 'MB810', 'GT-P1000', 'DESIRE',
+            'SGH-T849', '_MB300']
     WINDOWS_CARD_A_MEM = ['ANDROID_PHONE', 'GT-I9000_CARD', 'SGH-I897',
             'FILE-STOR_GADGET', 'SGH-T959', 'SAMSUNG_ANDROID', 'GT-P1000_CARD']
 

src/calibre/devices/bambook/driver.py

@@ -37,7 +37,8 @@ class BAMBOOK(DeviceConfig, DevicePlugin):
     BCD = None
     CAN_SET_METADATA = False
     THUMBNAIL_HEIGHT = 155
-    EXTRA_CUSTOMIZATION_MESSAGE = _("Device IP Address")
+    EXTRA_CUSTOMIZATION_MESSAGE = \
+        _("Device IP Address (restart calibre after changing)")
 
     icon = I("devices/bambook.png")
 #    OPEN_FEEDBACK_MESSAGE = _(
@@ -70,7 +71,7 @@ class BAMBOOK(DeviceConfig, DevicePlugin):
         self.bambook.Connect(ip = self.ip, timeout = 10000)
         if self.bambook.GetState() != CONN_CONNECTED:
             self.bambook = None
-            raise OpenFeedback(_("Unable to connect to Bambook. \n" 
+            raise OpenFeedback(_("Unable to connect to Bambook. \n"
                                  "If you are trying to connect via Wi-Fi, "
                                  "please make sure the IP address of Bambook has been correctly configured."))
         self.is_connected = True

src/calibre/devices/eb600/driver.py

@@ -230,7 +230,7 @@ class POCKETBOOK301(USBMS):
 class POCKETBOOK602(USBMS):
 
     name = 'PocketBook Pro 602/902 Device Interface'
-    description    = _('Communicate with the PocketBook 602 reader.')
+    description    = _('Communicate with the PocketBook 602/603/902/903 reader.')
     author         = 'Kovid Goyal'
     supported_platforms = ['windows', 'osx', 'linux']
     FORMATS = ['epub', 'fb2', 'prc', 'mobi', 'pdf', 'djvu', 'rtf', 'chm',
@@ -244,7 +244,7 @@ class POCKETBOOK602(USBMS):
     BCD         = [0x0324]
 
     VENDOR_NAME = ''
-    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = ['PB602', 'PB902']
+    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = ['PB602', 'PB603', 'PB902', 'PB903']
 
 class POCKETBOOK701(USBMS):
 

src/calibre/devices/folder_device/driver.py

@@ -18,9 +18,9 @@ class FOLDER_DEVICE_FOR_CONFIG(USBMS):
     supported_platforms = ['windows', 'osx', 'linux']
     FORMATS     = ['epub', 'fb2', 'mobi', 'azw', 'lrf', 'tcr', 'pmlz', 'lit',
                    'rtf', 'rb', 'pdf', 'oeb', 'txt', 'pdb', 'prc']
-    VENDOR_ID   = 0xffff
+    VENDOR_ID   = [0xffff]
-    PRODUCT_ID  = 0xffff
+    PRODUCT_ID  = [0xffff]
-    BCD         = 0xffff
+    BCD         = [0xffff]
     DEVICE_PLUGBOARD_NAME = 'FOLDER_DEVICE'
 
 
@@ -34,9 +34,9 @@ class FOLDER_DEVICE(USBMS):
     supported_platforms = ['windows', 'osx', 'linux']
     FORMATS     = FOLDER_DEVICE_FOR_CONFIG.FORMATS
 
-    VENDOR_ID   = 0xffff
+    VENDOR_ID   = [0xffff]
-    PRODUCT_ID  = 0xffff
+    PRODUCT_ID  = [0xffff]
-    BCD         = 0xffff
+    BCD         = [0xffff]
     DEVICE_PLUGBOARD_NAME = 'FOLDER_DEVICE'
 
     THUMBNAIL_HEIGHT = 68 # Height for thumbnails on device

src/calibre/devices/iriver/driver.py

@@ -20,11 +20,11 @@ class IRIVER_STORY(USBMS):
     FORMATS     = ['epub', 'fb2', 'pdf', 'djvu', 'txt']
 
     VENDOR_ID   = [0x1006]
-    PRODUCT_ID  = [0x4023, 0x4025]
+    PRODUCT_ID  = [0x4023, 0x4024, 0x4025]
     BCD         = [0x0323]
 
     VENDOR_NAME = 'IRIVER'
-    WINDOWS_MAIN_MEM = ['STORY', 'STORY_EB05']
+    WINDOWS_MAIN_MEM = ['STORY', 'STORY_EB05', 'STORY_WI-FI']
     WINDOWS_CARD_A_MEM = ['STORY', 'STORY_SD']
 
     #OSX_MAIN_MEM = 'Kindle Internal Storage Media'

src/calibre/devices/kobo/driver.py

@@ -33,7 +33,7 @@ class KOBO(USBMS):
     booklist_class = CollectionsBookList
 
     # Ordered list of supported formats
-    FORMATS     = ['epub', 'pdf']
+    FORMATS     = ['epub', 'pdf', 'txt', 'cbz', 'cbr']
     CAN_SET_METADATA = ['collections']
 
     VENDOR_ID   = [0x2237]
@@ -409,7 +409,7 @@ class KOBO(USBMS):
             else:
                 ContentType = 901
         else: # if extension == '.html' or extension == '.txt':
-            ContentType = 999 # Yet another hack: to get around Kobo changing how ContentID is stored
+            ContentType = 901 # Yet another hack: to get around Kobo changing how ContentID is stored
         return ContentType
 
     def path_from_contentid(self, ContentID, ContentType, MimeType, oncard):

src/calibre/devices/misc.py

@@ -264,3 +264,23 @@ class EEEREADER(USBMS):
     VENDOR_NAME = 'LINUX'
     WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = 'FILE-STOR_GADGET'
 
+class NEXTBOOK(USBMS):
+
+    name           = 'Nextbook device interface'
+    gui_name       = 'Nextbook'
+    description    = _('Communicate with the Nextbook Reader')
+    author         = 'Kovid Goyal'
+    supported_platforms = ['windows', 'osx', 'linux']
+
+    # Ordered list of supported formats
+    FORMATS     = ['epub', 'fb2', 'txt', 'pdf']
+
+    VENDOR_ID   = [0x05e3]
+    PRODUCT_ID  = [0x0726]
+    BCD         = [0x021a]
+
+    EBOOK_DIR_MAIN = ''
+
+    VENDOR_NAME = 'NEXT2'
+    WINDOWS_MAIN_MEM = '1.0.14'
+

src/calibre/devices/scanner.py

@@ -30,6 +30,12 @@ class Drive(str):
         typ.order = order
         return typ
 
+def drivecmp(a, b):
+    ans = cmp(getattr(a, 'order', 0), getattr(b, 'order', 0))
+    if ans == 0:
+        ans = cmp(a, b)
+    return ans
+
 
 class WinPNPScanner(object):
 
@@ -57,7 +63,13 @@ class WinPNPScanner(object):
         order = 0
         match = re.search(r'REV_.*?&(\d+)#', pnp_id)
         if match is None:
-            match = re.search(r'REV_.*?&(\d+)', pnp_id)
+            # Windows XP
+            # On the Nook Color this is the last digit
+            #
+            # USBSTOR\DISK&VEN_B&N&PROD_EBOOK_DISK&REV_0100\7&13EAFDB8&0&2004760017462009&1
+            # USBSTOR\DISK&VEN_B&N&PROD_EBOOK_DISK&REV_0100\7&13EAFDB8&0&2004760017462009&0
+            #
+            match = re.search(r'REV_.*&(\d+)', pnp_id)
         if match is not None:
             order = int(match.group(1))
         return order

src/calibre/devices/usbms/books.py

@@ -140,11 +140,19 @@ class CollectionsBookList(BookList):
         all_by_author = ''
         all_by_title = ''
         ca = []
+        all_by_something = []
         for c in collection_attributes:
-            if c.startswith('aba:') and c[4:]:
+            if c.startswith('aba:') and c[4:].strip():
                 all_by_author = c[4:].strip()
-            elif c.startswith('abt:') and c[4:]:
+            elif c.startswith('abt:') and c[4:].strip():
                 all_by_title = c[4:].strip()
+            elif c.startswith('abs:') and c[4:].strip():
+                name = c[4:].strip()
+                sby = self.in_category_sort_rules(name)
+                if sby is None:
+                    sby = name
+                if name and sby:
+                    all_by_something.append((name, sby))
             else:
                 ca.append(c.lower())
         collection_attributes = ca
@@ -251,6 +259,10 @@ class CollectionsBookList(BookList):
                 if all_by_title not in collections:
                     collections[all_by_title] = {}
                 collections[all_by_title][lpath] = (book, tsval, asval)
+            for (n, sb) in all_by_something:
+                if n not in collections:
+                    collections[n] = {}
+                collections[n][lpath] = (book, book.get(sb, ''), tsval)
 
         # Sort collections
         result = {}

src/calibre/devices/usbms/device.py

@@ -11,7 +11,7 @@ intended to be subclassed with the relevant parts implemented for a particular
 device. This class handles device detection.
 '''
 
-import os, subprocess, time, re, sys, glob, operator
+import os, subprocess, time, re, sys, glob
 from itertools import repeat
 
 from calibre.devices.interface import DevicePlugin
@@ -225,7 +225,7 @@ class Device(DeviceConfig, DevicePlugin):
         return False
 
     def open_windows(self):
-        from calibre.devices.scanner import win_pnp_drives
+        from calibre.devices.scanner import win_pnp_drives, drivecmp
 
         time.sleep(5)
         drives = {}
@@ -263,7 +263,7 @@ class Device(DeviceConfig, DevicePlugin):
         if self.WINDOWS_MAIN_MEM in (self.WINDOWS_CARD_A_MEM,
                 self.WINDOWS_CARD_B_MEM) or \
                 self.WINDOWS_CARD_A_MEM == self.WINDOWS_CARD_B_MEM:
-            letters = sorted(drives.values(), key=operator.attrgetter('order'))
+            letters = sorted(drives.values(), cmp=drivecmp)
             drives = {}
             for which, letter in zip(['main', 'carda', 'cardb'], letters):
                 drives[which] = letter

src/calibre/ebooks/compression/tcr.py

@@ -6,11 +6,118 @@ __docformat__ = 'restructuredtext en'
 
 import re
 
+class TCRCompressor(object):
+    '''
+    TCR compression takes the form header+code_dict+coded_text.
+    The header is always "!!8-Bit!!". The code dict is a list of 256 strings.
+    The list takes the form 1 byte length and then a string. Each position in
+    The list corresponds to a code found in the file. The coded text is
+    string of characters values. for instance the character Q represents the
+    value 81 which corresponds to the string in the code list at position 81.
+    '''
+    
+    def _reset(self):
+        # List of indexes in the codes list that are empty and can hold new codes
+        self.unused_codes = set()
+        self.coded_txt = ''
+        # Generate initial codes from text.
+        # The index of the list will be the code that represents the characters at that location
+        # in the list
+        self.codes = []
+        
+    def _combine_codes(self):
+        '''
+        Combine two codes that always appear in pair into a single code.
+        The intent is to create more unused codes.
+        '''
+        possible_codes = []
+        a_code = set(re.findall('(?msu).', self.coded_txt))
+        
+        for code in a_code:
+            single_code = set(re.findall('(?msu)%s.' % re.escape(code), self.coded_txt))
+            if len(single_code) == 1:
+                possible_codes.append(single_code.pop())
+                
+        for code in possible_codes:
+            self.coded_txt = self.coded_txt.replace(code, code[0])
+            self.codes[ord(code[0])] = '%s%s' % (self.codes[ord(code[0])], self.codes[ord(code[1])])
+        
+    def _free_unused_codes(self):
+        '''
+        Look for codes that do no not appear in the coded text and add them to
+        the list of free codes.
+        '''
+        for i in xrange(256):
+            if i not in self.unused_codes:
+                if chr(i) not in self.coded_txt:
+                    self.unused_codes.add(i)
+    
+    def _new_codes(self):
+        '''
+        Create new codes from codes that occur in pairs often.
+        '''
+        possible_new_codes = list(set(re.findall('(?msu)..', self.coded_txt)))
+        new_codes_count = []
+        
+        for c in possible_new_codes:
+            count = self.coded_txt.count(c)
+            # Less than 3 occurrences will not produce any size reduction.
+            if count > 2:
+                new_codes_count.append((c, count))
+        
+        # Arrange the codes in order of least to most occurring. 
+        possible_new_codes = [x[0] for x in sorted(new_codes_count, key=lambda c: c[1])]
+        
+        return possible_new_codes
+        
+    def compress(self, txt):
+        self._reset()
+        
+        self.codes = list(set(re.findall('(?msu).', txt)))
+        
+        # Replace the text with their corresponding code
+        for c in txt:
+            self.coded_txt += chr(self.codes.index(c))
+        
+        # Zero the unused codes and record which are unused.
+        for i in range(len(self.codes), 256):
+            self.codes.append('')
+            self.unused_codes.add(i)
+            
+        self._combine_codes()
+        possible_codes = self._new_codes()
+
+        while possible_codes and self.unused_codes:
+            while possible_codes and self.unused_codes:
+                unused_code = self.unused_codes.pop()
+                # Take the last possible codes and split it into individual
+                # codes. The last possible code is the most often occurring.
+                code1, code2 = possible_codes.pop()
+                self.codes[unused_code] = '%s%s' % (self.codes[ord(code1)], self.codes[ord(code2)])
+                self.coded_txt = self.coded_txt.replace('%s%s' % (code1, code2), chr(unused_code))
+            self._combine_codes()
+            self._free_unused_codes()
+            possible_codes = self._new_codes()
+
+        self._free_unused_codes()
+
+        # Generate the code dictionary.
+        code_dict = []
+        for i in xrange(0, 256):
+            if i in self.unused_codes:
+                code_dict.append(chr(0))
+            else:
+                code_dict.append(chr(len(self.codes[i])) + self.codes[i])
+    
+        # Join the identifier with the dictionary and coded text.
+        return '!!8-Bit!!'+''.join(code_dict)+self.coded_txt
+
+
 def decompress(stream):
         txt = []
         stream.seek(0)
         if stream.read(9) != '!!8-Bit!!':
-            raise ValueError('File %s contaions an invalid TCR header.' % stream.name)
+            raise ValueError('File %s contains an invalid TCR header.' % stream.name)
 
         # Codes that the file contents are broken down into.
         entries = []
@@ -26,101 +133,6 @@ def decompress(stream):
 
         return ''.join(txt)
 
-
+def compress(txt):
-def compress(txt, level=5):
+    t = TCRCompressor()
-    '''
+    return t.compress(txt)
-    TCR compression takes the form header+code_list+coded_text.
-    The header is always "!!8-Bit!!". The code list is a list of 256 strings.
-    The list takes the form 1 byte length and then a string. Each position in
-    The list corresponds to a code found in the file. The coded text is
-    string of characters vaules. for instance the character Q represents the
-    value 81 which corresponds to the string in the code list at position 81.
-    '''
-    # Turn each unique character into a coded value.
-    # The code of the string at a given position are represented by the position
-    # they occupy in the list.
-    codes = list(set(re.findall('(?msu).', txt)))
-    for i in range(len(codes), 256):
-        codes.append('')
-    # Set the compression level.
-    if level <= 1:
-        new_length = 256
-    if level >= 10:
-        new_length = 1
-    else:
-        new_length = int(256 * (10 - level) * .1)
-    new_length = 1 if new_length < 1 else new_length
-    # Replace txt with codes.
-    coded_txt = ''
-    for c in txt:
-        coded_txt += chr(codes.index(c))
-    txt = coded_txt
-    # Start compressing the text.
-    new = True
-    merged = True
-    while new or merged:
-        # Merge codes that always follow another code
-        merge = []
-        merged = False
-        for i in xrange(256):
-            if codes[i] != '':
-                # Find all codes that are next to i.
-                fall = list(set(re.findall('(?msu)%s.' % re.escape(chr(i)), txt)))
-                # 1 if only one code comes after i.
-                if len(fall) == 1:
-                    # We are searching codes and each code is always 1 character.
-                    j = ord(fall[0][1:2])
-                    # Only merge if the total length of the string represented by
-                    # code is less than 256.
-                    if len(codes[i]) + len(codes[j]) < 256:
-                        merge.append((i, j))
-        if merge:
-            merged = True
-            for i, j in merge:
-                # Merge the string for j into the string for i.
-                if i == j:
-                    # Don't use += here just in case something goes wrong. This
-                    # will prevent out of control memory consumption. This is
-                    # unecessary but when creating this routine it happened due
-                    # to an error.
-                    codes[i] = codes[i] + codes[i]
-                else:
-                    codes[i] = codes[i] + codes[j]
-                txt = txt.replace(chr(i)+chr(j), chr(i))
-                if chr(j) not in txt:
-                    codes[j] = ''
-        new = False
-        if '' in codes:
-            # Create a list of codes based on combinations of codes that are next
-            # to each other. The amount of savings for the new code is calculated.
-            new_codes = []
-            for c in list(set(re.findall('(?msu)..', txt))):
-                i = ord(c[0:1])
-                j = ord(c[1:2])
-                if codes[i]+codes[j] in codes:
-                    continue
-                savings = txt.count(chr(i)+chr(j)) - len(codes[i]) - len(codes[j])
-                if savings > 2 and len(codes[i]) + len(codes[j]) < 256:
-                    new_codes.append((savings, i, j, codes[i], codes[j]))
-            if new_codes:
-                new = True
-                # Sort the codes from highest savings to lowest.
-                new_codes.sort(lambda x, y: -1 if x[0] > y[0] else 1 if x[0] < y[0] else 0)
-                # The shorter new_length the more chances time merging will happen
-                # giving more changes for better codes to be created. However,
-                # the shorter new_lengh the longer it will take to compress.
-                new_codes = new_codes[:new_length]
-                for code in new_codes:
-                    if '' not in codes:
-                        break
-                    c = codes.index('')
-                    codes[c] = code[3]+code[4]
-                    txt = txt.replace(chr(code[1])+chr(code[2]), chr(c))
-    # Generate the code dictionary.
-    header = []
-    for code in codes:
-        header.append(chr(len(code))+code)
-    for i in xrange(len(header), 256):
-        header.append(chr(0))
-    # Join the identifier with the dictionary and coded text.
-    return '!!8-Bit!!'+''.join(header)+txt

src/calibre/ebooks/fb2/fb2ml.py

@@ -200,8 +200,10 @@ class FB2MLizer(object):
                         im = Image()
                         im.load(item.data)
                         im.set_compression_quality(70)
-                        data = im.export('jpg')
+                        imdata = im.export('jpg')
-                    raw_data = b64encode(data)
+                        raw_data = b64encode(imdata)
+                    else:
+                        raw_data = b64encode(item.data)
                     # Don't put the encoded image on a single line.
                     data = ''
                     col = 1

src/calibre/ebooks/fb2/input.py

@@ -41,9 +41,12 @@ class FB2Input(InputFormatPlugin):
         from calibre.ebooks.metadata.opf2 import OPFCreator
         from calibre.ebooks.metadata.meta import get_metadata
         from calibre.ebooks.oeb.base import XLINK_NS, XHTML_NS, RECOVER_PARSER
+        from calibre.ebooks.chardet import xml_to_unicode
         NAMESPACES = {'f':FB2NS, 'l':XLINK_NS}
         log.debug('Parsing XML...')
         raw = stream.read().replace('\0', '')
+        raw = xml_to_unicode(raw, strip_encoding_pats=True,
+            assume_utf8=True)[0]
         try:
             doc = etree.fromstring(raw)
         except etree.XMLSyntaxError:

src/calibre/ebooks/fb2/output.py

@@ -35,7 +35,7 @@ class FB2Output(OutputFormatPlugin):
             rasterizer = SVGRasterizer()
             rasterizer(oeb_book, opts)
         except Unavailable:
-            self.log.warn('SVG rasterizer unavailable, SVG will not be converted')
+            log.warn('SVG rasterizer unavailable, SVG will not be converted')
 
         linearize_jacket(oeb_book)
 

src/calibre/ebooks/metadata/book/base.py

@@ -159,6 +159,11 @@ class Metadata(object):
         try:
             return self.__getattribute__(field)
         except AttributeError:
+            if field.startswith('#') and field.endswith('_index'):
+                try:
+                    return self.get_extra(field[:-6])
+                except:
+                    pass
             return default
 
     def get_extra(self, field):

src/calibre/ebooks/metadata/fb2.py

@@ -9,6 +9,7 @@ import mimetypes, os
 from base64 import b64decode
 from lxml import etree
 from calibre.ebooks.metadata import MetaInformation
+from calibre.ebooks.chardet import xml_to_unicode
 
 XLINK_NS     = 'http://www.w3.org/1999/xlink'
 def XLINK(name):
@@ -23,7 +24,10 @@ def get_metadata(stream):
     tostring = lambda x : etree.tostring(x, method='text',
             encoding=unicode).strip()
     parser = etree.XMLParser(recover=True, no_network=True)
-    root = etree.fromstring(stream.read(), parser=parser)
+    raw = stream.read()
+    raw = xml_to_unicode(raw, strip_encoding_pats=True,
+            assume_utf8=True)[0]
+    root = etree.fromstring(raw, parser=parser)
     authors, author_sort = [], None
     for au in XPath('//fb2:author')(root):
         fname = lname = author = None

src/calibre/ebooks/metadata/zip.py

@@ -27,7 +27,39 @@ def get_metadata(stream):
                 with TemporaryDirectory() as tdir:
                     with CurrentDir(tdir):
                         path = zf.extract(f)
-                        return get_metadata(open(path, 'rb'), stream_type)
+                        mi = get_metadata(open(path,'rb'), stream_type)
+                        if stream_type == 'opf' and mi.application_id == None:
+                            try:
+                                # zip archive opf files without an application_id were assumed not to have a cover
+                                # reparse the opf and if cover exists read its data from zip archive for the metadata
+                                nmi = zip_opf_metadata(path, zf)
+                                return nmi
+                            except:
+                                pass
+                        return mi
     raise ValueError('No ebook found in ZIP archive')
 
 
+def zip_opf_metadata(opfpath, zf):
+    from calibre.ebooks.metadata.opf2 import OPF
+    if hasattr(opfpath, 'read'):
+        f = opfpath
+        opfpath = getattr(f, 'name', os.getcwd())
+    else:
+        f = open(opfpath, 'rb')
+    opf = OPF(f, os.path.dirname(opfpath))
+    mi = opf.to_book_metadata()
+    # This is broken, in that it only works for
+    # when both the OPF file and the cover file are in the root of the
+    # zip file and the cover is an actual raster image, but I don't care
+    # enough to make it more robust
+    if getattr(mi, 'cover', None):
+        covername = os.path.basename(mi.cover)
+        mi.cover = None
+        names = zf.namelist()
+        if covername in names:
+            fmt = covername.rpartition('.')[-1]
+            data = zf.read(covername)
+            mi.cover_data = (fmt, data)
+    return mi
+

src/calibre/ebooks/mobi/reader.py

@@ -513,11 +513,14 @@ class MobiReader(object):
         mobi_version = self.book_header.mobi_version
         for x in root.xpath('//ncx'):
             x.getparent().remove(x)
+        svg_tags = []
         for i, tag in enumerate(root.iter(etree.Element)):
             tag.attrib.pop('xmlns', '')
             for x in tag.attrib:
                 if ':' in x:
                     del tag.attrib[x]
+            if tag.tag and barename(tag.tag) == 'svg':
+                svg_tags.append(tag)
             if tag.tag and barename(tag.tag.lower()) in \
                 ('country-region', 'place', 'placetype', 'placename',
                     'state', 'city', 'street', 'address', 'content', 'form'):
@@ -628,6 +631,11 @@ class MobiReader(object):
                 cls = cls + (' ' if cls else '') + ncls
                 attrib['class'] = cls
 
+        for tag in svg_tags:
+            p = tag.getparent()
+            if hasattr(p, 'remove'):
+                p.remove(tag)
+
     def create_opf(self, htmlfile, guide=None, root=None):
         mi = getattr(self.book_header.exth, 'mi', self.embedded_mi)
         if mi is None:

src/calibre/ebooks/pdf/writer.py

@@ -25,10 +25,6 @@ from PyQt4.QtWebKit import QWebView
 
 from pyPdf import PdfFileWriter, PdfFileReader
 
-def get_pdf_printer():
-    return QPrinter(QPrinter.HighResolution)
-
-
 def get_custom_size(opts):
     custom_size = None
     if opts.custom_size != None:
@@ -42,12 +38,12 @@ def get_custom_size(opts):
                 custom_size = None
     return custom_size
 
-def setup_printer(opts, for_comic=False):
+def get_pdf_printer(opts, for_comic=False):
     from calibre.gui2 import is_ok_to_use_qt
     if not is_ok_to_use_qt():
         raise Exception('Not OK to use Qt')
 
-    printer = get_pdf_printer()
+    printer = QPrinter(QPrinter.HighResolution)
     custom_size = get_custom_size(opts)
 
     if opts.output_profile.short_name == 'default':
@@ -61,15 +57,22 @@ def setup_printer(opts, for_comic=False):
         h = opts.output_profile.comic_screen_size[1] if for_comic else \
                 opts.output_profile.height
         dpi = opts.output_profile.dpi
-        printer.setPaperSize(QSizeF(float(w) / dpi, float(h)/dpi), QPrinter.Inch)
+        printer.setPaperSize(QSizeF(float(w) / dpi, float(h) / dpi), QPrinter.Inch)
 
-    printer.setPageMargins(0, 0, 0, 0, QPrinter.Point)
+    if for_comic:
+        # Comic pages typically have their own margins, or their background
+        # color is not white, in which case the margin looks bad
+        printer.setPageMargins(0, 0, 0, 0, QPrinter.Point)
+    else:
+        printer.setPageMargins(opts.margin_left, opts.margin_top,
+                opts.margin_right, opts.margin_bottom, QPrinter.Point)
     printer.setOrientation(orientation(opts.orientation))
     printer.setOutputFormat(QPrinter.PdfFormat)
+    printer.setFullPage(True)
     return printer
 
 def get_printer_page_size(opts, for_comic=False):
-    printer = setup_printer(opts, for_comic=for_comic)
+    printer = get_pdf_printer(opts, for_comic=for_comic)
     size =  printer.paperSize(QPrinter.Millimeter)
     return size.width() / 10., size.height() / 10.
 
@@ -154,24 +157,11 @@ class PDFWriter(QObject): # {{{
 
         self.view.load(QUrl.fromLocalFile(item))
 
-    def get_printer(self, set_horz_margins=False):
-        printer = get_pdf_printer()
-        printer.setPaperSize(QSizeF(self.size[0] * 10, self.size[1] * 10), QPrinter.Millimeter)
-        if set_horz_margins:
-            printer.setPageMargins(0., self.opts.margin_top, 0.,
-                    self.opts.margin_bottom, QPrinter.Point)
-        else:
-            printer.setPageMargins(0, 0, 0, 0, QPrinter.Point)
-        printer.setOrientation(orientation(self.opts.orientation))
-        printer.setOutputFormat(QPrinter.PdfFormat)
-        printer.setFullPage(not set_horz_margins)
-        return printer
-
     def _render_html(self, ok):
         if ok:
             item_path = os.path.join(self.tmp_path, '%i.pdf' % len(self.combine_queue))
-            self.logger.debug('\tRendering item %s as %i' % (os.path.basename(str(self.view.url().toLocalFile())), len(self.combine_queue)))
+            self.logger.debug('\tRendering item %s as %i.pdf' % (os.path.basename(str(self.view.url().toLocalFile())), len(self.combine_queue)))
-            printer = self.get_printer(set_horz_margins=True)
+            printer = get_pdf_printer(self.opts)
             printer.setOutputFileName(item_path)
             self.view.print_(printer)
         self._render_book()
@@ -185,7 +175,7 @@ class PDFWriter(QObject): # {{{
         if self.cover_data is None:
             return
         item_path = os.path.join(self.tmp_path, 'cover.pdf')
-        printer = self.get_printer()
+        printer = get_pdf_printer(self.opts)
         printer.setOutputFileName(item_path)
         self.combine_queue.insert(0, item_path)
         p = QPixmap()
@@ -233,16 +223,11 @@ class ImagePDFWriter(object):
             os.remove(f.name)
 
     def render_images(self, outpath, mi, items):
-        printer = get_pdf_printer()
+        printer = get_pdf_printer(self.opts, for_comic=True)
-        printer.setPaperSize(QSizeF(self.size[0] * 10, self.size[1] * 10), QPrinter.Millimeter)
-        printer.setPageMargins(0, 0, 0, 0, QPrinter.Point)
-        printer.setOrientation(orientation(self.opts.orientation))
-        printer.setOutputFormat(QPrinter.PdfFormat)
         printer.setOutputFileName(outpath)
         printer.setDocName(mi.title)
         printer.setCreator(u'%s [%s]'%(__appname__, __version__))
         # Seems to be no way to set author
-        printer.setFullPage(True)
 
         painter = QPainter(printer)
         painter.setRenderHints(QPainter.Antialiasing|QPainter.SmoothPixmapTransform)

src/calibre/ebooks/rtf2xml/field_strings.py

@@ -402,7 +402,7 @@ class FieldStrings:
         Logic:
         self.__link_switch = re.compile(r'\\l\s{1,}(.*?)\s')
         """
-        self.__link_switch = re.compile(r'\\l\s{1,}(.*?)\s')
+        self.__link_switch = re.compile(r'\\l\s{1,}"{0,1}(.*?)"{0,1}\s')
         the_string = name
         match_group = re.search(self.__link_switch, line)
         if match_group:

src/calibre/ebooks/tcr/output.py

@@ -22,11 +22,6 @@ class TCROutput(OutputFormatPlugin):
             level=OptionRecommendation.LOW,
             help=_('Specify the character encoding of the output document. ' \
             'The default is utf-8.')),
-        OptionRecommendation(name='compression_level', recommended_value=5,
-            level=OptionRecommendation.LOW,
-            help=_('Specify the compression level to use. Scale 1 - 10. 1 ' \
-            'being the lowest compression but the fastest and 10 being the ' \
-            'highest compression but the slowest.')),
     ])
 
     def convert(self, oeb_book, output_path, input_plugin, opts, log):
@@ -48,7 +43,7 @@ class TCROutput(OutputFormatPlugin):
         txt = writer.extract_content(oeb_book, opts).encode(opts.output_encoding, 'replace')
 
         log.info('Compressing text...')
-        txt = compress(txt, opts.compression_level)
+        txt = compress(txt)
 
         out_stream.seek(0)
         out_stream.truncate()

src/calibre/ebooks/txt/output.py

@@ -29,8 +29,7 @@ class TXTOutput(OutputFormatPlugin):
         OptionRecommendation(name='output_encoding', recommended_value='utf-8',
             level=OptionRecommendation.LOW,
             help=_('Specify the character encoding of the output document. ' \
-            'The default is utf-8. Note: This option is not honored by all ' \
+            'The default is utf-8.')),
-            'formats.')),
         OptionRecommendation(name='inline_toc',
             recommended_value=False, level=OptionRecommendation.LOW,
             help=_('Add Table of Contents to beginning of the book.')),

src/calibre/gui2/__init__.py

@@ -53,6 +53,8 @@ gprefs.defaults['toolbar_icon_size'] = 'medium'
 gprefs.defaults['toolbar_text'] = 'auto'
 gprefs.defaults['show_child_bar'] = False
 gprefs.defaults['font'] = None
+gprefs.defaults['tags_browser_partition_method'] = 'first letter'
+gprefs.defaults['tags_browser_collapse_at'] = 100
 
 # }}}
 

src/calibre/gui2/actions/catalog.py

@@ -57,7 +57,7 @@ class GenerateCatalogAction(InterfaceAction):
         if job.result:
             # Search terms nulled catalog results
             return error_dialog(self.gui, _('No books found'),
-                    _("No books to catalog\nCheck exclude tags"),
+                    _("No books to catalog\nCheck exclusion criteria"),
                     show=True)
         if job.failed:
             return self.gui.job_exception(job)

src/calibre/gui2/actions/device.py

@@ -42,8 +42,13 @@ class ShareConnMenu(QMenu): # {{{
         mitem.setEnabled(True)
         mitem.triggered.connect(lambda x : self.connect_to_bambook.emit())
         self.connect_to_bambook_action = mitem
-        if is_disabled(BAMBOOK):
+        bambook_visible = False
-            mitem.setVisible(False)
+        if not is_disabled(BAMBOOK):
+            device_ip = BAMBOOK.settings().extra_customization
+            if device_ip:
+                bambook_visible = True
+        self.connect_to_bambook_action.setVisible(bambook_visible)
+
         self.addSeparator()
         self.toggle_server_action = \
             self.addAction(QIcon(I('network-server.png')),
@@ -99,12 +104,6 @@ class ShareConnMenu(QMenu): # {{{
         self.connect_to_folder_action.setEnabled(not device_connected)
         self.connect_to_itunes_action.setEnabled(not device_connected)
         self.connect_to_bambook_action.setEnabled(not device_connected)
-        bambook_visible = False
-        if not is_disabled(BAMBOOK):
-            device_ip = BAMBOOK.settings().extra_customization
-            if device_ip != None and device_ip != '':
-                bambook_visible = True
-        self.connect_to_bambook_action.setVisible(bambook_visible)
 
 
 # }}}

src/calibre/gui2/book_details.py

@@ -8,7 +8,7 @@ __docformat__ = 'restructuredtext en'
 import os, collections, sys
 from Queue import Queue
 
-from PyQt4.Qt import QPixmap, QSize, QWidget, Qt, pyqtSignal, \
+from PyQt4.Qt import QPixmap, QSize, QWidget, Qt, pyqtSignal, QUrl, \
     QPropertyAnimation, QEasingCurve, QThread, QApplication, QFontInfo, \
     QSizePolicy, QPainter, QRect, pyqtProperty, QLayout, QPalette
 from PyQt4.QtWebKit import QWebView
@@ -18,7 +18,7 @@ from calibre.gui2.widgets import IMAGE_EXTENSIONS
 from calibre.ebooks import BOOK_EXTENSIONS
 from calibre.constants import preferred_encoding
 from calibre.library.comments import comments_to_html
-from calibre.gui2 import config, open_local_file
+from calibre.gui2 import config, open_local_file, open_url
 from calibre.utils.icu import sort_key
 
 # render_rows(data) {{{
@@ -412,6 +412,12 @@ class BookDetails(QWidget): # {{{
             self.view_specific_format.emit(int(id_), fmt)
         elif typ == 'devpath':
             open_local_file(val)
+        else:
+            try:
+                open_url(QUrl(link, QUrl.TolerantMode))
+            except:
+                import traceback
+                traceback.print_exc()
 
 
     def mouseDoubleClickEvent(self, ev):

src/calibre/gui2/catalog/catalog_csv_xml.py

@@ -6,9 +6,9 @@ __license__   = 'GPL v3'
 __copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
-
 from calibre.gui2 import gprefs
 from calibre.gui2.catalog.catalog_csv_xml_ui import Ui_Form
+from calibre.library import db as db_
 from PyQt4.Qt import QWidget, QListWidgetItem
 
 class PluginWidget(QWidget, Ui_Form):
@@ -28,6 +28,12 @@ class PluginWidget(QWidget, Ui_Form):
                 self.all_fields.append(x)
                 QListWidgetItem(x, self.db_fields)
 
+        db = db_()
+        for x in  sorted(db.custom_field_keys()):
+            self.all_fields.append(x)
+            QListWidgetItem(x, self.db_fields)
+
+
     def initialize(self, name, db):
         self.name = name
         fields = gprefs.get(name+'_db_fields', self.all_fields)

src/calibre/gui2/catalog/catalog_epub_mobi.py

@@ -17,18 +17,55 @@ class PluginWidget(QWidget,Ui_Form):
 
     TITLE = _('E-book options')
     HELP  = _('Options specific to')+' EPUB/MOBI '+_('output')
-    OPTION_FIELDS = [('exclude_genre','\[.+\]'),
-                     ('exclude_tags','~,'+_('Catalog')),
-                     ('generate_titles', True),
-                     ('generate_series', True),
-                     ('generate_recently_added', True),
-                     ('note_tag','*'),
-                     ('numbers_as_text', False),
-                     ('read_pattern','+'),
-                     ('read_source_field_cb','Tag'),
-                     ('wishlist_tag','Wishlist'),
-                     ]
 
+    CheckBoxControls = [
+                        'generate_titles',
+                        'generate_series',
+                        'generate_genres',
+                        'generate_recently_added',
+                        'generate_descriptions',
+                        'include_hr'
+                        ]
+    ComboBoxControls = [
+                        'read_source_field',
+                        'exclude_source_field',
+                        'header_note_source_field',
+                        'merge_source_field'
+                        ]
+    LineEditControls = [
+                        'exclude_genre',
+                        'exclude_pattern',
+                        'exclude_tags',
+                        'read_pattern',
+                        'wishlist_tag'
+                        ]
+    RadioButtonControls = [
+                        'merge_before',
+                        'merge_after'
+                        ]
+    SpinBoxControls = [
+                        'thumb_width'
+                        ]
+
+    OPTION_FIELDS = zip(CheckBoxControls,
+                        [True for i in CheckBoxControls],
+                        ['check_box' for i in CheckBoxControls])
+    OPTION_FIELDS += zip(ComboBoxControls,
+                        [None for i in ComboBoxControls],
+                        ['combo_box' for i in ComboBoxControls])
+    OPTION_FIELDS += zip(RadioButtonControls,
+                        [None for i in RadioButtonControls],
+                        ['radio_button' for i in RadioButtonControls])
+
+    # LineEditControls
+    OPTION_FIELDS += zip(['exclude_genre'],['\[.+\]'],['line_edit'])
+    OPTION_FIELDS += zip(['exclude_pattern'],[None],['line_edit'])
+    OPTION_FIELDS += zip(['exclude_tags'],['~,'+_('Catalog')],['line_edit'])
+    OPTION_FIELDS += zip(['read_pattern'],['+'],['line_edit'])
+    OPTION_FIELDS += zip(['wishlist_tag'],['Wishlist'],['line_edit'])
+
+    # SpinBoxControls
+    OPTION_FIELDS += zip(['thumb_width'],[1.00],['spin_box'])
 
     # Output synced to the connected device?
     sync_enabled = True
@@ -42,105 +79,203 @@ class PluginWidget(QWidget,Ui_Form):
 
     def initialize(self, name, db):
         self.name = name
-
+        self.db = db
-        # Populate the 'Read book' source fields
+        self.populateComboBoxes()
-        all_custom_fields = db.custom_field_keys()
-        custom_fields = {}
-        custom_fields['Tag'] = {'field':'tag', 'datatype':u'text'}
-        for custom_field in all_custom_fields:
-            field_md = db.metadata_for_field(custom_field)
-            if field_md['datatype'] in ['bool','composite','datetime','text']:
-                custom_fields[field_md['name']] = {'field':custom_field,
-                                                   'datatype':field_md['datatype']}
-
-        # Add the sorted eligible fields to the combo box
-        for cf in sorted(custom_fields):
-            self.read_source_field_cb.addItem(cf)
-
-        self.read_source_fields = custom_fields
-        self.read_source_field_cb.currentIndexChanged.connect(self.read_source_field_changed)
 
         # Update dialog fields from stored options
         for opt in self.OPTION_FIELDS:
-            opt_value = gprefs.get(self.name + '_' + opt[0], opt[1])
+            c_name, c_def, c_type = opt
-            if opt[0] in [
+            opt_value = gprefs.get(self.name + '_' + c_name, c_def)
-                          'generate_recently_added',
+            if c_type in ['check_box']:
-                          'generate_series',
+                getattr(self, c_name).setChecked(eval(str(opt_value)))
-                          'generate_titles',
+            elif c_type in ['combo_box'] and opt_value is not None:
-                          'numbers_as_text',
+                # *** Test this code with combo boxes ***
-                          ]:
+                #index = self.read_source_field.findText(opt_value)
-                getattr(self, opt[0]).setChecked(opt_value)
+                index = getattr(self,c_name).findText(opt_value)
+                if index == -1 and c_name == 'read_source_field':
+                    index = self.read_source_field.findText('Tag')
+                #self.read_source_field.setCurrentIndex(index)
+                getattr(self,c_name).setCurrentIndex(index)
+            elif c_type in ['line_edit']:
+                getattr(self, c_name).setText(opt_value if opt_value else '')
+            elif c_type in ['radio_button'] and opt_value is not None:
+                getattr(self, c_name).setChecked(opt_value)
+            elif c_type in ['spin_box']:
+                getattr(self, c_name).setValue(float(opt_value))
 
-            # Combo box
+        # Init self.read_source_field_name
-            elif opt[0] in ['read_source_field_cb']:
+        cs = unicode(self.read_source_field.currentText())
-                # Look for last-stored combo box value
-                index = self.read_source_field_cb.findText(opt_value)
-                if index == -1:
-                    index = self.read_source_field_cb.findText('Tag')
-                self.read_source_field_cb.setCurrentIndex(index)
-
-            # Text fields
-            else:
-                getattr(self, opt[0]).setText(opt_value)
-
-        # Init self.read_source_field
-        cs = unicode(self.read_source_field_cb.currentText())
         read_source_spec = self.read_source_fields[cs]
-        self.read_source_field = read_source_spec['field']
+        self.read_source_field_name = read_source_spec['field']
+
+        # Init self.exclude_source_field_name
+        self.exclude_source_field_name = ''
+        cs = unicode(self.exclude_source_field.currentText())
+        if cs > '':
+            exclude_source_spec = self.exclude_source_fields[cs]
+            self.exclude_source_field_name = exclude_source_spec['field']
+
+        # Init self.merge_source_field_name
+        self.merge_source_field_name = ''
+        cs = unicode(self.merge_source_field.currentText())
+        if cs > '':
+            merge_source_spec = self.merge_source_fields[cs]
+            self.merge_source_field_name = merge_source_spec['field']
+
+        # Init self.header_note_source_field_name
+        self.header_note_source_field_name = ''
+        cs = unicode(self.header_note_source_field.currentText())
+        if cs > '':
+            header_note_source_spec = self.header_note_source_fields[cs]
+            self.header_note_source_field_name = header_note_source_spec['field']
+
+        # Hook changes to thumb_width
+        self.thumb_width.valueChanged.connect(self.thumb_width_changed)
 
     def options(self):
         # Save/return the current options
         # exclude_genre stores literally
         # generate_titles, generate_recently_added, numbers_as_text stores as True/False
         # others store as lists
+
         opts_dict = {}
+        # Save values to gprefs
         for opt in self.OPTION_FIELDS:
-            # Save values to gprefs
+            c_name, c_def, c_type = opt
-            if opt[0] in [
+            if c_type in ['check_box', 'radio_button']:
-                          'generate_recently_added',
+                opt_value = getattr(self, c_name).isChecked()
-                          'generate_series',
+            elif c_type in ['combo_box']:
-                          'generate_titles',
+                opt_value = unicode(getattr(self,c_name).currentText()).strip()
-                          'numbers_as_text',
+            elif c_type in ['line_edit']:
-                          ]:
+                opt_value = unicode(getattr(self, c_name).text()).strip()
-                opt_value = getattr(self,opt[0]).isChecked()
+            elif c_type in ['spin_box']:
+                opt_value = unicode(getattr(self, c_name).value())
+            gprefs.set(self.name + '_' + c_name, opt_value)
 
-            # Combo box uses .currentText()
+            # Construct opts object
-            elif opt[0] in ['read_source_field_cb']:
+            if c_name == 'exclude_tags':
-                opt_value = unicode(getattr(self, opt[0]).currentText())
+                # store as list
-
+                opts_dict[c_name] = opt_value.split(',')
-            # text fields use .text()
             else:
-                opt_value = unicode(getattr(self, opt[0]).text())
+                opts_dict[c_name] = opt_value
-            gprefs.set(self.name + '_' + opt[0], opt_value)
 
-            # Construct opts
+        # Generate markers for hybrids
-            if opt[0] in [
+        opts_dict['read_book_marker'] = "%s:%s" % (self.read_source_field_name,
-                          'exclude_genre',
+                                                   self.read_pattern.text())
-                          'generate_recently_added',
+        opts_dict['exclude_book_marker'] = "%s:%s" % (self.exclude_source_field_name,
-                          'generate_series',
+                                                       self.exclude_pattern.text())
-                          'generate_titles',
-                          'numbers_as_text',
-                          ]:
-                opts_dict[opt[0]] = opt_value
-            else:
-                opts_dict[opt[0]] = opt_value.split(',')
 
-        # Generate read_book_marker
+        # Generate specs for merge_comments, header_note_source_field
-        opts_dict['read_book_marker'] = "%s:%s" % (self.read_source_field, self.read_pattern.text())
+        checked = ''
+        if self.merge_before.isChecked():
+            checked = 'before'
+        elif self.merge_after.isChecked():
+            checked = 'after'
+        include_hr = self.include_hr.isChecked()
+        opts_dict['merge_comments'] = "%s:%s:%s" % \
+            (self.merge_source_field_name, checked, include_hr)
+
+        opts_dict['header_note_source_field'] = self.header_note_source_field_name
 
         # Append the output profile
         opts_dict['output_profile'] = [load_defaults('page_setup')['output_profile']]
+        if False:
+            print "opts_dict"
+            for opt in sorted(opts_dict.keys()):
+                print " %s: %s" % (opt, repr(opts_dict[opt]))
         return opts_dict
 
+    def populateComboBoxes(self):
+        # Custom column types declared in
+        #  gui2.preferences.create_custom_column:CreateCustomColumn()
+        # As of 0.7.34:
+        #  bool         Yes/No
+        #  comments     Long text, like comments, not shown in tag browser
+        #  composite    Column built from other columns
+        #  datetime     Date
+        #  enumeration  Text, but with a fixed set of permitted values
+        #  float        Floating point numbers
+        #  int          Integers
+        #  rating       Ratings, shown with stars
+        #  series       Text column for keeping series-like information
+        #  text         Column shown in the tag browser
+        #  *text        Comma-separated text, like tags, shown in tag browser
+
+        all_custom_fields = self.db.custom_field_keys()
+        # Populate the 'Read book' hybrid
+        custom_fields = {}
+        custom_fields['Tag'] = {'field':'tag', 'datatype':u'text'}
+        for custom_field in all_custom_fields:
+            field_md = self.db.metadata_for_field(custom_field)
+            if field_md['datatype'] in ['bool','composite','datetime','enumeration','text']:
+                custom_fields[field_md['name']] = {'field':custom_field,
+                                                   'datatype':field_md['datatype']}
+        # Add the sorted eligible fields to the combo box
+        for cf in sorted(custom_fields):
+            self.read_source_field.addItem(cf)
+        self.read_source_fields = custom_fields
+        self.read_source_field.currentIndexChanged.connect(self.read_source_field_changed)
+
+
+        # Populate the 'Excluded books' hybrid
+        custom_fields = {}
+        for custom_field in all_custom_fields:
+            field_md = self.db.metadata_for_field(custom_field)
+            if field_md['datatype'] in ['bool','composite','datetime','enumeration','text']:
+                custom_fields[field_md['name']] = {'field':custom_field,
+                                                   'datatype':field_md['datatype']}
+        # Blank field first
+        self.exclude_source_field.addItem('')
+        # Add the sorted eligible fields to the combo box
+        for cf in sorted(custom_fields):
+            self.exclude_source_field.addItem(cf)
+        self.exclude_source_fields = custom_fields
+        self.exclude_source_field.currentIndexChanged.connect(self.exclude_source_field_changed)
+
+
+        # Populate the 'Header note' combo box
+        custom_fields = {}
+        for custom_field in all_custom_fields:
+            field_md = self.db.metadata_for_field(custom_field)
+            if field_md['datatype'] in ['bool','composite','datetime','enumeration','text']:
+                custom_fields[field_md['name']] = {'field':custom_field,
+                                                   'datatype':field_md['datatype']}
+        # Blank field first
+        self.header_note_source_field.addItem('')
+        # Add the sorted eligible fields to the combo box
+        for cf in sorted(custom_fields):
+            self.header_note_source_field.addItem(cf)
+        self.header_note_source_fields = custom_fields
+        self.header_note_source_field.currentIndexChanged.connect(self.header_note_source_field_changed)
+
+
+        # Populate the 'Merge with Comments' combo box
+        custom_fields = {}
+        for custom_field in all_custom_fields:
+            field_md = self.db.metadata_for_field(custom_field)
+            if field_md['datatype'] in ['text','comments']:
+                custom_fields[field_md['name']] = {'field':custom_field,
+                                                   'datatype':field_md['datatype']}
+        # Blank field first
+        self.merge_source_field.addItem('')
+        # Add the sorted eligible fields to the combo box
+        for cf in sorted(custom_fields):
+            self.merge_source_field.addItem(cf)
+        self.merge_source_fields = custom_fields
+        self.merge_source_field.currentIndexChanged.connect(self.merge_source_field_changed)
+        self.merge_before.setEnabled(False)
+        self.merge_after.setEnabled(False)
+        self.include_hr.setEnabled(False)
+
     def read_source_field_changed(self,new_index):
         '''
         Process changes in the read_source_field combo box
         Currently using QLineEdit for all field types
         Possible to modify to switch QWidget type
         '''
-        new_source = str(self.read_source_field_cb.currentText())
+        new_source = str(self.read_source_field.currentText())
         read_source_spec = self.read_source_fields[str(new_source)]
-        self.read_source_field = read_source_spec['field']
+        self.read_source_field_name = read_source_spec['field']
 
         # Change pattern input widget to match the source field datatype
         if read_source_spec['datatype'] in ['bool','composite','datetime','text']:
@@ -152,3 +287,63 @@ class PluginWidget(QWidget,Ui_Form):
                 self.read_pattern = dw
                 self.read_spec_hl.addWidget(dw)
 
+    def exclude_source_field_changed(self,new_index):
+        '''
+        Process changes in the exclude_source_field combo box
+        Currently using QLineEdit for all field types
+        Possible to modify to switch QWidget type
+        '''
+        new_source = str(self.exclude_source_field.currentText())
+        self.exclude_source_field_name = new_source
+        if new_source > '':
+            exclude_source_spec = self.exclude_source_fields[str(new_source)]
+            self.exclude_source_field_name = exclude_source_spec['field']
+            self.exclude_pattern.setEnabled(True)
+
+            # Change pattern input widget to match the source field datatype
+            if exclude_source_spec['datatype'] in ['bool','composite','datetime','text']:
+                if not isinstance(self.exclude_pattern, QLineEdit):
+                    self.exclude_spec_hl.removeWidget(self.exclude_pattern)
+                    dw = QLineEdit(self)
+                    dw.setObjectName('exclude_pattern')
+                    dw.setToolTip('Exclusion pattern')
+                    self.exclude_pattern = dw
+                    self.exclude_spec_hl.addWidget(dw)
+        else:
+            self.exclude_pattern.setEnabled(False)
+
+    def header_note_source_field_changed(self,new_index):
+        '''
+        Process changes in the header_note_source_field combo box
+        '''
+        new_source = str(self.header_note_source_field.currentText())
+        self.header_note_source_field_name = new_source
+        if new_source > '':
+            header_note_source_spec = self.header_note_source_fields[str(new_source)]
+            self.header_note_source_field_name = header_note_source_spec['field']
+
+    def merge_source_field_changed(self,new_index):
+        '''
+        Process changes in the header_note_source_field combo box
+        '''
+        new_source = str(self.merge_source_field.currentText())
+        self.merge_source_field_name = new_source
+        if new_source > '':
+            merge_source_spec = self.merge_source_fields[str(new_source)]
+            self.merge_source_field_name = merge_source_spec['field']
+            if not self.merge_before.isChecked() and not self.merge_after.isChecked():
+                self.merge_after.setChecked(True)
+            self.merge_before.setEnabled(True)
+            self.merge_after.setEnabled(True)
+            self.include_hr.setEnabled(True)
+
+        else:
+            self.merge_before.setEnabled(False)
+            self.merge_after.setEnabled(False)
+            self.include_hr.setEnabled(False)
+
+    def thumb_width_changed(self,new_value):
+        '''
+        Process changes in the thumb_width spin box
+        '''
+        pass

src/calibre/gui2/catalog/catalog_epub_mobi.ui

@@ -6,163 +6,681 @@
    <rect>
     <x>0</x>
     <y>0</y>
-    <width>627</width>
+    <width>650</width>
-    <height>549</height>
+    <height>582</height>
    </rect>
   </property>
+  <property name="sizePolicy">
+   <sizepolicy hsizetype="Preferred" vsizetype="Preferred">
+    <horstretch>0</horstretch>
+    <verstretch>0</verstretch>
+   </sizepolicy>
+  </property>
   <property name="windowTitle">
    <string>Form</string>
   </property>
-  <layout class="QGridLayout" name="gridLayout">
+  <layout class="QVBoxLayout" name="verticalLayout">
-   <item row="0" column="0">
+   <item>
-    <widget class="QLabel" name="label_2">
+    <widget class="QGroupBox" name="includedSections">
-     <property name="text">
+     <property name="sizePolicy">
-      <string>'Don't include this book' tag:</string>
+      <sizepolicy hsizetype="Preferred" vsizetype="Maximum">
+       <horstretch>0</horstretch>
+       <verstretch>0</verstretch>
+      </sizepolicy>
      </property>
-    </widget>
+     <property name="minimumSize">
-   </item>
-   <item row="0" column="1">
-    <widget class="QLineEdit" name="exclude_tags">
-     <property name="toolTip">
-      <string extracomment="Default: ~,Catalog"/>
-     </property>
-    </widget>
-   </item>
-   <item row="4" column="0">
-    <widget class="QLabel" name="label_4">
-     <property name="text">
-      <string>Additional note tag prefix:</string>
-     </property>
-    </widget>
-   </item>
-   <item row="4" column="1">
-    <widget class="QLineEdit" name="note_tag">
-     <property name="toolTip">
-      <string extracomment="Default: *"/>
-     </property>
-    </widget>
-   </item>
-   <item row="6" column="1">
-    <widget class="QLineEdit" name="exclude_genre">
-     <property name="toolTip">
-      <string extracomment="Default: \[[\w]*\]"/>
-     </property>
-    </widget>
-   </item>
-   <item row="6" column="0">
-    <widget class="QLabel" name="label">
-     <property name="text">
-      <string>Regex pattern describing tags to exclude as genres:</string>
-     </property>
-     <property name="textFormat">
-      <enum>Qt::LogText</enum>
-     </property>
-     <property name="wordWrap">
-      <bool>true</bool>
-     </property>
-    </widget>
-   </item>
-   <item row="7" column="1">
-    <widget class="QLabel" name="label_6">
-     <property name="text">
-      <string>Regex tips:
-- The default regex - \[.+\] - excludes genre tags of the form [tag], e.g., [Amazon Freebie]
-- A regex pattern of a single dot excludes all genre tags, generating no Genre Section</string>
-     </property>
-     <property name="wordWrap">
-      <bool>true</bool>
-     </property>
-    </widget>
-   </item>
-   <item row="8" column="0">
-    <spacer name="verticalSpacer">
-     <property name="orientation">
-      <enum>Qt::Vertical</enum>
-     </property>
-     <property name="sizeHint" stdset="0">
       <size>
-       <width>20</width>
+       <width>0</width>
-       <height>40</height>
+       <height>0</height>
       </size>
      </property>
-    </spacer>
+     <property name="toolTip">
-   </item>
+      <string>Sections to include in catalog.  All catalogs include 'Books by Author'.</string>
-   <item row="10" column="0">
-    <widget class="QCheckBox" name="generate_titles">
-     <property name="text">
-      <string>Include 'Titles' Section</string>
      </property>
+     <property name="title">
+      <string>Included sections</string>
+     </property>
+     <layout class="QGridLayout" name="gridLayout_2">
+      <item row="0" column="0">
+       <widget class="QCheckBox" name="generate_titles">
+        <property name="text">
+         <string>Books by &amp;Title</string>
+        </property>
+       </widget>
+      </item>
+      <item row="4" column="0">
+       <widget class="QCheckBox" name="generate_series">
+        <property name="text">
+         <string>Books by &amp;Series</string>
+        </property>
+       </widget>
+      </item>
+      <item row="0" column="2">
+       <widget class="QCheckBox" name="generate_recently_added">
+        <property name="text">
+         <string>Recently &amp;Added</string>
+        </property>
+       </widget>
+      </item>
+      <item row="5" column="0">
+       <widget class="QCheckBox" name="generate_genres">
+        <property name="text">
+         <string>Books by &amp;Genre</string>
+        </property>
+       </widget>
+      </item>
+      <item row="4" column="2">
+       <widget class="QCheckBox" name="generate_descriptions">
+        <property name="text">
+         <string>&amp;Descriptions</string>
+        </property>
+       </widget>
+      </item>
+     </layout>
     </widget>
    </item>
-   <item row="12" column="0">
+   <item>
-    <widget class="QCheckBox" name="generate_recently_added">
+    <widget class="QGroupBox" name="excludedGenres">
-     <property name="text">
+     <property name="sizePolicy">
-      <string>Include 'Recently Added' Section</string>
+      <sizepolicy hsizetype="Preferred" vsizetype="Maximum">
+       <horstretch>0</horstretch>
+       <verstretch>0</verstretch>
+      </sizepolicy>
      </property>
+     <property name="minimumSize">
+      <size>
+       <width>0</width>
+       <height>0</height>
+      </size>
+     </property>
+     <property name="toolTip">
+      <string>&lt;!DOCTYPE HTML PUBLIC &quot;-//W3C//DTD HTML 4.0//EN&quot; &quot;http://www.w3.org/TR/REC-html40/strict.dtd&quot;&gt;
+&lt;html&gt;&lt;head&gt;&lt;meta name=&quot;qrichtext&quot; content=&quot;1&quot; /&gt;&lt;style type=&quot;text/css&quot;&gt;
+p, li { white-space: pre-wrap; }
+&lt;/style&gt;&lt;/head&gt;&lt;body style=&quot; font-family:'Lucida Grande'; font-size:13pt; font-weight:400; font-style:normal;&quot;&gt;
+&lt;p style=&quot; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;Default pattern &lt;/p&gt;
+&lt;p style=&quot; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;&lt;span style=&quot; font-family:'Courier New,courier';&quot;&gt;\[.+\]&lt;/span&gt;&lt;/p&gt;
+&lt;p style=&quot; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;excludes tags of the form [&lt;span style=&quot; font-family:'Courier New,courier';&quot;&gt;tag&lt;/span&gt;], &lt;/p&gt;
+&lt;p style=&quot; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;e.g., [Project Gutenberg]&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</string>
+     </property>
+     <property name="title">
+      <string>Excluded genres</string>
+     </property>
+     <layout class="QFormLayout" name="formLayout_3">
+      <property name="fieldGrowthPolicy">
+       <enum>QFormLayout::FieldsStayAtSizeHint</enum>
+      </property>
+      <item row="0" column="0" colspan="2">
+       <layout class="QHBoxLayout" name="horizontalLayout">
+        <property name="spacing">
+         <number>-1</number>
+        </property>
+        <property name="leftMargin">
+         <number>0</number>
+        </property>
+        <item>
+         <widget class="QLabel" name="label">
+          <property name="minimumSize">
+           <size>
+            <width>175</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="maximumSize">
+           <size>
+            <width>200</width>
+            <height>16777215</height>
+           </size>
+          </property>
+          <property name="text">
+           <string>Tags to &amp;exclude</string>
+          </property>
+          <property name="textFormat">
+           <enum>Qt::AutoText</enum>
+          </property>
+          <property name="alignment">
+           <set>Qt::AlignRight|Qt::AlignTrailing|Qt::AlignVCenter</set>
+          </property>
+          <property name="wordWrap">
+           <bool>true</bool>
+          </property>
+          <property name="buddy">
+           <cstring>exclude_genre</cstring>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QLineEdit" name="exclude_genre">
+          <property name="sizePolicy">
+           <sizepolicy hsizetype="Expanding" vsizetype="Fixed">
+            <horstretch>0</horstretch>
+            <verstretch>0</verstretch>
+           </sizepolicy>
+          </property>
+          <property name="minimumSize">
+           <size>
+            <width>0</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="toolTip">
+           <string extracomment="Default: \[[\w]*\]"/>
+          </property>
+         </widget>
+        </item>
+       </layout>
+      </item>
+     </layout>
     </widget>
    </item>
-   <item row="13" column="0">
+   <item>
-    <widget class="QCheckBox" name="numbers_as_text">
+    <widget class="QGroupBox" name="excludedBooks">
-     <property name="text">
+     <property name="sizePolicy">
-      <string>Sort numbers as text</string>
+      <sizepolicy hsizetype="Preferred" vsizetype="Maximum">
+       <horstretch>0</horstretch>
+       <verstretch>0</verstretch>
+      </sizepolicy>
      </property>
+     <property name="minimumSize">
+      <size>
+       <width>0</width>
+       <height>0</height>
+      </size>
+     </property>
+     <property name="toolTip">
+      <string>Books matching either pattern will not be included in generated catalog. </string>
+     </property>
+     <property name="title">
+      <string>Excluded books</string>
+     </property>
+     <layout class="QFormLayout" name="formLayout">
+      <item row="0" column="0" colspan="2">
+       <layout class="QHBoxLayout" name="horizontalLayout_3">
+        <item>
+         <widget class="QLabel" name="label_2">
+          <property name="sizePolicy">
+           <sizepolicy hsizetype="Preferred" vsizetype="Preferred">
+            <horstretch>0</horstretch>
+            <verstretch>0</verstretch>
+           </sizepolicy>
+          </property>
+          <property name="minimumSize">
+           <size>
+            <width>175</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="maximumSize">
+           <size>
+            <width>200</width>
+            <height>16777215</height>
+           </size>
+          </property>
+          <property name="text">
+           <string>Tags to &amp;exclude</string>
+          </property>
+          <property name="alignment">
+           <set>Qt::AlignRight|Qt::AlignTrailing|Qt::AlignVCenter</set>
+          </property>
+          <property name="wordWrap">
+           <bool>true</bool>
+          </property>
+          <property name="buddy">
+           <cstring>exclude_tags</cstring>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QLineEdit" name="exclude_tags">
+          <property name="minimumSize">
+           <size>
+            <width>0</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="toolTip">
+           <string>&lt;!DOCTYPE HTML PUBLIC &quot;-//W3C//DTD HTML 4.0//EN&quot; &quot;http://www.w3.org/TR/REC-html40/strict.dtd&quot;&gt;
+&lt;html&gt;&lt;head&gt;&lt;meta name=&quot;qrichtext&quot; content=&quot;1&quot; /&gt;&lt;style type=&quot;text/css&quot;&gt;
+p, li { white-space: pre-wrap; }
+&lt;/style&gt;&lt;/head&gt;&lt;body style=&quot; font-family:'Lucida Grande'; font-size:13pt; font-weight:400; font-style:normal;&quot;&gt;
+&lt;p style=&quot; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;&lt;span style=&quot; font-size:12pt;&quot;&gt;Comma-separated list of tags to exclude.&lt;/span&gt;&lt;/p&gt;
+&lt;p style=&quot; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;&lt;span style=&quot; font-size:12pt;&quot;&gt;Default:&lt;/span&gt;&lt;span style=&quot; font-family:'Courier New,courier'; font-size:12pt;&quot;&gt; ~,Catalog&lt;/span&gt;&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</string>
+          </property>
+         </widget>
+        </item>
+       </layout>
+      </item>
+      <item row="2" column="0" colspan="2">
+       <layout class="QHBoxLayout" name="exclude_spec_hl">
+        <item>
+         <widget class="QLabel" name="label_7">
+          <property name="minimumSize">
+           <size>
+            <width>175</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="maximumSize">
+           <size>
+            <width>200</width>
+            <height>16777215</height>
+           </size>
+          </property>
+          <property name="text">
+           <string>&amp;Column/value</string>
+          </property>
+          <property name="alignment">
+           <set>Qt::AlignRight|Qt::AlignTrailing|Qt::AlignVCenter</set>
+          </property>
+          <property name="wordWrap">
+           <bool>true</bool>
+          </property>
+          <property name="buddy">
+           <cstring>exclude_source_field</cstring>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QComboBox" name="exclude_source_field">
+          <property name="minimumSize">
+           <size>
+            <width>0</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="toolTip">
+           <string>Column containing additional exclusion criteria</string>
+          </property>
+          <property name="sizeAdjustPolicy">
+           <enum>QComboBox::AdjustToMinimumContentsLengthWithIcon</enum>
+          </property>
+          <property name="minimumContentsLength">
+           <number>18</number>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QLineEdit" name="exclude_pattern">
+          <property name="minimumSize">
+           <size>
+            <width>150</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="toolTip">
+           <string>Exclusion pattern</string>
+          </property>
+         </widget>
+        </item>
+       </layout>
+      </item>
+     </layout>
     </widget>
    </item>
-   <item row="11" column="0">
+   <item>
-    <widget class="QCheckBox" name="generate_series">
+    <widget class="QGroupBox" name="readBooks">
-     <property name="text">
+     <property name="sizePolicy">
-      <string>Include 'Series' Section</string>
+      <sizepolicy hsizetype="Preferred" vsizetype="Maximum">
+       <horstretch>0</horstretch>
+       <verstretch>0</verstretch>
+      </sizepolicy>
      </property>
+     <property name="minimumSize">
+      <size>
+       <width>0</width>
+       <height>0</height>
+      </size>
+     </property>
+     <property name="toolTip">
+      <string>Matching books will be displayed with ✓</string>
+     </property>
+     <property name="title">
+      <string>Read books</string>
+     </property>
+     <layout class="QFormLayout" name="formLayout_2">
+      <item row="0" column="0" colspan="2">
+       <layout class="QHBoxLayout" name="read_spec_hl">
+        <property name="sizeConstraint">
+         <enum>QLayout::SetDefaultConstraint</enum>
+        </property>
+        <item>
+         <widget class="QLabel" name="label_3">
+          <property name="minimumSize">
+           <size>
+            <width>175</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="maximumSize">
+           <size>
+            <width>200</width>
+            <height>16777215</height>
+           </size>
+          </property>
+          <property name="text">
+           <string>&amp;Column/value</string>
+          </property>
+          <property name="alignment">
+           <set>Qt::AlignRight|Qt::AlignTrailing|Qt::AlignVCenter</set>
+          </property>
+          <property name="wordWrap">
+           <bool>true</bool>
+          </property>
+          <property name="buddy">
+           <cstring>read_source_field</cstring>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QComboBox" name="read_source_field">
+          <property name="minimumSize">
+           <size>
+            <width>0</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="toolTip">
+           <string>Column containing 'read' status</string>
+          </property>
+          <property name="statusTip">
+           <string/>
+          </property>
+          <property name="sizeAdjustPolicy">
+           <enum>QComboBox::AdjustToMinimumContentsLengthWithIcon</enum>
+          </property>
+          <property name="minimumContentsLength">
+           <number>18</number>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QLineEdit" name="read_pattern">
+          <property name="minimumSize">
+           <size>
+            <width>150</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="toolTip">
+           <string>'read book' pattern</string>
+          </property>
+          <property name="statusTip">
+           <string/>
+          </property>
+         </widget>
+        </item>
+       </layout>
+      </item>
+     </layout>
     </widget>
    </item>
-   <item row="3" column="1">
+   <item>
-    <widget class="QLineEdit" name="wishlist_tag"/>
+    <widget class="QGroupBox" name="otherOptions">
-   </item>
+     <property name="sizePolicy">
-   <item row="3" column="0">
+      <sizepolicy hsizetype="Preferred" vsizetype="Maximum">
-    <widget class="QLabel" name="label_5">
+       <horstretch>0</horstretch>
-     <property name="text">
+       <verstretch>0</verstretch>
-      <string>Wishlist tag:</string>
+      </sizepolicy>
      </property>
-    </widget>
+     <property name="minimumSize">
-   </item>
+      <size>
-   <item row="2" column="1">
+       <width>0</width>
-    <layout class="QHBoxLayout" name="read_spec_hl">
+       <height>0</height>
-     <property name="sizeConstraint">
+      </size>
-      <enum>QLayout::SetMinimumSize</enum>
      </property>
-     <item>
+     <property name="title">
-      <widget class="QComboBox" name="read_source_field_cb">
+      <string>Other options</string>
-       <property name="sizePolicy">
-        <sizepolicy hsizetype="MinimumExpanding" vsizetype="Fixed">
-         <horstretch>0</horstretch>
-         <verstretch>0</verstretch>
-        </sizepolicy>
-       </property>
-       <property name="toolTip">
-        <string>Source column for read book</string>
-       </property>
-       <property name="statusTip">
-        <string/>
-       </property>
-      </widget>
-     </item>
-     <item>
-      <widget class="QLineEdit" name="read_pattern">
-       <property name="toolTip">
-        <string>Pattern for read book</string>
-       </property>
-       <property name="statusTip">
-        <string/>
-       </property>
-      </widget>
-     </item>
-    </layout>
-   </item>
-   <item row="2" column="0">
-    <widget class="QLabel" name="label_3">
-     <property name="text">
-      <string>Books marked as read:</string>
      </property>
+     <layout class="QFormLayout" name="formLayout_4">
+      <property name="fieldGrowthPolicy">
+       <enum>QFormLayout::FieldsStayAtSizeHint</enum>
+      </property>
+      <item row="1" column="0" colspan="2">
+       <layout class="QHBoxLayout" name="horizontalLayout_2">
+        <item>
+         <widget class="QLabel" name="label_5">
+          <property name="minimumSize">
+           <size>
+            <width>175</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="maximumSize">
+           <size>
+            <width>200</width>
+            <height>16777215</height>
+           </size>
+          </property>
+          <property name="toolTip">
+           <string/>
+          </property>
+          <property name="text">
+           <string>&amp;Wishlist tag</string>
+          </property>
+          <property name="alignment">
+           <set>Qt::AlignRight|Qt::AlignTrailing|Qt::AlignVCenter</set>
+          </property>
+          <property name="wordWrap">
+           <bool>true</bool>
+          </property>
+          <property name="buddy">
+           <cstring>wishlist_tag</cstring>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QLineEdit" name="wishlist_tag">
+          <property name="toolTip">
+           <string>Books tagged as Wishlist items will be displayed with ✕</string>
+          </property>
+         </widget>
+        </item>
+       </layout>
+      </item>
+      <item row="2" column="0" colspan="2">
+       <layout class="QHBoxLayout" name="horizontalLayout_4">
+        <item>
+         <widget class="QLabel" name="label_4">
+          <property name="minimumSize">
+           <size>
+            <width>175</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="maximumSize">
+           <size>
+            <width>200</width>
+            <height>16777215</height>
+           </size>
+          </property>
+          <property name="text">
+           <string>&amp;Thumbnail width</string>
+          </property>
+          <property name="alignment">
+           <set>Qt::AlignRight|Qt::AlignTrailing|Qt::AlignVCenter</set>
+          </property>
+          <property name="wordWrap">
+           <bool>true</bool>
+          </property>
+          <property name="buddy">
+           <cstring>thumb_width</cstring>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QDoubleSpinBox" name="thumb_width">
+          <property name="sizePolicy">
+           <sizepolicy hsizetype="Expanding" vsizetype="Fixed">
+            <horstretch>0</horstretch>
+            <verstretch>0</verstretch>
+           </sizepolicy>
+          </property>
+          <property name="toolTip">
+           <string>Size hint for Description cover thumbnails</string>
+          </property>
+          <property name="suffix">
+           <string> inch</string>
+          </property>
+          <property name="decimals">
+           <number>2</number>
+          </property>
+          <property name="minimum">
+           <double>1.000000000000000</double>
+          </property>
+          <property name="maximum">
+           <double>2.000000000000000</double>
+          </property>
+          <property name="singleStep">
+           <double>0.100000000000000</double>
+          </property>
+         </widget>
+        </item>
+       </layout>
+      </item>
+      <item row="3" column="0" colspan="2">
+       <layout class="QHBoxLayout" name="horizontalLayout_5">
+        <item>
+         <widget class="QLabel" name="label_6">
+          <property name="sizePolicy">
+           <sizepolicy hsizetype="Preferred" vsizetype="Preferred">
+            <horstretch>0</horstretch>
+            <verstretch>0</verstretch>
+           </sizepolicy>
+          </property>
+          <property name="minimumSize">
+           <size>
+            <width>175</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="maximumSize">
+           <size>
+            <width>200</width>
+            <height>16777215</height>
+           </size>
+          </property>
+          <property name="toolTip">
+           <string/>
+          </property>
+          <property name="text">
+           <string>&amp;Description note</string>
+          </property>
+          <property name="alignment">
+           <set>Qt::AlignRight|Qt::AlignTrailing|Qt::AlignVCenter</set>
+          </property>
+          <property name="buddy">
+           <cstring>header_note_source_field</cstring>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QComboBox" name="header_note_source_field">
+          <property name="sizePolicy">
+           <sizepolicy hsizetype="Expanding" vsizetype="Fixed">
+            <horstretch>0</horstretch>
+            <verstretch>0</verstretch>
+           </sizepolicy>
+          </property>
+          <property name="minimumSize">
+           <size>
+            <width>0</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="toolTip">
+           <string>Custom column source for note to include in Description header area</string>
+          </property>
+         </widget>
+        </item>
+       </layout>
+      </item>
+      <item row="4" column="0" colspan="2">
+       <layout class="QHBoxLayout" name="horizontalLayout_7">
+        <item>
+         <widget class="QLabel" name="label_9">
+          <property name="minimumSize">
+           <size>
+            <width>175</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="maximumSize">
+           <size>
+            <width>200</width>
+            <height>16777215</height>
+           </size>
+          </property>
+          <property name="text">
+           <string>&amp;Merge with Comments</string>
+          </property>
+          <property name="alignment">
+           <set>Qt::AlignRight|Qt::AlignTrailing|Qt::AlignVCenter</set>
+          </property>
+          <property name="buddy">
+           <cstring>merge_source_field</cstring>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QComboBox" name="merge_source_field">
+          <property name="minimumSize">
+           <size>
+            <width>0</width>
+            <height>0</height>
+           </size>
+          </property>
+          <property name="toolTip">
+           <string>Additional content merged with Comments during catalog generation</string>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="Line" name="line_2">
+          <property name="orientation">
+           <enum>Qt::Vertical</enum>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QRadioButton" name="merge_before">
+          <property name="toolTip">
+           <string>Merge additional content before Comments</string>
+          </property>
+          <property name="text">
+           <string>&amp;Before</string>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QRadioButton" name="merge_after">
+          <property name="toolTip">
+           <string>Merge additional content after Comments</string>
+          </property>
+          <property name="text">
+           <string>&amp;After</string>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="Line" name="line">
+          <property name="orientation">
+           <enum>Qt::Vertical</enum>
+          </property>
+         </widget>
+        </item>
+        <item>
+         <widget class="QCheckBox" name="include_hr">
+          <property name="toolTip">
+           <string>Separate Comments and additional content with horizontal rule</string>
+          </property>
+          <property name="text">
+           <string>&amp;Separator</string>
+          </property>
+         </widget>
+        </item>
+       </layout>
+      </item>
+     </layout>
     </widget>
    </item>
   </layout>

src/calibre/gui2/catalog/catalog_tab_template.ui

@@ -6,8 +6,8 @@
    <rect>
     <x>0</x>
     <y>0</y>
-    <width>579</width>
+    <width>650</width>
-    <height>411</height>
+    <height>575</height>
    </rect>
   </property>
   <property name="windowTitle">

src/calibre/gui2/comments_editor.py

@@ -259,6 +259,19 @@ class EditorWidget(QWebView): # {{{
 
         return property(fget=fget, fset=fset)
 
+    def keyPressEvent(self, ev):
+        if ev.key() in (Qt.Key_Tab, Qt.Key_Escape, Qt.Key_Backtab):
+            ev.ignore()
+        else:
+            return QWebView.keyPressEvent(self, ev)
+
+    def keyReleaseEvent(self, ev):
+        if ev.key() in (Qt.Key_Tab, Qt.Key_Escape, Qt.Key_Backtab):
+            ev.ignore()
+        else:
+            return QWebView.keyReleaseEvent(self, ev)
+
+
 # }}}
 
 # Highlighter {{{
@@ -480,6 +493,9 @@ class Editor(QWidget): # {{{
         self.toolbar1 = QToolBar(self)
         self.toolbar2 = QToolBar(self)
         self.toolbar3 = QToolBar(self)
+        for i in range(1, 4):
+            t = getattr(self, 'toolbar%d'%i)
+            t.setIconSize(QSize(18, 18))
         self.editor = EditorWidget(self)
         self.tabs = QTabWidget(self)
         self.tabs.setTabPosition(self.tabs.South)

src/calibre/gui2/convert/gui_conversion.py

@@ -30,7 +30,7 @@ def gui_catalog(fmt, title, dbspec, ids, out_file_name, sync, fmt_options, conne
     from calibre.library import db
     from calibre.utils.config import prefs
     prefs.refresh()
-    db = db()
+    db = db(read_only=True)
     db.catalog_plugin_on_device_temp_mapping = dbspec
 
     # Create a minimal OptionParser that we can append to

src/calibre/gui2/convert/pdb_output.py

@@ -19,7 +19,7 @@ class PluginWidget(Widget, Ui_Form):
     ICON = I('mimetypes/unknown.png')
 
     def __init__(self, parent, get_option, get_help, db=None, book_id=None):
-        Widget.__init__(self, parent, ['format', 'inline_toc'])
+        Widget.__init__(self, parent, ['format', 'inline_toc', 'output_encoding'])
         self.db, self.book_id = db, book_id
         self.initialize_options(get_option, get_help, db, book_id)
 

src/calibre/gui2/convert/pdb_output.ui

@@ -27,7 +27,7 @@
    <item row="0" column="1">
     <widget class="QComboBox" name="opt_format"/>
    </item>
-   <item row="2" column="0">
+   <item row="3" column="0">
     <spacer name="verticalSpacer">
      <property name="orientation">
       <enum>Qt::Vertical</enum>
@@ -40,13 +40,23 @@
      </property>
     </spacer>
    </item>
-   <item row="1" column="0">
+   <item row="2" column="0">
     <widget class="QCheckBox" name="opt_inline_toc">
      <property name="text">
       <string>&amp;Inline TOC</string>
      </property>
     </widget>
    </item>
+   <item row="1" column="0">
+    <widget class="QLabel" name="label_2">
+     <property name="text">
+      <string>Output Encoding:</string>
+     </property>
+    </widget>
+   </item>
+   <item row="1" column="1">
+    <widget class="QLineEdit" name="opt_output_encoding"/>
+   </item>
   </layout>
  </widget>
  <resources/>

src/calibre/gui2/convert/pml_output.py

@@ -17,6 +17,7 @@ class PluginWidget(Widget, Ui_Form):
     ICON = I('mimetypes/unknown.png')
 
     def __init__(self, parent, get_option, get_help, db=None, book_id=None):
-        Widget.__init__(self, parent, ['inline_toc', 'full_image_depth'])
+        Widget.__init__(self, parent, ['inline_toc', 'full_image_depth',
+            'output_encoding'])
         self.db, self.book_id = db, book_id
         self.initialize_options(get_option, get_help, db, book_id)

src/calibre/gui2/convert/pmlz_output.ui

@@ -14,7 +14,7 @@
    <string>Form</string>
   </property>
   <layout class="QGridLayout" name="gridLayout">
-   <item row="2" column="0">
+   <item row="3" column="0">
     <spacer name="verticalSpacer">
      <property name="orientation">
       <enum>Qt::Vertical</enum>
@@ -27,20 +27,30 @@
      </property>
     </spacer>
    </item>
-   <item row="0" column="0">
+   <item row="1" column="0">
     <widget class="QCheckBox" name="opt_inline_toc">
      <property name="text">
       <string>&amp;Inline TOC</string>
      </property>
     </widget>
    </item>
-   <item row="1" column="0">
+   <item row="2" column="0">
     <widget class="QCheckBox" name="opt_full_image_depth">
      <property name="text">
       <string>Do not reduce image size and depth</string>
      </property>
     </widget>
    </item>
+   <item row="0" column="0">
+    <widget class="QLabel" name="label">
+     <property name="text">
+      <string>Output Encoding:</string>
+     </property>
+    </widget>
+   </item>
+   <item row="0" column="1">
+    <widget class="QLineEdit" name="opt_output_encoding"/>
+   </item>
   </layout>
  </widget>
  <resources/>

src/calibre/gui2/convert/txt_output.py

@@ -21,7 +21,8 @@ class PluginWidget(Widget, Ui_Form):
     def __init__(self, parent, get_option, get_help, db=None, book_id=None):
         Widget.__init__(self, parent,
         ['newline', 'max_line_length', 'force_max_line_length',
-        'inline_toc', 'markdown_format', 'keep_links', 'keep_image_references'])
+        'inline_toc', 'markdown_format', 'keep_links', 'keep_image_references',
+        'output_encoding'])
         self.db, self.book_id = db, book_id
         self.initialize_options(get_option, get_help, db, book_id)
 

src/calibre/gui2/convert/txt_output.ui

@@ -27,7 +27,7 @@
    <item row="0" column="1">
     <widget class="QComboBox" name="opt_newline"/>
    </item>
-   <item row="7" column="0">
+   <item row="8" column="0">
     <spacer name="verticalSpacer">
      <property name="orientation">
       <enum>Qt::Vertical</enum>
@@ -40,7 +40,7 @@
      </property>
     </spacer>
    </item>
-   <item row="3" column="0" colspan="2">
+   <item row="4" column="0" colspan="2">
     <widget class="QCheckBox" name="opt_inline_toc">
      <property name="text">
       <string>&amp;Inline TOC</string>
@@ -60,34 +60,44 @@
      </property>
     </widget>
    </item>
-   <item row="2" column="0" colspan="2">
+   <item row="3" column="0" colspan="2">
     <widget class="QCheckBox" name="opt_force_max_line_length">
      <property name="text">
       <string>Force maximum line length</string>
      </property>
     </widget>
    </item>
-   <item row="4" column="0">
+   <item row="5" column="0">
     <widget class="QCheckBox" name="opt_markdown_format">
      <property name="text">
       <string>Apply Markdown formatting to text</string>
      </property>
     </widget>
    </item>
-   <item row="5" column="0">
+   <item row="6" column="0">
     <widget class="QCheckBox" name="opt_keep_links">
      <property name="text">
       <string>Do not remove links (&lt;a&gt; tags) before processing</string>
      </property>
     </widget>
    </item>
-   <item row="6" column="0">
+   <item row="7" column="0">
     <widget class="QCheckBox" name="opt_keep_image_references">
      <property name="text">
       <string>Do not remove image references before processing</string>
      </property>
     </widget>
    </item>
+   <item row="2" column="0">
+    <widget class="QLabel" name="label_3">
+     <property name="text">
+      <string>Output Encoding:</string>
+     </property>
+    </widget>
+   </item>
+   <item row="2" column="1">
+    <widget class="QLineEdit" name="opt_output_encoding"/>
+   </item>
   </layout>
  </widget>
  <resources/>

src/calibre/gui2/dialogs/book_info.py

@@ -9,7 +9,7 @@ from PyQt4.Qt import QCoreApplication, SIGNAL, QModelIndex, QTimer, Qt, \
     QDialog, QPixmap, QGraphicsScene, QIcon, QSize
 
 from calibre.gui2.dialogs.book_info_ui import Ui_BookInfo
-from calibre.gui2 import dynamic, open_local_file
+from calibre.gui2 import dynamic, open_local_file, open_url
 from calibre import fit_image
 from calibre.library.comments import comments_to_html
 from calibre.utils.icu import sort_key
@@ -22,6 +22,8 @@ class BookInfo(QDialog, Ui_BookInfo):
         self.setupUi(self)
         self.cover_pixmap = None
         self.comments.sizeHint = self.comments_size_hint
+        self.comments.page().setLinkDelegationPolicy(self.comments.page().DelegateAllLinks)
+        self.comments.linkClicked.connect(self.link_clicked)
         self.view_func = view_func
 
 
@@ -41,6 +43,8 @@ class BookInfo(QDialog, Ui_BookInfo):
         screen_height = desktop.availableGeometry().height() - 100
         self.resize(self.size().width(), screen_height)
 
+    def link_clicked(self, url):
+        open_url(url)
 
     def comments_size_hint(self):
         return QSize(350, 250)
@@ -115,6 +119,7 @@ class BookInfo(QDialog, Ui_BookInfo):
             lines = [x if x.strip() else '<br><br>' for x in lines]
             comments = '\n'.join(lines)
         self.comments.setHtml('<div>%s</div>' % comments)
+        self.comments.page().setLinkDelegationPolicy(self.comments.page().DelegateAllLinks)
         cdata = info.pop('cover', '')
         self.cover_pixmap = QPixmap.fromImage(cdata)
         self.resize_cover()

src/calibre/gui2/dialogs/catalog.ui

@@ -6,8 +6,8 @@
    <rect>
     <x>0</x>
     <y>0</y>
-    <width>611</width>
+    <width>674</width>
-    <height>514</height>
+    <height>660</height>
    </rect>
   </property>
   <property name="windowTitle">
@@ -33,6 +33,18 @@
    </item>
    <item row="1" column="0" colspan="2">
     <widget class="QTabWidget" name="tabs">
+     <property name="sizePolicy">
+      <sizepolicy hsizetype="Expanding" vsizetype="Expanding">
+       <horstretch>0</horstretch>
+       <verstretch>0</verstretch>
+      </sizepolicy>
+     </property>
+     <property name="minimumSize">
+      <size>
+       <width>650</width>
+       <height>575</height>
+      </size>
+     </property>
      <property name="currentIndex">
       <number>0</number>
      </property>

src/calibre/gui2/dialogs/tag_editor.py

@@ -4,7 +4,7 @@ from PyQt4.QtCore import SIGNAL, Qt
 from PyQt4.QtGui import QDialog
 
 from calibre.gui2.dialogs.tag_editor_ui import Ui_TagEditor
-from calibre.gui2 import question_dialog, error_dialog
+from calibre.gui2 import question_dialog, error_dialog, gprefs
 from calibre.constants import islinux
 from calibre.utils.icu import sort_key
 
@@ -49,6 +49,10 @@ class TagEditor(QDialog, Ui_TagEditor):
             self.connect(self.available_tags, SIGNAL('itemActivated(QListWidgetItem*)'), self.apply_tags)
         self.connect(self.applied_tags,   SIGNAL('itemActivated(QListWidgetItem*)'), self.unapply_tags)
 
+        geom = gprefs.get('tag_editor_geometry', None)
+        if geom is not None:
+            self.restoreGeometry(geom)
+
 
     def delete_tags(self, item=None):
         confirms, deletes = [], []
@@ -121,3 +125,15 @@ class TagEditor(QDialog, Ui_TagEditor):
             self.applied_tags.addItem(tag)
 
         self.add_tag_input.setText('')
+
+    def accept(self):
+        self.save_state()
+        return QDialog.accept(self)
+
+    def reject(self):
+        self.save_state()
+        return QDialog.reject(self)
+
+    def save_state(self):
+        gprefs['tag_editor_geometry'] = bytearray(self.saveGeometry())
+

src/calibre/gui2/dialogs/template_dialog.py

@@ -19,7 +19,7 @@ class TemplateDialog(QDialog, Ui_TemplateDialog):
 
         if text is not None:
             self.textbox.setPlainText(text)
-        self.textbox.setTabChangesFocus(True)
+        self.textbox.setTabStopWidth(50)
         self.buttonBox.button(QDialogButtonBox.Ok).setText(_('&OK'))
         self.buttonBox.button(QDialogButtonBox.Cancel).setText(_('&Cancel'))
 

src/calibre/gui2/dialogs/template_dialog.ui

@@ -6,7 +6,7 @@
    <rect>
     <x>0</x>
     <y>0</y>
-    <width>336</width>
+    <width>500</width>
     <height>235</height>
    </rect>
   </property>

src/calibre/gui2/email.py

@@ -22,6 +22,7 @@ from calibre.customize.ui import available_input_formats, available_output_forma
 from calibre.ebooks.metadata import authors_to_string
 from calibre.constants import preferred_encoding
 from calibre.gui2 import config, Dispatcher, warning_dialog
+from calibre.utils.config import tweaks
 
 class EmailJob(BaseJob): # {{{
 
@@ -83,7 +84,7 @@ class Emailer(Thread): # {{{
         rh = opts.relay_host
         if rh and (
             'gmail.com' in rh or 'live.com' in rh):
-            self.rate_limit = 301
+            self.rate_limit = tweaks['public_smtp_relay_delay']
 
     def stop(self):
         self._run = False

src/calibre/gui2/library/__init__.py

@@ -5,6 +5,4 @@ __license__   = 'GPL v3'
 __copyright__ = '2010, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
-from PyQt4.Qt import Qt
+DEFAULT_SORT = ('timestamp', False)
-
-DEFAULT_SORT = ('timestamp', Qt.DescendingOrder)

src/calibre/gui2/library/models.py

@@ -247,9 +247,10 @@ class BooksModel(QAbstractTableModel): # {{{
         if not self.db:
             return
         self.about_to_be_sorted.emit(self.db.id)
-        ascending = order == Qt.AscendingOrder
+        if not isinstance(order, bool):
+            order = order == Qt.AscendingOrder
         label = self.column_map[col]
-        self.db.sort(label, ascending)
+        self.db.sort(label, order)
         if reset:
             self.reset()
         self.sorted_on = (label, order)

src/calibre/gui2/library/views.py

@@ -165,7 +165,7 @@ class BooksView(QTableView): # {{{
                     partial(self.column_header_context_handler,
                         action='descending', column=col))
             if self._model.sorted_on[0] == col:
-                ac = a if self._model.sorted_on[1] == Qt.AscendingOrder else d
+                ac = a if self._model.sorted_on[1] else d
                 ac.setCheckable(True)
                 ac.setChecked(True)
             if col not in ('ondevice', 'rating', 'inlibrary') and \
@@ -282,17 +282,21 @@ class BooksView(QTableView): # {{{
     def cleanup_sort_history(self, sort_history):
         history = []
         for col, order in sort_history:
+            if not isinstance(order, bool):
+                continue
             if col == 'date':
                 col = 'timestamp'
-            if col in self.column_map and (not history or history[0][0] != col):
+            if col in self.column_map:
-                history.append([col, order])
+                if (not history or history[-1][0] != col):
+                    history.append([col, order])
         return history
 
     def apply_sort_history(self, saved_history):
         if not saved_history:
             return
         for col, order in reversed(self.cleanup_sort_history(saved_history)[:3]):
-            self.sortByColumn(self.column_map.index(col), order)
+            self.sortByColumn(self.column_map.index(col),
+                              Qt.AscendingOrder if order else Qt.DescendingOrder)
 
     def apply_state(self, state):
         h = self.column_header
@@ -382,7 +386,12 @@ class BooksView(QTableView): # {{{
             old_state = self.get_default_state()
 
         if tweaks['sort_columns_at_startup'] is not None:
-            old_state['sort_history'] = tweaks['sort_columns_at_startup']
+            sh = []
+            for c,d in tweaks['sort_columns_at_startup']:
+                if not isinstance(d, bool):
+                    d = True if d == 0 else False
+                sh.append((c, d))
+            old_state['sort_history'] = sh
 
         self.apply_state(old_state)
 

src/calibre/gui2/main.py

@@ -303,7 +303,7 @@ def run_gui(opts, args, actions, listener, app, gui_debug=None):
                 runner.main.system_tray_icon.hide()
             except:
                 pass
-    if runner.main.gui_debug is not None:
+    if getattr(runner.main, 'gui_debug', None) is not None:
         e = sys.executable if getattr(sys, 'frozen', False) else sys.argv[0]
         import subprocess
         creationflags = 0

src/calibre/gui2/preferences/look_feel.py

@@ -57,6 +57,11 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
             (_('Never'), 'never')]
         r('toolbar_text', gprefs, choices=choices)
 
+        choices = [(_('Disabled'), 'disabled'), (_('By first letter'), 'first letter'),
+                   (_('Partitioned'), 'partition')]
+        r('tags_browser_partition_method', gprefs, choices=choices)
+        r('tags_browser_collapse_at', gprefs)
+
         self.current_font = None
         self.change_font_button.clicked.connect(self.change_font)
 
@@ -113,6 +118,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
     def refresh_gui(self, gui):
         gui.search.search_as_you_type(config['search_as_you_type'])
         self.update_font_display()
+        gui.tags_view.reread_collapse_parameters()
 
 if __name__ == '__main__':
     app = QApplication([])

src/calibre/gui2/preferences/look_feel.ui

@@ -7,7 +7,7 @@
     <x>0</x>
     <y>0</y>
     <width>670</width>
-    <height>385</height>
+    <height>392</height>
    </rect>
   </property>
   <property name="windowTitle">
@@ -142,6 +142,65 @@
     </widget>
    </item>
    <item row="7" column="0" colspan="2">
+    <layout class="QHBoxLayout">
+     <item>
+      <widget class="QLabel" name="label_6">
+       <property name="text">
+        <string>Tags browser category partitioning method:</string>
+       </property>
+       <property name="buddy">
+        <cstring>opt_tags_browser_partition_method</cstring>
+       </property>
+      </widget>
+     </item>
+     <item>
+      <widget class="QComboBox" name="opt_tags_browser_partition_method">
+       <property name="toolTip">
+        <string>Choose how tag browser subcategories are displayed when
+there are more items than the limit. Select by first
+letter to see an A, B, C list. Choose partitioned to
+have a list of fixed-sized groups. Set to disabled
+if you never want subcategories</string>
+       </property>
+      </widget>
+     </item>
+     <item>
+      <widget class="QLabel" name="label_6">
+       <property name="text">
+        <string>Collapse when more items than:</string>
+       </property>
+       <property name="buddy">
+        <cstring>opt_tags_browser_collapse_at</cstring>
+       </property>
+      </widget>
+     </item>
+     <item>
+      <widget class="QSpinBox" name="opt_tags_browser_collapse_at">
+       <property name="toolTip">
+           <string>If a Tag Browser category has more than this number of items, it is divided
+up into sub-categories. If the partition method is set to disable, this value is ignored.</string>
+       </property>
+       <property name="maximum">
+        <number>10000</number>
+       </property>
+      </widget>
+     </item>
+     <item>
+      <spacer name="verticalSpacer">
+       <property name="orientation">
+        <enum>Qt::Horizontal</enum>
+       </property>
+       <property name="sizeHint" stdset="0">
+        <size>
+         <width>20</width>
+         <height>5</height>
+        </size>
+       </property>
+      </spacer>
+     </item>
+    </layout>
+   </item>
+   <item row="15" column="0" colspan="2">
     <widget class="QGroupBox" name="groupBox_2">
      <property name="title">
       <string>&amp;Toolbar</string>
@@ -183,7 +242,7 @@
      </layout>
     </widget>
    </item>
-   <item row="8" column="0">
+   <item row="16" column="0">
     <layout class="QHBoxLayout" name="horizontalLayout">
      <item>
       <widget class="QLabel" name="label_2">
@@ -204,14 +263,14 @@
      </item>
     </layout>
    </item>
-   <item row="8" column="1">
+   <item row="16" column="1">
     <widget class="QPushButton" name="change_font_button">
      <property name="text">
       <string>Change &amp;font (needs restart)</string>
      </property>
     </widget>
    </item>
-   <item row="9" column="0" colspan="2">
+   <item row="17" column="0" colspan="2">
     <spacer name="verticalSpacer">
      <property name="orientation">
       <enum>Qt::Vertical</enum>

src/calibre/gui2/tag_view.py

@@ -10,22 +10,25 @@ Browsing book collection by tags.
 from itertools import izip
 from functools import partial
 
-from PyQt4.Qt import Qt, QTreeView, QApplication, pyqtSignal, \
+from PyQt4.Qt import Qt, QTreeView, QApplication, pyqtSignal, QFont, QSize, \
-                     QFont, QSize, QIcon, QPoint, QVBoxLayout, QComboBox, \
+                     QIcon, QPoint, QVBoxLayout, QHBoxLayout, QComboBox, QTimer,\
-                     QAbstractItemModel, QVariant, QModelIndex, QMenu, \
+                     QAbstractItemModel, QVariant, QModelIndex, QMenu, QFrame,\
-                     QPushButton, QWidget, QItemDelegate
+                     QPushButton, QWidget, QItemDelegate, QString, QLabel, \
+                     QShortcut, QKeySequence, SIGNAL
 
 from calibre.ebooks.metadata import title_sort
-from calibre.gui2 import config, NONE
+from calibre.gui2 import config, NONE, gprefs
 from calibre.library.field_metadata import TagsIcons, category_icon_map
 from calibre.utils.config import tweaks
-from calibre.utils.icu import sort_key
+from calibre.utils.icu import sort_key, upper, lower, strcmp
 from calibre.utils.search_query_parser import saved_searches
+from calibre.utils.formatter import eval_formatter
 from calibre.gui2 import error_dialog
 from calibre.gui2.dialogs.confirm_delete import confirm
 from calibre.gui2.dialogs.tag_categories import TagCategories
 from calibre.gui2.dialogs.tag_list_editor import TagListEditor
 from calibre.gui2.dialogs.edit_authors_dialog import EditAuthorsDialog
+from calibre.gui2.widgets import HistoryLineEdit
 
 class TagDelegate(QItemDelegate): # {{{
 
@@ -52,6 +55,8 @@ class TagDelegate(QItemDelegate): # {{{
             painter.setClipRect(r)
 
         # Paint the text
+        if item.boxed:
+            painter.drawRoundedRect(r.adjusted(1,1,-1,-1), 5, 5)
         r.setLeft(r.left()+r.height()+3)
         painter.drawText(r, Qt.AlignLeft|Qt.AlignVCenter,
                         model.data(index, Qt.DisplayRole).toString())
@@ -89,6 +94,10 @@ class TagsView(QTreeView): # {{{
         self.setDropIndicatorShown(True)
         self.setAutoExpandDelay(500)
         self.pane_is_visible = False
+        if gprefs['tags_browser_collapse_at'] == 0:
+            self.collapse_model = 'disable'
+        else:
+            self.collapse_model = gprefs['tags_browser_partition_method']
 
     def set_pane_is_visible(self, to_what):
         pv = self.pane_is_visible
@@ -96,12 +105,20 @@ class TagsView(QTreeView): # {{{
         if to_what and not pv:
             self.recount()
 
+    def reread_collapse_parameters(self):
+        if gprefs['tags_browser_collapse_at'] == 0:
+            self.collapse_model = 'disable'
+        else:
+            self.collapse_model = gprefs['tags_browser_partition_method']
+        self.set_new_model(self._model.get_filter_categories_by())
+
     def set_database(self, db, tag_match, sort_by):
         self.hidden_categories = config['tag_browser_hidden_categories']
         self._model = TagsModel(db, parent=self,
                                 hidden_categories=self.hidden_categories,
                                 search_restriction=None,
-                                drag_drop_finished=self.drag_drop_finished)
+                                drag_drop_finished=self.drag_drop_finished,
+                                collapse_model=self.collapse_model)
         self.pane_is_visible = True # because TagsModel.init did a recount
         self.sort_by = sort_by
         self.tag_match = tag_match
@@ -189,6 +206,12 @@ class TagsView(QTreeView): # {{{
                 self.hidden_categories.add(category)
             elif action == 'show':
                 self.hidden_categories.discard(category)
+            elif action == 'categorization':
+                changed = self.collapse_model != category
+                self.collapse_model = category
+                if changed:
+                    self.set_new_model(self._model.get_filter_categories_by())
+                    gprefs['tags_browser_partition_method'] = category
             elif action == 'defaults':
                 self.hidden_categories.clear()
             config.set('tag_browser_hidden_categories', self.hidden_categories)
@@ -211,6 +234,8 @@ class TagsView(QTreeView): # {{{
                 item = item.parent
 
             if item.type == TagTreeItem.CATEGORY:
+                while item.parent != self._model.root_item:
+                    item = item.parent
                 category = unicode(item.name.toString())
                 key = item.category_key
                 # Verify that we are working with a field that we know something about
@@ -272,6 +297,23 @@ class TagsView(QTreeView): # {{{
             self.context_menu.addAction(_('Show all categories'),
                         partial(self.context_menu_handler, action='defaults'))
 
+        m = self.context_menu.addMenu(_('Change sub-categorization scheme'))
+        da = m.addAction('Disable',
+            partial(self.context_menu_handler, action='categorization', category='disable'))
+        fla = m.addAction('By first letter',
+            partial(self.context_menu_handler, action='categorization', category='first letter'))
+        pa = m.addAction('Partition',
+            partial(self.context_menu_handler, action='categorization', category='partition'))
+        if self.collapse_model == 'disable':
+            da.setCheckable(True)
+            da.setChecked(True)
+        elif self.collapse_model == 'first letter':
+            fla.setCheckable(True)
+            fla.setChecked(True)
+        else:
+            pa.setCheckable(True)
+            pa.setChecked(True)
+
         if not self.context_menu.isEmpty():
             self.context_menu.popup(self.mapToGlobal(point))
         return True
@@ -322,21 +364,19 @@ class TagsView(QTreeView): # {{{
                 path = None
         except: #Database connection could be closed if an integrity check is happening
             pass
-        if path:
+        self._model.show_item_at_path(path)
-            idx = self.model().index_for_path(path)
-            if idx.isValid():
-                self.setCurrentIndex(idx)
-                self.scrollTo(idx, QTreeView.PositionAtCenter)
 
     # If the number of user categories changed,  if custom columns have come or
     # gone, or if columns have been hidden or restored, we must rebuild the
     # model. Reason: it is much easier than reconstructing the browser tree.
-    def set_new_model(self):
+    def set_new_model(self, filter_categories_by=None):
         try:
             self._model = TagsModel(self.db, parent=self,
                                     hidden_categories=self.hidden_categories,
                                     search_restriction=self.search_restriction,
-                                    drag_drop_finished=self.drag_drop_finished)
+                                    drag_drop_finished=self.drag_drop_finished,
+                                    filter_categories_by=filter_categories_by,
+                                    collapse_model=self.collapse_model)
             self.setModel(self._model)
         except:
             # The DB must be gone. Set the model to None and hope that someone
@@ -355,6 +395,7 @@ class TagTreeItem(object): # {{{
                  parent=None, tooltip=None, category_key=None):
         self.parent = parent
         self.children = []
+        self.boxed = False
         if self.parent is not None:
             self.parent.append(self)
         if data is None:
@@ -371,7 +412,13 @@ class TagTreeItem(object): # {{{
         elif self.type == self.TAG:
             icon_map[0] = data.icon
             self.tag, self.icon_state_map = data, list(map(QVariant, icon_map))
-        self.tooltip = tooltip
+        if tooltip:
+            if tooltip.endswith(':'):
+                self.tooltip = tooltip + ' '
+            else:
+                self.tooltip = tooltip + ': '
+        else:
+            self.tooltip = ''
 
     def __str__(self):
         if self.type == self.ROOT:
@@ -400,7 +447,7 @@ class TagTreeItem(object): # {{{
 
     def category_data(self, role):
         if role == Qt.DisplayRole:
-            return QVariant(self.py_name + ' [%d]'%len(self.children))
+            return QVariant(self.py_name + ' [%d]'%len(self.child_tags()))
         if role == Qt.DecorationRole:
             return self.icon
         if role == Qt.FontRole:
@@ -433,20 +480,32 @@ class TagTreeItem(object): # {{{
                     return QVariant('(%s) %s'%(tag.name, tag.tooltip))
                 else:
                     return QVariant(tag.name)
-            if tag.tooltip is not None:
+            if tag.tooltip:
-                return QVariant(tag.tooltip)
+                return QVariant(self.tooltip + tag.tooltip)
+            else:
+                return QVariant(self.tooltip)
         return NONE
 
     def toggle(self):
         if self.type == self.TAG:
             self.tag.state = (self.tag.state + 1)%3
 
+    def child_tags(self):
+        res = []
+        for t in self.children:
+            if t.type == TagTreeItem.CATEGORY:
+                for c in t.children:
+                    res.append(c)
+            else:
+                res.append(t)
+        return res
     # }}}
 
 class TagsModel(QAbstractItemModel): # {{{
 
     def __init__(self, db, parent, hidden_categories=None,
-            search_restriction=None, drag_drop_finished=None):
+            search_restriction=None, drag_drop_finished=None,
+            filter_categories_by=None, collapse_model='disable'):
         QAbstractItemModel.__init__(self, parent)
 
         # must do this here because 'QPixmap: Must construct a QApplication
@@ -466,6 +525,8 @@ class TagsModel(QAbstractItemModel): # {{{
         self.hidden_categories = hidden_categories
         self.search_restriction = search_restriction
         self.row_map = []
+        self.filter_categories_by = filter_categories_by
+        self.collapse_model = collapse_model
 
         # get_node_tree cannot return None here, because row_map is empty
         data = self.get_node_tree(config['sort_tags_by'])
@@ -477,19 +538,11 @@ class TagsModel(QAbstractItemModel): # {{{
                 tt = _('The lookup/search name is "{0}"').format(r)
             else:
                 tt = ''
-            c = TagTreeItem(parent=self.root_item,
+            TagTreeItem(parent=self.root_item,
                     data=self.categories[i],
                     category_icon=self.category_icon_map[r],
                     tooltip=tt, category_key=r)
-            # This duplicates code in refresh(). Having it here as well
+        self.refresh(data=data)
-            # can save seconds during startup, because we avoid a second
-            # call to get_node_tree.
-            for tag in data[r]:
-                if r not in self.categories_with_ratings and \
-                            not self.db.field_metadata[r]['is_custom'] and \
-                            not self.db.field_metadata[r]['kind'] == 'user':
-                    tag.avg_rating = None
-                TagTreeItem(parent=c, data=tag, icon_map=self.icon_state_map)
 
     def mimeTypes(self):
         return ["application/calibre+from_library"]
@@ -641,6 +694,11 @@ class TagsModel(QAbstractItemModel): # {{{
         else:
             data = self.db.get_categories(sort=sort, icon_map=self.category_icon_map)
 
+        if self.filter_categories_by:
+            for category in data.keys():
+                data[category] = [t for t in data[category]
+                        if lower(t.name).find(self.filter_categories_by) >= 0]
+
         tb_categories = self.db.field_metadata
         for category in tb_categories:
             if category in data: # The search category can come and go
@@ -652,35 +710,91 @@ class TagsModel(QAbstractItemModel): # {{{
             return None
         return data
 
-    def refresh(self):
+    def refresh(self, data=None):
-        data = self.get_node_tree(config['sort_tags_by']) # get category data
+        sort_by = config['sort_tags_by']
+        if data is None:
+            data = self.get_node_tree(sort_by) # get category data
         if data is None:
             return False
         row_index = -1
+        collapse = gprefs['tags_browser_collapse_at']
+        collapse_model = self.collapse_model
+        if collapse == 0:
+            collapse_model = 'disable'
+        elif collapse_model != 'disable':
+            if sort_by == 'name':
+                collapse_template = tweaks['categories_collapsed_name_template']
+            elif sort_by == 'rating':
+                collapse_model = 'partition'
+                collapse_template = tweaks['categories_collapsed_rating_template']
+            else:
+                collapse_model = 'partition'
+                collapse_template = tweaks['categories_collapsed_popularity_template']
+        collapse_letter = None
+
         for i, r in enumerate(self.row_map):
             if self.hidden_categories and self.categories[i] in self.hidden_categories:
                 continue
             row_index += 1
             category = self.root_item.children[row_index]
-            names = [t.tag.name for t in category.children]
+            names = []
-            states = [t.tag.state for t in category.children]
+            states = []
+            children = category.child_tags()
+            states = [t.tag.state for t in children]
+            names = [t.tag.name for names in children]
             state_map = dict(izip(names, states))
             category_index = self.index(row_index, 0, QModelIndex())
+            category_node = category_index.internalPointer()
             if len(category.children) > 0:
                 self.beginRemoveRows(category_index, 0,
                         len(category.children)-1)
                 category.children = []
                 self.endRemoveRows()
-            if len(data[r]) > 0:
+            cat_len = len(data[r])
-                self.beginInsertRows(category_index, 0, len(data[r])-1)
+            if cat_len <= 0:
-                for tag in data[r]:
+                continue
-                    if r not in self.categories_with_ratings and \
+
+            self.beginInsertRows(category_index, 0, len(data[r])-1)
+            clear_rating = True if r not in self.categories_with_ratings and \
                                 not self.db.field_metadata[r]['is_custom'] and \
-                                not self.db.field_metadata[r]['kind'] == 'user':
+                                not self.db.field_metadata[r]['kind'] == 'user' \
-                        tag.avg_rating = None
+                            else False
-                    tag.state = state_map.get(tag.name, 0)
+            for idx,tag in enumerate(data[r]):
-                    t = TagTreeItem(parent=category, data=tag, icon_map=self.icon_state_map)
+                if clear_rating:
-                self.endInsertRows()
+                    tag.avg_rating = None
+                tag.state = state_map.get(tag.name, 0)
+
+                if collapse_model != 'disable' and cat_len > collapse:
+                    if collapse_model == 'partition':
+                        if (idx % collapse) == 0:
+                            d = {'first': tag}
+                            if cat_len > idx + collapse:
+                                d['last'] = data[r][idx+collapse-1]
+                            else:
+                                d['last'] = data[r][cat_len-1]
+                            name = eval_formatter.safe_format(collapse_template,
+                                                              d, 'TAG_VIEW', None)
+                            sub_cat = TagTreeItem(parent=category,
+                                     data = name, tooltip = None,
+                                     category_icon = category_node.icon,
+                                     category_key=category_node.category_key)
+                    else:
+                        ts = tag.sort
+                        if not ts:
+                            ts = ' '
+                        if upper(ts[0]) != collapse_letter:
+                            collapse_letter = upper(ts[0])
+                            sub_cat = TagTreeItem(parent=category,
+                                     data = collapse_letter,
+                                     category_icon = category_node.icon,
+                                     tooltip = None,
+                                     category_key=category_node.category_key)
+                    t = TagTreeItem(parent=sub_cat, data=tag, tooltip=r,
+                                        icon_map=self.icon_state_map)
+                else:
+                    t = TagTreeItem(parent=category, data=tag, tooltip=r,
+                                    icon_map=self.icon_state_map)
+            self.endInsertRows()
         return True
 
     def columnCount(self, parent):
@@ -737,11 +851,7 @@ class TagsModel(QAbstractItemModel): # {{{
             self.tags_view.tag_item_renamed.emit()
             item.tag.name = val
             self.refresh() # Should work, because no categories can have disappeared
-        if path:
+        self.show_item_at_path(path)
-            idx = self.index_for_path(path)
-            if idx.isValid():
-                self.tags_view.setCurrentIndex(idx)
-                self.tags_view.scrollTo(idx, QTreeView.PositionAtCenter)
         return True
 
     def headerData(self, *args):
@@ -824,19 +934,27 @@ class TagsModel(QAbstractItemModel): # {{{
 
     def reset_all_states(self, except_=None):
         update_list = []
-        for i in xrange(self.rowCount(QModelIndex())):
+        def process_tag(tag_index, tag_item):
-            category_index = self.index(i, 0, QModelIndex())
+            tag = tag_item.tag
+            if tag is except_:
+                self.dataChanged.emit(tag_index, tag_index)
+                return
+            if tag.state != 0 or tag in update_list:
+                tag.state = 0
+                update_list.append(tag)
+                self.dataChanged.emit(tag_index, tag_index)
+
+        def process_level(category_index):
             for j in xrange(self.rowCount(category_index)):
                 tag_index = self.index(j, 0, category_index)
                 tag_item = tag_index.internalPointer()
-                tag = tag_item.tag
+                if tag_item.type == TagTreeItem.CATEGORY:
-                if tag is except_:
+                    process_level(tag_index)
-                    self.dataChanged.emit(tag_index, tag_index)
+                else:
-                    continue
+                    process_tag(tag_index, tag_item)
-                if tag.state != 0 or tag in update_list:
+
-                    tag.state = 0
+        for i in xrange(self.rowCount(QModelIndex())):
-                    update_list.append(tag)
+            process_level(self.index(i, 0, QModelIndex()))
-                    self.dataChanged.emit(tag_index, tag_index)
 
     def clear_state(self):
         self.reset_all_states()
@@ -856,14 +974,16 @@ class TagsModel(QAbstractItemModel): # {{{
         ans = []
         tags_seen = set()
         row_index = -1
+
         for i, key in enumerate(self.row_map):
             if self.hidden_categories and self.categories[i] in self.hidden_categories:
                 continue
             row_index += 1
-            if key.endswith(':'): # User category, so skip it. The tag will be marked in its real category
+            if key.endswith(':'):
+                # User category, so skip it. The tag will be marked in its real category
                 continue
             category_item = self.root_item.children[row_index]
-            for tag_item in category_item.children:
+            for tag_item in category_item.child_tags():
                 tag = tag_item.tag
                 if tag.state > 0:
                     prefix = ' not ' if tag.state == 2 else ''
@@ -878,6 +998,102 @@ class TagsModel(QAbstractItemModel): # {{{
                         ans.append('%s%s:"=%s"'%(prefix, category, tag.name))
         return ans
 
+    def find_node(self, key, txt, start_path):
+        '''
+        Search for an item (a node) in the tags browser list that matches both
+        the key (exact case-insensitive match) and txt (contains case-
+        insensitive match). Returns the path to the node. Note that paths are to
+        a location (second item, fourth item, 25 item), not to a node. If
+        start_path is None, the search starts with the topmost node. If the tree
+        is changed subsequent to calling this method, the path can easily refer
+        to a different node or no node at all.
+        '''
+        if not txt:
+            return None
+        txt = lower(txt)
+        self.path_found = None
+        if start_path is None:
+            start_path = []
+
+        def process_tag(depth, tag_index, tag_item, start_path):
+            path = self.path_for_index(tag_index)
+            if depth < len(start_path) and path[depth] <= start_path[depth]:
+                return False
+            tag = tag_item.tag
+            if tag is None:
+                return False
+            if lower(tag.name).find(txt) >= 0:
+                self.path_found = path
+                return True
+            return False
+
+        def process_level(depth, category_index, start_path):
+            path = self.path_for_index(category_index)
+            if depth < len(start_path):
+                if path[depth] < start_path[depth]:
+                    return False
+                if path[depth] > start_path[depth]:
+                    start_path = path
+            if key and strcmp(category_index.internalPointer().category_key, key) != 0:
+                return False
+            for j in xrange(self.rowCount(category_index)):
+                tag_index = self.index(j, 0, category_index)
+                tag_item = tag_index.internalPointer()
+                if tag_item.type == TagTreeItem.CATEGORY:
+                    if process_level(depth+1, tag_index, start_path):
+                        return True
+                else:
+                    if process_tag(depth+1, tag_index, tag_item, start_path):
+                        return True
+            return False
+
+        for i in xrange(self.rowCount(QModelIndex())):
+            if process_level(0, self.index(i, 0, QModelIndex()), start_path):
+                break
+        return self.path_found
+
+    def show_item_at_path(self, path, box=False):
+        '''
+        Scroll the browser and open categories to show the item referenced by
+        path. If possible, the item is placed in the center. If box=True, a
+        box is drawn around the item.
+        '''
+        if path:
+            self.show_item_at_index(self.index_for_path(path), box)
+
+    def show_item_at_index(self, idx, box=False):
+        if idx.isValid():
+            self.tags_view.setCurrentIndex(idx)
+            self.tags_view.scrollTo(idx, QTreeView.PositionAtCenter)
+            if box:
+                tag_item = idx.internalPointer()
+                tag_item.boxed = True
+                self.dataChanged.emit(idx, idx)
+
+    def clear_boxed(self):
+        '''
+        Clear all boxes around items.
+        '''
+        def process_tag(tag_index, tag_item):
+            if tag_item.boxed:
+                tag_item.boxed = False
+                self.dataChanged.emit(tag_index, tag_index)
+
+        def process_level(category_index):
+            for j in xrange(self.rowCount(category_index)):
+                tag_index = self.index(j, 0, category_index)
+                tag_item = tag_index.internalPointer()
+                if tag_item.type == TagTreeItem.CATEGORY:
+                    process_level(tag_index)
+                else:
+                    process_tag(tag_index, tag_item)
+
+        for i in xrange(self.rowCount(QModelIndex())):
+            process_level(self.index(i, 0, QModelIndex()))
+
+    def get_filter_categories_by(self):
+        return self.filter_categories_by
+
     # }}}
 
 class TagBrowserMixin(object): # {{{
@@ -993,14 +1209,73 @@ class TagBrowserWidget(QWidget): # {{{
 
     def __init__(self, parent):
         QWidget.__init__(self, parent)
+        self.parent = parent
         self._layout = QVBoxLayout()
         self.setLayout(self._layout)
         self._layout.setContentsMargins(0,0,0,0)
 
+        # Set up the find box & button
+        search_layout = QHBoxLayout()
+        self._layout.addLayout(search_layout)
+        self.item_search = HistoryLineEdit(parent)
+        try:
+            self.item_search.lineEdit().setPlaceholderText(
+                                                _('Find item in tag browser'))
+        except:
+            pass             # Using Qt < 4.7
+        self.item_search.setToolTip(_(
+        'Search for items. This is a "contains" search; items containing the\n'
+        'text anywhere in the name will be found. You can limit the search\n'
+        'to particular categories using syntax similar to search. For example,\n'
+        'tags:foo will find foo in any tag, but not in authors etc. Entering\n'
+        '*foo will filter all categories at once, showing only those items\n'
+        'containing the text "foo"'))
+        search_layout.addWidget(self.item_search)
+        # Not sure if the shortcut should be translatable ...
+        sc = QShortcut(QKeySequence(_('ALT+f')), parent)
+        sc.connect(sc, SIGNAL('activated()'), self.set_focus_to_find_box)
+
+        self.search_button = QPushButton()
+        self.search_button.setText(_('F&ind'))
+        self.search_button.setToolTip(_('Find the first/next matching item'))
+        self.search_button.setFixedWidth(40)
+        search_layout.addWidget(self.search_button)
+
+        self.expand_button = QPushButton()
+        self.expand_button.setText('-')
+        self.expand_button.setFixedWidth(20)
+        self.expand_button.setToolTip(_('Collapse all categories'))
+        search_layout.addWidget(self.expand_button)
+
+        self.current_find_position = None
+        self.search_button.clicked.connect(self.find)
+        self.item_search.initialize('tag_browser_search')
+        self.item_search.lineEdit().returnPressed.connect(self.do_find)
+        self.item_search.lineEdit().textEdited.connect(self.find_text_changed)
+        self.item_search.activated[QString].connect(self.do_find)
+        self.item_search.completer().setCaseSensitivity(Qt.CaseSensitive)
+
         parent.tags_view = TagsView(parent)
         self.tags_view = parent.tags_view
+        self.expand_button.clicked.connect(self.tags_view.collapseAll)
         self._layout.addWidget(parent.tags_view)
 
+        # Now the floating 'not found' box
+        l = QLabel(self.tags_view)
+        self.not_found_label = l
+        l.setFrameStyle(QFrame.StyledPanel)
+        l.setAutoFillBackground(True)
+        l.setText('<p><b>'+_('No More Matches.</b><p> Click Find again to go to first match'))
+        l.setAlignment(Qt.AlignVCenter)
+        l.setWordWrap(True)
+        l.resize(l.sizeHint())
+        l.move(10,20)
+        l.setVisible(False)
+        self.not_found_label_timer = QTimer()
+        self.not_found_label_timer.setSingleShot(True)
+        self.not_found_label_timer.timeout.connect(self.not_found_label_timer_event,
+                                                   type=Qt.QueuedConnection)
+
         parent.sort_by = QComboBox(parent)
         # Must be in the same order as db2.CATEGORY_SORTS
         for x in (_('Sort by name'), _('Sort by popularity'),
@@ -1031,6 +1306,63 @@ class TagBrowserWidget(QWidget): # {{{
     def set_pane_is_visible(self, to_what):
         self.tags_view.set_pane_is_visible(to_what)
 
+    def find_text_changed(self, str):
+        self.current_find_position = None
+
+    def set_focus_to_find_box(self):
+        self.item_search.setFocus()
+        self.item_search.lineEdit().selectAll()
+
+    def do_find(self, str=None):
+        self.current_find_position = None
+        self.find()
+
+    def find(self):
+        model = self.tags_view.model()
+        model.clear_boxed()
+        txt = unicode(self.item_search.currentText()).strip()
+
+        if txt.startswith('*'):
+            self.tags_view.set_new_model(filter_categories_by=txt[1:])
+            self.current_find_position = None
+            return
+        if model.get_filter_categories_by():
+            self.tags_view.set_new_model(filter_categories_by=None)
+            self.current_find_position = None
+            model = self.tags_view.model()
+
+        if not txt:
+            return
+
+        self.item_search.lineEdit().blockSignals(True)
+        self.search_button.setFocus(True)
+        self.item_search.lineEdit().blockSignals(False)
+
+        colon = txt.find(':')
+        key = None
+        if colon > 0:
+            key = self.parent.library_view.model().db.\
+                        field_metadata.search_term_to_field_key(txt[:colon])
+            txt = txt[colon+1:]
+
+        self.current_find_position = model.find_node(key, txt,
+                                                     self.current_find_position)
+        if self.current_find_position:
+            model.show_item_at_path(self.current_find_position, box=True)
+        elif self.item_search.text():
+            self.not_found_label.setVisible(True)
+            if self.tags_view.verticalScrollBar().isVisible():
+                sbw = self.tags_view.verticalScrollBar().width()
+            else:
+                sbw = 0
+            width = self.width() - 8 - sbw
+            height = self.not_found_label.heightForWidth(width) + 20
+            self.not_found_label.resize(width, height)
+            self.not_found_label.move(4, 10)
+            self.not_found_label_timer.start(2000)
+
+    def not_found_label_timer_event(self):
+        self.not_found_label.setVisible(False)
 
 # }}}
 

src/calibre/gui2/widgets.py

@@ -551,7 +551,11 @@ class HistoryLineEdit(QComboBox):
             item = unicode(self.itemText(i))
             if item not in items:
                 items.append(item)
-
+        self.blockSignals(True)
+        self.clear()
+        self.addItems(items)
+        self.setEditText(ct)
+        self.blockSignals(False)
         history.set(self.store_name, items)
 
     def setText(self, t):

src/calibre/gui2/wizard/__init__.py

@@ -174,7 +174,7 @@ class CybookOrizon(CybookOpus):
 class PocketBook360(CybookOpus):
 
     manufacturer = 'PocketBook'
-    name = 'PocketBook 360'
+    name = 'PocketBook 360 and newer models'
     id = 'pocketbook360'
     output_profile = 'cybook_opus'
 

src/calibre/gui2/wizard/send_email.py

@@ -144,8 +144,10 @@ class SendEmail(QWidget, Ui_Form):
         bb = QDialogButtonBox(QDialogButtonBox.Ok|QDialogButtonBox.Cancel)
         bb.accepted.connect(d.accept)
         bb.rejected.connect(d.reject)
-        d.tl = QLabel('<p>'+_('You can sign up for a free {name} email '
+        d.tl = QLabel(('<p>'+_('Setup sending email using') +
-            'account at <a href="http://{url}">http://{url}</a>. {extra}').format(
+                ' <b>{name}</b><p>' +
+            _('If you don\'t have an account, you can sign up for a free {name} email '
+            'account at <a href="http://{url}">http://{url}</a>. {extra}')).format(
                 **service))
         l.addWidget(d.tl, 0, 0, 3, 0)
         d.tl.setWordWrap(True)

src/calibre/library/__init__.py

@@ -2,10 +2,11 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
 ''' Code to manage ebook library'''
 
-def db(path=None):
+def db(path=None, read_only=False):
     from calibre.library.database2 import LibraryDatabase2
     from calibre.utils.config import prefs
-    return LibraryDatabase2(path if path else prefs['library_path'])
+    return LibraryDatabase2(path if path else prefs['library_path'],
+            read_only=read_only)
 
 
 def generate_test_db(library_path, # {{{

src/calibre/library/caches.py

@@ -669,6 +669,9 @@ class ResultCache(SearchQueryParser): # {{{
             fields = [('timestamp', False)]
 
         keyg = SortKeyGenerator(fields, self.field_metadata, self._data)
+        # For efficiency, the key generator returns a plain value if only one
+        # field is in the sort field list. Because the normal cmp function will
+        # always assume asc, we must deal with asc/desc here.
         if len(fields) == 1:
             self._map.sort(key=keyg, reverse=not fields[0][1])
         else:
@@ -697,7 +700,7 @@ class SortKeyGenerator(object):
     def __init__(self, fields, field_metadata, data):
         from calibre.utils.icu import sort_key
         self.field_metadata = field_metadata
-        self.orders = [-1 if x[1] else 1 for x in fields]
+        self.orders = [1 if x[1] else -1 for x in fields]
         self.entries = [(x[0], field_metadata[x[0]]) for x in fields]
         self.library_order = tweaks['title_series_sorting'] == 'library_order'
         self.data = data

src/calibre/library/database2.py

@@ -6,7 +6,7 @@ __docformat__ = 'restructuredtext en'
 '''
 The database used to store ebook metadata
 '''
-import os, sys, shutil, cStringIO, glob, time, functools, traceback, re
+import os, sys, shutil, cStringIO, glob, time, functools, traceback, re, json
 from itertools import repeat
 from math import ceil
 from Queue import Queue
@@ -32,7 +32,7 @@ from calibre.customize.ui import run_plugins_on_import
 from calibre import isbytestring
 from calibre.utils.filenames import ascii_filename
 from calibre.utils.date import utcnow, now as nowf, utcfromtimestamp
-from calibre.utils.config import prefs, tweaks
+from calibre.utils.config import prefs, tweaks, from_json, to_json
 from calibre.utils.icu import sort_key
 from calibre.utils.search_query_parser import saved_searches, set_saved_searches
 from calibre.ebooks import BOOK_EXTENSIONS, check_ebook_format
@@ -102,7 +102,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         if self.user_version == 0:
             self.initialize_database()
         # remember to add any filter to the connect method in sqlite.py as well
-        # so that various code taht connects directly will not complain about
+        # so that various code that connects directly will not complain about
         # missing functions
         self.books_list_filter = self.conn.create_dynamic_filter('books_list_filter')
         # Store temporary tables in memory
@@ -113,7 +113,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
     def exists_at(cls, path):
         return path and os.path.exists(os.path.join(path, 'metadata.db'))
 
-    def __init__(self, library_path, row_factory=False, default_prefs=None):
+    def __init__(self, library_path, row_factory=False, default_prefs=None,
+            read_only=False):
         self.field_metadata = FieldMetadata()
         self.dirtied_queue = Queue()
         if not os.path.exists(library_path):
@@ -127,6 +128,14 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         if isinstance(self.dbpath, unicode) and not iswindows:
             self.dbpath = self.dbpath.encode(filesystem_encoding)
 
+        if read_only and os.path.exists(self.dbpath):
+            # Work on only a copy of metadata.db to ensure that
+            # metadata.db is not changed
+            pt = PersistentTemporaryFile('_metadata_ro.db')
+            pt.close()
+            shutil.copyfile(self.dbpath, pt.name)
+            self.dbpath = pt.name
+
         apply_default_prefs = not os.path.exists(self.dbpath)
         self.connect()
 
@@ -1243,7 +1252,6 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                 else:
                     icon = icon_map[':custom']
                     icon_map[category] = icon
-                    tooltip = self.custom_column_label_map[label]['name']
 
             datatype = cat['datatype']
             avgr = lambda x: 0.0 if x.rc == 0 else x.rt/x.rc
@@ -2700,6 +2708,38 @@ books_series_link      feeds
 
         return duplicates
 
+    def add_custom_book_data(self, book_id, name, val):
+        x = self.conn.get('SELECT id FROM books WHERE ID=?', (book_id,), all=False)
+        if x is None:
+            raise ValueError('add_custom_book_data: no such book_id %d'%book_id)
+        # Do the json encode first, in case it throws an exception
+        s = json.dumps(val, default=to_json)
+        self.conn.execute('DELETE FROM books_plugin_data WHERE book=? AND name=?',
+                          (book_id, name))
+        self.conn.execute('''INSERT INTO books_plugin_data(book, name, val)
+                             VALUES(?, ?, ?)''', (book_id, name, s))
+        self.commit()
+
+    def get_custom_book_data(self, book_id, name, default=None):
+        try:
+            s = self.conn.get('''select val FROM books_plugin_data
+                    WHERE book=? AND name=?''', (book_id, name), all=False)
+            if s is None:
+                return default
+            return json.loads(s, object_hook=from_json)
+        except:
+            pass
+        return default
+
+    def delete_custom_book_data(self, book_id, name):
+        self.conn.execute('DELETE FROM books_plugin_data WHERE book=? AND name=?',
+                          (book_id, name))
+        self.commit()
+
+    def get_ids_for_custom_book_data(self, name):
+        s = self.conn.get('''SELECT book FROM books_plugin_data WHERE name=?''', (name,))
+        return [x[0] for x in s]
+
     def get_custom_recipes(self):
         for id, title, script in self.conn.get('SELECT id,title,script FROM feeds'):
             yield id, title, script
@@ -2738,7 +2778,6 @@ books_series_link      feeds
                 os.remove(dest)
             raise
         else:
-            os.remove(self.dbpath)
             shutil.copyfile(dest, self.dbpath)
             self.connect()
             self.initialize_dynamic()

src/calibre/library/schema_upgrades.py

@@ -441,3 +441,31 @@ class SchemaUpgrade(object):
                          WHERE id=NEW.id AND OLD.title <> NEW.title;
             END;
         ''')
+
+    def upgrade_version_17(self):
+        'custom book data table (for plugins)'
+        script = '''
+        DROP TABLE IF EXISTS books_plugin_data;
+        CREATE TABLE books_plugin_data(id INTEGER PRIMARY KEY,
+                                     book INTEGER NON NULL,
+                                     name TEXT NON NULL,
+                                     val TEXT NON NULL,
+                                     UNIQUE(book,name));
+        DROP TRIGGER IF EXISTS books_delete_trg;
+        CREATE TRIGGER books_delete_trg
+            AFTER DELETE ON books
+            BEGIN
+                DELETE FROM books_authors_link WHERE book=OLD.id;
+                DELETE FROM books_publishers_link WHERE book=OLD.id;
+                DELETE FROM books_ratings_link WHERE book=OLD.id;
+                DELETE FROM books_series_link WHERE book=OLD.id;
+                DELETE FROM books_tags_link WHERE book=OLD.id;
+                DELETE FROM data WHERE book=OLD.id;
+                DELETE FROM comments WHERE book=OLD.id;
+                DELETE FROM conversion_options WHERE book=OLD.id;
+                DELETE FROM books_plugin_data WHERE book=OLD.id;
+        END;
+        '''
+        self.conn.executescript(script)
+
+

src/calibre/manual/customize.rst

@@ -21,6 +21,7 @@ Environment variables
 -----------------------
 
     * ``CALIBRE_CONFIG_DIRECTORY`` - sets the directory where configuration files are stored/read.
+    * ``CALIBRE_TEMP_DIR`` - sets the temporary directory used by calibre
     * ``CALIBRE_OVERRIDE_DATABASE_PATH`` - allows you to specify the full path to metadata.db. Using this variable you can have metadata.db be in a location other than the library folder. Useful if your library folder is on a networked drive that does not support file locking.
     * ``CALIBRE_DEVELOP_FROM`` - Used to run from a calibre development environment. See :ref:`develop`.
     * ``CALIBRE_OVERRIDE_LANG`` - Used to force the language used by the interface (ISO 639 language code)

src/calibre/manual/faq.rst

@@ -273,7 +273,7 @@ will always be under /dev by examining the output of::
     mount
 
 
-Why does |app| not support collection on the Kindle or shelves on the Nook?
+Why does |app| not support collections on the Kindle or shelves on the Nook?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Neither the Kindle nor the Nook provide any way to manipulate collections over a USB connection.

src/calibre/manual/template_lang.rst

@@ -137,8 +137,8 @@ Note that you can use the prefix and suffix as well. If you want the number to a
     {#myint:0>3s:ifempty(0)|[|]}
     
 
-Using functions in templates - program mode
+Using functions in templates - template program mode
--------------------------------------------
+----------------------------------------------------
 
 The template language program mode differs from single-function mode in that it permits you to write template expressions that refer to other metadata fields, modify values, and do arithmetic. It is a reasonably complete programming language.
 
@@ -161,10 +161,13 @@ The syntax of the language is shown by the following grammar::
     constant   ::= " string " | ' string ' | number
     identifier ::= sequence of letters or ``_`` characters
     function   ::= identifier ( statement [ , statement ]* )
-    expression ::= identifier | constant | function
+    expression ::= identifier | constant | function | assignment
+    assignment ::= identifier '=' expression
     statement  ::= expression [ ; expression ]*
     program    ::= statement
 
+Comments are lines with a '#' character at the beginning of the line.
+
 An ``expression`` always has a value, either the value of the constant, the value contained in the identifier, or the value returned by a function. The value of a ``statement`` is the value of the last expression in the sequence of statements. As such, the value of the program (statement)::
 
     1; 2; 'foobar'; 3
@@ -208,13 +211,102 @@ The following functions are available in addition to those described in single-f
     * ``cmp(x, y, lt, eq, gt)`` -- compares x and y after converting both to numbers. Returns ``lt`` if x < y. Returns ``eq`` if x == y. Otherwise returns ``gt``.
     * ``divide(x, y)`` -- returns x / y. Throws an exception if either x or y are not numbers.
     * ``field(name)`` -- returns the metadata field named by ``name``.
+    * ``eval(string)`` -- evaluates the string as a program, passing the local variables (those ``assign`` ed to). This permits using the template processor to construct complex results from local variables.
     * ``multiply(x, y)`` -- returns x * y. Throws an exception if either x or y are not numbers.
+    * ``print(a, b, ...)`` -- prints the arguments to standard output. Unless you start calibre from the command line (``calibre-debug -g``), the output will go to a black hole.
     * ``strcat(a, b, ...)`` -- can take any number of arguments. Returns a string formed by concatenating all the arguments.
     * ``strcmp(x, y, lt, eq, gt)`` -- does a case-insensitive comparison x and y as strings. Returns ``lt`` if x < y. Returns ``eq`` if x == y. Otherwise returns ``gt``.
     * ``substr(str, start, end)`` -- returns the ``start``'th through the ``end``'th characters of ``str``. The first character in ``str`` is the zero'th character. If end is negative, then it indicates that many characters counting from the right. If end is zero, then it indicates the last character. For example, ``substr('12345', 1, 0)`` returns ``'2345'``, and ``substr('12345', 1, -1)`` returns ``'234'``.
     * ``subtract(x, y)`` -- returns x - y. Throws an exception if either x or y are not numbers.
     * ``template(x)`` -- evaluates x as a template. The evaluation is done in its own context, meaning that variables are not shared between the caller and the template evaluation. Because the `{` and `}` characters are special, you must use `[[` for the `{` character and `]]` for the '}' character; they are converted automatically. For example, ``template('[[title_sort]]') will evaluate the template ``{title_sort}`` and return its value.
-    
+
+Using general program mode
+-----------------------------------
+
+For more complicated template programs, it is sometimes easier to avoid template syntax (all the `{` and `}` characters), instead writing a more classical-looking program. You can do this in |app| by beginning the template with `program:`. In this case, no template processing is done. The special variable `$` is not set. It is up to your program to produce the correct results.
+
+One advantage of `program:` mode is that the brackets are no longer special. For example, it is not necessary to use `[[` and `]]` when using the `template()` function.
+
+The following example is a `program:` mode implementation of a recipe on the MobileRead forum: "Put series into the title, using either initials or a shortened form. Strip leading articles from the series name (any)." For example, for the book The Two Towers in the Lord of the Rings series, the recipe gives `LotR [02] The Two Towers`. Using standard templates, the recipe requires three custom columns and a plugboard, as explained in the following:
+
+The solution requires creating three composite columns. The first column is used to remove the leading articles. The second is used to compute the 'shorten' form. The third is to compute the 'initials' form. Once you have these columns, the plugboard selects between them. You can hide any or all of the three columns on the library view.
+
+    First column:
+    Name: #stripped_series. 
+    Template: {series:re(^(A|The|An)\s+,)||}
+
+    Second column (the shortened form):
+    Name: #shortened. 
+    Template: {#stripped_series:shorten(4,-,4)}
+
+    Third column (the initials form):
+    Name: #initials. 
+    Template: {#stripped_series:re(([^\s])[^\s]+(\s|$),\1)}
+
+    Plugboard expression:
+    Template:{#stripped_series:lookup(.\s,#initials,.,#shortened,series)}{series_index:0>2.0f| [|] }{title}
+    Destination field: title
+
+    This set of fields and plugboard produces:
+    Series: The Lord of the Rings
+    Series index: 2
+    Title: The Two Towers
+    Output: LotR [02] The Two Towers
+
+    Series: Dahak
+    Series index: 1
+    Title: Mutineers Moon
+    Output: Dahak [01] Mutineers Moon
+
+    Series: Berserkers
+    Series Index: 4
+    Title: Berserker Throne
+    Output: Bers-kers [04] Berserker Throne
+
+    Series: Meg Langslow Mysteries
+    Series Index: 3
+    Title: Revenge of the Wrought-Iron Flamingos
+    Output: MLM [03] Revenge of the Wrought-Iron Flamingos
+
+The following program produces the same results as the original recipe, using only one custom column to hold the results of a program that computes the special title value::
+
+    Custom column: 
+    Name: #special_title
+    Template: (the following with all leading spaces removed)
+        program:
+        #	compute the equivalent of the composite fields and store them in local variables
+            stripped = re(field('series'), '^(A|The|An)\s+', '');
+            shortened = shorten(stripped, 4, '-' ,4);
+            initials = re(stripped, '[^\w]*(\w?)[^\s]+(\s|$)', '\1');
+
+        #	Format the series index. Ends up as empty if there is no series index.
+        #	Note that leading and trailing spaces will be removed by the formatter,
+        # 	so we cannot add them here. We will do that in the strcat below.
+        #	Also note that because we are in 'program' mode, we can freely use
+        #	curly brackets in strings, something we cannot do in template mode.
+            s_index = template('{series_index:0>2.0f}');
+
+        #	print(stripped, shortened, initials, s_index);
+
+        #	Now concatenate all the bits together. The switch picks between 
+        # 	initials and shortened, depending on whether there is a space
+        #	in stripped. We then add the brackets around s_index if it is
+        #	not empty. Finally, add the title. As this is the last function in
+        # 	the program, its value will be returned.
+            strcat(
+                switch(	stripped, 
+                        '.\s', initials, 
+                        '.', shortened,
+                        field('series')),
+                test(s_index, strcat(' [', s_index, '] '), ''),
+                field('title'));
+
+    Plugboard expression:
+    Template:{#special_title}
+    Destination field: title
+
+It would be possible to do the above with no custom columns by putting the program into the template box of the plugboard. However, to do so, all comments must be removed because the plugboard text box does not support multi-line editing. It is debatable whether the gain of not having the custom column is worth the vast increase in difficulty caused by the program being one giant line.
+
 Special notes for save/send templates
 -------------------------------------
 
@@ -257,4 +349,4 @@ You might find the following tips useful.
     * Templates can use other templates by referencing a composite custom column.
     * In a plugboard, you can set a field to empty (or whatever is equivalent to empty) by using the special template ``{null}``. This template will always evaluate to an empty string.
     * The technique described above to show numbers even if they have a zero value works with the standard field series_index.
-    
+    

src/calibre/ptempfile.py

@@ -40,7 +40,7 @@ def base_dir():
             _base_dir = td
         else:
             _base_dir = tempfile.mkdtemp(prefix='%s_%s_tmp_'%(__appname__,
-                __version__))
+                __version__), dir=os.environ.get('CALIBRE_TEMP_DIR', None))
             atexit.register(remove_dir, _base_dir)
     return _base_dir