Merge from trunk

Changelog.yaml

@@ -19,6 +19,81 @@
 #  new recipes:
 #    - title: 
 
+- version: 0.7.47
+  date: 2011-02-25
+
+  new features:
+    - title: "Tag Browser: Support the creation of nested User Categories"
+      description: "See http://calibre-ebook.com/user_manual/gui.html#tag-browser for details"
+      type: major
+
+    - title: "Disable Kent District Library plugin to download series information. The website could not handle the load calibre's 2 million users put on it. You can manually re-enable it if you really want series information, but it is very slow"
+
+    - title: "Drivers for the Wexler T7001, Archos 7, Wink and Xperia X10"
+
+    - title: "Comic Input: Add option to not add links to individual pages to the Table of Contents when converting CBC files"
+
+    - title: "EPUB Output: Try to ensure that the cover image always has an id='cover' to workaround Nook cover reading bug."
+      tickets: [8182]
+
+    - title: "ODT input: Update odfpy library to latest version, adds support for bookmarks"
+
+    - title: "EPUB Output: Remove unnecessary CSS page breaks as they confuse the latest release of iBooks"
+  
+  bug fixes:
+    - title: "Fix regression in 0.7.46 that broke creating date and composite custom columns"
+
+    - title: "Linux binary build: Fix ImageMagick trying to load system modules instead of bundled modules"
+
+    - title: "Kobo driver: Handle missing firmware version file"
+
+    - title: "ODT Input: Do not force the background color to white."
+      tickets: [9118]
+
+    - title: "MOBI Input: Do not speciy text-align for every paragraph. Fixes text-align inheritance issues for newer MOBIs with nested divs."
+      tickets: [9098]
+
+    - title: "EPUB Output: Do not set the file-as attribute on title elements in the OPF as the current OPF spec does not support file-as. Instead use a calibre extension to OPF."
+      tickets: [9109]
+
+    - title: "Content server: Fix regression that broke browsing User Categories via OPDS"
+      tickets: [9090]
+
+    - title: "Update the book details panel after adding books incase automerge is turned on and the current book is affected"
+      tickets: [9073]
+
+    - title: "FB2 Output: Fix paragraph spacing sometime incorrect."
+      tickets: [8927]
+
+    - title: "Tag Browser: Fix generation of search query for authors with quote characters in their names"
+      tickets: [9071]
+
+    - title: "Fix bug that could cause download of cover/social metadata from Amazon to sometimes fail"
+
+    - title: "LRF Input: Workaround for broken LRF files from BookDesigner that have incomplete TextStyle elements"
+
+  improved recipes:
+    - Le Monde
+    - Gizmodo
+    - Lifehacker
+    - ESPN
+    - Adevarul
+    - gsp.ro
+    - Ming Pao
+
+  new recipes:
+    - title: "Flickr Blog"
+      author: Ricardo Jurado
+
+    - title: "Various Romanian news sources"
+      author: Silviu Coatara
+
+    - title: "Osnews.pl and SwiatKindle"
+      author: Mori
+
+    - title: "Roger Ebert Journal"
+      author: Shane Erstad
+
 - version: 0.7.46
   date: 2011-02-18
 

resources/recipes/deadspin.recipe

@@ -16,14 +16,9 @@ class Deadspin(BasicNewsRecipe):
     max_articles_per_feed = 100
     no_stylesheets        = True
     encoding              = 'utf-8'
-    use_embedded_content  = False
+    use_embedded_content  = True
     language              = 'en'
     masthead_url          = 'http://cache.gawkerassets.com/assets/deadspin.com/img/logo.png'
-    extra_css             = '''
-	body{font-family: "Lucida Grande",Helvetica,Arial,sans-serif}
-	img{margin-bottom: 1em}
-	h1{font-family :Arial,Helvetica,sans-serif; font-size:large}
-	              '''
     conversion_options = {
                           'comment'   : description
                         , 'tags'      : category
@@ -31,13 +26,11 @@ class Deadspin(BasicNewsRecipe):
                         , 'language'  : language
                         }
 
-    remove_attributes  = ['width','height']
+    remove_tags = [
-    keep_only_tags     = [dict(attrs={'class':'content permalink'})]
+            {'class': 'feedflare'},
-    remove_tags_before = dict(name='h1')
+    ]
-    remove_tags        = [dict(attrs={'class':'contactinfo'})]
-    remove_tags_after  = dict(attrs={'class':'contactinfo'})
 
-    feeds = [(u'Articles', u'http://feeds.gawker.com/deadspin/full')]
+    feeds = [(u'Articles', u'http://feeds.gawker.com/deadspin/vip?format=xml')]
 
     def preprocess_html(self, soup):
         return self.adeify_images(soup)

resources/recipes/gawker.recipe

@@ -16,14 +16,10 @@ class Gawker(BasicNewsRecipe):
     max_articles_per_feed = 100
     no_stylesheets        = True
     encoding              = 'utf-8'
-    use_embedded_content  = False
+    use_embedded_content  = True
     language              = 'en'
     masthead_url          = 'http://cache.gawkerassets.com/assets/gawker.com/img/logo.png'
-    extra_css             = '''
+
-	body{font-family: "Lucida Grande",Helvetica,Arial,sans-serif}
-	img{margin-bottom: 1em}
-	h1{font-family :Arial,Helvetica,sans-serif; font-size:large}
-	              '''
     conversion_options = {
                           'comment'   : description
                         , 'tags'      : category
@@ -31,13 +27,11 @@ class Gawker(BasicNewsRecipe):
                         , 'language'  : language
                         }
 
-    remove_attributes  = ['width','height']
+    remove_tags = [
-    keep_only_tags     = [dict(attrs={'class':'content permalink'})]
+            {'class': 'feedflare'},
-    remove_tags_before = dict(name='h1')
+    ]
-    remove_tags        = [dict(attrs={'class':'contactinfo'})]
-    remove_tags_after  = dict(attrs={'class':'contactinfo'})
 
-    feeds = [(u'Articles', u'http://feeds.gawker.com/gawker/full')]
+    feeds = [(u'Articles', u'http://feeds.gawker.com/gawker/vip?format=xml')]
 
     def preprocess_html(self, soup):
         return self.adeify_images(soup)

resources/recipes/swiatkindle.recipe

@@ -3,22 +3,20 @@
 __license__   = 'GPL v3'
 __copyright__ = u'2011, Tomasz Dlugosz <tomek3d@gmail.com>'
 '''
-swiatkindle.pl
+swiatczytnikow.pl
 '''
 
 import re
 
-from calibre.web.feeds.news import BasicNewsRecipe
+class swiatczytnikow(BasicNewsRecipe):
-
+    title          = u'Swiat Czytnikow'
-class swiatkindle(BasicNewsRecipe):
+    description    = u'Czytniki e-książek w Polsce. Jak wybrać, kupić i korzystać z Amazon Kindle i innych'
-    title          = u'Swiat Kindle'
-    description    = u'Blog o czytniku Amazon Kindle. Wersje, ksi\u0105\u017cki, kupowanie i korzystanie w Polsce'
     language = 'pl'
     __author__ = u'Tomasz D\u0142ugosz'
     oldest_article = 7
     max_articles_per_feed = 100
 
-    feeds       = [(u'\u015awiat Kindle - wpisy', u'http://swiatkindle.pl/feed')]
+    feeds       = [(u'Świat Czytników - wpisy', u'http://swiatczytnikow.pl/feed')]
 
     remove_tags = [dict(name = 'ul', attrs = {'class' : 'similar-posts'})]
 

resources/recipes/zeitde_sub.recipe

@@ -2,9 +2,9 @@
 # -*- coding: utf-8 mode: python -*-
 
 __license__   = 'GPL v3'
-__copyright__ = '2010, Steffen Siebert <calibre at steffensiebert.de>'
+__copyright__ = '2010-2011, Steffen Siebert <calibre at steffensiebert.de>'
 __docformat__ = 'restructuredtext de'
-__version__   = '1.1'
+__version__   = '1.2'
 
 """
 Die Zeit EPUB
@@ -13,21 +13,43 @@ Die Zeit EPUB
 import os, urllib2, zipfile, re
 from calibre.web.feeds.news import BasicNewsRecipe
 from calibre.ptempfile import PersistentTemporaryFile
+from calibre import walk
 
 class ZeitEPUBAbo(BasicNewsRecipe):
 
-    title = u'Zeit Online Premium'
+    title = u'Die Zeit'
     description = u'Das EPUB Abo der Zeit (needs subscription)'
     language = 'de'
     lang = 'de-DE'
 
-    __author__ = 'Steffen Siebert'
+    __author__ = 'Steffen Siebert and Tobias Isenberg'
     needs_subscription = True
 
     conversion_options = {
-        'no_default_epub_cover' : True
+        'no_default_epub_cover' : True,
+        # fixing the wrong left margin
+        'mobi_ignore_margins' : True,
     }
 
+    preprocess_regexps    = [
+        # filtering for correct dashes
+        (re.compile(r' - '), lambda match: ' – '), # regular "Gedankenstrich"
+        (re.compile(r' -,'), lambda match: ' –,'), # "Gedankenstrich" before a comma
+        (re.compile(r'(?<=\d)-(?=\d)'), lambda match: '–'), # number-number
+        # filtering for unicode characters that are missing on the Kindle,
+        # try to replace them with meaningful work-arounds
+        (re.compile(u'\u2080'), lambda match: '<span style="font-size: 50%;">0</span>'), # subscript-0
+        (re.compile(u'\u2081'), lambda match: '<span style="font-size: 50%;">1</span>'), # subscript-1
+        (re.compile(u'\u2082'), lambda match: '<span style="font-size: 50%;">2</span>'), # subscript-2
+        (re.compile(u'\u2083'), lambda match: '<span style="font-size: 50%;">3</span>'), # subscript-3
+        (re.compile(u'\u2084'), lambda match: '<span style="font-size: 50%;">4</span>'), # subscript-4
+        (re.compile(u'\u2085'), lambda match: '<span style="font-size: 50%;">5</span>'), # subscript-5
+        (re.compile(u'\u2086'), lambda match: '<span style="font-size: 50%;">6</span>'), # subscript-6
+        (re.compile(u'\u2087'), lambda match: '<span style="font-size: 50%;">7</span>'), # subscript-7
+        (re.compile(u'\u2088'), lambda match: '<span style="font-size: 50%;">8</span>'), # subscript-8
+        (re.compile(u'\u2089'), lambda match: '<span style="font-size: 50%;">9</span>'), # subscript-9
+    ]
+
     def build_index(self):
         domain = "http://premium.zeit.de"
         url = domain + "/abovorteile/cgi-bin/_er_member/p4z.fpl?ER_Do=getUserData&ER_NextTemplate=login_ok"
@@ -55,9 +77,36 @@ class ZeitEPUBAbo(BasicNewsRecipe):
         zfile.extractall(self.output_dir)
 
         tmp.close()
+
         index = os.path.join(self.output_dir, 'content.opf')
 
         self.report_progress(1,_('epub downloaded and extracted'))
 
+        # doing regular expression filtering
+        for path in walk('.'):
+            (shortname, extension) = os.path.splitext(path)
+            if extension.lower() in ('.html', '.htm', '.xhtml'):
+                with open(path, 'r+b') as f:
+                    raw = f.read()
+                    raw = raw.decode('utf-8')
+                    for pat, func in self.preprocess_regexps:
+                        raw = pat.sub(func, raw)
+                    f.seek(0)
+                    f.truncate()
+                    f.write(raw.encode('utf-8'))
+
+        # adding real cover
+        self.report_progress(0,_('trying to download cover image (titlepage)'))
+        self.download_cover()
+        self.conversion_options["cover"] = self.cover_path
+
         return index
 
+    # getting url of the cover
+    def get_cover_url(self):
+        try:
+            inhalt = self.index_to_soup('http://www.zeit.de/inhalt')
+            cover_url = inhalt.find('div', attrs={'class':'singlearchive clearfix'}).img['src'].replace('icon_','')
+        except:
+            cover_url = 'http://images.zeit.de/bilder/titelseiten_zeit/1946/001_001.jpg'
+        return cover_url

src/calibre/constants.py

@@ -2,7 +2,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = 'calibre'
-__version__   = '0.7.46'
+__version__   = '0.7.47'
 __author__    = "Kovid Goyal <kovid@kovidgoyal.net>"
 
 import re

src/calibre/manual/gui.rst

@@ -40,7 +40,7 @@ The actions toolbar provides convenient shortcuts to commonly used actions. Most
 Add books
 ~~~~~~~~~~~~~~~~~~
 .. |adbi| image:: images/add_books.png
-    :class: float-left-img
+    :class: float-right-img
 
 |adbi| The :guilabel:`Add books` action has five variations, accessed by the clicking the down arrow on the right side of the button.
 
@@ -65,7 +65,7 @@ To add an additional format for an existing book, use the :ref:`edit_meta_inform
 Edit metadata
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 .. |emii| image:: images/edit_meta_information.png
-    :class: float-left-img
+    :class: float-right-img
 
 |emii| The :guilabel:`Edit metadata` action has six variations, which can be accessed by clicking the down arrow on the right side of the button.
 
@@ -83,7 +83,7 @@ Edit metadata
 Convert e-books
 ~~~~~~~~~~~~~~~~~~~~~~
 .. |cei| image:: images/convert_ebooks.png
-    :class: float-left-img
+    :class: float-right-img
 
 |cei| Ebooks can be converted from a number of formats into whatever format your e-book reader prefers.
 Note that ebooks you purchase will typically have `Digital Rights Management <http://bugs.calibre-ebook.com/wiki/DRM>`_ *(DRM)*.
@@ -110,7 +110,7 @@ The :guilabel:`Convert E-books` action has three variations, accessed by the arr
 View
 ~~~~~~~~~~~
 .. |vi| image:: images/view.png
-    :class: float-left-img
+    :class: float-right-img
 
 |vi| The :guilabel:`View` action displays the book in an ebook viewer program. |app| has a builtin viewer for the most e-book formats.
 For other formats it uses the default operating system application. You can configure which formats should open with the internal viewer via
@@ -123,7 +123,7 @@ on the right of the :guilabel:`View` button.
 Send to device
 ~~~~~~~~~~~~~~~~~~~~~~~~
 .. |stdi| image:: images/send_to_device.png
-    :class: float-left-img
+    :class: float-right-img
 
 |stdi| The :guilabel:`Send to device` action has eight variations, accessed by clicking the down arrow on the right of the button.
 
@@ -144,7 +144,7 @@ You can control the file name and folder structure of files sent to the device b
 Fetch news
 ~~~~~~~~~~~~~~~~~
 .. |fni| image:: images/fetch_news.png
-    :class: float-left-img
+    :class: float-right-img
 
 |fni| The :guilabel:`Fetch news` action downloads news from various websites and converts it into an ebook that can be read on your ebook reader. Normally, the newly created ebook is added to your ebook library, but if an ebook reader is connected at the time the download finishes, the news is also uploaded to the reader automatically.
             
@@ -162,7 +162,7 @@ The :guilabel:`Fetch news` action has three variations, accessed by clicking the
 Library
 ~~~~~~~~~~~~~~~~~
 .. |lii| image:: images/library.png
-    :class: float-left-img
+    :class: float-right-img
 
 |lii| The :guilabel: `Library` action allows you to create, switch between, rename or delete a Library.  |app| allows you to create as many libraries as you wish. You coudl for instance create a fiction library, a non fiction library, a foreign language library a project library, basically any structure that suits your needs. Libraries are the highest organizational structure within |app|, each library has its own set of books, tags, categories and base storage location.
 
@@ -177,7 +177,7 @@ Library
 Device
 ~~~~~~~~~~~~~~~~~
 .. |dvi| image:: images/device.png
-    :class: float-left-img
+    :class: float-right-img
 
 |dvi| The :guilabel:`Device` action allows you to view the books in the main memory or storage cards of your device, or to eject the device (detach it from |app|).
 This icon shows up automatically on the main |app| toolbar when you connect a supported device. You can click on it to see the books on your device. You can also drag  and drop books from your |app| library onto the icon to transfer them to your device. Conversely, you can drag and drop books from your device onto the |app| icon on the toolbar to transfer books from your device to the |app| library. 
@@ -188,7 +188,7 @@ This icon shows up automatically on the main |app| toolbar when you connect a su
 Save to disk
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 .. |svdi| image:: images/save_to_disk.png
-    :class: float-left-img
+    :class: float-right-img
 
 |svdi| The :guilabel:`Save to disk` action has five variations, accessed by the arrow next to the button.
 
@@ -222,7 +222,7 @@ Save to disk
 Connect/Share
 ~~~~~~~~~~~~~~~~~
 .. |csi| image:: images/connect_share.png
-    :class: float-left-img
+    :class: float-right-img
 
 |csi| The :guilabel:`Connect/Share` action allows you to manually connect to a device or folder on your computer, it also allows you to set up you |app| library for access via a web browser, or email.
 
@@ -241,7 +241,7 @@ Connect/Share
 Remove books
 ~~~~~~~~~~~~~~~~~~~~~
 .. |rbi| image:: images/remove_books.png
-    :class: float-left-img
+    :class: float-right-img
 
 |rbi| The :guilabel:`Remove books` action **deletes books permanently**, so use it with care. It is *context sensitive*, i.e. it depends on which :ref:`catalog <catalogs>` you have selected. If you have selected the :guilabel:`Library`, books will be removed from the library. If you have selected the ebook reader device, the books will be removed from the device. To remove only a particular format for a given book use the :ref:`edit_meta_information` action.  Remove books also has five variations which can be accessed by clicking the down arrow on the right side of the button.
 

src/calibre/manual/news.rst

@@ -33,6 +33,7 @@ Lets pick a couple of feeds that look interesting:
 I got the URLs by clicking the little orange RSS icon next to each feed name. To make |app| download the feeds and convert them into an e-book, you should click the :guilabel:`Fetch news` button and then the :guilabel:`Add a custom news source` menu item. A dialog similar to that shown below should open up.
 
 .. image:: images/custom_news.png
+    :align: center
 
 First enter ``Portfolio`` into the :guilabel:`Recipe title` field. This will be the title of the e-book that will be created from the articles in the above feeds. 
 
@@ -81,6 +82,7 @@ So it looks like to get the print version, we need to prefix every article URL w
 Now in the :guilabel:`Advanced Mode` of the Custom  news sources dialog, you should see something like (remember to select *The BBC* recipe before switching to advanced mode):
 
 .. image:: images/bbc_advanced.png
+    :align: center
 
 You can see that the fields from the :guilabel:`Basic mode` have been translated to python code in a straightforward manner. We need to add instructions to this recipe to use the print version of the articles. All that's needed is to add the following two lines:
 
@@ -92,6 +94,7 @@ You can see that the fields from the :guilabel:`Basic mode` have been translated
 This is python, so indentation is important. After you've added the lines, it should look like:
 
 .. image:: images/bbc_altered.png
+    :align: center
 
 In the above, ``def print_version(self, url)`` defines a *method* that is called by |app| for every article. ``url`` is the URL of the original article. What ``print_version`` does is take that url and replace it with the new URL that points to the print version of the article. To learn about `python <http://www.python.org>`_ see the `tutorial <http://docs.python.org/tut/>`_.
 
@@ -109,6 +112,7 @@ The recipe now looks like:
 .. _bbc1:
 
 .. image:: images/bbc_altered1.png
+    :align: center
 
 The new version looks pretty good. If you're a perfectionist, you'll want to read the next section, which deals with actually modifying the downloaded content.
 

src/calibre/manual/templates/layout.html

@@ -20,6 +20,7 @@
 
     <style type="text/css">
         .float-left-img { float: left; margin-right: 1em; margin-bottom: 1em }
+        .float-right-img { float: right; margin-left: 1em; margin-bottom: 1em }
     </style>
 
 {% endblock %}
@@ -31,7 +32,7 @@
       {%- endif %}
         <div class="body">
             {% if not embedded %}
-            <div id="ad-container">
+            <div id="ad-container" style="text-align:center">
                     <script type="text/javascript"><!--
                         google_ad_client = "ca-pub-2595272032872519";
                         /* User Manual horizontal */

src/calibre/manual/viewer.rst

@@ -66,6 +66,7 @@ reference mode. You can turn it on by clicking the Reference Mode button |refmi|
 mouse over a paragraph, calibre will display a unique number made up of the section and paragraph numbers. 
 
 .. image:: images/ref_mode.png
+    :align: center
 
 You can use this number to unambiguously refer to parts of the books when discussing it with friends or referring to it
 in other works. You can enter these numbers in the box marked Go to at the top of the window to go to a particular