Merge from trunk

Changelog.yaml

@@ -4,6 +4,118 @@
 # for important features/bug fixes.
 # Also, each release can have new and improved recipes.
 
+- version: 0.7.21
+  date: 2010-10-1
+
+  new features:
+    - title: "Automatic backup of the calibre metadata database"
+      type: major
+      desc: >
+        "calibre now automatically backups up the metadata for each book in the library into an individual OPF file in that books' folder. This means that if the calibre metadata database is corrupted, for example by a hard disk failure, you can reconstruct it from these OPF files, without losing any metadata. For the moment, only the backup is implemented, restore will be implemented in the future. The backup happens automatically in the background while calibre is running. The first time you start calibre, all the books will need to be backed up, so you may notice calibre running a little slower than usual."
+
+    - title: "Virtual columns"
+      type: major
+      desc: >
+        "You can now add virtual columns to the calibre book list. These are built fro other columns using templates and can be used to, for example, create columns to show the books isbn and avaialbale formats. You can do this by right clicking on a column header and select 'Add your own columns'"
+
+    - title: "calibre templates now much more powerful"
+      type: major
+      desc: >
+        "The templates used in calibre in send to device and save to disk have now beome much ore powerful. They can use conditinal text and functions to transforms the replacement text. Also they now have access t metadata in user defined columns. For details see the tutorials section of the User Manual."
+
+    - title: "Metadata plugboards: Allow you to perform sophisticated transformations on the metadata of a book when exporting it from the calibre library."
+      type: major
+      desc: > 
+        "For example, you can add the series informtion to the title when sendig books to a device. This functionality is accessed from Preferences->Import/Export->Metadata plugboards"
+
+    - title: "User defined columns are now fully integrated into calibre"
+      type: major
+      desc: >
+        "User defined columns can nw be used everywhere. In the content server, Search and Replace, to create ondevice collections, and in the save to disk and send to device templates for creating filenames. In addition, user defined metadata is saved to an read back from EPUB/OPF files."
+
+    - title: "Driver for the jetBook Mini"
+
+    - title: "Add tweaks to control which custom columns the content server displays."
+
+    - title: "Bulk downloading of metadata/covers now shows progress and can be canceled"
+
+    - title: "New plugin to download covers from douban.com. It is disabled by default and must be enabled via Preferences->Advanced->Plugins->Cover download plugins"
+
+    - title: "Add option to change titles to title case in the Bulk metadata edit dialog"
+
+    - title: "Add option to bulk metadata edit dialog to force series renumbering to start with a specified value"
+
+  bug fixes:
+    - title: "Fix various bugs that could lead to stale files being left in the calbre library when editing title/author metadata on windows"
+   
+    - title: "Fix various regression in the preprocess and de-hyphenation code that broke conversion of some files, especially PDF ones."
+
+    - title: "Alex driver: Fix books not being placed in sub directories. Send covers. And allow sending of FB2"
+      tickets: [6956]
+
+    - title: "MOBI Output: Fix bug that could caused left margins in the MOBI file to have twice the size of the left margins in the input document, when viewed on the pathetic Kindle MOBI renderer"
+
+    - title: "MOBI Input: Interpret blockquotes as having a left margin of 2em not 1em to reflect recent Amazon practice"
+
+    - title: "MOBI Output: Remove transparencies from images. Pathetic Kindle MOBI renderer strikes again"
+
+    - title: "Revert removal of inline toc from news downloaded in MOBI format as this makes it unusable with the pathetic Kindle For PC application"
+
+    - title: "Content server: Remove special characters from filenames in download links to accomodate broken browsers like the one in the Kindle"
+
+    - title: "Conversion pipeline: When rescaling images, dont replace gif image data with jpeg data"
+
+    - title: "EPUB Input: Ignore OPF files in the EPUB whose names start with a period"
+
+    - title: "RTF Output: Handle a larger set of broken images in the input document"
+      tickets: [7003]
+
+    - title: "epub-fix: Handle dates before 1900"
+      tickets: [7002]
+
+    - title: "Welcome wizard: Prevent the user from choosing a non empty folder as her calibre library"
+
+    - title: "Automatically enable the Douban metadata download plugins if the user choose chinese as the interface language in the welcome wizard"
+
+    - title: "Linux DBUS notifier: Fix causing freezes on some DBUS implementations"
+      tickets: [6969]
+
+    - title: "Workaround for windows limitation when reading from network sockets. Should fix issues with large files in calibre libraries on network shares."
+      tickets: [3248]
+
+  new recipes:
+    - title: "BBC Sport"
+      author: "limawhiskey"
+      
+    - title: "Revista Muy Interesante "
+      author: "Jefferson Frantz"
+
+    - title: "El Universo - Ecuador and Frederik Pohl's Blog"
+      author: "Darko Miletic"
+
+    - title: "Science News"
+      author: "Starson17"
+
+    - title: "Various Belgian news sources"
+      author: "Lionel Bergeret"
+
+    - title: "Oriental Daily"
+      author: "Larry Chan"
+
+    - title: "Rmf24 - Opinie"
+      author: "Tomasz Dlugosz"
+
+    - title: "Jerusalem Post - French and Howto Geek"
+      author: "Tony Stegall"
+
+
+  improved recipes:
+    - Peter Schiff
+    - Telegraph UK
+    - AJC
+    - Boortz
+    - Scientific American
+
 - version: 0.7.20
   date: 2010-09-24
 

resources/recipes/bbc_sport.recipe

@@ -0,0 +1,65 @@
+__license__   = 'GPL v3'
+__copyright__ = '2010, limawhiskey <limawhiskey at gmail.com>'
+'''
+news.bbc.co.uk/sport/
+'''
+import re
+from calibre.web.feeds.recipes import BasicNewsRecipe
+
+class BBC(BasicNewsRecipe):
+    title                  = 'BBC Sport'
+    __author__             = 'limawhiskey, Darko Miletic, Starson17'
+    description            = 'Sports news from UK. A fast version that does not download pictures'
+    oldest_article         = 2
+    max_articles_per_feed  = 100
+    no_stylesheets         = True
+    use_embedded_content   = False
+    encoding               = 'utf8'
+    publisher              = 'BBC'
+    category               = 'sport, news, UK, world'
+    language               = 'en_GB'
+    publication_type       = 'newsportal'
+    extra_css              = ' body{ font-family: Verdana,Helvetica,Arial,sans-serif } .introduction{font-weight: bold} .story-feature{display: block; padding: 0; border: 1px solid; width: 40%; font-size: small} .story-feature h2{text-align: center; text-transform: uppercase} '
+    preprocess_regexps     = [(re.compile(r'<!--.*?-->', re.DOTALL), lambda m: '')]
+    conversion_options = {
+                             'comments'        : description
+                            ,'tags'            : category
+                            ,'language'        : language
+                            ,'publisher'       : publisher
+                            ,'linearize_tables': True
+                         }
+
+    keep_only_tags  = [
+                       dict(name='div', attrs={'class':['ds','mxb']}),
+                       dict(attrs={'class':['story-body','storybody']})
+                      ]
+
+    remove_tags     = [
+                       dict(name='div', attrs={'class':['storyextra', 'share-help', 'embedded-hyper', \
+                       'story-feature wide ', 'story-feature narrow', 'cap', 'caption', 'q1', 'sihf', \
+                       'mva', 'videoInStoryC', 'sharesb', 'mvtb']}),
+                       dict(name=['img']), dict(name=['br'])
+                      ]
+
+    remove_attributes = ['width','height']
+
+    feeds          = [
+                      ('Sport Front Page', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/front_page/rss.xml'),
+                      ('Football', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/football/rss.xml'),
+                      ('Cricket', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/cricket/rss.xml'),
+                      ('Formula 1', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/motorsport/formula_one/rss.xml'),
+                      ('Commonwealth Games', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/commonwealth_games/delhi_2010/rss.xml'),
+                      ('Golf', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/golf/rss.xml'),
+                      ('Rugby Union', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/rugby_union/rss.xml'),
+                      ('Rugby League', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/rugby_league/rss.xml'),
+                      ('Tennis', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/tennis/rss.xml'),
+                      ('Motorsport', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/motorsport/rss.xml'),
+                      ('Boxing', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/boxing/rss.xml'),
+                      ('Athletics', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/athletics/rss.xml'),
+                      ('Snooker', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/other_sports/snooker/rss.xml'),
+                      ('Horse Racing', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/other_sports/horse_racing/rss.xml'),
+                      ('Cycling', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/other_sports/cycling/rss.xml'),
+                      ('Disability Sport', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/other_sports/disability_sport/rss.xml'),
+                      ('Other Sport', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/other_sports/rss.xml'),
+                      ('Olympics 2012', 'http://newsrss.bbc.co.uk/rss/sportonline_uk_edition/olympics/london_2012/rss.xml'),
+                     ]

resources/recipes/eluniverso_ec.recipe

@@ -0,0 +1,63 @@
+__license__   = 'GPL v3'
+__copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
+'''
+eluniverso.com
+'''
+
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class ElUniverso_Ecuador(BasicNewsRecipe):
+    title                 = 'El Universo - Ecuador'
+    __author__            = 'Darko Miletic'
+    description           = 'Noticias del Ecuador y el resto del mundo'
+    publisher             = 'El Universo'
+    category              = 'news, politics, Ecuador'
+    oldest_article        = 2
+    max_articles_per_feed = 200
+    no_stylesheets        = True
+    encoding              = 'utf8'
+    use_embedded_content  = False
+    language              = 'es'
+    remove_empty_feeds    = True
+    publication_type      = 'newspaper'
+    masthead_url          = 'http://servicios2.eluniverso.com/versiones/v1/img/Hd/lg_ElUniverso.gif'
+    extra_css             = """
+                                body{font-family: Verdana,Arial,Helvetica,sans-serif; color: #333333 }
+                                h2{font-family: Georgia,"Times New Roman",Times,serif; color: #1B2D60}
+                            """
+
+    conversion_options = {
+                          'comment'   : description
+                        , 'tags'      : category
+                        , 'publisher' : publisher
+                        , 'language'  : language
+                        }
+
+    remove_tags        = [
+                            dict(attrs={'class':['flechs','multiBox','colRecursos']})
+                           ,dict(name=['meta','link','embed','object','iframe','base'])
+                         ]
+    keep_only_tags     = [dict(attrs={'class':'Nota'})]
+    remove_tags_after  = dict(attrs={'id':'TextoPrint'})
+    remove_tags_before = dict(attrs={'id':'FechaPrint'})
+
+    feeds = [
+              (u'Portada'          , u'http://www.eluniverso.com/rss/portada.xml'           )
+             ,(u'Politica'         , u'http://www.eluniverso.com/rss/politica.xml'          )
+             ,(u'Economia'         , u'http://www.eluniverso.com/rss/economia.xml'          )
+             ,(u'Sucesos'          , u'http://www.eluniverso.com/rss/sucesos.xml'           )
+             ,(u'Migracion'        , u'http://www.eluniverso.com/rss/migrantes_tema.xml'    )
+             ,(u'El Pais'          , u'http://www.eluniverso.com/rss/elpais.xml'            )
+             ,(u'Internacionales'  , u'http://www.eluniverso.com/rss/internacionales.xml'   )
+             ,(u'Deportes'         , u'http://www.eluniverso.com/rss/deportes.xml'          )
+             ,(u'Gran Guayaquill'  , u'http://www.eluniverso.com/rss/gran_guayaquil.xml'    )
+             ,(u'Entretenimiento'  , u'http://www.eluniverso.com/rss/arteyespectaculos.xml' )
+             ,(u'Vida'             , u'http://www.eluniverso.com/rss/tuvida.xml'            )
+             ,(u'Opinion'          , u'http://www.eluniverso.com/rss/opinion.xml'           )
+            ]
+
+    def preprocess_html(self, soup):
+        for item in soup.findAll(style=True):
+            del item['style']
+        return soup
+

resources/recipes/revista_muy.recipe

@@ -0,0 +1,78 @@
+from calibre.web.feeds.recipes import BasicNewsRecipe
+from BeautifulSoup import Tag
+
+class RevistaMuyInteresante(BasicNewsRecipe):
+
+    title       = 'Revista Muy Interesante'
+    __author__  = 'Jefferson Frantz'
+    description = 'Revista de divulgacion'
+    timefmt = ' [%d %b, %Y]'
+    language = 'es'
+
+    no_stylesheets = True
+    remove_attributes = ['style', 'font']
+
+    #then we add our own style(s) like this:
+    extra_css = '''
+                       .contentheading{font-weight: bold}
+                       p {font-size: 4px;font-family: Times New Roman;}
+                    '''
+
+    def preprocess_html(self, soup):
+            for img_tag in soup.findAll('img'):
+                parent_tag = img_tag.parent
+                if parent_tag.name == 'td':
+                    if not parent_tag.get('class') == 'txt_articulo': break
+                    imagen = img_tag
+                    new_tag = Tag(soup,'p')
+                    img_tag.replaceWith(new_tag)
+                    div = soup.find(attrs={'class':'article_category'})
+                    div.insert(0,imagen)
+            return soup
+
+    keep_only_tags = [dict(name='div', attrs={'class':['article']}),dict(name='td', attrs={'class':['txt_articulo']})]
+
+    remove_tags        = [
+                             dict(name=['object','link','script','ul'])
+                            ,dict(name='div', attrs={'id':['comment']})
+                            ,dict(name='td', attrs={'class':['buttonheading']})
+                            ,dict(name='div', attrs={'class':['tags_articles']})
+                         ]
+
+    remove_tags_after = dict(name='div', attrs={'class':'tags_articles'})
+
+
+    #TO GET ARTICLES IN SECTION
+    def nz_parse_section(self, url):
+            soup = self.index_to_soup(url)
+            div = soup.find(attrs={'class':'contenido'})
+            current_articles = []
+            for x in div.findAllNext(attrs={'class':['headline']}):
+                    a = x.find('a', href=True)
+                    if a is None:
+                        continue
+                    title = self.tag_to_string(a)
+                    url = a.get('href', False)
+                    if not url or not title:
+                        continue
+                    if url.startswith('/'):
+                         url = 'http://www.muyinteresante.es'+url
+#                    self.log('\t\tFound article:', title)
+#                    self.log('\t\t\t', url)
+                    current_articles.append({'title': title, 'url':url,
+                        'description':'', 'date':''})
+
+            return current_articles
+
+
+    # To GET SECTIONS
+    def parse_index(self):
+            feeds = []
+            for title, url in [
+                ('Historia',
+                 'http://www.muyinteresante.es/historia-articulos'),
+             ]:
+               articles = self.nz_parse_section(url)
+               if articles:
+                   feeds.append((title, articles))
+            return feeds

src/calibre/constants.py

@@ -2,7 +2,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = 'calibre'
-__version__   = '0.7.20'
+__version__   = '0.7.21'
 __author__    = "Kovid Goyal <kovid@kovidgoyal.net>"
 
 import re

src/calibre/devices/hanvon/driver.py

@@ -7,7 +7,7 @@ __docformat__ = 'restructuredtext en'
 '''
 Device driver for Hanvon devices
 '''
-import re
+import re, os
 
 from calibre.devices.usbms.driver import USBMS
 
@@ -67,10 +67,51 @@ class ALEX(N516):
 
     EBOOK_DIR_MAIN = 'eBooks'
     SUPPORTS_SUB_DIRS = False
+    THUMBNAIL_HEIGHT = 120
 
     def can_handle(self, device_info, debug=False):
         return is_alex(device_info)
 
+    def alex_cpath(self, file_abspath):
+        base = os.path.dirname(file_abspath)
+        name = os.path.splitext(os.path.basename(file_abspath))[0] + '.png'
+        return os.path.join(base, 'covers', name)
+
+    def upload_cover(self, path, filename, metadata):
+        from calibre.ebooks import calibre_cover
+        from calibre.utils.magick.draw import thumbnail
+        coverdata = getattr(metadata, 'thumbnail', None)
+        if coverdata and coverdata[2]:
+            cover = coverdata[2]
+        else:
+            cover = calibre_cover(metadata.get('title', _('Unknown')),
+                    metadata.get('authors', _('Unknown')))
+
+        cover = thumbnail(cover, width=self.THUMBNAIL_HEIGHT,
+                height=self.THUMBNAIL_HEIGHT, fmt='png')[-1]
+
+        cpath = self.alex_cpath(os.path.join(path, filename))
+        cdir = os.path.dirname(cpath)
+        if not os.path.exists(cdir):
+            os.makedirs(cdir)
+        with open(cpath, 'wb') as coverfile:
+            coverfile.write(cover)
+
+    def delete_books(self, paths, end_session=True):
+        for i, path in enumerate(paths):
+            self.report_progress((i+1) / float(len(paths)), _('Removing books from device...'))
+            path = self.normalize_path(path)
+            if os.path.exists(path):
+                # Delete the ebook
+                os.unlink(path)
+            try:
+                cpath = self.alex_cpath(path)
+                if os.path.exists(cpath):
+                    os.remove(cpath)
+            except:
+                pass
+        self.report_progress(1.0, _('Removing books from device...'))
+
 class AZBOOKA(ALEX):
 
     name = 'Azbooka driver'
@@ -83,10 +124,13 @@ class AZBOOKA(ALEX):
     MAIN_MEMORY_VOLUME_LABEL  = 'Azbooka Internal Memory'
 
     EBOOK_DIR_MAIN = ''
+    SUPPORTS_SUB_DIRS = True
 
     def can_handle(self, device_info, debug=False):
         return not is_alex(device_info)
 
+    def upload_cover(self, path, filename, metadata):
+        pass
 
 class EB511(USBMS):
     name           = 'Elonex EB 511 driver'

src/calibre/manual/metadata.rst

@@ -4,3 +4,59 @@
 
 Editing E-book Metadata
 ========================
+
+.. contents:: Contents
+  :depth: 2
+  :local:
+
+E-books come in all shapes and sizes and more often than not, their metadata (things like title/author/series/publisher) is incomplete or incorrect. 
+The simplest way to change metadata in |app| is to simply double click on an entry and type in the correct replacement.
+For more sophisticated, "power editing" use the edit metadata tools discussed below.
+
+Editing the metadata of one book at a time
+-------------------------------------------
+
+Click the book you want to edit and then click the :guilabel:`Edit metadata` button or press the ``E`` key. A dialog opens that allows you to edit all aspects of the metadata. It has various features to make editing faster and more efficient. A list of the commonly used tips:
+
+    * You can click the button in between title and authors to swap them automatically. Or 
+    * You can click the button next to author sort to automatically to have |app| automatically fill it from the author name.
+    * You can click the button next to tags to use the Tag Editor to manage the tags associated with the book.
+    * The ISBN box will have a red background if you enter an invalid ISBN. It will be green for valid ISBNs
+    * The author sort box will be red if the author sort value differs from what |app| thinks it should be.
+
+Downloading metadata
+^^^^^^^^^^^^^^^^^^^^^
+
+The nicest feature of the edit metadata dialog is its ability to automatically fill in many metadata fields by getting metadata from various websites. Currently, |app| uses isbndb.com, Google Books, Amazon and Library Thing. The metadata download can fill in Title, author, series, tags, rating, description and ISBN for you. 
+
+To use the download, fill in the title and author fields and click the :guilabel:`Fetch metadata` button. |app| will present you with a list of books that most closely match the title and author. If you fill in the ISBN field first, it will be used in preference to the title and author. If no matches are found, try making your search a little less specific by including only some key words in the title and only the author last name. 
+
+Managing book formats
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+In |app|, a single book entry can have many different *formats* associated with it. For example you may have obtained the Complete Works of Shakespeare in EPUB format and later converted it to MOBI to read on your Kindle. |app| automatically manages multiple formats for you. In the :guilabel:`Available formats` section of the Edit metadata dialog, you can manage these formats. You can add a new format, delete an existing format and also ask |app| to set the metadata and cover for the book entry from the metadata in one of the formats.
+
+All about covers
+^^^^^^^^^^^^^^^^^^^^^
+
+You can ask |app| to download book covers for you, provided the book has a known ISBN. Alternatively you can specify a file on your computer to use as the cover. |app| can even generate a default cover with basic metadata on it for you. You can drag and drop images onto the cover to change it and also right click to copy/paste cover images.
+
+In addition, there is a button to automatically trim borders from the cover, in case your cover image has an ugly border.
+
+
+Editing the metadata of many books at a time
+---------------------------------------------
+
+First select the books you want to edit by holding Ctrl or Shift and clicking on them. If you select more than one book, clicking the :guilabel:`Edit metadata` button will cause a new *Bulk* metadata edit dialog to open. Using this dialog, you can quickly set the author/publisher/rating/tags/series etc of a bunch of books to the same value. This is particularly useful if you have just imported a number of books that have some metadata in common. You can also click the arrow next to the :guilabel:`Edit metadata` button and select :guilabel:`Edit metadata individually` to use the powerful single book edit dialog from above for all the selected books in succession.
+
+Search and replace
+^^^^^^^^^^^^^^^^^^^^
+
+The Bulk metadata edit dialog allows you to perform arbitrarily powerful search and replace operations on the selected books. By default it uses a simple text search and replace, but it also support *regular expressions*. For more on regular expressions, see :ref:`regexptutorial`.
+
+Bulk downloading of metadata
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you want to download the metadata for multiple books at once, click the arrow next to the :guilabel:`Edit metadata` button and select :guilabel:`Download metadata and covers`. You can choose to download only metadata, only covers, both or only social metadata (tags/rating/series). 
+
+