Sync to trunk.

Changelog.yaml

@@ -4,6 +4,65 @@
 # for important features/bug fixes.
 # Also, each release can have new and improved recipes.
 
+- version: 0.6.52
+  date: 2010-07-30
+
+  new features:
+    - title: "Support for the Kobo Reader and the HTC Desire"
+
+    - title: "PDB Input: Add support for PDB files created with incorrect encodings"
+
+    - title: "EPUB Output: Make the file size splitting algorithm more intelligent. If a split results in a tree that is very small, choose another split point."
+
+    - title: "Add an entry to the Fetch News menu to fetch all scheduled news"
+      tickets: [5436]
+
+    - title: "MOBI Output: When inserting metadata, hide the searchable version of the tags (append with ttt). They are still searchable, but not visible."
+
+  bug fixes:
+    - title: "HTML Input: Handle case sensitive file systems on OS X"
+      tickets: [5083]
+
+    - title: "EPUB Output: When rescaling PNG images, write out the rescaled data in PNG format as ADE cannot handle JPEG data in a PNG file"
+
+    - title: "SONY drivers: Fix corner case that could cause errors when transferring books with series to the device."
+      tickets: [5431]
+
+    - title: "Allow users to set the content server port to < 1025 (system ports) with a warning."
+      tickets: [5470]
+
+    - title: "Don't popup an error message when adding books if the cover is not a valid image"
+
+    - title: "SONY drivers: Fix bug that prevented detection SD cards that had the Sony Reader/database folders, but no cache.xml"
+
+    - title: "EPUB Input: Handle invalid EPUB files that have manifest entries pointing to non existent files"
+      tickets: [5444]
+
+    - title: "Fix the get cover/metadata from specific format buttons when the user has chosen to read metadata only from filenames"
+      tickets: [5445]
+
+    - title: "Conversion pipeline: More robust conversion of HTML tags to text when detecting structure"
+
+    - title: "Fix 'Open containing folder' does not work if the path name contains '#'"
+      tickets: [5424]
+
+
+  new recipes:
+    - title: The Old New Thing, Berlingske, ABC, Ultima Hora, China Daily, Dani
+      author: Darko Miletic
+
+    - title: Arbetaren, Ekot and Fria Tidningen
+      author: Joakim Lindskog
+
+    - title: Il Messaggero, Il Giornale and ADN Kronos
+      author: Gabriele Marini
+
+    - title: Onion AV Club
+      author: Stephen Williams
+
+  improved recipes:
+    - Jerusalem Post
+
 - version: 0.6.51
   date: 2010-04-30
 

resources/recipes/arbetaren.recipe

@@ -0,0 +1,34 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class Arbetaren_SE(BasicNewsRecipe):
+    title          = u'Arbetaren'
+    __author__            = 'Joakim Lindskog'
+    description           = 'Nyheter fr\xc3\xa5n Arbetaren'
+    publisher             = 'Arbetaren'
+    category              = 'news, politics, socialism, Sweden'
+    oldest_article        = 7
+    delay                 = 1
+    max_articles_per_feed = 100
+    no_stylesheets        = True
+    use_embedded_content  = False
+    encoding              = 'utf-8'
+    language              = 'sv'
+
+    conversion_options = {
+                          'comment'   : description
+                        , 'tags'      : category
+                        , 'publisher' : publisher
+                        , 'language'  : language
+                        }
+
+    keep_only_tags = [dict(name='div', attrs={'id':'article'})]
+    remove_tags_before = dict(name='div', attrs={'id':'article'})
+    remove_tags_after = dict(name='p',attrs={'id':'byline'})
+    remove_tags = [
+                     dict(name=['object','link','base']),
+                     dict(name='p', attrs={'class':'print'}),
+                     dict(name='a', attrs={'class':'addthis_button_compact'}),
+                     dict(name='script')
+                  ]
+
+    feeds          = [(u'Nyheter', u'http://www.arbetaren.se/rss/arbetaren.rss?rev=123')]

resources/recipes/ekot.recipe

@@ -0,0 +1,39 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class Ekot_SE(BasicNewsRecipe):
+    title                 = 'Ekot'
+    __author__            = 'Joakim Lindskog'
+    description           = 'Nyheter fr\xc3\xa5n Ekot'
+    publisher             = 'Ekot'
+    category              = 'news, politics, Sweden'
+    oldest_article        = 7
+    delay                 = 1
+    max_articles_per_feed = 100
+    no_stylesheets        = True
+    use_embedded_content  = False
+    encoding              = 'utf-8'
+    language              = 'sv'
+
+    conversion_options = {
+                          'comment'   : description
+                        , 'tags'      : category
+                        , 'publisher' : publisher
+                        , 'language'  : language
+                        }
+
+    keep_only_tags = [dict(name='h1', attrs={'class':'newsH2'}),
+                               dict(name='div', attrs={'class':'articleTop'}),
+                               dict(name='div', attrs={'class':'newsIntro'}),
+                               dict(name='div', attrs={'class':'newsText'})]
+    remove_tags = [
+                     dict(name=['object','link','base'])
+                    ,dict(name='span',attrs={'class':'relLink'})
+                  ]
+
+    feeds          = [(u'Ekot', u'http://api.sr.se/api/rssfeed/rssfeed.aspx?rssfeed=83'),
+                          (u'Utrikes', u'http://api.sr.se/api/rssfeed/rssfeed.aspx?rssfeed=3304'),
+                          (u'Radiosporten', u'http://api.sr.se/api/rssfeed/rssfeed.aspx?rssfeed=179')]
+
+    def print_version(self, url):
+        return url.replace('http://sverigesradio.se/cgi-bin/ekot/artikel.asp',
+                'http://sverigesradio.se/cgi-bin/isidorpub/PrinterFriendlyArticle.asp')+'&ProgramID=83'

resources/recipes/fria.recipe

@@ -0,0 +1,66 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class FriaTidningen_SE(BasicNewsRecipe):
+    title          = u'Fria Tidningen'
+    __author__            = 'Joakim Lindskog'
+    description           = 'Nyheter fr\xc3\xa5n Fria Tidningen'
+    publisher             = 'Fria Tidningen'
+    category              = 'news, politics, Sweden'
+    oldest_article        = 7
+    delay                 = 1
+    max_articles_per_feed = 100
+    no_stylesheets        = True
+    use_embedded_content  = False
+    encoding              = 'utf-8'
+    language              = 'sv'
+
+    conversion_options = {
+                          'comment'   : description
+                        , 'tags'      : category
+                        , 'publisher' : publisher
+                        , 'language'  : language
+                        }
+
+    keep_only_tags = [dict(name='div', attrs={'id':'content-area'})]
+    remove_tags_before = dict(name='div', attrs={'id':'content-area'})
+    remove_tags_after = dict(name='div',attrs={'id':'byline'})
+    remove_tags = [
+                     dict(name=['object','link','base']),
+                     dict(name='div', attrs={'id':'comments'}),
+                     dict(name='div', attrs={'id':'block-block-21'}),
+                     dict(name='div', attrs={'id':'block-block-22'}),
+                     dict(name='div', attrs={'id':'block-block-23'}),
+                     dict(name='div', attrs={'id':'block-block-24'}),
+                     dict(name='div', attrs={'id':'block-block-25'}),
+                     dict(name='div', attrs={'id':'block-block-26'}),
+                     dict(name='div', attrs={'id':'block-block-27'}),
+                     dict(name='div', attrs={'id':'block-block-28'}),
+                     dict(name='div', attrs={'id':'block-block-29'}),
+                     dict(name='div', attrs={'id':'block-block-30'}),
+                     dict(name='div', attrs={'id':'block-block-40'})
+                  ]
+
+    feeds          = [(u'Allt', u'http://www.fria.nu/feed'),
+                          (u'Nyheter', u'http://www.fria.nu/taxonomy/term/13/feed/feed'),
+                          (u'Inrikes', u'http://www.fria.nu/taxonomy/term/14/0/feed'),
+                          (u'Utrikes', u'http://www.fria.nu/taxonomy/term/15/0/feed'),
+                          (u'Ekonomi', u'http://www.fria.nu/taxonomy/term/27047/0/feed'),
+                          (u'Opinion', u'http://www.fria.nu/taxonomy/term/22/0/feed'),
+                          (u'Inledaren', u'http://www.fria.nu/taxonomy/term/24/0/feed'),
+                          (u'Argument', u'http://www.fria.nu/taxonomy/term/23/0/feed'),
+                          (u'Synpunkten', u'http://www.fria.nu/taxonomy/term/26/0/feed'),
+                          (u'Debatt', u'http://www.fria.nu/taxonomy/term/25/0/feed'),
+                          (u'Kultur', u'http://www.fria.nu/taxonomy/term/19/0/feed'),
+                          (u'Kulturnyheter', u'http://www.fria.nu/taxonomy/term/24534/0/feed'),
+                          (u'Recensioner', u'http://www.fria.nu/taxonomy/term/24535/0/feed'),
+                          (u'BAK', u'http://www.fria.nu/taxonomy/term/27/0/feed'),
+                          (u'Sport & H\xc3\xa4lsa' u'http://www.fria.nu/taxonomy/term/27215/0/feed'),
+                          (u'Sport', u'http://www.fria.nu/taxonomy/term/20/0/feed'),
+                          (u'H\xc3\xa4lsa', u'http://www.fria.nu/taxonomy/term/21/0/feed'),
+                          (u'F\xc3\xb6rdjupning', u'http://www.fria.nu/taxonomy/term/24994/0/feed'),
+                          (u'Fokus', u'http://www.fria.nu/taxonomy/term/24864/0/feed'),
+                          (u'Samtal', u'http://www.fria.nu/taxonomy/term/28/0/feed'),
+                          (u'Stockholm', u'http://www.fria.nu/taxonomy/term/122/0/feed'),
+                          (u'G\xc3\xb6teborg', u'http://www.fria.nu/taxonomy/term/73/0/feed'),
+                          (u'Uppsala', u'http://www.fria.nu/taxonomy/term/27324/0/feed'),
+                          (u'Malm\xc3\xb6', u'http://www.fria.nu/taxonomy/term/28031/0/feed')]

src/calibre/constants.py

@@ -2,7 +2,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = 'calibre'
-__version__   = '0.6.51'
+__version__   = '0.6.52'
 __author__    = "Kovid Goyal <kovid@kovidgoyal.net>"
 
 import re

src/calibre/customize/profiles.py

@@ -261,6 +261,18 @@ class SonyReaderOutput(OutputProfile):
     fbase                     = 12
     fsizes                    = [7.5, 9, 10, 12, 15.5, 20, 22, 24]
 
+class KoboReaderOutput(OutputProfile):
+
+    name = 'Kobo Reader'
+    short_name = 'kobo'
+
+    description = _('This profile is intended for the Kobo Reader.')
+
+    screen_size               = (590, 775)
+    dpi                       = 168.451
+    fbase                     = 12
+    fsizes                    = [7.5, 9, 10, 12, 15.5, 20, 22, 24]
+
 class SonyReader300Output(SonyReaderOutput):
 
     author      = 'John Schember'
@@ -461,7 +473,7 @@ class NookOutput(OutputProfile):
 output_profiles = [OutputProfile, SonyReaderOutput, SonyReader300Output,
         SonyReader900Output, MSReaderOutput, MobipocketOutput, HanlinV3Output,
         HanlinV5Output, CybookG3Output, CybookOpusOutput, KindleOutput,
-        iPadOutput,
+        iPadOutput, KoboReaderOutput,
         SonyReaderLandscapeOutput, KindleDXOutput, IlliadOutput,
         IRexDR1000Output, IRexDR800Output, JetBook5Output, NookOutput,]
 

src/calibre/devices/prs505/books.py

@@ -384,7 +384,7 @@ class BookList(_BookList):
             if not pl:
                 continue
             db_ids = [i.getAttribute('id') for i in pl.childNodes if hasattr(i, 'getAttribute')]
-            pl_book_ids = [self.book_by_id(i.getAttribute('id')).db_id for i in pl.childNodes  if hasattr(i, 'getAttribute')]
+            pl_book_ids = [getattr(self.book_by_id(i), 'db_id', None) for i in db_ids]
             map = {}
             for i, j in zip(pl_book_ids, db_ids):
                 map[i] = j

src/calibre/ebooks/epub/output.py

@@ -171,6 +171,10 @@ class EPUBOutput(OutputFormatPlugin):
 
         self.workaround_sony_quirks()
 
+        if self.oeb.toc.count() == 0:
+            self.log.warn('This EPUB file has no Table of Contents. It will '
+                    'not validate via epubcheck')
+
         from calibre.ebooks.oeb.base import OPF
         identifiers = oeb.metadata['identifier']
         uuid = None

src/calibre/ebooks/oeb/transforms/rescale.py

@@ -37,6 +37,11 @@ class RescaleImages(object):
             page_height -= (self.opts.margin_top + self.opts.margin_bottom) * self.opts.dest.dpi/72.
         for item in self.oeb.manifest:
             if item.media_type.startswith('image'):
+                ext = item.media_type.split('/')[-1].upper()
+                if ext == 'JPG': ext = 'JPEG'
+                if ext not in ('PNG', 'JPEG'):
+                    ext = 'JPEG'
+
                 raw = item.data
                 if not raw: continue
                 if qt:
@@ -65,11 +70,11 @@ class RescaleImages(object):
                     if qt:
                         img = img.scaled(new_width, new_height,
                                 Qt.IgnoreAspectRatio, Qt.SmoothTransformation)
-                        data = pixmap_to_data(img)
+                        data = pixmap_to_data(img, format=ext)
                     else:
                         im = im.resize((int(new_width), int(new_height)), PILImage.ANTIALIAS)
                         of = cStringIO.StringIO()
-                        im.convert('RGB').save(of, 'JPEG')
+                        im.convert('RGB').save(of, ext)
                         data = of.getvalue()
                     if data is not None:
                         item.data = data

src/calibre/gui2/dialogs/config/__init__.py

@@ -361,7 +361,10 @@ class ConfigDialog(ResizableDialog, Ui_Dialog):
 
         column_map = config['column_map']
         for col in column_map + [i for i in ALL_COLUMNS if i not in column_map]:
-            item = QListWidgetItem(BooksModel.headers[col], self.columns)
+            try:
+                item = QListWidgetItem(BooksModel.headers[col], self.columns)
+            except KeyError:
+                continue
             item.setData(Qt.UserRole, QVariant(col))
             item.setFlags(Qt.ItemIsEnabled|Qt.ItemIsUserCheckable|Qt.ItemIsSelectable)
             item.setCheckState(Qt.Checked if col in column_map else Qt.Unchecked)
@@ -461,6 +464,17 @@ class ConfigDialog(ResizableDialog, Ui_Dialog):
         self.opt_overwrite_author_title_metadata.setChecked(config['overwrite_author_title_metadata'])
         self.opt_enforce_cpu_limit.setChecked(config['enforce_cpu_limit'])
         self.device_detection_button.clicked.connect(self.debug_device_detection)
+        self.port.editingFinished.connect(self.check_port_value)
+
+    def check_port_value(self, *args):
+        port = self.port.value()
+        if port < 1025:
+            warning_dialog(self, _('System port selected'), '<p>'+
+                    _('The value <b>%d</b> you have chosen for the content '
+                        'server port is a system port. You operating '
+                        'system <b>may</b> not allow the server to run on this '
+                        'port. To be safe choose a port number larger than '
+                        '1024.')%port, show=True)
 
     def debug_device_detection(self):
         from calibre.gui2.dialogs.config.device_debug import DebugDevice

src/calibre/gui2/dialogs/config/config.ui

@@ -719,9 +719,6 @@
                </item>
                <item row="0" column="1">
                 <widget class="QSpinBox" name="port">
-                 <property name="minimum">
-                  <number>1025</number>
-                 </property>
                  <property name="maximum">
                   <number>65535</number>
                  </property>

src/calibre/gui2/library.py

@@ -186,6 +186,7 @@ class BooksModel(QAbstractTableModel):
         QAbstractTableModel.__init__(self, parent)
         self.db = None
         self.column_map = config['column_map']
+        self.column_map = [x for x in self.column_map if x in self.headers]
         self.editable_cols = ['title', 'authors', 'rating', 'publisher',
                               'tags', 'series', 'timestamp', 'pubdate']
         self.default_image = QImage(I('book.svg'))
@@ -202,6 +203,7 @@ class BooksModel(QAbstractTableModel):
     def read_config(self):
         self.use_roman_numbers = config['use_roman_numerals_for_series_number']
         cols = config['column_map']
+        cols = [x for x in cols if x in self.headers]
         if cols != self.column_map:
             self.column_map = cols
             self.reset()

src/calibre/gui2/ui.py

@@ -355,7 +355,7 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
         self.view_menu = QMenu()
         self.view_menu.addAction(_('View'))
         ac = self.view_menu.addAction(_('View specific format'))
-        ac.setShortcut(Qt.AltModifier+Qt.Key_V)
+        ac.setShortcut((Qt.ControlModifier if isosx else Qt.AltModifier)+Qt.Key_V)
         self.action_view.setMenu(self.view_menu)
 
         self.delete_menu = QMenu()

src/calibre/gui2/wizard/__init__.py

@@ -92,6 +92,14 @@ class Sony505(Sony500):
     name = 'SONY Reader 6" and Touch Edition'
     id = 'prs505'
 
+class Kobo(Device):
+    name = 'Kobo Reader'
+    manufacturer = 'Kobo'
+    output_profile = 'kobo'
+    output_format = 'EPUB'
+    name = 'Kobo Reader'
+    id = 'kobo'
+
 class Sony300(Sony505):
 
     name = 'SONY Reader Pocket Edition'
@@ -125,6 +133,20 @@ class CybookOpus(CybookG3):
     output_profile = 'cybook_opus'
     id = 'cybook_opus'
 
+class PocketBook360(CybookOpus):
+
+    manufacturer = 'PocketBook'
+    name = 'PocketBook 360'
+    id = 'pocketbook360'
+    output_profile = 'cybook_opus'
+
+class PocketBook(CybookG3):
+
+    manufacturer = 'PocketBook'
+    name = 'PocketBook 301/302'
+    id = 'pocketbook'
+    output_profile = 'cybookg3'
+
 class iPhone(Device):
 
     name = 'iPad or iPhone/iTouch + Stanza'

src/calibre/library/database2.py

@@ -1050,7 +1050,10 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
             mi.pubdate = utcnow()
         self.set_metadata(id, mi)
         if cover is not None:
-            self.set_cover(id, cover)
+            try:
+                self.set_cover(id, cover)
+            except:
+                traceback.print_exc()
         return id
 
 

src/calibre/manual/conversion.rst

@@ -513,3 +513,23 @@ removed from the text it can throw off the paragraph unwrapping.
 Some limitations of PDF input is complex, multi-column, and image based documents are not supported.
 Extraction of vector images and tables from within the document is also not supported.
 
+Comic Book Collections
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A comic book collection is a .cbc file. A .cbc file is a zip file that contains other CBZ/CBR files. In addition the
+.cbc file must contain a simple text file called comics.txt, encoded in UTF-8. The comics.txt file must contain
+a list of the comics files inside the .cbc file, in the form filename:title, as shown below::
+
+    one.cbz:Chapter One
+    two.cbz:Chapter Two
+    three.cbz:Chapter Three
+
+The .cbc file will then contain::
+
+    comics.txt
+    one.cbz
+    two.cbz
+    three.cbz
+
+|app| will automatically convert this .cbc file into a e-book with a Table of Contents pointing to each entry in comics.txt.
+

src/calibre/manual/faq.rst

@@ -81,7 +81,7 @@ Device Integration
 
 What devices does |app| support?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-At the moment |app| has full support for the SONY PRS 300/500/505/600/700/900, Barnes & Noble Nook, Cybook Gen 3/Opus, Amazon Kindle 1/2/DX, Entourage Edge, Longshine ShineBook, Ectaco Jetbook, BeBook/BeBook Mini, Irex Illiad/DR1000, Foxit eSlick, PocketBook 360, Italica, eClicto, Iriver Story, Airis dBook, Hanvon N515, Binatone Readme, Teclast K3, SpringDesign Alex, various Android phones and the iPhone. In addition, using the :guilabel:`Save to disk` function you can use it with any ebook reader that exports itself as a USB disk.
+At the moment |app| has full support for the SONY PRS 300/500/505/600/700/900, Barnes & Noble Nook, Cybook Gen 3/Opus, Amazon Kindle 1/2/DX, Entourage Edge, Longshine ShineBook, Ectaco Jetbook, BeBook/BeBook Mini, Irex Illiad/DR1000, Foxit eSlick, PocketBook 360, Italica, eClicto, Iriver Story, Airis dBook, Hanvon N515, Binatone Readme, Teclast K3, SpringDesign Alex, Kobo Reader, various Android phones and the iPhone/iPad. In addition, using the :guilabel:`Save to disk` function you can use it with any ebook reader that exports itself as a USB disk.
 
 How can I help get my device supported in |app|?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

src/calibre/utils/pdftk.py

@@ -54,7 +54,7 @@ def set_metadata(stream, mi):
 
         while True:
             try:
-                ret = p.wait()
+                p.wait()
                 break
             except OSError, e:
                 if e.errno == errno.EINTR: