Merge from trunk

Changelog.yaml

@@ -19,6 +19,50 @@
 #   new recipes:
 #     - title: 
 
+- version: 0.8.62
+  date: 2012-07-27
+
+  new features:
+    - title: "Book details panel: Allow right clicking on a format to delete it."
+
+    - title: "When errors occur in lots of background jobs, add an option to the error message to temporarily suppress subsequent error messages."
+      tickets: [886904]
+
+    - title: "E-book viewer full screen mode: Allow clicking in the left and right page margins to turn pages."
+      tickets: [1024819]
+
+    - title: "Drivers for various Android devices"
+      tickets: [1028690,1027431]
+
+    - title: "Advanced search dialog: When starting on the title/author/etc. tab, restore the previously used search kind as well."
+      tickets: [1029745]
+
+    - title: "When presenting the calibre must be restarted warning after installing a new plugin, add a restart now button so that the user can conveniently restart calibre. Currently only works when going vie Preferences->Plugins->Get new plugins"
+ 
+  bug fixes:
+    - title: "Fix main window layout state being saved incorrectly if calibre is killed without a proper shutdown"
+
+    - title: "Fix boolean and date searching in non english calibre installs."
+
+    - title: "Conversion: Ignore invalid chapter detection and level n ToC expressions instead of erroring out"
+
+  improved recipes:
+    - Psychology Today
+    - The Smithsonian
+    - The New Republic
+    - Various updated Polish news sources
+    - The Sun
+    - San Francisco Bay Guardian
+    - AnandTech
+    - Smashing Magazine
+
+  new recipes:
+    - title: Linux Journal and Conowego.pl 
+      author: fenuks
+
+    - title: A list apart and .net magazine 
+      author: Marc Busque
+
 - version: 0.8.61
   date: 2012-07-20
 

manual/conversion.rst

@@ -710,3 +710,31 @@ EPUB from the ZIP file are::
 
 Note that because this file explores the potential of EPUB, most of the advanced formatting is not going to work on readers less capable than |app|'s built-in EPUB viewer. 
 
+
+Convert ODT documents
+~~~~~~~~~~~~~~~~~~~~~
+
+|app| can directly convert ODT (OpenDocument Text) files. You should use styles to format your document and minimize the use of direct formatting.
+When inserting images into your document you need to anchor them to the paragraph, images anchored to a page will all end up in the front of the conversion.
+
+To enable automatic detection of chapters, you need to mark them with the build-in styles called 'Heading 1', 'Heading 2', ..., 'Heading 6' ('Heading 1' equates to the HTML tag <h1>, 'Heading 2' to <h2> etc). When you convert in |app| you can enter which style you used into the 'Detect chapters at' box. Example:
+
+  * If you mark Chapters with style 'Heading 2', you have to set the 'Detect chapters at' box to ``//h:h2``
+  * For a nested TOC with Sections marked with 'Heading 2' and the Chapters marked with 'Heading 3' you need to enter ``//h:h2|//h:h3``. On the Convert - TOC page set the 'Level 1 TOC' box to ``//h:h2`` and the 'Level 2 TOC' box to ``//h:h3``.
+
+Well-known document properties (Title, Keywords, Description, Creator) are recognized and |app| will use the first image (not to small, and with good aspect-ratio) as the cover image.
+
+There is also an advanced property conversion mode, which is activated by setting the custom property ``opf.metadata`` ('Yes or No' type) to Yes in your ODT document (File->Properties->Custom Properties).
+If this property is detected by |app|, the following custom properties are recognized (``opf.authors`` overrides document creator)::
+
+    opf.titlesort
+    opf.authors
+    opf.authorsort
+    opf.publisher
+    opf.pubdate
+    opf.isbn
+    opf.language
+
+In addition to this, you can specify the picture to use as the cover by naming it ``opf.cover`` (right click, Picture->Options->Name) in the ODT. If no picture with this name is found, the 'smart' method is used.
+To prevent this you can set the custom property ``opf.nocover`` ('Yes or No' type) to Yes.
+

manual/develop.rst

@@ -149,23 +149,25 @@ the previously checked out |app| code directory, for example::
 
 calibre is the directory that contains the src and resources sub-directories. Ensure you have installed the |app| commandline tools via :guilabel:`Preferences->Advanced->Miscellaneous` in the |app| GUI.
 
-The next step is to set the environment variable ``CALIBRE_DEVELOP_FROM`` to the absolute path of the src directory.
+The next step is to create a bash script that will set the environment variable ``CALIBRE_DEVELOP_FROM`` to the absolute path of the src directory when running calibre in debug mode.
-So, following the example above, it would be ``/Users/kovid/work/calibre/src``. 
-`How to set environment variables <http://www.dowdandassociates.com/content/howto-set-environment-variable-mac-os-x-etclaunchdconf>`_.
 
-Once you have set the environment variable, open a new Terminal and check that it was correctly set by using
+Create a plain text file::
-the command::
 
-    echo $CALIBRE_DEVELOP_FROM
+    #!/bin/sh
+    export CALIBRE_DEVELOP_FROM="/Users/kovid/work/calibre/src"
+    calibre-debug -g
 
-Setting this environment variable means that |app| will now load all its Python code from the specified location.
+Save this file as ``/usr/bin/calibre-develop``, then set its permissions so that it can be executed::
 
-That's it! You are now ready to start hacking on the |app| code. For example, open the file :file:`src/calibre/__init__.py`
+    chmod +x /usr/bin/calibre-develop
-in your favorite editor and add the line::
 
-    print ("Hello, world!")
+Once you have done this, run::
 
-near the top of the file. Now run the command :command:`calibredb`. The very first line of output should be ``Hello, world!``.
+    calibre-develop
+
+You should see some diagnostic information in the Terminal window as calibre
+starts up, and you should see an asterisk after the version number in the GUI
+window, indicating that you are running from source.
 
 Linux development environment
 ------------------------------

recipes/benchmark_pl.recipe

@@ -1,6 +1,6 @@
 from calibre.web.feeds.news import BasicNewsRecipe
 import re
-class Benchmark_pl(BasicNewsRecipe):
+class BenchmarkPl(BasicNewsRecipe):
     title          = u'Benchmark.pl'
     __author__        = 'fenuks'
     description   = u'benchmark.pl -IT site'
@@ -14,7 +14,7 @@ class Benchmark_pl(BasicNewsRecipe):
     preprocess_regexps = [(re.compile(ur'<h3><span style="font-size: small;">&nbsp;Zobacz poprzednie <a href="http://www.benchmark.pl/news/zestawienie/grupa_id/135">Opinie dnia:</a></span>.*</body>', re.DOTALL|re.IGNORECASE), lambda match: '</body>'), (re.compile(ur'Więcej o .*?</ul>', re.DOTALL|re.IGNORECASE), lambda match: '')]
     keep_only_tags=[dict(name='div', attrs={'class':['m_zwykly', 'gallery']})]
     remove_tags_after=dict(name='div', attrs={'class':'body'})
-    remove_tags=[dict(name='div', attrs={'class':['kategoria', 'socialize', 'thumb', 'panelOcenaObserwowane', 'categoryNextToSocializeGallery']}), dict(name='table', attrs={'background':'http://www.benchmark.pl/uploads/backend_img/a/fotki_newsy/opinie_dnia/bg.png'}), dict(name='table', attrs={'width':'210', 'cellspacing':'1', 'cellpadding':'4', 'border':'0', 'align':'right'})]
+    remove_tags=[dict(name='div', attrs={'class':['kategoria', 'socialize', 'thumb', 'panelOcenaObserwowane', 'categoryNextToSocializeGallery', 'breadcrumb']}), dict(name='table', attrs={'background':'http://www.benchmark.pl/uploads/backend_img/a/fotki_newsy/opinie_dnia/bg.png'}), dict(name='table', attrs={'width':'210', 'cellspacing':'1', 'cellpadding':'4', 'border':'0', 'align':'right'})]
     INDEX= 'http://www.benchmark.pl'
     feeds          = [(u'Aktualności', u'http://www.benchmark.pl/rss/aktualnosci-pliki.xml'), 
                           (u'Testy i recenzje', u'http://www.benchmark.pl/rss/testy-recenzje-minirecenzje.xml')]

recipes/conowego_pl.recipe

@@ -0,0 +1,38 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+from calibre.ebooks.BeautifulSoup import BeautifulSoup
+class CoNowegoPl(BasicNewsRecipe):
+    title          = u'conowego.pl'
+    __author__        = 'fenuks'
+    description   = u'Nowy wortal technologiczny oraz gazeta internetowa. Testy najnowszych produktów, fachowe porady i recenzje. U nas znajdziesz wszystko o elektronice użytkowej !'
+    cover_url      = 'http://www.conowego.pl/fileadmin/templates/main/images/logo_top.png'
+    category       = 'IT, news'
+    language       = 'pl'
+    oldest_article = 7
+    max_articles_per_feed = 100
+    no_stylesheets = True
+    remove_empty_feeds = True
+    use_embedded_content = False
+    keep_only_tags = [dict(name='div', attrs={'class':'news_list single_view'})]
+    remove_tags = [dict(name='div', attrs={'class':['ni_bottom', 'ni_rank', 'ni_date']})]
+    feeds          = [(u'Aktualno\u015bci', u'http://www.conowego.pl/rss/aktualnosci-5/?type=100'), (u'Gaming', u'http://www.conowego.pl/rss/gaming-6/?type=100'), (u'Porady', u'http://www.conowego.pl/rss/porady-3/?type=100'), (u'Testy', u'http://www.conowego.pl/rss/testy-2/?type=100')]
+
+    def preprocess_html(self, soup):
+        for i in soup.findAll('img'):
+            i.parent.insert(0, BeautifulSoup('<br />'))
+            i.insert(len(i), BeautifulSoup('<br />'))
+        self.append_page(soup, soup.body)
+        return soup
+
+
+    def append_page(self, soup, appendtag):
+        tag = appendtag.find('div', attrs={'class':'pages'})
+        if tag:
+            nexturls=tag.findAll('a')
+            for nexturl in nexturls[:-1]:
+                soup2 = self.index_to_soup('http://www.conowego.pl/' + nexturl['href'])
+                pagetext = soup2.find(attrs={'class':'ni_content'})
+                pos = len(appendtag.contents)
+                appendtag.insert(pos, pagetext)
+
+            for r in appendtag.findAll(attrs={'class':['pages', 'paginationWrap']}):
+                r.extract()

recipes/film_web.recipe

@@ -1,6 +1,7 @@
 from calibre.web.feeds.news import BasicNewsRecipe
-
+import re
-class Filmweb_pl(BasicNewsRecipe):
+from calibre.ebooks.BeautifulSoup import BeautifulSoup
+class FilmWebPl(BasicNewsRecipe):
     title          = u'FilmWeb'
     __author__        = 'fenuks'
     description   = 'FilmWeb - biggest polish movie site'
@@ -12,8 +13,9 @@ class Filmweb_pl(BasicNewsRecipe):
     max_articles_per_feed = 100
     no_stylesheets= True
     remove_empty_feeds=True
+    preprocess_regexps = [(re.compile(u'\(kliknij\,\ aby powiększyć\)', re.IGNORECASE), lambda m: ''), ]#(re.compile(ur' | ', re.IGNORECASE), lambda m: '')]
     extra_css      = '.hdrBig {font-size:22px;} ul {list-style-type:none; padding: 0; margin: 0;}'
-    remove_tags= [dict(name='div', attrs={'class':['recommendOthers']}), dict(name='ul', attrs={'class':'fontSizeSet'})]
+    remove_tags= [dict(name='div', attrs={'class':['recommendOthers']}), dict(name='ul', attrs={'class':'fontSizeSet'}), dict(attrs={'class':'userSurname anno'})]
     keep_only_tags= [dict(name='h1', attrs={'class':['hdrBig', 'hdrEntity']}), dict(name='div', attrs={'class':['newsInfo', 'newsInfoSmall', 'reviewContent description']})]
     feeds          = [(u'Wszystkie newsy', u'http://www.filmweb.pl/feed/news/latest'),
                          (u'News / Filmy w produkcji', 'http://www.filmweb.pl/feed/news/category/filminproduction'),
@@ -31,13 +33,12 @@ class Filmweb_pl(BasicNewsRecipe):
                          (u'News / Kino polskie', u'http://www.filmweb.pl/feed/news/category/polish.cinema'),
                          (u'News / Telewizja', u'http://www.filmweb.pl/feed/news/category/tv'),
                          (u'Recenzje redakcji', u'http://www.filmweb.pl/feed/reviews/latest'),
-                         (u'Recenzje użytkowników', u'http://www.filmweb.pl/feed/user-reviews/latest')]
+                         (u'Recenzje użytkowników', u'http://www.filmweb.pl/feed/user-reviews/latest')
+                          ]
 
     def skip_ad_pages(self, soup):
         skip_tag = soup.find('a', attrs={'class':'welcomeScreenButton'})
         if skip_tag is not None:
-            self.log.warn('skip_tag')
-            self.log.warn(skip_tag)
             return self.index_to_soup(skip_tag['href'], raw=True)
 
 
@@ -45,4 +46,9 @@ class Filmweb_pl(BasicNewsRecipe):
         for a in soup('a'):
             if a.has_key('href') and 'http://' not in a['href'] and 'https://' not in a['href']:
                 a['href']=self.index + a['href']
+        for i in soup.findAll('a', attrs={'class':'fn'}):
+            i.insert(len(i), BeautifulSoup('<br />'))
+        for i in soup.findAll('sup'):
+            if not i.string or i.string.startswith('(kliknij'):
+                i.extract()
         return soup

recipes/gry_online_pl.recipe

@@ -1,6 +1,6 @@
 from calibre.web.feeds.recipes import BasicNewsRecipe
 
-class Gry_online_pl(BasicNewsRecipe):
+class GryOnlinePl(BasicNewsRecipe):
     title          = u'Gry-Online.pl'
     __author__        = 'fenuks'
     description   = 'Gry-Online.pl - computer games'
@@ -21,17 +21,18 @@ class Gry_online_pl(BasicNewsRecipe):
         tag = appendtag.find('div', attrs={'class':'n5p'})
         if tag:
             nexturls=tag.findAll('a')
-            for nexturl in nexturls[1:]:
+            url_part = soup.find('link', attrs={'rel':'canonical'})['href']
-                try:
+            url_part = url_part[25:].rpartition('?')[0]
-                    soup2 = self.index_to_soup('http://www.gry-online.pl/S020.asp'+ nexturl['href'])
+            for nexturl in nexturls[1:-1]:
-                except:
+                soup2 = self.index_to_soup('http://www.gry-online.pl/' + url_part + nexturl['href'])
-                    soup2 = self.index_to_soup('http://www.gry-online.pl/S022.asp'+ nexturl['href'])
                 pagetext = soup2.find(attrs={'class':'gc660'})
                 for r in pagetext.findAll(name='header'):
                     r.extract()
+                for r in pagetext.findAll(attrs={'itemprop':'description'}):
+                    r.extract()
                 pos = len(appendtag.contents)
                 appendtag.insert(pos, pagetext)
-            for r in appendtag.findAll(attrs={'class':['n5p', 'add-info', 'twitter-share-button']}):
+            for r in appendtag.findAll(attrs={'class':['n5p', 'add-info', 'twitter-share-button', 'lista lista3 lista-gry']}):
                 r.extract()
 
 

recipes/linux_journal.recipe

@@ -0,0 +1,36 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class LinuxJournal(BasicNewsRecipe):
+    title          = u'Linux Journal'
+    __author__        = 'fenuks'
+    description   = u'The monthly magazine of the Linux community, promoting the use of Linux worldwide.'
+    cover_url      = 'http://www.linuxjournal.com/files/linuxjournal.com/ufiles/logo-lj.jpg'
+    category       = 'IT, Linux'
+    language       = 'en'
+    oldest_article = 7
+    max_articles_per_feed = 100
+    no_stylesheets = True
+    use_embedded_content = False
+    remove_empty_feeds = True
+    keep_only_tags=[dict(id='content-inner')]
+    remove_tags_after= dict(attrs={'class':'user-signature clear-block'})
+    remove_tags=[dict(attrs={'class':['user-signature clear-block', 'breadcrumb',  'terms terms-inline']})]
+    feeds          = [(u'Front Page', u'http://feeds.feedburner.com/linuxjournalcom'), (u'News', u'http://feeds.feedburner.com/LinuxJournal-BreakingNews'), (u'Blogs', u'http://www.linuxjournal.com/blog/feed'), (u'Audio/Video', u'http://www.linuxjournal.com/taxonomy/term/28/0/feed'), (u'Community', u'http://www.linuxjournal.com/taxonomy/term/18/0/feed'), (u'Education', u'http://www.linuxjournal.com/taxonomy/term/25/0/feed'), (u'Embedded', u'http://www.linuxjournal.com/taxonomy/term/27/0/feed'), (u'Hardware', u'http://www.linuxjournal.com/taxonomy/term/23/0/feed'), (u'HOWTOs', u'http://www.linuxjournal.com/taxonomy/term/19/0/feed'), (u'International', u'http://www.linuxjournal.com/taxonomy/term/30/0/feed'), (u'Security', u'http://www.linuxjournal.com/taxonomy/term/31/0/feed'), (u'Software', u'http://www.linuxjournal.com/taxonomy/term/17/0/feed'), (u'Sysadmin', u'http://www.linuxjournal.com/taxonomy/term/21/0/feed'), (u'Webmaster', u'http://www.linuxjournal.com/taxonomy/term/24/0/feed')]
+
+    def append_page(self, soup, appendtag):
+        next = appendtag.find('li', attrs={'class':'pager-next'})
+        while next:    
+            nexturl = next.a['href']
+            appendtag.find('div', attrs={'class':'links'}).extract()
+            soup2 = self.index_to_soup('http://www.linuxjournal.com'+ nexturl)
+            pagetext = soup2.find(attrs={'class':'node-inner'}).find(attrs={'class':'content'})
+            next = appendtag.find('li', attrs={'class':'pager-next'})
+            pos = len(appendtag.contents)
+            appendtag.insert(pos, pagetext)
+        tag = appendtag.find('div', attrs={'class':'links'})
+        if tag:
+            tag.extract()
+
+    def preprocess_html(self, soup):
+        self.append_page(soup, soup.body)
+        return soup

recipes/natemat_pl.recipe

@@ -1,3 +1,4 @@
+import re
 from calibre.web.feeds.news import BasicNewsRecipe
 
 class NaTemat(BasicNewsRecipe):
@@ -8,8 +9,9 @@ class NaTemat(BasicNewsRecipe):
     description   = u'informacje, komentarze, opinie'
     category       = 'news'
     language       = 'pl'
+    preprocess_regexps = [(re.compile(ur'Czytaj też\:.*?</a>', re.IGNORECASE), lambda m: ''), (re.compile(ur'Zobacz też\:.*?</a>', re.IGNORECASE), lambda m: ''), (re.compile(ur'Czytaj więcej\:.*?</a>', re.IGNORECASE), lambda m: ''), (re.compile(ur'Czytaj również\:.*?</a>', re.IGNORECASE), lambda m: '')]
     cover_url= 'http://blog.plona.pl/wp-content/uploads/2012/05/natemat.png'
     no_stylesheets = True
     keep_only_tags= [dict(id='main')]
-    remove_tags= [dict(attrs={'class':['button', 'block-inside style_default', 'article-related']})]
+    remove_tags= [dict(attrs={'class':['button', 'block-inside style_default', 'article-related', 'user-header', 'links']}), dict(name='img', attrs={'class':'indent'})]
     feeds          = [(u'Artyku\u0142y', u'http://natemat.pl/rss/wszystkie')]

recipes/psych.recipe

@@ -1,44 +1,79 @@
+import re
+from calibre.web.feeds.recipes import BasicNewsRecipe
 
-from calibre.ptempfile import PersistentTemporaryFile
-from calibre.web.feeds.news import BasicNewsRecipe
 
-class AdvancedUserRecipe1275708473(BasicNewsRecipe):
+class PsychologyToday(BasicNewsRecipe):
-    title          = u'Psychology Today'
+
-    _author__ = 'rty'
+    title       = 'Psychology Today'
-    publisher = u'www.psychologytoday.com'
+    __author__  = 'Rick Shang'
-    category = u'Psychology'
+
-    max_articles_per_feed = 100
+    description = 'This magazine takes information from the latest research in the field of psychology and makes it useful to people in their everyday lives. Its coverage encompasses self-improvement, relationships, the mind-body connection, health, family, the workplace and culture.'
-    remove_javascript = True
-    use_embedded_content   = False
-    no_stylesheets = True
     language = 'en'
-    temp_files = []
+    category = 'news'
-    articles_are_obfuscated = True
+    encoding = 'UTF-8'
-    remove_tags = [
+    keep_only_tags = [dict(attrs={'class':['print-title', 'print-submitted', 'print-content', 'print-footer', 'print-source_url', 'print-links']})]
-                    dict(name='div', attrs={'class':['print-source_url','field-items','print-footer']}),
+    no_javascript = True
-                    dict(name='span', attrs={'class':'print-footnote'}),
+    no_stylesheets = True
-                  ]
-    remove_tags_before  = dict(name='h1', attrs={'class':'print-title'})
-    remove_tags_after     = dict(name='div', attrs={'class':['field-items','print-footer']})
 
-    feeds          = [(u'Contents', u'http://www.psychologytoday.com/articles/index.rss')]
 
-    def get_article_url(self, article):
+    def parse_index(self):
-       return article.get('link',  None)
+        articles = []
+        soup = self.index_to_soup('http://www.psychologytoday.com/magazine')
+
+
+        #Go to the main body
+        div = soup.find('div',attrs={'id':'content-content'})
+        #Find cover & date
+        cover_item = div.find('div', attrs={'class':'collections-header-image'})
+        cover = cover_item.find('img',src=True)
+        self.cover_url = cover['src']
+        date = self.tag_to_string(cover['title'])
+        self.timefmt = u' [%s]'%date
+
+        articles = []
+        for post in div.findAll('div', attrs={'class':'collections-node-feature-info'}):
+            title = self.tag_to_string(post.find('h2'))
+            author_item=post.find('div', attrs={'class':'collection-node-byline'})
+            author = re.sub(r'.*by\s',"",self.tag_to_string(author_item).strip())
+            title = title + u' (%s)'%author
+            article_page= self.index_to_soup('http://www.psychologytoday.com'+post.find('a', href=True)['href'])
+            print_page=article_page.find('li', attrs={'class':'print_html first'})
+            url='http://www.psychologytoday.com'+print_page.find('a',href=True)['href']
+            desc = self.tag_to_string(post.find('div', attrs={'class':'collection-node-description'})).strip()
+            self.log('Found article:', title)
+            self.log('\t', url)
+            self.log('\t', desc)
+            articles.append({'title':title, 'url':url, 'date':'','description':desc})
+
+        for post in div.findAll('div', attrs={'class':'collections-node-thumbnail-info'}):
+            title = self.tag_to_string(post.find('h2'))
+            author_item=post.find('div', attrs={'class':'collection-node-byline'})
+            article_page= self.index_to_soup('http://www.psychologytoday.com'+post.find('a', href=True)['href'])
+            print_page=article_page.find('li', attrs={'class':'print_html first'})
+            description = post.find('div', attrs={'class':'collection-node-description'})
+            author = re.sub(r'.*by\s',"",self.tag_to_string(description.nextSibling).strip())
+            desc = self.tag_to_string(description).strip()
+            url='http://www.psychologytoday.com'+print_page.find('a',href=True)['href']
+            title = title + u' (%s)'%author
+            self.log('Found article:', title)
+            self.log('\t', url)
+            self.log('\t', desc)
+            articles.append({'title':title, 'url':url, 'date':'','description':desc})
+
+        for post in div.findAll('li', attrs={'class':['collection-item-list-odd','collection-item-list-even']}):
+            title = self.tag_to_string(post.find('h2'))
+            author_item=post.find('div', attrs={'class':'collection-node-byline'})
+            author = re.sub(r'.*by\s',"",self.tag_to_string(author_item).strip())
+            title = title + u' (%s)'%author
+            article_page= self.index_to_soup('http://www.psychologytoday.com'+post.find('a', href=True)['href'])
+            print_page=article_page.find('li', attrs={'class':'print_html first'})
+            url='http://www.psychologytoday.com'+print_page.find('a',href=True)['href']
+            desc = self.tag_to_string(post.find('div', attrs={'class':'collection-node-description'})).strip()
+            self.log('Found article:', title)
+            self.log('\t', url)
+            self.log('\t', desc)
+            articles.append({'title':title, 'url':url, 'date':'','description':desc})
+
+        return [('Current Issue', articles)]
 
-    def get_obfuscated_article(self, url):
-        br = self.get_browser()
-        br.open(url)
-        response = br.follow_link(url_regex = r'/print/[0-9]+', nr = 0)
-        html = response.read()
-        self.temp_files.append(PersistentTemporaryFile('_fa.html'))
-        self.temp_files[-1].write(html)
-        self.temp_files[-1].close()
-        return self.temp_files[-1].name
 
-    def get_cover_url(self):
-        index = 'http://www.psychologytoday.com/magazine/'
-        soup = self.index_to_soup(index)
-        for image in soup.findAll('img',{ "class" : "imagefield imagefield-field_magazine_cover" }):
-              return image['src'] + '.jpg'
-        return None

recipes/smith.recipe

@@ -1,61 +1,67 @@
 import re
-from calibre.web.feeds.news import BasicNewsRecipe
+from calibre.web.feeds.recipes import BasicNewsRecipe
-from calibre.ebooks.BeautifulSoup import BeautifulSoup
+from collections import OrderedDict
 
-class SmithsonianMagazine(BasicNewsRecipe):
+class Smithsonian(BasicNewsRecipe):
-    title          = u'Smithsonian Magazine'
-    language       = 'en'
-    __author__     = 'Krittika Goyal and TerminalVeracity'
-    oldest_article = 31#days
-    max_articles_per_feed = 50
-    use_embedded_content = False
-    recursions = 1
-    cover_url = 'http://sphotos.xx.fbcdn.net/hphotos-snc7/431147_10150602715983253_764313347_n.jpg'
-    match_regexps = ['&page=[2-9]$']
-    preprocess_regexps = [
-        (re.compile(r'for more of Smithsonian\'s coverage on history, science and nature.', re.DOTALL), lambda m: '')
-        ]
-    extra_css             = """
-                               h1{font-size: large; margin: .2em 0}
-                               h2{font-size: medium; margin: .2em 0}
-                               h3{font-size: medium; margin: .2em 0}
-                               #byLine{margin: .2em 0}
-                               .articleImageCaptionwide{font-style: italic}
-                               .wp-caption-text{font-style: italic}
-                               img{display: block}
-                            """
 
+    title       = 'Smithsonian Magazine'
+    __author__  = 'Rick Shang'
 
-    remove_stylesheets = True
+    description = 'This magazine chronicles the arts, environment, sciences and popular culture of the times. It is edited for modern, well-rounded individuals with diverse, general interests. With your order, you become a National Associate Member of the Smithsonian. Membership benefits include your subscription to Smithsonian magazine, a personalized membership card, discounts from the Smithsonian catalog, and more.'
-    remove_tags_after  = dict(name='div', attrs={'class':['post','articlePaginationWrapper']})
+    language = 'en'
-    remove_tags = [
+    category = 'news'
-       dict(name='iframe'),
+    encoding = 'UTF-8'
-       dict(name='div', attrs={'class':['article_sidebar_border','viewMorePhotos','addtoany_share_save_container','meta','social','OUTBRAIN','related-articles-inpage']}),
+    keep_only_tags = [dict(attrs={'id':['articleTitle', 'subHead', 'byLine', 'articleImage', 'article-text']})]
-       dict(name='div', attrs={'id':['article_sidebar_border', 'most-popular_large', 'most-popular-body_large','comment_section','article-related']}),
+    remove_tags = [dict(attrs={'class':['related-articles-inpage', 'viewMorePhotos']})]
-       dict(name='ul', attrs={'class':'cat-breadcrumb col three last'}),
+    no_javascript = True
-       dict(name='h4', attrs={'id':'related-topics'}),
+    no_stylesheets = True
-       dict(name='table'),
-       dict(name='a', attrs={'href':['/subArticleBottomWeb','/subArticleTopWeb','/subArticleTopMag','/subArticleBottomMag']}),
-       dict(name='a', attrs={'name':'comments_shaded'}),
-    ]
 
+    def parse_index(self):
+        #Go to the issue
+        soup0 = self.index_to_soup('http://www.smithsonianmag.com/issue/archive/')
+        div = soup0.find('div',attrs={'id':'archives'})
+        issue = div.find('ul',attrs={'class':'clear-both'})
+        current_issue_url = issue.find('a', href=True)['href']
+        soup = self.index_to_soup(current_issue_url)
 
-    feeds          = [
+        #Go to the main body
-('History and Archeology',
+        div = soup.find ('div', attrs={'id':'content-inset'})
- 'http://feeds.feedburner.com/smithsonianmag/history-archaeology'),
+
-('People and Places',
+        #Find date
- 'http://feeds.feedburner.com/smithsonianmag/people-places'),
+        date = re.sub('.*\:\W*', "", self.tag_to_string(div.find('h2')).strip())
-('Science and Nature',
+        self.timefmt = u' [%s]'%date
- 'http://feeds.feedburner.com/smithsonianmag/science-nature'),
+
-('Arts and Culture',
+        #Find cover
- 'http://feeds.feedburner.com/smithsonianmag/arts-culture'),
+        self.cover_url = div.find('img',src=True)['src']
-('Travel',
+
- 'http://feeds.feedburner.com/smithsonianmag/travel'),
+        feeds = OrderedDict()
-]
+        section_title = ''
+        subsection_title = ''
+        for post in div.findAll('div', attrs={'class':['plainModule', 'departments plainModule']}):
+            articles = []
+            prefix = ''
+            h3=post.find('h3')
+            if h3 is not None:
+                section_title = self.tag_to_string(h3)
+            else:
+                subsection=post.find('p',attrs={'class':'article-cat'})
+                link=post.find('a',href=True)
+                url=link['href']+'?c=y&story=fullstory'
+                if subsection is not None:
+                    subsection_title = self.tag_to_string(subsection)
+                    prefix = (subsection_title+': ')
+                    description=self.tag_to_string(post('p', limit=2)[1]).strip()
+                else:
+                    description=self.tag_to_string(post.find('p')).strip()
+                desc=re.sub('\sBy\s.*', '', description, re.DOTALL)
+                author=re.sub('.*By\s', '', description, re.DOTALL)
+                title=prefix + self.tag_to_string(link).strip()+ u' (%s)'%author
+                articles.append({'title':title, 'url':url, 'description':desc, 'date':''})
+
+            if articles:
+                if section_title not in feeds:
+                    feeds[section_title] = []
+                feeds[section_title] += articles
+        ans = [(key, val) for key, val in feeds.iteritems()]
+        return ans
 
-    def preprocess_html(self, soup):
-        story = soup.find(name='div', attrs={'id':'article-body'})
-        soup = BeautifulSoup('<html><head><title>t</title></head><body></body></html>')
-        body = soup.find(name='body')
-        body.insert(0, story)
-        return soup

recipes/the_new_republic.recipe

@@ -1,45 +1,64 @@
-from calibre.web.feeds.news import BasicNewsRecipe
+import re
+from calibre.web.feeds.recipes import BasicNewsRecipe
+from collections import OrderedDict
 
-class The_New_Republic(BasicNewsRecipe):
+class TNR(BasicNewsRecipe):
-    title = 'The New Republic'
+
-    __author__ = 'cix3'
+    title       = 'The New Republic'
+    __author__  = 'Rick Shang'
+
+    description = 'The New Republic is a journal of opinion with an emphasis on politics and domestic and international affairs. It carries feature articles by staff and contributing editors. The second half of each issue is devoted to book and the arts, theater, motion pictures, music and art.'
     language = 'en'
-    description = 'Intelligent, stimulating and rigorous examination of American politics, foreign policy and culture'
+    category = 'news'
-    timefmt = ' [%b %d, %Y]'
+    encoding = 'UTF-8'
-
+    remove_tags = [dict(attrs={'class':['print-logo','print-site_name','print-hr']})]
-    oldest_article = 7
+    no_javascript = True
-    max_articles_per_feed = 100
     no_stylesheets = True
 
-    remove_tags = [
-            dict(name='div', attrs={'class':['print-logo', 'print-site_name', 'img-left', 'print-source_url']}),
-            dict(name='hr', attrs={'class':'print-hr'}), dict(name='img')
-            ]
 
-    feeds = [
+    def parse_index(self):
-        ('Politics', 'http://www.tnr.com/rss/articles/Politics'),
-        ('Books and Arts', 'http://www.tnr.com/rss/articles/Books-and-Arts'),
-        ('Economy', 'http://www.tnr.com/rss/articles/Economy'),
-        ('Environment and Energy', 'http://www.tnr.com/rss/articles/Environment-%2526-Energy'),
-        ('Health Care', 'http://www.tnr.com/rss/articles/Health-Care'),
-        ('Metro Policy', 'http://www.tnr.com/rss/articles/Metro-Policy'),
-        ('World', 'http://www.tnr.com/rss/articles/World'),
-        ('Film', 'http://www.tnr.com/rss/articles/Film'),
-        ('Books', 'http://www.tnr.com/rss/articles/books'),
-        ('The Book', 'http://www.tnr.com/rss/book'),
-        ('Jonathan Chait', 'http://www.tnr.com/rss/blogs/Jonathan-Chait'),
-        ('The Plank', 'http://www.tnr.com/rss/blogs/The-Plank'),
-        ('The Treatment', 'http://www.tnr.com/rss/blogs/The-Treatment'),
-        ('The Spine', 'http://www.tnr.com/rss/blogs/The-Spine'),
-        ('The Vine', 'http://www.tnr.com/rss/blogs/The-Vine'),
-        ('The Avenue', 'http://www.tnr.com/rss/blogs/The-Avenue'),
-        ('William Galston', 'http://www.tnr.com/rss/blogs/William-Galston'),
-        ('Simon Johnson', 'http://www.tnr.com/rss/blogs/Simon-Johnson'),
-        ('Ed Kilgore', 'http://www.tnr.com/rss/blogs/Ed-Kilgore'),
-        ('Damon Linker', 'http://www.tnr.com/rss/blogs/Damon-Linker'),
-        ('John McWhorter', 'http://www.tnr.com/rss/blogs/John-McWhorter')
-            ]
 
-    def print_version(self, url):
+        #Go to the issue
-        return url.replace('http://www.tnr.com/', 'http://www.tnr.com/print/')
+        soup0 = self.index_to_soup('http://www.tnr.com/magazine-issues')
+        issue = soup0.find('div',attrs={'id':'current_issue'})
 
+        #Find date
+        date = self.tag_to_string(issue.find('div',attrs={'class':'date'})).strip()
+        self.timefmt = u' [%s]'%date
+
+        #Go to the main body
+        current_issue_url = 'http://www.tnr.com' + issue.find('a', href=True)['href']
+        soup = self.index_to_soup(current_issue_url)
+        div = soup.find ('div', attrs={'class':'article_detail_body'})
+
+
+
+        #Find cover
+        self.cover_url = div.find('img',src=True)['src']
+
+        feeds = OrderedDict()
+        section_title = ''
+        subsection_title = ''
+        for post in div.findAll('p'):
+            articles = []
+            em=post.find('em')
+            b=post.find('b')
+            a=post.find('a',href=True)
+            if em is not None:
+                section_title = self.tag_to_string(em).strip()
+                subsection_title = ''
+            elif b is not None:
+                subsection_title=self.tag_to_string(b).strip()
+            elif a is not None:
+                prefix = (subsection_title+': ') if subsection_title else ''
+                url=re.sub('www.tnr.com','www.tnr.com/print', a['href'])
+                author=re.sub('.*by\s', '', self.tag_to_string(post), re.DOTALL)
+                title=prefix + self.tag_to_string(a).strip()+ u' (%s)'%author
+                articles.append({'title':title, 'url':url, 'description':'', 'date':''})
+
+            if articles:
+                if section_title not in feeds:
+                    feeds[section_title] = []
+                feeds[section_title] += articles
+        ans = [(key, val) for key, val in feeds.iteritems()]
+        return ans

recipes/wnp.recipe

@@ -1,7 +1,7 @@
 from calibre.web.feeds.news import BasicNewsRecipe
 import re
 
-class AdvancedUserRecipe1312886443(BasicNewsRecipe):
+class WNP(BasicNewsRecipe):
     title          = u'WNP'
     cover_url= 'http://k.wnp.pl/images/wnpLogo.gif'
     __author__        = 'fenuks'
@@ -12,7 +12,7 @@ class AdvancedUserRecipe1312886443(BasicNewsRecipe):
     oldest_article = 8
     max_articles_per_feed = 100
     no_stylesheets= True
-    remove_tags=[dict(attrs={'class':'printF'})]
+    remove_tags=[dict(attrs={'class':['printF', 'border3B2 clearfix', 'articleMenu clearfix']})]
     feeds          = [(u'Wiadomości gospodarcze', u'http://www.wnp.pl/rss/serwis_rss.xml'),
                           (u'Serwis Energetyka - Gaz', u'http://www.wnp.pl/rss/serwis_rss_1.xml'),
           (u'Serwis Nafta - Chemia', u'http://www.wnp.pl/rss/serwis_rss_2.xml'),

src/calibre/constants.py

@@ -4,7 +4,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = u'calibre'
-numeric_version = (0, 8, 61)
+numeric_version = (0, 8, 62)
 __version__   = u'.'.join(map(unicode, numeric_version))
 __author__    = u"Kovid Goyal <kovid@kovidgoyal.net>"
 

src/calibre/customize/conversion.py

@@ -91,6 +91,37 @@ class DummyReporter(object):
     def __call__(self, percent, msg=''):
         pass
 
+def gui_configuration_widget(name, parent, get_option_by_name,
+        get_option_help, db, book_id, for_output=True):
+    import importlib
+
+    def widget_factory(cls):
+        return cls(parent, get_option_by_name,
+            get_option_help, db, book_id)
+
+    if for_output:
+        try:
+            output_widget = importlib.import_module(
+                    'calibre.gui2.convert.'+name)
+            pw = output_widget.PluginWidget
+            pw.ICON = I('back.png')
+            pw.HELP = _('Options specific to the output format.')
+            return widget_factory(pw)
+        except ImportError:
+            pass
+    else:
+        try:
+            input_widget = importlib.import_module(
+                    'calibre.gui2.convert.'+name)
+            pw = input_widget.PluginWidget
+            pw.ICON = I('forward.png')
+            pw.HELP = _('Options specific to the input format.')
+            return widget_factory(pw)
+        except ImportError:
+            pass
+    return None
+
+
 class InputFormatPlugin(Plugin):
     '''
     InputFormatPlugins are responsible for converting a document into
@@ -225,6 +256,17 @@ class InputFormatPlugin(Plugin):
         '''
         pass
 
+    def gui_configuration_widget(self, parent, get_option_by_name,
+            get_option_help, db, book_id):
+        '''
+        Called to create the widget used for configuring this plugin in the
+        calibre GUI. The widget must be an instance of the PluginWidget class.
+        See the builting input plugins for examples.
+        '''
+        name = self.name.lower().replace(' ', '_')
+        return gui_configuration_widget(name, parent, get_option_by_name,
+                get_option_help, db, book_id, for_output=False)
+
 
 class OutputFormatPlugin(Plugin):
     '''
@@ -308,4 +350,16 @@ class OutputFormatPlugin(Plugin):
         '''
         pass
 
+    def gui_configuration_widget(self, parent, get_option_by_name,
+            get_option_help, db, book_id):
+        '''
+        Called to create the widget used for configuring this plugin in the
+        calibre GUI. The widget must be an instance of the PluginWidget class.
+        See the builtin output plugins for examples.
+        '''
+        name = self.name.lower().replace(' ', '_')
+        return gui_configuration_widget(name, parent, get_option_by_name,
+                get_option_help, db, book_id, for_output=True)
+
+
 

src/calibre/ebooks/conversion/plugins/epub_input.py

@@ -198,11 +198,13 @@ class EPUBInput(InputFormatPlugin):
                 ('application/vnd.adobe-page-template+xml','application/text'):
                     not_for_spine.add(id_)
 
+        seen = set()
         for x in list(opf.iterspine()):
             ref = x.get('idref', None)
-            if ref is None or ref in not_for_spine:
+            if not ref or ref in not_for_spine or ref in seen:
                 x.getparent().remove(x)
                 continue
+            seen.add(ref)
 
         if len(list(opf.iterspine())) == 0:
             raise ValueError('No valid entries in the spine of this EPUB')

src/calibre/ebooks/metadata/odt.py

@@ -1,5 +1,7 @@
 #!/usr/bin/python
 # -*- coding: utf-8 -*-
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai
+#
 # Copyright (C) 2006 Søren Roug, European Environment Agency
 #
 # This is free software.  You may redistribute it under the terms
@@ -17,12 +19,20 @@
 #
 # Contributor(s):
 #
+from __future__ import division
+
 import zipfile, re
 import xml.sax.saxutils
 from cStringIO import StringIO
 
 from odf.namespaces import OFFICENS, DCNS, METANS
-from calibre.ebooks.metadata import MetaInformation, string_to_authors
+from odf.opendocument import load as odLoad
+from odf.draw import Image as odImage, Frame as odFrame
+
+from calibre.ebooks.metadata import MetaInformation, string_to_authors, check_isbn
+from calibre.utils.magick.draw import identify_data
+from calibre.utils.date import parse_date
+from calibre.utils.localization import canonicalize_lang
 
 whitespace = re.compile(r'\s+')
 
@@ -125,6 +135,10 @@ class odfmetaparser(xml.sax.saxutils.XMLGenerator):
         else:
             texttag = self._tag
         self.seenfields[texttag] = self.data()
+        # OpenOffice has the habit to capitalize custom properties, so we add a
+        # lowercase version for easy access
+        if texttag[:4].lower() == u'opf.':
+            self.seenfields[texttag.lower()] = self.data()
 
         if field in self.deletefields:
             self.output.dowrite = True
@@ -141,7 +155,7 @@ class odfmetaparser(xml.sax.saxutils.XMLGenerator):
     def data(self):
         return normalize(''.join(self._data))
 
-def get_metadata(stream):
+def get_metadata(stream, extract_cover=True):
     zin = zipfile.ZipFile(stream, 'r')
     odfs = odfmetaparser()
     parser = xml.sax.make_parser()
@@ -162,7 +176,82 @@ def get_metadata(stream):
     if data.has_key('language'):
         mi.language = data['language']
     if data.get('keywords', ''):
-        mi.tags = data['keywords'].split(',')
+        mi.tags = [x.strip() for x in data['keywords'].split(',') if x.strip()]
+    opfmeta = False # we need this later for the cover
+    opfnocover = False
+    if data.get('opf.metadata','') == 'true':
+        # custom metadata contains OPF information
+        opfmeta = True
+        if data.get('opf.titlesort', ''):
+            mi.title_sort = data['opf.titlesort']
+        if data.get('opf.authors', ''):
+            mi.authors = string_to_authors(data['opf.authors'])
+        if data.get('opf.authorsort', ''):
+            mi.author_sort = data['opf.authorsort']
+        if data.get('opf.isbn', ''):
+            isbn = check_isbn(data['opf.isbn'])
+            if isbn is not None:
+                mi.isbn = isbn
+        if data.get('opf.publisher', ''):
+            mi.publisher = data['opf.publisher']
+        if data.get('opf.pubdate', ''):
+            mi.pubdate = parse_date(data['opf.pubdate'], assume_utc=True)
+        if data.get('opf.language', ''):
+            cl = canonicalize_lang(data['opf.language'])
+            if cl:
+                mi.languages = [cl]
+        opfnocover = data.get('opf.nocover', 'false') == 'true'
+    if not opfnocover:
+        try:
+            read_cover(stream, zin, mi, opfmeta, extract_cover)
+        except:
+            pass # Do not let an error reading the cover prevent reading other data
 
     return mi
 
+def read_cover(stream, zin, mi, opfmeta, extract_cover):
+    # search for an draw:image in a draw:frame with the name 'opf.cover'
+    # if opf.metadata prop is false, just use the first image that
+    # has a proper size (borrowed from docx)
+    otext = odLoad(stream)
+    cover_href = None
+    cover_data = None
+    # check that it's really a ODT
+    if otext.mimetype == u'application/vnd.oasis.opendocument.text':
+        for elem in otext.text.getElementsByType(odFrame):
+            img = elem.getElementsByType(odImage)
+            if len(img) > 0: # there should be only one
+                i_href = img[0].getAttribute('href')
+                try:
+                    raw = zin.read(i_href)
+                except KeyError:
+                    continue
+                try:
+                    width, height, fmt = identify_data(raw)
+                except:
+                    continue
+            else:
+                continue
+            if opfmeta and elem.getAttribute('name').lower() == u'opf.cover':
+                cover_href = i_href
+                cover_data = (fmt, raw)
+                break
+            if cover_href is None and 0.8 <= height/width <= 1.8 and height*width >= 12000:
+                cover_href = i_href
+                cover_data = (fmt, raw)
+                if not opfmeta:
+                    break
+
+    if cover_href is not None:
+        mi.cover = cover_href
+        if extract_cover:
+            if not cover_data:
+                raw = zin.read(cover_href)
+                try:
+                    width, height, fmt = identify_data(raw)
+                except:
+                    pass
+                else:
+                    cover_data = (fmt, raw)
+            mi.cover_data = cover_data
+

src/calibre/ebooks/metadata/opf2.py

@@ -286,15 +286,17 @@ class Spine(ResourceCollection): # {{{
     @staticmethod
     def from_opf_spine_element(itemrefs, manifest):
         s = Spine(manifest)
+        seen = set()
         for itemref in itemrefs:
             idref = itemref.get('idref', None)
             if idref is not None:
                 path = s.manifest.path_for_id(idref)
-                if path:
+                if path and path not in seen:
                     r = Spine.Item(lambda x:idref, path, is_path=True)
                     r.is_linear = itemref.get('linear', 'yes') == 'yes'
                     r.idref = idref
                     s.append(r)
+                    seen.add(path)
         return s
 
     @staticmethod

src/calibre/ebooks/mobi/writer2/serializer.py

@@ -12,7 +12,7 @@ import re
 from calibre.ebooks.oeb.base import (OEB_DOCS, XHTML, XHTML_NS, XML_NS,
         namespace, prefixname, urlnormalize)
 from calibre.ebooks.mobi.mobiml import MBP_NS
-from calibre.ebooks.mobi.utils import is_guide_ref_start
+from calibre.ebooks.mobi.utils import is_guide_ref_start, utf8_text
 
 from collections import defaultdict
 from urlparse import urldefrag
@@ -355,7 +355,7 @@ class Serializer(object):
         text = text.replace(u'\u00AD', '') # Soft-hyphen
         if quot:
             text = text.replace('"', '"')
-        self.buf.write(text.encode('utf-8'))
+        self.buf.write(utf8_text(text))
 
     def fixup_links(self):
         '''

src/calibre/ebooks/odt/input.py

@@ -142,7 +142,7 @@ class Extract(ODF2XHTML):
         from calibre.utils.zipfile import ZipFile
         from calibre.ebooks.metadata.meta import get_metadata
         from calibre.ebooks.metadata.opf2 import OPFCreator
-
+        from calibre.customize.ui import quick_metadata
 
         if not os.path.exists(odir):
             os.makedirs(odir)
@@ -163,7 +163,10 @@ class Extract(ODF2XHTML):
             zf = ZipFile(stream, 'r')
             self.extract_pictures(zf)
             stream.seek(0)
-            mi = get_metadata(stream, 'odt')
+            with quick_metadata:
+                # We dont want the cover, as it will lead to a duplicated image
+                # if no external cover is specified.
+                mi = get_metadata(stream, 'odt')
             if not mi.title:
                 mi.title = _('Unknown')
             if not mi.authors:

src/calibre/ebooks/oeb/display/paged.coffee

@@ -26,6 +26,7 @@ class PagedDisplay
         this.current_margin_side = 0
         this.is_full_screen_layout = false
         this.max_col_width = -1
+        this.current_page_height = null
         this.document_margins = null
         this.use_document_margins = false
 
@@ -74,25 +75,12 @@ class PagedDisplay
         # start_time = new Date().getTime()
         body_style = window.getComputedStyle(document.body)
         bs = document.body.style
-        # When laying body out in columns, webkit bleeds the top margin of the
-        # first block element out above the columns, leading to an extra top
-        # margin for the page. We compensate for that here. Computing the
-        # boundingrect of body is very expensive with column layout, so we do
-        # it before the column layout is applied.
         first_layout = false
         if not this.in_paged_mode
-            bs.setProperty('margin-top', '0px')
-            extra_margin = document.body.getBoundingClientRect().top
-            if extra_margin <= this.margin_top
-                extra_margin = 0
-            margin_top = (this.margin_top - extra_margin) + 'px'
             # Check if the current document is a full screen layout like
             # cover, if so we treat it specially.
             single_screen = (document.body.scrollWidth < window.innerWidth + 25 and document.body.scrollHeight < window.innerHeight + 25)
             first_layout = true
-        else
-            # resize event
-            margin_top = body_style.marginTop
 
         ww = window.innerWidth
 
@@ -116,16 +104,23 @@ class PagedDisplay
             col_width = Math.max(100, ((ww - adjust)/n) - 2*sm)
         this.page_width = col_width + 2*sm
         this.screen_width = this.page_width * this.cols_per_screen
+        this.current_page_height = window.innerHeight - this.margin_top - this.margin_bottom
 
         fgcolor = body_style.getPropertyValue('color')
 
         bs.setProperty('-webkit-column-gap', (2*sm)+'px')
         bs.setProperty('-webkit-column-width', col_width+'px')
         bs.setProperty('-webkit-column-rule-color', fgcolor)
+
+        # Without this, webkit bleeds the margin of the first block(s) of body
+        # above the columns, which causes them to effectively be added to the
+        # page margins (the margin collapse algorithm)
+        bs.setProperty('-webkit-margin-collapse', 'separate')
+
         bs.setProperty('overflow', 'visible')
         bs.setProperty('height', (window.innerHeight - this.margin_top - this.margin_bottom) + 'px')
         bs.setProperty('width', (window.innerWidth - 2*sm)+'px')
-        bs.setProperty('margin-top', margin_top)
+        bs.setProperty('margin-top', this.margin_top + 'px')
         bs.setProperty('margin-bottom', this.margin_bottom+'px')
         bs.setProperty('margin-left', sm+'px')
         bs.setProperty('margin-right', sm+'px')
@@ -167,9 +162,15 @@ class PagedDisplay
         # that this method use getBoundingClientRect() which means it will
         # force a relayout if the render tree is dirty.
         images = []
+        vimages = []
+        maxh = this.current_page_height
         for img in document.getElementsByTagName('img')
             previously_limited = calibre_utils.retrieve(img, 'width-limited', false)
+            data = calibre_utils.retrieve(img, 'img-data', null)
             br = img.getBoundingClientRect()
+            if data == null
+                data = {'left':br.left, 'right':br.right, 'height':br.height, 'display': img.style.display}
+                calibre_utils.store(img, 'img-data', data)
             left = calibre_utils.viewport_to_document(br.left, 0, doc=img.ownerDocument)[0]
             col = this.column_at(left) * this.page_width
             rleft = left - col - this.current_margin_side
@@ -178,23 +179,28 @@ class PagedDisplay
             col_width = this.page_width - 2*this.current_margin_side
             if previously_limited or rright > col_width
                 images.push([img, col_width - rleft])
+            previously_limited = calibre_utils.retrieve(img, 'height-limited', false)
+            if previously_limited or br.height > maxh
+                vimages.push(img)
+            if previously_limited
+                img.style.setProperty('-webkit-column-break-before', 'auto')
+                img.style.setProperty('display', data.display)
+            img.style.setProperty('-webkit-column-break-inside', 'avoid')
 
         for [img, max_width] in images
             img.style.setProperty('max-width', max_width+'px')
             calibre_utils.store(img, 'width-limited', true)
 
-    check_top_margin: () ->
+        for img in vimages
-        # This is needed to handle the case when a descendant of body specifies
+            data = calibre_utils.retrieve(img, 'img-data', null)
-        # a top margin as a percentage, which messes up the top margin
+            img.style.setProperty('-webkit-column-break-before', 'always')
-        # calculations above
+            img.style.setProperty('max-height', maxh+'px')
-        tm = document.body.getBoundingClientRect().top
+            if data.height > maxh
-        if tm != this.margin_top
+                # This is needed to force the image onto a new page, without
-            document.body.style.setProperty('margin-top', '0px')
+                # it, the webkit algorithm may still decide to split the image
-            tm = document.body.getBoundingClientRect().top
+                # by keeping it part of its parent block
-            if tm <= this.margin_top
+                img.style.setProperty('display', 'block')
-                tm = 0
+            calibre_utils.store(img, 'height-limited', true)
-            m = this.margin_top - tm
-            document.body.style.setProperty('margin-top', m+'px')
 
     scroll_to_pos: (frac) ->
         # Scroll to the position represented by frac (number between 0 and 1)

src/calibre/ebooks/pdf/writer.py

@@ -202,7 +202,6 @@ class PDFWriter(QObject): # {{{
         paged_display.set_geometry(1, 0, 0, 0);
         paged_display.layout();
         paged_display.fit_images();
-        paged_display.check_top_margin();
         ''')
         mf = self.view.page().mainFrame()
         while True:
@@ -221,7 +220,7 @@ class PDFWriter(QObject): # {{{
             self.tmp_path = PersistentTemporaryDirectory('_pdf_output_parts')
 
     def insert_cover(self):
-        if self.cover_data is None:
+        if not isinstance(self.cover_data, bytes):
             return
         item_path = os.path.join(self.tmp_path, 'cover.pdf')
         printer = get_pdf_printer(self.opts, output_file_name=item_path,

src/calibre/gui2/actions/plugin_updates.py

@@ -31,3 +31,5 @@ class PluginUpdaterAction(InterfaceAction):
 
         d = PluginUpdaterDialog(self.gui, initial_filter=initial_filter)
         d.exec_()
+        if d.do_restart:
+            self.gui.quit(restart=True)

src/calibre/gui2/actions/preferences.py

@@ -45,6 +45,8 @@ class PreferencesAction(InterfaceAction):
         d = PluginUpdaterDialog(self.gui,
                 initial_filter=FILTER_NOT_INSTALLED)
         d.exec_()
+        if d.do_restart:
+            self.gui.quit(restart=True)
 
     def do_config(self, checked=False, initial_plugin=None,
             close_after_initial=False):

src/calibre/gui2/comments_editor.py

@@ -8,6 +8,7 @@ __docformat__ = 'restructuredtext en'
 import re, os
 
 from lxml import html
+import sip
 
 from PyQt4.Qt import (QApplication, QFontInfo, QSize, QWidget, QPlainTextEdit,
     QToolBar, QVBoxLayout, QAction, QIcon, Qt, QTabWidget, QUrl,
@@ -42,6 +43,7 @@ class PageAction(QAction): # {{{
         self.page_action.trigger()
 
     def update_state(self, *args):
+        if sip.isdeleted(self) or sip.isdeleted(self.page_action): return
         if self.isCheckable():
             self.setChecked(self.page_action.isChecked())
         self.setEnabled(self.page_action.isEnabled())

src/calibre/gui2/convert/bulk.py

@@ -4,7 +4,7 @@ __license__ = 'GPL 3'
 __copyright__ = '2009, John Schember <john@nachtimwald.com>'
 __docformat__ = 'restructuredtext en'
 
-import shutil, importlib
+import shutil
 
 from PyQt4.Qt import QString, SIGNAL
 
@@ -86,17 +86,9 @@ class BulkConfig(Config):
         sd = widget_factory(StructureDetectionWidget)
         toc = widget_factory(TOCWidget)
 
-        output_widget = None
+        output_widget = self.plumber.output_plugin.gui_configuration_widget(
-        name = self.plumber.output_plugin.name.lower().replace(' ', '_')
+                self.stack, self.plumber.get_option_by_name,
-        try:
+                self.plumber.get_option_help, self.db)
-            output_widget = importlib.import_module(
-                    'calibre.gui2.convert.'+name)
-            pw = output_widget.PluginWidget
-            pw.ICON = I('back.png')
-            pw.HELP = _('Options specific to the output format.')
-            output_widget = widget_factory(pw)
-        except ImportError:
-            pass
 
         while True:
             c = self.stack.currentWidget()

src/calibre/gui2/convert/single.py

@@ -6,7 +6,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
-import cPickle, shutil, importlib
+import cPickle, shutil
 
 from PyQt4.Qt import QString, SIGNAL, QAbstractListModel, Qt, QVariant, QFont
 
@@ -187,29 +187,12 @@ class Config(ResizableDialog, Ui_Dialog):
         toc = widget_factory(TOCWidget)
         debug = widget_factory(DebugWidget)
 
-        output_widget = None
+        output_widget = self.plumber.output_plugin.gui_configuration_widget(
-        name = self.plumber.output_plugin.name.lower().replace(' ', '_')
+                self.stack, self.plumber.get_option_by_name,
-        try:
+                self.plumber.get_option_help, self.db, self.book_id)
-            output_widget = importlib.import_module(
+        input_widget = self.plumber.input_plugin.gui_configuration_widget(
-                    'calibre.gui2.convert.'+name)
+                self.stack, self.plumber.get_option_by_name,
-            pw = output_widget.PluginWidget
+                self.plumber.get_option_help, self.db, self.book_id)
-            pw.ICON = I('back.png')
-            pw.HELP = _('Options specific to the output format.')
-            output_widget = widget_factory(pw)
-        except ImportError:
-            pass
-        input_widget = None
-        name = self.plumber.input_plugin.name.lower().replace(' ', '_')
-        try:
-            input_widget = importlib.import_module(
-                    'calibre.gui2.convert.'+name)
-            pw = input_widget.PluginWidget
-            pw.ICON = I('forward.png')
-            pw.HELP = _('Options specific to the input format.')
-            input_widget = widget_factory(pw)
-        except ImportError:
-            pass
-
         while True:
             c = self.stack.currentWidget()
             if not c: break

src/calibre/gui2/device.py

@@ -30,7 +30,6 @@ from calibre.constants import DEBUG
 from calibre.utils.config import prefs, tweaks
 from calibre.utils.magick.draw import thumbnail
 from calibre.library.save_to_disk import find_plugboard
-from calibre.gui2 import is_gui_thread
 # }}}
 
 class DeviceJob(BaseJob): # {{{

src/calibre/gui2/dialogs/message_box.py

@@ -9,7 +9,7 @@ import sys
 
 from PyQt4.Qt import (QDialog, QIcon, QApplication, QSize, QKeySequence,
     QAction, Qt, QTextBrowser, QDialogButtonBox, QVBoxLayout, QGridLayout,
-    QLabel, QPlainTextEdit, QTextDocument)
+    QLabel, QPlainTextEdit, QTextDocument, QCheckBox, pyqtSignal)
 
 from calibre.constants import __version__, isfrozen
 from calibre.gui2.dialogs.message_box_ui import Ui_Dialog
@@ -270,21 +270,23 @@ class ErrorNotification(MessageBox): # {{{
 class JobError(QDialog): # {{{
 
     WIDTH = 600
+    do_pop = pyqtSignal()
 
-    def __init__(self, gui):
+    def __init__(self, parent):
-        QDialog.__init__(self, gui)
+        QDialog.__init__(self, parent)
         self.setAttribute(Qt.WA_DeleteOnClose, False)
-        self.gui = gui
         self.queue = []
+        self.do_pop.connect(self.pop, type=Qt.QueuedConnection)
 
         self._layout = l = QGridLayout()
         self.setLayout(l)
         self.icon = QIcon(I('dialog_error.png'))
         self.setWindowIcon(self.icon)
         self.icon_label = QLabel()
-        self.icon_label.setPixmap(self.icon.pixmap(128, 128))
+        self.icon_label.setPixmap(self.icon.pixmap(68, 68))
-        self.icon_label.setMaximumSize(QSize(128, 128))
+        self.icon_label.setMaximumSize(QSize(68, 68))
         self.msg_label = QLabel('<p>&nbsp;')
+        self.msg_label.setStyleSheet('QLabel { margin-top: 1ex; }')
         self.msg_label.setWordWrap(True)
         self.msg_label.setTextFormat(Qt.RichText)
         self.det_msg = QPlainTextEdit(self)
@@ -302,15 +304,23 @@ class JobError(QDialog): # {{{
         self.det_msg_toggle.clicked.connect(self.toggle_det_msg)
         self.det_msg_toggle.setToolTip(
                 _('Show detailed information about this error'))
+        self.suppress = QCheckBox(self)
 
         l.addWidget(self.icon_label, 0, 0, 1, 1)
-        l.addWidget(self.msg_label,  0, 1, 1, 1, Qt.AlignLeft|Qt.AlignTop)
+        l.addWidget(self.msg_label,  0, 1, 1, 1)
-        l.addWidget(self.det_msg, 1, 0, 1, 2)
+        l.addWidget(self.det_msg,    1, 0, 1, 2)
-
+        l.addWidget(self.suppress,   2, 0, 1, 2, Qt.AlignLeft|Qt.AlignBottom)
-        l.addWidget(self.bb, 2, 0, 1, 2, Qt.AlignRight|Qt.AlignBottom)
+        l.addWidget(self.bb,         3, 0, 1, 2, Qt.AlignRight|Qt.AlignBottom)
+        l.setColumnStretch(1, 100)
 
         self.setModal(False)
-        self.base_height = max(200, self.sizeHint().height() + 20)
+        self.suppress.setVisible(False)
+        self.do_resize()
+
+    def update_suppress_state(self):
+        self.suppress.setText(_(
+            'Hide the remaining %d error messages'%len(self.queue)))
+        self.suppress.setVisible(len(self.queue) > 3)
         self.do_resize()
 
     def copy_to_clipboard(self, *args):
@@ -332,9 +342,11 @@ class JobError(QDialog): # {{{
         self.do_resize()
 
     def do_resize(self):
-        h = self.base_height
+        h = self.sizeHint().height()
-        if self.det_msg.isVisible():
+        self.setMinimumHeight(0) # Needed as this gets set if det_msg is shown
-            h += 250
+        # Needed otherwise re-showing the box after showing det_msg causes the box
+        # to not reduce in height
+        self.setMaximumHeight(h)
         self.resize(QSize(self.WIDTH, h))
 
     def showEvent(self, ev):
@@ -342,16 +354,50 @@ class JobError(QDialog): # {{{
         self.bb.button(self.bb.Close).setFocus(Qt.OtherFocusReason)
         return ret
 
+    def show_error(self, title, msg, det_msg=u''):
+        self.queue.append((title, msg, det_msg))
+        self.update_suppress_state()
+        self.pop()
+
+    def pop(self):
+        if not self.queue or self.isVisible(): return
+        title, msg, det_msg = self.queue.pop(0)
+        self.setWindowTitle(title)
+        self.msg_label.setText(msg)
+        self.det_msg.setPlainText(det_msg)
+        self.det_msg.setVisible(False)
+        self.det_msg_toggle.setText(self.show_det_msg)
+        self.det_msg_toggle.setVisible(True)
+        self.suppress.setChecked(False)
+        self.update_suppress_state()
+        if not det_msg:
+            self.det_msg_toggle.setVisible(False)
+        self.do_resize()
+        self.show()
+
+    def done(self, r):
+        if self.suppress.isChecked():
+            self.queue = []
+        QDialog.done(self, r)
+        self.do_pop.emit()
+
 # }}}
 
 if __name__ == '__main__':
     app = QApplication([])
-    from calibre.gui2.preferences import init_gui
+    d = JobError(None)
-    gui = init_gui()
+    d.show_error('test title', 'some long meaningless test message', 'det msg')
-    d = JobError(gui)
+    d.show_error('test title', 'some long meaningless test message', 'det msg')
-    d.show()
+    d.show_error('test title', 'some long meaningless test message', 'det msg')
+    d.show_error('test title', 'some long meaningless test message', 'det msg')
+    d.show_error('test title', 'some long meaningless test message', 'det msg')
+    d.show_error('test title', 'some long meaningless test message', 'det msg')
+    app.setQuitOnLastWindowClosed(False)
+    def checkd():
+        if not d.queue:
+            app.quit()
+    app.lastWindowClosed.connect(checkd)
     app.exec_()
-    gui.shutdown()
 
 # if __name__ == '__main__':
 #     app = QApplication([])

src/calibre/gui2/dialogs/plugin_updater.py

@@ -456,6 +456,7 @@ class PluginUpdaterDialog(SizePersistedDialog):
         self.gui = gui
         self.forum_link = None
         self.model = None
+        self.do_restart = False
         self._initialize_controls()
         self._create_context_menu()
 
@@ -720,6 +721,7 @@ class PluginUpdaterDialog(SizePersistedDialog):
             prints('Installing plugin: ', zip_path)
         self.gui.status_bar.showMessage(_('Installing plugin: %s') % zip_path)
 
+        do_restart = False
         try:
             try:
                 plugin = add_plugin(zip_path)
@@ -731,11 +733,21 @@ class PluginUpdaterDialog(SizePersistedDialog):
             widget.gui = self.gui
             widget.check_for_add_to_toolbars(plugin)
             self.gui.status_bar.showMessage(_('Plugin installed: %s') % display_plugin.name)
-            info_dialog(self.gui, _('Success'),
+            d = info_dialog(self.gui, _('Success'),
                     _('Plugin <b>{0}</b> successfully installed under <b>'
                         ' {1} plugins</b>. You may have to restart calibre '
                         'for the plugin to take effect.').format(plugin.name, plugin.type),
-                    show=True, show_copy_button=False)
+                    show_copy_button=False)
+            b = d.bb.addButton(_('Restart calibre now'), d.bb.AcceptRole)
+            b.setIcon(QIcon(I('lt.png')))
+            d.do_restart = False
+            def rf():
+                d.do_restart = True
+            b.clicked.connect(rf)
+            d.set_details('')
+            d.exec_()
+            b.clicked.disconnect()
+            do_restart = d.do_restart
 
             display_plugin.plugin = plugin
             # We cannot read the 'actual' version information as the plugin will not be loaded yet
@@ -762,6 +774,9 @@ class PluginUpdaterDialog(SizePersistedDialog):
         else:
             self.model.refresh_plugin(display_plugin)
             self._select_and_focus_view(change_selection=False)
+        if do_restart:
+            self.do_restart = True
+            self.accept()
 
     def _history_clicked(self):
         display_plugin = self._selected_display_plugin()

src/calibre/gui2/dialogs/search.py

@@ -12,6 +12,7 @@ from calibre.utils.icu import sort_key
 from calibre.utils.config import tweaks
 
 box_values = {}
+last_matchkind = CONTAINS_MATCH
 
 class SearchDialog(QDialog, Ui_Dialog):
 
@@ -57,6 +58,9 @@ class SearchDialog(QDialog, Ui_Dialog):
 
         current_tab = gprefs.get('advanced search dialog current tab', 0)
         self.tabWidget.setCurrentIndex(current_tab)
+        if current_tab == 1:
+            self.matchkind.setCurrentIndex(last_matchkind)
+
         self.tabWidget.currentChanged[int].connect(self.tab_changed)
         self.tab_changed(current_tab)
 
@@ -173,7 +177,9 @@ class SearchDialog(QDialog, Ui_Dialog):
         general_index = unicode(self.general_combo.currentText())
         self.box_last_values['general_index'] = general_index
         global box_values
+        global last_matchkind
         box_values = copy.deepcopy(self.box_last_values)
+        last_matchkind = mk
         if general:
             ans.append(unicode(self.general_combo.currentText()) + ':"' +
                     self.mc + general + '"')

src/calibre/gui2/dialogs/smartdevice.py

@@ -1,11 +1,10 @@
 __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
-import re
+
 from PyQt4.QtGui import QDialog, QLineEdit
 from PyQt4.QtCore import SIGNAL, Qt
 
 from calibre.gui2.dialogs.smartdevice_ui import Ui_Dialog
-from calibre.gui2 import dynamic
 
 class SmartdeviceDialog(QDialog, Ui_Dialog):
 

src/calibre/gui2/preferences/__init__.py

@@ -236,6 +236,7 @@ class ConfigWidgetBase(QWidget, ConfigWidgetInterface):
     '''
 
     changed_signal = pyqtSignal()
+    restart_now = pyqtSignal()
     supports_restoring_to_defaults = True
     restart_critical = False
 

src/calibre/gui2/preferences/main.py

@@ -290,6 +290,7 @@ class Preferences(QMainWindow):
         self.apply_action.setEnabled(False)
         self.showing_widget.changed_signal.connect(lambda :
                 self.apply_action.setEnabled(True))
+        self.showing_widget.restart_now.connect(self.restart_now)
         self.restore_action.setEnabled(self.showing_widget.supports_restoring_to_defaults)
         tt = self.showing_widget.restore_defaults_desc
         if not self.restore_action.isEnabled():
@@ -319,6 +320,15 @@ class Preferences(QMainWindow):
         elif self.stack.currentIndex() == 0:
             self.close()
 
+    def restart_now(self):
+        try:
+            self.showing_widget.commit()
+        except AbortCommit:
+            return
+        self.hide_plugin()
+        self.close()
+        self.gui.quit(restart=True)
+
     def commit(self, *args):
         try:
             must_restart = self.showing_widget.commit()

src/calibre/gui2/preferences/plugins.py

@@ -384,6 +384,8 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         self._plugin_model.populate()
         self._plugin_model.reset()
         self.changed_signal.emit()
+        if d.do_restart:
+            self.restart_now.emit()
 
     def reload_store_plugins(self):
         self.gui.load_store_plugins()

src/calibre/gui2/store/stores/sony_plugin.py

@@ -32,6 +32,8 @@ class SonyStore(BasicStoreConfig, StorePlugin):
                 d.setWindowTitle(self.name)
                 d.set_tags(self.config.get('tags', ''))
                 d.exec_()
+        else:
+            open_url(QUrl('http://ebookstore.sony.com'))
 
     def search(self, query, max_results=10, timeout=60):
         url = 'http://ebookstore.sony.com/search?keyword=%s'%urllib.quote_plus(

src/calibre/gui2/ui.py

@@ -44,6 +44,7 @@ from calibre.gui2.keyboard import Manager
 from calibre.gui2.auto_add import AutoAdder
 from calibre.library.sqlite import sqlite, DatabaseException
 from calibre.gui2.proceed import ProceedQuestion
+from calibre.gui2.dialogs.message_box import JobError
 
 class Listener(Thread): # {{{
 
@@ -111,6 +112,7 @@ class Main(MainWindow, MainWindowMixin, DeviceMixin, EmailMixin, # {{{
         self.proceed_requested.connect(self.do_proceed,
                 type=Qt.QueuedConnection)
         self.proceed_question = ProceedQuestion(self)
+        self.job_error_dialog = JobError(self)
         self.keyboard = Manager(self)
         _gui = self
         self.opts = opts
@@ -690,12 +692,9 @@ class Main(MainWindow, MainWindowMixin, DeviceMixin, EmailMixin, # {{{
         except:
             pass
         if not minz:
-            d = error_dialog(self, dialog_title,
+            self.job_error_dialog.show_error(dialog_title,
                     _('<b>Failed</b>')+': '+unicode(job.description),
                     det_msg=job.details)
-            d.setModal(False)
-            d.show()
-            self._modeless_dialogs.append(d)
 
     def read_settings(self):
         geometry = config['main_window_geometry']

src/calibre/gui2/update.py

@@ -175,6 +175,8 @@ class UpdateMixin(object):
                 d = PluginUpdaterDialog(self,
                         initial_filter=FILTER_UPDATE_AVAILABLE)
                 d.exec_()
+                if d.do_restart:
+                    self.quit(restart=True)
 
     def plugin_update_found(self, number_of_updates):
         # Change the plugin icon to indicate there are updates available

src/calibre/gui2/viewer/documentview.py

@@ -243,7 +243,6 @@ class Document(QWebPage): # {{{
             sz.setWidth(scroll_width+side_margin)
             self.setPreferredContentsSize(sz)
         self.javascript('window.paged_display.fit_images()')
-        self.javascript('window.paged_display.check_top_margin()')
 
     @property
     def column_boundaries(self):

src/calibre/gui2/viewer/printing.py

@@ -71,7 +71,6 @@ class Printing(QObject):
                 paged_display.set_geometry(1, 0, 0, 0);
                 paged_display.layout();
                 paged_display.fit_images();
-                paged_display.check_top_margin();
             ''')
 
             while True:

src/calibre/utils/icu.c

@@ -4,6 +4,7 @@
 #include <unicode/utypes.h>
 #include <unicode/uclean.h>
 #include <unicode/ucol.h>
+#include <unicode/ucoleitr.h>
 #include <unicode/ustring.h>
 #include <unicode/usearch.h>
 
@@ -310,6 +311,41 @@ icu_Collator_startswith(icu_Collator *self, PyObject *args, PyObject *kwargs) {
     Py_RETURN_FALSE;
 } // }}}
 
+// Collator.startswith {{{
+static PyObject *
+icu_Collator_collation_order(icu_Collator *self, PyObject *args, PyObject *kwargs) {
+    PyObject *a_;
+    size_t asz;
+    int32_t actual_a;
+    UChar *a;
+    wchar_t *aw;
+    UErrorCode status = U_ZERO_ERROR;
+    UCollationElements *iter = NULL;
+    int order = 0, len = -1;
+  
+    if (!PyArg_ParseTuple(args, "U", &a_)) return NULL;
+    asz = PyUnicode_GetSize(a_); 
+    
+    a = (UChar*)calloc(asz*4 + 2, sizeof(UChar));
+    aw = (wchar_t*)calloc(asz*4 + 2, sizeof(wchar_t));
+
+    if (a == NULL || aw == NULL ) return PyErr_NoMemory();
+
+    actual_a = (int32_t)PyUnicode_AsWideChar((PyUnicodeObject*)a_, aw, asz*4+1);
+    if (actual_a > -1) {
+        u_strFromWCS(a, asz*4 + 1, &actual_a, aw, -1, &status);
+        iter = ucol_openElements(self->collator, a, actual_a, &status);
+        if (iter != NULL && U_SUCCESS(status)) {
+            order = ucol_next(iter, &status);
+            len = ucol_getOffset(iter);
+            ucol_closeElements(iter); iter = NULL;
+        }
+    }
+
+    free(a); free(aw); 
+    return Py_BuildValue("ii", order, len);
+} // }}}
+
 static PyObject*
 icu_Collator_clone(icu_Collator *self, PyObject *args, PyObject *kwargs);
 
@@ -338,6 +374,10 @@ static PyMethodDef icu_Collator_methods[] = {
         "startswith(a, b) -> returns True iff a startswith b, following the current collation rules."
     },
 
+    {"collation_order", (PyCFunction)icu_Collator_collation_order, METH_VARARGS,
+        "collation_order(string) -> returns (order, length) where order is an integer that gives the position of string in a list. length gives the number of characters used for order."
+    },
+
     {NULL}  /* Sentinel */
 };
 

src/calibre/utils/icu.py

@@ -75,6 +75,7 @@ def icu_sort_key(collator, obj):
     except AttributeError:
         return secondary_collator().sort_key(obj)
 
+
 def py_find(pattern, source):
     pos = source.find(pattern)
     if pos > -1:
@@ -126,6 +127,12 @@ def icu_contractions(collator):
         _cmap[collator] = ans
     return ans
 
+def icu_collation_order(collator, a):
+    try:
+        return collator.collation_order(a)
+    except TypeError:
+        return collator.collation_order(unicode(a))
+
 load_icu()
 load_collator()
 _icu_not_ok = _icu is None or _collator is None
@@ -205,6 +212,14 @@ def primary_startswith(a, b):
     except AttributeError:
         return icu_startswith(primary_collator(), a, b)
 
+def collation_order(a):
+    if _icu_not_ok:
+        return (ord(a[0]), 1) if a else (0, 0)
+    try:
+        return icu_collation_order(_secondary_collator, a)
+    except AttributeError:
+        return icu_collation_order(secondary_collator(), a)
+
 ################################################################################
 
 def test(): # {{{