Sync to trunk.

Changelog.yaml

@@ -19,6 +19,69 @@
 #  new recipes:
 #    - title: 
 
+- version: 0.8.6
+  date: 2011-06-17
+
+  new features:
+    - title: "Builtin support for downloading and installing/updating calibre plugins. Go to Preferences->Plugins and click 'Get new plugins'"
+      description: "When updates for installed plugins are available, calibre will automatically (unobtrusively) notify you"
+      type: major
+
+    - title: "Metadata download configuration: Allow defining a set of 'default' fields for metadata download and quichly switching to/from them"
+
+    - title: "Allow clicking on the news category in the Tag Browser to display all downloaded periodicals"
+
+    - title: "Driver for the Libre Air"
+
+    - title: "Email sending: Allow user to stop email jobs (note that stopping may not actually prevent the email from being sent, depending on when the stop happens). Also automatically abort email sending if it takes longer than 15mins."
+      tickets: [795960]
+  
+  bug fixes:
+    - title: "MOBI Output: Allow setting of background color on tables also set the border attribute on the table if the table has any border related css defined."
+      tickets: [797580]
+
+    - title: "Nook TSR: Put news sent to the device in My Files/Newspapers instaed of My Files/Books."
+      tickets: [796674]
+
+    - title: "MOBI Output: Fix a bug where linking to the very first element in an HTML file could sometimes result in the link pointing to the last element in the previous file."
+      tickets: [797214]
+
+    - title: "CSV catalog: Convert HTML comments to plain text"
+
+    - title: "HTML Input: Ignore links to text files."
+      tickets: [791568]
+
+    - title: "EPUB Output: Change orphaned <td> tags to <div> as they cause ADE to crash."
+
+    - title: "Fix 'Stop selected jobs' button trying to stop the same job multiple times"
+
+    - title: "Database: Explicitly test for case sensitivity on OS X instead of assuming a case insensitive filesystem."
+      tickets: [796258]
+
+    - title: "Get Books: More fixes to the Amazon store plugin"
+
+    - title: "FB2 Input: Do not specify font families/background colors"
+
+
+  improved recipes:
+    - Philadelphia Inquirer
+    - Macleans Magazone
+    - Metro UK
+
+  new recipes:
+    - title: "Christian Post, Down To Earth and Words Without Borders"
+      author: sexymax15
+
+    - title: "Noticias R7"
+      author: Diniz Bortolotto
+
+    - title: "UK Daily Mirror"
+      author: Dave Asbury
+
+    - title: "New Musical Express Magazine"
+      author: scissors
+
+
 - version: 0.8.5
   date: 2011-06-10
 

recipes/christian_post.recipe

@@ -0,0 +1,37 @@
+#created by sexymax15 ....sexymax15@gmail.com
+#christian post recipe
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class ChristianPost(BasicNewsRecipe):
+
+    title = 'The Christian Post'
+    __author__ = 'sexymax15'
+    description = 'Homepage'
+    language = 'en'
+    no_stylesheets = True
+    use_embedded_content	= False
+    oldest_article	= 30
+    max_articles_per_feed = 15
+
+    remove_empty_feeds = True
+    no_stylesheets = True
+    remove_javascript = True
+
+    extra_css = '''
+              h1 {color:#008852;font-family:Arial,Helvetica,sans-serif; font-size:20px; font-size-adjust:none; font-stretch:normal; font-style:normal; font-variant:normal; font-weight:bold; line-height:18px;}
+              h2 {color:#4D4D4D;font-family:Arial,Helvetica,sans-serif; font-size:16px; font-size-adjust:none; font-stretch:normal; font-style:normal; font-variant:normal; font-weight:bold; line-height:16px; } '''
+
+
+    feeds =  [
+             ('Homepage', 'http://www.christianpost.com/services/rss/feed/'),
+             ('Most Popular', 'http://www.christianpost.com/services/rss/feed/most-popular'),
+             ('Entertainment', 'http://www.christianpost.com/services/rss/feed/entertainment/'),
+             ('Politics', 'http://www.christianpost.com/services/rss/feed/politics/'),
+             ('Living', 'http://www.christianpost.com/services/rss/feed/living/'),
+             ('Business', 'http://www.christianpost.com/services/rss/feed/business/'),
+             ('Opinion', 'http://www.christianpost.com/services/rss/feed/opinion/')
+             ]
+
+    def print_version(self, url):
+        return url +'print.html'
+

recipes/down_to_earth.recipe

@@ -0,0 +1,18 @@
+from calibre.web.feeds.recipes import BasicNewsRecipe
+
+class AdvancedUserRecipe1307834113(BasicNewsRecipe):
+
+    title          = u'Down To Earth'
+    oldest_article = 300
+    __author__ = 'sexymax15'
+    max_articles_per_feed = 30
+    no_stylesheets = True
+    remove_javascript     = True
+    remove_attributes = ['width','height']
+    use_embedded_content  = False
+    language              = 'en_IN'
+    remove_empty_feeds    = True
+    remove_tags_before = dict(name='div', id='PageContent')
+    remove_tags_after = [dict(name='div'),{'class':'box'}]
+    remove_tags =[{'class':'box'}]
+    feeds          = [(u'editor', u'http://www.downtoearth.org.in/taxonomy/term/20348/0/feed'), (u'cover story', u'http://www.downtoearth.org.in/taxonomy/term/20345/0/feed'), (u'special report', u'http://www.downtoearth.org.in/taxonomy/term/20384/0/feed'), (u'features', u'http://www.downtoearth.org.in/taxonomy/term/20350/0/feed'), (u'news', u'http://www.downtoearth.org.in/taxonomy/term/20366/0/feed'), (u'debate', u'http://www.downtoearth.org.in/taxonomy/term/20347/0/feed'), (u'natural disasters', u'http://www.downtoearth.org.in/taxonomy/term/20822/0/feed')]

recipes/frontlineonnet.recipe

@@ -0,0 +1,81 @@
+__license__   = 'GPL v3'
+__copyright__ = '2011, Darko Miletic <darko.miletic at gmail.com>'
+'''
+frontlineonnet.com
+'''
+
+import re
+from calibre import strftime
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class Frontlineonnet(BasicNewsRecipe):
+    title                = 'Frontline'
+    __author__           = 'Darko Miletic'
+    description          = "India's national magazine"
+    publisher            = 'Frontline'
+    category             = 'news, politics, India'
+    no_stylesheets       = True
+    delay                = 1
+    INDEX                = 'http://frontlineonnet.com/'
+    use_embedded_content = False
+    encoding             = 'cp1252'
+    language             = 'en_IN'
+    publication_type     = 'magazine'
+    masthead_url         = 'http://frontlineonnet.com/images/newfline.jpg'
+    extra_css            = """
+                              body{font-family: Verdana,Arial,Helvetica,sans-serif}
+                              img{margin-top:0.5em; margin-bottom: 0.7em; display: block}
+                           """
+
+    conversion_options = {
+                          'comment'          : description
+                        , 'tags'             : category
+                        , 'publisher'        : publisher
+                        , 'language'         : language
+                        , 'linearize_tables' : True
+                        }
+
+    preprocess_regexps = [
+                           (re.compile(r'.*?<base', re.DOTALL|re.IGNORECASE),lambda match: '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html dir="ltr" xml:lang="en-IN"><head><title>title</title><base')
+                          ,(re.compile(r'<base .*?>', re.DOTALL|re.IGNORECASE),lambda match: '</head><body>')
+                          ,(re.compile(r'<byline>', re.DOTALL|re.IGNORECASE),lambda match: '<div class="byline">')
+                          ,(re.compile(r'</byline>', re.DOTALL|re.IGNORECASE),lambda match: '</div>')
+                          ,(re.compile(r'<center>', re.DOTALL|re.IGNORECASE),lambda match: '<div class="ctr">')
+                          ,(re.compile(r'</center>', re.DOTALL|re.IGNORECASE),lambda match: '</div>')
+                         ]
+
+    keep_only_tags= [
+                      dict(name='font', attrs={'class':'storyhead'})
+                     ,dict(attrs={'class':'byline'})
+                    ]
+    remove_attributes=['size','noshade','border']
+
+    def preprocess_html(self, soup):
+        for item in soup.findAll(style=True):
+            del item['style']
+        for item in soup.findAll('img'):
+            if not item.has_key('alt'):
+               item['alt'] = 'image'
+        return soup
+
+    def parse_index(self):
+        articles = []
+        soup = self.index_to_soup(self.INDEX)
+        for feed_link in soup.findAll('a',href=True):
+            if feed_link['href'].startswith('stories/'):
+                url   = self.INDEX + feed_link['href']
+                title = self.tag_to_string(feed_link)
+                date  = strftime(self.timefmt)
+                articles.append({
+                                  'title'      :title
+                                 ,'date'       :date
+                                 ,'url'        :url
+                                 ,'description':''
+                                })
+        return [('Frontline', articles)]
+
+    def print_version(self, url):
+        return "http://www.hinduonnet.com/thehindu/thscrip/print.pl?prd=fline&file=" + url.rpartition('/')[2]
+
+    def image_url_processor(self, baseurl, url):
+        return url.replace('../images/', self.INDEX + 'images/').strip()

recipes/macleans.recipe

@@ -1,239 +1,28 @@
 #!/usr/bin/env  python
+from calibre.web.feeds.news import BasicNewsRecipe
 
-__license__   = 'GPL v3'
+class AdvancedUserRecipe1308306308(BasicNewsRecipe):
-
-'''
-macleans.ca
-'''
-from calibre.web.feeds.recipes import BasicNewsRecipe
-from calibre.ebooks.BeautifulSoup import Tag
-from datetime import timedelta, date
-
-class Macleans(BasicNewsRecipe):
     title          = u'Macleans Magazine'
-    __author__     = 'Nick Redding'
     language = 'en_CA'
-    description = ('Macleans Magazine')
+    __author__ = 'sexymax15'
+    oldest_article = 30
+    max_articles_per_feed = 12
 
+    use_embedded_content = False
+
+    remove_empty_feeds = True
     no_stylesheets = True
-    timefmt = ' [%b %d]'
+    remove_javascript = True
+    remove_tags = [dict(name ='img'),dict (id='header'),{'class':'postmetadata'}]
+    remove_tags_after = {'class':'postmetadata'}
 
-    # customization notes: delete sections you are not interested in
+    feeds          = [(u'Blog Central', u'http://www2.macleans.ca/category/blog-central/feed/'),
-    # set oldest_article to the maximum number of days back from today to include articles
+ (u'Canada', u'http://www2.macleans.ca/category/canada/feed/'),
-    sectionlist = [
+(u'World', u'http://www2.macleans.ca/category/world-from-the-magazine/feed/'),
-                        ['http://www2.macleans.ca/','Front Page'],
+(u'Business', u'http://www2.macleans.ca/category/business/feed/'),
-                        ['http://www2.macleans.ca/category/canada/','Canada'],
+(u'Arts & Culture', u'http://www2.macleans.ca/category/arts-culture/feed/'),
-                        ['http://www2.macleans.ca/category/world-from-the-magazine/','World'],
+(u'Opinion', u'http://www2.macleans.ca/category/opinion/feed/'),
-                        ['http://www2.macleans.ca/category/business','Business'],
+(u'Health', u'http://www2.macleans.ca/category/health-from-the-magazine/feed/'),
-                        ['http://www2.macleans.ca/category/arts-culture/','Culture'],
+ (u'Environment', u'http://www2.macleans.ca/category/environment-from-the-magazine/feed/')]
-                        ['http://www2.macleans.ca/category/opinion','Opinion'],
+    def print_version(self, url):
-                        ['http://www2.macleans.ca/category/health-from-the-magazine/','Health'],
+        return url + 'print/'
-                        ['http://www2.macleans.ca/category/environment-from-the-magazine/','Environment'],
-                        ['http://www2.macleans.ca/category/education/','On Campus'],
-                        ['http://www2.macleans.ca/category/travel-from-the-magazine/','Travel']
-                    ]
-    oldest_article = 7
-
-    # formatting for print version of articles
-    extra_css   =   '''h2{font-family:Times,serif; font-size:large;}
-                        small {font-family:Times,serif; font-size:xx-small; list-style-type: none;}
-                    '''
-
-    # tag handling for print version of articles
-    keep_only_tags = [dict(id='tw-print')]
-    remove_tags =   [dict({'class':'postmetadata'})]
-
-
-    def preprocess_html(self,soup):
-        for img_tag in soup.findAll('img'):
-            parent_tag = img_tag.parent
-            if parent_tag.name == 'a':
-                new_tag = Tag(soup,'p')
-                new_tag.insert(0,img_tag)
-                parent_tag.replaceWith(new_tag)
-            elif parent_tag.name == 'p':
-                if not self.tag_to_string(parent_tag) == '':
-                    new_div = Tag(soup,'div')
-                    new_tag = Tag(soup,'p')
-                    new_tag.insert(0,img_tag)
-                    parent_tag.replaceWith(new_div)
-                    new_div.insert(0,new_tag)
-                    new_div.insert(1,parent_tag)
-        return soup
-
-    def parse_index(self):
-
-
-
-        articles = {}
-        key = None
-        ans = []
-
-        def parse_index_page(page_url,page_title):
-
-            def decode_date(datestr):
-                dmysplit = datestr.strip().lower().split(',')
-                mdsplit = dmysplit[1].split()
-                m = ['january','february','march','april','may','june','july','august','september','october','november','december'].index(mdsplit[0])+1
-                d = int(mdsplit[1])
-                y = int(dmysplit[2].split()[0])
-                return date(y,m,d)
-
-            def article_title(tag):
-                atag = tag.find('a',href=True)
-                if not atag:
-                    return ''
-                return self.tag_to_string(atag)
-
-            def article_url(tag):
-                atag = tag.find('a',href=True)
-                if not atag:
-                    return ''
-                return atag['href']+'print/'
-
-            def article_description(tag):
-                for p_tag in tag.findAll('p'):
-                    d = self.tag_to_string(p_tag,False)
-                    if not d == '':
-                        return d
-                return ''
-
-            def compound_h4_h3_title(tag):
-                if tag.h4:
-                    if tag.h3:
-                        return self.tag_to_string(tag.h4,False)+u'\u2014'+self.tag_to_string(tag.h3,False)
-                    else:
-                        return self.tag_to_string(tag.h4,False)
-                elif tag.h3:
-                    return self.tag_to_string(tag.h3,False)
-                else:
-                    return ''
-
-            def compound_h2_h4_title(tag):
-                if tag.h2:
-                    if tag.h4:
-                        return self.tag_to_string(tag.h2,False)+u'\u2014'+self.tag_to_string(tag.h4,False)
-                    else:
-                        return self.tag_to_string(tag.h2,False)
-                elif tag.h4:
-                    return self.tag_to_string(tag.h4,False)
-                else:
-                    return ''
-
-
-            def handle_article(header_tag, outer_tag):
-                if header_tag:
-                    url = article_url(header_tag)
-                    title = article_title(header_tag)
-                    author_date_tag = outer_tag.h4
-                    if author_date_tag:
-                        author_date = self.tag_to_string(author_date_tag,False).split(' - ')
-                        author = author_date[0].strip()
-                        article_date = decode_date(author_date[1])
-                        earliest_date = date.today() - timedelta(days=self.oldest_article)
-                        if article_date < earliest_date:
-                            self.log("Skipping article dated %s" % author_date[1])
-                        else:
-                            excerpt_div = outer_tag.find('div','excerpt')
-                            if excerpt_div:
-                                description = article_description(excerpt_div)
-                            else:
-                                description = ''
-                            if not articles.has_key(page_title):
-                                articles[page_title] = []
-                            articles[page_title].append(dict(title=title,url=url,date=author_date[1],description=description,author=author,content=''))
-
-            def handle_category_article(cat, header_tag, outer_tag):
-                url = article_url(header_tag)
-                title = article_title(header_tag)
-                if not title == '':
-                    title = cat+u'\u2014'+title
-                a_tag = outer_tag.find('span','authorLink')
-                if a_tag:
-                    author = self.tag_to_string(a_tag,False)
-                    a_tag.parent.extract()
-                else:
-                    author = ''
-                description = article_description(outer_tag)
-                if not articles.has_key(page_title):
-                    articles[page_title] = []
-                articles[page_title].append(dict(title=title,url=url,date='',description=description,author=author,content=''))
-
-
-            soup = self.index_to_soup(page_url)
-
-            if page_title == 'Front Page':
-                # special processing for the front page
-                top_stories = soup.find('div',{ "id" : "macleansFeatured" })
-                if top_stories:
-                    for div_slide in top_stories.findAll('div','slide'):
-                        url = article_url(div_slide)
-                        div_title = div_slide.find('div','header')
-                        if div_title:
-                            title = self.tag_to_string(div_title,False)
-                        else:
-                            title = ''
-                        description = article_description(div_slide)
-                        if not articles.has_key(page_title):
-                             articles[page_title] = []
-                        articles[page_title].append(dict(title=title,url=url,date='',description=description,author='',content=''))
-
-                from_macleans = soup.find('div',{ "id" : "fromMacleans" })
-                if from_macleans:
-                    for li_tag in from_macleans.findAll('li','fromMacleansArticle'):
-                        title = compound_h4_h3_title(li_tag)
-                        url = article_url(li_tag)
-                        description = article_description(li_tag)
-                        if not articles.has_key(page_title):
-                            articles[page_title] = []
-                        articles[page_title].append(dict(title=title,url=url,date='',description=description,author='',content=''))
-
-                blog_central = soup.find('div',{ "id" : "bloglist" })
-                if blog_central:
-                    for li_tag in blog_central.findAll('li'):
-                        title = compound_h2_h4_title(li_tag)
-                        if li_tag.h4:
-                            url = article_url(li_tag.h4)
-                            if not articles.has_key(page_title):
-                                articles[page_title] = []
-                            articles[page_title].append(dict(title=title,url=url,date='',description='',author='',content=''))
-
-#                need_to_know = soup.find('div',{ "id" : "needToKnow" })
-#                if need_to_know:
-#                    for div_tag in need_to_know('div',attrs={'class' : re.compile("^needToKnowArticle")}):
-#                        title = compound_h4_h3_title(div_tag)
-#                        url = article_url(div_tag)
-#                        description = article_description(div_tag)
-#                        if not articles.has_key(page_title):
-#                            articles[page_title] = []
-#                        articles[page_title].append(dict(title=title,url=url,date='',description=description,author='',content=''))
-
-                for news_category in soup.findAll('div','newsCategory'):
-                    news_cat = self.tag_to_string(news_category.h4,False)
-                    handle_category_article(news_cat, news_category.find('h2'), news_category.find('div'))
-                    for news_item in news_category.findAll('li'):
-                        handle_category_article(news_cat,news_item.h3,news_item)
-
-                return
-
-            # find the div containing the highlight article
-            div_post = soup.find('div','post')
-            if div_post:
-                h1_tag = div_post.h1
-                handle_article(h1_tag,div_post)
-
-            # find the divs containing the rest of the articles
-            div_other = div_post.find('div', { "id" : "categoryOtherPosts" })
-            if div_other:
-                for div_entry in div_other.findAll('div','entry'):
-                    h2_tag = div_entry.h2
-                    handle_article(h2_tag,div_entry)
-
-
-
-        for page_name,page_title in self.sectionlist:
-            parse_index_page(page_name,page_title)
-            ans.append(page_title)
-
-        ans = [(key, articles[key]) for key in ans if articles.has_key(key)]
-        return ans

recipes/mail_and_guardian.recipe

@@ -3,7 +3,7 @@ from calibre.web.feeds.news import BasicNewsRecipe
 class AdvancedUserRecipe1295081935(BasicNewsRecipe):
      title          = u'Mail & Guardian ZA News'
      __author__     = '77ja65'
-     language = 'en'
+     language = 'en_ZA'
      oldest_article = 7
      max_articles_per_feed = 30
      no_stylesheets = True

recipes/metro_uk.recipe

@@ -1,29 +1,34 @@
+import re
 from calibre.web.feeds.news import BasicNewsRecipe
 class AdvancedUserRecipe1306097511(BasicNewsRecipe):
     title          = u'Metro UK'
-
+    description = 'News as provide by The Metro -UK'
-    no_stylesheets = True
-    oldest_article = 1
-    max_articles_per_feed = 200
 
     __author__ = 'Dave Asbury'
+    no_stylesheets = True
+    oldest_article = 1
+    max_articles_per_feed = 25
+    remove_empty_feeds = True
+    remove_javascript     = True
+
+    preprocess_regexps = [(re.compile(r'Tweet'), lambda  a : '')]
+
     language = 'en_GB'
-    simultaneous_downloads= 3
+
 
     masthead_url        = 'http://e-edition.metro.co.uk/images/metro_logo.gif'
 
+    extra_css = 'h2 {font: sans-serif medium;}'
     keep_only_tags = [
+	dict(name='h1'),dict(name='h2', attrs={'class':'h2'}),
                     dict(attrs={'class':['img-cnt figure']}),
-    dict(attrs={'class':['art-img']}),
+    	dict(attrs={'class':['art-img']}),
-                    dict(name='h1'),
+
-    dict(name='h2', attrs={'class':'h2'}),
                     dict(name='div', attrs={'class':'art-lft'})
     ]
-    remove_tags    = [dict(name='div', attrs={'class':[ 'metroCommentFormWrap',
+    remove_tags    = [dict(name='div', attrs={'class':[ 'news m12 clrd clr-b p5t shareBtm', 'commentForm', 'metroCommentInnerWrap',
-                                                                             'commentForm', 'metroCommentInnerWrap',
+                             'art-rgt','pluck-app pluck-comm','news m12 clrd clr-l p5t', 'flt-r' ]}),
-                             'art-rgt','pluck-app pluck-comm','news m12 clrd clr-l p5t', 'flt-r' ]})]
+	dict(attrs={'class':[ 'metroCommentFormWrap','commentText','commentsNav','avatar','submDateAndTime']})
-
+	          ]
     feeds          = [
         (u'News', u'http://www.metro.co.uk/rss/news/'), (u'Money', u'http://www.metro.co.uk/rss/money/'), (u'Sport', u'http://www.metro.co.uk/rss/sport/'), (u'Film', u'http://www.metro.co.uk/rss/metrolife/film/'), (u'Music', u'http://www.metro.co.uk/rss/metrolife/music/'), (u'TV', u'http://www.metro.co.uk/rss/tv/'), (u'Showbiz', u'http://www.metro.co.uk/rss/showbiz/'), (u'Weird News', u'http://www.metro.co.uk/rss/weird/'), (u'Travel', u'http://www.metro.co.uk/rss/travel/'), (u'Lifestyle', u'http://www.metro.co.uk/rss/lifestyle/'), (u'Books', u'http://www.metro.co.uk/rss/lifestyle/books/'), (u'Food', u'http://www.metro.co.uk/rss/lifestyle/restaurants/')]
-
-

recipes/noticias_r7.recipe

@@ -0,0 +1,40 @@
+import re
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class PortalR7(BasicNewsRecipe):
+    title                  = 'Noticias R7'
+    __author__             = 'Diniz Bortolotto'
+    description            = 'Noticias Portal R7'
+    oldest_article         = 2
+    max_articles_per_feed  = 20
+    encoding               = 'utf8'
+    publisher              = 'Rede Record'
+    category               = 'news, Brazil'
+    language               = 'pt_BR'
+    publication_type       = 'newsportal'
+    use_embedded_content   = False
+    no_stylesheets         = True
+    remove_javascript      = True
+    remove_attributes      = ['style']
+
+    feeds                  = [
+                              (u'Brasil', u'http://www.r7.com/data/rss/brasil.xml'),
+                              (u'Economia', u'http://www.r7.com/data/rss/economia.xml'),
+                              (u'Internacional', u'http://www.r7.com/data/rss/internacional.xml'),
+                              (u'Tecnologia e Ci\xeancia', u'http://www.r7.com/data/rss/tecnologiaCiencia.xml')
+                             ]
+    reverse_article_order  = True
+
+    keep_only_tags         = [dict(name='div', attrs={'class':'materia'})]
+    remove_tags            = [
+                              dict(id=['espalhe', 'report-erro']),
+                              dict(name='ul', attrs={'class':'controles'}),
+                              dict(name='ul', attrs={'class':'relacionados'}),
+                              dict(name='div', attrs={'class':'materia_banner'}),
+                              dict(name='div', attrs={'class':'materia_controles'})
+                             ]
+
+    preprocess_regexps     = [
+                              (re.compile(r'<div class="materia">.*<div class="materia_cabecalho">',re.DOTALL|re.IGNORECASE),
+                              lambda match: '<div class="materia"><div class="materia_cabecalho">')
+                             ]

recipes/perfil.recipe

@@ -26,6 +26,7 @@ class Perfil(BasicNewsRecipe):
                                 .foto1 h1{font-size: x-small}
                                 h1{font-family: Georgia,"Times New Roman",serif}
                                 img{margin-bottom: 0.4em}
+                                .hora{font-size: x-small; color: red}
                             """
 
     conversion_options = {
@@ -60,7 +61,26 @@ class Perfil(BasicNewsRecipe):
              ,(u'Tecnologia'     , u'http://www.perfil.com/rss/tecnologia.xml'   )
             ]
 
+    def get_article_url(self, article):
+        return article.get('guid',  None)
+            
     def preprocess_html(self, soup):
         for item in soup.findAll(style=True):
             del item['style']
+        for item in soup.findAll('a'):
+            limg = item.find('img')
+            if item.string is not None:
+               str = item.string
+               item.replaceWith(str)
+            else:
+               if limg:
+                  item.name = 'div'
+                  item.attrs = []
+               else:
+                   str = self.tag_to_string(item)
+                   item.replaceWith(str)
+        for item in soup.findAll('img'):
+            if not item.has_key('alt'):
+               item['alt'] = 'image'
         return soup
+        

recipes/philly.recipe

@@ -1,85 +1,45 @@
 #!/usr/bin/env  python
-__license__   = 'GPL v3'
+from calibre.web.feeds.news import BasicNewsRecipe
-'''
-philly.com/inquirer/
-'''
-from calibre.web.feeds.recipes import BasicNewsRecipe
 
-class Philly(BasicNewsRecipe):
+class AdvancedUserRecipe1308312288(BasicNewsRecipe):
-
+    title          = u'Philadelphia Inquirer'
-    title       = 'Philadelphia Inquirer'
+    __author__ = 'sexymax15'
-    __author__  = 'RadikalDissent and Sujata Raman'
     language = 'en'
     description = 'Daily news from the Philadelphia Inquirer'
-    no_stylesheets        = True
+    oldest_article = 15
-    use_embedded_content  = False
+    max_articles_per_feed = 20
-    oldest_article = 1
+    use_embedded_content = False
-    max_articles_per_feed = 25
+    remove_empty_feeds = True
+    no_stylesheets = True
+    remove_javascript = True
 
-    extra_css = '''
+   # remove_tags_before = {'class':'article_timestamp'}
-        h1{font-family:verdana,arial,helvetica,sans-serif; font-size: large;}
+    #remove_tags_after = {'class':'graylabel'}
-        h2{font-family:verdana,arial,helvetica,sans-serif; font-size: small;}
+    keep_only_tags= [dict(name=['h1','p'])]
-        .body-content{font-family:verdana,arial,helvetica,sans-serif; font-size: small;}
+    remove_tags = [dict(name=['hr','dl','dt','img','meta','iframe','link','script','form','input','label']),
-        .byline {font-size: small; color: #666666; font-style:italic; }
+dict(id=['toggleConfirmEmailDiv','toggleTOS','toggleUsernameMsgDiv','toggleConfirmYear','navT1_philly','secondaryNav','navPlacement','globalPrimaryNav'
-        .lastline {font-size: small; color: #666666; font-style:italic;}
+,'ugc-footer-philly','bv_footer_include','footer','header',
-        .contact {font-size: small; color: #666666;}
+'container_rag_bottom','section_rectangle','contentrightside'])
-        .contact p {font-size: small; color: #666666;}
+,{'class':['megamenu3 megamenu','container misc','container_inner misc_inner'
-        #photoCaption { font-family:verdana,arial,helvetica,sans-serif; font-size:x-small;}
+,'misccontainer_left_32','headlineonly','misccontainer_middle_32'
-        .photoCaption { font-family:verdana,arial,helvetica,sans-serif; font-size:x-small;}
+,'misccontainer_right_32','headline formBegin',
-        #photoCredit{ font-family:verdana,arial,helvetica,sans-serif; font-size:x-small; color:#666666;}
+'post_balloon','relatedlist','linkssubhead','b_sq','dotted-rule-above'
-        .photoCredit{ font-family:verdana,arial,helvetica,sans-serif; font-size:x-small; color:#666666;}
+,'container','headlines-digest','graylabel','container_inner'
-        .article_timestamp{font-size:x-small; color:#666666;}
+,'rlinks_colorbar1','rlinks_colorbar2','supercontainer','container_5col_left','container_image_left',
-        a {font-family:verdana,arial,helvetica,sans-serif; font-size: x-small;}
+'digest-headline2','digest-lead','container_5col_leftmiddle',
-                '''
+'container_5col_middlemiddle','container_5col_rightmiddle'
+,'container_5col_right','divclear','supercontainer_outer force-width',
+'supercontainer','containertitle  kicker-title',
+'pollquestion','pollchoice','photomore','pollbutton','container rssbox','containertitle video ',
+'containertitle_image ','container_tabtwo','selected'
+,'shadetabs','selected','tabcontentstyle','tabcontent','inner_container'
+,'arrow','container_ad','containertitlespacer','adUnit','tracking','sitemsg_911 clearfix']}]
 
-    keep_only_tags = [
+    extra_css             = """
-               dict(name='div', attrs={'class':'story-content'}),
+                               h1{font-family: Georgia,serif; font-size: xx-large}
-               dict(name='div', attrs={'id': 'contentinside'})
-                    ]
 
-    remove_tags = [
+                            """
-         dict(name='div', attrs={'class':['linkssubhead','post_balloon','relatedlist','pollquestion','b_sq']}),
-         dict(name='dl', attrs={'class':'relatedlist'}),
-        dict(name='div', attrs={'id':['photoNav','sidebar_adholder']}),
-        dict(name='a', attrs={'class': ['headlineonly','bl']}),
-         dict(name='img', attrs={'class':'img_noborder'})
-    ]
-   # def print_version(self, url):
-   #     return url + '?viewAll=y'
 
 
-    feeds = [
+    feeds          = [(u'News', u'http://www.philly.com/philly_news.rss')]
-        ('Front Page', 'http://www.philly.com/inquirer_front_page.rss'),
-        ('Business', 'http://www.philly.com/inq_business.rss'),
-        #('News', 'http://www.philly.com/inquirer/news/index.rss'),
-        ('Nation', 'http://www.philly.com/inq_news_world_us.rss'),
-        ('Local', 'http://www.philly.com/inquirer_local.rss'),
-        ('Health', 'http://www.philly.com/inquirer_health_science.rss'),
-        ('Education', 'http://www.philly.com/inquirer_education.rss'),
-        ('Editorial and opinion', 'http://www.philly.com/inq_news_editorial.rss'),
-        ('Sports', 'http://www.philly.com/inquirer_sports.rss')
-        ]
 
-    def get_article_url(self, article):
-        ans = article.link
-
-        try:
-            self.log('Looking for full story link in', ans)
-            soup = self.index_to_soup(ans)
-            x = soup.find(text="View All")
-
-            if x is not None:
-                ans = ans + '?viewAll=y'
-                self.log('Found full story link', ans)
-        except:
-            pass
-        return ans
-
-    def postprocess_html(self, soup,first):
-
-         for tag in soup.findAll(name='div',attrs={'class':"container_ate_qandatitle"}):
-                tag.extract()
-         for tag in soup.findAll(name='br'):
-                tag.extract()
-
-         return soup

recipes/words_without_borders.recipe

@@ -0,0 +1,25 @@
+#recipe created by sexymax15.....sexymax15@gmail.com
+#Words without Borders recipe
+
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class AdvancedUserRecipe1308302002(BasicNewsRecipe):
+    title          = u'Words Without Borders'
+    language = 'en'
+    __author__ = 'sexymax15'
+    oldest_article = 90
+    max_articles_per_feed = 30
+    use_embedded_content = False
+
+    remove_empty_feeds = True
+    no_stylesheets = True
+    remove_javascript = True
+    keep_only_tags = {'class':'span-14 article'}
+    remove_tags_after = [{'class':'addthis_toolbox addthis_default_style no_print'}]
+    remove_tags = [{'class':['posterous_quote_citation','button']}]
+    extra_css             = """
+                               h1{font-family: Georgia,serif; font-size: large}h2{font-family: Georgia,serif; font-size: large} """
+
+
+
+    feeds          = [(u'wwb', u'http://feeds.feedburner.com/wwborders?format=xml')]

recipes/wsj.recipe

@@ -51,7 +51,7 @@ class WallStreetJournal(BasicNewsRecipe):
             br['password'] = self.password
             res = br.submit()
             raw = res.read()
-            if 'Welcome,' not in raw:
+            if 'Welcome,' not in raw and '>Logout<' not in raw:
                 raise ValueError('Failed to log in to wsj.com, check your '
                         'username and password')
         return br

resources/content_server/read/monocle.js

@@ -1,5 +1,5 @@
 Monocle = {
-  VERSION: "1.0.0"
+  VERSION: "2.0.0"
 };
 
 
@@ -170,7 +170,8 @@ Monocle.Browser.has.iframeTouchBug = Monocle.Browser.iOSVersionBelow("4.2");
 Monocle.Browser.has.selectThruBug = Monocle.Browser.iOSVersionBelow("4.2");
 
 Monocle.Browser.has.mustScrollSheaf = Monocle.Browser.is.MobileSafari;
-Monocle.Browser.has.iframeDoubleWidthBug = Monocle.Browser.has.mustScrollSheaf;
+Monocle.Browser.has.iframeDoubleWidthBug =
+  Monocle.Browser.has.mustScrollSheaf || Monocle.Browser.on.Kindle3;
 
 Monocle.Browser.has.floatColumnBug = Monocle.Browser.is.WebKit;
 
@@ -181,6 +182,11 @@ Monocle.Browser.has.jumpFlickerBug =
   Monocle.Browser.on.MacOSX && Monocle.Browser.is.WebKit;
 
 
+Monocle.Browser.has.columnOverflowPaintBug = Monocle.Browser.is.WebKit &&
+  !Monocle.Browser.is.MobileSafari &&
+  navigator.userAgent.indexOf("AppleWebKit/534") > 0;
+
+
 if (typeof window.console == "undefined") {
   window.console = {
     messages: [],
@@ -241,6 +247,7 @@ Monocle.Factory = function (element, label, index, reader) {
 
 
   function initialize() {
+    if (!p.label) { return; }
     var node = p.reader.properties.graph;
     node[p.label] = node[p.label] || [];
     if (typeof p.index == 'undefined' && node[p.label][p.index]) {
@@ -274,7 +281,11 @@ Monocle.Factory = function (element, label, index, reader) {
 
   function make(tagName, oLabel, index_or_options, or_options) {
     var oIndex, options;
-    if (arguments.length == 2) {
+    if (arguments.length == 1) {
+      oLabel = null,
+      oIndex = 0;
+      options = {};
+    } else if (arguments.length == 2) {
       oIndex = 0;
       options = {};
     } else if (arguments.length == 4) {
@@ -376,6 +387,22 @@ Monocle.pieceLoaded('factory');
 Monocle.Events = {}
 
 
+Monocle.Events.dispatch = function (elem, evtType, data, cancelable) {
+  if (!document.createEvent) {
+    return true;
+  }
+  var evt = document.createEvent("Events");
+  evt.initEvent(evtType, false, cancelable || false);
+  evt.m = data;
+  try {
+    return elem.dispatchEvent(evt);
+  } catch(e) {
+    console.warn("Failed to dispatch event: "+evtType);
+    return false;
+  }
+}
+
+
 Monocle.Events.listen = function (elem, evtType, fn, useCapture) {
   if (elem.addEventListener) {
     return elem.addEventListener(evtType, fn, useCapture || false);
@@ -405,7 +432,7 @@ Monocle.Events.listenForContact = function (elem, fns, options) {
       pageY: ci.pageY
     };
 
-    var target = evt.target || window.srcElement;
+    var target = evt.target || evt.srcElement;
     while (target.nodeType != 1 && target.parentNode) {
       target = target.parentNode;
     }
@@ -527,13 +554,18 @@ Monocle.Events.deafenForContact = function (elem, listeners) {
 }
 
 
-Monocle.Events.listenForTap = function (elem, fn) {
+Monocle.Events.listenForTap = function (elem, fn, activeClass) {
   var startPos;
 
   if (Monocle.Browser.on.Kindle3) {
     Monocle.Events.listen(elem, 'click', function () {});
   }
 
+  var annul = function () {
+    startPos = null;
+    if (activeClass && elem.dom) { elem.dom.removeClass(activeClass); }
+  }
+
   var annulIfOutOfBounds = function (evt) {
     if (evt.type.match(/^mouse/)) {
       return;
@@ -545,7 +577,7 @@ Monocle.Events.listenForTap = function (elem, fn) {
       evt.m.registrantX < 0 || evt.m.registrantX > elem.offsetWidth ||
       evt.m.registrantY < 0 || evt.m.registrantY > elem.offsetHeight
     ) {
-      startPos = null;
+      annul();
     } else {
       evt.preventDefault();
     }
@@ -557,6 +589,7 @@ Monocle.Events.listenForTap = function (elem, fn) {
       start: function (evt) {
         startPos = [evt.m.pageX, evt.m.pageY];
         evt.preventDefault();
+        if (activeClass && elem.dom) { elem.dom.addClass(activeClass); }
       },
       move: annulIfOutOfBounds,
       end: function (evt) {
@@ -565,10 +598,9 @@ Monocle.Events.listenForTap = function (elem, fn) {
           evt.m.startOffset = startPos;
           fn(evt);
         }
+        annul();
       },
-      cancel: function (evt) {
+      cancel: annul
-        startPos = null;
-      }
     },
     {
       useCapture: false
@@ -997,6 +1029,9 @@ Monocle.Reader = function (node, bookData, options, onLoadCallback) {
     createReaderElements();
 
     p.defaultStyles = addPageStyles(k.DEFAULT_STYLE_RULES, false);
+    if (options.stylesheet) {
+      p.initialStyles = addPageStyles(options.stylesheet, false);
+    }
 
     primeFrames(options.primeURL, function () {
       applyStyles();
@@ -1077,6 +1112,7 @@ Monocle.Reader = function (node, bookData, options, onLoadCallback) {
       if (Monocle.Browser.is.WebKit) {
         frame.contentDocument.documentElement.style.overflow = "hidden";
       }
+      dispatchEvent('monocle:frameprimed', { frame: frame, pageIndex: pageCount });
       if ((pageCount += 1) == pageMax) {
         Monocle.defer(callback);
       }
@@ -1131,6 +1167,7 @@ Monocle.Reader = function (node, bookData, options, onLoadCallback) {
     var pageCount = 0;
     if (typeof callback == 'function') {
       var watcher = function (evt) {
+        dispatchEvent('monocle:firstcomponentchange', evt.m);
         if ((pageCount += 1) == p.flipper.pageCount) {
           deafen('monocle:componentchange', watcher);
           callback();
@@ -1239,7 +1276,7 @@ Monocle.Reader = function (node, bookData, options, onLoadCallback) {
         page.appendChild(runner);
         ctrlData.elements.push(runner);
       }
-    } else if (cType == "modal" || cType == "popover") {
+    } else if (cType == "modal" || cType == "popover" || cType == "hud") {
       ctrlElem = ctrl.createControlElements(overlay);
       overlay.appendChild(ctrlElem);
       ctrlData.elements.push(ctrlElem);
@@ -1312,24 +1349,33 @@ Monocle.Reader = function (node, bookData, options, onLoadCallback) {
     var controlData = dataForControl(ctrl);
     if (!controlData) {
       console.warn("No data for control: " + ctrl);
-      return;
+      return false;
     }
-    if (controlData.hidden == false) {
+
-      return;
+    if (showingControl(ctrl)) {
+      return false;
     }
+
+    var overlay = dom.find('overlay');
+    if (controlData.usesOverlay && controlData.controlType != "hud") {
+      for (var i = 0, ii = p.controls.length; i < ii; ++i) {
+        if (p.controls[i].usesOverlay && !p.controls[i].hidden) {
+          return false;
+        }
+      }
+      overlay.style.display = "block";
+    }
+
     for (var i = 0; i < controlData.elements.length; ++i) {
       controlData.elements[i].style.display = "block";
     }
-    var overlay = dom.find('overlay');
+
-    if (controlData.usesOverlay) {
-      overlay.style.display = "block";
-    }
     if (controlData.controlType == "popover") {
       overlay.listeners = Monocle.Events.listenForContact(
         overlay,
         {
           start: function (evt) {
-            obj = evt.target || window.event.srcElement;
+            var obj = evt.target || window.event.srcElement;
             do {
               if (obj == controlData.elements[0]) { return true; }
             } while (obj && (obj = obj.parentNode));
@@ -1346,22 +1392,18 @@ Monocle.Reader = function (node, bookData, options, onLoadCallback) {
       ctrl.properties.hidden = false;
     }
     dispatchEvent('controlshow', ctrl, false);
+    return true;
+  }
+
+
+  function showingControl(ctrl) {
+    var controlData = dataForControl(ctrl);
+    return controlData.hidden == false;
   }
 
 
   function dispatchEvent(evtType, data, cancelable) {
-    if (!document.createEvent) {
+    return Monocle.Events.dispatch(dom.find('box'), evtType, data, cancelable);
-      return true;
-    }
-    var evt = document.createEvent("Events");
-    evt.initEvent(evtType, false, cancelable || false);
-    evt.m = data;
-    try {
-      return dom.find('box').dispatchEvent(evt);
-    } catch(e) {
-      console.warn("Failed to dispatch event: " + evtType);
-      return false;
-    }
   }
 
 
@@ -1502,6 +1544,7 @@ Monocle.Reader = function (node, bookData, options, onLoadCallback) {
   API.addControl = addControl;
   API.hideControl = hideControl;
   API.showControl = showControl;
+  API.showingControl = showingControl;
   API.dispatchEvent = dispatchEvent;
   API.listen = listen;
   API.deafen = deafen;
@@ -1527,22 +1570,32 @@ Monocle.Reader.DEFAULT_CLASS_PREFIX = 'monelem_'
 Monocle.Reader.FLIPPER_DEFAULT_CLASS = "Slider";
 Monocle.Reader.FLIPPER_LEGACY_CLASS = "Legacy";
 Monocle.Reader.DEFAULT_STYLE_RULES = [
-  "html * {" +
+  "html#RS\\:monocle * {" +
+    "-webkit-font-smoothing: subpixel-antialiased;" +
     "text-rendering: auto !important;" +
     "word-wrap: break-word !important;" +
+    "overflow: visible !important;" +
     (Monocle.Browser.has.floatColumnBug ? "float: none !important;" : "") +
-  "}" +
+  "}",
-  "body {" +
+  "html#RS\\:monocle body {" +
     "margin: 0 !important;" +
     "padding: 0 !important;" +
     "-webkit-text-size-adjust: none;" +
-  "}" +
+  "}",
-  "table, img {" +
+  "html#RS\\:monocle body * {" +
     "max-width: 100% !important;" +
-    "max-height: 90% !important;" +
+  "}",
+  "html#RS\\:monocle img, html#RS\\:monocle video, html#RS\\:monocle object {" +
+    "max-height: 95% !important;" +
   "}"
 ]
 
+if (Monocle.Browser.has.columnOverflowPaintBug) {
+  Monocle.Reader.DEFAULT_STYLE_RULES.push(
+    "::-webkit-scrollbar { width: 0; height: 0; }"
+  )
+}
+
 
 Monocle.pieceLoaded('reader');
 /* BOOK */
@@ -1586,6 +1639,16 @@ Monocle.Book = function (dataSource) {
       locus.load = true;
       locus.componentId = p.componentIds[0];
       return locus;
+    } else if (
+      cIndex < 0 &&
+      locus.componentId &&
+      currComponent.properties.id != locus.componentId
+    ) {
+      pageDiv.m.reader.dispatchEvent(
+        "monocle:notfound",
+        { href: locus.componentId }
+      );
+      return null;
     } else if (cIndex < 0) {
       component = currComponent;
       locus.componentId = pageDiv.m.activeFrame.m.component.properties.id;
@@ -1619,6 +1682,8 @@ Monocle.Book = function (dataSource) {
       result.page += locus.direction;
     } else if (typeof(locus.anchor) == "string") {
       result.page = component.pageForChapter(locus.anchor, pageDiv);
+    } else if (typeof(locus.xpath) == "string") {
+      result.page = component.pageForXPath(locus.xpath, pageDiv);
     } else if (typeof(locus.position) == "string") {
       if (locus.position == "start") {
         result.page = 1;
@@ -1638,6 +1703,7 @@ Monocle.Book = function (dataSource) {
     if (result.page < 1) {
       if (cIndex == 0) {
         result.page = 1;
+        result.boundarystart = true;
       } else {
         result.load = true;
         result.componentId = p.componentIds[cIndex - 1];
@@ -1647,6 +1713,7 @@ Monocle.Book = function (dataSource) {
     } else if (result.page > lastPageNum['new']) {
       if (cIndex == p.lastCIndex) {
         result.page = lastPageNum['new'];
+        result.boundaryend = true;
       } else {
         result.load = true;
         result.componentId = p.componentIds[cIndex + 1];
@@ -1660,18 +1727,25 @@ Monocle.Book = function (dataSource) {
 
   function setPageAt(pageDiv, locus) {
     locus = pageNumberAt(pageDiv, locus);
-    if (!locus.load) {
+    if (locus && !locus.load) {
-      var component = p.components[p.componentIds.indexOf(locus.componentId)];
+      var evtData = { locus: locus, page: pageDiv }
-      pageDiv.m.place = pageDiv.m.place || new Monocle.Place();
+      if (locus.boundarystart) {
-      pageDiv.m.place.setPlace(component, locus.page);
+        pageDiv.m.reader.dispatchEvent('monocle:boundarystart', evtData);
+      } else if (locus.boundaryend) {
+        pageDiv.m.reader.dispatchEvent('monocle:boundaryend', evtData);
+      } else {
+        var component = p.components[p.componentIds.indexOf(locus.componentId)];
+        pageDiv.m.place = pageDiv.m.place || new Monocle.Place();
+        pageDiv.m.place.setPlace(component, locus.page);
 
-      var evtData = {
+        var evtData = {
-        page: pageDiv,
+          page: pageDiv,
-        locus: locus,
+          locus: locus,
-        pageNumber: pageDiv.m.place.pageNumber(),
+          pageNumber: pageDiv.m.place.pageNumber(),
-        componentId: locus.componentId
+          componentId: locus.componentId
+        }
+        pageDiv.m.reader.dispatchEvent("monocle:pagechange", evtData);
       }
-      pageDiv.m.reader.dispatchEvent("monocle:pagechange", evtData);
     }
     return locus;
   }
@@ -1683,6 +1757,10 @@ Monocle.Book = function (dataSource) {
       locus = pageNumberAt(pageDiv, locus);
     }
 
+    if (!locus) {
+      return;
+    }
+
     if (!locus.load) {
       callback(locus);
       return;
@@ -1690,7 +1768,9 @@ Monocle.Book = function (dataSource) {
 
     var findPageNumber = function () {
       locus = setPageAt(pageDiv, locus);
-      if (locus.load) {
+      if (!locus) {
+        return;
+      } else if (locus.load) {
         loadPageAt(pageDiv, locus, callback, progressCallback)
       } else {
         callback(locus);
@@ -1715,10 +1795,12 @@ Monocle.Book = function (dataSource) {
   }
 
 
-  function setOrLoadPageAt(pageDiv, locus, callback, progressCallback) {
+  function setOrLoadPageAt(pageDiv, locus, callback, onProgress, onFail) {
     locus = setPageAt(pageDiv, locus);
-    if (locus.load) {
+    if (!locus) {
-      loadPageAt(pageDiv, locus, callback, progressCallback);
+      if (onFail) { onFail(); }
+    } else if (locus.load) {
+      loadPageAt(pageDiv, locus, callback, onProgress);
     } else {
       callback(locus);
     }
@@ -1864,13 +1946,18 @@ Monocle.Place = function () {
   }
 
 
-  function percentageThrough() {
+  function percentAtTopOfPage() {
+    return p.percent - 1.0 / p.component.lastPageNumber();
+  }
+
+
+  function percentAtBottomOfPage() {
     return p.percent;
   }
 
 
-  function pageAtPercentageThrough(pc) {
+  function pageAtPercentageThrough(percent) {
-    return Math.max(Math.round(p.component.lastPageNumber() * pc), 1);
+    return Math.max(Math.round(p.component.lastPageNumber() * percent), 1);
   }
 
 
@@ -1911,6 +1998,8 @@ Monocle.Place = function () {
     }
     if (options.direction) {
       locus.page += options.direction;
+    } else {
+      locus.percent = percentAtBottomOfPage();
     }
     return locus;
   }
@@ -1942,7 +2031,9 @@ Monocle.Place = function () {
   API.setPlace = setPlace;
   API.setPercentageThrough = setPercentageThrough;
   API.componentId = componentId;
-  API.percentageThrough = percentageThrough;
+  API.percentAtTopOfPage = percentAtTopOfPage;
+  API.percentAtBottomOfPage = percentAtBottomOfPage;
+  API.percentageThrough = percentAtBottomOfPage;
   API.pageAtPercentageThrough = pageAtPercentageThrough;
   API.pageNumber = pageNumber;
   API.chapterInfo = chapterInfo;
@@ -2158,11 +2249,13 @@ Monocle.Component = function (book, id, index, chapters, source) {
     if (p.chapters[0] && typeof p.chapters[0].percent == "number") {
       return;
     }
+    var doc = pageDiv.m.activeFrame.contentDocument;
     for (var i = 0; i < p.chapters.length; ++i) {
       var chp = p.chapters[i];
       chp.percent = 0;
       if (chp.fragment) {
-        chp.percent = pageDiv.m.dimensions.percentageThroughOfId(chp.fragment);
+        var node = doc.getElementById(chp.fragment);
+        chp.percent = pageDiv.m.dimensions.percentageThroughOfNode(node);
       }
     }
     return p.chapters;
@@ -2187,14 +2280,37 @@ Monocle.Component = function (book, id, index, chapters, source) {
     if (!fragment) {
       return 1;
     }
-    var pc2pn = function (pc) { return Math.floor(pc * p.pageLength) + 1 }
     for (var i = 0; i < p.chapters.length; ++i) {
       if (p.chapters[i].fragment == fragment) {
-        return pc2pn(p.chapters[i].percent);
+        return percentToPageNumber(p.chapters[i].percent);
       }
     }
-    var percent = pageDiv.m.dimensions.percentageThroughOfId(fragment);
+    var doc = pageDiv.m.activeFrame.contentDocument;
-    return pc2pn(percent);
+    var node = doc.getElementById(fragment);
+    var percent = pageDiv.m.dimensions.percentageThroughOfNode(node);
+    return percentToPageNumber(percent);
+  }
+
+
+  function pageForXPath(xpath, pageDiv) {
+    var doc = pageDiv.m.activeFrame.contentDocument;
+    var percent = 0;
+    if (typeof doc.evaluate == "function") {
+      var node = doc.evaluate(
+        xpath,
+        doc,
+        null,
+        9,
+        null
+      ).singleNodeValue;
+      var percent = pageDiv.m.dimensions.percentageThroughOfNode(node);
+    }
+    return percentToPageNumber(percent);
+  }
+
+
+  function percentToPageNumber(pc) {
+    return Math.floor(pc * p.pageLength) + 1;
   }
 
 
@@ -2207,6 +2323,7 @@ Monocle.Component = function (book, id, index, chapters, source) {
   API.updateDimensions = updateDimensions;
   API.chapterForPage = chapterForPage;
   API.pageForChapter = pageForChapter;
+  API.pageForXPath = pageForXPath;
   API.lastPageNumber = lastPageNumber;
 
   return API;
@@ -2415,9 +2532,11 @@ Monocle.Dimensions.Vert = function (pageDiv) {
   }
 
 
-  function percentageThroughOfId(id) {
+  function percentageThroughOfNode(target) {
+    if (!target) {
+      return 0;
+    }
     var doc = p.page.m.activeFrame.contentDocument;
-    var target = doc.getElementById(id);
     var offset = 0;
     if (target.getBoundingClientRect) {
       offset = target.getBoundingClientRect().top;
@@ -2456,7 +2575,7 @@ Monocle.Dimensions.Vert = function (pageDiv) {
   API.hasChanged = hasChanged;
   API.measure = measure;
   API.pages = pages;
-  API.percentageThroughOfId = percentageThroughOfId;
+  API.percentageThroughOfNode = percentageThroughOfNode;
   API.locusToOffset = locusToOffset;
 
   initialize();
@@ -2713,8 +2832,7 @@ Monocle.Dimensions.Columns = function (pageDiv) {
       (!p.measurements) ||
       (p.measurements.width != newMeasurements.width) ||
       (p.measurements.height != newMeasurements.height) ||
-      (p.measurements.scrollWidth != newMeasurements.scrollWidth) ||
+      (p.measurements.scrollWidth != newMeasurements.scrollWidth)
-      (p.measurements.fontSize != newMeasurements.fontSize)
     );
   }
 
@@ -2736,10 +2854,16 @@ Monocle.Dimensions.Columns = function (pageDiv) {
       if (!lc || !lc.getBoundingClientRect) {
         console.warn('Empty document for page['+p.page.m.pageIndex+']');
         p.measurements.scrollWidth = p.measurements.width;
-      } else if (lc.getBoundingClientRect().bottom > p.measurements.height) {
-        p.measurements.scrollWidth = p.measurements.width * 2;
       } else {
-        p.measurements.scrollWidth = p.measurements.width;
+        var bcr = lc.getBoundingClientRect();
+        if (
+          bcr.right > p.measurements.width ||
+          bcr.bottom > p.measurements.height
+        ) {
+          p.measurements.scrollWidth = p.measurements.width * 2;
+        } else {
+          p.measurements.scrollWidth = p.measurements.width;
+        }
       }
     }
 
@@ -2758,12 +2882,11 @@ Monocle.Dimensions.Columns = function (pageDiv) {
   }
 
 
-  function percentageThroughOfId(id) {
+  function percentageThroughOfNode(target) {
-    var doc = p.page.m.activeFrame.contentDocument;
-    var target = doc.getElementById(id);
     if (!target) {
       return 0;
     }
+    var doc = p.page.m.activeFrame.contentDocument;
     var offset = 0;
     if (target.getBoundingClientRect) {
       offset = target.getBoundingClientRect().left;
@@ -2785,20 +2908,30 @@ Monocle.Dimensions.Columns = function (pageDiv) {
   function componentChanged(evt) {
     if (evt.m['page'] != p.page) { return; }
     var doc = evt.m['document'];
-    Monocle.Styles.applyRules(doc.body, k.BODY_STYLES);
+    if (Monocle.Browser.has.columnOverflowPaintBug) {
+      var div = doc.createElement('div');
+      Monocle.Styles.applyRules(div, k.BODY_STYLES);
+      div.style.cssText += "overflow: scroll !important;";
+      while (doc.body.childNodes.length) {
+        div.appendChild(doc.body.firstChild);
+      }
+      doc.body.appendChild(div);
+    } else {
+      Monocle.Styles.applyRules(doc.body, k.BODY_STYLES);
 
-    if (Monocle.Browser.is.WebKit) {
+      if (Monocle.Browser.is.WebKit) {
-      doc.documentElement.style.overflow = 'hidden';
+        doc.documentElement.style.overflow = 'hidden';
+      }
     }
+
     p.dirty = true;
   }
 
 
   function setColumnWidth() {
     var cw = p.page.m.sheafDiv.clientWidth;
-    var doc = p.page.m.activeFrame.contentDocument;
     if (currBodyStyleValue('column-width') != cw+"px") {
-      Monocle.Styles.affix(doc.body, 'column-width', cw+"px");
+      Monocle.Styles.affix(columnedElement(), 'column-width', cw+"px");
       p.dirty = true;
     }
   }
@@ -2809,8 +2942,7 @@ Monocle.Dimensions.Columns = function (pageDiv) {
     return {
       width: sheaf.clientWidth,
       height: sheaf.clientHeight,
-      scrollWidth: scrollerWidth(),
+      scrollWidth: scrollerWidth()
-      fontSize: currBodyStyleValue('font-size')
     }
   }
 
@@ -2819,16 +2951,24 @@ Monocle.Dimensions.Columns = function (pageDiv) {
     if (Monocle.Browser.has.mustScrollSheaf) {
       return p.page.m.sheafDiv;
     } else {
-      return p.page.m.activeFrame.contentDocument.body;
+      return columnedElement();
     }
   }
 
 
+  function columnedElement() {
+    var elem = p.page.m.activeFrame.contentDocument.body;
+    return Monocle.Browser.has.columnOverflowPaintBug ? elem.firstChild : elem;
+  }
+
+
   function scrollerWidth() {
     var bdy = p.page.m.activeFrame.contentDocument.body;
     if (Monocle.Browser.has.iframeDoubleWidthBug) {
-      if (Monocle.Browser.on.Android) {
+      if (Monocle.Browser.on.Kindle3) {
-        return bdy.scrollWidth * 1.5; // I actually have no idea why 1.5.
+        return scrollerElement().scrollWidth;
+      } else if (Monocle.Browser.on.Android) {
+        return bdy.scrollWidth;
       } else if (Monocle.Browser.iOSVersion < "4.1") {
         var hbw = bdy.scrollWidth / 2;
         var sew = scrollerElement().scrollWidth;
@@ -2838,15 +2978,18 @@ Monocle.Dimensions.Columns = function (pageDiv) {
         var hbw = bdy.scrollWidth / 2;
         return hbw;
       }
-    } else if (Monocle.Browser.is.Gecko) {
+    } else if (bdy.getBoundingClientRect) {
-      var lc = bdy.lastChild;
+      var elems = bdy.getElementsByTagName('*');
-      while (lc && lc.nodeType != 1) {
+      var bdyRect = bdy.getBoundingClientRect();
-        lc = lc.previousSibling;
+      var l = bdyRect.left, r = bdyRect.right;
-      }
+      for (var i = elems.length - 1; i >= 0; --i) {
-      if (lc && lc.getBoundingClientRect) {
+        var rect = elems[i].getBoundingClientRect();
-        return lc.getBoundingClientRect().right;
+        l = Math.min(l, rect.left);
+        r = Math.max(r, rect.right);
       }
+      return Math.abs(l) + Math.abs(r);
     }
+
     return scrollerElement().scrollWidth;
   }
 
@@ -2867,8 +3010,14 @@ Monocle.Dimensions.Columns = function (pageDiv) {
 
   function translateToLocus(locus) {
     var offset = locusToOffset(locus);
-    var bdy = p.page.m.activeFrame.contentDocument.body;
+    p.page.m.offset = 0 - offset;
-    Monocle.Styles.affix(bdy, "transform", "translateX("+offset+"px)");
+    if (k.SETX && !Monocle.Browser.has.columnOverflowPaintBug) {
+      var bdy = p.page.m.activeFrame.contentDocument.body;
+      Monocle.Styles.affix(bdy, "transform", "translateX("+offset+"px)");
+    } else {
+      var scrElem = scrollerElement();
+      scrElem.scrollLeft = 0 - offset;
+    }
     return offset;
   }
 
@@ -2876,7 +3025,7 @@ Monocle.Dimensions.Columns = function (pageDiv) {
   API.hasChanged = hasChanged;
   API.measure = measure;
   API.pages = pages;
-  API.percentageThroughOfId = percentageThroughOfId;
+  API.percentageThroughOfNode = percentageThroughOfNode;
 
   API.locusToOffset = locusToOffset;
   API.translateToLocus = translateToLocus;
@@ -2898,6 +3047,8 @@ Monocle.Dimensions.Columns.BODY_STYLES = {
   "column-fill": "auto"
 }
 
+Monocle.Dimensions.Columns.SETX = true; // Set to false for scrollLeft.
+
 if (Monocle.Browser.has.iframeDoubleWidthBug) {
   Monocle.Dimensions.Columns.BODY_STYLES["min-width"] = "200%";
 } else {
@@ -2924,6 +3075,8 @@ Monocle.Flippers.Slider = function (reader) {
 
   function addPage(pageDiv) {
     pageDiv.m.dimensions = new Monocle.Dimensions.Columns(pageDiv);
+
+    Monocle.Styles.setX(pageDiv, "0px");
   }
 
 
@@ -2963,6 +3116,7 @@ Monocle.Flippers.Slider = function (reader) {
 
 
   function interactiveMode(bState) {
+    p.reader.dispatchEvent('monocle:interactive:'+(bState ? 'on' : 'off'));
     if (!Monocle.Browser.has.selectThruBug) {
       return;
     }
@@ -2994,10 +3148,10 @@ Monocle.Flippers.Slider = function (reader) {
 
   function moveTo(locus, callback) {
     var fn = function () {
-      prepareNextPage(announceTurn);
+      prepareNextPage(function () {
-      if (typeof callback == "function") {
+        if (typeof callback == "function") { callback(); }
-        callback();
+        announceTurn();
-      }
+      });
     }
     setPage(upperPage(), locus, fn);
   }
@@ -3045,12 +3199,26 @@ Monocle.Flippers.Slider = function (reader) {
 
     if (dir == k.FORWARDS) {
       if (getPlace().onLastPageOfBook()) {
+        p.reader.dispatchEvent(
+          'monocle:boundaryend',
+          {
+            locus: getPlace().getLocus({ direction : dir }),
+            page: upperPage()
+          }
+        );
         resetTurnData();
         return;
       }
       onGoingForward(boxPointX);
     } else if (dir == k.BACKWARDS) {
       if (getPlace().onFirstPageOfBook()) {
+        p.reader.dispatchEvent(
+          'monocle:boundarystart',
+          {
+            locus: getPlace().getLocus({ direction : dir }),
+            page: upperPage()
+          }
+        );
         resetTurnData();
         return;
       }
@@ -3215,14 +3383,14 @@ Monocle.Flippers.Slider = function (reader) {
 
 
   function announceTurn() {
-    hideWaitControl(upperPage());
-    hideWaitControl(lowerPage());
     p.reader.dispatchEvent('monocle:turn');
     resetTurnData();
   }
 
 
   function resetTurnData() {
+    hideWaitControl(upperPage());
+    hideWaitControl(lowerPage());
     p.turnData = {};
   }
 
@@ -3268,7 +3436,7 @@ Monocle.Flippers.Slider = function (reader) {
           (new Date()).getTime() - stamp > duration ||
           Math.abs(currX - finalX) <= Math.abs((currX + step) - finalX)
         ) {
-          clearTimeout(elem.setXTransitionInterval)
+          clearTimeout(elem.setXTransitionInterval);
           Monocle.Styles.setX(elem, finalX);
           if (elem.setXTCB) {
             elem.setXTCB();
@@ -3366,13 +3534,17 @@ Monocle.Flippers.Slider = function (reader) {
 
   function jumpIn(pageDiv, callback) {
     var dur = Monocle.Browser.has.jumpFlickerBug ? 1 : 0;
-    setX(pageDiv, 0, { duration: dur }, callback);
+    Monocle.defer(function () {
+      setX(pageDiv, 0, { duration: dur }, callback);
+    });
   }
 
 
   function jumpOut(pageDiv, callback) {
     var dur = Monocle.Browser.has.jumpFlickerBug ? 1 : 0;
-    setX(pageDiv, 0 - pageDiv.offsetWidth, { duration: dur }, callback);
+    Monocle.defer(function () {
+      setX(pageDiv, 0 - pageDiv.offsetWidth, { duration: dur }, callback);
+    });
   }
 
 
@@ -3382,7 +3554,9 @@ Monocle.Flippers.Slider = function (reader) {
       duration: k.durations.SLIDE,
       timing: 'ease-in'
     };
-    setX(upperPage(), 0, slideOpts, callback);
+    Monocle.defer(function () {
+      setX(upperPage(), 0, slideOpts, callback);
+    });
   }
 
 
@@ -3391,7 +3565,9 @@ Monocle.Flippers.Slider = function (reader) {
       duration: k.durations.SLIDE,
       timing: 'ease-in'
     };
-    setX(upperPage(), 0 - upperPage().offsetWidth, slideOpts, callback);
+    Monocle.defer(function () {
+      setX(upperPage(), 0 - upperPage().offsetWidth, slideOpts, callback);
+    });
   }
 
 
@@ -3418,13 +3594,13 @@ Monocle.Flippers.Slider = function (reader) {
 
   function showWaitControl(page) {
     var ctrl = p.reader.dom.find('flippers_slider_wait', page.m.pageIndex);
-    ctrl.style.opacity = 0.5;
+    ctrl.style.visibility = "visible";
   }
 
 
   function hideWaitControl(page) {
     var ctrl = p.reader.dom.find('flippers_slider_wait', page.m.pageIndex);
-    ctrl.style.opacity = 0;
+    ctrl.style.visibility = "hidden";
   }
 
   API.pageCount = p.pageCount;

setup/installer/windows/portable.c

@@ -95,6 +95,11 @@ void launch_calibre(LPCTSTR exe, LPCTSTR config_dir, LPCTSTR library_dir) {
         ExitProcess(1);
     }
 
+    if (! SetEnvironmentVariable(TEXT("CALIBRE_PORTABLE_BUILD"), exe)) {
+        show_last_error(TEXT("Failed to set environment variables"));
+        ExitProcess(1);
+    }
+
     dwFlags = CREATE_UNICODE_ENVIRONMENT | CREATE_NEW_PROCESS_GROUP;
     _sntprintf_s(cmdline, BUFSIZE, _TRUNCATE, TEXT(" \"--with-library=%s\""), library_dir);
 

src/calibre/constants.py

@@ -4,7 +4,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = u'calibre'
-numeric_version = (0, 8, 5)
+numeric_version = (0, 8, 6)
 __version__   = u'.'.join(map(unicode, numeric_version))
 __author__    = u"Kovid Goyal <kovid@kovidgoyal.net>"
 
@@ -32,6 +32,7 @@ isbsd = isfreebsd or isnetbsd
 islinux   = not(iswindows or isosx or isbsd)
 isfrozen  = hasattr(sys, 'frozen')
 isunix = isosx or islinux
+isportable = os.environ.get('CALIBRE_PORTABLE_BUILD', None) is not None
 
 try:
     preferred_encoding = locale.getpreferredencoding()

src/calibre/customize/builtins.py

@@ -594,7 +594,7 @@ from calibre.devices.iliad.driver import ILIAD
 from calibre.devices.irexdr.driver import IREXDR1000, IREXDR800
 from calibre.devices.jetbook.driver import JETBOOK, MIBUK, JETBOOK_MINI
 from calibre.devices.kindle.driver import KINDLE, KINDLE2, KINDLE_DX
-from calibre.devices.nook.driver import NOOK, NOOK_COLOR, NOOK_TSR
+from calibre.devices.nook.driver import NOOK, NOOK_COLOR
 from calibre.devices.prs505.driver import PRS505
 from calibre.devices.user_defined.driver import USER_DEFINED
 from calibre.devices.android.driver import ANDROID, S60
@@ -694,7 +694,7 @@ plugins += [
     KINDLE,
     KINDLE2,
     KINDLE_DX,
-    NOOK, NOOK_COLOR, NOOK_TSR,
+    NOOK, NOOK_COLOR,
     PRS505,
     ANDROID,
     S60,
@@ -762,99 +762,127 @@ plugins += input_profiles + output_profiles
 class ActionAdd(InterfaceActionBase):
     name = 'Add Books'
     actual_plugin = 'calibre.gui2.actions.add:AddAction'
+    description = _('Add books to calibre or the connected device')
 
 class ActionFetchAnnotations(InterfaceActionBase):
     name = 'Fetch Annotations'
     actual_plugin = 'calibre.gui2.actions.annotate:FetchAnnotationsAction'
+    description = _('Fetch annotations from a connected Kindle (experimental)')
 
 class ActionGenerateCatalog(InterfaceActionBase):
     name = 'Generate Catalog'
     actual_plugin = 'calibre.gui2.actions.catalog:GenerateCatalogAction'
+    description = _('Generate a catalog of the books in your calibre library')
 
 class ActionConvert(InterfaceActionBase):
     name = 'Convert Books'
     actual_plugin = 'calibre.gui2.actions.convert:ConvertAction'
+    description = _('Convert books to various ebook formats')
 
 class ActionDelete(InterfaceActionBase):
     name = 'Remove Books'
     actual_plugin = 'calibre.gui2.actions.delete:DeleteAction'
+    description = _('Delete books from your calibre library or connected device')
 
 class ActionEditMetadata(InterfaceActionBase):
     name = 'Edit Metadata'
     actual_plugin = 'calibre.gui2.actions.edit_metadata:EditMetadataAction'
+    description = _('Edit the metadata of books in your calibre library')
 
 class ActionView(InterfaceActionBase):
     name = 'View'
     actual_plugin = 'calibre.gui2.actions.view:ViewAction'
+    description = _('Read books in your calibre library')
 
 class ActionFetchNews(InterfaceActionBase):
     name = 'Fetch News'
     actual_plugin = 'calibre.gui2.actions.fetch_news:FetchNewsAction'
+    description = _('Download news from the internet in ebook form')
 
 class ActionSaveToDisk(InterfaceActionBase):
     name = 'Save To Disk'
     actual_plugin = 'calibre.gui2.actions.save_to_disk:SaveToDiskAction'
+    description = _('Export books from your calibre library to the hard disk')
 
 class ActionShowBookDetails(InterfaceActionBase):
     name = 'Show Book Details'
     actual_plugin = 'calibre.gui2.actions.show_book_details:ShowBookDetailsAction'
+    description = _('Show book details in a separate popup')
 
 class ActionRestart(InterfaceActionBase):
     name = 'Restart'
     actual_plugin = 'calibre.gui2.actions.restart:RestartAction'
+    description = _('Restart calibre')
 
 class ActionOpenFolder(InterfaceActionBase):
     name = 'Open Folder'
     actual_plugin = 'calibre.gui2.actions.open:OpenFolderAction'
+    description = _('Open the folder that contains the book files in your'
+            ' calibre library')
 
 class ActionSendToDevice(InterfaceActionBase):
     name = 'Send To Device'
     actual_plugin = 'calibre.gui2.actions.device:SendToDeviceAction'
+    description = _('Send books to the connected device')
 
 class ActionConnectShare(InterfaceActionBase):
     name = 'Connect Share'
     actual_plugin = 'calibre.gui2.actions.device:ConnectShareAction'
+    description = _('Send books via email or the web also connect to iTunes or'
+            ' folders on your computer as if they are devices')
 
 class ActionHelp(InterfaceActionBase):
     name = 'Help'
     actual_plugin = 'calibre.gui2.actions.help:HelpAction'
+    description = _('Browse the calibre User Manual')
 
 class ActionPreferences(InterfaceActionBase):
     name = 'Preferences'
     actual_plugin = 'calibre.gui2.actions.preferences:PreferencesAction'
+    description = _('Customize calibre')
 
 class ActionSimilarBooks(InterfaceActionBase):
     name = 'Similar Books'
     actual_plugin = 'calibre.gui2.actions.similar_books:SimilarBooksAction'
+    description = _('Easily find books similar to the currently selected one')
 
 class ActionChooseLibrary(InterfaceActionBase):
     name = 'Choose Library'
     actual_plugin = 'calibre.gui2.actions.choose_library:ChooseLibraryAction'
+    description = _('Switch between different calibre libraries and perform'
+            ' maintenance on them')
 
 class ActionAddToLibrary(InterfaceActionBase):
     name = 'Add To Library'
     actual_plugin = 'calibre.gui2.actions.add_to_library:AddToLibraryAction'
+    description = _('Copy books from the devce to your calibre library')
 
 class ActionEditCollections(InterfaceActionBase):
     name = 'Edit Collections'
     actual_plugin = 'calibre.gui2.actions.edit_collections:EditCollectionsAction'
+    description = _('Edit the collections in which books are placed on your device')
 
 class ActionCopyToLibrary(InterfaceActionBase):
     name = 'Copy To Library'
     actual_plugin = 'calibre.gui2.actions.copy_to_library:CopyToLibraryAction'
+    description = _('Copy a book from one calibre library to another')
 
 class ActionTweakEpub(InterfaceActionBase):
     name = 'Tweak ePub'
     actual_plugin = 'calibre.gui2.actions.tweak_epub:TweakEpubAction'
+    description = _('Make small twekas to epub files in your calibre library')
 
 class ActionNextMatch(InterfaceActionBase):
     name = 'Next Match'
     actual_plugin = 'calibre.gui2.actions.next_match:NextMatchAction'
+    description = _('Find the next or previous match when searching in '
+            'your calibre library in highlight mode')
 
 class ActionStore(InterfaceActionBase):
     name = 'Store'
     author = 'John Schember'
     actual_plugin = 'calibre.gui2.actions.store:StoreAction'
+    description = _('Search for books from different book sellers')
 
     def customization_help(self, gui=False):
         return 'Customize the behavior of the store search.'
@@ -867,13 +895,20 @@ class ActionStore(InterfaceActionBase):
         from calibre.gui2.store.config.store import save_settings as save
         save(config_widget)
 
+class ActionPluginUpdater(InterfaceActionBase):
+    name = 'Plugin Updater'
+    author = 'Grant Drake'
+    description = _('Get new calibre plugins or update your existing ones')
+    actual_plugin = 'calibre.gui2.actions.plugin_updates:PluginUpdaterAction'
+
 plugins += [ActionAdd, ActionFetchAnnotations, ActionGenerateCatalog,
         ActionConvert, ActionDelete, ActionEditMetadata, ActionView,
         ActionFetchNews, ActionSaveToDisk, ActionShowBookDetails,
         ActionRestart, ActionOpenFolder, ActionConnectShare,
         ActionSendToDevice, ActionHelp, ActionPreferences, ActionSimilarBooks,
         ActionAddToLibrary, ActionEditCollections, ActionChooseLibrary,
-        ActionCopyToLibrary, ActionTweakEpub, ActionNextMatch, ActionStore]
+        ActionCopyToLibrary, ActionTweakEpub, ActionNextMatch, ActionStore,
+        ActionPluginUpdater]
 
 # }}}
 

src/calibre/customize/conversion.py

@@ -259,6 +259,10 @@ class OutputFormatPlugin(Plugin):
     #: (option_name, recommended_value, recommendation_level)
     recommendations = set([])
 
+    @property
+    def description(self):
+        return _('Convert ebooks to the %s format'%self.file_type)
+
     def __init__(self, *args):
         Plugin.__init__(self, *args)
         self.report_progress = DummyReporter()

src/calibre/customize/ui.py

@@ -493,6 +493,8 @@ def initialize_plugin(plugin, path_to_zip_file):
         raise InvalidPlugin((_('Initialization of plugin %s failed with traceback:')
                             %tb) + '\n'+tb)
 
+def has_external_plugins():
+    return bool(config['plugins'])
 
 def initialize_plugins():
     global _initialized_plugins

src/calibre/db/__init__.py

@@ -0,0 +1,67 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+
+'''
+Rewrite of the calibre database backend.
+
+Broad Objectives:
+
+    * Use the sqlite db only as a datastore. i.e. do not do
+      sorting/searching/concatenation or anything else in sqlite. Instead
+      mirror the sqlite tables in memory, create caches and lookup maps from
+      them and create a set_* API that updates the memory caches and the sqlite
+      correctly.
+
+    * Move from keeping a list of books in memory as a cache to a per table
+      cache. This allows much faster search and sort operations at the expense
+      of slightly slower lookup operations. That slowdown can be mitigated by
+      keeping lots of maps and updating them in the set_* API. Also
+      get_categories becomes blazingly fast.
+
+    * Separate the database layer from the cache layer more cleanly. Rather
+      than having the db layer refer to the cache layer and vice versa, the
+      cache layer will refer to the db layer only and the new API will be
+      defined on the cache layer.
+
+    * Get rid of index_is_id and other poor design decisions
+
+    * Minimize the API as much as possible and define it cleanly
+
+    * Do not change the on disk format of metadata.db at all (this is for
+      backwards compatibility)
+
+    * Get rid of the need for a separate db access thread by switching to apsw
+      to access sqlite, which is thread safe
+
+    * The new API will have methods to efficiently do bulk operations and will
+      use shared/exclusive/pending locks to serialize access to the in-mem data
+      structures. Use the same locking scheme as sqlite itself does.
+
+How this will proceed:
+
+    1. Create the new API
+    2. Create a test suite for it
+    3. Write a replacement for LibraryDatabase2 that uses the new API
+       internally
+    4. Lots of testing of calibre with the new LibraryDatabase2
+    5. Gradually migrate code to use the (much faster) new api wherever possible (the new api
+       will be exposed via db.new_api)
+
+    I plan to work on this slowly, in parallel to normal calibre development
+    work.
+
+Various things that require other things before they can be migrated:
+    1. From initialize_dynamic(): set_saved_searches,
+                    load_user_template_functions. Also add custom
+                    columns/categories/searches info into
+                    self.field_metadata. Finally, implement metadata dirtied
+                    functionality.
+
+'''

src/calibre/db/backend.py

@@ -0,0 +1,404 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+# Imports {{{
+import os, shutil, uuid, json
+from functools import partial
+
+import apsw
+
+from calibre import isbytestring, force_unicode, prints
+from calibre.constants import (iswindows, filesystem_encoding,
+        preferred_encoding)
+from calibre.ptempfile import PersistentTemporaryFile
+from calibre.library.schema_upgrades import SchemaUpgrade
+from calibre.library.field_metadata import FieldMetadata
+from calibre.ebooks.metadata import title_sort, author_to_author_sort
+from calibre.utils.icu import strcmp
+from calibre.utils.config import to_json, from_json, prefs, tweaks
+from calibre.utils.date import utcfromtimestamp
+# }}}
+
+'''
+Differences in semantics from pysqlite:
+
+    1. execute/executemany/executescript operate in autocommit mode
+
+'''
+
+class DynamicFilter(object): # {{{
+
+    'No longer used, present for legacy compatibility'
+
+    def __init__(self, name):
+        self.name = name
+        self.ids = frozenset([])
+
+    def __call__(self, id_):
+        return int(id_ in self.ids)
+
+    def change(self, ids):
+        self.ids = frozenset(ids)
+# }}}
+
+class DBPrefs(dict): # {{{
+
+    'Store preferences as key:value pairs in the db'
+
+    def __init__(self, db):
+        dict.__init__(self)
+        self.db = db
+        self.defaults = {}
+        self.disable_setting = False
+        for key, val in self.db.conn.get('SELECT key,val FROM preferences'):
+            try:
+                val = self.raw_to_object(val)
+            except:
+                prints('Failed to read value for:', key, 'from db')
+                continue
+            dict.__setitem__(self, key, val)
+
+    def raw_to_object(self, raw):
+        if not isinstance(raw, unicode):
+            raw = raw.decode(preferred_encoding)
+        return json.loads(raw, object_hook=from_json)
+
+    def to_raw(self, val):
+        return json.dumps(val, indent=2, default=to_json)
+
+    def __getitem__(self, key):
+        try:
+            return dict.__getitem__(self, key)
+        except KeyError:
+            return self.defaults[key]
+
+    def __delitem__(self, key):
+        dict.__delitem__(self, key)
+        self.db.conn.execute('DELETE FROM preferences WHERE key=?', (key,))
+
+    def __setitem__(self, key, val):
+        if self.disable_setting:
+            return
+        raw = self.to_raw(val)
+        self.db.conn.execute('INSERT OR REPLACE INTO preferences (key,val) VALUES (?,?)', (key,
+            raw))
+        dict.__setitem__(self, key, val)
+
+    def set(self, key, val):
+        self.__setitem__(key, val)
+
+# }}}
+
+# Extra collators {{{
+def pynocase(one, two, encoding='utf-8'):
+    if isbytestring(one):
+        try:
+            one = one.decode(encoding, 'replace')
+        except:
+            pass
+    if isbytestring(two):
+        try:
+            two = two.decode(encoding, 'replace')
+        except:
+            pass
+    return cmp(one.lower(), two.lower())
+
+def _author_to_author_sort(x):
+    if not x: return ''
+    return author_to_author_sort(x.replace('|', ','))
+
+def icu_collator(s1, s2):
+    return strcmp(force_unicode(s1, 'utf-8'), force_unicode(s2, 'utf-8'))
+# }}}
+
+class Connection(apsw.Connection): # {{{
+
+    BUSY_TIMEOUT = 2000 # milliseconds
+
+    def __init__(self, path):
+        apsw.Connection.__init__(self, path)
+
+        self.setbusytimeout(self.BUSY_TIMEOUT)
+        self.execute('pragma cache_size=5000')
+        self.conn.execute('pragma temp_store=2')
+
+        encoding = self.execute('pragma encoding').fetchone()[0]
+        self.conn.create_collation('PYNOCASE', partial(pynocase,
+            encoding=encoding))
+
+        self.conn.create_function('title_sort', 1, title_sort)
+        self.conn.create_function('author_to_author_sort', 1,
+                _author_to_author_sort)
+
+        self.conn.create_function('uuid4', 0, lambda : str(uuid.uuid4()))
+
+        # Dummy functions for dynamically created filters
+        self.conn.create_function('books_list_filter', 1, lambda x: 1)
+        self.conn.create_collation('icucollate', icu_collator)
+
+    def create_dynamic_filter(self, name):
+        f = DynamicFilter(name)
+        self.conn.create_function(name, 1, f)
+
+    def get(self, *args, **kw):
+        ans = self.cursor().execute(*args)
+        if kw.get('all', True):
+            return ans.fetchall()
+        for row in ans:
+            return ans[0]
+
+    def execute(self, sql, bindings=None):
+        cursor = self.cursor()
+        return cursor.execute(sql, bindings)
+
+    def executemany(self, sql, sequence_of_bindings):
+        return self.cursor().executemany(sql, sequence_of_bindings)
+
+    def executescript(self, sql):
+        with self:
+            # Use an explicit savepoint so that even if this is called
+            # while a transaction is active, it is atomic
+            return self.cursor().execute(sql)
+# }}}
+
+class DB(SchemaUpgrade):
+
+    PATH_LIMIT = 40 if iswindows else 100
+    WINDOWS_LIBRARY_PATH_LIMIT = 75
+
+    # Initialize database {{{
+
+    def __init__(self, library_path, default_prefs=None, read_only=False):
+        try:
+            if isbytestring(library_path):
+                library_path = library_path.decode(filesystem_encoding)
+        except:
+            import traceback
+            traceback.print_exc()
+
+        self.field_metadata = FieldMetadata()
+
+        self.library_path = os.path.abspath(library_path)
+        self.dbpath = os.path.join(library_path, 'metadata.db')
+        self.dbpath = os.environ.get('CALIBRE_OVERRIDE_DATABASE_PATH',
+                self.dbpath)
+
+        if iswindows and len(self.library_path) + 4*self.PATH_LIMIT + 10 > 259:
+            raise ValueError(_(
+                'Path to library too long. Must be less than'
+                ' %d characters.')%(259-4*self.PATH_LIMIT-10))
+        exists = self._exists = os.path.exists(self.dbpath)
+        if not exists:
+            # Be more strict when creating new libraries as the old calculation
+            # allowed for max path lengths of 265 chars.
+            if (iswindows and len(self.library_path) >
+                    self.WINDOWS_LIBRARY_PATH_LIMIT):
+                raise ValueError(_(
+                    'Path to library too long. Must be less than'
+                    ' %d characters.')%self.WINDOWS_LIBRARY_PATH_LIMIT)
+
+        if read_only and os.path.exists(self.dbpath):
+            # Work on only a copy of metadata.db to ensure that
+            # metadata.db is not changed
+            pt = PersistentTemporaryFile('_metadata_ro.db')
+            pt.close()
+            shutil.copyfile(self.dbpath, pt.name)
+            self.dbpath = pt.name
+
+        self.is_case_sensitive = (not iswindows and
+            not os.path.exists(self.dbpath.replace('metadata.db',
+                'MeTAdAtA.dB')))
+
+        self._conn = None
+
+        if self.user_version == 0:
+            self.initialize_database()
+
+        SchemaUpgrade.__init__(self)
+        # Guarantee that the library_id is set
+        self.library_id
+
+        self.initialize_prefs(default_prefs)
+
+        # Fix legacy triggers and columns
+        self.conn.executescript('''
+        DROP TRIGGER IF EXISTS author_insert_trg;
+        CREATE TEMP TRIGGER author_insert_trg
+            AFTER INSERT ON authors
+            BEGIN
+            UPDATE authors SET sort=author_to_author_sort(NEW.name) WHERE id=NEW.id;
+        END;
+        DROP TRIGGER IF EXISTS author_update_trg;
+        CREATE TEMP TRIGGER author_update_trg
+            BEFORE UPDATE ON authors
+            BEGIN
+            UPDATE authors SET sort=author_to_author_sort(NEW.name)
+            WHERE id=NEW.id AND name <> NEW.name;
+        END;
+        UPDATE authors SET sort=author_to_author_sort(name) WHERE sort IS NULL;
+        ''')
+
+    def initialize_prefs(self, default_prefs):
+        self.prefs = DBPrefs(self)
+
+        if default_prefs is not None and not self._exists:
+            # Only apply default prefs to a new database
+            for key in default_prefs:
+                # be sure that prefs not to be copied are listed below
+                if key not in frozenset(['news_to_be_synced']):
+                    self.prefs[key] = default_prefs[key]
+            if 'field_metadata' in default_prefs:
+                fmvals = [f for f in default_prefs['field_metadata'].values()
+                                if f['is_custom']]
+                for f in fmvals:
+                    self.create_custom_column(f['label'], f['name'],
+                            f['datatype'], f['is_multiple'] is not None,
+                            f['is_editable'], f['display'])
+
+        defs = self.prefs.defaults
+        defs['gui_restriction'] = defs['cs_restriction'] = ''
+        defs['categories_using_hierarchy'] = []
+        defs['column_color_rules'] = []
+
+        # Migrate the bool tristate tweak
+        defs['bools_are_tristate'] = \
+                tweaks.get('bool_custom_columns_are_tristate', 'yes') == 'yes'
+        if self.prefs.get('bools_are_tristate') is None:
+            self.prefs.set('bools_are_tristate', defs['bools_are_tristate'])
+
+        # Migrate column coloring rules
+        if self.prefs.get('column_color_name_1', None) is not None:
+            from calibre.library.coloring import migrate_old_rule
+            old_rules = []
+            for i in range(1, 6):
+                col = self.prefs.get('column_color_name_'+str(i), None)
+                templ = self.prefs.get('column_color_template_'+str(i), None)
+                if col and templ:
+                    try:
+                        del self.prefs['column_color_name_'+str(i)]
+                        rules = migrate_old_rule(self.field_metadata, templ)
+                        for templ in rules:
+                            old_rules.append((col, templ))
+                    except:
+                        pass
+            if old_rules:
+                self.prefs['column_color_rules'] += old_rules
+
+        # Migrate saved search and user categories to db preference scheme
+        def migrate_preference(key, default):
+            oldval = prefs[key]
+            if oldval != default:
+                self.prefs[key] = oldval
+                prefs[key] = default
+            if key not in self.prefs:
+                self.prefs[key] = default
+
+        migrate_preference('user_categories', {})
+        migrate_preference('saved_searches', {})
+
+        # migrate grouped_search_terms
+        if self.prefs.get('grouped_search_terms', None) is None:
+            try:
+                ogst = tweaks.get('grouped_search_terms', {})
+                ngst = {}
+                for t in ogst:
+                    ngst[icu_lower(t)] = ogst[t]
+                self.prefs.set('grouped_search_terms', ngst)
+            except:
+                pass
+
+        # Rename any user categories with names that differ only in case
+        user_cats = self.prefs.get('user_categories', [])
+        catmap = {}
+        for uc in user_cats:
+            ucl = icu_lower(uc)
+            if ucl not in catmap:
+                catmap[ucl] = []
+            catmap[ucl].append(uc)
+        cats_changed = False
+        for uc in catmap:
+            if len(catmap[uc]) > 1:
+                prints('found user category case overlap', catmap[uc])
+                cat = catmap[uc][0]
+                suffix = 1
+                while icu_lower((cat + unicode(suffix))) in catmap:
+                    suffix += 1
+                prints('Renaming user category %s to %s'%(cat, cat+unicode(suffix)))
+                user_cats[cat + unicode(suffix)] = user_cats[cat]
+                del user_cats[cat]
+                cats_changed = True
+        if cats_changed:
+            self.prefs.set('user_categories', user_cats)
+
+    @property
+    def conn(self):
+        if self._conn is None:
+            self._conn = apsw.Connection(self.dbpath)
+            if self._exists and self.user_version == 0:
+                self._conn.close()
+                os.remove(self.dbpath)
+                self._conn = apsw.Connection(self.dbpath)
+        return self._conn
+
+    @dynamic_property
+    def user_version(self):
+        doc = 'The user version of this database'
+
+        def fget(self):
+            return self.conn.get('pragma user_version;', all=False)
+
+        def fset(self, val):
+            self.conn.execute('pragma user_version=%d'%int(val))
+
+        return property(doc=doc, fget=fget, fset=fset)
+
+    def initialize_database(self):
+        metadata_sqlite = P('metadata_sqlite.sql', data=True,
+                allow_user_override=False).decode('utf-8')
+        self.conn.executescript(metadata_sqlite)
+        if self.user_version == 0:
+            self.user_version = 1
+    # }}}
+
+    # Database layer API {{{
+
+    @classmethod
+    def exists_at(cls, path):
+        return path and os.path.exists(os.path.join(path, 'metadata.db'))
+
+    @dynamic_property
+    def library_id(self):
+        doc = ('The UUID for this library. As long as the user only operates'
+                ' on libraries with calibre, it will be unique')
+
+        def fget(self):
+            if getattr(self, '_library_id_', None) is None:
+                ans = self.conn.get('SELECT uuid FROM library_id', all=False)
+                if ans is None:
+                    ans = str(uuid.uuid4())
+                    self.library_id = ans
+                else:
+                    self._library_id_ = ans
+            return self._library_id_
+
+        def fset(self, val):
+            self._library_id_ = unicode(val)
+            self.conn.execute('''
+                    DELETE FROM library_id;
+                    INSERT INTO library_id (uuid) VALUES (?);
+                    ''', self._library_id_)
+
+        return property(doc=doc, fget=fget, fset=fset)
+
+    def last_modified(self):
+        ''' Return last modified time as a UTC datetime object '''
+        return utcfromtimestamp(os.stat(self.dbpath).st_mtime)
+
+   # }}}
+

src/calibre/devices/apple/driver.py

@@ -135,7 +135,8 @@ class ITUNES(DriverBase):
     '''
     Calling sequences:
     Initialization:
-        can_handle() or can_handle_windows()
+        can_handle() | can_handle_windows()
+         _launch_iTunes()
         reset()
         open()
         card_prefix()

src/calibre/devices/hanvon/driver.py

@@ -61,7 +61,7 @@ class LIBREAIR(N516):
 
     BCD = [0x399]
     VENDOR_NAME      = 'ALURATEK'
-    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = '_FILE-STOR_GADGET'
+    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = 'FILE-STOR_GADGET'
     EBOOK_DIR_MAIN = 'Books'
 
 class ALEX(N516):

src/calibre/devices/nook/driver.py

@@ -81,55 +81,28 @@ class NOOK(USBMS):
         return [x.replace('#', '_') for x in components]
 
 class NOOK_COLOR(NOOK):
-    gui_name       = _('Nook Color')
+    description    = _('Communicate with the Nook Color and TSR eBook readers.')
-    description    = _('Communicate with the Nook Color eBook reader.')
 
-    PRODUCT_ID  = [0x002]
+    PRODUCT_ID  = [0x002, 0x003]
     BCD         = [0x216]
-    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = 'EBOOK_DISK'
 
+    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = 'EBOOK_DISK'
     EBOOK_DIR_MAIN = 'My Files'
+    NEWS_IN_FOLDER = False
+
+    def upload_cover(self, path, filename, metadata, filepath):
+        pass
+
+    def get_carda_ebook_dir(self, for_upload=False):
+        if for_upload:
+            return self.EBOOK_DIR_MAIN
+        return ''
 
     def create_upload_path(self, path, mdata, fname, create_dirs=True):
-        filepath = NOOK.create_upload_path(self, path, mdata, fname,
+        is_news = mdata.tags and _('News') in mdata.tags
-                create_dirs=False)
+        subdir = 'Magazines' if is_news else 'Books'
-        edm = self.EBOOK_DIR_MAIN
+        path = os.path.join(path, subdir)
-        subdir = 'Books'
+        return USBMS.create_upload_path(self, path, mdata, fname,
-        if mdata.tags:
+                create_dirs=create_dirs)
-            if _('News') in mdata.tags:
-                subdir = 'Magazines'
-        filepath = filepath.replace(os.sep+edm+os.sep,
-                os.sep+edm+os.sep+subdir+os.sep)
-        filedir = os.path.dirname(filepath)
-        if create_dirs and not os.path.exists(filedir):
-            os.makedirs(filedir)
-
-        return filepath
-
-    def upload_cover(self, path, filename, metadata, filepath):
-        pass
-
-    def get_carda_ebook_dir(self, for_upload=False):
-        if for_upload:
-            return 'My Files/Books'
-        return ''
-
-class NOOK_TSR(NOOK):
-    gui_name       = _('Nook Simple')
-    description    = _('Communicate with the Nook TSR eBook reader.')
-
-    PRODUCT_ID  = [0x003]
-    BCD         = [0x216]
-
-    EBOOK_DIR_MAIN = 'My Files/Books'
-    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = 'EBOOK_DISK'
-
-    def upload_cover(self, path, filename, metadata, filepath):
-        pass
-
-    def get_carda_ebook_dir(self, for_upload=False):
-        if for_upload:
-            return 'My Files/Books'
-        return ''
 
 

src/calibre/devices/usbms/device.py

@@ -101,6 +101,9 @@ class Device(DeviceConfig, DevicePlugin):
     #: The maximum length of paths created on the device
     MAX_PATH_LEN = 250
 
+    #: Put news in its own folder
+    NEWS_IN_FOLDER = True
+
     def reset(self, key='-1', log_packets=False, report_progress=None,
             detected_device=None):
         self._main_prefix = self._card_a_prefix = self._card_b_prefix = None
@@ -946,7 +949,8 @@ class Device(DeviceConfig, DevicePlugin):
             extra_components = []
             tag = special_tag
             if tag.startswith(_('News')):
-                extra_components.append('News')
+                if self.NEWS_IN_FOLDER:
+                    extra_components.append('News')
             else:
                 for c in tag.split('/'):
                     c = sanitize(c)

src/calibre/ebooks/epub/fix/epubcheck.py

@@ -26,6 +26,10 @@ class Epubcheck(ePubFixer):
                 'significant changes to your epub, complain to the epubcheck '
                 'project.')
 
+    @property
+    def description(self):
+        return self.long_description
+
     @property
     def fix_name(self):
         return 'epubcheck'

src/calibre/ebooks/epub/fix/unmanifested.py

@@ -22,6 +22,10 @@ class Unmanifested(ePubFixer):
         'the manifest or delete them as specified by the '
         'delete unmanifested option.')
 
+    @property
+    def description(self):
+        return self.long_description
+
     @property
     def fix_name(self):
         return 'unmanifested'

src/calibre/ebooks/html/input.py

@@ -455,13 +455,16 @@ class HTMLInput(InputFormatPlugin):
             bhref = os.path.basename(link)
             id, href = self.oeb.manifest.generate(id='added',
                     href=bhref)
+            guessed = self.guess_type(href)[0]
+            media_type = guessed or self.BINARY_MIME
+            if media_type == 'text/plain':
+                self.log.warn('Ignoring link to text file %r'%link_)
+                return None
+
             self.oeb.log.debug('Added', link)
             self.oeb.container = self.DirContainer(os.path.dirname(link),
                     self.oeb.log, ignore_opf=True)
             # Load into memory
-            guessed = self.guess_type(href)[0]
-            media_type = guessed or self.BINARY_MIME
-
             item = self.oeb.manifest.add(id, href, media_type)
             item.html_input_href = bhref
             if guessed in self.OEB_STYLES:

src/calibre/ebooks/metadata/sources/base.py

@@ -21,6 +21,7 @@ from calibre.ebooks.metadata import check_isbn
 msprefs = JSONConfig('metadata_sources/global.json')
 msprefs.defaults['txt_comments'] = False
 msprefs.defaults['ignore_fields'] = []
+msprefs.defaults['user_default_ignore_fields'] = []
 msprefs.defaults['max_tags'] = 20
 msprefs.defaults['wait_after_first_identify_result'] = 30 # seconds
 msprefs.defaults['wait_after_first_cover_result'] = 60 # seconds

src/calibre/ebooks/metadata/sources/identify.py

@@ -85,7 +85,11 @@ class ISBNMerge(object):
                 isbns, min_year = xisbn.get_isbn_pool(isbn)
                 if not isbns:
                     isbns = frozenset([isbn])
-                self.pools[isbns] = pool = (min_year, [])
+                if isbns in self.pools:
+                    # xISBN had a brain fart
+                    pool = self.pools[isbns]
+                else:
+                    self.pools[isbns] = pool = (min_year, [])
 
             if not self.pool_has_result_from_same_source(pool, result):
                 pool[1].append(result)

src/calibre/ebooks/metadata/xisbn.py

@@ -45,6 +45,11 @@ class xISBN(object):
                 ans.append(rec)
         return ans
 
+    def isbns_in_data(self, data):
+        for rec in data:
+            for i in rec.get('isbn', []):
+                yield i
+
     def get_data(self, isbn):
         isbn = self.purify(isbn)
         with self.lock:
@@ -57,9 +62,8 @@ class xISBN(object):
                     data = []
                 id_ = len(self._data)
                 self._data.append(data)
-                for rec in data:
+                for i in self.isbns_in_data(data):
-                    for i in rec.get('isbn', []):
+                    self._map[i] = id_
-                        self._map[i] = id_
                 self._map[isbn] = id_
             return self._data[self._map[isbn]]
 

src/calibre/ebooks/mobi/mobiml.py

@@ -443,11 +443,15 @@ class MobiMLizer(object):
             tag = 'span' if tag == 'td' else 'div'
 
         if tag == 'table':
+            col = style.backgroundColor
+            if col:
+                elem.set('bgcolor', col)
             css = style.cssdict()
             if 'border' in css or 'border-width' in css:
                 elem.set('border', '1')
         if tag in TABLE_TAGS:
-            for attr in ('rowspan', 'colspan', 'width', 'border', 'scope'):
+            for attr in ('rowspan', 'colspan', 'width', 'border', 'scope',
+                    'bgcolor'):
                 if attr in elem.attrib:
                     istate.attrib[attr] = elem.attrib[attr]
         if tag == 'q':

src/calibre/ebooks/mobi/writer.py

@@ -241,6 +241,7 @@ class Serializer(object):
         if self.write_page_breaks_after_item:
             buffer.write('<mbp:pagebreak/>')
         buffer.write('</div>')
+        self.anchor_offset = None
 
     def serialize_elem(self, elem, item, nsrmap=NSRMAP):
         buffer = self.buffer

src/calibre/ebooks/oeb/base.py

@@ -1055,6 +1055,12 @@ class Manifest(object):
                         and len(a) == 0 and not a.text:
                     remove_elem(a)
 
+            # Convert <br>s with content into paragraphs as ADE can't handle
+            # them
+            for br in xpath(data, '//h:br'):
+                if len(br) > 0 or br.text:
+                    br.tag = XHTML('div')
+
             return data
 
         def _parse_txt(self, data):
@@ -1156,7 +1162,7 @@ class Manifest(object):
                     data = self._parse_xml(data)
                 elif self.media_type.lower() in OEB_STYLES:
                     data = self._parse_css(data)
-                elif 'text' in self.media_type.lower():
+                elif self.media_type.lower() == 'text/plain':
                     self.oeb.log.warn('%s contains data in TXT format'%self.href,
                             'converting to HTML')
                     data = self._parse_txt(data)

src/calibre/ebooks/oeb/stylizer.py

@@ -11,7 +11,6 @@ __copyright__ = '2008, Marshall T. Vandegrift <llasram@gmail.com>'
 import os, itertools, re, logging, copy, unicodedata
 from weakref import WeakKeyDictionary
 from xml.dom import SyntaxErr as CSSSyntaxError
-import cssutils
 from cssutils.css import (CSSStyleRule, CSSPageRule, CSSStyleDeclaration,
     CSSFontFaceRule, cssproperties)
 try:
@@ -20,7 +19,8 @@ try:
 except ImportError:
     # cssutils >= 0.9.8
     from cssutils.css import PropertyValue as CSSValueList
-from cssutils import profile as cssprofiles
+from cssutils import (profile as cssprofiles, parseString, parseStyle, log as
+        cssutils_log, CSSParser, profiles)
 from lxml import etree
 from lxml.cssselect import css_to_xpath, ExpressionError, SelectorSyntaxError
 from calibre import force_unicode
@@ -28,7 +28,7 @@ from calibre.ebooks import unit_convert
 from calibre.ebooks.oeb.base import XHTML, XHTML_NS, CSS_MIME, OEB_STYLES
 from calibre.ebooks.oeb.base import XPNSMAP, xpath, urlnormalize
 
-cssutils.log.setLevel(logging.WARN)
+cssutils_log.setLevel(logging.WARN)
 
 _html_css_stylesheet = None
 
@@ -36,7 +36,7 @@ def html_css_stylesheet():
     global _html_css_stylesheet
     if _html_css_stylesheet is None:
         html_css = open(P('templates/html.css'), 'rb').read()
-        _html_css_stylesheet = cssutils.parseString(html_css)
+        _html_css_stylesheet = parseString(html_css)
         _html_css_stylesheet.namespaces['h'] = XHTML_NS
     return _html_css_stylesheet
 
@@ -157,11 +157,11 @@ class Stylizer(object):
 
         # Add cssutils parsing profiles from output_profile
         for profile in self.opts.output_profile.extra_css_modules:
-            cssutils.profile.addProfile(profile['name'],
+            cssprofiles.addProfile(profile['name'],
                                         profile['props'],
                                         profile['macros'])
 
-        parser = cssutils.CSSParser(fetcher=self._fetch_css_file,
+        parser = CSSParser(fetcher=self._fetch_css_file,
                 log=logging.getLogger('calibre.css'))
         self.font_face_rules = []
         for elem in head:
@@ -473,6 +473,7 @@ class Style(object):
         self._width = None
         self._height = None
         self._lineHeight = None
+        self._bgcolor = None
         stylizer._styles[element] = self
 
     def set(self, prop, val):
@@ -533,6 +534,48 @@ class Style(object):
     def pt_to_px(self, value):
         return (self._profile.dpi / 72.0) * value
 
+    @property
+    def backgroundColor(self):
+        '''
+        Return the background color by parsing both the background-color and
+        background shortcut properties. Note that inheritance/default values
+        are not used. None is returned if no background color is set.
+        '''
+
+        def validate_color(col):
+            return cssprofiles.validateWithProfile('color',
+                        col,
+                        profiles=[profiles.Profiles.CSS_LEVEL_2])[1]
+
+        if self._bgcolor is None:
+            col = None
+            val = self._style.get('background-color', None)
+            if val and validate_color(val):
+                col = val
+            else:
+                val = self._style.get('background', None)
+                if val is not None:
+                    try:
+                        style = parseStyle('background: '+val)
+                        val = style.getProperty('background').cssValue
+                        try:
+                            val = list(val)
+                        except:
+                            # val is CSSPrimitiveValue
+                            val = [val]
+                        for c in val:
+                            c = c.cssText
+                            if validate_color(c):
+                                col = c
+                                break
+                    except:
+                        pass
+            if col is None:
+                self._bgcolor = False
+            else:
+                self._bgcolor = col
+        return self._bgcolor if self._bgcolor else None
+
     @property
     def fontSize(self):
         def normalize_fontsize(value, base):

src/calibre/gui2/actions/choose_library.py

@@ -287,6 +287,18 @@ class ChooseLibraryAction(InterfaceAction):
               'rate of approximately 1 book every three seconds.'), show=True)
 
     def restore_database(self):
+        m = self.gui.library_view.model()
+        db = m.db
+        if (iswindows and len(db.library_path) >
+                LibraryDatabase2.WINDOWS_LIBRARY_PATH_LIMIT):
+            return error_dialog(self.gui, _('Too long'),
+                    _('Path to library too long. Must be less than'
+                    ' %d characters. Move your library to a location with'
+                    ' a shorter path using Windows Explorer, then point'
+                    ' calibre to the new location and try again.')%
+                    LibraryDatabase2.WINDOWS_LIBRARY_PATH_LIMIT,
+                    show=True)
+
         from calibre.gui2.dialogs.restore_library import restore_database
         m = self.gui.library_view.model()
         m.stop_metadata_backup()

src/calibre/gui2/actions/plugin_updates.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2011, Grant Drake <grant.drake@gmail.com>'
+__docformat__ = 'restructuredtext en'
+
+from PyQt4.Qt import QApplication, Qt, QIcon
+from calibre.gui2.actions import InterfaceAction
+from calibre.gui2.dialogs.plugin_updater import (PluginUpdaterDialog,
+                                                 FILTER_ALL, FILTER_UPDATE_AVAILABLE)
+
+class PluginUpdaterAction(InterfaceAction):
+
+    name = 'Plugin Updater'
+    action_spec = (_('Plugin Updater'), None, None, None)
+    action_type = 'current'
+
+    def genesis(self):
+        self.qaction.setIcon(QIcon(I('plugins/plugin_updater.png')))
+        self.qaction.triggered.connect(self.check_for_plugin_updates)
+
+    def check_for_plugin_updates(self):
+        # Get the user to choose a plugin to install
+        initial_filter = FILTER_UPDATE_AVAILABLE
+        mods = QApplication.keyboardModifiers()
+        if mods & Qt.ControlModifier or mods & Qt.ShiftModifier:
+            initial_filter = FILTER_ALL
+
+        d = PluginUpdaterDialog(self.gui, initial_filter=initial_filter)
+        d.exec_()

src/calibre/gui2/actions/preferences.py

@@ -24,6 +24,8 @@ class PreferencesAction(InterfaceAction):
             pm.addAction(QIcon(I('config.png')), _('Change calibre behavior'), self.do_config)
         pm.addAction(QIcon(I('wizard.png')), _('Run welcome wizard'),
                 self.gui.run_wizard)
+        pm.addAction(QIcon(I('plugins/plugin_updater.png')),
+                _('Get plugins to enhance calibre'), self.get_plugins)
         if not DEBUG:
             pm.addSeparator()
             ac = pm.addAction(QIcon(I('debug.png')), _('Restart in debug mode'),
@@ -36,6 +38,12 @@ class PreferencesAction(InterfaceAction):
         for x in (self.gui.preferences_action, self.qaction):
             x.triggered.connect(self.do_config)
 
+    def get_plugins(self):
+        from calibre.gui2.dialogs.plugin_updater import (PluginUpdaterDialog,
+                FILTER_NOT_INSTALLED)
+        d = PluginUpdaterDialog(self.gui,
+                initial_filter=FILTER_NOT_INSTALLED)
+        d.exec_()
 
     def do_config(self, checked=False, initial_plugin=None,
             close_after_initial=False):

src/calibre/gui2/device.py

@@ -49,16 +49,26 @@ class DeviceJob(BaseJob): # {{{
         self._aborted = False
 
     def start_work(self):
+        if DEBUG:
+            prints('Job:', self.id, self.description, 'started',
+                safe_encode=True)
         self.start_time = time.time()
         self.job_manager.changed_queue.put(self)
 
     def job_done(self):
         self.duration = time.time() - self.start_time
         self.percent = 1
+        if DEBUG:
+            prints('DeviceJob:', self.id, self.description,
+                    'done, calling callback', safe_encode=True)
+
         try:
             self.callback_on_done(self)
         except:
             pass
+        if DEBUG:
+            prints('DeviceJob:', self.id, self.description,
+                    'callback returned', safe_encode=True)
         self.job_manager.changed_queue.put(self)
 
     def report_progress(self, percent, msg=''):
@@ -119,6 +129,7 @@ class DeviceManager(Thread): # {{{
         self.sleep_time     = sleep_time
         self.connected_slot = connected_slot
         self.jobs           = Queue.Queue(0)
+        self.job_steps      = Queue.Queue(0)
         self.keep_going     = True
         self.job_manager    = job_manager
         self.reported_errors = set([])
@@ -235,6 +246,12 @@ class DeviceManager(Thread): # {{{
                 self.connected_device.unmount_device()
 
     def next(self):
+        if not self.job_steps.empty():
+            try:
+                return self.job_steps.get_nowait()
+            except Queue.Empty:
+                pass
+
         if not self.jobs.empty():
             try:
                 return self.jobs.get_nowait()
@@ -271,13 +288,20 @@ class DeviceManager(Thread): # {{{
                     break
             time.sleep(self.sleep_time)
 
-    def create_job(self, func, done, description, args=[], kwargs={}):
+    def create_job_step(self, func, done, description, to_job, args=[], kwargs={}):
         job = DeviceJob(func, done, self.job_manager,
                         args=args, kwargs=kwargs, description=description)
         self.job_manager.add_job(job)
-        self.jobs.put(job)
+        if (done is None or isinstance(done, FunctionDispatcher)) and \
+                    (to_job is not None and to_job == self.current_job):
+            self.job_steps.put(job)
+        else:
+            self.jobs.put(job)
         return job
 
+    def create_job(self, func, done, description, args=[], kwargs={}):
+        return self.create_job_step(func, done, description, None, args, kwargs)
+
     def has_card(self):
         try:
             return bool(self.device.card_prefix())
@@ -295,10 +319,10 @@ class DeviceManager(Thread): # {{{
         self._device_information = {'info': info, 'prefixes': cp, 'freespace': fs}
         return info, cp, fs
 
-    def get_device_information(self, done):
+    def get_device_information(self, done, add_as_step_to_job=None):
         '''Get device information and free space on device'''
-        return self.create_job(self._get_device_information, done,
+        return self.create_job_step(self._get_device_information, done,
-                    description=_('Get device information'))
+                    description=_('Get device information'), to_job=add_as_step_to_job)
 
     def get_current_device_information(self):
         return self._device_information
@@ -310,36 +334,38 @@ class DeviceManager(Thread): # {{{
         cardblist = self.device.books(oncard='cardb')
         return (mainlist, cardalist, cardblist)
 
-    def books(self, done):
+    def books(self, done, add_as_step_to_job=None):
         '''Return callable that returns the list of books on device as two booklists'''
-        return self.create_job(self._books, done, description=_('Get list of books on device'))
+        return self.create_job_step(self._books, done,
+                description=_('Get list of books on device'), to_job=add_as_step_to_job)
 
     def _annotations(self, path_map):
         return self.device.get_annotations(path_map)
 
-    def annotations(self, done, path_map):
+    def annotations(self, done, path_map, add_as_step_to_job=None):
         '''Return mapping of ids to annotations. Each annotation is of the
         form (type, location_info, content). path_map is a mapping of
         ids to paths on the device.'''
-        return self.create_job(self._annotations, done, args=[path_map],
+        return self.create_job_step(self._annotations, done, args=[path_map],
-                description=_('Get annotations from device'))
+                description=_('Get annotations from device'), to_job=add_as_step_to_job)
 
     def _sync_booklists(self, booklists):
         '''Sync metadata to device'''
         self.device.sync_booklists(booklists, end_session=False)
         return self.device.card_prefix(end_session=False), self.device.free_space()
 
-    def sync_booklists(self, done, booklists, plugboards):
+    def sync_booklists(self, done, booklists, plugboards, add_as_step_to_job=None):
         if hasattr(self.connected_device, 'set_plugboards') and \
                 callable(self.connected_device.set_plugboards):
             self.connected_device.set_plugboards(plugboards, find_plugboard)
-        return self.create_job(self._sync_booklists, done, args=[booklists],
+        return self.create_job_step(self._sync_booklists, done, args=[booklists],
-                        description=_('Send metadata to device'))
+                        description=_('Send metadata to device'), to_job=add_as_step_to_job)
 
-    def upload_collections(self, done, booklist, on_card):
+    def upload_collections(self, done, booklist, on_card, add_as_step_to_job=None):
-        return self.create_job(booklist.rebuild_collections, done,
+        return self.create_job_step(booklist.rebuild_collections, done,
                                args=[booklist, on_card],
-                        description=_('Send collections to device'))
+                        description=_('Send collections to device'),
+                        to_job=add_as_step_to_job)
 
     def _upload_books(self, files, names, on_card=None, metadata=None, plugboards=None):
         '''Upload books to device: '''
@@ -374,11 +400,12 @@ class DeviceManager(Thread): # {{{
                                         metadata=metadata, end_session=False)
 
     def upload_books(self, done, files, names, on_card=None, titles=None,
-                     metadata=None, plugboards=None):
+                     metadata=None, plugboards=None, add_as_step_to_job=None):
         desc = _('Upload %d books to device')%len(names)
         if titles:
             desc += u':' + u', '.join(titles)
-        return self.create_job(self._upload_books, done, args=[files, names],
+        return self.create_job_step(self._upload_books, done, to_job=add_as_step_to_job,
+                               args=[files, names],
                 kwargs={'on_card':on_card,'metadata':metadata,'plugboards':plugboards}, description=desc)
 
     def add_books_to_metadata(self, locations, metadata, booklists):
@@ -388,9 +415,10 @@ class DeviceManager(Thread): # {{{
         '''Remove books from device'''
         self.device.delete_books(paths, end_session=True)
 
-    def delete_books(self, done, paths):
+    def delete_books(self, done, paths, add_as_step_to_job=None):
-        return self.create_job(self._delete_books, done, args=[paths],
+        return self.create_job_step(self._delete_books, done, args=[paths],
-                        description=_('Delete books from device'))
+                        description=_('Delete books from device'),
+                        to_job=add_as_step_to_job)
 
     def remove_books_from_metadata(self, paths, booklists):
         self.device.remove_books_from_metadata(paths, booklists)
@@ -405,9 +433,10 @@ class DeviceManager(Thread): # {{{
                 self.device.get_file(path, f)
                 f.close()
 
-    def save_books(self, done, paths, target):
+    def save_books(self, done, paths, target, add_as_step_to_job=None):
-        return self.create_job(self._save_books, done, args=[paths, target],
+        return self.create_job_step(self._save_books, done, args=[paths, target],
-                        description=_('Download books from device'))
+                        description=_('Download books from device'),
+                        to_job=add_as_step_to_job)
 
     def _view_book(self, path, target):
         f = open(target, 'wb')
@@ -415,9 +444,9 @@ class DeviceManager(Thread): # {{{
         f.close()
         return target
 
-    def view_book(self, done, path, target):
+    def view_book(self, done, path, target, add_as_step_to_job=None):
-        return self.create_job(self._view_book, done, args=[path, target],
+        return self.create_job_step(self._view_book, done, args=[path, target],
-                        description=_('View book on device'))
+                        description=_('View book on device'), to_job=add_as_step_to_job)
 
     def set_current_library_uuid(self, uuid):
         self.current_library_uuid = uuid
@@ -778,7 +807,8 @@ class DeviceMixin(object): # {{{
                 self.device_manager.device.icon)
         self.bars_manager.update_bars()
         self.status_bar.device_connected(info[0])
-        self.device_manager.books(FunctionDispatcher(self.metadata_downloaded))
+        self.device_manager.books(FunctionDispatcher(self.metadata_downloaded),
+                                  add_as_step_to_job=job)
 
     def metadata_downloaded(self, job):
         '''
@@ -788,7 +818,7 @@ class DeviceMixin(object): # {{{
             self.device_job_exception(job)
             return
         # set_books_in_library might schedule a sync_booklists job
-        self.set_books_in_library(job.result, reset=True)
+        self.set_books_in_library(job.result, reset=True, add_as_step_to_job=job)
         mainlist, cardalist, cardblist = job.result
         self.memory_view.set_database(mainlist)
         self.memory_view.set_editable(self.device_manager.device.CAN_SET_METADATA,
@@ -843,8 +873,8 @@ class DeviceMixin(object): # {{{
         # set_books_in_library even though books were not added because
         # the deleted book might have been an exact match. Upload the booklists
         # if set_books_in_library did not.
-        if not self.set_books_in_library(self.booklists(), reset=True):
+        if not self.set_books_in_library(self.booklists(), reset=True, add_as_step_to_job=job):
-            self.upload_booklists()
+            self.upload_booklists(job)
         self.book_on_device(None, reset=True)
         # We need to reset the ondevice flags in the library. Use a big hammer,
         # so we don't need to worry about whether some succeeded or not.
@@ -1193,13 +1223,14 @@ class DeviceMixin(object): # {{{
         self.device_manager.sync_booklists(Dispatcher(lambda x: x),
                                            self.booklists(), plugboards)
 
-    def upload_booklists(self):
+    def upload_booklists(self, add_as_step_to_job=None):
         '''
         Upload metadata to device.
         '''
         plugboards = self.library_view.model().db.prefs.get('plugboards', {})
         self.device_manager.sync_booklists(FunctionDispatcher(self.metadata_synced),
-                                           self.booklists(), plugboards)
+                                           self.booklists(), plugboards,
+                                           add_as_step_to_job=add_as_step_to_job)
 
     def metadata_synced(self, job):
         '''
@@ -1274,8 +1305,8 @@ class DeviceMixin(object): # {{{
         # because the UUID changed. Force both the device and the library view
         # to refresh the flags. Set_books_in_library could upload the booklists.
         # If it does not, then do it here.
-        if not self.set_books_in_library(self.booklists(), reset=True):
+        if not self.set_books_in_library(self.booklists(), reset=True, add_as_step_to_job=job):
-            self.upload_booklists()
+            self.upload_booklists(job)
         with self.library_view.preserve_selected_books:
             self.book_on_device(None, reset=True)
             self.refresh_ondevice()
@@ -1335,7 +1366,7 @@ class DeviceMixin(object): # {{{
                 loc[4] |= self.book_db_uuid_path_map[id]
         return loc
 
-    def set_books_in_library(self, booklists, reset=False):
+    def set_books_in_library(self, booklists, reset=False, add_as_step_to_job=None):
         '''
         Set the ondevice indications in the device database.
         This method should be called before book_on_device is called, because
@@ -1487,7 +1518,7 @@ class DeviceMixin(object): # {{{
                 plugboards = self.library_view.model().db.prefs.get('plugboards', {})
                 self.device_manager.sync_booklists(
                                     FunctionDispatcher(self.metadata_synced), booklists,
-                                    plugboards)
+                                    plugboards, add_as_step_to_job)
         return update_metadata
     # }}}
 

src/calibre/gui2/dialogs/plugin_updater.py

@@ -0,0 +1,869 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2011, Grant Drake <grant.drake@gmail.com>'
+__docformat__ = 'restructuredtext en'
+
+import re, datetime, traceback
+from lxml import html
+from PyQt4.Qt import (Qt, QUrl, QFrame, QVBoxLayout, QLabel, QBrush, QTextEdit,
+                      QComboBox, QAbstractItemView, QHBoxLayout, QDialogButtonBox,
+                      QAbstractTableModel, QVariant, QTableView, QModelIndex,
+                      QSortFilterProxyModel, QAction, QIcon, QDialog,
+                      QFont, QPixmap, QSize)
+from calibre import browser, prints
+from calibre.constants import numeric_version, iswindows, isosx, DEBUG
+from calibre.customize.ui import (initialized_plugins, is_disabled, remove_plugin,
+                                  add_plugin, enable_plugin, disable_plugin,
+                                  NameConflict, has_external_plugins)
+from calibre.gui2 import error_dialog, question_dialog, info_dialog, NONE, open_url, gprefs
+from calibre.gui2.preferences.plugins import ConfigWidget
+from calibre.utils.date import UNDEFINED_DATE, format_date
+
+
+MR_URL = 'http://www.mobileread.com/forums/'
+MR_INDEX_URL = MR_URL + 'showpost.php?p=1362767&postcount=1'
+
+FILTER_ALL = 0
+FILTER_INSTALLED = 1
+FILTER_UPDATE_AVAILABLE = 2
+FILTER_NOT_INSTALLED = 3
+
+def get_plugin_updates_available():
+    '''
+    API exposed to read whether there are updates available for any
+    of the installed user plugins.
+    Returns None if no updates found
+    Returns list(DisplayPlugin) of plugins installed that have a new version
+    '''
+    if not has_external_plugins():
+        return None
+    display_plugins = read_available_plugins()
+    if display_plugins:
+        update_plugins = filter(filter_upgradeable_plugins, display_plugins)
+        if len(update_plugins) > 0:
+            return update_plugins
+    return None
+
+def filter_upgradeable_plugins(display_plugin):
+    return display_plugin.is_upgrade_available()
+
+def filter_not_installed_plugins(display_plugin):
+    return not display_plugin.is_installed()
+
+def read_available_plugins():
+    display_plugins = []
+    br = browser()
+    br.set_handle_gzip(True)
+    try:
+        raw = br.open_novisit(MR_INDEX_URL).read()
+        if not raw:
+            return
+    except:
+        traceback.print_exc()
+        return
+    raw = raw.decode('utf-8', errors='replace')
+    root = html.fromstring(raw)
+    list_nodes = root.xpath('//div[@id="post_message_1362767"]/ul/li')
+    # Add our deprecated plugins which are nested in a grey span
+    list_nodes.extend(root.xpath('//div[@id="post_message_1362767"]/span/ul/li'))
+    for list_node in list_nodes:
+        try:
+            display_plugin = DisplayPlugin(list_node)
+            get_installed_plugin_status(display_plugin)
+            display_plugins.append(display_plugin)
+        except:
+            if DEBUG:
+                prints('======= MobileRead Parse Error =======')
+                traceback.print_exc()
+                prints(html.tostring(list_node))
+    display_plugins = sorted(display_plugins, key=lambda k: k.name)
+    return display_plugins
+
+def get_installed_plugin_status(display_plugin):
+    display_plugin.installed_version = None
+    display_plugin.plugin = None
+    for plugin in initialized_plugins():
+        if plugin.name == display_plugin.name:
+            display_plugin.plugin = plugin
+            display_plugin.installed_version = plugin.version
+            break
+    if display_plugin.uninstall_plugins:
+        # Plugin requires a specific plugin name to be uninstalled first
+        # This could occur when a plugin is renamed (Kindle Collections)
+        # or multiple plugins deprecated into a newly named one.
+        # Check whether user has the previous version(s) installed
+        plugins_to_remove = list(display_plugin.uninstall_plugins)
+        for plugin_to_uninstall in plugins_to_remove:
+            found = False
+            for plugin in initialized_plugins():
+                if plugin.name == plugin_to_uninstall:
+                    found = True
+                    break
+            if not found:
+                display_plugin.uninstall_plugins.remove(plugin_to_uninstall)
+
+
+class ImageTitleLayout(QHBoxLayout):
+    '''
+    A reusable layout widget displaying an image followed by a title
+    '''
+    def __init__(self, parent, icon_name, title):
+        QHBoxLayout.__init__(self)
+        title_font = QFont()
+        title_font.setPointSize(16)
+        title_image_label = QLabel(parent)
+        pixmap = QPixmap()
+        pixmap.load(I(icon_name))
+        if pixmap is None:
+            error_dialog(parent, _('Restart required'),
+                         _('You must restart Calibre before using this plugin!'), show=True)
+        else:
+            title_image_label.setPixmap(pixmap)
+        title_image_label.setMaximumSize(32, 32)
+        title_image_label.setScaledContents(True)
+        self.addWidget(title_image_label)
+        shelf_label = QLabel(title, parent)
+        shelf_label.setFont(title_font)
+        self.addWidget(shelf_label)
+        self.insertStretch(-1)
+
+
+class SizePersistedDialog(QDialog):
+    '''
+    This dialog is a base class for any dialogs that want their size/position
+    restored when they are next opened.
+    '''
+
+    initial_extra_size = QSize(0, 0)
+
+    def __init__(self, parent, unique_pref_name):
+        QDialog.__init__(self, parent)
+        self.unique_pref_name = unique_pref_name
+        self.geom = gprefs.get(unique_pref_name, None)
+        self.finished.connect(self.dialog_closing)
+
+    def resize_dialog(self):
+        if self.geom is None:
+            self.resize(self.sizeHint()+self.initial_extra_size)
+        else:
+            self.restoreGeometry(self.geom)
+
+    def dialog_closing(self, result):
+        geom = bytearray(self.saveGeometry())
+        gprefs[self.unique_pref_name] = geom
+
+
+class VersionHistoryDialog(SizePersistedDialog):
+
+    def __init__(self, parent, plugin_name, html):
+        SizePersistedDialog.__init__(self, parent, 'Plugin Updater plugin:version history dialog')
+        self.setWindowTitle(_('Version History for %s')%plugin_name)
+
+        layout = QVBoxLayout(self)
+        self.setLayout(layout)
+
+        self.notes = QTextEdit(html, self)
+        self.notes.setReadOnly(True)
+        layout.addWidget(self.notes)
+
+        self.button_box = QDialogButtonBox(QDialogButtonBox.Close)
+        self.button_box.rejected.connect(self.reject)
+        layout.addWidget(self.button_box)
+
+        # Cause our dialog size to be restored from prefs or created on first usage
+        self.resize_dialog()
+
+
+class PluginFilterComboBox(QComboBox):
+    def __init__(self, parent):
+        QComboBox.__init__(self, parent)
+        items = [_('All'), _('Installed'), _('Update available'), _('Not installed')]
+        self.addItems(items)
+
+
+class DisplayPlugin(object):
+
+    def __init__(self, list_node):
+        # The html from the index web page looks like this:
+        '''
+<li><a href="http://www.mobileread.com/forums/showthread.php?t=121787">Book Sync</a><br />
+<i>Add books to a list to be automatically sent to your device the next time it is connected.<br />
+<span class="resize_1">Version: 1.1; Released: 02-22-2011; Calibre: 0.7.42; Author: kiwidude; <br />
+Platforms: Windows, OSX, Linux; History: Yes;</span></i></li>
+        '''
+        self.name = list_node.xpath('a')[0].text_content().strip()
+        self.forum_link = list_node.xpath('a/@href')[0].strip()
+        self.installed_version = None
+
+        description_text = list_node.xpath('i')[0].text_content()
+        description_parts = description_text.partition('Version:')
+        self.description = description_parts[0].strip()
+
+        details_text = description_parts[1] + description_parts[2].replace('\r\n','')
+        details_pairs = details_text.split(';')
+        details = {}
+        for details_pair in details_pairs:
+            pair = details_pair.split(':')
+            if len(pair) == 2:
+                key = pair[0].strip().lower()
+                value = pair[1].strip()
+                details[key] = value
+
+        donation_node = list_node.xpath('i/span/a/@href')
+        self.donation_link = donation_node[0] if donation_node else None
+
+        self.available_version = self._version_text_to_tuple(details.get('version', None))
+
+        release_date = details.get('released', '01-01-0101').split('-')
+        date_parts = [int(re.search(r'(\d+)', x).group(1)) for x in release_date]
+        self.release_date = datetime.date(date_parts[2], date_parts[0], date_parts[1])
+
+        self.calibre_required_version = self._version_text_to_tuple(details.get('calibre', None))
+        self.author = details.get('author', '')
+        self.platforms = [p.strip().lower() for p in details.get('platforms', '').split(',')]
+        # Optional pairing just for plugins which require checking for uninstall first
+        self.uninstall_plugins = []
+        uninstall = details.get('uninstall', None)
+        if uninstall:
+            self.uninstall_plugins = [i.strip() for i in uninstall.split(',')]
+        self.has_changelog = details.get('history', 'No').lower() in ['yes', 'true']
+        self.is_deprecated = details.get('deprecated', 'No').lower() in ['yes', 'true']
+
+    def _version_text_to_tuple(self, version_text):
+        if version_text:
+            ver = version_text.split('.')
+            while len(ver) < 3:
+                ver.append('0')
+            ver = [int(re.search(r'(\d+)', x).group(1)) for x in ver]
+            return tuple(ver)
+        else:
+            return None
+
+    def is_disabled(self):
+        if self.plugin is None:
+            return False
+        return is_disabled(self.plugin)
+
+    def is_installed(self):
+        return self.installed_version is not None
+
+    def is_upgrade_available(self):
+        return self.is_installed() and (self.installed_version < self.available_version \
+                or self.is_deprecated)
+
+    def is_valid_platform(self):
+        if iswindows:
+            return 'windows' in self.platforms
+        if isosx:
+            return 'osx' in self.platforms
+        return 'linux' in self.platforms
+
+    def is_valid_calibre(self):
+        return numeric_version >= self.calibre_required_version
+
+    def is_valid_to_install(self):
+        return self.is_valid_platform() and self.is_valid_calibre() and not self.is_deprecated
+
+
+class DisplayPluginSortFilterModel(QSortFilterProxyModel):
+
+    def __init__(self, parent):
+        QSortFilterProxyModel.__init__(self, parent)
+        self.setSortRole(Qt.UserRole)
+        self.filter_criteria = FILTER_ALL
+
+    def filterAcceptsRow(self, sourceRow, sourceParent):
+        index = self.sourceModel().index(sourceRow, 0, sourceParent)
+        display_plugin = self.sourceModel().display_plugins[index.row()]
+        if self.filter_criteria == FILTER_ALL:
+            return not (display_plugin.is_deprecated and not display_plugin.is_installed())
+        if self.filter_criteria == FILTER_INSTALLED:
+            return display_plugin.is_installed()
+        if self.filter_criteria == FILTER_UPDATE_AVAILABLE:
+            return display_plugin.is_upgrade_available()
+        if self.filter_criteria == FILTER_NOT_INSTALLED:
+            return not display_plugin.is_installed() and not display_plugin.is_deprecated
+        return False
+
+    def set_filter_criteria(self, filter_value):
+        self.filter_criteria = filter_value
+        self.invalidateFilter()
+
+
+class DisplayPluginModel(QAbstractTableModel):
+
+    def __init__(self, display_plugins):
+        QAbstractTableModel.__init__(self)
+        self.display_plugins = display_plugins
+        self.headers = map(QVariant, [_('Plugin Name'), _('Donate'), _('Status'), _('Installed'),
+                                      _('Available'), _('Released'), _('Calibre'), _('Author')])
+
+    def rowCount(self, *args):
+        return len(self.display_plugins)
+
+    def columnCount(self, *args):
+        return len(self.headers)
+
+    def headerData(self, section, orientation, role):
+        if role == Qt.DisplayRole and orientation == Qt.Horizontal:
+            return self.headers[section]
+        return NONE
+
+    def data(self, index, role):
+        if not index.isValid():
+            return NONE;
+        row, col = index.row(), index.column()
+        if row < 0 or row >= self.rowCount():
+            return NONE
+        display_plugin = self.display_plugins[row]
+        if role in [Qt.DisplayRole, Qt.UserRole]:
+            if col == 0:
+                return QVariant(display_plugin.name)
+            if col == 1:
+                if display_plugin.donation_link:
+                    return QVariant(_('PayPal'))
+            if col == 2:
+                return self._get_status(display_plugin)
+            if col == 3:
+                return QVariant(self._get_display_version(display_plugin.installed_version))
+            if col == 4:
+                return QVariant(self._get_display_version(display_plugin.available_version))
+            if col == 5:
+                if role == Qt.UserRole:
+                    return self._get_display_release_date(display_plugin.release_date, 'yyyyMMdd')
+                else:
+                    return self._get_display_release_date(display_plugin.release_date)
+            if col == 6:
+                return QVariant(self._get_display_version(display_plugin.calibre_required_version))
+            if col == 7:
+                return QVariant(display_plugin.author)
+        elif role == Qt.DecorationRole:
+            if col == 0:
+                return self._get_status_icon(display_plugin)
+            if col == 1:
+                if display_plugin.donation_link:
+                    return QIcon(I('donate.png'))
+        elif role == Qt.ToolTipRole:
+            if col == 1 and display_plugin.donation_link:
+                return QVariant(_('This plugin is FREE but you can reward the developer for their effort\n'
+                                  'by donating to them via PayPal.\n\n'
+                                  'Right-click and choose Donate to reward: ')+display_plugin.author)
+            else:
+                return self._get_status_tooltip(display_plugin)
+        elif role == Qt.ForegroundRole:
+            if col != 1: # Never change colour of the donation column
+                if display_plugin.is_deprecated:
+                    return QVariant(QBrush(Qt.blue))
+                if display_plugin.is_disabled():
+                    return QVariant(QBrush(Qt.gray))
+        return NONE
+
+    def plugin_to_index(self, display_plugin):
+        for i, p in enumerate(self.display_plugins):
+            if display_plugin == p:
+                return self.index(i, 0, QModelIndex())
+        return QModelIndex()
+
+    def refresh_plugin(self, display_plugin):
+        idx = self.plugin_to_index(display_plugin)
+        self.dataChanged.emit(idx, idx)
+
+    def _get_display_release_date(self, date_value, format='dd MMM yyyy'):
+        if date_value and date_value != UNDEFINED_DATE:
+            return QVariant(format_date(date_value, format))
+        return NONE
+
+    def _get_display_version(self, version):
+        if version is None:
+            return ''
+        return '.'.join([str(v) for v in list(version)])
+
+    def _get_status(self, display_plugin):
+        if not display_plugin.is_valid_platform():
+            return _('Platform unavailable')
+        if not display_plugin.is_valid_calibre():
+            return _('Calibre upgrade required')
+        if display_plugin.is_installed():
+            if display_plugin.is_deprecated:
+                return _('Plugin deprecated')
+            elif display_plugin.is_upgrade_available():
+                return _('New version available')
+            else:
+                return _('Latest version installed')
+        return _('Not installed')
+
+    def _get_status_icon(self, display_plugin):
+        if display_plugin.is_deprecated:
+            icon_name = 'plugin_deprecated.png'
+        elif display_plugin.is_disabled():
+            if display_plugin.is_upgrade_available():
+                if display_plugin.is_valid_to_install():
+                    icon_name = 'plugin_disabled_valid.png'
+                else:
+                    icon_name = 'plugin_disabled_invalid.png'
+            else:
+                icon_name = 'plugin_disabled_ok.png'
+        elif display_plugin.is_installed():
+            if display_plugin.is_upgrade_available():
+                if display_plugin.is_valid_to_install():
+                    icon_name = 'plugin_upgrade_valid.png'
+                else:
+                    icon_name = 'plugin_upgrade_invalid.png'
+            else:
+                icon_name = 'plugin_upgrade_ok.png'
+        else: # A plugin available not currently installed
+            if display_plugin.is_valid_to_install():
+                icon_name = 'plugin_new_valid.png'
+            else:
+                icon_name = 'plugin_new_invalid.png'
+        return QIcon(I('plugins/' + icon_name))
+
+    def _get_status_tooltip(self, display_plugin):
+        if display_plugin.is_deprecated:
+            return QVariant(_('This plugin has been deprecated and should be uninstalled')+'\n\n'+
+                            _('Right-click to see more options'))
+        if not display_plugin.is_valid_platform():
+            return QVariant(_('This plugin can only be installed on: %s') % \
+                            ', '.join(display_plugin.platforms)+'\n\n'+
+                            _('Right-click to see more options'))
+        if numeric_version < display_plugin.calibre_required_version:
+            return QVariant(_('You must upgrade to at least Calibre %s before installing this plugin') % \
+                            self._get_display_version(display_plugin.calibre_required_version)+'\n\n'+
+                            _('Right-click to see more options'))
+        if display_plugin.installed_version < display_plugin.available_version:
+            if display_plugin.installed_version is None:
+                return QVariant(_('You can install this plugin')+'\n\n'+
+                                _('Right-click to see more options'))
+            else:
+                return QVariant(_('A new version of this plugin is available')+'\n\n'+
+                                _('Right-click to see more options'))
+        return QVariant(_('This plugin is installed and up-to-date')+'\n\n'+
+                        _('Right-click to see more options'))
+
+
+class PluginUpdaterDialog(SizePersistedDialog):
+
+    initial_extra_size = QSize(350, 100)
+
+    def __init__(self, gui, initial_filter=FILTER_UPDATE_AVAILABLE):
+        SizePersistedDialog.__init__(self, gui, 'Plugin Updater plugin:plugin updater dialog')
+        self.gui = gui
+        self.forum_link = None
+        self.model = None
+        self._initialize_controls()
+        self._create_context_menu()
+
+        display_plugins = read_available_plugins()
+
+        if display_plugins:
+            self.model = DisplayPluginModel(display_plugins)
+            self.proxy_model = DisplayPluginSortFilterModel(self)
+            self.proxy_model.setSourceModel(self.model)
+            self.plugin_view.setModel(self.proxy_model)
+            self.plugin_view.resizeColumnsToContents()
+            self.plugin_view.selectionModel().currentRowChanged.connect(self._plugin_current_changed)
+            self.plugin_view.doubleClicked.connect(self.install_button.click)
+            self.filter_combo.setCurrentIndex(initial_filter)
+            self._select_and_focus_view()
+        else:
+            error_dialog(self.gui, _('Update Check Failed'),
+                        _('Unable to reach the MobileRead plugins forum index page.'),
+                        det_msg=MR_INDEX_URL, show=True)
+            self.filter_combo.setEnabled(False)
+        # Cause our dialog size to be restored from prefs or created on first usage
+        self.resize_dialog()
+
+    def _initialize_controls(self):
+        self.setWindowTitle(_('User plugins'))
+        self.setWindowIcon(QIcon(I('plugins/plugin_updater.png')))
+        layout = QVBoxLayout(self)
+        self.setLayout(layout)
+        title_layout = ImageTitleLayout(self, 'plugins/plugin_updater.png',
+                _('User Plugins'))
+        layout.addLayout(title_layout)
+
+        header_layout = QHBoxLayout()
+        layout.addLayout(header_layout)
+        self.filter_combo = PluginFilterComboBox(self)
+        self.filter_combo.setMinimumContentsLength(20)
+        self.filter_combo.currentIndexChanged[int].connect(self._filter_combo_changed)
+        header_layout.addWidget(QLabel(_('Filter list of plugins')+':', self))
+        header_layout.addWidget(self.filter_combo)
+        header_layout.addStretch(10)
+
+        self.plugin_view = QTableView(self)
+        self.plugin_view.horizontalHeader().setStretchLastSection(True)
+        self.plugin_view.setSelectionBehavior(QAbstractItemView.SelectRows)
+        self.plugin_view.setSelectionMode(QAbstractItemView.SingleSelection)
+        self.plugin_view.setAlternatingRowColors(True)
+        self.plugin_view.setSortingEnabled(True)
+        self.plugin_view.setIconSize(QSize(28, 28))
+        layout.addWidget(self.plugin_view)
+
+        details_layout = QHBoxLayout()
+        layout.addLayout(details_layout)
+        forum_label = QLabel('<a href="http://www.foo.com/">Plugin Forum Thread</a>', self)
+        forum_label.setTextInteractionFlags(Qt.LinksAccessibleByMouse | Qt.LinksAccessibleByKeyboard)
+        forum_label.linkActivated.connect(self._forum_label_activated)
+        details_layout.addWidget(QLabel(_('Description')+':', self), 0, Qt.AlignLeft)
+        details_layout.addWidget(forum_label, 1, Qt.AlignRight)
+
+        self.description = QLabel(self)
+        self.description.setFrameStyle(QFrame.Panel | QFrame.Sunken)
+        self.description.setAlignment(Qt.AlignTop | Qt.AlignLeft)
+        self.description.setMinimumHeight(40)
+        layout.addWidget(self.description)
+
+        self.button_box = QDialogButtonBox(QDialogButtonBox.Close)
+        self.button_box.rejected.connect(self._close_clicked)
+        self.install_button = self.button_box.addButton(_('&Install'), QDialogButtonBox.AcceptRole)
+        self.install_button.setToolTip(_('Install the selected plugin'))
+        self.install_button.clicked.connect(self._install_clicked)
+        self.install_button.setEnabled(False)
+        self.configure_button = self.button_box.addButton(' '+_('&Customize plugin ')+' ', QDialogButtonBox.ResetRole)
+        self.configure_button.setToolTip(_('Customize the options for this plugin'))
+        self.configure_button.clicked.connect(self._configure_clicked)
+        self.configure_button.setEnabled(False)
+        layout.addWidget(self.button_box)
+
+    def _create_context_menu(self):
+        self.plugin_view.setContextMenuPolicy(Qt.ActionsContextMenu)
+        self.install_action = QAction(QIcon(I('plugins/plugin_upgrade_ok.png')), _('&Install'), self)
+        self.install_action.setToolTip(_('Install the selected plugin'))
+        self.install_action.triggered.connect(self._install_clicked)
+        self.install_action.setEnabled(False)
+        self.plugin_view.addAction(self.install_action)
+        self.history_action = QAction(QIcon(I('chapters.png')), _('Version &History'), self)
+        self.history_action.setToolTip(_('Show history of changes to this plugin'))
+        self.history_action.triggered.connect(self._history_clicked)
+        self.history_action.setEnabled(False)
+        self.plugin_view.addAction(self.history_action)
+        self.forum_action = QAction(QIcon(I('plugins/mobileread.png')), _('Plugin &Forum Thread'), self)
+        self.forum_action.triggered.connect(self._forum_label_activated)
+        self.forum_action.setEnabled(False)
+        self.plugin_view.addAction(self.forum_action)
+
+        sep1 = QAction(self)
+        sep1.setSeparator(True)
+        self.plugin_view.addAction(sep1)
+
+        self.toggle_enabled_action = QAction(_('Enable/&Disable plugin'), self)
+        self.toggle_enabled_action.setToolTip(_('Enable or disable this plugin'))
+        self.toggle_enabled_action.triggered.connect(self._toggle_enabled_clicked)
+        self.toggle_enabled_action.setEnabled(False)
+        self.plugin_view.addAction(self.toggle_enabled_action)
+        self.uninstall_action = QAction(_('&Remove plugin'), self)
+        self.uninstall_action.setToolTip(_('Uninstall the selected plugin'))
+        self.uninstall_action.triggered.connect(self._uninstall_clicked)
+        self.uninstall_action.setEnabled(False)
+        self.plugin_view.addAction(self.uninstall_action)
+
+        sep2 = QAction(self)
+        sep2.setSeparator(True)
+        self.plugin_view.addAction(sep2)
+
+        self.donate_enabled_action = QAction(QIcon(I('donate.png')), _('Donate to developer'), self)
+        self.donate_enabled_action.setToolTip(_('Donate to the developer of this plugin'))
+        self.donate_enabled_action.triggered.connect(self._donate_clicked)
+        self.donate_enabled_action.setEnabled(False)
+        self.plugin_view.addAction(self.donate_enabled_action)
+
+        sep3 = QAction(self)
+        sep3.setSeparator(True)
+        self.plugin_view.addAction(sep3)
+
+        self.configure_action = QAction(QIcon(I('config.png')), _('&Customize plugin'), self)
+        self.configure_action.setToolTip(_('Customize the options for this plugin'))
+        self.configure_action.triggered.connect(self._configure_clicked)
+        self.configure_action.setEnabled(False)
+        self.plugin_view.addAction(self.configure_action)
+
+    def _close_clicked(self):
+        # Force our toolbar/action to be updated based on uninstalled updates
+        if self.model:
+            update_plugins = filter(filter_upgradeable_plugins, self.model.display_plugins)
+            self.gui.recalc_update_label(len(update_plugins))
+        self.reject()
+
+    def _plugin_current_changed(self, current, previous):
+        if current.isValid():
+            actual_idx = self.proxy_model.mapToSource(current)
+            display_plugin = self.model.display_plugins[actual_idx.row()]
+            self.description.setText(display_plugin.description)
+            self.forum_link = display_plugin.forum_link
+            self.forum_action.setEnabled(bool(self.forum_link))
+            self.install_button.setEnabled(display_plugin.is_valid_to_install())
+            self.install_action.setEnabled(self.install_button.isEnabled())
+            self.uninstall_action.setEnabled(display_plugin.is_installed())
+            self.history_action.setEnabled(display_plugin.has_changelog)
+            self.configure_button.setEnabled(display_plugin.is_installed())
+            self.configure_action.setEnabled(self.configure_button.isEnabled())
+            self.toggle_enabled_action.setEnabled(display_plugin.is_installed())
+            self.donate_enabled_action.setEnabled(bool(display_plugin.donation_link))
+        else:
+            self.description.setText('')
+            self.forum_link = None
+            self.forum_action.setEnabled(False)
+            self.install_button.setEnabled(False)
+            self.install_action.setEnabled(False)
+            self.uninstall_action.setEnabled(False)
+            self.history_action.setEnabled(False)
+            self.configure_button.setEnabled(False)
+            self.configure_action.setEnabled(False)
+            self.toggle_enabled_action.setEnabled(False)
+            self.donate_enabled_action.setEnabled(False)
+
+    def _donate_clicked(self):
+        plugin = self._selected_display_plugin()
+        if plugin and plugin.donation_link:
+            open_url(QUrl(plugin.donation_link))
+
+    def _select_and_focus_view(self, change_selection=True):
+        if change_selection and self.plugin_view.model().rowCount() > 0:
+            self.plugin_view.selectRow(0)
+        else:
+            idx = self.plugin_view.selectionModel().currentIndex()
+            self._plugin_current_changed(idx, 0)
+        self.plugin_view.setFocus()
+
+    def _filter_combo_changed(self, idx):
+        self.proxy_model.set_filter_criteria(idx)
+        if idx == FILTER_NOT_INSTALLED:
+            self.plugin_view.sortByColumn(5, Qt.DescendingOrder)
+        else:
+            self.plugin_view.sortByColumn(0, Qt.AscendingOrder)
+        self._select_and_focus_view()
+
+    def _forum_label_activated(self):
+        if self.forum_link:
+            open_url(QUrl(self.forum_link))
+
+    def _selected_display_plugin(self):
+        idx = self.plugin_view.selectionModel().currentIndex()
+        actual_idx = self.proxy_model.mapToSource(idx)
+        return self.model.display_plugins[actual_idx.row()]
+
+    def _uninstall_plugin(self, name_to_remove):
+        if DEBUG:
+            prints('Removing plugin: ', name_to_remove)
+        remove_plugin(name_to_remove)
+        # Make sure that any other plugins that required this plugin
+        # to be uninstalled first have the requirement removed
+        for display_plugin in self.model.display_plugins:
+            # Make sure we update the status and display of the
+            # plugin we just uninstalled
+            if name_to_remove in display_plugin.uninstall_plugins:
+                if DEBUG:
+                    prints('Removing uninstall dependency for: ', display_plugin.name)
+                display_plugin.uninstall_plugins.remove(name_to_remove)
+            if display_plugin.name == name_to_remove:
+                if DEBUG:
+                    prints('Resetting plugin to uninstalled status: ', display_plugin.name)
+                display_plugin.installed_version = None
+                display_plugin.plugin = None
+                display_plugin.uninstall_plugins = []
+                if self.proxy_model.filter_criteria not in [FILTER_INSTALLED, FILTER_UPDATE_AVAILABLE]:
+                    self.model.refresh_plugin(display_plugin)
+
+    def _uninstall_clicked(self):
+        display_plugin = self._selected_display_plugin()
+        if not question_dialog(self, _('Are you sure?'), '<p>'+
+                   _('Are you sure you want to uninstall the <b>%s</b> plugin?')%display_plugin.name,
+                   show_copy_button=False):
+            return
+        self._uninstall_plugin(display_plugin.name)
+        if self.proxy_model.filter_criteria in [FILTER_INSTALLED, FILTER_UPDATE_AVAILABLE]:
+            self.model.reset()
+            self._select_and_focus_view()
+        else:
+            self._select_and_focus_view(change_selection=False)
+
+    def _install_clicked(self):
+        display_plugin = self._selected_display_plugin()
+        if not question_dialog(self, _('Install %s')%display_plugin.name, '<p>' + \
+                _('Installing plugins is a <b>security risk</b>. '
+                'Plugins can contain a virus/malware. '
+                    'Only install it if you got it from a trusted source.'
+                    ' Are you sure you want to proceed?'),
+                show_copy_button=False):
+            return
+
+        if display_plugin.uninstall_plugins:
+            uninstall_names = list(display_plugin.uninstall_plugins)
+            if DEBUG:
+                prints('Uninstalling plugin: ', ', '.join(uninstall_names))
+            for name_to_remove in uninstall_names:
+                self._uninstall_plugin(name_to_remove)
+
+        if DEBUG:
+            prints('Locating zip file for %s: %s'% (display_plugin.name, display_plugin.forum_link))
+        self.gui.status_bar.showMessage(_('Locating zip file for %s: %s') % (display_plugin.name, display_plugin.forum_link))
+        plugin_zip_url = self._read_zip_attachment_url(display_plugin.forum_link)
+        if not plugin_zip_url:
+            return error_dialog(self.gui, _('Install Plugin Failed'),
+                        _('Unable to locate a plugin zip file for <b>%s</b>') % display_plugin.name,
+                        det_msg=display_plugin.forum_link, show=True)
+
+        if DEBUG:
+            prints('Downloading plugin zip attachment: ', plugin_zip_url)
+        self.gui.status_bar.showMessage(_('Downloading plugin zip attachment: %s') % plugin_zip_url)
+        zip_path = self._download_zip(plugin_zip_url)
+
+        if DEBUG:
+            prints('Installing plugin: ', zip_path)
+        self.gui.status_bar.showMessage(_('Installing plugin: %s') % zip_path)
+
+        try:
+            try:
+                plugin = add_plugin(zip_path)
+            except NameConflict as e:
+                return error_dialog(self.gui, _('Already exists'),
+                        unicode(e), show=True)
+            # Check for any toolbars to add to.
+            widget = ConfigWidget(self.gui)
+            widget.gui = self.gui
+            widget.check_for_add_to_toolbars(plugin)
+            self.gui.status_bar.showMessage(_('Plugin installed: %s') % display_plugin.name)
+            info_dialog(self.gui, _('Success'),
+                    _('Plugin <b>{0}</b> successfully installed under <b>'
+                        ' {1} plugins</b>. You may have to restart calibre '
+                        'for the plugin to take effect.').format(plugin.name, plugin.type),
+                    show=True, show_copy_button=False)
+
+            display_plugin.plugin = plugin
+            # We cannot read the 'actual' version information as the plugin will not be loaded yet
+            display_plugin.installed_version = display_plugin.available_version
+        except:
+            if DEBUG:
+                prints('ERROR occurred while installing plugin: %s'%display_plugin.name)
+                traceback.print_exc()
+            error_dialog(self.gui, _('Install Plugin Failed'),
+                         _('A problem occurred while installing this plugin.'
+                           ' This plugin will now be uninstalled.'
+                           ' Please post the error message in details below into'
+                           ' the forum thread for this plugin and restart Calibre.'),
+                         det_msg=traceback.format_exc(), show=True)
+            if DEBUG:
+                prints('Due to error now uninstalling plugin: %s'%display_plugin.name)
+                remove_plugin(display_plugin.name)
+                display_plugin.plugin = None
+
+        display_plugin.uninstall_plugins = []
+        if self.proxy_model.filter_criteria in [FILTER_NOT_INSTALLED, FILTER_UPDATE_AVAILABLE]:
+            self.model.reset()
+            self._select_and_focus_view()
+        else:
+            self.model.refresh_plugin(display_plugin)
+            self._select_and_focus_view(change_selection=False)
+
+    def _history_clicked(self):
+        display_plugin = self._selected_display_plugin()
+        text = self._read_version_history_html(display_plugin.forum_link)
+        if text:
+            dlg = VersionHistoryDialog(self, display_plugin.name, text)
+            dlg.exec_()
+        else:
+            return error_dialog(self, _('Version history missing'),
+                _('Unable to find the version history for %s')%display_plugin.name,
+                show=True)
+
+    def _configure_clicked(self):
+        display_plugin = self._selected_display_plugin()
+        plugin = display_plugin.plugin
+        if not plugin.is_customizable():
+            return info_dialog(self, _('Plugin not customizable'),
+                _('Plugin: %s does not need customization')%plugin.name, show=True)
+        from calibre.customize import InterfaceActionBase
+        if isinstance(plugin, InterfaceActionBase) and not getattr(plugin,
+                'actual_iaction_plugin_loaded', False):
+            return error_dialog(self, _('Must restart'),
+                    _('You must restart calibre before you can'
+                        ' configure the <b>%s</b> plugin')%plugin.name, show=True)
+        plugin.do_user_config(self.parent())
+
+    def _toggle_enabled_clicked(self):
+        display_plugin = self._selected_display_plugin()
+        plugin = display_plugin.plugin
+        if not plugin.can_be_disabled:
+            return error_dialog(self,_('Plugin cannot be disabled'),
+                         _('The plugin: %s cannot be disabled')%plugin.name, show=True)
+        if is_disabled(plugin):
+            enable_plugin(plugin)
+        else:
+            disable_plugin(plugin)
+        self.model.refresh_plugin(display_plugin)
+
+    def _read_version_history_html(self, forum_link):
+        br = browser()
+        br.set_handle_gzip(True)
+        try:
+            raw = br.open_novisit(forum_link).read()
+            if not raw:
+                return None
+        except:
+            traceback.print_exc()
+            return None
+        raw = raw.decode('utf-8', errors='replace')
+        root = html.fromstring(raw)
+        spoiler_nodes = root.xpath('//div[@class="smallfont" and strong="Spoiler"]')
+        for spoiler_node in spoiler_nodes:
+            try:
+                if spoiler_node.getprevious() is None:
+                    # This is a spoiler node that has been indented using [INDENT]
+                    # Need to go up to parent div, then previous node to get header
+                    heading_node = spoiler_node.getparent().getprevious()
+                else:
+                    # This is a spoiler node after a BR tag from the heading
+                    heading_node = spoiler_node.getprevious().getprevious()
+                if heading_node is None:
+                    continue
+                if heading_node.text_content().lower().find('version history') != -1:
+                    div_node = spoiler_node.xpath('div')[0]
+                    text = html.tostring(div_node, method='html', encoding=unicode)
+                    return re.sub('<div\s.*?>', '<div>', text)
+            except:
+                if DEBUG:
+                    prints('======= MobileRead Parse Error =======')
+                    traceback.print_exc()
+                    prints(html.tostring(spoiler_node))
+        return None
+
+    def _read_zip_attachment_url(self, forum_link):
+        br = browser()
+        br.set_handle_gzip(True)
+        try:
+            raw = br.open_novisit(forum_link).read()
+            if not raw:
+                return None
+        except:
+            traceback.print_exc()
+            return None
+        raw = raw.decode('utf-8', errors='replace')
+        root = html.fromstring(raw)
+        attachment_nodes = root.xpath('//fieldset/table/tr/td/a')
+        for attachment_node in attachment_nodes:
+            try:
+                filename = attachment_node.text_content().lower()
+                if filename.find('.zip') != -1:
+                    full_url = MR_URL + attachment_node.attrib['href']
+                    return full_url
+            except:
+                if DEBUG:
+                    prints('======= MobileRead Parse Error =======')
+                    traceback.print_exc()
+                    prints(html.tostring(attachment_node))
+        return None
+
+    def _download_zip(self, plugin_zip_url):
+        from calibre.ptempfile import PersistentTemporaryFile
+        br = browser()
+        br.set_handle_gzip(True)
+        raw = br.open_novisit(plugin_zip_url).read()
+        pt = PersistentTemporaryFile('.zip')
+        pt.write(raw)
+        pt.close()
+        return pt.name

src/calibre/gui2/init.py

@@ -27,7 +27,6 @@ def partial(*args, **kwargs):
     _keep_refs.append(ans)
     return ans
 
-
 class LibraryViewMixin(object): # {{{
 
     def __init__(self, db):
@@ -145,6 +144,7 @@ class UpdateLabel(QLabel): # {{{
 
     def __init__(self, *args, **kwargs):
         QLabel.__init__(self, *args, **kwargs)
+        self.setCursor(Qt.PointingHandCursor)
 
     def contextMenuEvent(self, e):
         pass
@@ -182,14 +182,6 @@ class StatusBar(QStatusBar): # {{{
         self.defmsg.setText(self.default_message)
         self.clearMessage()
 
-    def new_version_available(self, ver, url):
-        msg = (u'<span style="color:red; font-weight: bold">%s: <a'
-               ' href="update:%s">%s<a></span>') % (
-                _('Update found'), ver, ver)
-        self.update_label.setText(msg)
-        self.update_label.setCursor(Qt.PointingHandCursor)
-        self.update_label.setVisible(True)
-
     def get_version(self):
         dv = os.environ.get('CALIBRE_DEVELOP_FROM', None)
         v = __version__
@@ -257,12 +249,6 @@ class LayoutMixin(object): # {{{
         self.setStatusBar(self.status_bar)
         self.status_bar.update_label.linkActivated.connect(self.update_link_clicked)
 
-    def update_link_clicked(self, url):
-        url = unicode(url)
-        if url.startswith('update:'):
-            version = url.partition(':')[-1]
-            self.update_found(version, force=True)
-
     def finalize_layout(self):
         self.status_bar.initialize(self.system_tray_icon)
         self.book_details.show_book_info.connect(self.iactions['Show Book Details'].show_book_info)

src/calibre/gui2/jobs.py

@@ -432,6 +432,10 @@ class JobsDialog(QDialog, Ui_JobsDialog):
             self.jobs_view.horizontalHeader().restoreState(QByteArray(state))
         except:
             pass
+        idx = self.jobs_view.model().index(0, 0)
+        if idx.isValid():
+            sm = self.jobs_view.selectionModel()
+            sm.select(idx, sm.ClearAndSelect|sm.Rows)
 
     def save_state(self):
         try:

src/calibre/gui2/metadata/single.py

@@ -388,6 +388,10 @@ class MetadataSingleDialogBase(ResizableDialog):
 
     def apply_changes(self):
         self.changed.add(self.book_id)
+        if self.db is None:
+            # break_cycles has already been called, don't know why this should
+            # happen but a user reported it
+            return True
         for widget in self.basic_metadata_widgets:
             try:
                 if not widget.commit(self.db, self.book_id):

src/calibre/gui2/metadata/single_download.py

@@ -236,6 +236,11 @@ class ResultsView(QTableView): # {{{
         self.resizeRowsToContents()
         self.resizeColumnsToContents()
         self.setFocus(Qt.OtherFocusReason)
+        idx = self.model().index(0, 0)
+        if idx.isValid() and self.model().rowCount() > 0:
+            self.show_details(idx)
+            sm = self.selectionModel()
+            sm.select(idx, sm.ClearAndSelect|sm.Rows)
 
     def currentChanged(self, current, previous):
         ret = QTableView.currentChanged(self, current, previous)
@@ -480,12 +485,6 @@ class IdentifyWidget(QWidget): # {{{
             return
 
         self.results_view.show_results(self.worker.results)
-
-        self.comments_view.show_data('''
-            <div style="margin-bottom:2ex">Found <b>%d</b> results</div>
-            <div>To see <b>details</b>, click on any result</div>''' %
-                len(self.worker.results))
-
         self.results_found.emit()
 
 

src/calibre/gui2/preferences/coloring.py

@@ -22,11 +22,7 @@ from calibre.library.coloring import (Rule, conditionable_columns,
 
 class ConditionEditor(QWidget): # {{{
 
-    def __init__(self, fm, parent=None):
+    ACTION_MAP = {
-        QWidget.__init__(self, parent)
-        self.fm = fm
-
-        self.action_map = {
             'bool' : (
                     (_('is true'), 'is true',),
                     (_('is false'), 'is false'),
@@ -61,10 +57,17 @@ class ConditionEditor(QWidget): # {{{
                 (_('is set'), 'is set'),
                 (_('is not set'), 'is not set'),
             ),
-        }
+    }
 
-        for x in ('float', 'rating', 'datetime'):
+    for x in ('float', 'rating', 'datetime'):
-            self.action_map[x] = self.action_map['int']
+        ACTION_MAP[x] = ACTION_MAP['int']
+
+
+    def __init__(self, fm, parent=None):
+        QWidget.__init__(self, parent)
+        self.fm = fm
+
+        self.action_map = self.ACTION_MAP
 
         self.l = l = QGridLayout(self)
         self.setLayout(l)
@@ -446,9 +449,15 @@ class RulesModel(QAbstractListModel): # {{{
 
     def condition_to_html(self, condition):
         c, a, v = condition
+        action_name = a
+        for actions in ConditionEditor.ACTION_MAP.itervalues():
+            for trans, ac in actions:
+                if ac == a:
+                    action_name = trans
+
         return (
             _('<li>If the <b>%s</b> column <b>%s</b> value: <b>%s</b>') %
-                (c, a, prepare_string_for_xml(v)))
+                (c, action_name, prepare_string_for_xml(v)))
 
 # }}}
 

src/calibre/gui2/preferences/metadata_sources.py

@@ -224,6 +224,20 @@ class FieldsModel(QAbstractListModel): # {{{
             Qt.Unchecked])
         msprefs['ignore_fields'] = list(ignored_fields.union(changed))
 
+    def user_default_state(self, field):
+        return (Qt.Unchecked if field in msprefs.get('user_default_ignore_fields',[])
+                    else Qt.Checked)
+
+    def select_user_defaults(self):
+        self.overrides = dict([(f, self.user_default_state(f)) for f in self.fields])
+        self.reset()
+
+    def commit_user_defaults(self):
+        default_ignored_fields = set([x for x in msprefs['user_default_ignore_fields'] if x not in
+            self.overrides])
+        changed = set([k for k, v in self.overrides.iteritems() if v ==
+            Qt.Unchecked])
+        msprefs['user_default_ignore_fields'] = list(default_ignored_fields.union(changed))
 
 # }}}
 
@@ -286,6 +300,9 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         self.select_all_button.clicked.connect(self.changed_signal)
         self.clear_all_button.clicked.connect(self.fields_model.clear_all)
         self.clear_all_button.clicked.connect(self.changed_signal)
+        self.select_default_button.clicked.connect(self.fields_model.select_user_defaults)
+        self.select_default_button.clicked.connect(self.changed_signal)
+        self.set_as_default_button.clicked.connect(self.fields_model.commit_user_defaults)
 
     def configure_plugin(self):
         for index in self.sources_view.selectionModel().selectedRows():

src/calibre/gui2/preferences/metadata_sources.ui

@@ -102,6 +102,26 @@
             </property>
            </widget>
           </item>
+          <item row="2" column="0">
+           <widget class="QPushButton" name="select_default_button">
+            <property name="text">
+             <string>&amp;Select default</string>
+            </property>
+             <property name="toolTip">
+             <string>Restore your own subset of checked fields that you define using the 'Set as default' button</string>
+            </property>
+           </widget>
+          </item>
+          <item row="2" column="1">
+           <widget class="QPushButton" name="set_as_default_button">
+            <property name="text">
+             <string>&amp;Set as default</string>
+            </property>
+             <property name="toolTip">
+             <string>Store the currently checked fields as a default you can restore using the 'Select default' button</string>
+            </property>
+           </widget>
+          </item>
          </layout>
         </widget>
        </item>

src/calibre/gui2/preferences/plugins.py

@@ -8,16 +8,16 @@ __docformat__ = 'restructuredtext en'
 import textwrap, os
 from collections import OrderedDict
 
-from PyQt4.Qt import Qt, QModelIndex, QAbstractItemModel, QVariant, QIcon, \
+from PyQt4.Qt import (Qt, QModelIndex, QAbstractItemModel, QVariant, QIcon,
-        QBrush
+        QBrush)
 
 from calibre.gui2.preferences import ConfigWidgetBase, test_widget
 from calibre.gui2.preferences.plugins_ui import Ui_Form
 from calibre.customize.ui import (initialized_plugins, is_disabled, enable_plugin,
                                  disable_plugin, plugin_customization, add_plugin,
                                  remove_plugin, NameConflict)
-from calibre.gui2 import NONE, error_dialog, info_dialog, choose_files, \
+from calibre.gui2 import (NONE, error_dialog, info_dialog, choose_files,
-        question_dialog, gprefs
+        question_dialog, gprefs)
 from calibre.utils.search_query_parser import SearchQueryParser
 from calibre.utils.icu import lower
 
@@ -217,6 +217,8 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         self.customize_plugin_button.clicked.connect(self.customize_plugin)
         self.remove_plugin_button.clicked.connect(self.remove_plugin)
         self.button_plugin_add.clicked.connect(self.add_plugin)
+        self.button_plugin_updates.clicked.connect(self.update_plugins)
+        self.button_plugin_new.clicked.connect(self.get_plugins)
         self.search.initialize('plugin_search_history',
                 help_text=_('Search for plugin'))
         self.search.search.connect(self.find)
@@ -353,6 +355,19 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
                          plugin.name + _(' cannot be removed. It is a '
                          'builtin plugin. Try disabling it instead.')).exec_()
 
+    def get_plugins(self):
+        self.update_plugins(not_installed=True)
+
+    def update_plugins(self, not_installed=False):
+        from calibre.gui2.dialogs.plugin_updater import (PluginUpdaterDialog,
+                                FILTER_UPDATE_AVAILABLE, FILTER_NOT_INSTALLED)
+        mode = FILTER_NOT_INSTALLED if not_installed else FILTER_UPDATE_AVAILABLE
+        d = PluginUpdaterDialog(self.gui, initial_filter=mode)
+        d.exec_()
+        self._plugin_model.populate()
+        self._plugin_model.reset()
+        self.changed_signal.emit()
+
     def reload_store_plugins(self):
         self.gui.load_store_plugins()
         if self.gui.iactions.has_key('Store'):

src/calibre/gui2/preferences/plugins.ui

@@ -113,16 +113,49 @@
     </layout>
    </item>
    <item>
-    <widget class="QPushButton" name="button_plugin_add">
+    <widget class="QFrame" name="frame">
-     <property name="text">
+     <property name="frameShape">
-      <string>&amp;Add a new plugin</string>
+      <enum>QFrame::HLine</enum>
-     </property>
-     <property name="icon">
-      <iconset resource="../../../../resources/images.qrc">
-       <normaloff>:/images/plugins.png</normaloff>:/images/plugins.png</iconset>
      </property>
     </widget>
    </item>
+   <item>
+    <layout class="QHBoxLayout" name="horizontalLayout_7">
+     <item>
+      <widget class="QPushButton" name="button_plugin_new">
+       <property name="text">
+        <string>Get &amp;new plugins</string>
+       </property>
+       <property name="icon">
+        <iconset resource="../../../../resources/images.qrc">
+         <normaloff>:/images/plugins/plugin_new.png</normaloff>:/images/plugins/plugin_new.png</iconset>
+       </property>
+      </widget>
+     </item>
+     <item>
+      <widget class="QPushButton" name="button_plugin_updates">
+       <property name="text">
+        <string>Check for &amp;updated plugins</string>
+       </property>
+       <property name="icon">
+        <iconset resource="../../../../resources/images.qrc">
+         <normaloff>:/images/plugins/plugin_updater.png</normaloff>:/images/plugins/plugin_updater.png</iconset>
+       </property>
+      </widget>
+     </item>
+     <item>
+      <widget class="QPushButton" name="button_plugin_add">
+       <property name="text">
+        <string>&amp;Load plugin from file</string>
+       </property>
+       <property name="icon">
+        <iconset resource="../../../../resources/images.qrc">
+         <normaloff>:/images/document_open.png</normaloff>:/images/document_open.png</iconset>
+       </property>
+      </widget>
+     </item>
+    </layout>
+   </item>
   </layout>
  </widget>
  <customwidgets>

src/calibre/gui2/tag_view.py

@@ -727,6 +727,15 @@ class TagTreeItem(object): # {{{
         else:
             self.tag.state = set_to
 
+    def all_children(self):
+        res = []
+        def recurse(nodes, res):
+            for t in nodes:
+                res.append(t)
+                recurse(t.children, res)
+        recurse(self.children, res)
+        return res
+
     def child_tags(self):
         res = []
         def recurse(nodes, res):
@@ -1269,6 +1278,7 @@ class TagsModel(QAbstractItemModel): # {{{
                                      category_icon = category_node.icon,
                                      category_key=category_node.category_key,
                                      icon_map=self.icon_state_map)
+                            sub_cat.tag.is_searchable = False
                             self.endInsertRows()
                     else: # by 'first letter'
                         cl = cl_list[idx]
@@ -1644,14 +1654,23 @@ class TagsModel(QAbstractItemModel): # {{{
             if node.tag.state:
                 if node.category_key == "news":
                     if node_searches[node.tag.state] == 'true':
-                        ans.append('tags:=news')
+                        ans.append('tags:"=' + _('News') + '"')
                     else:
-                        ans.append('( not tags:=news )')
+                        ans.append('( not tags:"=' + _('News') + '")')
                 else:
                     ans.append('%s:%s'%(node.category_key, node_searches[node.tag.state]))
 
             key = node.category_key
-            for tag_item in node.child_tags():
+            for tag_item in node.all_children():
+                if tag_item.type == TagTreeItem.CATEGORY:
+                    if self.collapse_model == 'first letter' and \
+                            tag_item.temporary and not key.startswith('@') \
+                            and tag_item.tag.state:
+                        if node_searches[tag_item.tag.state] == 'true':
+                            ans.append('%s:~^%s'%(key, tag_item.py_name))
+                        else:
+                            ans.append('(not %s:~^%s )'%(key, tag_item.py_name))
+                    continue
                 tag = tag_item.tag
                 if tag.state != TAG_SEARCH_STATES['clear']:
                     if tag.state == TAG_SEARCH_STATES['mark_minus'] or \

src/calibre/gui2/update.py

@@ -3,16 +3,30 @@ __copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
 
 import traceback
 
-from PyQt4.Qt import QThread, pyqtSignal, Qt, QUrl, QDialog, QGridLayout, \
+from PyQt4.Qt import (QThread, pyqtSignal, Qt, QUrl, QDialog, QGridLayout,
-        QLabel, QCheckBox, QDialogButtonBox, QIcon, QPixmap
+        QLabel, QCheckBox, QDialogButtonBox, QIcon, QPixmap)
 import mechanize
 
-from calibre.constants import __appname__, __version__, iswindows, isosx
+from calibre.constants import (__appname__, __version__, iswindows, isosx,
+        isportable)
 from calibre import browser
 from calibre.utils.config import prefs
 from calibre.gui2 import config, dynamic, open_url
+from calibre.gui2.dialogs.plugin_updater import get_plugin_updates_available
 
 URL = 'http://status.calibre-ebook.com/latest'
+NO_CALIBRE_UPDATE = '-0.0.0'
+VSEP = '|'
+
+def get_newest_version():
+    br = browser()
+    req = mechanize.Request(URL)
+    req.add_header('CALIBRE_VERSION', __version__)
+    req.add_header('CALIBRE_OS',
+            'win' if iswindows else 'osx' if isosx else 'oth')
+    req.add_header('CALIBRE_INSTALL_UUID', prefs['installation_uuid'])
+    version = br.open(req).read().strip()
+    return version
 
 class CheckForUpdates(QThread):
 
@@ -24,23 +38,29 @@ class CheckForUpdates(QThread):
 
     def run(self):
         while True:
+            calibre_update_version = NO_CALIBRE_UPDATE
+            plugins_update_found = 0
             try:
-                br = browser()
+                version = get_newest_version()
-                req = mechanize.Request(URL)
-                req.add_header('CALIBRE_VERSION', __version__)
-                req.add_header('CALIBRE_OS',
-                        'win' if iswindows else 'osx' if isosx else 'oth')
-                req.add_header('CALIBRE_INSTALL_UUID', prefs['installation_uuid'])
-                version = br.open(req).read().strip()
                 if version and version != __version__ and len(version) < 10:
-                    self.update_found.emit(version)
+                    calibre_update_version = version
             except:
                 traceback.print_exc()
+            try:
+                update_plugins = get_plugin_updates_available()
+                if update_plugins is not None:
+                    plugins_update_found = len(update_plugins)
+            except:
+                traceback.print_exc()
+            if (calibre_update_version != NO_CALIBRE_UPDATE or
+                    plugins_update_found > 0):
+                self.update_found.emit('%s%s%d'%(calibre_update_version,
+                    VSEP, plugins_update_found))
             self.sleep(self.INTERVAL)
 
 class UpdateNotification(QDialog):
 
-    def __init__(self, version, parent=None):
+    def __init__(self, calibre_version, plugin_updates, parent=None):
         QDialog.__init__(self, parent)
         self.resize(400, 250)
         self.l = QGridLayout()
@@ -54,7 +74,8 @@ class UpdateNotification(QDialog):
             'See the <a href="http://calibre-ebook.com/whats-new'
             '">new features</a>.') + '<p>'+_('Update <b>only</b> if one of the '
             'new features or bug fixes is important to you. '
-            'If the current version works well for you, do not update.'))%(__appname__, version))
+            'If the current version works well for you, do not update.'))%(
+                __appname__, calibre_version))
         self.label.setOpenExternalLinks(True)
         self.label.setWordWrap(True)
         self.setWindowTitle(_('Update available!'))
@@ -70,18 +91,30 @@ class UpdateNotification(QDialog):
         b = self.bb.addButton(_('&Get update'), self.bb.AcceptRole)
         b.setDefault(True)
         b.setIcon(QIcon(I('arrow-down.png')))
+        if plugin_updates > 0:
+            b = self.bb.addButton(_('Update &plugins'), self.bb.ActionRole)
+            b.setIcon(QIcon(I('plugins/plugin_updater.png')))
+            b.clicked.connect(self.get_plugins, type=Qt.QueuedConnection)
         self.bb.addButton(self.bb.Cancel)
         self.l.addWidget(self.bb, 2, 0, 1, -1)
         self.bb.accepted.connect(self.accept)
         self.bb.rejected.connect(self.reject)
-        dynamic.set('update to version %s'%version, False)
+        dynamic.set('update to version %s'%calibre_version, False)
+
+    def get_plugins(self):
+        from calibre.gui2.dialogs.plugin_updater import (PluginUpdaterDialog,
+            FILTER_UPDATE_AVAILABLE)
+        d = PluginUpdaterDialog(self.parent(),
+                initial_filter=FILTER_UPDATE_AVAILABLE)
+        d.exec_()
 
     def show_future(self, *args):
         config.set('new_version_notification', bool(self.cb.isChecked()))
 
     def accept(self):
-        url = 'http://calibre-ebook.com/download_'+\
+        url = ('http://calibre-ebook.com/download_' +
-            ('windows' if iswindows else 'osx' if isosx else 'linux')
+            ('portable' if isportable else 'windows' if iswindows
+                else 'osx' if isosx else 'linux'))
         open_url(QUrl(url))
 
         QDialog.accept(self)
@@ -89,21 +122,79 @@ class UpdateNotification(QDialog):
 class UpdateMixin(object):
 
     def __init__(self, opts):
+        self.last_newest_calibre_version = NO_CALIBRE_UPDATE
         if not opts.no_update_check:
             self.update_checker = CheckForUpdates(self)
             self.update_checker.update_found.connect(self.update_found,
                     type=Qt.QueuedConnection)
             self.update_checker.start()
 
-    def update_found(self, version, force=False):
+    def recalc_update_label(self, number_of_plugin_updates):
-        os = 'windows' if iswindows else 'osx' if isosx else 'linux'
+        self.update_found('%s%s%d'%(self.last_newest_calibre_version, VSEP,
-        url = 'http://calibre-ebook.com/download_%s'%os
+            number_of_plugin_updates), no_show_popup=True)
-        self.status_bar.new_version_available(version, url)
 
-        if force or (config.get('new_version_notification') and \
+    def update_found(self, version, force=False, no_show_popup=False):
-                dynamic.get('update to version %s'%version, True)):
+        try:
-            self._update_notification__ = UpdateNotification(version,
+            calibre_version, plugin_updates = version.split(VSEP)
-                    parent=self)
+            plugin_updates = int(plugin_updates)
-            self._update_notification__.show()
+        except:
+            traceback.print_exc()
+            return
+        self.last_newest_calibre_version = calibre_version
+        has_calibre_update = calibre_version and calibre_version != NO_CALIBRE_UPDATE
+        has_plugin_updates = plugin_updates > 0
+        self.plugin_update_found(plugin_updates)
+
+        if not has_calibre_update and not has_plugin_updates:
+            self.status_bar.update_label.setVisible(False)
+            return
+        if has_calibre_update:
+            plt = u''
+            if has_plugin_updates:
+                plt = _(' (%d plugin updates)')%plugin_updates
+            msg = (u'<span style="color:red; font-weight: bold">%s: '
+                    u'<a href="update:%s">%s%s</a></span>') % (
+                        _('Update found'), version, calibre_version, plt)
+        else:
+            msg = (u'<a href="update:%s">%d %s</a>')%(version, plugin_updates,
+                    _('updated plugins'))
+        self.status_bar.update_label.setText(msg)
+        self.status_bar.update_label.setVisible(True)
 
 
+        if has_calibre_update:
+            if (force or (config.get('new_version_notification') and
+                    dynamic.get('update to version %s'%calibre_version, True))):
+                if not no_show_popup:
+                    self._update_notification__ = UpdateNotification(calibre_version,
+                            plugin_updates, parent=self)
+                    self._update_notification__.show()
+        elif has_plugin_updates:
+            if force:
+                from calibre.gui2.dialogs.plugin_updater import (PluginUpdaterDialog,
+                    FILTER_UPDATE_AVAILABLE)
+                d = PluginUpdaterDialog(self,
+                        initial_filter=FILTER_UPDATE_AVAILABLE)
+                d.exec_()
+
+    def plugin_update_found(self, number_of_updates):
+        # Change the plugin icon to indicate there are updates available
+        plugin = self.iactions.get('Plugin Updater', None)
+        if not plugin:
+            return
+        if number_of_updates:
+            plugin.qaction.setText(_('Plugin Updates')+'*')
+            plugin.qaction.setIcon(QIcon(I('plugins/plugin_updater_updates.png')))
+            plugin.qaction.setToolTip(
+                _('There are %d plugin updates available')%number_of_updates)
+        else:
+            plugin.qaction.setText(_('Plugin Updates'))
+            plugin.qaction.setIcon(QIcon(I('plugins/plugin_updater.png')))
+            plugin.qaction.setToolTip(_('Install and configure user plugins'))
+
+    def update_link_clicked(self, url):
+        url = unicode(url)
+        if url.startswith('update:'):
+            version = url[len('update:'):]
+            self.update_found(version, force=True)
+

src/calibre/gui2/widgets.py

@@ -5,15 +5,15 @@ Miscellaneous widgets used in the GUI
 '''
 import re, traceback
 
-from PyQt4.Qt import QIcon, QFont, QLabel, QListWidget, QAction, \
+from PyQt4.Qt import (QIcon, QFont, QLabel, QListWidget, QAction,
-                        QListWidgetItem, QTextCharFormat, QApplication, \
+                        QListWidgetItem, QTextCharFormat, QApplication,
-                        QSyntaxHighlighter, QCursor, QColor, QWidget, \
+                        QSyntaxHighlighter, QCursor, QColor, QWidget,
-                        QPixmap, QSplitterHandle, QToolButton, \
+                        QPixmap, QSplitterHandle, QToolButton,
-                        QAbstractListModel, QVariant, Qt, SIGNAL, pyqtSignal, \
+                        QAbstractListModel, QVariant, Qt, SIGNAL, pyqtSignal,
-                        QRegExp, QSettings, QSize, QSplitter, \
+                        QRegExp, QSettings, QSize, QSplitter,
-                        QPainter, QLineEdit, QComboBox, QPen, QGraphicsScene, \
+                        QPainter, QLineEdit, QComboBox, QPen, QGraphicsScene,
-                        QMenu, QStringListModel, QCompleter, QStringList, \
+                        QMenu, QStringListModel, QCompleter, QStringList,
-                        QTimer, QRect, QFontDatabase, QGraphicsView
+                        QTimer, QRect, QFontDatabase, QGraphicsView)
 
 from calibre.gui2 import NONE, error_dialog, pixmap_to_data, gprefs
 from calibre.gui2.filename_pattern_ui import Ui_Form
@@ -21,12 +21,12 @@ from calibre import fit_image
 from calibre.ebooks import BOOK_EXTENSIONS
 from calibre.utils.config import prefs, XMLConfig, tweaks
 from calibre.gui2.progress_indicator import ProgressIndicator as _ProgressIndicator
-from calibre.gui2.dnd import dnd_has_image, dnd_get_image, dnd_get_files, \
+from calibre.gui2.dnd import (dnd_has_image, dnd_get_image, dnd_get_files,
-    IMAGE_EXTENSIONS, dnd_has_extension, DownloadDialog
+    IMAGE_EXTENSIONS, dnd_has_extension, DownloadDialog)
 
 history = XMLConfig('history')
 
-class ProgressIndicator(QWidget):
+class ProgressIndicator(QWidget): # {{{
 
     def __init__(self, *args):
         QWidget.__init__(self, *args)
@@ -57,8 +57,9 @@ class ProgressIndicator(QWidget):
     def stop(self):
         self.pi.stopAnimation()
         self.setVisible(False)
+# }}}
 
-class FilenamePattern(QWidget, Ui_Form):
+class FilenamePattern(QWidget, Ui_Form): # {{{
 
     changed_signal = pyqtSignal()
 
@@ -148,8 +149,9 @@ class FilenamePattern(QWidget, Ui_Form):
 
         return pat
 
+# }}}
 
-class FormatList(QListWidget):
+class FormatList(QListWidget): # {{{
     DROPABBLE_EXTENSIONS = BOOK_EXTENSIONS
     formats_dropped = pyqtSignal(object, object)
     delete_format = pyqtSignal()
@@ -188,6 +190,8 @@ class FormatList(QListWidget):
         else:
             return QListWidget.keyPressEvent(self, event)
 
+# }}}
+
 class ImageDropMixin(object): # {{{
     '''
     Adds support for dropping images onto widgets and a context menu for
@@ -262,7 +266,7 @@ class ImageDropMixin(object): # {{{
                     pixmap_to_data(pmap))
 # }}}
 
-class ImageView(QWidget, ImageDropMixin):
+class ImageView(QWidget, ImageDropMixin): # {{{
 
     BORDER_WIDTH = 1
     cover_changed = pyqtSignal(object)
@@ -314,8 +318,9 @@ class ImageView(QWidget, ImageDropMixin):
             p.drawRect(target)
         #p.drawRect(self.rect())
         p.end()
+# }}}
 
-class CoverView(QGraphicsView, ImageDropMixin):
+class CoverView(QGraphicsView, ImageDropMixin): # {{{
 
     cover_changed = pyqtSignal(object)
 
@@ -333,7 +338,9 @@ class CoverView(QGraphicsView, ImageDropMixin):
         self.scene.addPixmap(pmap)
         self.setScene(self.scene)
 
-class FontFamilyModel(QAbstractListModel):
+# }}}
+
+class FontFamilyModel(QAbstractListModel): # {{{
 
     def __init__(self, *args):
         QAbstractListModel.__init__(self, *args)
@@ -371,7 +378,9 @@ class FontFamilyModel(QAbstractListModel):
 
     def index_of(self, family):
         return self.families.index(family.strip())
+# }}}
 
+# BasicList {{{
 class BasicListItem(QListWidgetItem):
 
     def __init__(self, text, user_data=None):
@@ -404,9 +413,9 @@ class BasicList(QListWidget):
     def items(self):
         for i in range(self.count()):
             yield self.item(i)
+# }}}
 
-
+class LineEditECM(object): # {{{
-class LineEditECM(object):
 
     '''
     Extend the context menu of a QLineEdit to include more actions.
@@ -449,8 +458,9 @@ class LineEditECM(object):
         from calibre.utils.icu import capitalize
         self.setText(capitalize(unicode(self.text())))
 
+# }}}
 
-class EnLineEdit(LineEditECM, QLineEdit):
+class EnLineEdit(LineEditECM, QLineEdit): # {{{
 
     '''
     Enhanced QLineEdit.
@@ -459,9 +469,9 @@ class EnLineEdit(LineEditECM, QLineEdit):
     '''
 
     pass
+# }}}
 
-
+class ItemsCompleter(QCompleter): # {{{
-class ItemsCompleter(QCompleter):
 
     '''
     A completer object that completes a list of tags. It is used in conjunction
@@ -486,8 +496,9 @@ class ItemsCompleter(QCompleter):
         model = QStringListModel(items, self)
         self.setModel(model)
 
+# }}}
 
-class CompleteLineEdit(EnLineEdit):
+class CompleteLineEdit(EnLineEdit):  # {{{
 
     '''
     A QLineEdit that can complete parts of text separated by separator.
@@ -550,8 +561,9 @@ class CompleteLineEdit(EnLineEdit):
         self.setText(complete_text_pat % (before_text[:cursor_pos - prefix_len], text, self.separator, after_text))
         self.setCursorPosition(cursor_pos - prefix_len + len(text) + len_extra)
 
+# }}}
 
-class EnComboBox(QComboBox):
+class EnComboBox(QComboBox): # {{{
 
     '''
     Enhanced QComboBox.
@@ -575,7 +587,9 @@ class EnComboBox(QComboBox):
             idx = 0
         self.setCurrentIndex(idx)
 
-class CompleteComboBox(EnComboBox):
+# }}}
+
+class CompleteComboBox(EnComboBox): # {{{
 
     def __init__(self, *args):
         EnComboBox.__init__(self, *args)
@@ -590,8 +604,9 @@ class CompleteComboBox(EnComboBox):
     def set_space_before_sep(self, space_before):
         self.lineEdit().set_space_before_sep(space_before)
 
+# }}}
 
-class HistoryLineEdit(QComboBox):
+class HistoryLineEdit(QComboBox): # {{{
 
     lost_focus = pyqtSignal()
 
@@ -638,7 +653,9 @@ class HistoryLineEdit(QComboBox):
         QComboBox.focusOutEvent(self, e)
         self.lost_focus.emit()
 
-class ComboBoxWithHelp(QComboBox):
+# }}}
+
+class ComboBoxWithHelp(QComboBox): # {{{
     '''
     A combobox where item 0 is help text. CurrentText will return '' for item 0.
     Be sure to always fetch the text with currentText. Don't use the signals
@@ -685,8 +702,9 @@ class ComboBoxWithHelp(QComboBox):
         QComboBox.hidePopup(self)
         self.set_state()
 
+# }}}
 
-class EncodingComboBox(QComboBox):
+class EncodingComboBox(QComboBox): # {{{
     '''
     A combobox that holds text encodings support
     by Python. This is only populated with the most
@@ -709,8 +727,9 @@ class EncodingComboBox(QComboBox):
         for item in self.ENCODINGS:
             self.addItem(item)
 
+# }}}
 
-class PythonHighlighter(QSyntaxHighlighter):
+class PythonHighlighter(QSyntaxHighlighter): # {{{
 
     Rules = []
     Formats = {}
@@ -948,6 +967,9 @@ class PythonHighlighter(QSyntaxHighlighter):
         QSyntaxHighlighter.rehighlight(self)
         QApplication.restoreOverrideCursor()
 
+# }}}
+
+# Splitter {{{
 class SplitterHandle(QSplitterHandle):
 
     double_clicked = pyqtSignal(object)
@@ -1179,3 +1201,5 @@ class Splitter(QSplitter):
 
     # }}}
 
+# }}}
+

src/calibre/library/caches.py

@@ -145,7 +145,7 @@ def _match(query, value, matchkind):
                             return True
                 elif query == t:
                     return True
-            elif ((matchkind == REGEXP_MATCH and re.search(query, t, re.I)) or ### search unanchored
+            elif ((matchkind == REGEXP_MATCH and re.search(query, t, re.I|re.UNICODE)) or ### search unanchored
                   (matchkind == CONTAINS_MATCH and query in t)):
                     return True
         except re.error:

src/calibre/library/catalog.py

@@ -3,7 +3,7 @@
 __license__   = 'GPL v3'
 __copyright__ = '2010, Greg Riker'
 
-import codecs, datetime, htmlentitydefs, os, re, shutil, time, zlib
+import codecs, datetime, htmlentitydefs, os, re, shutil, zlib
 from collections import namedtuple
 from copy import deepcopy
 from xml.sax.saxutils import escape
@@ -25,7 +25,7 @@ from calibre.utils.html2text import html2text
 from calibre.utils.icu import capitalize
 from calibre.utils.logging import default_log as log
 from calibre.utils.magick.draw import thumbnail
-from calibre.utils.zipfile import ZipFile, ZipInfo
+from calibre.utils.zipfile import ZipFile
 
 FIELDS = ['all', 'title', 'title_sort', 'author_sort', 'authors', 'comments',
           'cover', 'formats','id', 'isbn', 'ondevice', 'pubdate', 'publisher',
@@ -149,6 +149,15 @@ class CSV_XML(CatalogPlugin): # {{{
                     elif field == 'comments':
                         item = item.replace(u'\r\n',u' ')
                         item = item.replace(u'\n',u' ')
+
+                    # Convert HTML to markdown text
+                    if type(item) is unicode:
+                        opening_tag = re.search('<(\w+)(\x20|>)',item)
+                        if opening_tag:
+                            closing_tag = re.search('<\/%s>$' % opening_tag.group(1), item)
+                            if closing_tag:
+                                item = html2text(item)
+
                     outstr.append(u'"%s"' % unicode(item).replace('"','""'))
 
                 outfile.write(u','.join(outstr) + u'\n')
@@ -4695,24 +4704,33 @@ Author '{0}':
             to be replaced.
             '''
 
+            def open_archive(mode='r'):
+                try:
+                    return ZipFile(self.__archive_path, mode=mode)
+                except:
+                    # Happens on windows if the file is opened by another
+                    # process
+                    pass
+
             # Generate crc for current cover
             #self.opts.log.info(" generateThumbnail():")
-            data = open(title['cover'], 'rb').read()
+            with open(title['cover'], 'rb') as f:
+                data = f.read()
             cover_crc = hex(zlib.crc32(data))
 
             # Test cache for uuid
-            with ZipFile(self.__archive_path, mode='r') as zfr:
+            zf = open_archive()
-                try:
+            if zf is not None:
-                    t_info = zfr.getinfo(title['uuid'])
+                with zf:
-                except:
+                    try:
-                    pass
+                        zf.getinfo(title['uuid']+cover_crc)
-                else:
+                    except:
-                    if t_info.comment == cover_crc:
+                        pass
+                    else:
                         # uuid found in cache with matching crc
-                        thumb_data = zfr.read(title['uuid'])
+                        thumb_data = zf.read(title['uuid']+cover_crc)
-                        zfr.extract(title['uuid'],image_dir)
+                        with open(os.path.join(image_dir, thumb_file), 'wb') as f:
-                        os.rename(os.path.join(image_dir,title['uuid']),
+                            f.write(thumb_data)
-                                    os.path.join(image_dir,thumb_file))
                         return
 
 
@@ -4723,10 +4741,13 @@ Author '{0}':
                 f.write(thumb_data)
 
             # Save thumb to archive
-            t_info = ZipInfo(title['uuid'],time.localtime()[0:6])
+            if zf is not None: # Ensure that the read succeeded
-            t_info.comment = cover_crc
+                # If we failed to open the zip file for reading,
-            with ZipFile(self.__archive_path, mode='a') as zfw:
+                # we dont know if it contained the thumb or not
-                zfw.writestr(t_info, thumb_data)
+                zf = open_archive('a')
+                if zf is not None:
+                    with zf:
+                        zf.writestr(title['uuid']+cover_crc, thumb_data)
 
         def getFriendlyGenreTag(self, genre):
             # Find the first instance of friendly_tag matching genre

src/calibre/library/database2.py

@@ -201,16 +201,14 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
             dbprefs = DBPrefs(self)
             for key in default_prefs:
                 # be sure that prefs not to be copied are listed below
-                if key in ['news_to_be_synced']:
+                if key in frozenset(['news_to_be_synced']):
                     continue
-                try:
+                dbprefs[key] = default_prefs[key]
-                    dbprefs[key] = default_prefs[key]
+            if 'field_metadata' in default_prefs:
-                except:
+                fmvals = [f for f in default_prefs['field_metadata'].values() if f['is_custom']]
-                    pass # ignore options that don't exist anymore
+                for f in fmvals:
-            fmvals = [f for f in default_prefs['field_metadata'].values() if f['is_custom']]
+                    self.create_custom_column(f['label'], f['name'], f['datatype'],
-            for f in fmvals:
+                            f['is_multiple'] is not None, f['is_editable'], f['display'])
-                self.create_custom_column(f['label'], f['name'], f['datatype'],
-                        f['is_multiple'] is not None, f['is_editable'], f['display'])
         self.initialize_dynamic()
 
     def get_property(self, idx, index_is_id=False, loc=-1):

src/calibre/library/restore.py

@@ -24,6 +24,7 @@ NON_EBOOK_EXTENSIONS = frozenset([
 class RestoreDatabase(LibraryDatabase2):
 
     PATH_LIMIT = 10
+    WINDOWS_LIBRARY_PATH_LIMIT = 180
 
     def set_path(self, *args, **kwargs):
         pass

src/calibre/manual/conversion.rst

@@ -657,6 +657,7 @@ Some limitations of PDF input are:
     * Some PDFs store their images upside down with a rotation instruction, |app| currently doesn't support that instruction, so the images will be rotated in the output as well. 
     * Links and Tables of Contents are not supported
     * PDFs that use embedded non-unicode fonts to represent non-English characters will result in garbled output for those characters
+    * Some PDFs are made up of photographs of the page with OCRed text behind them. In such cases |app| uses the OCRed text, which can be very different from what you see when you view the PDF file
 
 To re-iterate **PDF is a really, really bad** format to use as input. If you absolutely must use PDF, then be prepared for an
 output ranging anywhere from decent to unusable, depending on the input PDF.

src/calibre/manual/develop.rst

@@ -28,7 +28,7 @@ For example, adding support for a new device to |app| typically involves writing
 a device driver plugin. You can browse the 
 `built-in drivers <http://bazaar.launchpad.net/%7Ekovid/calibre/trunk/files/head%3A/src/calibre/devices/>`_. Similarly, adding support 
 for new conversion formats involves writing input/output format plugins. Another example of the modular design is the :ref:`recipe system <news>` for 
-fetching news. 
+fetching news. For more examples of plugins designed to add features to |app|, see the `plugin index <http://www.mobileread.com/forums/showthread.php?t=118764>`_.
 
 Code layout
 ^^^^^^^^^^^^^^
@@ -36,10 +36,21 @@ Code layout
 All the |app| python code is in the ``calibre`` package. This package contains the following main sub-packages
 
     * devices - All the device drivers. Just look through some of the built-in drivers to get an idea for how they work.
-    * ebooks  - All the ebook conversion code. A good starting point is ``calibre.ebooks.conversion.cli`` which is the
+
-      module powering the :command:`ebook-convert` command.
+      * For details, see: devices.interface which defines the interface supported by device drivers and devices.usbms which
-    * library - The database backed and the content server. 
+        defines a generic driver that connects to a USBMS device. All USBMS based drivers in calibre inherit from it.
-    * gui2 - The Graphical User Interface. 
+
+    * ebooks  - All the ebook conversion/metadata code. A good starting point is ``calibre.ebooks.conversion.cli`` which is the
+      module powering the :command:`ebook-convert` command. The conversion process is controlled via conversion.plumber.
+      The format independent code is all in ebooks.oeb and the format dependent stuff is in ebooks.format_name.
+
+        * Metadata reading writing and downloading is all in ebooks.metadata
+
+    * library - The database backed and the content server. See library.database2 for the interface to the calibre library. library.server is the calibre Content Server.
+    * gui2 - The Graphical User Interface. GUI initialization happens in gui2.main and gui2.ui. The ebook-viewer is in gui2.viewer.
+
+If you need help understanding the code, post in the `development forum <http://www.mobileread.com/forums/forumdisplay.php?f=240>`_
+and you will most likely get help from one of |app|'s many developers.
 
 Getting the code
 ------------------
@@ -82,9 +93,9 @@ Now whenever you commit changes to your branch with the command::
 
     bzr commit -m "Comment describing your change"
 
-I can merge it directly from you branch into the main |app| source tree. You should also subscribe to the |app|
+I can merge it directly from you branch into the main |app| source tree. You should also keep an eye on the |app|
-developers mailing list `calibre-devs <https://launchpad.net/~calibre-devs>`_. Before making major changes, you should
+`development forum <http://www.mobileread.com/forums/forumdisplay.php?f=240>`. Before making major changes, you should
-discuss them on the mailing list or the #calibre IRC channel on Freenode to ensure that the changes will be accepted once you're done. 
+discuss them in the forum or contact Kovid directly (his email address is all over the source code). 
 
 Windows development environment
 ---------------------------------

src/calibre/manual/faq.rst

@@ -131,7 +131,7 @@ Follow these steps to find the problem:
     * Make sure that you are connecting only a single device to your computer at a time. Do not have another |app| supported device like an iPhone/iPad etc. at the same time.
     * If you are connecting an Apple iDevice (iPad, iPod Touch, iPhone), use the 'Connect to iTunes' method in the 'Getting started' instructions in `Calibre + Apple iDevices: Start here <http://www.mobileread.com/forums/showthread.php?t=118559>`_.
     * Make sure you are running the latest version of |app|. The latest version can always be downloaded from `the calibre website <http://calibre-ebook.com/download>`_.
-    * Ensure your operating system is seeing the device. That is, the device should be mounted as a disk that you can access using Windows explorer or whatever the file management program on your computer is.
+    * Ensure your operating system is seeing the device. That is, the device should be mounted as a disk, that you can access using Windows explorer or whatever the file management program on your computer is. On Windows your device **must have been assigned a drive letter**, like K:.
     * In calibre, go to Preferences->Plugins->Device Interface plugin and make sure the plugin for your device is enabled, the plugin icon next to it should be green when it is enabled.
     * If all the above steps fail, go to Preferences->Miscellaneous and click debug device detection with your device attached and post the output as a ticket on `the calibre bug tracker <http://bugs.calibre-ebook.com>`_.
 
@@ -515,7 +515,7 @@ Downloading from the internet can sometimes result in a corrupted download. If t
     * Try rebooting your computer and running a registry cleaner like `Wise registry cleaner <http://www.wisecleaner.com>`_.
     * Try downloading the installer with an alternate browser. For example if you are using Internet Explorer, try using Firefox or Chrome instead.
       
-Best place to ask for more help is in the `forums <http://www.mobileread.com/forums/forumdisplay.php?f=166>`_.
+If you still cannot get the installer to work and you are on windows, you can use the `calibre portable install <http://calibre-ebook.com/download_portable>`_, which does not need an installer (it is just a zip file).
 
 My antivirus program claims |app| is a virus/trojan?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~