diff --git a/Changelog.yaml b/Changelog.yaml index b297823841..c5eadc5e65 100644 --- a/Changelog.yaml +++ b/Changelog.yaml @@ -19,9 +19,93 @@ # new recipes: # - title: -# - title: "Launch of a new website that catalogues DRM free books. http://drmfree.calibre-ebook.com" -# description: "A growing catalogue of DRM free books. Books that you actually own after buying instead of renting." -# type: major +- version: 0.7.48 + date: 2011-03-04 + + new features: + - title: "Changes to the internal database structure used by calibre" + description: > + "These changes will allow calibre, in the future, to support book language, arbitrary book identifiers and keep track of when the metadata for a book was last modified. WARNING: Because of these changes, if you downgrade calibre versions after upgrading to 0.7.48, you will lose any changes you make to the ISBN of book entries in your calibre database, so do not downgrade unless you really have to. Also note that the first time you start calibre after this update, the startup will be slow as the database structure is being changed." + + - title: "Launch of a new website that catalogues DRM free ebooks. http://drmfree.calibre-ebook.com" + description: "A growing catalogue of DRM free ebooks. Ebooks that you actually own after paying, instead of just renting." + type: major + + - title: "News download: Add an option to keep at most x issues of a particular periodical in the calibre library. Use the Advanced tab in the Fetch news dialog for your news source to set this option." + tickets: [9168] + + - title: "You can now right click on the cover in the book details panel to copy/paste a new cover." + tickets: [9255] + + - title: "Add an entry to the add books drop down menu to easily add formats to an existing book record" + + - title: "Tag browser: Clicking on a nested category now searches for the category alone. Clicking twice searches for the category and all its descendants and so on." + tickets: [9166, 9169] + + - title: "Add a button to the Manage authors dialog to copy author sort values to author" + + - title: "Decrease startup times on large libraries by using a faster algorithm to parse stored dates" + + - title: "Add quick create links to easily create custom columns of commonly used types to the add custom column dialog" + + - title: "Allow drag drop of images to change cover in book details window." + tickets: [9226] + + - title: "Device susbsytem: Create a drive info file named driveinfo.calibre in the root of each device drive for USB connected devices. This file contains various useful data. API Change: The open method of the device plugins now accepts an extra parameter library_uuid which is the id of the calibre library connected tot eh device" + + bug fixes: + - title: "Conversion pipeline: Fix regression in 0.7.46 that caused loss of some CSS information when converting HTML produced by Microsoft Word. Also remove empty tags from microsoft namespaces when parsing HTML" + + - title: "Try harder to ensure that the worker log temporary files are deleted in windows" + + - title: "CHM Input: Handle CHM files that dont specify a topics file." + tickets: [9253] + + - title: "Fix regression that caused memory leak in Tag Browser. This would show up as the memory usage of calibre increasing when switching libraries." + tickets: [9246] + + - title: "Fix bug that caused preferences->behavior to not show the output format set by the welcome wizard, and instead default to showing EPUB" + + - title: "Fix bug that caused wrong books to be deleted from library if you choose 'delete from library and device' while the library is sorted by the On device column" + + - title: "MOBI Input: Ignore all ASCII control codes except CR, NL and Tab." + tickets: [9219] + + improved recipes: + - Credit Slips + - Seattle Times + - MacWorld + - Austin Statesman + - EPL Talk + - Gawker + - Deadspin + + new recipes: + - title: "Thai Post Today and Daily Post" + author: "Chotechai P." + + - title: "RBC.ru" + author: Chewi + + - title: Helsingin Sanomat + author: oneillpt + + - title: "LWN Weekly" + author: David Cavalca + + - title: "New York Times Sports and Technology Blogs" + author: rylsfan + + - title: "Historia and Buctaras" + author: Silviu Cotoara + + - title: "Buffalo News" + author: ChappyOnIce + + - title: "Dotpod" + author: Federico Escalada + + - version: 0.7.47 date: 2011-02-25 @@ -90,7 +174,7 @@ author: Ricardo Jurado - title: "Various Romanian news sources" - author: Silviu Coatara + author: Silviu Cotoara - title: "Osnews.pl and SwiatCzytnikow" author: Tomasz Dlugosz diff --git a/resources/default_tweaks.py b/resources/default_tweaks.py index 2303c6c108..38c1685b7c 100644 --- a/resources/default_tweaks.py +++ b/resources/default_tweaks.py @@ -349,3 +349,9 @@ public_smtp_relay_delay = 301 # after a restart of calibre. draw_hidden_section_indicators = True +#: The maximum width and height for covers saved in the calibre library +# All covers in the calibre library will be resized, preserving aspect ratio, +# to fit within this size. This is to prevent slowdowns caused by extremely +# large covers +maximum_cover_size = (1200, 1600) + diff --git a/resources/images/id_card.png b/resources/images/id_card.png new file mode 100644 index 0000000000..80ac5fda11 Binary files /dev/null and b/resources/images/id_card.png differ diff --git a/resources/images/news/cotidianul.png b/resources/images/news/cotidianul.png new file mode 100644 index 0000000000..2e57dbde54 Binary files /dev/null and b/resources/images/news/cotidianul.png differ diff --git a/resources/images/news/credit_slips.png b/resources/images/news/credit_slips.png new file mode 100644 index 0000000000..50ac1dc02e Binary files /dev/null and b/resources/images/news/credit_slips.png differ diff --git a/resources/images/news/ele.png b/resources/images/news/ele.png new file mode 100644 index 0000000000..82f66b5caa Binary files /dev/null and b/resources/images/news/ele.png differ diff --git a/resources/images/news/felicia.png b/resources/images/news/felicia.png new file mode 100644 index 0000000000..4bc1fd35d8 Binary files /dev/null and b/resources/images/news/felicia.png differ diff --git a/resources/images/news/financiarul.png b/resources/images/news/financiarul.png new file mode 100644 index 0000000000..1d91a72a34 Binary files /dev/null and b/resources/images/news/financiarul.png differ diff --git a/resources/images/news/hitro.png b/resources/images/news/hitro.png new file mode 100644 index 0000000000..75c08a1c25 Binary files /dev/null and b/resources/images/news/hitro.png differ diff --git a/resources/images/news/imperatortravel.png b/resources/images/news/imperatortravel.png new file mode 100644 index 0000000000..c459759ed0 Binary files /dev/null and b/resources/images/news/imperatortravel.png differ diff --git a/resources/images/news/kamikaze.png b/resources/images/news/kamikaze.png new file mode 100644 index 0000000000..49ef2f50a1 Binary files /dev/null and b/resources/images/news/kamikaze.png differ diff --git a/resources/images/news/kompiutierra.png b/resources/images/news/kompiutierra.png new file mode 100644 index 0000000000..272e3d905f Binary files /dev/null and b/resources/images/news/kompiutierra.png differ diff --git a/resources/images/news/lwn_weekly.png b/resources/images/news/lwn_weekly.png new file mode 100644 index 0000000000..0fc654add9 Binary files /dev/null and b/resources/images/news/lwn_weekly.png differ diff --git a/resources/images/news/monden.png b/resources/images/news/monden.png new file mode 100644 index 0000000000..fcf8ad42ae Binary files /dev/null and b/resources/images/news/monden.png differ diff --git a/resources/images/news/nytimes_sports.png b/resources/images/news/nytimes_sports.png new file mode 100644 index 0000000000..b587be8de0 Binary files /dev/null and b/resources/images/news/nytimes_sports.png differ diff --git a/resources/images/news/nytimes_tech.png b/resources/images/news/nytimes_tech.png new file mode 100644 index 0000000000..64ff8b5eb2 Binary files /dev/null and b/resources/images/news/nytimes_tech.png differ diff --git a/resources/images/news/promotor.png b/resources/images/news/promotor.png new file mode 100644 index 0000000000..a479cf135b Binary files /dev/null and b/resources/images/news/promotor.png differ diff --git a/resources/images/news/rbc_ru.png b/resources/images/news/rbc_ru.png new file mode 100644 index 0000000000..46c5d3fdce Binary files /dev/null and b/resources/images/news/rbc_ru.png differ diff --git a/resources/images/news/timesnewroman.png b/resources/images/news/timesnewroman.png new file mode 100644 index 0000000000..6ba02939b4 Binary files /dev/null and b/resources/images/news/timesnewroman.png differ diff --git a/resources/images/news/trombon.png b/resources/images/news/trombon.png new file mode 100644 index 0000000000..641b04f1b7 Binary files /dev/null and b/resources/images/news/trombon.png differ diff --git a/resources/images/news/wallstreetro.png b/resources/images/news/wallstreetro.png new file mode 100644 index 0000000000..d72bc70ca0 Binary files /dev/null and b/resources/images/news/wallstreetro.png differ diff --git a/resources/recipes/buffalo_news.recipe b/resources/recipes/buffalo_news.recipe index 92c96757ae..51985a3c51 100644 --- a/resources/recipes/buffalo_news.recipe +++ b/resources/recipes/buffalo_news.recipe @@ -1,8 +1,8 @@ __license__ = 'GPL v3' __author__ = 'Todd Chapman' __copyright__ = 'Todd Chapman' -__version__ = 'v0.1' -__date__ = '26 February 2011' +__version__ = 'v0.2' +__date__ = '2 March 2011' ''' http://www.buffalonews.com/RSS/ @@ -12,12 +12,16 @@ from calibre.web.feeds.news import BasicNewsRecipe class AdvancedUserRecipe1298680852(BasicNewsRecipe): title = u'Buffalo News' - __author__ = 'ChappyOnIce' - language = 'en' oldest_article = 2 + language = 'en' + __author__ = 'ChappyOnIce' max_articles_per_feed = 20 encoding = 'utf-8' + masthead_url = 'http://www.buffalonews.com/buffalonews/skins/buffalonews/images/masthead/the_buffalo_news_logo.png' remove_javascript = True + extra_css = 'body {text-align: justify;}\n \ + p {text-indent: 20px;}' + keep_only_tags = [ dict(name='div', attrs={'class':['main-content-left']}) ] @@ -28,9 +32,7 @@ class AdvancedUserRecipe1298680852(BasicNewsRecipe): ] remove_tags_after = dict(name='div', attrs={'class':['body storyContent']}) - conversion_options = { - 'base_font_size' : 14, - } + feeds = [(u'City of Buffalo', u'http://www.buffalonews.com/city/communities/buffalo/?widget=rssfeed&view=feed&contentId=77944'), (u'Southern Erie County', u'http://www.buffalonews.com/city/communities/southern-erie/?widget=rssfeed&view=feed&contentId=77944'), (u'Eastern Erie County', u'http://www.buffalonews.com/city/communities/eastern-erie/?widget=rssfeed&view=feed&contentId=77944'), diff --git a/resources/recipes/cotidianul.recipe b/resources/recipes/cotidianul.recipe new file mode 100644 index 0000000000..f00196532c --- /dev/null +++ b/resources/recipes/cotidianul.recipe @@ -0,0 +1,69 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = u'2011, Silviu Cotoar\u0103' +''' +cotidianul.ro +''' + +from calibre.web.feeds.news import BasicNewsRecipe + +class Cotidianul(BasicNewsRecipe): + title = u'Cotidianul' + __author__ = u'Silviu Cotoar\u0103' + description = u'' + publisher = u'Cotidianul' + oldest_article = 25 + language = 'ro' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare,Stiri' + encoding = 'utf-8' + cover_url = 'http://www.cotidianul.ro/images/cotidianul.png' + + conversion_options = { + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } + + extra_css = ''' + h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;} + h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;} + .byline {font-family:Arial,Helvetica,sans-serif; font-size:xx-small;} + .date {font-family:Arial,Helvetica,sans-serif; font-size:xx-small;} + p{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .copyright {font-family:Arial,Helvetica,sans-serif;font-size:xx-small;text-align:center} + .story{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .entry-asset asset hentry{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .pagebody{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .maincontentcontainer{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .story-body{font-family:Arial,Helvetica,sans-serif;font-size:small;} + body{font-family:Helvetica,Arial,sans-serif;font-size:small;} + ''' + + keep_only_tags = [ + dict(name='div', attrs={'class':'titlu'}) + , dict(name='div', attrs={'class':'gallery clearfix'}) + , dict(name='div', attrs={'align':'justify'}) + ] + + remove_tags = [ + dict(name='div', attrs={'class':['space']}) + , dict(name='div', attrs={'id':['title_desc']}) + ] + + remove_tags_after = [ + dict(name='div', attrs={'class':['space']}) + , dict(name='span', attrs={'class':['date']}) + ] + + feeds = [ + (u'Feeds', u'http://www.cotidianul.ro/rssfeed/ToateStirile.xml') + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) diff --git a/resources/recipes/credit_slips.recipe b/resources/recipes/credit_slips.recipe index 19e19ca2fb..d4fb3a94c0 100644 --- a/resources/recipes/credit_slips.recipe +++ b/resources/recipes/credit_slips.recipe @@ -1,35 +1,44 @@ #!/usr/bin/env python __license__ = 'GPL 3' -__copyright__ = 'zotzot' +__copyright__ = 'zotzo' __docformat__ = 'restructuredtext en' from calibre.web.feeds.news import BasicNewsRecipe class CreditSlips(BasicNewsRecipe): - __license__ = 'GPL v3' - __author__ = 'zotzot' language = 'en' - version = 1 + __author__ = 'zotzot' + version = 2 title = u'Credit Slips.org' publisher = u'Bankr-L' category = u'Economic blog' - description = u'All things about credit.' - cover_url = 'http://bit.ly/hyZSTr' - oldest_article = 50 + description = u'A discussion on credit and bankruptcy' + cover_url = 'http://bit.ly/eAKNCB' + oldest_article = 15 max_articles_per_feed = 100 use_embedded_content = True + no_stylesheets = True + remove_javascript = True + + conversion_options = { + 'comments': description, + 'tags': category, + 'language': 'en', + 'publisher': publisher, + } feeds = [ -(u'Credit Slips', u'http://www.creditslips.org/creditslips/atom.xml') -] - conversion_options = { -'comments': description, -'tags': category, -'language': 'en', -'publisher': publisher -} - extra_css = ''' - body{font-family:verdana,arial,helvetica,geneva,sans-serif;} - img {float: left; margin-right: 0.5em;} - ''' + (u'Credit Slips', u'http://www.creditslips.org/creditslips/atom.xml') + ] + + extra_css = ''' + .author {font-family:Helvetica,sans-serif; font-weight:normal;font-size:small;} + h1 {font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;} + p {font-family:Helvetica,Arial,sans-serif;font-size:small;} + body {font-family:Helvetica,Arial,sans-serif;font-size:small;} + ''' + + def populate_article_metadata(self, article, soup, first): + h2 = soup.find('h2') + h2.replaceWith(h2.prettify() + '

Posted by ' + article.author + '

') diff --git a/resources/recipes/economist.recipe b/resources/recipes/economist.recipe index 17bf4c8c20..9447fe2193 100644 --- a/resources/recipes/economist.recipe +++ b/resources/recipes/economist.recipe @@ -24,7 +24,7 @@ class Economist(BasicNewsRecipe): cover_url = 'http://www.economist.com/images/covers/currentcoverus_large.jpg' remove_tags = [ dict(name=['script', 'noscript', 'title', 'iframe', 'cf_floatingcontent']), - dict(attrs={'class':['dblClkTrk', 'ec-article-info']}), + dict(attrs={'class':['dblClkTrk', 'ec-article-info', 'share_inline_header']}), {'class': lambda x: x and 'share-links-header' in x}, ] keep_only_tags = [dict(id='ec-article-body')] diff --git a/resources/recipes/economist_free.recipe b/resources/recipes/economist_free.recipe index f4a4efd932..d1766211d7 100644 --- a/resources/recipes/economist_free.recipe +++ b/resources/recipes/economist_free.recipe @@ -18,7 +18,8 @@ class Economist(BasicNewsRecipe): cover_url = 'http://www.economist.com/images/covers/currentcoverus_large.jpg' remove_tags = [ dict(name=['script', 'noscript', 'title', 'iframe', 'cf_floatingcontent']), - dict(attrs={'class':['dblClkTrk', 'ec-article-info']}), + dict(attrs={'class':['dblClkTrk', 'ec-article-info', + 'share_inline_header']}), {'class': lambda x: x and 'share-links-header' in x}, ] keep_only_tags = [dict(id='ec-article-body')] diff --git a/resources/recipes/el_pais_babelia.recipe b/resources/recipes/el_pais_babelia.recipe new file mode 100644 index 0000000000..31b983ec0b --- /dev/null +++ b/resources/recipes/el_pais_babelia.recipe @@ -0,0 +1,49 @@ +from calibre.web.feeds.news import BasicNewsRecipe + +class ElPaisBabelia(BasicNewsRecipe): + + title = 'El Pais Babelia' + __author__ = 'oneillpt' + description = 'El Pais Babelia' + INDEX = 'http://www.elpais.com/suple/babelia/' + language = 'es' + + remove_tags_before = dict(name='div', attrs={'class':'estructura_2col'}) + keep_tags = [dict(name='div', attrs={'class':'estructura_2col'})] + remove_tags = [dict(name='div', attrs={'class':'votos estirar'}), + dict(name='div', attrs={'id':'utilidades'}), + dict(name='div', attrs={'class':'info_relacionada'}), + dict(name='div', attrs={'class':'mod_apoyo'}), + dict(name='div', attrs={'class':'contorno_f'}), + dict(name='div', attrs={'class':'pestanias'}), + dict(name='div', attrs={'class':'otros_webs'}), + dict(name='div', attrs={'id':'pie'}) + ] + #no_stylesheets = True + remove_javascript = True + + def parse_index(self): + articles = [] + soup = self.index_to_soup(self.INDEX) + feeds = [] + for section in soup.findAll('div', attrs={'class':'contenedor_nuevo'}): + section_title = self.tag_to_string(section.find('h1')) + articles = [] + for post in section.findAll('a', href=True): + url = post['href'] + if url.startswith('/'): + url = 'http://www.elpais.es'+url + title = self.tag_to_string(post) + if str(post).find('class=') > 0: + klass = post['class'] + if klass != "": + self.log() + self.log('--> post: ', post) + self.log('--> url: ', url) + self.log('--> title: ', title) + self.log('--> class: ', klass) + articles.append({'title':title, 'url':url}) + if articles: + feeds.append((section_title, articles)) + return feeds + diff --git a/resources/recipes/ele.recipe b/resources/recipes/ele.recipe new file mode 100644 index 0000000000..ea8954366b --- /dev/null +++ b/resources/recipes/ele.recipe @@ -0,0 +1,58 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = u'2011, Silviu Cotoar\u0103' +''' +ele.ro +''' + +from calibre.web.feeds.news import BasicNewsRecipe + +class Ele(BasicNewsRecipe): + title = u'Ele' + __author__ = u'Silviu Cotoar\u0103' + description = u'Dezv\u0103luie ceea ce e\u015fti' + publisher = u'Ele' + oldest_article = 25 + language = 'ro' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare,Femei' + encoding = 'utf-8' + cover_url = 'http://www.tripmedia.ro/tripadmin/photos/logo_ele_mare.jpg' + + conversion_options = { + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } + + extra_css = ''' + h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;} + h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;} + .byline {font-family:Arial,Helvetica,sans-serif; font-size:xx-small;} + .date {font-family:Arial,Helvetica,sans-serif; font-size:xx-small;} + p{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .copyright {font-family:Arial,Helvetica,sans-serif;font-size:xx-small;text-align:center} + .story{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .entry-asset asset hentry{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .pagebody{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .maincontentcontainer{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .story-body{font-family:Arial,Helvetica,sans-serif;font-size:small;} + body{font-family:Helvetica,Arial,sans-serif;font-size:small;} + ''' + + keep_only_tags = [ + dict(name='h1', attrs={'class':'article_title'}) + , dict(name='div', attrs={'class':'article_text'}) + ] + + feeds = [ + (u'Feeds', u'http://www.ele.ro/rss_must_read') + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) diff --git a/resources/recipes/epl_talk.recipe b/resources/recipes/epl_talk.recipe index 297dffd89c..432b396339 100644 --- a/resources/recipes/epl_talk.recipe +++ b/resources/recipes/epl_talk.recipe @@ -1,6 +1,6 @@ #!/usr/bin/env python __license__ = 'GPL 3' -__copyright__ = 'zotzot' +__copyright__ = 'zotzo' __docformat__ = 'restructuredtext en' ''' http://www.epltalk.com @@ -9,10 +9,9 @@ from calibre.web.feeds.news import BasicNewsRecipe class EPLTalkRecipe(BasicNewsRecipe): - __license__ = 'GPL v3' - __author__ = u'The Gaffer' language = 'en' - version = 1 + version = 2 + __author__ = 'rylsfan' title = u'EPL Talk' publisher = u'The Gaffer' @@ -21,17 +20,40 @@ class EPLTalkRecipe(BasicNewsRecipe): description = u'News and Analysis from the English Premier League' cover_url = 'http://bit.ly/hJxZPu' - oldest_article = 45 - max_articles_per_feed = 150 + oldest_article = 3 + max_articles_per_feed = 100 use_embedded_content = True remove_javascript = True encoding = 'utf8' - remove_tags_after = [dict(name='div', attrs={'class':'pd-rating'})] + conversion_options = { + 'comment' : description + , 'tags' : category + , 'publisher' : publisher + , 'language' : language + } - feeds = [(u'EPL Talk', u'http://feeds.feedburner.com/EPLTalk')] + remove_tags = [ + {'class': 'feedflare'}, + {'class': 'tweetmeme_button'}, + {'class': 'eplrelated'}, + {'p': 'Related posts:
    '}, + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) + + feeds =[ + (u'EPL Talk', u'http://feeds.feedburner.com/EPLTalk'), + (u'MLS Talk', u'http://feeds.feedburner.com/majorleaguesoccertalksite'), + #(), + #(), + #(), + ] extra_css = ''' - body{font-family:verdana,arial,helvetica,geneva,sans-serif;} - img {float: left; margin-right: 0.5em;} - ''' + h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;} + h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;} + p{font-family:Arial,Helvetica,sans-serif;font-size:small;} + body{font-family:Helvetica,Arial,sans-serif;font-size:small;} + ''' diff --git a/resources/recipes/evz.ro.recipe b/resources/recipes/evz.ro.recipe index bce151d1fc..841dc80429 100644 --- a/resources/recipes/evz.ro.recipe +++ b/resources/recipes/evz.ro.recipe @@ -1,52 +1,54 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + __license__ = 'GPL v3' -__copyright__ = '2010, Darko Miletic ' +__copyright__ = u'2011, Silviu Cotoar\u0103' ''' evz.ro ''' -import re from calibre.web.feeds.news import BasicNewsRecipe -class EVZ_Ro(BasicNewsRecipe): - title = 'evz.ro' - __author__ = 'Darko Miletic' - description = 'News from Romania' - publisher = 'evz.ro' - category = 'news, politics, Romania' - oldest_article = 2 - max_articles_per_feed = 200 - no_stylesheets = True - encoding = 'utf8' - use_embedded_content = False +class EvenimentulZilei(BasicNewsRecipe): + title = u'Evenimentul Zilei' + __author__ = u'Silviu Cotoar\u0103' + description = '' + publisher = u'Evenimentul Zilei' + oldest_article = 5 language = 'ro' - masthead_url = 'http://www.evz.ro/fileadmin/images/logo.gif' - extra_css = ' body{font-family: Georgia,Arial,Helvetica,sans-serif } .firstP{font-size: 1.125em} .author,.articleInfo{font-size: small} ' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare,Stiri' + encoding = 'utf-8' + cover_url = 'http://www.evz.ro/fileadmin/images/evzLogo.png' conversion_options = { - 'comment' : description - , 'tags' : category - , 'publisher' : publisher - , 'language' : language - } + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } - preprocess_regexps = [ - (re.compile(r'.*?', re.DOTALL|re.IGNORECASE),lambda match: '<head><title>') - ,(re.compile(r'.*?', re.DOTALL|re.IGNORECASE),lambda match: '') - ] + keep_only_tags = [ + dict(name='div', attrs={'class':'single'}) + , dict(name='img', attrs={'id':'placeholder'}) + , dict(name='a', attrs={'id':'holderlink'}) + ] - remove_tags = [ - dict(name=['form','embed','iframe','object','base','link','script','noscript']) - ,dict(attrs={'class':['section','statsInfo','email il']}) - ,dict(attrs={'id' :'gallery'}) - ] + remove_tags = [ + dict(name='p', attrs={'class':['articleInfo']}) + , dict(name='div', attrs={'id':['bannerAddoceansArticleJos']}) + , dict(name='div', attrs={'id':['bannerAddoceansArticle']}) + ] - remove_tags_after = dict(attrs={'class':'section'}) - keep_only_tags = [dict(attrs={'class':'single'})] - remove_attributes = ['height','width'] + remove_tags_after = [ + dict(name='div', attrs={'id':['bannerAddoceansArticleJos']}) + ] - feeds = [(u'Articles', u'http://www.evz.ro/rss.xml')] + feeds = [ + (u'Feeds', u'http://www.evz.ro/rss.xml') + ] def preprocess_html(self, soup): - for item in soup.findAll(style=True): - del item['style'] - return soup + return self.adeify_images(soup) diff --git a/resources/recipes/felicia.recipe b/resources/recipes/felicia.recipe new file mode 100644 index 0000000000..0772e38494 --- /dev/null +++ b/resources/recipes/felicia.recipe @@ -0,0 +1,48 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = u'2011, Silviu Cotoar\u0103' +''' +revistafelicia.ro +''' + +from calibre.web.feeds.news import BasicNewsRecipe + +class Felicia(BasicNewsRecipe): + title = u'Revista Felicia' + __author__ = u'Silviu Cotoar\u0103' + description = u'O revist\u0103 pentru sufletul t\u0103u' + publisher = u'Revista Felicia' + oldest_article = 25 + language = 'ro' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare,Reviste' + encoding = 'utf-8' + cover_url = 'http://www.3waves.net/uploads/image/logo-revista-felicia_03.jpg' + + conversion_options = { + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } + + keep_only_tags = [ + dict(name='div', attrs={'class':'header'}) + , dict(name='div', attrs={'id':'contentArticol'}) + ] + + remove_tags = [ + dict(name='img',attrs={'src':['http://www.revistafelicia.ro/templates/default/images/hdr_ultimul_nr.jpg']}) + , dict(name='div',attrs={'class':['content']}) + ] + + feeds = [ + (u'Feeds', u'http://www.revistafelicia.ro/rss') + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) diff --git a/resources/recipes/financiarul.recipe b/resources/recipes/financiarul.recipe new file mode 100644 index 0000000000..807f771408 --- /dev/null +++ b/resources/recipes/financiarul.recipe @@ -0,0 +1,55 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = u'2011, Silviu Cotoar\u0103' +''' +financiarul.com +''' + +from calibre.web.feeds.news import BasicNewsRecipe + +class Financiarul(BasicNewsRecipe): + title = u'Financiarul' + __author__ = u'Silviu Cotoar\u0103' + description = u'FIN.ro' + publisher = u'Financiarul' + oldest_article = 25 + language = 'ro' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare,Stiri' + encoding = 'utf-8' + cover_url = 'http://www.financiarul.com/templates/default/images/logo.png' + + conversion_options = { + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } + + keep_only_tags = [ + dict(name='div', attrs={'class':'col2ContentLeftL'}) + ] + + remove_tags = [ + dict(name='div',attrs={'class':['infoArticol']}) + , dict(name='ul', attrs={'class':'navSectiuni'}) + , dict(name='div', attrs={'class':'separator separatorTop'}) + , dict(name='div', attrs={'class':'infoArticol infoArticolBottom'}) + , dict(name='ul', attrs={'class':['related']}) + , dict(name='div', attrs={'class':['slot panel300 panelGri300 panelGri300s panelGri300sm']}) + ] + + remove_tags_after = [ + dict(name='ul', attrs={'class':['related']}) + ] + + feeds = [ + (u'Feeds', u'http://www.financiarul.com/rss') + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) diff --git a/resources/recipes/helsingin_sanomat.recipe b/resources/recipes/helsingin_sanomat.recipe new file mode 100644 index 0000000000..6099a1fda8 --- /dev/null +++ b/resources/recipes/helsingin_sanomat.recipe @@ -0,0 +1,31 @@ +from calibre.web.feeds.news import BasicNewsRecipe + +class AdvancedUserRecipe1298137661(BasicNewsRecipe): + title = u'Helsingin Sanomat' + __author__ = 'oneillpt' + language = 'fi' + oldest_article = 7 + max_articles_per_feed = 100 + no_stylesheets = True + remove_javascript = True + conversion_options = { + 'linearize_tables' : True + } + remove_tags = [ + dict(name='a', attrs={'id':'articleCommentUrl'}), + dict(name='p', attrs={'class':'newsSummary'}), + dict(name='div', attrs={'class':'headerTools'}) + ] + + feeds = [(u'Uutiset - HS.fi', u'http://www.hs.fi/uutiset/rss/'), (u'Politiikka - HS.fi', u'http://www.hs.fi/politiikka/rss/'), + (u'Ulkomaat - HS.fi', u'http://www.hs.fi/ulkomaat/rss/'), (u'Kulttuuri - HS.fi', u'http://www.hs.fi/kulttuuri/rss/'), + (u'Kirjat - HS.fi', u'http://www.hs.fi/kulttuuri/kirjat/rss/'), (u'Elokuvat - HS.fi', u'http://www.hs.fi/kulttuuri/elokuvat/rss/') + ] + + def print_version(self, url): + j = url.rfind("/") + s = url[j:] + i = s.rfind("?ref=rss") + if i > 0: + s = s[:i] + return "http://www.hs.fi/tulosta" + s diff --git a/resources/recipes/hitro.recipe b/resources/recipes/hitro.recipe new file mode 100644 index 0000000000..3a85847c81 --- /dev/null +++ b/resources/recipes/hitro.recipe @@ -0,0 +1,43 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = u'2011, Silviu Cotoar\u0103' +''' +hit.ro +''' + +from calibre.web.feeds.news import BasicNewsRecipe + +class Hit(BasicNewsRecipe): + title = u'HIT' + __author__ = u'Silviu Cotoar\u0103' + description = 'IT' + publisher = 'HIT' + oldest_article = 5 + language = 'ro' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare,Reviste,IT' + encoding = 'utf-8' + cover_url = 'http://www.hit.ro/lib/images/frontend/hit_logo.png' + + conversion_options = { + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } + + keep_only_tags = [ + dict(name='h1', attrs={'class':'art_titl'}) + , dict(name='div', attrs={'id':'continut_articol'}) + ] + + feeds = [ + (u'Feeds', u'http://www.hit.ro/rss') + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) diff --git a/resources/recipes/imperatortravel.recipe b/resources/recipes/imperatortravel.recipe new file mode 100644 index 0000000000..2b6d323bf5 --- /dev/null +++ b/resources/recipes/imperatortravel.recipe @@ -0,0 +1,68 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = u'2011, Silviu Cotoar\u0103' +''' +imperatortravel.ro +''' + +from calibre.web.feeds.news import BasicNewsRecipe + +class Imperatortravel(BasicNewsRecipe): + title = u'Imperator Travel' + __author__ = u'Silviu Cotoar\u0103' + description = u'C\u0103l\u0103torii' + publisher = u'Imperator Travel' + oldest_article = 25 + language = 'ro' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare,Stiri,Turism,Calatorii' + encoding = 'utf-8' + cover_url = 'http://www.imperatortravel.ro/images/header-1.jpg' + + conversion_options = { + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } + + extra_css = ''' + h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;} + h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;} + .byline {font-family:Arial,Helvetica,sans-serif; font-size:xx-small;} + .date {font-family:Arial,Helvetica,sans-serif; font-size:xx-small;} + p{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .copyright {font-family:Arial,Helvetica,sans-serif;font-size:xx-small;text-align:center} + .story{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .entry-asset asset hentry{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .pagebody{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .maincontentcontainer{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .story-body{font-family:Arial,Helvetica,sans-serif;font-size:small;} + body{font-family:Helvetica,Arial,sans-serif;font-size:small;} + ''' + + keep_only_tags = [ + dict(name='div', attrs={'class':'article first_main_article'}) + ] + + remove_tags = [ + dict(name='div', attrs={'class':['meta']}) + , dict(name='body', attrs={'class':['transparent_widget ff3 win Locale_en_US']}) + , dict(name='div', attrs={'class':['connect_widget']}) + , dict(name='ul', attrs={'class':['similar-posts']}) + ] + + remove_tags_after = [ + dict(name='ul', attrs={'class':['similar-posts']}) + ] + + feeds = [ + (u'Feeds', u'http://feeds.feedburner.com/ImperatorTravels') + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) diff --git a/resources/recipes/kamikaze.recipe b/resources/recipes/kamikaze.recipe new file mode 100644 index 0000000000..1369cb6f85 --- /dev/null +++ b/resources/recipes/kamikaze.recipe @@ -0,0 +1,53 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = u'2011, Silviu Cotoar\u0103' +''' +kamikazeonline.ro +''' + +from calibre.web.feeds.news import BasicNewsRecipe + +class Kamikaze(BasicNewsRecipe): + title = u'Kamikaze' + __author__ = u'Silviu Cotoar\u0103' + description = u'S\u0103pt\u0103m\u00e2nal sc\u0103pat de sub control' + publisher = 'Kamikaze' + oldest_article = 5 + language = 'ro' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare,Reviste' + encoding = 'utf-8' + cover_url = 'http://www.kamikazeonline.ro/wp-content/themes/kamikaze/images/kamikazeonline_header.gif' + + conversion_options = { + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } + + keep_only_tags = [ + dict(name='div', attrs={'id':'content'}) + ] + + remove_tags = [ + dict(name='div', attrs={'class':['connect_confirmation_cell connect_confirmation_cell_no_like']}) + , dict(name='h3', attrs={'id':['comments']}) + , dict(name='ul', attrs={'class':['addtoany_list']}) + , dict(name='p', attrs={'class':['postmetadata']}) + ] + + remove_tags_after = [ + dict(name='p', attrs={'class':['postmetadata']}) + ] + + feeds = [ + (u'Feeds', u'http://www.kamikazeonline.ro/feed/') + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) diff --git a/resources/recipes/komchadluek.recipe b/resources/recipes/komchadluek.recipe new file mode 100644 index 0000000000..5f0d2f58a2 --- /dev/null +++ b/resources/recipes/komchadluek.recipe @@ -0,0 +1,46 @@ +from calibre.web.feeds.recipes import BasicNewsRecipe + +class KomChadLuek(BasicNewsRecipe): + + title= 'KomChadLuek' + description = 'Komchadluek News' + __author__ = 'ballsaii and Chotechai' + __license__ = 'GPL v3' + publisher= 'Nation Media Group' + category = 'news, Thai' + language = 'th' + + oldest_article = 1 + max_articles_per_feed = 100 + no_stylesheets= True + remove_javascript=True + + cover_url = 'http://www.komchadluek.net/images_layout2/komchadluek_headerlogo.png' + + keep_only_tags = [] + keep_only_tags.append(dict(name = 'h2')) + keep_only_tags.append(dict(name = 'div', attrs={'id':'news_detail_news'})) + + remove_tags_after=[dict(name='hr')] + + feeds =( +(u'\u0e01\u0e32\u0e23\u0e40\u0e21\u0e37\u0e2d\u0e07','http://www.komchadluek.net/rss/politic.xml'), +(u'\u0e15\u0e48\u0e32\u0e07\u0e1b\u0e23\u0e30\u0e40\u0e17\u0e28','http://www.komchadluek.net/rss/sport.xml'), +(u'\u0e40\u0e01\u0e29\u0e15\u0e23','http://www.komchadluek.net/rss/agriculture.xml'), +(u'\u0e15\u0e48\u0e32\u0e07\u0e1b\u0e23\u0e30\u0e40\u0e17\u0e28','http://www.komchadluek.net/rss/foreign.xml'), +(u'\u0e1a\u0e31\u0e19\u0e40\u0e17\u0e34\u0e07','http://www.komchadluek.net/rss/entertainment.xml'), +(u'\u0e1c\u0e39\u0e49\u0e2b\u0e0d\u0e34\u0e07-\u0e41\u0e1f\u0e0a\u0e31\u0e48\u0e19','http://www.komchadluek.net/rss/fashion.xml'), +(u'\u0e1e\u0e23\u0e30\u0e40\u0e04\u0e23\u0e37\u0e48\u0e2d\u0e07','http://www.komchadluek.net/rss/amulet.xml'), +(u'\u0e20\u0e39\u0e21\u0e34\u0e20\u0e32\u0e04-\u0e1b\u0e23\u0e30\u0e0a\u0e32\u0e04\u0e21\u0e17\u0e49\u0e2d\u0e07\u0e16\u0e34\u0e48\u0e19','http://www.komchadluek.net/rss/local.xml'), +(u'\u0e25\u0e38\u0e07\u0e41\u0e08\u0e48\u0e21','http://www.komchadluek.net/rss/unclecham.xml'), +(u'\u0e44\u0e25\u0e1f\u0e4c\u0e2a\u0e44\u0e15\u0e25\u0e4c','http://www.komchadluek.net/rss/lifestyle.xml'), +(u'\u0e40\u0e28\u0e23\u0e29\u0e10\u0e01\u0e34\u0e08-\u0e01\u0e32\u0e23\u0e15\u0e25\u0e32\u0e14','http://www.komchadluek.net/rss/economic.xml'), +(u'\u0e2d\u0e32\u0e2b\u0e32\u0e23','http://www.komchadluek.net/rss/food.xml'), +(u'\u0e04\u0e19\u0e23\u0e31\u0e01\u0e1a\u0e49\u0e32\u0e19-\u0e22\u0e32\u0e19\u0e22\u0e19\u0e15\u0e4c','http://www.komchadluek.net/rss/homecar.xml'), +(u'\u0e14\u0e39\u0e14\u0e27\u0e07-\u0e42\u0e2b\u0e23\u0e32\u0e28\u0e32\u0e2a\u0e15\u0e23\u0e4c','http://www.komchadluek.net/rss/horoscope.xml'), +(u'\u0e27\u0e34\u0e17\u0e22\u0e4c\u0e28\u0e32\u0e2a\u0e15\u0e23\u0e4c-\u0e44\u0e2d\u0e17\u0e35','http://www.komchadluek.net/rss/scienceit.xml'), +(u'\u0e28\u0e32\u0e2a\u0e19\u0e32 \u0e28\u0e34\u0e25\u0e1b\u0e30-\u0e27\u0e31\u0e12\u0e19\u0e18\u0e23\u0e23\u0e21 \u0e2a\u0e32\u0e18\u0e32\u0e23\u0e13\u0e2a\u0e38\u0e02','http://www.komchadluek.net/rss/artculture.xml'), +(u'\u0e01\u0e32\u0e23\u0e28\u0e36\u0e01\u0e29\u0e32', 'http://www.komchadluek.net/rss/education.xml'), +(u'\u0e1a\u0e17\u0e04\u0e27\u0e32\u0e21','http://www.komchadluek.net/rss/article.xml'), +(u'\u0e2d\u0e32\u0e0a\u0e0d\u0e32\u0e01\u0e23\u0e23\u0e21', 'http://www.komchadluek.net/rss/crime.xml') +) diff --git a/resources/recipes/kompiutierra.recipe b/resources/recipes/kompiutierra.recipe index 0d30afa3a7..a82db9aced 100644 --- a/resources/recipes/kompiutierra.recipe +++ b/resources/recipes/kompiutierra.recipe @@ -1,36 +1,37 @@ -#!/usr/bin/python -# -*- coding: utf-8 -*- - -__license__ = 'GPL v3' -__copyright__ = '2010, Vadim Dyadkin, dyadkin@gmail.com' -__author__ = 'Vadim Dyadkin' - -from calibre.web.feeds.news import BasicNewsRecipe - -class Computerra(BasicNewsRecipe): - title = u'\u041a\u043e\u043c\u043f\u044c\u044e\u0442\u0435\u0440\u0440\u0430' - recursion = 50 - oldest_article = 100 - __author__ = 'Vadim Dyadkin' - max_articles_per_feed = 100 - use_embedded_content = False - simultaneous_downloads = 5 - language = 'ru' - description = u'\u041a\u043e\u043c\u043f\u044c\u044e\u0442\u0435\u0440\u044b, \u043e\u043a\u043e\u043b\u043e\u043d\u0430\u0443\u0447\u043d\u044b\u0435 \u0438 \u043e\u043a\u043e\u043b\u043e\u0444\u0438\u043b\u043e\u0441\u043e\u0444\u0441\u043a\u0438\u0435 \u0441\u0442\u0430\u0442\u044c\u0438, \u0433\u0430\u0434\u0436\u0435\u0442\u044b.' - - keep_only_tags = [dict(name='div', attrs={'id': 'content'}),] - - - feeds = [(u'\u041a\u043e\u043c\u043f\u044c\u044e\u0442\u0435\u0440\u0440\u0430', 'http://feeds.feedburner.com/ct_news/'),] - - remove_tags = [dict(name='div', attrs={'id': ['fin', 'idc-container', 'idc-noscript',]}), - dict(name='ul', attrs={'class': "related_post"}), - dict(name='p', attrs={'class': 'info'}), - dict(name='a', attrs={'rel': 'tag', 'class': 'twitter-share-button', 'type': 'button_count'}), - dict(name='h2', attrs={}),] - - extra_css = 'body { text-align: justify; }' - - def get_article_url(self, article): - return article.get('feedburner:origLink', article.get('guid')) - +#!/usr/bin/python +# -*- coding: utf-8 -*- + +__license__ = 'GPL v3' +__copyright__ = '2010, Vadim Dyadkin, dyadkin@gmail.com' +__author__ = 'Vadim Dyadkin' + +from calibre.web.feeds.news import BasicNewsRecipe + +class Computerra(BasicNewsRecipe): + title = u'\u041a\u043e\u043c\u043f\u044c\u044e\u0442\u0435\u0440\u0440\u0430' + oldest_article = 100 + __author__ = 'Vadim Dyadkin (edited by A. Chewi)' + max_articles_per_feed = 50 + use_embedded_content = False + remove_javascript = True + no_stylesheets = True + conversion_options = {'linearize_tables' : True} + simultaneous_downloads = 5 + language = 'ru' + description = u'Компьютерра: все новости про компьютеры, железо, новые технологии, информационные технологии' + + keep_only_tags = [dict(name='div', attrs={'id': 'content'}),] + + feeds = [(u'Компьютерра-Онлайн', 'http://feeds.feedburner.com/ct_news/'),] + + remove_tags = [ + dict(name='div', attrs={'id': ['fin', 'idc-container', 'idc-noscript',]}), + dict(name='ul', attrs={'class': "related_post"}), + dict(name='p', attrs={'class': 'info'}), + dict(name='a', attrs={'class': 'twitter-share-button'}), + dict(name='a', attrs={'type': 'button_count'}), + dict(name='h2', attrs={}) + ] + + def print_version(self, url): + return url + '?print=true' diff --git a/resources/recipes/lanacion.recipe b/resources/recipes/lanacion.recipe index 05e777ec67..425aa9b193 100644 --- a/resources/recipes/lanacion.recipe +++ b/resources/recipes/lanacion.recipe @@ -1,5 +1,5 @@ __license__ = 'GPL v3' -__copyright__ = '2008-2010, Darko Miletic ' +__copyright__ = '2008-2011, Darko Miletic ' ''' lanacion.com.ar ''' @@ -19,9 +19,10 @@ class Lanacion(BasicNewsRecipe): language = 'es_AR' publication_type = 'newspaper' remove_empty_feeds = True - masthead_url = 'http://www.lanacion.com.ar/imgs/layout/logos/ln341x47.gif' - extra_css = """ h1{font-family: Georgia,serif} - h2{color: #626262} + masthead_url = 'http://www.lanacion.com.ar/_ui/desktop/imgs/layout/logos/ln341x47.gif' + extra_css = """ + h1{font-family: Georgia,serif} + h2{color: #626262; font-weight: normal; font-size: 1.1em} body{font-family: Arial,sans-serif} img{margin-top: 0.5em; margin-bottom: 0.2em; display: block} .notaFecha{color: #808080} @@ -37,47 +38,78 @@ class Lanacion(BasicNewsRecipe): , 'language' : language } - keep_only_tags = [dict(name='div', attrs={'class':['nota floatFix','topNota','nota','post']})] + keep_only_tags = [dict(name='div', attrs={'id':'content'})] + remove_tags = [ dict(name='div' , attrs={'class':'notaComentario floatFix noprint' }) ,dict(name='ul' , attrs={'class':['cajaHerramientas cajaTop noprint','herramientas noprint']}) - ,dict(name='div' , attrs={'class':['cajaHerramientas noprint','cajaHerramientas floatFix'] }) - ,dict(attrs={'class':['titulosMultimedia','derecha','techo color','encuesta','izquierda compartir','floatFix','videoCentro']}) + ,dict(name='div' , attrs={'class':['titulosMultimedia','herramientas noprint','cajaHerramientas noprint','cajaHerramientas floatFix'] }) + ,dict(attrs={'class':['izquierda','espacio17','espacio10','espacio20','floatFix ultimasNoticias','relacionadas','titulosMultimedia','derecha','techo color','encuesta','izquierda compartir','floatFix','videoCentro']}) ,dict(name=['iframe','embed','object','form','base','hr','meta','link','input']) ] + remove_tags_after = dict(attrs={'class':['tags','nota-destacado']}) remove_attributes = ['height','width','visible','onclick','data-count','name'] feeds = [ - (u'Ultimas noticias' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?origen=2' ) - ,(u'Politica' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=30' ) - ,(u'Economia' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=272' ) - ,(u'Deportes' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=131' ) - ,(u'Informacion General' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=21' ) - ,(u'Cultura' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=1' ) - ,(u'Opinion' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=28' ) - ,(u'Espectaculos' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=120' ) - ,(u'Exterior' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=7' ) - ,(u'Ciencia&Salud' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=498' ) - ,(u'Revista' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=494' ) - ,(u'Enfoques' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=421' ) - ,(u'Comercio Exterior' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=347' ) - ,(u'Tecnologia' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=432' ) - ,(u'Arquitectura' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=366' ) - ,(u'Turismo' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=504' ) - ,(u'Al volante' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=371' ) - ,(u'El Campo' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=337' ) - ,(u'Moda y Belleza' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=1312' ) - ,(u'Inmuebles Comerciales', u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=1363' ) - ,(u'Countries' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=1348' ) - ,(u'adnCultura' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=6734' ) - ,(u'The Wall Street Journal Americas', u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=6373' ) - ,(u'Estilo de vida' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=7353' ) - ,(u'Management' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=7380' ) - ,(u'Bicentenario' , u'http://www.lanacion.com.ar/herramientas/rss/index.asp?categoria_id=7276' ) + (u'Ultimas Noticias' , u'http://servicios.lanacion.com.ar/herramientas/rss/origen=2' ) + ,(u'Politica' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=30' ) + ,(u'Deportes' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=131' ) + ,(u'Economia' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=272' ) + ,(u'Informacion General' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=21' ) + ,(u'Cultura' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=1' ) + ,(u'Opinion' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=28' ) + ,(u'Espectaculos' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=120' ) + ,(u'Exterior' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=7' ) + ,(u'Ciencia&Salud' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=498' ) + ,(u'Revista' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=494' ) + ,(u'Enfoques' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=421' ) + ,(u'Comercio Exterior' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=347' ) + ,(u'Tecnologia' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=432' ) + ,(u'Arquitectura' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=366' ) + ,(u'Turismo' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=504' ) + ,(u'Al volante' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=371' ) + ,(u'El Campo' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=337' ) + ,(u'Moda y Belleza' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=1312') + ,(u'Inmuebles Comerciales', u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=1363') + ,(u'Countries' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=1348') + ,(u'adnCultura' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=6734') + ,(u'The WSJ Americas' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=6373') + ,(u'Comunidad' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=1344') + ,(u'Management' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=7380') + ,(u'Bicentenario' , u'http://servicios.lanacion.com.ar/herramientas/rss/categoria_id=7276') ] + + def get_browser(self): + br = BasicNewsRecipe.get_browser() + br.set_debug_redirects(True) + br.set_debug_responses(True) + br.set_debug_http(True) + return br + + def get_article_url(self, article): + link = BasicNewsRecipe.get_article_url(self,article) + if link.startswith('http://blogs.lanacion') and not link.endswith('/'): + return None + return link + def preprocess_html(self, soup): for item in soup.findAll(style=True): del item['style'] - return self.adeify_images(soup) + for item in soup.findAll('a'): + limg = item.find('img') + if item.string is not None: + str = item.string + item.replaceWith(str) + else: + if limg: + item.name = 'div' + item.attrs = [] + else: + str = self.tag_to_string(item) + item.replaceWith(str) + for item in soup.findAll('img'): + if not item.has_key('alt'): + item['alt'] = 'image' + return soup diff --git a/resources/recipes/leduc.recipe b/resources/recipes/leduc.recipe index 79ab693115..1412551801 100644 --- a/resources/recipes/leduc.recipe +++ b/resources/recipes/leduc.recipe @@ -1,40 +1,52 @@ from calibre.web.feeds.news import BasicNewsRecipe class AdvancedUserRecipe1292550626(BasicNewsRecipe): - title = 'Leduc - Wetaskiwin Pipestone Flyer' - __author__ = 'Brian Hahn' - description = 'News from Alberta, Canada' - oldest_article = 56 - max_articles_per_feed = 100 - no_stylesheets = True - #delay = 1 - use_embedded_content = False - publisher = 'Pipestone Publishing' - category = 'News, Alberta, Canada' - language = 'en_CA' - encoding = 'iso-8859-1' - cover_url = 'http://www.pipestoneflyer.ca/images/calibre-cover.jpg' - remove_tags_before = dict(id='ContentPanel') - remove_tags_after = dict(id='ContentPanel') - remove_tags = [dict(name='div', attrs={'id':'StoryNav'}),dict(name='div', attrs={'id':'BottomAds'}),dict(name='div', attrs={'id':'MoreStoryLinks'})] - extra_css = 'img { margin:5px }' - feeds = [ -('Feature', 'http://www.pipestoneflyer.ca/Feature.rss'), -('Editors Desk', 'http://www.pipestoneflyer.ca/Editor%27s%20Desk.rss'), -('Letters', 'http://www.pipestoneflyer.ca/Letters.rss'), -('A Loco Viewpoint', 'http://www.pipestoneflyer.ca/A%20Loco%20Viewpoint.rss'), -('Lifes Doorway', 'http://www.pipestoneflyer.ca/Life%27s%20Doorway.rss'), -('From the Otherside', 'http://www.pipestoneflyer.ca/From%20the%20Otherside.rss'), -('Opinion', 'http://www.pipestoneflyer.ca/Opinion.rss'), -('Community', 'http://www.pipestoneflyer.ca/Community.rss'), -('Sports', 'http://www.pipestoneflyer.ca/Sports.rss'), -('Chambers', 'http://www.pipestoneflyer.ca/Chambers.rss'), -('Government', 'http://www.pipestoneflyer.ca/Government.rss'), -('Environment', 'http://www.pipestoneflyer.ca/Environment.rss'), -('Health', 'http://www.pipestoneflyer.ca/Health.rss'), -('Funnies', 'http://www.pipestoneflyer.ca/Funnies.rss'), -('Faith', 'http://www.pipestoneflyer.ca/Faith.rss'), -('News and Views', 'http://www.pipestoneflyer.ca/News%20and%20Views.rss'), -('Obituaries', 'http://www.pipestoneflyer.ca/Obituaries.rss'), -('Police Blotter', 'http://www.pipestoneflyer.ca/Police%20Blotter.rss'), -] + title = 'Leduc - Wetaskiwin Pipestone Flyer' + __author__ = 'Brian Hahn' + description = '''Provides news from central Alberta, Canada. This is a + weekly publication that provides coverage from the Cities of Leduc and + Wetaskiwin, including news from two complete counties, plus the towns and + villages within. The counties of Leduc and Wetaskiwin provide news + coverage of agriculture, sports, government, family, events and opinion. + This publication updated weekly every Thursday.''' + oldest_article = 13 + max_articles_per_feed = 100 + no_stylesheets = True + #delay = 1 + use_embedded_content = False + publisher = 'Pipestone Publishing' + category = 'News, Alberta, Canada' + language = 'en_CA' + encoding = 'iso-8859-1' + cover_url = 'http://www.pipestoneflyer.ca/images/calibre-cover.jpg' + remove_tags_before = dict(id='ContentPanel') + remove_tags_after = dict(id='ContentPanel') + remove_tags = [dict(name='div', + attrs={'id':'StoryNav'}),dict(name='div', + attrs={'id':'BottomAds'}),dict(name='div', attrs={'id':'MoreStoryLinks'})] + extra_css = 'img { margin:5px }' + feeds = [ + ('Feature', 'http://www.pipestoneflyer.ca/Feature.rss'), + ('Editors Desk', 'http://www.pipestoneflyer.ca/Editor%27s%20Desk.rss'), + ('Letters', 'http://www.pipestoneflyer.ca/Letters.rss'), + ('A Loco Viewpoint', + 'http://www.pipestoneflyer.ca/A%20Loco%20Viewpoint.rss'), + ('Lifes Doorway', 'http://www.pipestoneflyer.ca/Life%27s%20Doorway.rss'), + ('From the Otherside', + 'http://www.pipestoneflyer.ca/From%20the%20Otherside.rss'), + ('Opinion', 'http://www.pipestoneflyer.ca/Opinion.rss'), + ('Community', 'http://www.pipestoneflyer.ca/Community.rss'), + ('Sports', 'http://www.pipestoneflyer.ca/Sports.rss'), + ('Chambers', 'http://www.pipestoneflyer.ca/Chambers.rss'), + ('Government', 'http://www.pipestoneflyer.ca/Government.rss'), + ('Travel ', 'http://www.pipestoneflyer.ca/Travel%20.rss'), + ('Environment', 'http://www.pipestoneflyer.ca/Environment.rss'), + ('Health', 'http://www.pipestoneflyer.ca/Health.rss'), + ('Funnies', 'http://www.pipestoneflyer.ca/Funnies.rss'), + ('Events', 'http://www.pipestoneflyer.ca/Events.rss'), + ('Faith', 'http://www.pipestoneflyer.ca/Faith.rss'), + ('News and Views', 'http://www.pipestoneflyer.ca/News%20and%20Views.rss'), + ('Obituaries', 'http://www.pipestoneflyer.ca/Obituaries.rss'), + ('Police Blotter', 'http://www.pipestoneflyer.ca/Police%20Blotter.rss'), + ('Careers', 'http://www.pipestoneflyer.ca/Careers.rss'), + ] diff --git a/resources/recipes/lwn_weekly.recipe b/resources/recipes/lwn_weekly.recipe new file mode 100644 index 0000000000..28ee35802a --- /dev/null +++ b/resources/recipes/lwn_weekly.recipe @@ -0,0 +1,104 @@ +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = '2011, Davide Cavalca ' +''' +lwn.net +''' + +from calibre.web.feeds.news import BasicNewsRecipe +import re + +class WeeklyLWN(BasicNewsRecipe): + title = 'LWN.net Weekly Edition' + description = 'Weekly summary of what has happened in the free software world.' + __author__ = 'Davide Cavalca' + language = 'en' + + cover_url = 'http://lwn.net/images/lcorner.png' + #masthead_url = 'http://lwn.net/images/lcorner.png' + publication_type = 'magazine' + + remove_tags_before = dict(attrs={'class':'PageHeadline'}) + remove_tags_after = dict(attrs={'class':'ArticleText'}) + remove_tags = [dict(name=['h2', 'form'])] + + conversion_options = { 'linearize_tables' : True } + + oldest_article = 7.0 + needs_subscription = 'optional' + + def get_browser(self): + br = BasicNewsRecipe.get_browser() + if self.username is not None and self.password is not None: + br.open('https://lwn.net/login') + br.select_form(name='loginform') + br['Username'] = self.username + br['Password'] = self.password + br.submit() + return br + + def parse_index(self): + if self.username is not None and self.password is not None: + index_url = 'http://lwn.net/current/bigpage' + else: + index_url = 'http://lwn.net/free/bigpage' + soup = self.index_to_soup(index_url) + body = soup.body + + articles = {} + ans = [] + url_re = re.compile('^http://lwn.net/Articles/') + + while True: + tag_title = body.findNext(name='p', attrs={'class':'SummaryHL'}) + if tag_title == None: + break + + tag_section = tag_title.findPrevious(name='p', attrs={'class':'Cat1HL'}) + if tag_section == None: + section = 'Front Page' + else: + section = tag_section.string + + tag_section2 = tag_title.findPrevious(name='p', attrs={'class':'Cat2HL'}) + if tag_section2 != None: + if tag_section2.findPrevious(name='p', attrs={'class':'Cat1HL'}) == tag_section: + section = "%s: %s" %(section, tag_section2.string) + + if section not in articles.keys(): + articles[section] = [] + if section not in ans: + ans.append(section) + + body = tag_title + while True: + tag_url = body.findNext(name='a', attrs={'href':url_re}) + if tag_url == None: + break + body = tag_url + if tag_url.string == None: + continue + elif tag_url.string == 'Full Story': + break + elif tag_url.string.startswith('Comments ('): + break + else: + continue + + if tag_url == None: + break + + article = dict( + title=tag_title.string, + url=tag_url['href'].split('#')[0], + description='', content='', date='') + articles[section].append(article) + + ans = [(key, articles[key]) for key in ans if articles.has_key(key)] + if not ans: + raise Exception('Could not find any articles.') + + return ans + +# vim: expandtab:ts=4:sw=4 diff --git a/resources/recipes/mac_world.recipe b/resources/recipes/mac_world.recipe index bf1403820a..486aa9cb87 100644 --- a/resources/recipes/mac_world.recipe +++ b/resources/recipes/mac_world.recipe @@ -11,7 +11,6 @@ http://www.macworld.co.uk/ ''' from calibre.web.feeds.news import BasicNewsRecipe -from calibre.ptempfile import PersistentTemporaryFile temp_files = [] articles_are_obfuscated = True @@ -36,26 +35,17 @@ class macWorld(BasicNewsRecipe): remove_javascript = True no_stylesheets = True - def get_obfuscated_article(self, url): - br = self.get_browser() - br.open(url+'&print') - - response = br.follow_link(url, nr = 0) - html = response.read() - - self.temp_files.append(PersistentTemporaryFile('_fa.html')) - self.temp_files[-1].write(html) - self.temp_files[-1].close() - return self.temp_files[-1].name - keep_only_tags = [ - dict(name='div', attrs={'id':'article'}) + dict(name='div', attrs={'id':'content'}) ] remove_tags = [ - dict(name='div', attrs={'class':['toolBar','mac_tags','toolBar btmTools','textAds']}), + {'class':['toolBar','mac_tags','toolBar btmTools','textAds']}, dict(name='p', attrs={'class':'breadcrumbs'}), - dict(name='div', attrs={'id':['breadcrumb','sidebar','comments']}) + dict(id=['breadcrumb','sidebar','comments','topContentWrapper', + 'rightColumn', 'aboveFootPromo', 'storyCarousel']), + {'class':lambda x: x and ('tools' in x or 'toolBar' + in x)} ] diff --git a/resources/recipes/ming_pao.recipe b/resources/recipes/ming_pao.recipe index bbdbbf7ace..4a405a59dd 100644 --- a/resources/recipes/ming_pao.recipe +++ b/resources/recipes/ming_pao.recipe @@ -1,7 +1,20 @@ __license__ = 'GPL v3' __copyright__ = '2010-2011, Eddie Lau' + +# Users of Kindle 3 (with limited system-level CJK support) +# please replace the following "True" with "False". +__MakePeriodical__ = True +# Turn it to True if your device supports display of CJK titles +__UseChineseTitle__ = False + + ''' Change Log: +2011/03/06: add new articles for finance section, also a new section "Columns" +2011/02/28: rearrange the sections + [Disabled until Kindle has better CJK support and can remember last (section,article) read in Sections & Articles + View] make it the same title if generating a periodical, so past issue will be automatically put into "Past Issues" + folder in Kindle 3 2011/02/20: skip duplicated links in finance section, put photos which may extend a whole page to the back of the articles clean up the indentation 2010/12/07: add entertainment section, use newspaper front page as ebook cover, suppress date display in section list @@ -19,55 +32,58 @@ import os, datetime, re from calibre.web.feeds.recipes import BasicNewsRecipe from contextlib import nested - from calibre.ebooks.BeautifulSoup import BeautifulSoup from calibre.ebooks.metadata.opf2 import OPFCreator from calibre.ebooks.metadata.toc import TOC from calibre.ebooks.metadata import MetaInformation class MPHKRecipe(BasicNewsRecipe): - IsCJKWellSupported = True # Set to False to avoid generating periodical in which CJK characters can't be displayed in section/article view - title = 'Ming Pao - Hong Kong' - oldest_article = 1 - max_articles_per_feed = 100 - __author__ = 'Eddie Lau' - description = ('Hong Kong Chinese Newspaper (http://news.mingpao.com). If' - 'you are using a Kindle with firmware < 3.1, customize the' - 'recipe') - publisher = 'MingPao' - category = 'Chinese, News, Hong Kong' - remove_javascript = True - use_embedded_content = False - no_stylesheets = True - language = 'zh' - encoding = 'Big5-HKSCS' - recursions = 0 - conversion_options = {'linearize_tables':True} - timefmt = '' - extra_css = 'img {display: block; margin-left: auto; margin-right: auto; margin-top: 10px; margin-bottom: 10px;} font>b {font-size:200%; font-weight:bold;}' - masthead_url = 'http://news.mingpao.com/image/portals_top_logo_news.gif' - keep_only_tags = [dict(name='h1'), + title = 'Ming Pao - Hong Kong' + oldest_article = 1 + max_articles_per_feed = 100 + __author__ = 'Eddie Lau' + description = 'Hong Kong Chinese Newspaper (http://news.mingpao.com)' + publisher = 'MingPao' + category = 'Chinese, News, Hong Kong' + remove_javascript = True + use_embedded_content = False + no_stylesheets = True + language = 'zh' + encoding = 'Big5-HKSCS' + recursions = 0 + conversion_options = {'linearize_tables':True} + timefmt = '' + extra_css = 'img {display: block; margin-left: auto; margin-right: auto; margin-top: 10px; margin-bottom: 10px;} font>b {font-size:200%; font-weight:bold;}' + masthead_url = 'http://news.mingpao.com/image/portals_top_logo_news.gif' + keep_only_tags = [dict(name='h1'), dict(name='font', attrs={'style':['font-size:14pt; line-height:160%;']}), # for entertainment page title - dict(attrs={'id':['newscontent']}), # entertainment page content + dict(name='font', attrs={'color':['AA0000']}), # for column articles title + dict(attrs={'id':['newscontent']}), # entertainment and column page content dict(attrs={'id':['newscontent01','newscontent02']}), dict(attrs={'class':['photo']}) ] - remove_tags = [dict(name='style'), - dict(attrs={'id':['newscontent135']})] # for the finance page - remove_attributes = ['width'] - preprocess_regexps = [ + remove_tags = [dict(name='style'), + dict(attrs={'id':['newscontent135']}), # for the finance page + dict(name='table')] # for content fetched from life.mingpao.com + remove_attributes = ['width'] + preprocess_regexps = [ (re.compile(r'
    ', re.DOTALL|re.IGNORECASE), lambda match: '

    '), (re.compile(r'

    ', re.DOTALL|re.IGNORECASE), lambda match: ''), (re.compile(r'

    ', re.DOTALL|re.IGNORECASE), # for entertainment page - lambda match: '') + lambda match: ''), + # skip
    after title in life.mingpao.com fetched article + (re.compile(r"

    ", re.DOTALL|re.IGNORECASE), + lambda match: "
    "), + (re.compile(r"

    ", re.DOTALL|re.IGNORECASE), + lambda match: "") ] - def image_url_processor(cls, baseurl, url): - # trick: break the url at the first occurance of digit, add an additional - # '_' at the front - # not working, may need to move this to preprocess_html() method + def image_url_processor(cls, baseurl, url): + # trick: break the url at the first occurance of digit, add an additional + # '_' at the front + # not working, may need to move this to preprocess_html() method # minIdx = 10000 # i0 = url.find('0') # if i0 >= 0 and i0 < minIdx: @@ -99,253 +115,314 @@ class MPHKRecipe(BasicNewsRecipe): # i9 = url.find('9') # if i9 >= 0 and i9 < minIdx: # minIdx = i9 - return url + return url - def get_dtlocal(self): - dt_utc = datetime.datetime.utcnow() - # convert UTC to local hk time - at around HKT 6.00am, all news are available - dt_local = dt_utc - datetime.timedelta(-2.0/24) - return dt_local + def get_dtlocal(self): + dt_utc = datetime.datetime.utcnow() + # convert UTC to local hk time - at around HKT 6.00am, all news are available + dt_local = dt_utc - datetime.timedelta(-2.0/24) + return dt_local - def get_fetchdate(self): - return self.get_dtlocal().strftime("%Y%m%d") + def get_fetchdate(self): + return self.get_dtlocal().strftime("%Y%m%d") - def get_fetchformatteddate(self): - return self.get_dtlocal().strftime("%Y-%m-%d") + def get_fetchformatteddate(self): + return self.get_dtlocal().strftime("%Y-%m-%d") - def get_fetchday(self): - # convert UTC to local hk time - at around HKT 6.00am, all news are available - return self.get_dtlocal().strftime("%d") + def get_fetchday(self): + # convert UTC to local hk time - at around HKT 6.00am, all news are available + return self.get_dtlocal().strftime("%d") - def get_cover_url(self): - cover = 'http://news.mingpao.com/' + self.get_fetchdate() + '/' + self.get_fetchdate() + '_' + self.get_fetchday() + 'gacov.jpg' - br = BasicNewsRecipe.get_browser() - try: - br.open(cover) - except: - cover = None - return cover + def get_cover_url(self): + cover = 'http://news.mingpao.com/' + self.get_fetchdate() + '/' + self.get_fetchdate() + '_' + self.get_fetchday() + 'gacov.jpg' + br = BasicNewsRecipe.get_browser() + try: + br.open(cover) + except: + cover = None + return cover - def parse_index(self): - feeds = [] - dateStr = self.get_fetchdate() - for title, url in [(u'\u8981\u805e Headline', 'http://news.mingpao.com/' + dateStr + '/gaindex.htm'), - (u'\u6e2f\u805e Local', 'http://news.mingpao.com/' + dateStr + '/gbindex.htm'), - (u'\u793e\u8a55/\u7b46\u9663 Editorial', 'http://news.mingpao.com/' + dateStr + '/mrindex.htm'), - (u'\u8ad6\u58c7 Forum', 'http://news.mingpao.com/' + dateStr + '/faindex.htm'), + def parse_index(self): + feeds = [] + dateStr = self.get_fetchdate() + + for title, url in [(u'\u8981\u805e Headline', 'http://news.mingpao.com/' + dateStr + '/gaindex.htm'), + (u'\u6e2f\u805e Local', 'http://news.mingpao.com/' + dateStr + '/gbindex.htm'), + (u'\u6559\u80b2 Education', 'http://news.mingpao.com/' + dateStr + '/gfindex.htm')]: + articles = self.parse_section(url) + if articles: + feeds.append((title, articles)) + + # special- editorial + ed_articles = self.parse_ed_section('http://life.mingpao.com/cfm/dailynews2.cfm?Issue=' + dateStr +'&Category=nalmr') + if ed_articles: + feeds.append((u'\u793e\u8a55/\u7b46\u9663 Editorial', ed_articles)) + + for title, url in [(u'\u8ad6\u58c7 Forum', 'http://news.mingpao.com/' + dateStr + '/faindex.htm'), (u'\u4e2d\u570b China', 'http://news.mingpao.com/' + dateStr + '/caindex.htm'), - (u'\u570b\u969b World', 'http://news.mingpao.com/' + dateStr + '/taindex.htm'), - ('Tech News', 'http://news.mingpao.com/' + dateStr + '/naindex.htm'), - (u'\u6559\u80b2 Education', 'http://news.mingpao.com/' + dateStr + '/gfindex.htm'), - (u'\u9ad4\u80b2 Sport', 'http://news.mingpao.com/' + dateStr + '/spindex.htm'), - (u'\u526f\u520a Supplement', 'http://news.mingpao.com/' + dateStr + '/jaindex.htm'), + (u'\u570b\u969b World', 'http://news.mingpao.com/' + dateStr + '/taindex.htm')]: + articles = self.parse_section(url) + if articles: + feeds.append((title, articles)) + + # special - finance + #fin_articles = self.parse_fin_section('http://www.mpfinance.com/htm/Finance/' + dateStr + '/News/ea,eb,ecindex.htm') + fin_articles = self.parse_fin_section('http://life.mingpao.com/cfm/dailynews2.cfm?Issue=' + dateStr + '&Category=nalea') + if fin_articles: + feeds.append((u'\u7d93\u6fdf Finance', fin_articles)) + + for title, url in [('Tech News', 'http://news.mingpao.com/' + dateStr + '/naindex.htm'), + (u'\u9ad4\u80b2 Sport', 'http://news.mingpao.com/' + dateStr + '/spindex.htm')]: + articles = self.parse_section(url) + if articles: + feeds.append((title, articles)) + + # special - entertainment + ent_articles = self.parse_ent_section('http://ol.mingpao.com/cfm/star1.cfm') + if ent_articles: + feeds.append((u'\u5f71\u8996 Film/TV', ent_articles)) + + for title, url in [(u'\u526f\u520a Supplement', 'http://news.mingpao.com/' + dateStr + '/jaindex.htm'), (u'\u82f1\u6587 English', 'http://news.mingpao.com/' + dateStr + '/emindex.htm')]: - articles = self.parse_section(url) - if articles: - feeds.append((title, articles)) - # special - finance - fin_articles = self.parse_fin_section('http://www.mpfinance.com/htm/Finance/' + dateStr + '/News/ea,eb,ecindex.htm') - if fin_articles: - feeds.append((u'\u7d93\u6fdf Finance', fin_articles)) - # special - entertainment - ent_articles = self.parse_ent_section('http://ol.mingpao.com/cfm/star1.cfm') - if ent_articles: - feeds.append((u'\u5f71\u8996 Film/TV', ent_articles)) - return feeds + articles = self.parse_section(url) + if articles: + feeds.append((title, articles)) - def parse_section(self, url): - dateStr = self.get_fetchdate() - soup = self.index_to_soup(url) - divs = soup.findAll(attrs={'class': ['bullet','bullet_grey']}) - current_articles = [] - included_urls = [] - divs.reverse() - for i in divs: - a = i.find('a', href = True) - title = self.tag_to_string(a) - url = a.get('href', False) - url = 'http://news.mingpao.com/' + dateStr + '/' +url - if url not in included_urls and url.rfind('Redirect') == -1: - current_articles.append({'title': title, 'url': url, 'description':'', 'date':''}) - included_urls.append(url) - current_articles.reverse() - return current_articles - def parse_fin_section(self, url): - dateStr = self.get_fetchdate() - soup = self.index_to_soup(url) - a = soup.findAll('a', href= True) - current_articles = [] - included_urls = [] - for i in a: - url = 'http://www.mpfinance.com/cfm/' + i.get('href', False) - if url not in included_urls and not url.rfind(dateStr) == -1 and url.rfind('index') == -1: - title = self.tag_to_string(i) - current_articles.append({'title': title, 'url': url, 'description':''}) - included_urls.append(url) - return current_articles + # special- columns + col_articles = self.parse_col_section('http://life.mingpao.com/cfm/dailynews2.cfm?Issue=' + dateStr +'&Category=ncolumn') + if col_articles: + feeds.append((u'\u5c08\u6b04 Columns', col_articles)) - def parse_ent_section(self, url): - self.get_fetchdate() - soup = self.index_to_soup(url) - a = soup.findAll('a', href=True) - a.reverse() - current_articles = [] - included_urls = [] - for i in a: - title = self.tag_to_string(i) - url = 'http://ol.mingpao.com/cfm/' + i.get('href', False) - if (url not in included_urls) and (not url.rfind('.txt') == -1) and (not url.rfind('star') == -1): - current_articles.append({'title': title, 'url': url, 'description': ''}) - included_urls.append(url) - current_articles.reverse() - return current_articles + return feeds - def preprocess_html(self, soup): - for item in soup.findAll(style=True): - del item['style'] - for item in soup.findAll(style=True): - del item['width'] - for item in soup.findAll(stype=True): - del item['absmiddle'] - return soup + def parse_section(self, url): + dateStr = self.get_fetchdate() + soup = self.index_to_soup(url) + divs = soup.findAll(attrs={'class': ['bullet','bullet_grey']}) + current_articles = [] + included_urls = [] + divs.reverse() + for i in divs: + a = i.find('a', href = True) + title = self.tag_to_string(a) + url = a.get('href', False) + url = 'http://news.mingpao.com/' + dateStr + '/' +url + if url not in included_urls and url.rfind('Redirect') == -1: + current_articles.append({'title': title, 'url': url, 'description':'', 'date':''}) + included_urls.append(url) + current_articles.reverse() + return current_articles - def create_opf(self, feeds, dir=None): - if dir is None: - dir = self.output_dir - if self.IsCJKWellSupported == True: - # use Chinese title - title = u'\u660e\u5831 (\u9999\u6e2f) ' + self.get_fetchformatteddate() - else: - # use English title - title = self.short_title() + ' ' + self.get_fetchformatteddate() - if True: # force date in title - # title += strftime(self.timefmt) - mi = MetaInformation(title, [self.publisher]) - mi.publisher = self.publisher - mi.author_sort = self.publisher - if self.IsCJKWellSupported == True: - mi.publication_type = 'periodical:'+self.publication_type+':'+self.short_title() - else: - mi.publication_type = self.publication_type+':'+self.short_title() - #mi.timestamp = nowf() - mi.timestamp = self.get_dtlocal() - mi.comments = self.description - if not isinstance(mi.comments, unicode): - mi.comments = mi.comments.decode('utf-8', 'replace') - #mi.pubdate = nowf() - mi.pubdate = self.get_dtlocal() - opf_path = os.path.join(dir, 'index.opf') - ncx_path = os.path.join(dir, 'index.ncx') - opf = OPFCreator(dir, mi) - # Add mastheadImage entry to section - mp = getattr(self, 'masthead_path', None) - if mp is not None and os.access(mp, os.R_OK): - from calibre.ebooks.metadata.opf2 import Guide - ref = Guide.Reference(os.path.basename(self.masthead_path), os.getcwdu()) - ref.type = 'masthead' - ref.title = 'Masthead Image' - opf.guide.append(ref) + def parse_ed_section(self, url): + self.get_fetchdate() + soup = self.index_to_soup(url) + a = soup.findAll('a', href=True) + a.reverse() + current_articles = [] + included_urls = [] + for i in a: + title = self.tag_to_string(i) + url = 'http://life.mingpao.com/cfm/' + i.get('href', False) + if (url not in included_urls) and (not url.rfind('.txt') == -1) and (not url.rfind('nal') == -1): + current_articles.append({'title': title, 'url': url, 'description': ''}) + included_urls.append(url) + current_articles.reverse() + return current_articles - manifest = [os.path.join(dir, 'feed_%d'%i) for i in range(len(feeds))] - manifest.append(os.path.join(dir, 'index.html')) - manifest.append(os.path.join(dir, 'index.ncx')) + def parse_fin_section(self, url): + self.get_fetchdate() + soup = self.index_to_soup(url) + a = soup.findAll('a', href= True) + current_articles = [] + included_urls = [] + for i in a: + #url = 'http://www.mpfinance.com/cfm/' + i.get('href', False) + url = 'http://life.mingpao.com/cfm/' + i.get('href', False) + #if url not in included_urls and not url.rfind(dateStr) == -1 and url.rfind('index') == -1: + if url not in included_urls and (not url.rfind('txt') == -1) and (not url.rfind('nal') == -1): + title = self.tag_to_string(i) + current_articles.append({'title': title, 'url': url, 'description':''}) + included_urls.append(url) + return current_articles - # Get cover - cpath = getattr(self, 'cover_path', None) - if cpath is None: - pf = open(os.path.join(dir, 'cover.jpg'), 'wb') - if self.default_cover(pf): - cpath = pf.name - if cpath is not None and os.access(cpath, os.R_OK): - opf.cover = cpath - manifest.append(cpath) + def parse_ent_section(self, url): + self.get_fetchdate() + soup = self.index_to_soup(url) + a = soup.findAll('a', href=True) + a.reverse() + current_articles = [] + included_urls = [] + for i in a: + title = self.tag_to_string(i) + url = 'http://ol.mingpao.com/cfm/' + i.get('href', False) + if (url not in included_urls) and (not url.rfind('.txt') == -1) and (not url.rfind('star') == -1): + current_articles.append({'title': title, 'url': url, 'description': ''}) + included_urls.append(url) + current_articles.reverse() + return current_articles - # Get masthead - mpath = getattr(self, 'masthead_path', None) - if mpath is not None and os.access(mpath, os.R_OK): - manifest.append(mpath) + def parse_col_section(self, url): + self.get_fetchdate() + soup = self.index_to_soup(url) + a = soup.findAll('a', href=True) + a.reverse() + current_articles = [] + included_urls = [] + for i in a: + title = self.tag_to_string(i) + url = 'http://life.mingpao.com/cfm/' + i.get('href', False) + if (url not in included_urls) and (not url.rfind('.txt') == -1) and (not url.rfind('ncl') == -1): + current_articles.append({'title': title, 'url': url, 'description': ''}) + included_urls.append(url) + current_articles.reverse() + return current_articles - opf.create_manifest_from_files_in(manifest) - for mani in opf.manifest: - if mani.path.endswith('.ncx'): - mani.id = 'ncx' - if mani.path.endswith('mastheadImage.jpg'): - mani.id = 'masthead-image' - entries = ['index.html'] - toc = TOC(base_path=dir) - self.play_order_counter = 0 - self.play_order_map = {} + def preprocess_html(self, soup): + for item in soup.findAll(style=True): + del item['style'] + for item in soup.findAll(style=True): + del item['width'] + for item in soup.findAll(stype=True): + del item['absmiddle'] + return soup - def feed_index(num, parent): - f = feeds[num] - for j, a in enumerate(f): - if getattr(a, 'downloaded', False): - adir = 'feed_%d/article_%d/'%(num, j) - auth = a.author - if not auth: - auth = None - desc = a.text_summary - if not desc: - desc = None - else: - desc = self.description_limiter(desc) - entries.append('%sindex.html'%adir) - po = self.play_order_map.get(entries[-1], None) - if po is None: - self.play_order_counter += 1 - po = self.play_order_counter - parent.add_item('%sindex.html'%adir, None, a.title if a.title else _('Untitled Article'), + def create_opf(self, feeds, dir=None): + if dir is None: + dir = self.output_dir + if __UseChineseTitle__ == True: + title = u'\u660e\u5831 (\u9999\u6e2f)' + else: + title = self.short_title() + # if not generating a periodical, force date to apply in title + if __MakePeriodical__ == False: + title = title + ' ' + self.get_fetchformatteddate() + if True: + mi = MetaInformation(title, [self.publisher]) + mi.publisher = self.publisher + mi.author_sort = self.publisher + if __MakePeriodical__ == True: + mi.publication_type = 'periodical:'+self.publication_type+':'+self.short_title() + else: + mi.publication_type = self.publication_type+':'+self.short_title() + #mi.timestamp = nowf() + mi.timestamp = self.get_dtlocal() + mi.comments = self.description + if not isinstance(mi.comments, unicode): + mi.comments = mi.comments.decode('utf-8', 'replace') + #mi.pubdate = nowf() + mi.pubdate = self.get_dtlocal() + opf_path = os.path.join(dir, 'index.opf') + ncx_path = os.path.join(dir, 'index.ncx') + opf = OPFCreator(dir, mi) + # Add mastheadImage entry to section + mp = getattr(self, 'masthead_path', None) + if mp is not None and os.access(mp, os.R_OK): + from calibre.ebooks.metadata.opf2 import Guide + ref = Guide.Reference(os.path.basename(self.masthead_path), os.getcwdu()) + ref.type = 'masthead' + ref.title = 'Masthead Image' + opf.guide.append(ref) + + manifest = [os.path.join(dir, 'feed_%d'%i) for i in range(len(feeds))] + manifest.append(os.path.join(dir, 'index.html')) + manifest.append(os.path.join(dir, 'index.ncx')) + + # Get cover + cpath = getattr(self, 'cover_path', None) + if cpath is None: + pf = open(os.path.join(dir, 'cover.jpg'), 'wb') + if self.default_cover(pf): + cpath = pf.name + if cpath is not None and os.access(cpath, os.R_OK): + opf.cover = cpath + manifest.append(cpath) + + # Get masthead + mpath = getattr(self, 'masthead_path', None) + if mpath is not None and os.access(mpath, os.R_OK): + manifest.append(mpath) + + opf.create_manifest_from_files_in(manifest) + for mani in opf.manifest: + if mani.path.endswith('.ncx'): + mani.id = 'ncx' + if mani.path.endswith('mastheadImage.jpg'): + mani.id = 'masthead-image' + entries = ['index.html'] + toc = TOC(base_path=dir) + self.play_order_counter = 0 + self.play_order_map = {} + + def feed_index(num, parent): + f = feeds[num] + for j, a in enumerate(f): + if getattr(a, 'downloaded', False): + adir = 'feed_%d/article_%d/'%(num, j) + auth = a.author + if not auth: + auth = None + desc = a.text_summary + if not desc: + desc = None + else: + desc = self.description_limiter(desc) + entries.append('%sindex.html'%adir) + po = self.play_order_map.get(entries[-1], None) + if po is None: + self.play_order_counter += 1 + po = self.play_order_counter + parent.add_item('%sindex.html'%adir, None, a.title if a.title else _('Untitled Article'), play_order=po, author=auth, description=desc) - last = os.path.join(self.output_dir, ('%sindex.html'%adir).replace('/', os.sep)) - for sp in a.sub_pages: - prefix = os.path.commonprefix([opf_path, sp]) - relp = sp[len(prefix):] - entries.append(relp.replace(os.sep, '/')) - last = sp + last = os.path.join(self.output_dir, ('%sindex.html'%adir).replace('/', os.sep)) + for sp in a.sub_pages: + prefix = os.path.commonprefix([opf_path, sp]) + relp = sp[len(prefix):] + entries.append(relp.replace(os.sep, '/')) + last = sp - if os.path.exists(last): - with open(last, 'rb') as fi: - src = fi.read().decode('utf-8') - soup = BeautifulSoup(src) - body = soup.find('body') - if body is not None: - prefix = '/'.join('..'for i in range(2*len(re.findall(r'link\d+', last)))) - templ = self.navbar.generate(True, num, j, len(f), + if os.path.exists(last): + with open(last, 'rb') as fi: + src = fi.read().decode('utf-8') + soup = BeautifulSoup(src) + body = soup.find('body') + if body is not None: + prefix = '/'.join('..'for i in range(2*len(re.findall(r'link\d+', last)))) + templ = self.navbar.generate(True, num, j, len(f), not self.has_single_feed, a.orig_url, self.publisher, prefix=prefix, center=self.center_navbar) - elem = BeautifulSoup(templ.render(doctype='xhtml').decode('utf-8')).find('div') - body.insert(len(body.contents), elem) - with open(last, 'wb') as fi: - fi.write(unicode(soup).encode('utf-8')) - if len(feeds) == 0: - raise Exception('All feeds are empty, aborting.') + elem = BeautifulSoup(templ.render(doctype='xhtml').decode('utf-8')).find('div') + body.insert(len(body.contents), elem) + with open(last, 'wb') as fi: + fi.write(unicode(soup).encode('utf-8')) + if len(feeds) == 0: + raise Exception('All feeds are empty, aborting.') - if len(feeds) > 1: - for i, f in enumerate(feeds): - entries.append('feed_%d/index.html'%i) - po = self.play_order_map.get(entries[-1], None) - if po is None: - self.play_order_counter += 1 - po = self.play_order_counter - auth = getattr(f, 'author', None) - if not auth: - auth = None - desc = getattr(f, 'description', None) - if not desc: - desc = None - feed_index(i, toc.add_item('feed_%d/index.html'%i, None, + if len(feeds) > 1: + for i, f in enumerate(feeds): + entries.append('feed_%d/index.html'%i) + po = self.play_order_map.get(entries[-1], None) + if po is None: + self.play_order_counter += 1 + po = self.play_order_counter + auth = getattr(f, 'author', None) + if not auth: + auth = None + desc = getattr(f, 'description', None) + if not desc: + desc = None + feed_index(i, toc.add_item('feed_%d/index.html'%i, None, f.title, play_order=po, description=desc, author=auth)) - else: - entries.append('feed_%d/index.html'%0) - feed_index(0, toc) + else: + entries.append('feed_%d/index.html'%0) + feed_index(0, toc) - for i, p in enumerate(entries): - entries[i] = os.path.join(dir, p.replace('/', os.sep)) - opf.create_spine(entries) - opf.set_toc(toc) + for i, p in enumerate(entries): + entries[i] = os.path.join(dir, p.replace('/', os.sep)) + opf.create_spine(entries) + opf.set_toc(toc) - with nested(open(opf_path, 'wb'), open(ncx_path, 'wb')) as (opf_file, ncx_file): - opf.render(opf_file, ncx_file) + with nested(open(opf_path, 'wb'), open(ncx_path, 'wb')) as (opf_file, ncx_file): + opf.render(opf_file, ncx_file) diff --git a/resources/recipes/monden.recipe b/resources/recipes/monden.recipe new file mode 100644 index 0000000000..22764ffe47 --- /dev/null +++ b/resources/recipes/monden.recipe @@ -0,0 +1,66 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = u'2011, Silviu Cotoar\u0103' +''' +monden.info +''' + +from calibre.web.feeds.news import BasicNewsRecipe + +class Monden(BasicNewsRecipe): + title = u'Monden' + __author__ = u'Silviu Cotoar\u0103' + description = u'Arti\u015fti, interviuri, concerte.. MUZIC\u0102' + publisher = u'Monden' + oldest_article = 25 + language = 'ro' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare,Stiri,Muzica' + encoding = 'utf-8' + cover_url = 'http://www.monden.info/wp-content/uploads/2009/04/mondeninfo-logo.jpg' + + conversion_options = { + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } + + extra_css = ''' + h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;} + h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;} + .byline {font-family:Arial,Helvetica,sans-serif; font-size:xx-small;} + .date {font-family:Arial,Helvetica,sans-serif; font-size:xx-small;} + p{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .copyright {font-family:Arial,Helvetica,sans-serif;font-size:xx-small;text-align:center} + .story{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .entry-asset asset hentry{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .pagebody{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .maincontentcontainer{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .story-body{font-family:Arial,Helvetica,sans-serif;font-size:small;} + body{font-family:Helvetica,Arial,sans-serif;font-size:small;} + ''' + + keep_only_tags = [ + dict(name='div', attrs={'id':'content'}) + ] + + remove_tags = [ + dict(name='div', attrs={'class':['postAuthor']}) + , dict(name='div', attrs={'class':['postLike']}) + ] + + remove_tags_after = [ + dict(name='div', attrs={'class':['postLike']}) + ] + + feeds = [ + (u'Feeds', u'http://www.monden.info/feed/') + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) diff --git a/resources/recipes/nationalgeoro.recipe b/resources/recipes/nationalgeoro.recipe index a3c5727d38..8f989be74d 100644 --- a/resources/recipes/nationalgeoro.recipe +++ b/resources/recipes/nationalgeoro.recipe @@ -14,7 +14,7 @@ class NationalGeoRo(BasicNewsRecipe): __author__ = u'Silviu Cotoar\u0103' description = u'S\u0103 avem grij\u0103 de planet\u0103' publisher = 'National Geographic' - oldest_article = 5 + oldest_article = 35 language = 'ro' max_articles_per_feed = 100 no_stylesheets = True diff --git a/resources/recipes/nrc-nl-epub.recipe b/resources/recipes/nrc-nl-epub.recipe index da9b9195ce..2d190e4d0a 100644 --- a/resources/recipes/nrc-nl-epub.recipe +++ b/resources/recipes/nrc-nl-epub.recipe @@ -1,14 +1,14 @@ -#!/usr/bin/env python +#!/usr/bin/env python2 # -*- coding: utf-8 -*- -#Based on Lars Jacob's Taz Digiabo recipe +#Based on veezh's original recipe and Kovid Goyal's New York Times recipe __license__ = 'GPL v3' -__copyright__ = '2010, veezh' +__copyright__ = '2011, Snaab' ''' www.nrc.nl ''' -import os, urllib2, zipfile +import os, zipfile import time from calibre.web.feeds.news import BasicNewsRecipe from calibre.ptempfile import PersistentTemporaryFile @@ -17,41 +17,59 @@ from calibre.ptempfile import PersistentTemporaryFile class NRCHandelsblad(BasicNewsRecipe): title = u'NRC Handelsblad' - description = u'De EPUB-versie van NRC' + description = u'De ePaper-versie van NRC' language = 'nl' lang = 'nl-NL' + needs_subscription = True - __author__ = 'veezh' + __author__ = 'Snaab' conversion_options = { 'no_default_epub_cover' : True } + def get_browser(self): + br = BasicNewsRecipe.get_browser() + if self.username is not None and self.password is not None: + br.open('http://login.nrc.nl/login') + br.select_form(nr=0) + br['username'] = self.username + br['password'] = self.password + br.submit() + return br + def build_index(self): + today = time.strftime("%Y%m%d") + domain = "http://digitaleeditie.nrc.nl" url = domain + "/digitaleeditie/helekrant/epub/nrc_" + today + ".epub" -# print url + #print url try: - f = urllib2.urlopen(url) - except urllib2.HTTPError: + br = self.get_browser() + f = br.open(url) + except: self.report_progress(0,_('Kan niet inloggen om editie te downloaden')) raise ValueError('Krant van vandaag nog niet beschikbaar') + tmp = PersistentTemporaryFile(suffix='.epub') self.report_progress(0,_('downloading epub')) tmp.write(f.read()) - tmp.close() - - zfile = zipfile.ZipFile(tmp.name, 'r') - self.report_progress(0,_('extracting epub')) - - zfile.extractall(self.output_dir) + f.close() + br.close() + if zipfile.is_zipfile(tmp): + try: + zfile = zipfile.ZipFile(tmp.name, 'r') + zfile.extractall(self.output_dir) + self.report_progress(0,_('extracting epub')) + except zipfile.BadZipfile: + self.report_progress(0,_('BadZip error, continuing')) tmp.close() - index = os.path.join(self.output_dir, 'content.opf') + index = os.path.join(self.output_dir, 'metadata.opf') self.report_progress(1,_('epub downloaded and extracted')) diff --git a/resources/recipes/nytimes_sports.recipe b/resources/recipes/nytimes_sports.recipe new file mode 100644 index 0000000000..f394fc3232 --- /dev/null +++ b/resources/recipes/nytimes_sports.recipe @@ -0,0 +1,55 @@ +#!/usr/bin/env python +# encoding: utf-8 + +from __future__ import with_statement +__license__ = 'GPL 3' +__copyright__ = 'zotzo' +__docformat__ = 'restructuredtext en' +""" +http://fifthdown.blogs.nytimes.com/ +http://offthedribble.blogs.nytimes.com/ +http://thequad.blogs.nytimes.com/ +http://slapshot.blogs.nytimes.com/ +http://goal.blogs.nytimes.com/ +http://bats.blogs.nytimes.com/ +http://straightsets.blogs.nytimes.com/ +http://formulaone.blogs.nytimes.com/ +http://onpar.blogs.nytimes.com/ +""" + +from calibre.web.feeds.news import BasicNewsRecipe + + +class NYTimesSports(BasicNewsRecipe): + title = 'New York Times Sports Beat' + language = 'en' + __author__ = 'rylsfan' + description = 'Indepth sports from the New York Times' + publisher = 'The New York Times' + category = 'Sports' + oldest_article = 3 + max_articles_per_feed = 25 + no_stylesheets = True + language = 'en' + #cover_url ='http://bit.ly/h8F4DO' + feeds = [ + (u'The Fifth Down', u'http://fifthdown.blogs.nytimes.com/feed/'), + (u'Off The Dribble', u'http://offthedribble.blogs.nytimes.com/feed/'), + (u'The Quad', u'http://thequad.blogs.nytimes.com/feed/'), + (u'Slap Shot', u'http://slapshot.blogs.nytimes.com/feed/'), + (u'Goal', u'http://goal.blogs.nytimes.com/feed/'), + (u'Bats', u'http://bats.blogs.nytimes.com/feed/'), + (u'Straight Sets', u'http://straightsets.blogs.nytimes.com/feed/'), + (u'Formula One', u'http://formulaone.blogs.nytimes.com/feed/'), + (u'On Par', u'http://onpar.blogs.nytimes.com/feed/'), + ] + keep_only_tags = [dict(name='div', attrs={'id':'header'}), + dict(name='h1'), + dict(name='h2'), + dict(name='div', attrs={'class':'entry-content'})] + extra_css = ''' + h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;} + h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;} + p{font-family:Arial,Helvetica,sans-serif;font-size:small;} + body{font-family:Helvetica,Arial,sans-serif;font-size:small;} + ''' diff --git a/resources/recipes/nytimes_tech.recipe b/resources/recipes/nytimes_tech.recipe new file mode 100644 index 0000000000..7db2db9781 --- /dev/null +++ b/resources/recipes/nytimes_tech.recipe @@ -0,0 +1,46 @@ +#!/usr/bin/env python +# encoding: utf-8 + +from __future__ import with_statement +__license__ = 'GPL 3' +__copyright__ = 'zotzo' +__docformat__ = 'restructuredtext en' +""" +http://pogue.blogs.nytimes.com/ +""" + +from calibre.web.feeds.news import BasicNewsRecipe + + +class NYTimesTechnology(BasicNewsRecipe): + title = 'New York Times Technology Beat' + language = 'en' + __author__ = 'David Pogue' + description = 'The latest in technology from David Pogue' + publisher = 'The New York Times' + category = 'Technology' + oldest_article = 14 + max_articles_per_feed = 25 + no_stylesheets = True + language = 'en' + cover_url ='http://bit.ly/g0SKJT' + feeds = [ + (u'Pogues Posts', u'http://pogue.blogs.nytimes.com/feed/'), + (u'Bits', u'http://bits.blogs.nytimes.com/feed/'), + (u'Gadgetwise', u'http://gadgetwise.blogs.nytimes.com/feed/'), + (u'Open', u'http://open.blogs.nytimes.com/feed/') + ] + keep_only_tags = [dict(name='div', attrs={'id':'header'}), + dict(name='h1'), + dict(name='h2'), + dict(name='div', attrs={'class':'entry-content'})] + extra_css = ''' + h1{font-family:Arial,Helvetica,sans-serif; + font-weight:bold;font-size:large;} + + h2{font-family:Arial,Helvetica,sans-serif; + font-weight:normal;font-size:small;} + + p{font-family:Arial,Helvetica,sans-serif;font-size:small;} + body{font-family:Helvetica,Arial,sans-serif;font-size:small;} + ''' diff --git a/resources/recipes/post_today.recipe b/resources/recipes/post_today.recipe new file mode 100644 index 0000000000..a86e154b84 --- /dev/null +++ b/resources/recipes/post_today.recipe @@ -0,0 +1,21 @@ +from calibre.web.feeds.news import BasicNewsRecipe + +class AdvancedUserRecipe1299061355(BasicNewsRecipe): + title = u'Post Today' + language = 'th' + __author__ = "Chotechai P." + oldest_article = 7 + max_articles_per_feed = 100 + cover_url = 'http://upload.wikimedia.org/wikipedia/th/2/2e/Posttoday_Logo.png' + feeds = [(u'Breaking News', u'http://www.posttoday.com/rss/src/breakingnews.xml'), (u'\u0e02\u0e48\u0e32\u0e27', u'http://www.posttoday.com/rss/src/news.xml'), (u'\u0e27\u0e34\u0e40\u0e04\u0e23\u0e32\u0e30\u0e2b\u0e4c', u'http://www.posttoday.com/rss/src/analyse.xml'), (u'\u0e40\u0e21\u0e32\u0e17\u0e4c\u0e01\u0e31\u0e19\u0e43\u0e2b\u0e49 z', u'http://www.posttoday.com/rss/src/mouth.xml'), (u'\u0e44\u0e17\u0e22\u0e42\u0e0b\u0e44\u0e0b\u0e15\u0e35\u0e49', u'http://www.posttoday.com/rss/src/thaisociety.xml'), (u'\u0e44\u0e25\u0e1f\u0e4c\u0e2a\u0e44\u0e15\u0e25\u0e4c', u'http://www.posttoday.com/rss/src/lifestyle.xml'), (u'\u0e0a\u0e35\u0e49\u0e0a\u0e48\u0e2d\u0e07\u0e23\u0e27\u0e22', u'http://www.posttoday.com/rss/src/moneyguide.xml'), (u'\u0e1a\u0e49\u0e32\u0e19-\u0e04\u0e2d\u0e19\u0e42\u0e14', u'http://www.posttoday.com/rss/src/homecondo.xml'), (u'\u0e22\u0e32\u0e19\u0e22\u0e19\u0e15\u0e4c', u'http://www.posttoday.com/rss/src/motor.xml'), (u'\u0e14\u0e34\u0e08\u0e34\u0e15\u0e2d\u0e25\u0e44\u0e25\u0e1f\u0e4c', u'http://www.posttoday.com/rss/src/digitallife.xml'), (u'\u0e01\u0e35\u0e2c\u0e32', u'http://www.posttoday.com/rss/src/sport.xml'), (u'\u0e23\u0e2d\u0e1a\u0e42\u0e25\u0e01', u'http://www.posttoday.com/rss/src/world.xml'), (u'\u0e01\u0e34\u0e19-\u0e40\u0e17\u0e35\u0e48\u0e22\u0e27', u'http://www.posttoday.com/rss/src/eattravel.xml'), (u'Mind & Soul', u'http://www.posttoday.com/rss/src/mindsoul.xml'), (u'\u0e1a\u0e25\u0e47\u0e2d\u0e01 \u0e1a\u0e01.', u'http://www.posttoday.com/rss/src/blogs.xml')] + keep_only_tags = [] + keep_only_tags.append(dict(name = 'div', attrs = {'class' : +'articleContents'})) + + remove_tags = [] + remove_tags.append(dict(name = 'label')) + remove_tags.append(dict(name = 'span')) + remove_tags.append(dict(name = 'div', attrs = {'class' : +'socialBookmark'})) + remove_tags.append(dict(name = 'div', attrs = {'class' : +'misc'})) diff --git a/resources/recipes/promotor.recipe b/resources/recipes/promotor.recipe new file mode 100644 index 0000000000..11a8499d7b --- /dev/null +++ b/resources/recipes/promotor.recipe @@ -0,0 +1,70 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = u'2011, Silviu Cotoar\u0103' +''' +promotor.ro +''' + +from calibre.web.feeds.news import BasicNewsRecipe + +class Promotor(BasicNewsRecipe): + title = u'Promotor' + __author__ = u'Silviu Cotoar\u0103' + description = u'Auto-moto' + publisher = u'Promotor' + oldest_article = 25 + language = 'ro' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare,Reviste,TV,Auto' + encoding = 'utf-8' + cover_url = 'http://www.promotor.ro/images/logo_promotor.gif' + + conversion_options = { + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } + extra_css = ''' + h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;} + h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;} + .byline {font-family:Arial,Helvetica,sans-serif; font-size:xx-small;} + .date {font-family:Arial,Helvetica,sans-serif; font-size:xx-small;} + p{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .copyright {font-family:Arial,Helvetica,sans-serif;font-size:xx-small;text-align:center} + .story{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .entry-asset asset hentry{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .pagebody{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .maincontentcontainer{font-family:Arial,Helvetica,sans-serif;font-size:small;} + .story-body{font-family:Arial,Helvetica,sans-serif;font-size:small;} + body{font-family:Helvetica,Arial,sans-serif;font-size:small;} + ''' + + keep_only_tags = [ + dict(name='div', attrs={'class':'casetatitluarticol'}) + , dict(name='div', attrs={'style':'width: 273px; height: 210px; overflow: hidden; margin: 0pt auto;'}) + , dict(name='div', attrs={'class':'textb'}) + , dict(name='div', attrs={'class':'contentarticol'}) + ] + + remove_tags = [ + dict(name='td', attrs={'class':['connect_widget_vertical_center connect_widget_button_cell']}) + , dict(name='div', attrs={'class':['etichetagry']}) + , dict(name='span', attrs={'class':['textb']}) + ] + + remove_tags_after = [ + dict(name='div', attrs={'class':['etichetagry']}) + , dict(name='span', attrs={'class':['textb']}) + ] + + feeds = [ + (u'Feeds', u'http://www.promotor.ro/rss') + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) diff --git a/resources/recipes/rbc_ru.recipe b/resources/recipes/rbc_ru.recipe new file mode 100644 index 0000000000..4c377a334b --- /dev/null +++ b/resources/recipes/rbc_ru.recipe @@ -0,0 +1,49 @@ +from calibre.web.feeds.news import BasicNewsRecipe + +class AdvancedUserRecipe1286819935(BasicNewsRecipe): + title = u'RBC.ru' + __author__ = 'A. Chewi' + oldest_article = 7 + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + conversion_options = {'linearize_tables' : True} + remove_attributes = ['style'] + language = 'ru' + timefmt = ' [%a, %d %b, %Y]' + + keep_only_tags = [dict(name='h2', attrs={}), + dict(name='div', attrs={'class': 'box _ga1_on_'}), + dict(name='h1', attrs={'class': 'news_section'}), + dict(name='div', attrs={'class': 'news_body dotted_border_bottom'}), + dict(name='table', attrs={'class': 'newsBody'}), + dict(name='h2', attrs={'class': 'black'})] + + feeds = [(u'Главные новости', u'http://static.feed.rbc.ru/rbc/internal/rss.rbc.ru/rbc.ru/mainnews.rss'), + (u'Политика', u'http://static.feed.rbc.ru/rbc/internal/rss.rbc.ru/rbc.ru/politics.rss'), + (u'Экономика', u'http://static.feed.rbc.ru/rbc/internal/rss.rbc.ru/rbc.ru/economics.rss'), + (u'Общество', u'http://static.feed.rbc.ru/rbc/internal/rss.rbc.ru/rbc.ru/society.rss'), + (u'Происшествия', u'http://static.feed.rbc.ru/rbc/internal/rss.rbc.ru/rbc.ru/incidents.rss'), + (u'Финансовые новости Quote.rbc.ru', u'http://static.feed.rbc.ru/rbc/internal/rss.rbc.ru/quote.ru/mainnews.rss')] + + + remove_tags = [dict(name='div', attrs={'class': "video-frame"}), + dict(name='div', attrs={'class': "photo-container videoContainer videoSWFLinks videoPreviewSlideContainer notes"}), + dict(name='div', attrs={'class': "notes"}), + dict(name='div', attrs={'class': "publinks"}), + dict(name='a', attrs={'class': "print"}), + dict(name='div', attrs={'class': "photo-report_new notes newslider"}), + dict(name='div', attrs={'class': "videoContainer"}), + dict(name='div', attrs={'class': "videoPreviewSlideContainer"}), + dict(name='a', attrs={'class': "videoPreviewContainer"}), + dict(name='a', attrs={'class': "red"}),] + + def preprocess_html(self, soup): + for alink in soup.findAll('a'): + if alink.string is not None: + tstr = alink.string + alink.replaceWith(tstr) + return soup + + def print_version(self, url): + return url + '?print=true' diff --git a/resources/recipes/seattle_times.recipe b/resources/recipes/seattle_times.recipe index cd7f96fc8b..1d72df2991 100644 --- a/resources/recipes/seattle_times.recipe +++ b/resources/recipes/seattle_times.recipe @@ -69,12 +69,16 @@ class SeattleTimes(BasicNewsRecipe): u'http://seattletimes.nwsource.com/rss/mostreadarticles.xml'), ] + keep_only_tags = [dict(id='content')] remove_tags = [ - dict(name=['object','link','script']) - ,dict(name='p', attrs={'class':'permission'}) + dict(name=['object','link','script']), + {'class':['permission', 'note', 'bottomtools', + 'homedelivery']}, + dict(id=["rightcolumn", 'footer', 'adbottom']), ] def print_version(self, url): + return url start_url, sep, rest_url = url.rpartition('_') rurl, rsep, article_id = start_url.rpartition('/') return u'http://seattletimes.nwsource.com/cgi-bin/PrintStory.pl?document_id=' + article_id diff --git a/resources/recipes/statesman.recipe b/resources/recipes/statesman.recipe index 1bbf94fa5b..727df2ae61 100644 --- a/resources/recipes/statesman.recipe +++ b/resources/recipes/statesman.recipe @@ -10,12 +10,14 @@ class AdvancedUserRecipe1278049615(BasicNewsRecipe): max_articles_per_feed = 100 - feeds = [(u'News', u'http://www.statesman.com/section-rss.do?source=news&includeSubSections=true'), - (u'Business', u'http://www.statesman.com/section-rss.do?source=business&includeSubSections=true'), - (u'Life', u'http://www.statesman.com/section-rss.do?source=life&includesubsection=true'), - (u'Editorial', u'http://www.statesman.com/section-rss.do?source=opinion&includesubsections=true'), - (u'Sports', u'http://www.statesman.com/section-rss.do?source=sports&includeSubSections=true') - ] + feeds = [(u'News', + u'http://www.statesman.com/section-rss.do?source=news&includeSubSections=true'), + (u'Local', u'http://www.statesman.com/section-rss.do?source=local&includeSubSections=true'), + (u'Business', u'http://www.statesman.com/section-rss.do?source=business&includeSubSections=true'), + (u'Life', u'http://www.statesman.com/section-rss.do?source=life&includesubsection=true'), + (u'Editorial', u'http://www.statesman.com/section-rss.do?source=opinion&includesubsections=true'), + (u'Sports', u'http://www.statesman.com/section-rss.do?source=sports&includeSubSections=true') + ] masthead_url = "http://www.statesman.com/images/cmg-logo.gif" #temp_files = [] #articles_are_obfuscated = True @@ -28,8 +30,11 @@ class AdvancedUserRecipe1278049615(BasicNewsRecipe): conversion_options = {'linearize_tables':True} remove_tags = [ dict(name='div', attrs={'id':'cxArticleOptions'}), + {'class':['perma', 'comments', 'trail', 'share-buttons', + 'toggle_show_on']}, ] keep_only_tags = [ - dict(name='div', attrs={'class':'cxArticleHeader'}), - dict(name='div', attrs={'id':'cxArticleBodyText'}), + dict(name='div', attrs={'class':'cxArticleHeader'}), + dict(name='div', attrs={'id':['cxArticleBodyText', + 'content']}), ] diff --git a/resources/recipes/swiatkindle.recipe b/resources/recipes/swiatkindle.recipe index a6bf225294..9847d1359e 100644 --- a/resources/recipes/swiatkindle.recipe +++ b/resources/recipes/swiatkindle.recipe @@ -7,6 +7,7 @@ swiatczytnikow.pl ''' import re +from calibre.web.feeds.news import BasicNewsRecipe class swiatczytnikow(BasicNewsRecipe): title = u'Swiat Czytnikow' diff --git a/resources/recipes/thai_post_daily.recipe b/resources/recipes/thai_post_daily.recipe new file mode 100644 index 0000000000..2be17cc37f --- /dev/null +++ b/resources/recipes/thai_post_daily.recipe @@ -0,0 +1,17 @@ +from calibre.web.feeds.news import BasicNewsRecipe + +class AdvancedUserRecipe1299054026(BasicNewsRecipe): + title = u'Thai Post Daily' + __author__ = 'Chotechai P.' + oldest_article = 7 + max_articles_per_feed = 100 + cover_url = 'http://upload.wikimedia.org/wikipedia/th/1/10/ThaiPost_Logo.png' + feeds = [(u'\u0e02\u0e48\u0e32\u0e27\u0e2b\u0e19\u0e49\u0e32\u0e2b\u0e19\u0e36\u0e48\u0e07', u'http://thaipost.net/taxonomy/term/1/all/feed'), (u'\u0e1a\u0e17\u0e1a\u0e23\u0e23\u0e13\u0e32\u0e18\u0e34\u0e01\u0e32\u0e23', u'http://thaipost.net/taxonomy/term/11/all/feed'), (u'\u0e40\u0e1b\u0e25\u0e27 \u0e2a\u0e35\u0e40\u0e07\u0e34\u0e19', u'http://thaipost.net/taxonomy/term/2/all/feed'), (u'\u0e2a\u0e20\u0e32\u0e1b\u0e23\u0e30\u0e0a\u0e32\u0e0a\u0e19', u'http://thaipost.net/taxonomy/term/3/all/feed'), (u'\u0e16\u0e39\u0e01\u0e17\u0e38\u0e01\u0e02\u0e49\u0e2d', u'http://thaipost.net/taxonomy/term/4/all/feed'), (u'\u0e01\u0e32\u0e23\u0e40\u0e21\u0e37\u0e2d\u0e07', u'http://thaipost.net/taxonomy/term/5/all/feed'), (u'\u0e17\u0e48\u0e32\u0e19\u0e02\u0e38\u0e19\u0e19\u0e49\u0e2d\u0e22', u'http://thaipost.net/taxonomy/term/12/all/feed'), (u'\u0e1a\u0e17\u0e04\u0e27\u0e32\u0e21\u0e1e\u0e34\u0e40\u0e28\u0e29', u'http://thaipost.net/taxonomy/term/66/all/feed'), (u'\u0e23\u0e32\u0e22\u0e07\u0e32\u0e19\u0e1e\u0e34\u0e40\u0e28\u0e29', u'http://thaipost.net/taxonomy/term/67/all/feed'), (u'\u0e1a\u0e31\u0e19\u0e17\u0e36\u0e01\u0e2b\u0e19\u0e49\u0e32 4', u'http://thaipost.net/taxonomy/term/13/all/feed'), (u'\u0e40\u0e2a\u0e35\u0e22\u0e1a\u0e0b\u0e36\u0e48\u0e07\u0e2b\u0e19\u0e49\u0e32', u'http://thaipost.net/taxonomy/term/64/all/feed'), (u'\u0e04\u0e31\u0e19\u0e1b\u0e32\u0e01\u0e2d\u0e22\u0e32\u0e01\u0e40\u0e25\u0e48\u0e32', u'http://thaipost.net/taxonomy/term/65/all/feed'), (u'\u0e40\u0e28\u0e23\u0e29\u0e10\u0e01\u0e34\u0e08', u'http://thaipost.net/taxonomy/term/6/all/feed'), (u'\u0e01\u0e23\u0e30\u0e08\u0e01\u0e44\u0e23\u0e49\u0e40\u0e07\u0e32', u'http://thaipost.net/taxonomy/term/14/all/feed'), (u'\u0e01\u0e23\u0e30\u0e08\u0e01\u0e2b\u0e31\u0e01\u0e21\u0e38\u0e21', u'http://thaipost.net/taxonomy/term/71/all/feed'), (u'\u0e04\u0e34\u0e14\u0e40\u0e2b\u0e19\u0e37\u0e2d\u0e01\u0e23\u0e30\u0e41\u0e2a', u'http://thaipost.net/taxonomy/term/69/all/feed'), (u'\u0e23\u0e32\u0e22\u0e07\u0e32\u0e19', u'http://thaipost.net/taxonomy/term/68/all/feed'), (u'\u0e2d\u0e34\u0e42\u0e04\u0e42\u0e1f\u0e01\u0e31\u0e2a', u'http://thaipost.net/taxonomy/term/10/all/feed'), (u'\u0e01\u0e32\u0e23\u0e28\u0e36\u0e01\u0e29\u0e32-\u0e2a\u0e32\u0e18\u0e32\u0e23\u0e13\u0e2a\u0e38\u0e02', u'http://thaipost.net/taxonomy/term/7/all/feed'), (u'\u0e15\u0e48\u0e32\u0e07\u0e1b\u0e23\u0e30\u0e40\u0e17\u0e28', u'http://thaipost.net/taxonomy/term/8/all/feed'), (u'\u0e01\u0e35\u0e2c\u0e32', u'http://thaipost.net/taxonomy/term/9/all/feed')] + + def print_version(self, url): + return url.replace(url, 'http://www.thaipost.net/print/' + url [32:]) + + remove_tags = [] + remove_tags.append(dict(name = 'div', attrs = {'class' : 'print-logo'})) + remove_tags.append(dict(name = 'div', attrs = {'class' : 'print-site_name'})) + remove_tags.append(dict(name = 'div', attrs = {'class' : 'print-breadcrumb'})) diff --git a/resources/recipes/timesnewroman.recipe b/resources/recipes/timesnewroman.recipe new file mode 100644 index 0000000000..12672aa888 --- /dev/null +++ b/resources/recipes/timesnewroman.recipe @@ -0,0 +1,52 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = u'2011, Silviu Cotoar\u0103' +''' +timesnewroman.ro +''' + +from calibre.web.feeds.news import BasicNewsRecipe + +class TimesNewRoman(BasicNewsRecipe): + title = u'Times New Roman' + __author__ = u'Silviu Cotoar\u0103' + description = u'Cotidian independent de umor voluntar' + publisher = u'Times New Roman' + oldest_article = 25 + language = 'ro' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare,Reviste,Fun' + encoding = 'utf-8' + cover_url = 'http://www.timesnewroman.ro/templates/TNRV2/images/logo.gif' + + conversion_options = { + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } + + keep_only_tags = [ + dict(name='div', attrs={'id':'page'}) + ] + + remove_tags = [ + dict(name='p', attrs={'class':['articleinfo']}) + , dict(name='div',attrs={'class':['vergefacebooklike']}) + , dict(name='div', attrs={'class':'cleared'}) + ] + + remove_tags_after = [ + dict(name='div', attrs={'class':'cleared'}) + ] + + feeds = [ + (u'Feeds', u'http://www.timesnewroman.ro/index.php?format=feed&type=rss') + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) diff --git a/resources/recipes/trombon.recipe b/resources/recipes/trombon.recipe new file mode 100644 index 0000000000..1a4e488a43 --- /dev/null +++ b/resources/recipes/trombon.recipe @@ -0,0 +1,51 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = u'2011, Silviu Cotoar\u0103' +''' +trombon.ro +''' + +from calibre.web.feeds.news import BasicNewsRecipe + +class Trombon(BasicNewsRecipe): + title = u'Trombon' + __author__ = u'Silviu Cotoar\u0103' + description = u'Parodii si Pamflete' + publisher = u'Trombon' + oldest_article = 5 + language = 'ro' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare,Reviste,Fun' + encoding = 'utf-8' + cover_url = 'http://www.trombon.ro/i/trombon.gif' + + conversion_options = { + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } + + keep_only_tags = [ + dict(name='div', attrs={'class':'articol'}) + ] + + remove_tags = [ + dict(name='div', attrs={'class':['info_2']}) + , dict(name='iframe', attrs={'scrolling':['no']}) + ] + + remove_tags_after = [ + dict(name='div', attrs={'id':'article_vote'}) + ] + + feeds = [ + (u'Feeds', u'http://feeds.feedburner.com/trombon/ABWb?format=xml') + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) diff --git a/resources/recipes/wallstreetro.recipe b/resources/recipes/wallstreetro.recipe new file mode 100644 index 0000000000..8a66aa3673 --- /dev/null +++ b/resources/recipes/wallstreetro.recipe @@ -0,0 +1,54 @@ +# -*- coding: utf-8 -*- +#!/usr/bin/env python + +__license__ = 'GPL v3' +__copyright__ = u'2011, Silviu Cotoar\u0103' +''' +wall-street.ro +''' + +from calibre.web.feeds.news import BasicNewsRecipe + +class WallStreetRo(BasicNewsRecipe): + title = u'Wall Street' + __author__ = u'Silviu Cotoar\u0103' + description = '' + publisher = 'Wall Street' + oldest_article = 5 + language = 'ro' + max_articles_per_feed = 100 + no_stylesheets = True + use_embedded_content = False + category = 'Ziare' + encoding = 'utf-8' + cover_url = 'http://img.wall-street.ro/images/WS_new_logo.jpg' + + conversion_options = { + 'comments' : description + ,'tags' : category + ,'language' : language + ,'publisher' : publisher + } + + keep_only_tags = [ + dict(name='div', attrs={'class':'article_header'}) + , dict(name='div', attrs={'class':'article_text'}) + ] + + remove_tags = [ + dict(name='p', attrs={'class':['page_breadcrumbs']}) + , dict(name='div', attrs={'id':['article_user_toolbox']}) + , dict(name='p', attrs={'class':['comments_count_container']}) + , dict(name='div', attrs={'class':['article_left_column']}) + ] + + remove_tags_after = [ + dict(name='div', attrs={'class':'clearfloat'}) + ] + + feeds = [ + (u'Feeds', u'http://img.wall-street.ro/rssfeeds/wall-street.xml') + ] + + def preprocess_html(self, soup): + return self.adeify_images(soup) diff --git a/resources/template-functions.json b/resources/template-functions.json index 5d9b6a11a3..fe4379d701 100644 --- a/resources/template-functions.json +++ b/resources/template-functions.json @@ -5,8 +5,9 @@ "strcat": "def evaluate(self, formatter, kwargs, mi, locals, *args):\n i = 0\n res = ''\n for i in range(0, len(args)):\n res += args[i]\n return res\n", "substr": "def evaluate(self, formatter, kwargs, mi, locals, str_, start_, end_):\n return str_[int(start_): len(str_) if int(end_) == 0 else int(end_)]\n", "ifempty": "def evaluate(self, formatter, kwargs, mi, locals, val, value_if_empty):\n if val:\n return val\n else:\n return value_if_empty\n", + "select": "def evaluate(self, formatter, kwargs, mi, locals, val, key):\n if not val:\n return ''\n vals = [v.strip() for v in val.split(',')]\n for v in vals:\n if v.startswith(key+':'):\n return v[len(key)+1:]\n return ''\n", "field": "def evaluate(self, formatter, kwargs, mi, locals, name):\n return formatter.get_value(name, [], kwargs)\n", - "capitalize": "def evaluate(self, formatter, kwargs, mi, locals, val):\n return capitalize(val)\n", + "subtract": "def evaluate(self, formatter, kwargs, mi, locals, x, y):\n x = float(x if x else 0)\n y = float(y if y else 0)\n return unicode(x - y)\n", "list_item": "def evaluate(self, formatter, kwargs, mi, locals, val, index, sep):\n if not val:\n return ''\n index = int(index)\n val = val.split(sep)\n try:\n return val[index]\n except:\n return ''\n", "shorten": "def evaluate(self, formatter, kwargs, mi, locals,\n val, leading, center_string, trailing):\n l = max(0, int(leading))\n t = max(0, int(trailing))\n if len(val) > l + len(center_string) + t:\n return val[0:l] + center_string + ('' if t == 0 else val[-t:])\n else:\n return val\n", "re": "def evaluate(self, formatter, kwargs, mi, locals, val, pattern, replacement):\n return re.sub(pattern, replacement, val)\n", @@ -19,11 +20,13 @@ "test": "def evaluate(self, formatter, kwargs, mi, locals, val, value_if_set, value_not_set):\n if val:\n return value_if_set\n else:\n return value_not_set\n", "eval": "def evaluate(self, formatter, kwargs, mi, locals, template):\n from formatter import eval_formatter\n template = template.replace('[[', '{').replace(']]', '}')\n return eval_formatter.safe_format(template, locals, 'EVAL', None)\n", "multiply": "def evaluate(self, formatter, kwargs, mi, locals, x, y):\n x = float(x if x else 0)\n y = float(y if y else 0)\n return unicode(x * y)\n", - "subtract": "def evaluate(self, formatter, kwargs, mi, locals, x, y):\n x = float(x if x else 0)\n y = float(y if y else 0)\n return unicode(x - y)\n", + "format_date": "def evaluate(self, formatter, kwargs, mi, locals, val, format_string):\n if not val:\n return ''\n try:\n dt = parse_date(val)\n s = format_date(dt, format_string)\n except:\n s = 'BAD DATE'\n return s\n", + "capitalize": "def evaluate(self, formatter, kwargs, mi, locals, val):\n return capitalize(val)\n", "count": "def evaluate(self, formatter, kwargs, mi, locals, val, sep):\n return unicode(len(val.split(sep)))\n", "lowercase": "def evaluate(self, formatter, kwargs, mi, locals, val):\n return val.lower()\n", "assign": "def evaluate(self, formatter, kwargs, mi, locals, target, value):\n locals[target] = value\n return value\n", "switch": "def evaluate(self, formatter, kwargs, mi, locals, val, *args):\n if (len(args) % 2) != 1:\n raise ValueError(_('switch requires an odd number of arguments'))\n i = 0\n while i < len(args):\n if i + 1 >= len(args):\n return args[i]\n if re.search(args[i], val):\n return args[i+1]\n i += 2\n", "strcmp": "def evaluate(self, formatter, kwargs, mi, locals, x, y, lt, eq, gt):\n v = strcmp(x, y)\n if v < 0:\n return lt\n if v == 0:\n return eq\n return gt\n", + "raw_field": "def evaluate(self, formatter, kwargs, mi, locals, name):\n return unicode(getattr(mi, name, None))\n", "cmp": "def evaluate(self, formatter, kwargs, mi, locals, x, y, lt, eq, gt):\n x = float(x if x else 0)\n y = float(y if y else 0)\n if x < y:\n return lt\n if x == y:\n return eq\n return gt\n" } \ No newline at end of file diff --git a/resources/templates/fb2.xsl b/resources/templates/fb2.xsl index 77c03cdc74..273edd71ae 100644 --- a/resources/templates/fb2.xsl +++ b/resources/templates/fb2.xsl @@ -4,6 +4,7 @@ # # # # # copyright 2002 Paul Henry Tremblay # +# Copyright 2011 Kovid Goyal # # # This program is distributed in the hope that it will be useful, # # but WITHOUT ANY WARRANTY; without even the implied warranty of # @@ -19,21 +20,21 @@ ######################################################################### --> - - - - - - - - - - <xsl:value-of select="fb:description/fb:title-info/fb:book-title"/> - - + - - - -
    - -
    -
    -     - -
      - -
    -
    + + + +
    + +
    +
    +     + +
      + +
    +
    - - - -
    -     - -

    - -

    -     - - -     - - -     - - - - - + + + +
    +     + +

    + +

    +     + + +     + + +     + + + + + - -
    -     - - -
  1. - - - , # - - - -
      - - - -
    -     - - - - - - - - - -
  2. - - -
    -     -
    3. - - - - - - - - - -
    -     + +
    + + + +
  3. + + + , # + + + +
      + + + +
    +     + + + + + + + + + +
  4. + + +
    +     +
    5. + + +     + + + + + + +
    +     - + @@ -164,15 +165,15 @@ - - - - - + + + + + - - - + + + @@ -181,79 +182,79 @@ TOC_ - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + - + - -
    -     - - - - - - - -
    - -
    -     - - + +
    +     + + + + + + + +
    + +
    +     + + paragraph - - - - + + + + - - - - - - - - - - - - - - - -
    -     +     + + + + + + + + + + + + + + +
    +     @@ -261,123 +262,140 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    Annotation

    - -     - - -
    - - - - - - -
    -     - - -
    - -
    -     - - -
    - - - - - - -
    -     - - -
    -
    -     - - - - -     -
    -     - -     -
    -     -     -     - - -
    - - - - - - -
    -     + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

    Annotation

    + +     + + + + +
    +     + + + + + + + + + + + + + +
    + + + + + + +
    +     + + +
    + +
    +     + + +
    + + + + + + +
    +     + + +
    +
    +     + + + + +     +
    +     + +     +
    +     +     +     + + +
    + + + + + + +
    +     - - - -
    -     - - - - - - - -
    -     - - -
    - - - - - - - - - - -
    -     + + + +
    +     + + + + + + + +
    +     + + +
    + + + + + + + + + + +
    +     diff --git a/setup/extensions.py b/setup/extensions.py index 6a9cce7625..6e8e7ce4b7 100644 --- a/setup/extensions.py +++ b/setup/extensions.py @@ -68,6 +68,10 @@ if isosx: extensions = [ + Extension('speedup', + ['calibre/utils/speedup.c'], + ), + Extension('icu', ['calibre/utils/icu.c'], libraries=icu_libs, diff --git a/src/calibre/__init__.py b/src/calibre/__init__.py index 221f5911c6..fa9a8f2404 100644 --- a/src/calibre/__init__.py +++ b/src/calibre/__init__.py @@ -61,8 +61,9 @@ def osx_version(): if m: return int(m.group(1)), int(m.group(2)), int(m.group(3)) - _filename_sanitize = re.compile(r'[\xae\0\\|\?\*<":>\+/]') +_filename_sanitize_unicode = frozenset([u'\\', u'|', u'?', u'*', u'<', + u'"', u':', u'>', u'+', u'/'] + list(map(unichr, xrange(32)))) def sanitize_file_name(name, substitute='_', as_unicode=False): ''' @@ -83,8 +84,35 @@ def sanitize_file_name(name, substitute='_', as_unicode=False): one = one.decode(filesystem_encoding) one = one.replace('..', substitute) # Windows doesn't like path components that end with a period - if one.endswith('.'): + if one and one[-1] in ('.', ' '): one = one[:-1]+'_' + # Names starting with a period are hidden on Unix + if one.startswith('.'): + one = '_' + one[1:] + return one + +def sanitize_file_name_unicode(name, substitute='_'): + ''' + Sanitize the filename `name`. All invalid characters are replaced by `substitute`. + The set of invalid characters is the union of the invalid characters in Windows, + OS X and Linux. Also removes leading and trailing whitespace. + **WARNING:** This function also replaces path separators, so only pass file names + and not full paths to it. + ''' + if not isinstance(name, unicode): + return sanitize_file_name(name, substitute=substitute, as_unicode=True) + chars = [substitute if c in _filename_sanitize_unicode else c for c in + name] + one = u''.join(chars) + one = re.sub(r'\s', ' ', one).strip() + one = re.sub(r'^\.+$', '_', one) + one = one.replace('..', substitute) + # Windows doesn't like path components that end with a period or space + if one and one[-1] in ('.', ' '): + one = one[:-1]+'_' + # Names starting with a period are hidden on Unix + if one.startswith('.'): + one = '_' + one[1:] return one diff --git a/src/calibre/constants.py b/src/calibre/constants.py index def8e631c0..2c4ae9d512 100644 --- a/src/calibre/constants.py +++ b/src/calibre/constants.py @@ -2,7 +2,7 @@ __license__ = 'GPL v3' __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net' __docformat__ = 'restructuredtext en' __appname__ = 'calibre' -__version__ = '0.7.47' +__version__ = '0.7.48' __author__ = "Kovid Goyal " import re @@ -69,6 +69,7 @@ if plugins is None: 'chmlib', 'chm_extra', 'icu', + 'speedup', ] + \ (['winutil'] if iswindows else []) + \ (['usbobserver'] if isosx else []): diff --git a/src/calibre/devices/android/driver.py b/src/calibre/devices/android/driver.py index 3724f02ca2..e1a806af0f 100644 --- a/src/calibre/devices/android/driver.py +++ b/src/calibre/devices/android/driver.py @@ -57,7 +57,7 @@ class ANDROID(USBMS): 0x413c : { 0xb007 : [0x0100, 0x0224]}, # LG - 0x1004 : { 0x61cc : [0x100] }, + 0x1004 : { 0x61cc : [0x100], 0x61ce : [0x100] }, # Archos 0x0e79 : { @@ -78,6 +78,9 @@ class ANDROID(USBMS): # Xperia 0x13d3 : { 0x3304 : [0x0001, 0x0002] }, + # CREEL?? Also Nextbook + 0x5e3 : { 0x726 : [0x222] }, + } EBOOK_DIR_MAIN = ['eBooks/import', 'wordplayer/calibretransfer', 'Books'] EXTRA_CUSTOMIZATION_MESSAGE = _('Comma separated list of directories to ' diff --git a/src/calibre/devices/apple/driver.py b/src/calibre/devices/apple/driver.py index aaa9382612..76ecce3a8e 100644 --- a/src/calibre/devices/apple/driver.py +++ b/src/calibre/devices/apple/driver.py @@ -701,7 +701,7 @@ class ITUNES(DriverBase): self.log.info("ITUNES.get_file(): exporting '%s'" % path) outfile.write(open(self.cached_books[path]['lib_book'].location().path).read()) - def open(self): + def open(self, library_uuid): ''' Perform any device specific initialization. Called after the device is detected but before any other functions that communicate with the device. @@ -2512,7 +2512,12 @@ class ITUNES(DriverBase): # Refresh epub metadata with open(fpath,'r+b') as zfo: # Touch the OPF timestamp - zf_opf = ZipFile(fpath,'r') + try: + zf_opf = ZipFile(fpath,'r') + except: + raise UserFeedback("'%s' is not a valid EPUB" % metadata.title, + None, + level=UserFeedback.WARN) fnames = zf_opf.namelist() opf = [x for x in fnames if '.opf' in x][0] if opf: diff --git a/src/calibre/devices/bambook/driver.py b/src/calibre/devices/bambook/driver.py index 3cc0245cf7..f251310d77 100644 --- a/src/calibre/devices/bambook/driver.py +++ b/src/calibre/devices/bambook/driver.py @@ -61,7 +61,7 @@ class BAMBOOK(DeviceConfig, DevicePlugin): detected_device=None) : self.open() - def open(self): + def open(self, library_uuid): # Make sure the Bambook library is ready if not is_bambook_lib_ready(): raise OpenFeedback(_("Unable to connect to Bambook, you need to install Bambook library first.")) diff --git a/src/calibre/devices/folder_device/driver.py b/src/calibre/devices/folder_device/driver.py index d75697a6cb..c08448051d 100644 --- a/src/calibre/devices/folder_device/driver.py +++ b/src/calibre/devices/folder_device/driver.py @@ -47,6 +47,7 @@ class FOLDER_DEVICE(USBMS): #: Icon for this device icon = I('devices/folder.png') METADATA_CACHE = '.metadata.calibre' + DRIVEINFO = '.driveinfo.calibre' _main_prefix = '' _card_a_prefix = None @@ -77,7 +78,8 @@ class FOLDER_DEVICE(USBMS): only_presence=False): return self.is_connected, self - def open(self): + def open(self, library_uuid): + self.current_library_uuid = library_uuid if not self._main_prefix: return False return True diff --git a/src/calibre/devices/hanlin/driver.py b/src/calibre/devices/hanlin/driver.py index 37f8430a66..ba0cca954d 100644 --- a/src/calibre/devices/hanlin/driver.py +++ b/src/calibre/devices/hanlin/driver.py @@ -116,6 +116,7 @@ class BOOX(HANLINV3): author = 'Jesus Manuel Marinho Valcarce' supported_platforms = ['windows', 'osx', 'linux'] METADATA_CACHE = '.metadata.calibre' + DRIVEINFO = '.driveinfo.calibre' # Ordered list of supported formats FORMATS = ['epub', 'fb2', 'djvu', 'pdf', 'html', 'txt', 'rtf', 'mobi', diff --git a/src/calibre/devices/interface.py b/src/calibre/devices/interface.py index bc442f5853..b265331ace 100644 --- a/src/calibre/devices/interface.py +++ b/src/calibre/devices/interface.py @@ -215,7 +215,7 @@ class DevicePlugin(Plugin): return True - def open(self): + def open(self, library_uuid): ''' Perform any device specific initialization. Called after the device is detected but before any other functions that communicate with the device. @@ -260,6 +260,8 @@ class DevicePlugin(Plugin): Ask device for device information. See L{DeviceInfoQuery}. :return: (device name, device version, software version on device, mime type) + The tuple can optionally have a fifth element, which is a + drive information diction. See usbms.driver for an example. """ raise NotImplementedError() @@ -447,6 +449,15 @@ class DevicePlugin(Plugin): ''' pass + def set_driveinfo_name(self, location_code, name): + ''' + Set the device name in the driveinfo file to 'name'. This setting will + persist until the file is re-created or the name is changed again. + + Non-disk devices will ignore this request. + ''' + pass + class BookList(list): ''' A list of books. Each Book object must have the fields diff --git a/src/calibre/devices/misc.py b/src/calibre/devices/misc.py index 8cf0fb5a06..07b381d11a 100644 --- a/src/calibre/devices/misc.py +++ b/src/calibre/devices/misc.py @@ -272,6 +272,7 @@ class NEXTBOOK(USBMS): VENDOR_NAME = 'NEXT2' WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = '1.0.14' SUPPORTS_SUB_DIRS = True + THUMBNAIL_HEIGHT = 120 ''' def upload_cover(self, path, filename, metadata, filepath): diff --git a/src/calibre/devices/prs500/cli/main.py b/src/calibre/devices/prs500/cli/main.py index cd8395467b..6d568b01a2 100755 --- a/src/calibre/devices/prs500/cli/main.py +++ b/src/calibre/devices/prs500/cli/main.py @@ -213,7 +213,7 @@ def main(): for d in connected_devices: try: - d.open() + d.open(None) except: continue else: diff --git a/src/calibre/devices/prs500/driver.py b/src/calibre/devices/prs500/driver.py index 445ddd757b..65ecc98a81 100644 --- a/src/calibre/devices/prs500/driver.py +++ b/src/calibre/devices/prs500/driver.py @@ -240,7 +240,7 @@ class PRS500(DeviceConfig, DevicePlugin): def set_progress_reporter(self, report_progress): self.report_progress = report_progress - def open(self) : + def open(self, library_uuid) : """ Claim an interface on the device for communication. Requires write privileges to the device file. diff --git a/src/calibre/devices/prs505/driver.py b/src/calibre/devices/prs505/driver.py index 3768b8be62..9f17ea22a4 100644 --- a/src/calibre/devices/prs505/driver.py +++ b/src/calibre/devices/prs505/driver.py @@ -153,9 +153,6 @@ class PRS505(USBMS): # updated on every connect self.WANTS_UPDATED_THUMBNAILS = self.settings().extra_customization[2] - def get_device_information(self, end_session=True): - return (self.gui_name, '', '', '') - def filename_callback(self, fname, mi): if getattr(mi, 'application_id', None) is not None: base = fname.rpartition('.')[0] diff --git a/src/calibre/devices/usbms/device.py b/src/calibre/devices/usbms/device.py index b0857de909..37b2b061e5 100644 --- a/src/calibre/devices/usbms/device.py +++ b/src/calibre/devices/usbms/device.py @@ -700,7 +700,7 @@ class Device(DeviceConfig, DevicePlugin): - def open(self): + def open(self, library_uuid): time.sleep(5) self._main_prefix = self._card_a_prefix = self._card_b_prefix = None if islinux: @@ -722,6 +722,7 @@ class Device(DeviceConfig, DevicePlugin): time.sleep(7) self.open_osx() + self.current_library_uuid = library_uuid self.post_open_callback() def post_open_callback(self): diff --git a/src/calibre/devices/usbms/driver.py b/src/calibre/devices/usbms/driver.py index ef654ac428..a19df07abf 100644 --- a/src/calibre/devices/usbms/driver.py +++ b/src/calibre/devices/usbms/driver.py @@ -10,17 +10,18 @@ driver. It is intended to be subclassed with the relevant parts implemented for a particular device. ''' -import os -import re -import time +import os, re, time, json, uuid from itertools import cycle +from calibre.constants import numeric_version from calibre import prints, isbytestring from calibre.constants import filesystem_encoding, DEBUG from calibre.devices.usbms.cli import CLI from calibre.devices.usbms.device import Device from calibre.devices.usbms.books import BookList, Book from calibre.ebooks.metadata.book.json_codec import JsonCodec +from calibre.utils.config import from_json, to_json +from calibre.utils.date import now, isoformat BASE_TIME = None def debug_print(*args): @@ -52,10 +53,59 @@ class USBMS(CLI, Device): FORMATS = [] CAN_SET_METADATA = [] METADATA_CACHE = 'metadata.calibre' + DRIVEINFO = 'driveinfo.calibre' + + def _update_driveinfo_record(self, dinfo, prefix, location_code, name=None): + if not isinstance(dinfo, dict): + dinfo = {} + if dinfo.get('device_store_uuid', None) is None: + dinfo['device_store_uuid'] = unicode(uuid.uuid4()) + if dinfo.get('device_name') is None: + dinfo['device_name'] = self.get_gui_name() + if name is not None: + dinfo['device_name'] = name + dinfo['location_code'] = location_code + dinfo['last_library_uuid'] = getattr(self, 'current_library_uuid', None) + dinfo['calibre_version'] = '.'.join([unicode(i) for i in numeric_version]) + dinfo['date_last_connected'] = isoformat(now()) + dinfo['prefix'] = prefix.replace('\\', '/') + return dinfo + + def _update_driveinfo_file(self, prefix, location_code, name=None): + if os.path.exists(os.path.join(prefix, self.DRIVEINFO)): + with open(os.path.join(prefix, self.DRIVEINFO), 'rb') as f: + try: + driveinfo = json.loads(f.read(), object_hook=from_json) + except: + driveinfo = None + driveinfo = self._update_driveinfo_record(driveinfo, prefix, + location_code, name) + with open(os.path.join(prefix, self.DRIVEINFO), 'wb') as f: + f.write(json.dumps(driveinfo, default=to_json)) + else: + driveinfo = self._update_driveinfo_record({}, prefix, location_code, name) + with open(os.path.join(prefix, self.DRIVEINFO), 'wb') as f: + f.write(json.dumps(driveinfo, default=to_json)) + return driveinfo def get_device_information(self, end_session=True): self.report_progress(1.0, _('Get device information...')) - return (self.get_gui_name(), '', '', '') + self.driveinfo = {} + if self._main_prefix is not None: + self.driveinfo['main'] = self._update_driveinfo_file(self._main_prefix, 'main') + if self._card_a_prefix is not None: + self.driveinfo['A'] = self._update_driveinfo_file(self._card_a_prefix, 'A') + if self._card_b_prefix is not None: + self.driveinfo['B'] = self._update_driveinfo_file(self._card_b_prefix, 'B') + return (self.get_gui_name(), '', '', '', self.driveinfo) + + def set_driveinfo_name(self, location_code, name): + if location_code == 'main': + self._update_driveinfo_file(self._main_prefix, location_code, name) + elif location_code == 'A': + self._update_driveinfo_file(self._card_a_prefix, location_code, name) + elif location_code == 'B': + self._update_driveinfo_file(self._card_b_prefix, location_code, name) def books(self, oncard=None, end_session=True): from calibre.ebooks.metadata.meta import path_to_ext diff --git a/src/calibre/ebooks/__init__.py b/src/calibre/ebooks/__init__.py index dcd32811b3..c5bac936b5 100644 --- a/src/calibre/ebooks/__init__.py +++ b/src/calibre/ebooks/__init__.py @@ -25,10 +25,10 @@ class DRMError(ValueError): class ParserError(ValueError): pass -BOOK_EXTENSIONS = ['lrf', 'rar', 'zip', 'rtf', 'lit', 'txt', 'txtz', 'htm', 'xhtm', +BOOK_EXTENSIONS = ['lrf', 'rar', 'zip', 'rtf', 'lit', 'txt', 'txtz', 'text', 'htm', 'xhtm', 'html', 'xhtml', 'pdf', 'pdb', 'pdr', 'prc', 'mobi', 'azw', 'doc', 'epub', 'fb2', 'djvu', 'lrx', 'cbr', 'cbz', 'cbc', 'oebzip', - 'rb', 'imp', 'odt', 'chm', 'tpz', 'azw1', 'pml', 'mbp', 'tan', 'snb'] + 'rb', 'imp', 'odt', 'chm', 'tpz', 'azw1', 'pml', 'pmlz', 'mbp', 'tan', 'snb'] class HTMLRenderer(object): diff --git a/src/calibre/ebooks/chm/input.py b/src/calibre/ebooks/chm/input.py index 89efa2b4d1..f55a76d67e 100644 --- a/src/calibre/ebooks/chm/input.py +++ b/src/calibre/ebooks/chm/input.py @@ -22,7 +22,7 @@ class CHMInput(InputFormatPlugin): def _chmtohtml(self, output_dir, chm_path, no_images, log): from calibre.ebooks.chm.reader import CHMReader log.debug('Opening CHM file') - rdr = CHMReader(chm_path, log) + rdr = CHMReader(chm_path, log, self.opts) log.debug('Extracting CHM to %s' % output_dir) rdr.extract_content(output_dir) self._chm_reader = rdr @@ -32,13 +32,13 @@ class CHMInput(InputFormatPlugin): def convert(self, stream, options, file_ext, log, accelerators): from calibre.ebooks.chm.metadata import get_metadata_from_reader from calibre.customize.ui import plugin_for_input_format + self.opts = options log.debug('Processing CHM...') with TemporaryDirectory('_chm2oeb') as tdir: html_input = plugin_for_input_format('html') for opt in html_input.options: setattr(options, opt.option.name, opt.recommended_value) - options.input_encoding = 'utf-8' no_images = False #options.no_images chm_name = stream.name #chm_data = stream.read() @@ -54,6 +54,7 @@ class CHMInput(InputFormatPlugin): odi = options.debug_pipeline options.debug_pipeline = None + options.input_encoding = 'utf-8' # try a custom conversion: #oeb = self._create_oebbook(mainpath, tdir, options, log, metadata) # try using html converter: diff --git a/src/calibre/ebooks/chm/reader.py b/src/calibre/ebooks/chm/reader.py index 04ce6d5efe..34d228ef3b 100644 --- a/src/calibre/ebooks/chm/reader.py +++ b/src/calibre/ebooks/chm/reader.py @@ -40,13 +40,14 @@ class CHMError(Exception): pass class CHMReader(CHMFile): - def __init__(self, input, log): + def __init__(self, input, log, opts): CHMFile.__init__(self) if isinstance(input, unicode): input = input.encode(filesystem_encoding) if not self.LoadCHM(input): raise CHMError("Unable to open CHM file '%s'"%(input,)) self.log = log + self.opts = opts self._sourcechm = input self._contents = None self._playorder = 0 @@ -54,8 +55,12 @@ class CHMReader(CHMFile): self._extracted = False # location of '.hhc' file, which is the CHM TOC. - self.root, ext = os.path.splitext(self.topics.lstrip('/')) - self.hhc_path = self.root + ".hhc" + if self.topics is None: + self.root, ext = os.path.splitext(self.home.lstrip('/')) + self.hhc_path = self.root + ".hhc" + else: + self.root, ext = os.path.splitext(self.topics.lstrip('/')) + self.hhc_path = self.root + ".hhc" def _parse_toc(self, ul, basedir=os.getcwdu()): toc = TOC(play_order=self._playorder, base_path=basedir, text='') @@ -147,6 +152,8 @@ class CHMReader(CHMFile): break def _reformat(self, data, htmlpath): + if self.opts.input_encoding: + data = data.decode(self.opts.input_encoding) try: data = xml_to_unicode(data, strip_encoding_pats=True)[0] soup = BeautifulSoup(data) diff --git a/src/calibre/ebooks/comic/input.py b/src/calibre/ebooks/comic/input.py index 7710d41fb3..56fa123249 100755 --- a/src/calibre/ebooks/comic/input.py +++ b/src/calibre/ebooks/comic/input.py @@ -131,9 +131,12 @@ class PageProcessor(list): # {{{ newsizey = int(newsizex / aspect) deltax = 0 deltay = (SCRHEIGHT - newsizey) / 2 - wand.size = (newsizex, newsizey) - wand.set_border_color(pw) - wand.add_border(pw, deltax, deltay) + if newsizex < 20000 and newsizey < 20000: + # Too large and resizing fails, so better + # to leave it as original size + wand.size = (newsizex, newsizey) + wand.set_border_color(pw) + wand.add_border(pw, deltax, deltay) elif self.opts.wide: # Keep aspect and Use device height as scaled image width so landscape mode is clean aspect = float(sizex) / float(sizey) @@ -152,11 +155,15 @@ class PageProcessor(list): # {{{ newsizey = int(newsizex / aspect) deltax = 0 deltay = (wscreeny - newsizey) / 2 - wand.size = (newsizex, newsizey) - wand.set_border_color(pw) - wand.add_border(pw, deltax, deltay) + if newsizex < 20000 and newsizey < 20000: + # Too large and resizing fails, so better + # to leave it as original size + wand.size = (newsizex, newsizey) + wand.set_border_color(pw) + wand.add_border(pw, deltax, deltay) else: - wand.size = (SCRWIDTH, SCRHEIGHT) + if SCRWIDTH < 20000 and SCRHEIGHT < 20000: + wand.size = (SCRWIDTH, SCRHEIGHT) if not self.opts.dont_sharpen: wand.sharpen(0.0, 1.0) diff --git a/src/calibre/ebooks/conversion/preprocess.py b/src/calibre/ebooks/conversion/preprocess.py index 5f6402f746..a1d5fa94d8 100644 --- a/src/calibre/ebooks/conversion/preprocess.py +++ b/src/calibre/ebooks/conversion/preprocess.py @@ -265,16 +265,28 @@ class CSSPreProcessor(object): PAGE_PAT = re.compile(r'@page[^{]*?{[^}]*?}') # Remove some of the broken CSS Microsoft products - # create, slightly dangerous as it removes to end of line - # rather than semi-colon - MS_PAT = re.compile(r'^\s*(mso-|panose-).+?$', - re.MULTILINE|re.IGNORECASE) + # create + MS_PAT = re.compile(r''' + (?P^|;|\{)\s* # The end of the previous rule or block start + (%s).+? # The invalid selectors + (?P$|;|\}) # The end of the declaration + '''%'mso-|panose-|text-underline|tab-interval', + re.MULTILINE|re.IGNORECASE|re.VERBOSE) + + def ms_sub(self, match): + end = match.group('end') + try: + start = match.group('start') + except: + start = '' + if end == ';': + end = '' + return start + end def __call__(self, data, add_namespace=False): from calibre.ebooks.oeb.base import XHTML_CSS_NAMESPACE data = self.PAGE_PAT.sub('', data) - if '\n' in data: - data = self.MS_PAT.sub('', data) + data = self.MS_PAT.sub(self.ms_sub, data) if not add_namespace: return data ans, namespaced = [], False diff --git a/src/calibre/ebooks/metadata/book/__init__.py b/src/calibre/ebooks/metadata/book/__init__.py index 033a78d611..fae858aabd 100644 --- a/src/calibre/ebooks/metadata/book/__init__.py +++ b/src/calibre/ebooks/metadata/book/__init__.py @@ -18,14 +18,14 @@ SOCIAL_METADATA_FIELDS = frozenset([ 'series_index', # A floating point number # Of the form { scheme1:value1, scheme2:value2} # For example: {'isbn':'123456789', 'doi':'xxxx', ... } - 'classifiers', + 'identifiers', ]) ''' -The list of names that convert to classifiers when in get and set. +The list of names that convert to identifiers when in get and set. ''' -TOP_LEVEL_CLASSIFIERS = frozenset([ +TOP_LEVEL_IDENTIFIERS = frozenset([ 'isbn', ]) @@ -108,7 +108,7 @@ STANDARD_METADATA_FIELDS = SOCIAL_METADATA_FIELDS.union( SC_FIELDS_NOT_COPIED = frozenset(['title', 'title_sort', 'authors', 'author_sort', 'author_sort_map', 'cover_data', 'tags', 'language', - 'classifiers']) + 'identifiers']) # Metadata fields that smart update should copy only if the source is not None SC_FIELDS_COPY_NOT_NULL = frozenset(['lpath', 'size', 'comments', 'thumbnail']) diff --git a/src/calibre/ebooks/metadata/book/base.py b/src/calibre/ebooks/metadata/book/base.py index b47cc373a7..feb6ff4bb9 100644 --- a/src/calibre/ebooks/metadata/book/base.py +++ b/src/calibre/ebooks/metadata/book/base.py @@ -12,7 +12,7 @@ from calibre.constants import DEBUG from calibre.ebooks.metadata.book import SC_COPYABLE_FIELDS from calibre.ebooks.metadata.book import SC_FIELDS_COPY_NOT_NULL from calibre.ebooks.metadata.book import STANDARD_METADATA_FIELDS -from calibre.ebooks.metadata.book import TOP_LEVEL_CLASSIFIERS +from calibre.ebooks.metadata.book import TOP_LEVEL_IDENTIFIERS from calibre.ebooks.metadata.book import ALL_METADATA_FIELDS from calibre.library.field_metadata import FieldMetadata from calibre.utils.date import isoformat, format_date @@ -24,7 +24,7 @@ NULL_VALUES = { 'user_metadata': {}, 'cover_data' : (None, None), 'tags' : [], - 'classifiers' : {}, + 'identifiers' : {}, 'languages' : [], 'device_collections': [], 'author_sort_map': {}, @@ -41,7 +41,7 @@ class SafeFormat(TemplateFormatter): def get_value(self, key, args, kwargs): try: key = key.lower() - if key != 'title_sort': + if key != 'title_sort' and key not in TOP_LEVEL_IDENTIFIERS: key = field_metadata.search_term_to_field_key(key) b = self.book.get_user_metadata(key, False) if b and b['datatype'] == 'int' and self.book.get(key, 0) == 0: @@ -49,7 +49,7 @@ class SafeFormat(TemplateFormatter): elif b and b['datatype'] == 'float' and self.book.get(key, 0.0) == 0.0: v = '' else: - ign, v = self.book.format_field(key, series_with_index=False) + v = self.book.format_field(key, series_with_index=False)[1] if v is None: return '' if v == '': @@ -96,8 +96,8 @@ class Metadata(object): def __getattribute__(self, field): _data = object.__getattribute__(self, '_data') - if field in TOP_LEVEL_CLASSIFIERS: - return _data.get('classifiers').get(field, None) + if field in TOP_LEVEL_IDENTIFIERS: + return _data.get('identifiers').get(field, None) if field in STANDARD_METADATA_FIELDS: return _data.get(field, None) try: @@ -123,11 +123,14 @@ class Metadata(object): def __setattr__(self, field, val, extra=None): _data = object.__getattribute__(self, '_data') - if field in TOP_LEVEL_CLASSIFIERS: - _data['classifiers'].update({field: val}) + if field in TOP_LEVEL_IDENTIFIERS: + field, val = self._clean_identifier(field, val) + _data['identifiers'].update({field: val}) + elif field == 'identifiers': + self.set_identifiers(val) elif field in STANDARD_METADATA_FIELDS: if val is None: - val = NULL_VALUES.get(field, None) + val = copy.copy(NULL_VALUES.get(field, None)) _data[field] = val elif field in _data['user_metadata'].iterkeys(): _data['user_metadata'][field]['#value#'] = val @@ -176,17 +179,48 @@ class Metadata(object): def set(self, field, val, extra=None): self.__setattr__(field, val, extra) - def get_classifiers(self): + def get_identifiers(self): ''' - Return a copy of the classifiers dictionary. + Return a copy of the identifiers dictionary. The dict is small, and the penalty for using a reference where a copy is needed is large. Also, we don't want any manipulations of the returned dict to show up in the book. ''' - return copy.deepcopy(object.__getattribute__(self, '_data')['classifiers']) + ans = object.__getattribute__(self, + '_data')['identifiers'] + if not ans: + ans = {} + return copy.deepcopy(ans) - def set_classifiers(self, classifiers): - object.__getattribute__(self, '_data')['classifiers'] = classifiers + def _clean_identifier(self, typ, val): + typ = icu_lower(typ).strip().replace(':', '').replace(',', '') + val = val.strip().replace(',', '|').replace(':', '|') + return typ, val + + def set_identifiers(self, identifiers): + ''' + Set all identifiers. Note that if you previously set ISBN, calling + this method will delete it. + ''' + cleaned = {} + for key, val in identifiers.iteritems(): + key, val = self._clean_identifier(key, val) + if key and val: + cleaned[key] = val + object.__getattribute__(self, '_data')['identifiers'] = cleaned + + def set_identifier(self, typ, val): + 'If val is empty, deletes identifier of type typ' + typ, val = self._clean_identifier(typ, val) + if not typ: + return + identifiers = object.__getattribute__(self, + '_data')['identifiers'] + + if not val and typ in identifiers: + identifiers.pop(typ) + if val: + identifiers[typ] = val # field-oriented interface. Intended to be the same as in LibraryDatabase @@ -229,7 +263,7 @@ class Metadata(object): if v is not None: result[attr] = v # separate these because it uses the self.get(), not _data.get() - for attr in TOP_LEVEL_CLASSIFIERS: + for attr in TOP_LEVEL_IDENTIFIERS: v = self.get(attr, None) if v is not None: result[attr] = v @@ -400,8 +434,8 @@ class Metadata(object): self.set_all_user_metadata(other.get_all_user_metadata(make_copy=True)) for x in SC_FIELDS_COPY_NOT_NULL: copy_not_none(self, other, x) - if callable(getattr(other, 'get_classifiers', None)): - self.set_classifiers(other.get_classifiers()) + if callable(getattr(other, 'get_identifiers', None)): + self.set_identifiers(other.get_identifiers()) # language is handled below else: for attr in SC_COPYABLE_FIELDS: @@ -456,15 +490,15 @@ class Metadata(object): if len(other_comments.strip()) > len(my_comments.strip()): self.comments = other_comments - # Copy all the non-none classifiers - if callable(getattr(other, 'get_classifiers', None)): - d = self.get_classifiers() - s = other.get_classifiers() + # Copy all the non-none identifiers + if callable(getattr(other, 'get_identifiers', None)): + d = self.get_identifiers() + s = other.get_identifiers() d.update([v for v in s.iteritems() if v[1] is not None]) - self.set_classifiers(d) + self.set_identifiers(d) else: - # other structure not Metadata. Copy the top-level classifiers - for attr in TOP_LEVEL_CLASSIFIERS: + # other structure not Metadata. Copy the top-level identifiers + for attr in TOP_LEVEL_IDENTIFIERS: copy_not_none(self, other, attr) other_lang = getattr(other, 'language', None) @@ -544,9 +578,15 @@ class Metadata(object): res = res/2 return (name, unicode(res), orig_res, cmeta) + # convert top-level ids into their value + if key in TOP_LEVEL_IDENTIFIERS: + fmeta = field_metadata['identifiers'] + name = key + res = self.get(key, None) + return (name, res, res, fmeta) + # Translate aliases into the standard field name fmkey = field_metadata.search_term_to_field_key(key) - if fmkey in field_metadata and field_metadata[fmkey]['kind'] == 'field': res = self.get(key, None) fmeta = field_metadata[fmkey] @@ -561,6 +601,8 @@ class Metadata(object): elif key == 'series_index': res = self.format_series_index(res) elif datatype == 'text' and fmeta['is_multiple']: + if isinstance(res, dict): + res = [k + ':' + v for k,v in res.items()] res = u', '.join(sorted(res, key=sort_key)) elif datatype == 'series' and series_with_index: res = res + ' [%s]'%self.format_series_index() diff --git a/src/calibre/ebooks/metadata/book/json_codec.py b/src/calibre/ebooks/metadata/book/json_codec.py index c02d4e953d..1d93b5dece 100644 --- a/src/calibre/ebooks/metadata/book/json_codec.py +++ b/src/calibre/ebooks/metadata/book/json_codec.py @@ -123,6 +123,8 @@ class JsonCodec(object): if key == 'user_metadata': book.set_all_user_metadata(meta) else: + if key == 'classifiers': + key = 'identifiers' setattr(book, key, meta) booklist.append(book) except: @@ -130,6 +132,8 @@ class JsonCodec(object): traceback.print_exc() def decode_metadata(self, key, value): + if key == 'classifiers': + key = 'identifiers' if key == 'user_metadata': for k in value: if value[k]['datatype'] == 'datetime': diff --git a/src/calibre/ebooks/metadata/opf2.py b/src/calibre/ebooks/metadata/opf2.py index d34a563110..9c59692628 100644 --- a/src/calibre/ebooks/metadata/opf2.py +++ b/src/calibre/ebooks/metadata/opf2.py @@ -596,6 +596,9 @@ class OPF(object): # {{{ ans = MetaInformation(self) for n, v in self._user_metadata_.items(): ans.set_user_metadata(n, v) + + ans.set_identifiers(self.get_identifiers()) + return ans def write_user_metadata(self): @@ -855,6 +858,21 @@ class OPF(object): # {{{ return property(fget=fget, fset=fset) + def get_identifiers(self): + identifiers = {} + for x in self.XPath( + 'descendant::*[local-name() = "identifier" and text()]')( + self.metadata): + for attr, val in x.attrib.iteritems(): + if attr.endswith('scheme'): + typ = icu_lower(val) + val = etree.tostring(x, with_tail=False, encoding=unicode, + method='text').strip() + if val and typ not in ('calibre', 'uuid'): + identifiers[typ] = val + break + return identifiers + @dynamic_property def application_id(self): @@ -1166,8 +1184,8 @@ class OPFCreator(Metadata): a(DC_ELEM('description', self.comments)) if self.publisher: a(DC_ELEM('publisher', self.publisher)) - if self.isbn: - a(DC_ELEM('identifier', self.isbn, opf_attrs={'scheme':'ISBN'})) + for key, val in self.get_identifiers().iteritems(): + a(DC_ELEM('identifier', val, opf_attrs={'scheme':icu_upper(key)})) if self.rights: a(DC_ELEM('rights', self.rights)) if self.tags: @@ -1291,8 +1309,8 @@ def metadata_to_opf(mi, as_string=True): factory(DC('description'), mi.comments) if mi.publisher: factory(DC('publisher'), mi.publisher) - if mi.isbn: - factory(DC('identifier'), mi.isbn, scheme='ISBN') + for key, val in mi.get_identifiers().iteritems(): + factory(DC('identifier'), val, scheme=icu_upper(key)) if mi.rights: factory(DC('rights'), mi.rights) factory(DC('language'), mi.language if mi.language and mi.language.lower() @@ -1342,7 +1360,7 @@ def test_m2o(): mi.language = 'en' mi.comments = 'what a fun book\n\n' mi.publisher = 'publisher' - mi.isbn = 'boooo' + mi.set_identifiers({'isbn':'booo', 'dummy':'dummy'}) mi.tags = ['a', 'b'] mi.series = 's"c\'l&<>' mi.series_index = 3.34 @@ -1350,7 +1368,7 @@ def test_m2o(): mi.timestamp = nowf() mi.publication_type = 'ooooo' mi.rights = 'yes' - mi.cover = 'asd.jpg' + mi.cover = os.path.abspath('asd.jpg') opf = metadata_to_opf(mi) print opf newmi = MetaInformation(OPF(StringIO(opf))) @@ -1363,6 +1381,9 @@ def test_m2o(): o, n = getattr(mi, attr), getattr(newmi, attr) if o != n and o.strip() != n.strip(): print 'FAILED:', attr, getattr(mi, attr), '!=', getattr(newmi, attr) + if mi.get_identifiers() != newmi.get_identifiers(): + print 'FAILED:', 'identifiers', mi.get_identifiers(), + print '!=', newmi.get_identifiers() class OPFTest(unittest.TestCase): @@ -1378,6 +1399,7 @@ class OPFTest(unittest.TestCase): Next OneTwo 123456789 + dummy @@ -1405,6 +1427,8 @@ class OPFTest(unittest.TestCase): self.assertEqual(opf.rating, 4) self.assertEqual(opf.publication_type, 'test') self.assertEqual(list(opf.itermanifest())[0].get('href'), 'a ~ b') + self.assertEqual(opf.get_identifiers(), {'isbn':'123456789', + 'dummy':'dummy'}) def testWriting(self): for test in [('title', 'New & Title'), ('authors', ['One', 'Two']), @@ -1461,5 +1485,5 @@ def test_user_metadata(): if __name__ == '__main__': #test_user_metadata() - #test_m2o() + test_m2o() test() diff --git a/src/calibre/ebooks/metadata/sources/base.py b/src/calibre/ebooks/metadata/sources/base.py index 54d7d49d6d..142224c599 100644 --- a/src/calibre/ebooks/metadata/sources/base.py +++ b/src/calibre/ebooks/metadata/sources/base.py @@ -65,7 +65,8 @@ class Source(Plugin): parts = parts[1:] + parts[:1] for tok in parts: tok = pat.sub('', tok).strip() - yield tok + if len(tok) > 2 and tok.lower() not in ('von', ): + yield tok def get_title_tokens(self, title): diff --git a/src/calibre/ebooks/mobi/reader.py b/src/calibre/ebooks/mobi/reader.py index 9c52a18691..f1b1b1ef63 100644 --- a/src/calibre/ebooks/mobi/reader.py +++ b/src/calibre/ebooks/mobi/reader.py @@ -18,6 +18,7 @@ from calibre import xml_entity_to_unicode, CurrentDir, entity_to_unicode, \ replace_entities from calibre.utils.filenames import ascii_filename from calibre.utils.date import parse_date +from calibre.utils.cleantext import clean_ascii_chars from calibre.ptempfile import TemporaryDirectory from calibre.ebooks import DRMError from calibre.ebooks.chardet import ENCODING_PATS @@ -323,6 +324,7 @@ class MobiReader(object): self.cleanup_html() self.log.debug('Parsing HTML...') + self.processed_html = clean_ascii_chars(self.processed_html) try: root = html.fromstring(self.processed_html) if len(root.xpath('//html')) > 5: diff --git a/src/calibre/ebooks/oeb/base.py b/src/calibre/ebooks/oeb/base.py index ccc452f1f8..7e99916fc3 100644 --- a/src/calibre/ebooks/oeb/base.py +++ b/src/calibre/ebooks/oeb/base.py @@ -827,6 +827,24 @@ class Manifest(object): return None return etree.fromstring(data, parser=RECOVER_PARSER) + def clean_word_doc(self, data): + prefixes = [] + for match in re.finditer(r'xmlns:(\S+?)=".*?microsoft.*?"', data): + prefixes.append(match.group(1)) + if prefixes: + self.oeb.log.warn('Found microsoft markup, cleaning...') + # Remove empty tags as they are not rendered by browsers + # but can become renderable HTML tags like

    if the + # document is parsed by an HTML parser + pat = re.compile( + r'<(%s):([a-zA-Z0-9]+)[^>/]*?>'%('|'.join(prefixes)), + re.DOTALL) + data = pat.sub('', data) + pat = re.compile( + r'<(%s):([a-zA-Z0-9]+)[^>/]*?/>'%('|'.join(prefixes))) + data = pat.sub('', data) + return data + def _parse_xhtml(self, data): self.oeb.log.debug('Parsing', self.href, '...') # Convert to Unicode and normalize line endings @@ -884,6 +902,10 @@ class Manifest(object): except etree.XMLSyntaxError: data = etree.fromstring(data, parser=RECOVER_PARSER) return data + try: + data = self.clean_word_doc(data) + except: + pass data = first_pass(data) # Handle weird (non-HTML/fragment) files @@ -907,6 +929,7 @@ class Manifest(object): parent.append(child) data = nroot + # Force into the XHTML namespace if not namespace(data.tag): self.oeb.log.warn('Forcing', self.href, 'into XHTML namespace') diff --git a/src/calibre/ebooks/oeb/output.py b/src/calibre/ebooks/oeb/output.py index 6709141a01..fac6675904 100644 --- a/src/calibre/ebooks/oeb/output.py +++ b/src/calibre/ebooks/oeb/output.py @@ -59,20 +59,34 @@ class OEBOutput(OutputFormatPlugin): def workaround_nook_cover_bug(self, root): # {{{ cov = root.xpath('//*[local-name() = "meta" and @name="cover" and' ' @content != "cover"]') + + def manifest_items_with_id(id_): + return root.xpath('//*[local-name() = "manifest"]/*[local-name() = "item" ' + ' and @id="%s"]'%id_) + if len(cov) == 1: - manpath = ('//*[local-name() = "manifest"]/*[local-name() = "item" ' - ' and @id="%s" and @media-type]') cov = cov[0] - covid = cov.get('content') - manifest_item = root.xpath(manpath%covid) - has_cover = root.xpath(manpath%'cover') - if len(manifest_item) == 1 and not has_cover and \ - manifest_item[0].get('media-type', - '').startswith('image/'): - self.log.warn('The cover image has an id != "cover". Renaming' - ' to work around Nook Color bug') - manifest_item = manifest_item[0] - manifest_item.set('id', 'cover') - cov.set('content', 'cover') + covid = cov.get('content', '') + + if covid: + manifest_item = manifest_items_with_id(covid) + if len(manifest_item) == 1 and \ + manifest_item[0].get('media-type', + '').startswith('image/'): + self.log.warn('The cover image has an id != "cover". Renaming' + ' to work around bug in Nook Color') + + import uuid + newid = str(uuid.uuid4()) + + for item in manifest_items_with_id('cover'): + item.set('id', newid) + + for x in root.xpath('//*[@idref="cover"]'): + x.set('idref', newid) + + manifest_item = manifest_item[0] + manifest_item.set('id', 'cover') + cov.set('content', 'cover') # }}} diff --git a/src/calibre/ebooks/oeb/stylizer.py b/src/calibre/ebooks/oeb/stylizer.py index 849d161228..0cd17387fe 100644 --- a/src/calibre/ebooks/oeb/stylizer.py +++ b/src/calibre/ebooks/oeb/stylizer.py @@ -8,11 +8,7 @@ from __future__ import with_statement __license__ = 'GPL v3' __copyright__ = '2008, Marshall T. Vandegrift ' -import os -import itertools -import re -import logging -import copy +import os, itertools, re, logging, copy, unicodedata from weakref import WeakKeyDictionary from xml.dom import SyntaxErr as CSSSyntaxError import cssutils @@ -234,8 +230,18 @@ class Stylizer(object): for elem in matches: for x in elem.iter(): if x.text: - span = E.span(x.text[0]) - span.tail = x.text[1:] + punctuation_chars = [] + text = unicode(x.text) + while text: + if not unicodedata.category(text[0]).startswith('P'): + break + punctuation_chars.append(text[0]) + text = text[1:] + + special_text = u''.join(punctuation_chars) + \ + (text[0] if text else u'') + span = E.span(special_text) + span.tail = text[1:] x.text = None x.insert(0, span) self.style(span)._update_cssdict(cssdict) @@ -423,6 +429,7 @@ class Stylizer(object): class Style(object): UNIT_RE = re.compile(r'^(-*[0-9]*[.]?[0-9]*)\s*(%|em|ex|en|px|mm|cm|in|pt|pc)$') + MS_PAT = re.compile(r'^\s*(mso-|panose-|text-underline|tab-interval)') def __init__(self, element, stylizer): self._element = element @@ -447,6 +454,8 @@ class Style(object): return css = attrib['style'].split(';') css = filter(None, (x.strip() for x in css)) + css = [x.strip() for x in css] + css = [x for x in css if self.MS_PAT.match(x) is None] try: style = CSSStyleDeclaration('; '.join(css)) except CSSSyntaxError: diff --git a/src/calibre/ebooks/oeb/transforms/structure.py b/src/calibre/ebooks/oeb/transforms/structure.py index 0db9b153df..fc338da692 100644 --- a/src/calibre/ebooks/oeb/transforms/structure.py +++ b/src/calibre/ebooks/oeb/transforms/structure.py @@ -13,6 +13,7 @@ from urlparse import urlparse from calibre.ebooks.oeb.base import XPNSMAP, TOC, XHTML, xml2text from calibre.ebooks import ConversionError +from calibre.utils.ordered_dict import OrderedDict def XPath(x): try: @@ -95,10 +96,8 @@ class DetectStructure(object): self.log.exception('Failed to mark chapter') def create_level_based_toc(self): - if self.opts.level1_toc is None: - return - for item in self.oeb.spine: - self.add_leveled_toc_items(item) + if self.opts.level1_toc is not None: + self.add_leveled_toc_items() def create_toc_from_chapters(self): counter = self.oeb.toc.next_play_order() @@ -145,49 +144,57 @@ class DetectStructure(object): return text, href - def add_leveled_toc_items(self, item): - level1 = XPath(self.opts.level1_toc)(item.data) - level1_order = [] - document = item - + def add_leveled_toc_items(self): + added = OrderedDict() + added2 = OrderedDict() counter = 1 - if level1: - added = {} - for elem in level1: + for document in self.oeb.spine: + previous_level1 = list(added.itervalues())[-1] if added else None + previous_level2 = list(added2.itervalues())[-1] if added2 else None + + for elem in XPath(self.opts.level1_toc)(document.data): text, _href = self.elem_to_link(document, elem, counter) counter += 1 if text: node = self.oeb.toc.add(text, _href, play_order=self.oeb.toc.next_play_order()) - level1_order.append(node) added[elem] = node #node.add(_('Top'), _href) - if self.opts.level2_toc is not None: - added2 = {} - level2 = list(XPath(self.opts.level2_toc)(document.data)) - for elem in level2: + + if self.opts.level2_toc is not None and added: + for elem in XPath(self.opts.level2_toc)(document.data): level1 = None for item in document.data.iterdescendants(): - if item in added.keys(): + if item in added: level1 = added[item] - elif item == elem and level1 is not None: + elif item == elem: + if level1 is None: + if previous_level1 is None: + break + level1 = previous_level1 text, _href = self.elem_to_link(document, elem, counter) counter += 1 if text: added2[elem] = level1.add(text, _href, play_order=self.oeb.toc.next_play_order()) - if self.opts.level3_toc is not None: - level3 = list(XPath(self.opts.level3_toc)(document.data)) - for elem in level3: + break + + if self.opts.level3_toc is not None and added2: + for elem in XPath(self.opts.level3_toc)(document.data): level2 = None for item in document.data.iterdescendants(): - if item in added2.keys(): + if item in added2: level2 = added2[item] - elif item == elem and level2 is not None: + elif item == elem: + if level2 is None: + if previous_level2 is None: + break + level2 = previous_level2 text, _href = \ self.elem_to_link(document, elem, counter) counter += 1 if text: level2.add(text, _href, - play_order=self.oeb.toc.next_play_order()) + play_order=self.oeb.toc.next_play_order()) + break diff --git a/src/calibre/ebooks/pdf/reflow.cpp b/src/calibre/ebooks/pdf/reflow.cpp index 0c569fe0d1..e444c126ab 100644 --- a/src/calibre/ebooks/pdf/reflow.cpp +++ b/src/calibre/ebooks/pdf/reflow.cpp @@ -887,7 +887,7 @@ vector* Reflow::render_first_page(bool use_crop_box, double x_res, } pg_w *= x_res/72.; - pg_h *= x_res/72.; + pg_h *= y_res/72.; int x=0, y=0; this->doc->displayPageSlice(out, pg, x_res, y_res, 0, diff --git a/src/calibre/ebooks/pdf/writer.py b/src/calibre/ebooks/pdf/writer.py index b0884417f6..516509fdd7 100644 --- a/src/calibre/ebooks/pdf/writer.py +++ b/src/calibre/ebooks/pdf/writer.py @@ -46,7 +46,8 @@ def get_pdf_printer(opts, for_comic=False): printer = QPrinter(QPrinter.HighResolution) custom_size = get_custom_size(opts) - if opts.output_profile.short_name == 'default': + if opts.output_profile.short_name == 'default' or \ + opts.output_profile.width > 10000: if custom_size is None: printer.setPaperSize(paper_size(opts.paper_size)) else: diff --git a/src/calibre/ebooks/snb/snbfile.py b/src/calibre/ebooks/snb/snbfile.py index e42533f241..9a7d65e417 100644 --- a/src/calibre/ebooks/snb/snbfile.py +++ b/src/calibre/ebooks/snb/snbfile.py @@ -75,15 +75,20 @@ class SNBFile: for i in range(self.plainBlock): bzdc = bz2.BZ2Decompressor() if (i < self.plainBlock - 1): - bSize = self.blocks[self.binBlock + i + 1].Offset - self.blocks[self.binBlock + i].Offset; + bSize = self.blocks[self.binBlock + i + 1].Offset - self.blocks[self.binBlock + i].Offset else: - bSize = self.tailOffset - self.blocks[self.binBlock + i].Offset; - snbFile.seek(self.blocks[self.binBlock + i].Offset); + bSize = self.tailOffset - self.blocks[self.binBlock + i].Offset + snbFile.seek(self.blocks[self.binBlock + i].Offset) try: data = snbFile.read(bSize) - uncompressedData += bzdc.decompress(data) + if len(data) < 32768: + uncompressedData += bzdc.decompress(data) + else: + uncompressedData += data except Exception, e: print e + if len(uncompressedData) != self.plainStreamSizeUncompressed: + raise Exception() f.fileBody = uncompressedData[plainPos:plainPos+f.fileSize] plainPos += f.fileSize elif f.attr & 0x01000000 == 0x01000000: diff --git a/src/calibre/ebooks/txt/input.py b/src/calibre/ebooks/txt/input.py index 1c49eb9b35..3c256fda7a 100644 --- a/src/calibre/ebooks/txt/input.py +++ b/src/calibre/ebooks/txt/input.py @@ -22,7 +22,7 @@ class TXTInput(InputFormatPlugin): name = 'TXT Input' author = 'John Schember' description = 'Convert TXT files to HTML' - file_types = set(['txt', 'txtz']) + file_types = set(['txt', 'txtz', 'text']) options = set([ OptionRecommendation(name='paragraph_type', recommended_value='auto', diff --git a/src/calibre/gui2/actions/add.py b/src/calibre/gui2/actions/add.py index f99e48eb2b..cf67cd6cfa 100644 --- a/src/calibre/gui2/actions/add.py +++ b/src/calibre/gui2/actions/add.py @@ -20,9 +20,26 @@ from calibre.ebooks import BOOK_EXTENSIONS from calibre.utils.filenames import ascii_filename from calibre.constants import preferred_encoding, filesystem_encoding from calibre.gui2.actions import InterfaceAction -from calibre.gui2 import config +from calibre.gui2 import config, question_dialog from calibre.ebooks.metadata import MetaInformation +def get_filters(): + return [ + (_('Books'), BOOK_EXTENSIONS), + (_('EPUB Books'), ['epub']), + (_('LRF Books'), ['lrf']), + (_('HTML Books'), ['htm', 'html', 'xhtm', 'xhtml']), + (_('LIT Books'), ['lit']), + (_('MOBI Books'), ['mobi', 'prc', 'azw']), + (_('Topaz books'), ['tpz','azw1']), + (_('Text books'), ['txt', 'rtf']), + (_('PDF Books'), ['pdf']), + (_('SNB Books'), ['snb']), + (_('Comics'), ['cbz', 'cbr', 'cbc']), + (_('Archives'), ['zip', 'rar']), + ] + + class AddAction(InterfaceAction): name = 'Add Books' @@ -47,6 +64,10 @@ class AddAction(InterfaceAction): self.add_menu.addAction(_('Add Empty book. (Book entry with no ' 'formats)'), self.add_empty, _('Shift+Ctrl+E')) self.add_menu.addAction(_('Add from ISBN'), self.add_from_isbn) + self.add_menu.addSeparator() + self.add_menu.addAction(_('Add files to selected book records'), + self.add_formats, _('Shift+A')) + self.qaction.setMenu(self.add_menu) self.qaction.triggered.connect(self.add_books) @@ -55,6 +76,39 @@ class AddAction(InterfaceAction): for action in list(self.add_menu.actions())[1:]: action.setEnabled(enabled) + def add_formats(self, *args): + if self.gui.stack.currentIndex() != 0: + return + view = self.gui.library_view + rows = view.selectionModel().selectedRows() + if not rows: + return + ids = [view.model().id(r) for r in rows] + + if len(ids) > 1 and not question_dialog(self.gui, + _('Are you sure'), + _('Are you sure you want to add the same' + ' files to all %d books? If the format' + 'already exists for a book, it will be replaced.')%len(ids)): + return + + books = choose_files(self.gui, 'add formats dialog dir', + _('Select book files'), filters=get_filters()) + if not books: + return + + db = view.model().db + for id_ in ids: + for fpath in books: + fmt = os.path.splitext(fpath)[1][1:].upper() + if fmt: + db.add_format_with_hooks(id_, fmt, fpath, index_is_id=True, + notify=True) + current_idx = self.gui.library_view.currentIndex() + if current_idx.isValid(): + view.model().current_changed(current_idx, current_idx) + + def add_recursive(self, single): root = choose_dir(self.gui, 'recursive book import root dir dialog', 'Select root folder') @@ -150,15 +204,29 @@ class AddAction(InterfaceAction): to_device = self.gui.stack.currentIndex() != 0 self._add_books(paths, to_device) - def files_dropped_on_book(self, event, paths): + def remote_file_dropped_on_book(self, url, fname): + if self.gui.current_view() is not self.gui.library_view: + return + db = self.gui.library_view.model().db + current_idx = self.gui.library_view.currentIndex() + if not current_idx.isValid(): return + cid = db.id(current_idx.row()) + from calibre.gui2.dnd import DownloadDialog + d = DownloadDialog(url, fname, self.gui) + d.start_download() + if d.err is None: + self.files_dropped_on_book(None, [d.fpath], cid=cid) + + def files_dropped_on_book(self, event, paths, cid=None): accept = False if self.gui.current_view() is not self.gui.library_view: return db = self.gui.library_view.model().db cover_changed = False current_idx = self.gui.library_view.currentIndex() - if not current_idx.isValid(): return - cid = db.id(current_idx.row()) + if cid is None: + if not current_idx.isValid(): return + cid = db.id(current_idx.row()) if cid is None else cid for path in paths: ext = os.path.splitext(path)[1].lower() if ext: @@ -173,8 +241,9 @@ class AddAction(InterfaceAction): elif ext in BOOK_EXTENSIONS: db.add_format_with_hooks(cid, ext, path, index_is_id=True) accept = True - if accept: + if accept and event is not None: event.accept() + if current_idx.isValid(): self.gui.library_view.model().current_changed(current_idx, current_idx) if cover_changed: if self.gui.cover_flow: @@ -207,27 +276,14 @@ class AddAction(InterfaceAction): ''' Add books from the local filesystem to either the library or the device. ''' - filters = [ - (_('Books'), BOOK_EXTENSIONS), - (_('EPUB Books'), ['epub']), - (_('LRF Books'), ['lrf']), - (_('HTML Books'), ['htm', 'html', 'xhtm', 'xhtml']), - (_('LIT Books'), ['lit']), - (_('MOBI Books'), ['mobi', 'prc', 'azw']), - (_('Topaz books'), ['tpz','azw1']), - (_('Text books'), ['txt', 'rtf']), - (_('PDF Books'), ['pdf']), - (_('SNB Books'), ['snb']), - (_('Comics'), ['cbz', 'cbr', 'cbc']), - (_('Archives'), ['zip', 'rar']), - ] + filters = get_filters() to_device = self.gui.stack.currentIndex() != 0 if to_device: fmts = self.gui.device_manager.device.settings().format_map filters = [(_('Supported books'), fmts)] - books = choose_files(self.gui, 'add books dialog dir', 'Select books', - filters=filters) + books = choose_files(self.gui, 'add books dialog dir', + _('Select books'), filters=filters) if not books: return self._add_books(books, to_device) diff --git a/src/calibre/gui2/actions/choose_library.py b/src/calibre/gui2/actions/choose_library.py index f3a7f1742d..6f4ca624cb 100644 --- a/src/calibre/gui2/actions/choose_library.py +++ b/src/calibre/gui2/actions/choose_library.py @@ -355,6 +355,7 @@ class ChooseLibraryAction(InterfaceAction): print print 'before:', self.before_mem print 'after:', memory()/1024**2 + print self.dbref = self.before_mem = None diff --git a/src/calibre/gui2/actions/delete.py b/src/calibre/gui2/actions/delete.py index 9bc43f580b..a1fddd84b8 100644 --- a/src/calibre/gui2/actions/delete.py +++ b/src/calibre/gui2/actions/delete.py @@ -19,11 +19,11 @@ single_shot = partial(QTimer.singleShot, 10) class MultiDeleter(QObject): - def __init__(self, gui, rows, callback): + def __init__(self, gui, ids, callback): from calibre.gui2.dialogs.progress import ProgressDialog QObject.__init__(self, gui) self.model = gui.library_view.model() - self.ids = list(map(self.model.id, rows)) + self.ids = ids self.gui = gui self.failures = [] self.deleted_ids = [] @@ -231,6 +231,7 @@ class DeleteAction(InterfaceAction): return # Library view is visible. if self.gui.stack.currentIndex() == 0: + to_delete_ids = [view.model().id(r) for r in rows] # Ask the user if they want to delete the book from the library or device if it is in both. if self.gui.device_manager.is_device_connected: on_device = False @@ -264,10 +265,10 @@ class DeleteAction(InterfaceAction): if ci.isValid(): row = ci.row() if len(rows) < 5: - ids_deleted = view.model().delete_books(rows) - self.library_ids_deleted(ids_deleted, row) + view.model().delete_books_by_id(to_delete_ids) + self.library_ids_deleted(to_delete_ids, row) else: - self.__md = MultiDeleter(self.gui, rows, + self.__md = MultiDeleter(self.gui, to_delete_ids, partial(self.library_ids_deleted, current_row=row)) # Device view is visible. else: diff --git a/src/calibre/gui2/actions/fetch_news.py b/src/calibre/gui2/actions/fetch_news.py index fe51012e31..f7756efbab 100644 --- a/src/calibre/gui2/actions/fetch_news.py +++ b/src/calibre/gui2/actions/fetch_news.py @@ -67,7 +67,8 @@ class FetchNewsAction(InterfaceAction): keep_issues = 0 if keep_issues > 0: ids_with_tag = list(sorted(self.gui.library_view.model(). - db.tags_older_than(arg['title'], None), reverse=True)) + db.tags_older_than(arg['title'], + None, must_have_tag=_('News')), reverse=True)) ids_to_delete = ids_with_tag[keep_issues:] if ids_to_delete: self.gui.library_view.model().delete_books_by_id(ids_to_delete) diff --git a/src/calibre/gui2/book_details.py b/src/calibre/gui2/book_details.py index 2f7892692c..a28759486e 100644 --- a/src/calibre/gui2/book_details.py +++ b/src/calibre/gui2/book_details.py @@ -5,20 +5,21 @@ __license__ = 'GPL v3' __copyright__ = '2010, Kovid Goyal ' __docformat__ = 'restructuredtext en' -import os, collections, sys +import collections, sys from Queue import Queue from PyQt4.Qt import QPixmap, QSize, QWidget, Qt, pyqtSignal, QUrl, \ QPropertyAnimation, QEasingCurve, QThread, QApplication, QFontInfo, \ - QSizePolicy, QPainter, QRect, pyqtProperty, QLayout, QPalette + QSizePolicy, QPainter, QRect, pyqtProperty, QLayout, QPalette, QMenu from PyQt4.QtWebKit import QWebView from calibre import fit_image, prepare_string_for_xml -from calibre.gui2.widgets import IMAGE_EXTENSIONS +from calibre.gui2.dnd import dnd_has_image, dnd_get_image, dnd_get_files, \ + IMAGE_EXTENSIONS, dnd_has_extension from calibre.ebooks import BOOK_EXTENSIONS from calibre.constants import preferred_encoding from calibre.library.comments import comments_to_html -from calibre.gui2 import config, open_local_file, open_url +from calibre.gui2 import config, open_local_file, open_url, pixmap_to_data from calibre.utils.icu import sort_key # render_rows(data) {{{ @@ -70,6 +71,7 @@ def render_rows(data): class CoverView(QWidget): # {{{ + cover_changed = pyqtSignal(object, object) def __init__(self, vertical, parent=None): QWidget.__init__(self, parent) @@ -151,6 +153,36 @@ class CoverView(QWidget): # {{{ fset=setCurrentPixmapSize ) + def contextMenuEvent(self, ev): + cm = QMenu(self) + paste = cm.addAction(_('Paste Cover')) + copy = cm.addAction(_('Copy Cover')) + if not QApplication.instance().clipboard().mimeData().hasImage(): + paste.setEnabled(False) + copy.triggered.connect(self.copy_to_clipboard) + paste.triggered.connect(self.paste_from_clipboard) + cm.exec_(ev.globalPos()) + + def copy_to_clipboard(self): + QApplication.instance().clipboard().setPixmap(self.pixmap) + + def paste_from_clipboard(self, pmap=None): + if not isinstance(pmap, QPixmap): + cb = QApplication.instance().clipboard() + pmap = cb.pixmap() + if pmap.isNull() and cb.supportsSelection(): + pmap = cb.pixmap(cb.Selection) + if not pmap.isNull(): + self.pixmap = pmap + self.do_layout() + self.update() + if not config['disable_animations']: + self.animation.start() + id_ = self.data.get('id', None) + if id_ is not None: + self.cover_changed.emit(id_, + pixmap_to_data(pmap)) + # }}} @@ -196,6 +228,7 @@ class BookInfo(QWebView): self._link_clicked = False self.setAttribute(Qt.WA_OpaquePaintEvent, False) palette = self.palette() + self.setAcceptDrops(False) palette.setBrush(QPalette.Base, Qt.transparent) self.page().setPalette(palette) @@ -358,34 +391,50 @@ class BookDetails(QWidget): # {{{ show_book_info = pyqtSignal() open_containing_folder = pyqtSignal(int) view_specific_format = pyqtSignal(int, object) + remote_file_dropped = pyqtSignal(object, object) + files_dropped = pyqtSignal(object, object) + cover_changed = pyqtSignal(object, object) # Drag 'n drop {{{ DROPABBLE_EXTENSIONS = IMAGE_EXTENSIONS+BOOK_EXTENSIONS - files_dropped = pyqtSignal(object, object) - - @classmethod - def paths_from_event(cls, event): - ''' - Accept a drop event and return a list of paths that can be read from - and represent files with extensions. - ''' - if event.mimeData().hasFormat('text/uri-list'): - urls = [unicode(u.toLocalFile()) for u in event.mimeData().urls()] - urls = [u for u in urls if os.path.splitext(u)[1] and os.access(u, os.R_OK)] - return [u for u in urls if os.path.splitext(u)[1][1:].lower() in cls.DROPABBLE_EXTENSIONS] def dragEnterEvent(self, event): - if int(event.possibleActions() & Qt.CopyAction) + \ - int(event.possibleActions() & Qt.MoveAction) == 0: - return - paths = self.paths_from_event(event) - if paths: + md = event.mimeData() + if dnd_has_extension(md, self.DROPABBLE_EXTENSIONS) or \ + dnd_has_image(md): event.acceptProposedAction() def dropEvent(self, event): - paths = self.paths_from_event(event) event.setDropAction(Qt.CopyAction) - self.files_dropped.emit(event, paths) + md = event.mimeData() + + x, y = dnd_get_image(md) + if x is not None: + # We have an image, set cover + event.accept() + if y is None: + # Local image + self.cover_view.paste_from_clipboard(x) + else: + self.remote_file_dropped.emit(x, y) + # We do not support setting cover *and* adding formats for + # a remote drop, anyway, so return + return + + # Now look for ebook files + urls, filenames = dnd_get_files(md, BOOK_EXTENSIONS) + if not urls: + # Nothing found + return + + if not filenames: + # Local files + self.files_dropped.emit(event, urls) + else: + # Remote files, use the first file + self.remote_file_dropped.emit(urls[0], filenames[0]) + event.accept() + def dragMoveEvent(self, event): event.acceptProposedAction() @@ -399,6 +448,7 @@ class BookDetails(QWidget): # {{{ self.setLayout(self._layout) self.cover_view = CoverView(vertical, self) + self.cover_view.cover_changed.connect(self.cover_changed.emit) self._layout.addWidget(self.cover_view) self.book_info = BookInfo(vertical, self) self._layout.addWidget(self.book_info) diff --git a/src/calibre/gui2/convert/xexp_edit.ui b/src/calibre/gui2/convert/xexp_edit.ui index 18b7c39b52..68c0c8c98e 100644 --- a/src/calibre/gui2/convert/xexp_edit.ui +++ b/src/calibre/gui2/convert/xexp_edit.ui @@ -43,6 +43,9 @@ 0 + + QComboBox::AdjustToMinimumContentsLengthWithIcon + 30 diff --git a/src/calibre/gui2/custom_column_widgets.py b/src/calibre/gui2/custom_column_widgets.py index fa7ba3c56d..beaca77a38 100644 --- a/src/calibre/gui2/custom_column_widgets.py +++ b/src/calibre/gui2/custom_column_widgets.py @@ -5,7 +5,6 @@ __license__ = 'GPL v3' __copyright__ = '2010, Kovid Goyal ' __docformat__ = 'restructuredtext en' -import sys from functools import partial from PyQt4.Qt import QComboBox, QLabel, QSpinBox, QDoubleSpinBox, QDateEdit, \ @@ -85,7 +84,7 @@ class Int(Base): self.widgets = [QLabel('&'+self.col_metadata['name']+':', parent), QSpinBox(parent)] w = self.widgets[1] - w.setRange(-100, sys.maxint) + w.setRange(-100, 100000000) w.setSpecialValueText(_('Undefined')) w.setSingleStep(1) @@ -108,7 +107,7 @@ class Float(Int): self.widgets = [QLabel('&'+self.col_metadata['name']+':', parent), QDoubleSpinBox(parent)] w = self.widgets[1] - w.setRange(-100., float(sys.maxint)) + w.setRange(-100., float(100000000)) w.setDecimals(2) w.setSpecialValueText(_('Undefined')) w.setSingleStep(1) @@ -289,7 +288,7 @@ class Series(Base): self.widgets.append(QLabel('&'+self.col_metadata['name']+_(' index:'), parent)) w = QDoubleSpinBox(parent) - w.setRange(-100., float(sys.maxint)) + w.setRange(-100., float(100000000)) w.setDecimals(2) w.setSpecialValueText(_('Undefined')) w.setSingleStep(1) @@ -595,7 +594,7 @@ class BulkInt(BulkBase): def setup_ui(self, parent): self.make_widgets(parent, QSpinBox) - self.main_widget.setRange(-100, sys.maxint) + self.main_widget.setRange(-100, 100000000) self.main_widget.setSpecialValueText(_('Undefined')) self.main_widget.setSingleStep(1) @@ -617,7 +616,7 @@ class BulkFloat(BulkInt): def setup_ui(self, parent): self.make_widgets(parent, QDoubleSpinBox) - self.main_widget.setRange(-100., float(sys.maxint)) + self.main_widget.setRange(-100., float(100000000)) self.main_widget.setDecimals(2) self.main_widget.setSpecialValueText(_('Undefined')) self.main_widget.setSingleStep(1) @@ -795,6 +794,7 @@ class BulkEnumeration(BulkBase, Enumeration): return value def setup_ui(self, parent): + self.parent = parent self.make_widgets(parent, QComboBox) vals = self.col_metadata['display']['enum_values'] self.main_widget.blockSignals(True) diff --git a/src/calibre/gui2/device.py b/src/calibre/gui2/device.py index e4096f5761..2cbecc134c 100644 --- a/src/calibre/gui2/device.py +++ b/src/calibre/gui2/device.py @@ -140,6 +140,8 @@ class DeviceManager(Thread): # {{{ self.mount_connection_requests = Queue.Queue(0) self.open_feedback_slot = open_feedback_slot self.open_feedback_msg = open_feedback_msg + self._device_information = None + self.current_library_uuid = None def report_progress(self, *args): pass @@ -159,7 +161,7 @@ class DeviceManager(Thread): # {{{ try: dev.reset(detected_device=detected_device, report_progress=self.report_progress) - dev.open() + dev.open(self.current_library_uuid) except OpenFeedback, e: if dev not in self.ejected_devices: self.open_feedback_msg(dev.get_gui_name(), e.feedback_msg) @@ -194,6 +196,7 @@ class DeviceManager(Thread): # {{{ else: self.connected_slot(False, self.connected_device_kind) self.connected_device = None + self._device_information = None def detect_device(self): self.scanner.scan() @@ -292,9 +295,13 @@ class DeviceManager(Thread): # {{{ def _get_device_information(self): info = self.device.get_device_information(end_session=False) - info = [i.replace('\x00', '').replace('\x01', '') for i in info] + if len(info) < 5: + info = tuple(list(info) + [{}]) + info = [i.replace('\x00', '').replace('\x01', '') if isinstance(i, basestring) else i + for i in info] cp = self.device.card_prefix(end_session=False) fs = self.device.free_space() + self._device_information = {'info': info, 'prefixes': cp, 'freespace': fs} return info, cp, fs def get_device_information(self, done): @@ -302,6 +309,9 @@ class DeviceManager(Thread): # {{{ return self.create_job(self._get_device_information, done, description=_('Get device information')) + def get_current_device_information(self): + return self._device_information + def _books(self): '''Get metadata from device''' mainlist = self.device.books(oncard=None, end_session=False) @@ -417,6 +427,13 @@ class DeviceManager(Thread): # {{{ return self.create_job(self._view_book, done, args=[path, target], description=_('View book on device')) + def set_current_library_uuid(self, uuid): + self.current_library_uuid = uuid + + def set_driveinfo_name(self, location_code, name): + if self.connected_device: + self.connected_device.set_driveinfo_name(location_code, name) + # }}} class DeviceAction(QAction): # {{{ @@ -1143,6 +1160,14 @@ class DeviceMixin(object): # {{{ ), bad) d.exec_() + def upload_dirtied_booklists(self): + ''' + Upload metadata to device. + ''' + plugboards = self.library_view.model().db.prefs.get('plugboards', {}) + self.device_manager.sync_booklists(Dispatcher(lambda x: x), + self.booklists(), plugboards) + def upload_booklists(self): ''' Upload metadata to device. diff --git a/src/calibre/gui2/dialogs/book_info.py b/src/calibre/gui2/dialogs/book_info.py index 4da897920c..e860579fdf 100644 --- a/src/calibre/gui2/dialogs/book_info.py +++ b/src/calibre/gui2/dialogs/book_info.py @@ -6,7 +6,7 @@ __docformat__ = 'restructuredtext en' import textwrap, os, re from PyQt4.Qt import QCoreApplication, SIGNAL, QModelIndex, QTimer, Qt, \ - QDialog, QPixmap, QGraphicsScene, QIcon, QSize + QDialog, QPixmap, QIcon, QSize from calibre.gui2.dialogs.book_info_ui import Ui_BookInfo from calibre.gui2 import dynamic, open_local_file, open_url @@ -14,12 +14,14 @@ from calibre import fit_image from calibre.library.comments import comments_to_html from calibre.utils.icu import sort_key + class BookInfo(QDialog, Ui_BookInfo): def __init__(self, parent, view, row, view_func): QDialog.__init__(self, parent) Ui_BookInfo.__init__(self) self.setupUi(self) + self.gui = parent self.cover_pixmap = None self.comments.sizeHint = self.comments_size_hint self.comments.page().setLinkDelegationPolicy(self.comments.page().DelegateAllLinks) @@ -38,11 +40,26 @@ class BookInfo(QDialog, Ui_BookInfo): self.connect(self.text, SIGNAL('linkActivated(QString)'), self.open_book_path) self.fit_cover.stateChanged.connect(self.toggle_cover_fit) self.cover.resizeEvent = self.cover_view_resized + self.cover.cover_changed.connect(self.cover_changed) desktop = QCoreApplication.instance().desktop() screen_height = desktop.availableGeometry().height() - 100 self.resize(self.size().width(), screen_height) + def cover_changed(self, data): + if self.current_row is not None: + id_ = self.view.model().id(self.current_row) + self.view.model().db.set_cover(id_, data) + if self.gui.cover_flow: + self.gui.cover_flow.dataChanged() + ci = self.view.currentIndex() + if ci.isValid(): + self.view.model().current_changed(ci, ci) + self.cover_pixmap = QPixmap() + self.cover_pixmap.loadFromData(data) + if self.fit_cover.isChecked(): + self.resize_cover() + def link_clicked(self, url): open_url(url) @@ -83,7 +100,6 @@ class BookInfo(QDialog, Ui_BookInfo): if self.cover_pixmap is None: return self.setWindowIcon(QIcon(self.cover_pixmap)) - self.scene = QGraphicsScene() pixmap = self.cover_pixmap if self.fit_cover.isChecked(): scaled, new_width, new_height = fit_image(pixmap.width(), @@ -92,8 +108,7 @@ class BookInfo(QDialog, Ui_BookInfo): if scaled: pixmap = pixmap.scaled(new_width, new_height, Qt.KeepAspectRatio, Qt.SmoothTransformation) - self.scene.addPixmap(pixmap) - self.cover.setScene(self.scene) + self.cover.set_pixmap(pixmap) def refresh(self, row): if isinstance(row, QModelIndex): diff --git a/src/calibre/gui2/dialogs/book_info.ui b/src/calibre/gui2/dialogs/book_info.ui index 2902a2c917..412126a610 100644 --- a/src/calibre/gui2/dialogs/book_info.ui +++ b/src/calibre/gui2/dialogs/book_info.ui @@ -25,7 +25,7 @@ - + @@ -115,6 +115,11 @@ QWidget

    QtWebKit/QWebView
    + + CoverView + QGraphicsView +
    calibre/gui2/widgets.h
    +     diff --git a/src/calibre/gui2/dialogs/choose_plugin_toolbars.py b/src/calibre/gui2/dialogs/choose_plugin_toolbars.py new file mode 100644 index 0000000000..ddf8e162e8 --- /dev/null +++ b/src/calibre/gui2/dialogs/choose_plugin_toolbars.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python +# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai + +from __future__ import (unicode_literals, division, absolute_import, + print_function) + +__copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net' +__docformat__ = 'restructuredtext en' +__license__ = 'GPL v3' + + +from PyQt4.Qt import QDialog, QVBoxLayout, QLabel, QDialogButtonBox, \ + QListWidget, QAbstractItemView +from PyQt4 import QtGui + +class ChoosePluginToolbarsDialog(QDialog): + + def __init__(self, parent, plugin, locations): + QDialog.__init__(self, parent) + self.locations = locations + + self.setWindowTitle( + _('Add "%s" to toolbars or menus')%plugin.name) + + self._layout = QVBoxLayout(self) + self.setLayout(self._layout) + + self._header_label = QLabel( + _('Select the toolbars and/or menus to add %s to:') % + plugin.name) + self._layout.addWidget(self._header_label) + + self._locations_list = QListWidget(self) + self._locations_list.setSelectionMode(QAbstractItemView.MultiSelection) + sizePolicy = QtGui.QSizePolicy(QtGui.QSizePolicy.Preferred, + QtGui.QSizePolicy.Minimum) + sizePolicy.setHorizontalStretch(0) + sizePolicy.setVerticalStretch(0) + self._locations_list.setSizePolicy(sizePolicy) + for key, text in locations: + self._locations_list.addItem(text) + self._layout.addWidget(self._locations_list) + + self._footer_label = QLabel( + _('You can also customise the plugin locations ' + 'using Preferences -> Customise the toolbar')) + self._layout.addWidget(self._footer_label) + + button_box = QDialogButtonBox(QDialogButtonBox.Ok | + QDialogButtonBox.Cancel) + button_box.accepted.connect(self.accept) + button_box.rejected.connect(self.reject) + self._layout.addWidget(button_box) + self.resize(self.sizeHint()) + + def selected_locations(self): + selected = [] + for row in self._locations_list.selectionModel().selectedRows(): + selected.append(self.locations[row.row()]) + return selected + diff --git a/src/calibre/gui2/dialogs/edit_authors_dialog.py b/src/calibre/gui2/dialogs/edit_authors_dialog.py index 62721f4c33..eae189e04c 100644 --- a/src/calibre/gui2/dialogs/edit_authors_dialog.py +++ b/src/calibre/gui2/dialogs/edit_authors_dialog.py @@ -66,8 +66,8 @@ class EditAuthorsDialog(QDialog, Ui_EditAuthorsDialog): self.sort_by_author_sort.setChecked(True) self.author_sort_order = 1 - # set up author sort calc button self.recalc_author_sort.clicked.connect(self.do_recalc_author_sort) + self.auth_sort_to_author.clicked.connect(self.do_auth_sort_to_author) if select_item is not None: self.table.setCurrentItem(select_item) @@ -108,6 +108,17 @@ class EditAuthorsDialog(QDialog, Ui_EditAuthorsDialog): self.table.setFocus(Qt.OtherFocusReason) self.table.cellChanged.connect(self.cell_changed) + def do_auth_sort_to_author(self): + self.table.cellChanged.disconnect() + for row in range(0,self.table.rowCount()): + item = self.table.item(row, 1) + aus = unicode(item.text()).strip() + c = self.table.item(row, 0) + # Sometimes trailing commas are left by changing between copy algs + c.setText(aus) + self.table.setFocus(Qt.OtherFocusReason) + self.table.cellChanged.connect(self.cell_changed) + def cell_changed(self, row, col): if col == 0: item = self.table.item(row, 0) diff --git a/src/calibre/gui2/dialogs/edit_authors_dialog.ui b/src/calibre/gui2/dialogs/edit_authors_dialog.ui index 6518e6a1b0..3280245959 100644 --- a/src/calibre/gui2/dialogs/edit_authors_dialog.ui +++ b/src/calibre/gui2/dialogs/edit_authors_dialog.ui @@ -52,13 +52,26 @@ - Reset all the author sort values to a value automatically generated from the author. Exactly how this value is automatically generated can be controlled via Preferences->Advanced->Tweaks + Reset all the author sort values to a value automatically +generated from the author. Exactly how this value is automatically +generated can be controlled via Preferences->Advanced->Tweaks Recalculate all author sort values + + + + Copy author sort to author for every author. You typically use this button +after changing Preferences->Advanced->Tweaks->Author sort name algorithm + + + Copy all author sort values to author + + + diff --git a/src/calibre/gui2/dialogs/metadata_bulk.py b/src/calibre/gui2/dialogs/metadata_bulk.py index c1627d7e12..e270cd0a55 100644 --- a/src/calibre/gui2/dialogs/metadata_bulk.py +++ b/src/calibre/gui2/dialogs/metadata_bulk.py @@ -7,7 +7,7 @@ import re, os, inspect from PyQt4.Qt import Qt, QDialog, QGridLayout, QVBoxLayout, QFont, QLabel, \ pyqtSignal, QDialogButtonBox, QInputDialog, QLineEdit, \ - QDate + QDate, QCompleter from calibre.gui2.dialogs.metadata_bulk_ui import Ui_MetadataBulkDialog from calibre.gui2.dialogs.tag_editor import TagEditor @@ -364,7 +364,8 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog): (fm[f]['datatype'] in ['text', 'series', 'enumeration'] and fm[f].get('search_terms', None) and f not in ['formats', 'ondevice']) or - fm[f]['datatype'] in ['int', 'float', 'bool'] ): + (fm[f]['datatype'] in ['int', 'float', 'bool'] and + f not in ['id'])): self.all_fields.append(f) self.writable_fields.append(f) if fm[f]['datatype'] == 'composite': @@ -393,6 +394,14 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog): self.book_1_text.setObjectName(name) self.testgrid.addWidget(w, i+offset, 2, 1, 1) + ident_types = sorted(self.db.get_all_identifier_types(), key=sort_key) + self.s_r_dst_ident.setCompleter(QCompleter(ident_types)) + try: + self.s_r_dst_ident.setPlaceholderText(_('Enter an identifier type')) + except: + pass + self.s_r_src_ident.addItems(ident_types) + self.main_heading = _( 'You can destroy your library using this feature. ' 'Changes are permanent. There is no undo function. ' @@ -449,6 +458,8 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog): self.test_text.editTextChanged[str].connect(self.s_r_paint_results) self.comma_separated.stateChanged.connect(self.s_r_paint_results) self.case_sensitive.stateChanged.connect(self.s_r_paint_results) + self.s_r_src_ident.currentIndexChanged[int].connect(self.s_r_paint_results) + self.s_r_dst_ident.textChanged.connect(self.s_r_paint_results) self.s_r_template.lost_focus.connect(self.s_r_template_changed) self.central_widget.setCurrentIndex(0) @@ -471,6 +482,8 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog): self.query_field.addItems(sorted([q for q in self.queries], key=sort_key)) self.query_field.currentIndexChanged[str].connect(self.s_r_query_change) self.query_field.setCurrentIndex(0) + self.search_field.setCurrentIndex(0) + self.s_r_search_field_changed(0) def s_r_sf_itemdata(self, idx): if idx is None: @@ -495,6 +508,13 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog): val = mi.get(field, None) if isinstance(val, (int, float, bool)): val = str(val) + elif fm['is_csp']: + # convert the csp dict into a list + id_type = unicode(self.s_r_src_ident.currentText()) + if id_type: + val = [val.get(id_type, '')] + else: + val = [u'%s:%s'%(t[0], t[1]) for t in val.iteritems()] if val is None: val = [] if fm['is_multiple'] else [''] elif not fm['is_multiple']: @@ -512,12 +532,17 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog): self.s_r_search_field_changed(self.search_field.currentIndex()) def s_r_search_field_changed(self, idx): - if self.search_mode.currentIndex() != 0 and idx == 1: # Template + self.s_r_template.setVisible(False) + self.template_label.setVisible(False) + self.s_r_src_ident_label.setVisible(False) + self.s_r_src_ident.setVisible(False) + if idx == 1: # Template self.s_r_template.setVisible(True) self.template_label.setVisible(True) - else: - self.s_r_template.setVisible(False) - self.template_label.setVisible(False) + elif self.s_r_sf_itemdata(idx) == 'identifiers': + self.s_r_src_ident_label.setVisible(True) + self.s_r_src_ident.setVisible(True) + for i in range(0, self.s_r_number_of_books): w = getattr(self, 'book_%d_text'%(i+1)) mi = self.db.get_metadata(self.ids[i], index_is_id=True) @@ -535,10 +560,15 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog): self.s_r_paint_results(None) def s_r_destination_field_changed(self, idx): + self.s_r_dst_ident_label.setVisible(False) + self.s_r_dst_ident.setVisible(False) txt = self.s_r_df_itemdata(idx) if not txt: txt = self.s_r_sf_itemdata(None) if txt and txt in self.writable_fields: + if txt == 'identifiers': + self.s_r_dst_ident_label.setVisible(True) + self.s_r_dst_ident.setVisible(True) self.destination_field_fm = self.db.metadata_for_field(txt) self.s_r_paint_results(None) @@ -617,6 +647,10 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog): dest = src dest_mode = self.replace_mode.currentIndex() + if self.destination_field_fm['is_csp']: + if not unicode(self.s_r_dst_ident.text()): + raise Exception(_('You must specify a destination identifier type')) + if self.destination_field_fm['is_multiple']: if self.comma_separated.isChecked(): if dest == 'authors': @@ -635,6 +669,13 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog): if dest_mode != 0: dest_val = mi.get(dest, '') + if self.db.metadata_for_field(dest)['is_csp']: + dst_id_type = unicode(self.s_r_dst_ident.text()) + if dst_id_type: + dest_val = [dest_val.get(dst_id_type, '')] + else: + # convert the csp dict into a list + dest_val = [u'%s:%s'%(t[0], t[1]) for t in dest_val.iteritems()] if dest_val is None: dest_val = [] elif not isinstance(dest_val, list): @@ -717,6 +758,17 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog): 'Book title %s not processed')%mi.title, show=True) return + # convert the colon-separated pair strings back into a dict, which + # is what set_identifiers wants + if dfm['is_csp']: + dst_id_type = unicode(self.s_r_dst_ident.text()) + if dst_id_type: + v = ''.join(val) + ids = mi.get(dest) + ids[dst_id_type] = v + val = ids + else: + val = dict([(t.split(':')) for t in val]) else: val = self.s_r_replace_mode_separator().join(val) if dest == 'title' and len(val) == 0: diff --git a/src/calibre/gui2/dialogs/metadata_bulk.ui b/src/calibre/gui2/dialogs/metadata_bulk.ui index ae3445998b..59a68d6514 100644 --- a/src/calibre/gui2/dialogs/metadata_bulk.ui +++ b/src/calibre/gui2/dialogs/metadata_bulk.ui @@ -341,7 +341,7 @@ from the value in the box 1 - 990000 + 99000000 1 @@ -732,6 +732,29 @@ Future conversion of these books will use the default settings. + + + + Identifier type: + + + s_r_src_ident + + + + + + + + 100 + 0 + + + + Choose which identifier type to operate upon + + + @@ -910,7 +933,30 @@ not multiple and the destination field is multiple - + + + + Identifier type: + + + s_r_dst_ident + + + + + + + + 100 + 0 + + + + Choose which identifier type to operate upon + + + + @@ -996,7 +1042,7 @@ not multiple and the destination field is multiple - + QFrame::NoFrame @@ -1120,6 +1166,7 @@ not multiple and the destination field is multiple remove_button search_field search_mode + s_r_src_ident s_r_template search_for case_sensitive @@ -1128,6 +1175,7 @@ not multiple and the destination field is multiple destination_field replace_mode comma_separated + s_r_dst_ident results_count starting_from multiple_separator diff --git a/src/calibre/gui2/dialogs/metadata_single.ui b/src/calibre/gui2/dialogs/metadata_single.ui index 5bcf268aaa..ced5030f94 100644 --- a/src/calibre/gui2/dialogs/metadata_single.ui +++ b/src/calibre/gui2/dialogs/metadata_single.ui @@ -419,7 +419,7 @@ If the box is colored green, then text matches the individual author's sort stri Book - 9999.989999999999782 + 99999999.989999994635582 diff --git a/src/calibre/gui2/dialogs/scheduler.py b/src/calibre/gui2/dialogs/scheduler.py index 48d0d67255..b2561342b8 100644 --- a/src/calibre/gui2/dialogs/scheduler.py +++ b/src/calibre/gui2/dialogs/scheduler.py @@ -8,9 +8,11 @@ Scheduler for automated recipe downloads ''' from datetime import timedelta +import calendar, textwrap -from PyQt4.Qt import QDialog, SIGNAL, Qt, QTime, QObject, QMenu, \ - QAction, QIcon, QMutex, QTimer, pyqtSignal +from PyQt4.Qt import QDialog, Qt, QTime, QObject, QMenu, QHBoxLayout, \ + QAction, QIcon, QMutex, QTimer, pyqtSignal, QWidget, QGridLayout, \ + QCheckBox, QTimeEdit, QLabel, QLineEdit, QDoubleSpinBox from calibre.gui2.dialogs.scheduler_ui import Ui_Dialog from calibre.gui2 import config as gconf, error_dialog @@ -18,9 +20,173 @@ from calibre.web.feeds.recipes.model import RecipeModel from calibre.ptempfile import PersistentTemporaryFile from calibre.utils.date import utcnow from calibre.utils.network import internet_connected +from calibre.utils.ordered_dict import OrderedDict +from calibre import force_unicode + +def convert_day_time_schedule(val): + day_of_week, hour, minute = val + if day_of_week == -1: + return (tuple(xrange(7)), hour, minute) + return ((day_of_week,), hour, minute) + +class Base(QWidget): + + def __init__(self, parent=None): + QWidget.__init__(self, parent) + self.l = QGridLayout() + self.setLayout(self.l) + self.setToolTip(textwrap.dedent(self.HELP)) + +class DaysOfWeek(Base): + + HELP = _('''\ + Download this periodical every week on the specified days after + the specified time. For example, if you choose: Monday after + 9:00 AM, then the periodical will be download every Monday as + soon after 9:00 AM as possible. + ''') + + def __init__(self, parent=None): + Base.__init__(self, parent) + self.days = [QCheckBox(force_unicode(calendar.day_abbr[d]), + self) for d in xrange(7)] + for i, cb in enumerate(self.days): + row = i % 2 + col = i // 2 + self.l.addWidget(cb, row, col, 1, 1) + + self.time = QTimeEdit(self) + self.time.setDisplayFormat('hh:mm AP') + self.hl = QHBoxLayout() + self.l1 = QLabel(_('&Download after:')) + self.l1.setBuddy(self.time) + self.hl.addWidget(self.l1) + self.hl.addWidget(self.time) + self.l.addLayout(self.hl, 1, 3, 1, 1) + self.initialize() + + def initialize(self, typ=None, val=None): + if typ is None: + typ = 'day/time' + val = (-1, 9, 0) + if typ == 'day/time': + val = convert_day_time_schedule(val) + + days_of_week, hour, minute = val + for i, d in enumerate(self.days): + d.setChecked(i in days_of_week) + + self.time.setTime(QTime(hour, minute)) + + @property + def schedule(self): + days_of_week = tuple([i for i, d in enumerate(self.days) if + d.isChecked()]) + t = self.time.time() + hour, minute = t.hour(), t.minute() + return 'days_of_week', (days_of_week, int(hour), int(minute)) + +class DaysOfMonth(Base): + + HELP = _('''\ + Download this periodical every month, on the specified days. + The download will happen as soon after the specified time as + possible on the specified days of each month. For example, + if you choose the 1st and the 15th after 9:00 AM, the + periodical will be downloaded on the 1st and 15th of every + month, as soon after 9:00 AM as possible. + ''') + + def __init__(self, parent=None): + Base.__init__(self, parent) + + self.l1 = QLabel(_('&Days of the month:')) + self.days = QLineEdit(self) + self.days.setToolTip(_('Comma separated list of days of the month.' + ' For example: 1, 15')) + self.l1.setBuddy(self.days) + + self.l2 = QLabel(_('Download &after:')) + self.time = QTimeEdit(self) + self.time.setDisplayFormat('hh:mm AP') + self.l2.setBuddy(self.time) + + self.l.addWidget(self.l1, 0, 0, 1, 1) + self.l.addWidget(self.days, 0, 1, 1, 1) + self.l.addWidget(self.l2, 1, 0, 1, 1) + self.l.addWidget(self.time, 1, 1, 1, 1) + + def initialize(self, typ=None, val=None): + if val is None: + val = ((1,), 9, 0) + days_of_month, hour, minute = val + self.days.setText(', '.join(map(str, map(int, days_of_month)))) + self.time.setTime(QTime(hour, minute)) + + @property + def schedule(self): + parts = [x.strip() for x in unicode(self.days.text()).split(',') if + x.strip()] + try: + days_of_month = tuple(map(int, parts)) + except: + days_of_month = (1,) + if not days_of_month: + days_of_month = (1,) + t = self.time.time() + hour, minute = t.hour(), t.minute() + return 'days_of_month', (days_of_month, int(hour), int(minute)) + +class EveryXDays(Base): + + HELP = _('''\ + Download this periodical every x days. For example, if you + choose 30 days, the periodical will be downloaded every 30 + days. Note that you can set periods of less than a day, like + 0.1 days to download a periodical more than once a day. + ''') + + def __init__(self, parent=None): + Base.__init__(self, parent) + self.l1 = QLabel(_('&Download every:')) + self.interval = QDoubleSpinBox(self) + self.interval.setMinimum(0.04) + self.interval.setSpecialValueText(_('every hour')) + self.interval.setMaximum(1000.0) + self.interval.setValue(31.0) + self.interval.setSuffix(' ' + _('days')) + self.interval.setSingleStep(1.0) + self.interval.setDecimals(2) + self.l1.setBuddy(self.interval) + self.l2 = QLabel(_('Note: You can set intervals of less than a day,' + ' by typing the value manually.')) + self.l2.setWordWrap(True) + + self.l.addWidget(self.l1, 0, 0, 1, 1) + self.l.addWidget(self.interval, 0, 1, 1, 1) + self.l.addWidget(self.l2, 1, 0, 1, -1) + + def initialize(self, typ=None, val=None): + if val is None: + val = 31.0 + self.interval.setValue(val) + + @property + def schedule(self): + schedule = self.interval.value() + return 'interval', schedule + class SchedulerDialog(QDialog, Ui_Dialog): + SCHEDULE_TYPES = OrderedDict([ + ('days_of_week', DaysOfWeek), + ('days_of_month', DaysOfMonth), + ('every_x_days', EveryXDays), + ]) + + download = pyqtSignal(object) + def __init__(self, recipe_model, parent=None): QDialog.__init__(self, parent) self.setupUi(self) @@ -30,6 +196,11 @@ class SchedulerDialog(QDialog, Ui_Dialog): _('%s news sources') % self.recipe_model.showing_count) + self.schedule_widgets = [] + for key in reversed(self.SCHEDULE_TYPES): + self.schedule_widgets.insert(0, self.SCHEDULE_TYPES[key](self)) + self.schedule_stack.insertWidget(0, self.schedule_widgets[0]) + self.search.initialize('scheduler_search_history') self.search.setMinimumContentsLength(15) self.search.search.connect(self.recipe_model.search) @@ -43,29 +214,43 @@ class SchedulerDialog(QDialog, Ui_Dialog): self.detail_box.setVisible(False) self.download_button.setVisible(False) self.recipes.currentChanged = self.current_changed - self.interval_button.setChecked(True) + for b, c in self.SCHEDULE_TYPES.iteritems(): + b = getattr(self, b) + b.toggled.connect(self.schedule_type_selected) + b.setToolTip(textwrap.dedent(c.HELP)) + self.days_of_week.setChecked(True) - self.connect(self.schedule, SIGNAL('stateChanged(int)'), - self.toggle_schedule_info) - self.connect(self.show_password, SIGNAL('stateChanged(int)'), - lambda state: self.password.setEchoMode(self.password.Normal if state == Qt.Checked else self.password.Password)) - self.connect(self.download_button, SIGNAL('clicked()'), - self.download_clicked) - self.connect(self.download_all_button, SIGNAL('clicked()'), + self.schedule.stateChanged[int].connect(self.toggle_schedule_info) + self.show_password.stateChanged[int].connect(self.set_pw_echo_mode) + self.download_button.clicked.connect(self.download_clicked) + self.download_all_button.clicked.connect( self.download_all_clicked) self.old_news.setValue(gconf['oldest_news']) + def set_pw_echo_mode(self, state): + self.password.setEchoMode(self.password.Normal + if state == Qt.Checked else self.password.Password) + + + def schedule_type_selected(self, *args): + for i, st in enumerate(self.SCHEDULE_TYPES): + if getattr(self, st).isChecked(): + self.schedule_stack.setCurrentIndex(i) + break + def keyPressEvent(self, ev): if ev.key() not in (Qt.Key_Enter, Qt.Key_Return): return QDialog.keyPressEvent(self, ev) def break_cycles(self): - self.disconnect(self.recipe_model, SIGNAL('searched(PyQt_PyObject)'), - self.search_done) - self.disconnect(self.recipe_model, SIGNAL('searched(PyQt_PyObject)'), - self.search.search_done) - self.search.search.disconnect() + try: + self.recipe_model.searched.disconnect(self.search_done) + self.recipe_model.searched.disconnect(self.search.search_done) + self.search.search.disconnect() + self.download.disconnect() + except: + pass self.recipe_model = None def search_done(self, *args): @@ -74,8 +259,9 @@ class SchedulerDialog(QDialog, Ui_Dialog): def toggle_schedule_info(self, *args): enabled = self.schedule.isChecked() - for x in ('daily_button', 'day', 'time', 'interval_button', 'interval'): + for x in self.SCHEDULE_TYPES: getattr(self, x).setEnabled(enabled) + self.schedule_stack.setEnabled(enabled) self.last_downloaded.setVisible(enabled) def current_changed(self, current, previous): @@ -97,14 +283,14 @@ class SchedulerDialog(QDialog, Ui_Dialog): return False return QDialog.accept(self) - def download_clicked(self): + def download_clicked(self, *args): self.commit() if self.commit() and self.current_urn: - self.emit(SIGNAL('download(PyQt_PyObject)'), self.current_urn) + self.download.emit(self.current_urn) - def download_all_clicked(self): + def download_all_clicked(self, *args): if self.commit() and self.commit(): - self.emit(SIGNAL('download(PyQt_PyObject)'), None) + self.download.emit(None) @property def current_urn(self): @@ -130,16 +316,8 @@ class SchedulerDialog(QDialog, Ui_Dialog): self.recipe_model.set_account_info(urn, un, pw) if self.schedule.isChecked(): - schedule_type = 'interval' if self.interval_button.isChecked() else 'day/time' - if schedule_type == 'interval': - schedule = self.interval.value() - if schedule < 0.1: - schedule = 1./24. - else: - day_of_week = self.day.currentIndex() - 1 - t = self.time.time() - hour, minute = t.hour(), t.minute() - schedule = (day_of_week, hour, minute) + schedule_type, schedule = \ + self.schedule_stack.currentWidget().schedule self.recipe_model.schedule_recipe(urn, schedule_type, schedule) else: self.recipe_model.un_schedule_recipe(urn) @@ -192,27 +370,27 @@ class SchedulerDialog(QDialog, Ui_Dialog): self.schedule.setChecked(scheduled) self.toggle_schedule_info() self.last_downloaded.setText(_('Last downloaded: never')) + ld_text = _('never') if scheduled: typ, sch, last_downloaded = schedule_info - if typ == 'interval': - self.interval_button.setChecked(True) - self.interval.setValue(sch) - elif typ == 'day/time': - self.daily_button.setChecked(True) - day, hour, minute = sch - self.day.setCurrentIndex(day+1) - self.time.setTime(QTime(hour, minute)) - d = utcnow() - last_downloaded def hm(x): return (x-x%3600)//3600, (x%3600 - (x%3600)%60)//60 hours, minutes = hm(d.seconds) tm = _('%d days, %d hours and %d minutes ago')%(d.days, hours, minutes) if d < timedelta(days=366): - self.last_downloaded.setText(_('Last downloaded')+': '+tm) - + ld_text = tm + else: + typ, sch = 'day/time', (-1, 9, 0) + sch_widget = {'day/time': 0, 'days_of_week': 0, 'days_of_month':1, + 'interval':2}[typ] + rb = getattr(self, list(self.SCHEDULE_TYPES)[sch_widget]) + rb.setChecked(True) + self.schedule_stack.setCurrentIndex(sch_widget) + self.schedule_stack.currentWidget().initialize(typ, sch) add_title_tag, custom_tags, keep_issues = customize_info self.add_title_tag.setChecked(add_title_tag) self.custom_tags.setText(u', '.join(custom_tags)) + self.last_downloaded.setText(_('Last downloaded:') + ' ' + ld_text) try: keep_issues = int(keep_issues) except: @@ -233,7 +411,8 @@ class Scheduler(QObject): QObject.__init__(self, parent) self.internet_connection_failed = False self._parent = parent - self.recipe_model = RecipeModel(db) + self.recipe_model = RecipeModel() + self.db = db self.lock = QMutex(QMutex.Recursive) self.download_queue = set([]) @@ -241,9 +420,9 @@ class Scheduler(QObject): self.news_icon = QIcon(I('news.png')) self.scheduler_action = QAction(QIcon(I('scheduler.png')), _('Schedule news download'), self) self.news_menu.addAction(self.scheduler_action) - self.connect(self.scheduler_action, SIGNAL('triggered(bool)'), self.show_dialog) + self.scheduler_action.triggered[bool].connect(self.show_dialog) self.cac = QAction(QIcon(I('user_profile.png')), _('Add a custom news source'), self) - self.connect(self.cac, SIGNAL('triggered(bool)'), self.customize_feeds) + self.cac.triggered[bool].connect(self.customize_feeds) self.news_menu.addAction(self.cac) self.news_menu.addSeparator() self.all_action = self.news_menu.addAction( @@ -252,10 +431,12 @@ class Scheduler(QObject): self.timer = QTimer(self) self.timer.start(int(self.INTERVAL * 60 * 1000)) - self.connect(self.timer, SIGNAL('timeout()'), self.check) + self.timer.timeout.connect(self.check) self.oldest = gconf['oldest_news'] QTimer.singleShot(5 * 1000, self.oldest_check) - self.database_changed = self.recipe_model.database_changed + + def database_changed(self, db): + self.db = db def oldest_check(self): if self.oldest > 0: @@ -264,10 +445,8 @@ class Scheduler(QObject): ids = list(self.recipe_model.db.tags_older_than(_('News'), delta)) except: - # Should never happen + # Happens if library is being switched ids = [] - import traceback - traceback.print_exc() if ids: if ids: self.delete_old_news.emit(ids) @@ -278,8 +457,7 @@ class Scheduler(QObject): self.lock.lock() try: d = SchedulerDialog(self.recipe_model) - self.connect(d, SIGNAL('download(PyQt_PyObject)'), - self.download_clicked) + d.download.connect(self.download_clicked) d.exec_() gconf['oldest_news'] = self.oldest = d.old_news.value() d.break_cycles() @@ -374,7 +552,6 @@ class Scheduler(QObject): if __name__ == '__main__': from calibre.gui2 import is_ok_to_use_qt is_ok_to_use_qt() - from calibre.library.database2 import LibraryDatabase2 - d = SchedulerDialog(RecipeModel(LibraryDatabase2('/home/kovid/documents/library'))) + d = SchedulerDialog(RecipeModel()) d.exec_() diff --git a/src/calibre/gui2/dialogs/scheduler.ui b/src/calibre/gui2/dialogs/scheduler.ui index 26953bbe16..f295703b33 100644 --- a/src/calibre/gui2/dialogs/scheduler.ui +++ b/src/calibre/gui2/dialogs/scheduler.ui @@ -61,7 +61,7 @@ &Schedule - + @@ -76,141 +76,94 @@ - - - &Schedule for download: + + + Qt::Vertical + + + 20 + 40 + + + + + + + + QFrame::StyledPanel + + + QFrame::Raised + + + + + + &Schedule for download: + + + + + + + + + Days of week + + + + + + + Days of month + + + + + + + Every x days + + + + + + + + + + 16777215 + 75 + + + + + + + + + + + true + + + + - - - - - Every - - - - - - - - day - - - - - Monday - - - - - Tuesday - - - - - Wednesday - - - - - Thursday - - - - - Friday - - - - - Saturday - - - - - Sunday - - - - - - - - at - - - - - - - - - - Qt::Horizontal - - - - 40 - 20 - - - - - - - - - - - - Every - - - - - - - - 0 - 0 - - - - Interval at which to download this recipe. A value of zero means that the recipe will be downloaded every hour. - - - days - - - 1 - - - 0.000000000000000 - - - 365.100000000000023 - - - 1.000000000000000 - - - 1.000000000000000 - - - - - - - - - + + + Qt::Vertical - - true + + + 20 + 40 + - + @@ -343,6 +296,19 @@ + + + + Qt::Vertical + + + + 20 + 40 + + + + @@ -485,54 +451,6 @@ - - daily_button - toggled(bool) - day - setEnabled(bool) - - - 458 - 155 - - - 573 - 158 - - - - - daily_button - toggled(bool) - time - setEnabled(bool) - - - 458 - 155 - - - 684 - 157 - - - - - interval_button - toggled(bool) - interval - setEnabled(bool) - - - 458 - 212 - - - 752 - 215 - - - add_title_tag toggled(bool) diff --git a/src/calibre/gui2/dialogs/user_profiles.py b/src/calibre/gui2/dialogs/user_profiles.py index fe64deb430..5453a90766 100644 --- a/src/calibre/gui2/dialogs/user_profiles.py +++ b/src/calibre/gui2/dialogs/user_profiles.py @@ -4,13 +4,13 @@ __copyright__ = '2008, Kovid Goyal ' import time, os from PyQt4.Qt import SIGNAL, QUrl, QAbstractListModel, Qt, \ - QVariant + QVariant, QFont -from calibre.web.feeds.recipes import compile_recipe +from calibre.web.feeds.recipes import compile_recipe, custom_recipes from calibre.web.feeds.news import AutomaticNewsRecipe from calibre.gui2.dialogs.user_profiles_ui import Ui_Dialog from calibre.gui2 import error_dialog, question_dialog, open_url, \ - choose_files, ResizableDialog, NONE + choose_files, ResizableDialog, NONE, open_local_file from calibre.gui2.widgets import PythonHighlighter from calibre.ptempfile import PersistentTemporaryFile from calibre.utils.icu import sort_key @@ -83,6 +83,9 @@ class UserProfiles(ResizableDialog, Ui_Dialog): self._model = self.model = CustomRecipeModel(recipe_model) self.available_profiles.setModel(self._model) self.available_profiles.currentChanged = self.current_changed + f = QFont() + f.setStyleHint(f.Monospace) + self.source_code.setFont(f) self.connect(self.remove_feed_button, SIGNAL('clicked(bool)'), self.added_feeds.remove_selected_items) @@ -93,6 +96,7 @@ class UserProfiles(ResizableDialog, Ui_Dialog): self.connect(self.load_button, SIGNAL('clicked()'), self.load) self.connect(self.builtin_recipe_button, SIGNAL('clicked()'), self.add_builtin_recipe) self.connect(self.share_button, SIGNAL('clicked()'), self.share) + self.show_recipe_files_button.clicked.connect(self.show_recipe_files) self.connect(self.down_button, SIGNAL('clicked()'), self.down) self.connect(self.up_button, SIGNAL('clicked()'), self.up) self.connect(self.add_profile_button, SIGNAL('clicked(bool)'), @@ -102,6 +106,10 @@ class UserProfiles(ResizableDialog, Ui_Dialog): self.connect(self.toggle_mode_button, SIGNAL('clicked(bool)'), self.toggle_mode) self.clear() + def show_recipe_files(self, *args): + bdir = os.path.dirname(custom_recipes.file_path) + open_local_file(bdir) + def break_cycles(self): self.recipe_model = self._model.recipe_model = None self.available_profiles = None @@ -366,8 +374,7 @@ class %(classname)s(%(base_class)s): if __name__ == '__main__': from calibre.gui2 import is_ok_to_use_qt is_ok_to_use_qt() - from calibre.library.database2 import LibraryDatabase2 from calibre.web.feeds.recipes.model import RecipeModel - d=UserProfiles(None, RecipeModel(LibraryDatabase2('/home/kovid/documents/library'))) + d=UserProfiles(None, RecipeModel()) d.exec_() diff --git a/src/calibre/gui2/dialogs/user_profiles.ui b/src/calibre/gui2/dialogs/user_profiles.ui index 4ca47539d1..7631c74768 100644 --- a/src/calibre/gui2/dialogs/user_profiles.ui +++ b/src/calibre/gui2/dialogs/user_profiles.ui @@ -35,7 +35,7 @@ 0 0 730 - 600 + 601 @@ -102,6 +102,17 @@ + + + + S&how recipe files + + + + :/images/document_open.png:/images/document_open.png + + + @@ -399,11 +410,6 @@ p, li { white-space: pre-wrap; } 0 - - - DejaVu Sans Mono - - QTextEdit::NoWrap diff --git a/src/calibre/gui2/dnd.py b/src/calibre/gui2/dnd.py new file mode 100644 index 0000000000..928de72578 --- /dev/null +++ b/src/calibre/gui2/dnd.py @@ -0,0 +1,325 @@ +#!/usr/bin/env python +# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai +from __future__ import (unicode_literals, division, absolute_import, + print_function) + +__license__ = 'GPL v3' +__copyright__ = '2011, Kovid Goyal ' +__docformat__ = 'restructuredtext en' + +import posixpath, os, urllib, re +from urlparse import urlparse, urlunparse +from threading import Thread +from Queue import Queue, Empty + +from PyQt4.Qt import QPixmap, Qt, QDialog, QLabel, QVBoxLayout, \ + QDialogButtonBox, QProgressBar, QTimer + +from calibre.constants import DEBUG, iswindows +from calibre.ptempfile import PersistentTemporaryFile +from calibre import browser, as_unicode, prints +from calibre.gui2 import error_dialog + +IMAGE_EXTENSIONS = ['jpg', 'jpeg', 'gif', 'png', 'bmp'] + +class Worker(Thread): # {{{ + + def __init__(self, url, fpath, rq): + Thread.__init__(self) + self.url, self.fpath = url, fpath + self.daemon = True + self.rq = rq + self.err = self.tb = None + + def run(self): + try: + br = browser() + br.retrieve(self.url, self.fpath, self.callback) + except Exception, e: + self.err = as_unicode(e) + import traceback + self.tb = traceback.format_exc() + + def callback(self, a, b, c): + self.rq.put((a, b, c)) +# }}} + +class DownloadDialog(QDialog): # {{{ + + def __init__(self, url, fname, parent): + QDialog.__init__(self, parent) + self.setWindowTitle(_('Download %s')%fname) + self.l = QVBoxLayout(self) + self.purl = urlparse(url) + self.msg = QLabel(_('Downloading %s from %s')%(fname, + self.purl.netloc)) + self.msg.setWordWrap(True) + self.l.addWidget(self.msg) + self.pb = QProgressBar(self) + self.pb.setMinimum(0) + self.pb.setMaximum(0) + self.l.addWidget(self.pb) + self.bb = QDialogButtonBox(QDialogButtonBox.Cancel, Qt.Horizontal, self) + self.l.addWidget(self.bb) + self.bb.rejected.connect(self.reject) + sz = self.sizeHint() + self.resize(max(sz.width(), 400), sz.height()) + + fpath = PersistentTemporaryFile(os.path.splitext(fname)[1]) + fpath.close() + self.fpath = fpath.name + + self.worker = Worker(url, self.fpath, Queue()) + self.rejected = False + + def reject(self): + self.rejected = True + QDialog.reject(self) + + def start_download(self): + self.worker.start() + QTimer.singleShot(50, self.update) + self.exec_() + if self.worker.err is not None: + error_dialog(self.parent(), _('Download failed'), + _('Failed to download from %r with error: %s')%( + self.worker.url, self.worker.err), + det_msg=self.worker.tb, show=True) + + def update(self): + if self.rejected: + return + + try: + progress = self.worker.rq.get_nowait() + except Empty: + pass + else: + self.update_pb(progress) + + if not self.worker.is_alive(): + return self.accept() + QTimer.singleShot(50, self.update) + + def update_pb(self, progress): + transferred, block_size, total = progress + if total == -1: + self.pb.setMaximum(0) + self.pb.setMinimum(0) + self.pb.setValue(0) + else: + so_far = transferred * block_size + self.pb.setMaximum(max(total, so_far)) + self.pb.setValue(so_far) + + @property + def err(self): + return self.worker.err + +# }}} + +def dnd_has_image(md): + return md.hasImage() + +def data_as_string(f, md): + raw = bytes(md.data(f)) + if '/x-moz' in f: + try: + raw = raw.decode('utf-16') + except: + pass + return raw + +def dnd_has_extension(md, extensions): + if DEBUG: + prints('Debugging DND event') + for f in md.formats(): + f = unicode(f) + prints(f, repr(data_as_string(f, md))[:300], '\n') + print () + if has_firefox_ext(md, extensions): + return True + if md.hasUrls(): + urls = [unicode(u.toString()) for u in + md.urls()] + purls = [urlparse(u) for u in urls] + if DEBUG: + prints('URLS:', urls) + prints('Paths:', [u2p(x) for x in purls]) + + exts = frozenset([posixpath.splitext(u.path)[1][1:].lower() for u in + purls]) + return bool(exts.intersection(frozenset(extensions))) + return False + +def u2p(url): + path = url.path + if iswindows: + if path.startswith('/'): + path = path[1:] + ans = path.replace('/', os.sep) + if os.path.exists(ans): + return ans + # Try unquoting the URL + return urllib.unquote(ans) + +def dnd_get_image(md, image_exts=IMAGE_EXTENSIONS): + ''' + Get the image in the QMimeData object md. + + :return: None, None if no image is found + QPixmap, None if an image is found, the pixmap is guaranteed not + null + url, filename if a URL that points to an image is found + ''' + if dnd_has_image(md): + for x in md.formats(): + x = unicode(x) + if x.startswith('image/'): + cdata = bytes(md.data(x)) + pmap = QPixmap() + pmap.loadFromData(cdata) + if not pmap.isNull(): + return pmap, None + break + + # No image, look for a URL pointing to an image + if md.hasUrls(): + urls = [unicode(u.toString()) for u in + md.urls()] + purls = [urlparse(u) for u in urls] + # First look for a local file + images = [u2p(x) for x in purls if x.scheme in ('', 'file') and + posixpath.splitext(urllib.unquote(x.path))[1][1:].lower() in + image_exts] + images = [x for x in images if os.path.exists(x)] + p = QPixmap() + for path in images: + try: + with open(path, 'rb') as f: + p.loadFromData(f.read()) + except: + continue + if not p.isNull(): + return p, None + + # No local images, look for remote ones + + # First, see if this is from Firefox + rurl, fname = get_firefox_rurl(md, image_exts) + + if rurl and fname: + return rurl, fname + # Look through all remaining URLs + remote_urls = [x for x in purls if x.scheme in ('http', 'https', + 'ftp') and posixpath.splitext(x.path)[1][1:].lower() in image_exts] + if remote_urls: + rurl = remote_urls[0] + fname = posixpath.basename(urllib.unquote(rurl.path)) + return urlunparse(rurl), fname + + return None, None + +def dnd_get_files(md, exts): + ''' + Get the file in the QMimeData object md with an extension that is one of + the extensions in exts. + + :return: None, None if no file is found + [paths], None if a local file is found + [urls], [filenames] if URLs that point to a files are found + ''' + # Look for a URL pointing to a file + if md.hasUrls(): + urls = [unicode(u.toString()) for u in + md.urls()] + purls = [urlparse(u) for u in urls] + # First look for a local file + local_files = [u2p(x) for x in purls if x.scheme in ('', 'file') and + posixpath.splitext(urllib.unquote(x.path))[1][1:].lower() in + exts] + local_files = [x for x in local_files if os.path.exists(x)] + if local_files: + return local_files, None + + # No local files, look for remote ones + + # First, see if this is from Firefox + rurl, fname = get_firefox_rurl(md, exts) + if rurl and fname: + return [rurl], [fname] + + # Look through all remaining URLs + remote_urls = [x for x in purls if x.scheme in ('http', 'https', + 'ftp') and posixpath.splitext(x.path)[1][1:].lower() in exts] + if remote_urls: + filenames = [posixpath.basename(urllib.unquote(rurl.path)) for rurl in + remote_urls] + return [urlunparse(x) for x in remote_urls], filenames + + return None, None + +def _get_firefox_pair(md, exts, url, fname): + url = bytes(md.data(url)).decode('utf-16') + fname = bytes(md.data(fname)).decode('utf-16') + while url.endswith('\x00'): + url = url[:-1] + while fname.endswith('\x00'): + fname = fname[:-1] + if not url or not fname: + return None, None + ext = posixpath.splitext(fname)[1][1:].lower() + # Weird firefox bug on linux + ext = {'jpe':'jpg', 'epu':'epub', 'mob':'mobi'}.get(ext, ext) + fname = os.path.splitext(fname)[0] + '.' + ext + if DEBUG: + prints('Firefox file promise:', url, fname) + if ext not in exts: + fname = url = None + return url, fname + + +def get_firefox_rurl(md, exts): + formats = frozenset([unicode(x) for x in md.formats()]) + url = fname = None + if 'application/x-moz-file-promise-url' in formats and \ + 'application/x-moz-file-promise-dest-filename' in formats: + try: + url, fname = _get_firefox_pair(md, exts, + 'application/x-moz-file-promise-url', + 'application/x-moz-file-promise-dest-filename') + except: + if DEBUG: + import traceback + traceback.print_exc() + if url is None and 'text/x-moz-url-data' in formats and \ + 'text/x-moz-url-desc' in formats: + try: + url, fname = _get_firefox_pair(md, exts, + 'text/x-moz-url-data', 'text/x-moz-url-desc') + except: + if DEBUG: + import traceback + traceback.print_exc() + + if url is None and '_NETSCAPE_URL' in formats: + try: + raw = bytes(md.data('_NETSCAPE_URL')) + raw = raw.decode('utf-8') + lines = raw.splitlines() + if len(lines) > 1 and re.match(r'[a-z]+://', lines[1]) is None: + url, fname = lines[:2] + ext = posixpath.splitext(fname)[1][1:].lower() + if ext not in exts: + fname = url = None + except: + if DEBUG: + import traceback + traceback.print_exc() + if DEBUG: + prints('Firefox rurl:', url, fname) + return url, fname + +def has_firefox_ext(md, exts): + return bool(get_firefox_rurl(md, exts)[0]) + diff --git a/src/calibre/gui2/init.py b/src/calibre/gui2/init.py index 0ca58582b6..80f1f1c2cf 100644 --- a/src/calibre/gui2/init.py +++ b/src/calibre/gui2/init.py @@ -44,13 +44,13 @@ class LibraryViewMixin(object): # {{{ for view in (self.library_view, self.memory_view, self.card_a_view, self.card_b_view): getattr(view, func)(*args) - self.memory_view.connect_dirtied_signal(self.upload_booklists) + self.memory_view.connect_dirtied_signal(self.upload_dirtied_booklists) self.memory_view.connect_upload_collections_signal( func=self.upload_collections, oncard=None) - self.card_a_view.connect_dirtied_signal(self.upload_booklists) + self.card_a_view.connect_dirtied_signal(self.upload_dirtied_booklists) self.card_a_view.connect_upload_collections_signal( func=self.upload_collections, oncard='carda') - self.card_b_view.connect_dirtied_signal(self.upload_booklists) + self.card_b_view.connect_dirtied_signal(self.upload_dirtied_booklists) self.card_b_view.connect_upload_collections_signal( func=self.upload_collections, oncard='cardb') self.book_on_device(None, reset=True) @@ -141,6 +141,15 @@ class Stack(QStackedWidget): # {{{ # }}} +class UpdateLabel(QLabel): # {{{ + + def __init__(self, *args, **kwargs): + QLabel.__init__(self, *args, **kwargs) + + def contextMenuEvent(self, e): + pass +# }}} + class StatusBar(QStatusBar): # {{{ def __init__(self, parent=None): @@ -148,7 +157,7 @@ class StatusBar(QStatusBar): # {{{ self.default_message = __appname__ + ' ' + _('version') + ' ' + \ self.get_version() + ' ' + _('created by Kovid Goyal') self.device_string = '' - self.update_label = QLabel('') + self.update_label = UpdateLabel('') self.addPermanentWidget(self.update_label) self.update_label.setVisible(False) self._font = QFont() @@ -253,6 +262,11 @@ class LayoutMixin(object): # {{{ self.status_bar.initialize(self.system_tray_icon) self.book_details.show_book_info.connect(self.iactions['Show Book Details'].show_book_info) self.book_details.files_dropped.connect(self.iactions['Add Books'].files_dropped_on_book) + self.book_details.cover_changed.connect(self.bd_cover_changed, + type=Qt.QueuedConnection) + self.book_details.remote_file_dropped.connect( + self.iactions['Add Books'].remote_file_dropped_on_book, + type=Qt.QueuedConnection) self.book_details.open_containing_folder.connect(self.iactions['View'].view_folder_for_id) self.book_details.view_specific_format.connect(self.iactions['View'].view_format_by_id) @@ -263,6 +277,10 @@ class LayoutMixin(object): # {{{ self.library_view.currentIndex()) self.library_view.setFocus(Qt.OtherFocusReason) + def bd_cover_changed(self, id_, cdata): + self.library_view.model().db.set_cover(id_, cdata) + if self.cover_flow: + self.cover_flow.dataChanged() def save_layout_state(self): for x in ('library', 'memory', 'card_a', 'card_b'): diff --git a/src/calibre/gui2/library/delegates.py b/src/calibre/gui2/library/delegates.py index 87da6818eb..3a090f8102 100644 --- a/src/calibre/gui2/library/delegates.py +++ b/src/calibre/gui2/library/delegates.py @@ -5,7 +5,6 @@ __license__ = 'GPL v3' __copyright__ = '2010, Kovid Goyal ' __docformat__ = 'restructuredtext en' -import sys from math import cos, sin, pi from PyQt4.Qt import QColor, Qt, QModelIndex, QSize, \ @@ -245,13 +244,13 @@ class CcTextDelegate(QStyledItemDelegate): # {{{ typ = m.custom_columns[col]['datatype'] if typ == 'int': editor = QSpinBox(parent) - editor.setRange(-100, sys.maxint) + editor.setRange(-100, 100000000) editor.setSpecialValueText(_('Undefined')) editor.setSingleStep(1) elif typ == 'float': editor = QDoubleSpinBox(parent) editor.setSpecialValueText(_('Undefined')) - editor.setRange(-100., float(sys.maxint)) + editor.setRange(-100., 100000000) editor.setDecimals(2) else: editor = MultiCompleteLineEdit(parent) diff --git a/src/calibre/gui2/library/models.py b/src/calibre/gui2/library/models.py index a5e68ab6a6..0658c0604a 100644 --- a/src/calibre/gui2/library/models.py +++ b/src/calibre/gui2/library/models.py @@ -22,7 +22,7 @@ from calibre.utils.icu import sort_key, strcmp as icu_strcmp from calibre.ebooks.metadata.meta import set_metadata as _set_metadata from calibre.utils.search_query_parser import SearchQueryParser from calibre.library.caches import _match, CONTAINS_MATCH, EQUALS_MATCH, \ - REGEXP_MATCH, MetadataBackup + REGEXP_MATCH, MetadataBackup, force_to_bool from calibre import strftime, isbytestring, prepare_string_for_xml from calibre.constants import filesystem_encoding, DEBUG from calibre.gui2.library import DEFAULT_SORT @@ -268,6 +268,15 @@ class BooksModel(QAbstractTableModel): # {{{ return None return self.get_current_highlighted_id() + def highlight_ids(self, ids_to_highlight): + self.ids_to_highlight = ids_to_highlight + self.ids_to_highlight_set = set(self.ids_to_highlight) + if self.ids_to_highlight: + self.current_highlighted_idx = 0 + else: + self.current_highlighted_idx = None + self.reset() + def search(self, text, reset=True): try: if self.highlight_only: @@ -615,7 +624,7 @@ class BooksModel(QAbstractTableModel): # {{{ return None # displayed using a decorator def bool_type_decorator(r, idx=-1, bool_cols_are_tristate=True): - val = self.db.data[r][idx] + val = force_to_bool(self.db.data[r][idx]) if not bool_cols_are_tristate: if val is None or not val: return self.bool_no_icon @@ -674,8 +683,14 @@ class BooksModel(QAbstractTableModel): # {{{ idx = self.custom_columns[col]['rec_index'] datatype = self.custom_columns[col]['datatype'] if datatype in ('text', 'comments', 'composite', 'enumeration'): - self.dc[col] = functools.partial(text_type, idx=idx, - mult=self.custom_columns[col]['is_multiple']) + mult=self.custom_columns[col]['is_multiple'] + self.dc[col] = functools.partial(text_type, idx=idx, mult=mult) + if datatype in ['text', 'composite', 'enumeration'] and not mult: + if self.custom_columns[col]['display'].get('use_decorations', False): + self.dc_decorator[col] = functools.partial( + bool_type_decorator, idx=idx, + bool_cols_are_tristate= + tweaks['bool_custom_columns_are_tristate'] != 'no') elif datatype in ('int', 'float'): self.dc[col] = functools.partial(number_type, idx=idx) elif datatype == 'datetime': @@ -684,7 +699,8 @@ class BooksModel(QAbstractTableModel): # {{{ self.dc[col] = functools.partial(bool_type, idx=idx) self.dc_decorator[col] = functools.partial( bool_type_decorator, idx=idx, - bool_cols_are_tristate=tweaks['bool_custom_columns_are_tristate'] != 'no') + bool_cols_are_tristate= + tweaks['bool_custom_columns_are_tristate'] != 'no') elif datatype == 'rating': self.dc[col] = functools.partial(rating_type, idx=idx) elif datatype == 'series': diff --git a/src/calibre/gui2/metadata/basic_widgets.py b/src/calibre/gui2/metadata/basic_widgets.py index a135176daf..d5a8de7b67 100644 --- a/src/calibre/gui2/metadata/basic_widgets.py +++ b/src/calibre/gui2/metadata/basic_widgets.py @@ -351,7 +351,7 @@ class SeriesIndexEdit(QDoubleSpinBox): QDoubleSpinBox.__init__(self, parent) self.dialog = parent self.db = self.original_series_name = None - self.setMaximum(1000000) + self.setMaximum(10000000) self.series_edit = series_edit series_edit.currentIndexChanged.connect(self.enable) series_edit.editTextChanged.connect(self.enable) diff --git a/src/calibre/gui2/preferences/__init__.py b/src/calibre/gui2/preferences/__init__.py index 7267716ea8..54eb2f713c 100644 --- a/src/calibre/gui2/preferences/__init__.py +++ b/src/calibre/gui2/preferences/__init__.py @@ -8,7 +8,7 @@ __docformat__ = 'restructuredtext en' import textwrap from PyQt4.Qt import QWidget, pyqtSignal, QCheckBox, QAbstractSpinBox, \ - QLineEdit, QComboBox, QVariant + QLineEdit, QComboBox, QVariant, Qt from calibre.customize.ui import preferences_plugins from calibre.utils.config import ConfigProxy @@ -82,6 +82,8 @@ class ConfigWidgetInterface(object): class Setting(object): + CHOICES_SEARCH_FLAGS = Qt.MatchExactly | Qt.MatchCaseSensitive + def __init__(self, name, config_obj, widget, gui_name=None, empty_string_is_None=True, choices=None, restart_required=False): self.name, self.gui_name = name, gui_name @@ -168,7 +170,8 @@ class Setting(object): elif self.datatype == 'string': self.gui_obj.setText(val if val else '') elif self.datatype == 'choice': - idx = self.gui_obj.findData(QVariant(val)) + idx = self.gui_obj.findData(QVariant(val), role=Qt.UserRole, + flags=self.CHOICES_SEARCH_FLAGS) if idx == -1: idx = 0 self.gui_obj.setCurrentIndex(idx) diff --git a/src/calibre/gui2/preferences/behavior.py b/src/calibre/gui2/preferences/behavior.py index aeee6e5064..45a63ce529 100644 --- a/src/calibre/gui2/preferences/behavior.py +++ b/src/calibre/gui2/preferences/behavior.py @@ -9,7 +9,7 @@ import re from PyQt4.Qt import Qt, QVariant, QListWidgetItem -from calibre.gui2.preferences import ConfigWidgetBase, test_widget +from calibre.gui2.preferences import ConfigWidgetBase, test_widget, Setting from calibre.gui2.preferences.behavior_ui import Ui_Form from calibre.gui2 import config, info_dialog, dynamic from calibre.utils.config import prefs @@ -20,6 +20,10 @@ from calibre.ebooks.oeb.iterator import is_supported from calibre.constants import iswindows from calibre.utils.icu import sort_key +class OutputFormatSetting(Setting): + + CHOICES_SEARCH_FLAGS = Qt.MatchFixedString + class ConfigWidget(ConfigWidgetBase, Ui_Form): def genesis(self, gui): @@ -43,7 +47,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form): output_formats = list(sorted(available_output_formats())) output_formats.remove('oeb') choices = [(x.upper(), x) for x in output_formats] - r('output_format', prefs, choices=choices) + r('output_format', prefs, choices=choices, setting=OutputFormatSetting) restrictions = sorted(saved_searches().names(), key=sort_key) choices = [('', '')] + [(x, x) for x in restrictions] diff --git a/src/calibre/gui2/preferences/columns.ui b/src/calibre/gui2/preferences/columns.ui index b5dc9b8c90..a9d82530ec 100644 --- a/src/calibre/gui2/preferences/columns.ui +++ b/src/calibre/gui2/preferences/columns.ui @@ -38,6 +38,9 @@ + + Move column up + ... @@ -45,6 +48,12 @@ :/images/arrow-up.png:/images/arrow-up.png + + + 32 + 32 + + @@ -72,6 +81,12 @@ :/images/minus.png:/images/minus.png + + + 32 + 32 + + @@ -99,6 +114,12 @@ :/images/plus.png:/images/plus.png + + + 32 + 32 + + @@ -126,6 +147,12 @@ :/images/edit_input.png:/images/edit_input.png + + + 32 + 32 + + @@ -143,6 +170,9 @@ + + Move column down + ... @@ -150,6 +180,12 @@ :/images/arrow-down.png:/images/arrow-down.png + + + 32 + 32 + + diff --git a/src/calibre/gui2/preferences/create_custom_column.py b/src/calibre/gui2/preferences/create_custom_column.py index 9974de472f..50d567d239 100644 --- a/src/calibre/gui2/preferences/create_custom_column.py +++ b/src/calibre/gui2/preferences/create_custom_column.py @@ -6,7 +6,6 @@ __copyright__ = '2010, Kovid Goyal ' import re from functools import partial -from PyQt4.QtCore import SIGNAL from PyQt4.Qt import QDialog, Qt, QListWidgetItem, QVariant from calibre.gui2.preferences.create_custom_column_ui import Ui_QCreateCustomColumn @@ -48,6 +47,8 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn): QDialog.__init__(self, parent) Ui_QCreateCustomColumn.__init__(self) self.setupUi(self) + self.setWindowTitle(_('Create a custom column')) + self.heading_label.setText(_('Create a custom column')) # Remove help icon on title bar icon = self.windowIcon() self.setWindowFlags(self.windowFlags()&(~Qt.WindowContextHelpButtonHint)) @@ -55,8 +56,21 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn): self.simple_error = partial(error_dialog, self, show=True, show_copy_button=False) - self.connect(self.button_box, SIGNAL("accepted()"), self.accept) - self.connect(self.button_box, SIGNAL("rejected()"), self.reject) + self.button_box.accepted.connect(self.accept) + self.button_box.rejected.connect(self.reject) + self.shortcuts.linkActivated.connect(self.shortcut_activated) + text = '

    '+_('Quick create:') + for col, name in [('isbn', _('ISBN')), ('formats', _('Formats')), + ('last_modified', _('Modified Date')), ('yesno', _('Yes/No')), + ('tags', _('Tags')), ('series', _('Series')), ('rating', + _('Rating'))]: + text += ' %s,'%(col, name) + text = text[:-1] + self.shortcuts.setText(text) + + for sort_by in [_('Text'), _('Number'), _('Date'), _('Yes/No')]: + self.composite_sort_by.addItem(sort_by) + self.parent = parent self.editing_col = editing self.standard_colheads = standard_colheads @@ -69,6 +83,9 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn): self.datatype_changed() self.exec_() return + self.setWindowTitle(_('Edit a custom column')) + self.heading_label.setText(_('Edit a custom column')) + self.shortcuts.setVisible(False) idx = parent.opt_columns.currentRow() if idx < 0: self.simple_error(_('No column selected'), @@ -94,11 +111,47 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn): self.date_format_box.setText(c['display'].get('date_format', '')) elif ct == 'composite': self.composite_box.setText(c['display'].get('composite_template', '')) + sb = c['display'].get('composite_sort', 'text') + vals = ['text', 'number', 'date', 'bool'] + if sb in vals: + sb = vals.index(sb) + else: + sb = 0 + self.composite_sort_by.setCurrentIndex(sb) elif ct == 'enumeration': self.enum_box.setText(','.join(c['display'].get('enum_values', []))) self.datatype_changed() + if ct in ['text', 'composite', 'enumeration']: + self.use_decorations.setChecked(c['display'].get('use_decorations', False)) self.exec_() + def shortcut_activated(self, url): + which = unicode(url).split(':')[-1] + self.column_type_box.setCurrentIndex({ + 'yesno': 9, + 'tags' : 1, + 'series': 3, + 'rating': 8, + }.get(which, 10)) + self.column_name_box.setText(which) + self.column_heading_box.setText({ + 'isbn':'ISBN', + 'formats':_('Formats'), + 'yesno':_('Yes/No'), + 'tags': _('My Tags'), + 'series': _('My Series'), + 'rating': _('My Rating'), + 'last_modified':_('Modified Date')}[which]) + if self.composite_box.isVisible(): + self.composite_box.setText( + { + 'isbn': '{identifiers:select(isbn)}', + 'formats': '{formats}', + 'last_modified':'''{last_modified:'format_date($, "dd MMM yy")'}''' + }[which]) + self.composite_sort_by.setCurrentIndex(2 if which == 'last_modified' else 0) + + def datatype_changed(self, *args): try: col_type = self.column_types[self.column_type_box.currentIndex()]['datatype'] @@ -106,10 +159,11 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn): col_type = None for x in ('box', 'default_label', 'label'): getattr(self, 'date_format_'+x).setVisible(col_type == 'datetime') - for x in ('box', 'default_label', 'label'): + for x in ('box', 'default_label', 'label', 'sort_by', 'sort_by_label'): getattr(self, 'composite_'+x).setVisible(col_type == 'composite') for x in ('box', 'default_label', 'label'): getattr(self, 'enum_'+x).setVisible(col_type == 'enumeration') + self.use_decorations.setVisible(col_type in ['text', 'composite', 'enumeration']) def accept(self): col = unicode(self.column_name_box.text()).strip() @@ -130,10 +184,13 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn): is_multiple = False if not col_heading: return self.simple_error('', _('No column heading was provided')) + + db = self.parent.gui.library_view.model().db + key = db.field_metadata.custom_field_prefix+col bad_col = False - if col in self.parent.custcols: + if key in self.parent.custcols: if not self.editing_col or \ - self.parent.custcols[col]['colnum'] != self.orig_column_number: + self.parent.custcols[key]['colnum'] != self.orig_column_number: bad_col = True if bad_col: return self.simple_error('', _('The lookup name %s is already used')%col) @@ -161,7 +218,10 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn): if not unicode(self.composite_box.text()).strip(): return self.simple_error('', _('You must enter a template for' ' composite columns')) - display_dict = {'composite_template':unicode(self.composite_box.text()).strip()} + display_dict = {'composite_template':unicode(self.composite_box.text()).strip(), + 'composite_sort': ['text', 'number', 'date', 'bool'] + [self.composite_sort_by.currentIndex()] + } elif col_type == 'enumeration': if not unicode(self.enum_box.text()).strip(): return self.simple_error('', _('You must enter at least one' @@ -176,8 +236,9 @@ class CreateCustomColumn(QDialog, Ui_QCreateCustomColumn): 'list more than once').format(l[i])) display_dict = {'enum_values': l} - db = self.parent.gui.library_view.model().db - key = db.field_metadata.custom_field_prefix+col + if col_type in ['text', 'composite', 'enumeration']: + display_dict['use_decorations'] = self.use_decorations.checkState() + if not self.editing_col: db.field_metadata self.parent.custcols[key] = { diff --git a/src/calibre/gui2/preferences/create_custom_column.ui b/src/calibre/gui2/preferences/create_custom_column.ui index d4e85a24c9..aaa69f5e4b 100644 --- a/src/calibre/gui2/preferences/create_custom_column.ui +++ b/src/calibre/gui2/preferences/create_custom_column.ui @@ -9,8 +9,8 @@ 0 0 - 528 - 212 + 603 + 344 @@ -19,19 +19,20 @@ 0 - - Create or edit custom columns + + + :/images/column.png:/images/column.png - + QLayout::SetDefaultConstraint 5 - + 0 @@ -79,7 +80,7 @@ - Column &type + &Column type column_type_box @@ -87,23 +88,58 @@ - - - - 0 - 0 - - - - - 70 - 0 - - - - What kind of information will be kept in the column. - - + + + + + + 0 + 0 + + + + + 70 + 0 + + + + What kind of information will be kept in the column. + + + + + + + Show checkmarks + + + Show check marks in the GUI. Values of 'yes', 'checked', and 'true' +will show a green check. Values of 'no', 'unchecked', and 'false' will show a red X. +Everything else will show nothing. + + + + + + + Qt::Horizontal + + + + 10 + 0 + + + + + 20 + 0 + + + + + @@ -147,6 +183,16 @@ + + + + &Template + + + composite_box + + + @@ -174,16 +220,46 @@ - - + + - &Template + &Sort/search column by - composite_box + composite_sort_by + + + + + + How this column should handled in the GUI when sorting and searching + + + + + + + Qt::Horizontal + + + + 10 + 0 + + + + + 20 + 0 + + + + + + @@ -238,7 +314,7 @@ four values, the first of them being the empty value. - + Qt::Horizontal @@ -252,7 +328,7 @@ four values, the first of them being the empty value. - + 75 @@ -260,7 +336,31 @@ four values, the first of them being the empty value. - Create or edit custom columns + + + + + + + + + + + true + + + + + + + Qt::Vertical + + + + + + + Qt::Vertical @@ -276,6 +376,8 @@ four values, the first of them being the empty value. composite_box button_box - + + + diff --git a/src/calibre/gui2/preferences/look_feel.py b/src/calibre/gui2/preferences/look_feel.py index f7d76f2b70..15d5666978 100644 --- a/src/calibre/gui2/preferences/look_feel.py +++ b/src/calibre/gui2/preferences/look_feel.py @@ -66,7 +66,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form): choices = set([k for k in db.field_metadata.all_field_keys() if db.field_metadata[k]['is_category'] and db.field_metadata[k]['datatype'] in ['text', 'series', 'enumeration']]) - choices -= set(['authors', 'publisher', 'formats', 'news']) + choices -= set(['authors', 'publisher', 'formats', 'news', 'identifiers']) self.opt_categories_using_hierarchy.update_items_cache(choices) r('categories_using_hierarchy', db.prefs, setting=CommaSeparatedList, choices=sorted(list(choices), key=sort_key)) diff --git a/src/calibre/gui2/preferences/plugins.py b/src/calibre/gui2/preferences/plugins.py index acf42fee16..85221766f2 100644 --- a/src/calibre/gui2/preferences/plugins.py +++ b/src/calibre/gui2/preferences/plugins.py @@ -16,9 +16,10 @@ from calibre.customize.ui import initialized_plugins, is_disabled, enable_plugin disable_plugin, plugin_customization, add_plugin, \ remove_plugin from calibre.gui2 import NONE, error_dialog, info_dialog, choose_files, \ - question_dialog + question_dialog, gprefs from calibre.utils.search_query_parser import SearchQueryParser from calibre.utils.icu import lower +from calibre.utils.ordered_dict import OrderedDict class PluginModel(QAbstractItemModel, SearchQueryParser): # {{{ @@ -281,6 +282,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form): self._plugin_model.populate() self._plugin_model.reset() self.changed_signal.emit() + self.check_for_add_to_toolbars(plugin) info_dialog(self, _('Success'), _('Plugin {0} successfully installed under ' ' {1} plugins. You may have to restart calibre ' @@ -342,6 +344,37 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form): plugin.name + _(' cannot be removed. It is a ' 'builtin plugin. Try disabling it instead.')).exec_() + def check_for_add_to_toolbars(self, plugin): + from calibre.gui2.preferences.toolbar import ConfigWidget + from calibre.customize import InterfaceActionBase + + if not isinstance(plugin, InterfaceActionBase): + return + + all_locations = OrderedDict(ConfigWidget.LOCATIONS) + plugin_action = plugin.load_actual_plugin(self.gui) + installed_actions = OrderedDict([ + (key, list(gprefs.get('action-layout-'+key, []))) + for key in all_locations]) + + # If already installed in a GUI container, do nothing + for action_names in installed_actions.itervalues(): + if plugin_action.name in action_names: + return + + allowed_locations = [(key, text) for key, text in + all_locations.iteritems() if key + not in plugin_action.dont_add_to] + if not allowed_locations: + return # This plugin doesn't want to live in the GUI + + from calibre.gui2.dialogs.choose_plugin_toolbars import ChoosePluginToolbarsDialog + d = ChoosePluginToolbarsDialog(self, plugin_action, allowed_locations) + if d.exec_() == d.Accepted: + for key, text in d.selected_locations(): + installed_actions = list(gprefs.get('action-layout-'+key, [])) + installed_actions.append(plugin_action.name) + gprefs['action-layout-'+key] = tuple(installed_actions) if __name__ == '__main__': from PyQt4.Qt import QApplication diff --git a/src/calibre/gui2/search_box.py b/src/calibre/gui2/search_box.py index 5a4c34a5cd..5b8501ebb5 100644 --- a/src/calibre/gui2/search_box.py +++ b/src/calibre/gui2/search_box.py @@ -436,17 +436,18 @@ class SavedSearchBoxMixin(object): # {{{ b = getattr(self, x+'_search_button') b.setStatusTip(b.toolTip()) - def saved_searches_changed(self): + def saved_searches_changed(self, set_restriction=None): p = sorted(saved_searches().names(), key=sort_key) - t = unicode(self.search_restriction.currentText()) + if set_restriction is None: + set_restriction = unicode(self.search_restriction.currentText()) # rebuild the restrictions combobox using current saved searches self.search_restriction.clear() self.search_restriction.addItem('') self.tags_view.recount() for s in p: self.search_restriction.addItem(s) - if t: # redo the search restriction if there was one - self.apply_named_search_restriction(t) + if set_restriction: # redo the search restriction if there was one + self.apply_named_search_restriction(set_restriction) def do_saved_search_edit(self, search): d = SavedSearchEditor(self, search) diff --git a/src/calibre/gui2/tag_view.py b/src/calibre/gui2/tag_view.py index 8abe2d433d..c4871880a4 100644 --- a/src/calibre/gui2/tag_view.py +++ b/src/calibre/gui2/tag_view.py @@ -33,9 +33,6 @@ from calibre.gui2.dialogs.tag_list_editor import TagListEditor from calibre.gui2.dialogs.edit_authors_dialog import EditAuthorsDialog from calibre.gui2.widgets import HistoryLineEdit -def original_name(t): - return getattr(t, 'original_name', t.name) - class TagDelegate(QItemDelegate): # {{{ def paint(self, painter, option, index): @@ -240,16 +237,16 @@ class TagsView(QTreeView): # {{{ tag = index.tag if len(index.children) > 0: for c in index.children: - self.add_item_to_user_cat.emit(category, original_name(c.tag), + self.add_item_to_user_cat.emit(category, c.tag.original_name, c.tag.category) - self.add_item_to_user_cat.emit(category, original_name(tag), + self.add_item_to_user_cat.emit(category, tag.original_name, tag.category) return if action == 'add_subcategory': self.add_subcategory.emit(key) return if action == 'search_category': - self.tags_marked.emit(key + ':' + search_state) + self._toggle(index, set_to=search_state) return if action == 'delete_user_category': self.delete_user_category.emit(key) @@ -258,9 +255,9 @@ class TagsView(QTreeView): # {{{ tag = index.tag if len(index.children) > 0: for c in index.children: - self.del_item_from_user_cat.emit(key, original_name(c.tag), + self.del_item_from_user_cat.emit(key, c.tag.original_name, c.tag.category) - self.del_item_from_user_cat.emit(key, original_name(tag), tag.category) + self.del_item_from_user_cat.emit(key, tag.original_name, tag.category) return if action == 'manage_searches': self.saved_search_edit.emit(category) @@ -323,6 +320,9 @@ class TagsView(QTreeView): # {{{ self.context_menu.addAction(_('Edit sort for %s')%tag.name, partial(self.context_menu_handler, action='edit_author_sort', index=tag.id)) + + # is_editable is also overloaded to mean 'can be added + # to a user category' m = self.context_menu.addMenu(self.user_category_icon, _('Add %s to user category')%tag.name) nt = self.model().category_node_tree @@ -348,7 +348,7 @@ class TagsView(QTreeView): # {{{ partial(self.context_menu_handler, action='delete_item_from_user_category', key = key, index = tag_item)) - # Add the search for value items + # Add the search for value items. All leaf nodes are searchable self.context_menu.addAction(self.search_icon, _('Search for %s')%tag.name, partial(self.context_menu_handler, action='search', @@ -376,7 +376,6 @@ class TagsView(QTreeView): # {{{ action='delete_user_category', key=key)) self.context_menu.addSeparator() # Hide/Show/Restore categories - #if not key.startswith('@') or key.find('.') < 0: self.context_menu.addAction(_('Hide category %s') % category, partial(self.context_menu_handler, action='hide', category=key)) @@ -387,23 +386,28 @@ class TagsView(QTreeView): # {{{ m.addAction(self.db.field_metadata[col]['name'], partial(self.context_menu_handler, action='show', category=col)) - # search by category - if key != 'search': + # search by category. Some categories are not searchable, such + # as search and news + if item.tag.is_searchable: self.context_menu.addAction(self.search_icon, _('Search for books in category %s')%category, - partial(self.context_menu_handler, action='search_category', - key=key, search_state='true')) + partial(self.context_menu_handler, + action='search_category', + index=self._model.createIndex(item.row(), 0, item), + search_state=TAG_SEARCH_STATES['mark_plus'])) self.context_menu.addAction(self.search_icon, _('Search for books not in category %s')%category, - partial(self.context_menu_handler, action='search_category', - key=key, search_state='false')) + partial(self.context_menu_handler, + action='search_category', + index=self._model.createIndex(item.row(), 0, item), + search_state=TAG_SEARCH_STATES['mark_minus'])) # Offer specific editors for tags/series/publishers/saved searches self.context_menu.addSeparator() if key in ['tags', 'publisher', 'series'] or \ self.db.field_metadata[key]['is_custom']: self.context_menu.addAction(_('Manage %s')%category, partial(self.context_menu_handler, action='open_editor', - category=original_name(tag) if tag else None, + category=tag.original_name if tag else None, key=key)) elif key == 'authors': self.context_menu.addAction(_('Manage %s')%category, @@ -562,16 +566,27 @@ class TagTreeItem(object): # {{{ self.bold_font = QVariant(self.bold_font) self.category_key = category_key self.temporary = temporary - self.tag = Tag(data) - self.tag.is_hierarchical = category_key.startswith('@') + self.tag = Tag(data, category=category_key, + is_editable=category_key not in ['news', 'search', 'identifiers'], + is_searchable=category_key not in ['news', 'search']) + elif self.type == self.TAG: - icon_map[0] = data.icon + self.icon_state_map[0] = QVariant(data.icon) self.tag = data if tooltip: self.tooltip = tooltip + ' ' else: self.tooltip = '' + def break_cycles(self): + for x in self.children: + try: + x.break_cycles() + except: + pass + self.parent = self.icon_state_map = self.bold_font = self.tag = \ + self.icon = self.children = None + def __str__(self): if self.type == self.ROOT: return 'ROOT' @@ -623,7 +638,7 @@ class TagTreeItem(object): # {{{ while p.parent.type != self.ROOT: p = p.parent if not tag.is_hierarchical: - name = original_name(tag) + name = tag.original_name else: name = tag.name tt_author = False @@ -635,7 +650,7 @@ class TagTreeItem(object): # {{{ else: return QVariant('[%d] %s'%(count, name)) if role == Qt.EditRole: - return QVariant(original_name(tag)) + return QVariant(tag.original_name) if role == Qt.DecorationRole: return self.icon_state_map[tag.state] if role == Qt.ToolTipRole: @@ -659,7 +674,7 @@ class TagTreeItem(object): # {{{ self.tag.state = (self.tag.state + 1)%5 if self.tag.state == TAG_SEARCH_STATES['mark_plus'] or \ self.tag.state == TAG_SEARCH_STATES['mark_minus']: - if self.tag.is_editable: + if self.tag.is_searchable: break elif self.tag.state == TAG_SEARCH_STATES['mark_plusplus'] or\ self.tag.state == TAG_SEARCH_STATES['mark_minusminus']: @@ -758,6 +773,7 @@ class TagsModel(QAbstractItemModel): # {{{ self.category_nodes.append(node) node.can_be_edited = (not is_gst) and (i == (len(path_parts)-1)) node.is_gst = is_gst + node.tag.is_hierarchical = not is_gst if not is_gst: tree_root[p] = {} tree_root = tree_root[p] @@ -778,6 +794,7 @@ class TagsModel(QAbstractItemModel): # {{{ self.refresh(data=data) def break_cycles(self): + self.root_item.break_cycles() self.db = self.root_item = None def mimeTypes(self): @@ -798,7 +815,7 @@ class TagsModel(QAbstractItemModel): # {{{ p = node while p.type != TagTreeItem.CATEGORY: p = p.parent - d = (node.type, p.category_key, p.is_gst, original_name(t), + d = (node.type, p.category_key, p.is_gst, t.original_name, t.category, path) data.append(d) else: @@ -849,7 +866,7 @@ class TagsModel(QAbstractItemModel): # {{{ Copy/move an item and all its children to the destination ''' copied = False - src_name = original_name(node.tag) + src_name = node.tag.original_name src_cat = node.tag.category # delete the item if the source is a user category and action is move if is_uc and not src_parent_is_gst and src_parent in user_cats and \ @@ -1007,7 +1024,7 @@ class TagsModel(QAbstractItemModel): # {{{ fm = self.db.metadata_for_field(key) is_multiple = fm['is_multiple'] - val = original_name(on_node.tag) + val = on_node.tag.original_name for id in ids: mi = self.db.get_metadata(id, index_is_id=True) @@ -1123,7 +1140,7 @@ class TagsModel(QAbstractItemModel): # {{{ collapse_model = 'partition' collapse_template = tweaks['categories_collapsed_popularity_template'] - def process_one_node(category, state_map): + def process_one_node(category, state_map): # {{{ collapse_letter = None category_index = self.createIndex(category.row(), 0, category) category_node = category_index.internalPointer() @@ -1140,7 +1157,8 @@ class TagsModel(QAbstractItemModel): # {{{ not fm['is_custom'] and \ not fm['kind'] == 'user' \ else False - tt = key if fm['kind'] == 'user' else None + in_uc = fm['kind'] == 'user' + tt = key if in_uc else None if collapse_model == 'first letter': # Build a list of 'equal' first letters by looking for @@ -1215,11 +1233,10 @@ class TagsModel(QAbstractItemModel): # {{{ # category display order is important here. The following works # only of all the non-user categories are displayed before the # user categories - components = [t.strip() for t in original_name(tag).split('.') + components = [t.strip() for t in tag.original_name.split('.') if t.strip()] - if len(components) == 0 or '.'.join(components) != original_name(tag): - components = [original_name(tag)] - in_uc = fm['kind'] == 'user' + if len(components) == 0 or '.'.join(components) != tag.original_name: + components = [tag.original_name] if (not tag.is_hierarchical) and (in_uc or key in ['authors', 'publisher', 'news', 'formats', 'rating'] or key not in self.db.prefs.get('categories_using_hierarchy', []) or @@ -1231,9 +1248,6 @@ class TagsModel(QAbstractItemModel): # {{{ n.id_set |= tag.id_set category_child_map[tag.name, tag.category] = n self.endInsertRows() - tag.is_editable = key != 'formats' and (key == 'news' or \ - self.db.field_metadata[tag.category]['datatype'] in \ - ['text', 'series', 'enumeration']) else: for i,comp in enumerate(components): if i == 0: @@ -1249,12 +1263,13 @@ class TagsModel(QAbstractItemModel): # {{{ if i < len(components)-1: t = copy.copy(tag) t.original_name = '.'.join(components[:i+1]) + # This 'manufactured' intermediate node can + # be searched, but cannot be edited. t.is_editable = False else: t = tag if not in_uc: t.original_name = t.name - t.is_editable = True t.is_hierarchical = True t.name = comp self.beginInsertRows(category_index, 999999, 1) @@ -1265,6 +1280,7 @@ class TagsModel(QAbstractItemModel): # {{{ # This id_set must not be None node_parent.id_set |= tag.id_set return + # }}} for category in self.category_nodes: if len(category.children) > 0: @@ -1330,7 +1346,8 @@ class TagsModel(QAbstractItemModel): # {{{ for c in sorted(user_cats.keys(), key=sort_key): if icu_lower(c).startswith(ckey_lower): if len(c) == len(ckey): - if nkey_lower in user_cat_keys_lower: + if strcmp(ckey, nkey) != 0 and \ + nkey_lower in user_cat_keys_lower: error_dialog(self.tags_view, _('Rename user category'), _('The name %s is already used')%nkey, show=True) return False @@ -1338,7 +1355,8 @@ class TagsModel(QAbstractItemModel): # {{{ del user_cats[ckey] elif c[len(ckey)] == '.': rest = c[len(ckey):] - if icu_lower(nkey + rest) in user_cat_keys_lower: + if strcmp(ckey, nkey) != 0 and \ + icu_lower(nkey + rest) in user_cat_keys_lower: error_dialog(self.tags_view, _('Rename user category'), _('The name %s is already used')%(nkey+rest), show=True) return False @@ -1352,7 +1370,7 @@ class TagsModel(QAbstractItemModel): # {{{ return True key = item.tag.category - name = original_name(item.tag) + name = item.tag.original_name # make certain we know about the item's category if key not in self.db.field_metadata: return False @@ -1576,10 +1594,14 @@ class TagsModel(QAbstractItemModel): # {{{ else: prefix = '' category = tag.category if key != 'news' else 'tag' + add_colon = False + if self.db.field_metadata[tag.category]['is_csp']: + add_colon = True + if tag.name and tag.name[0] == u'\u2605': # char is a star. Assume rating ans.append('%s%s:%s'%(prefix, category, len(tag.name))) else: - name = original_name(tag) + name = tag.original_name use_prefix = tag.state in [TAG_SEARCH_STATES['mark_plusplus'], TAG_SEARCH_STATES['mark_minusminus']] if category == 'tags': @@ -1589,9 +1611,12 @@ class TagsModel(QAbstractItemModel): # {{{ if tag in nodes_seen: continue nodes_seen.add(tag) - ans.append('%s%s:"=%s%s"'%(prefix, category, - '.' if use_prefix else '', - name.replace(r'"', r'\"'))) + n = name.replace(r'"', r'\"') + if name.startswith('.'): + n = '.' + n + ans.append('%s%s:"=%s%s%s"'%(prefix, category, + '.' if use_prefix else '', n, + ':' if add_colon else '')) return ans def find_item_node(self, key, txt, start_path, equals_match=False): @@ -1619,7 +1644,7 @@ class TagsModel(QAbstractItemModel): # {{{ tag = tag_item.tag if tag is None: return False - name = original_name(tag) + name = tag.original_name if (equals_match and strcmp(name, txt) == 0) or \ (not equals_match and lower(name).find(txt) >= 0): self.path_found = path @@ -2061,6 +2086,10 @@ class TagBrowserWidget(QWidget): # {{{ _('Add your own categories to the Tag Browser')) parent.edit_categories.setStatusTip(parent.edit_categories.toolTip()) + # self.leak_test_timer = QTimer(self) + # self.leak_test_timer.timeout.connect(self.test_for_leak) + # self.leak_test_timer.start(5000) + def set_pane_is_visible(self, to_what): self.tags_view.set_pane_is_visible(to_what) @@ -2122,5 +2151,13 @@ class TagBrowserWidget(QWidget): # {{{ def not_found_label_timer_event(self): self.not_found_label.setVisible(False) + def test_for_leak(self): + from calibre.utils.mem import memory + import gc + before = memory() + self.tags_view.recount() + for i in xrange(3): gc.collect() + print 'Used memory:', memory(before)/(1024.), 'KB' + # }}} diff --git a/src/calibre/gui2/ui.py b/src/calibre/gui2/ui.py index 8844446de6..a2ec8c9846 100644 --- a/src/calibre/gui2/ui.py +++ b/src/calibre/gui2/ui.py @@ -296,6 +296,7 @@ class Main(MainWindow, MainWindowMixin, DeviceMixin, EmailMixin, # {{{ traceback.print_exc() if ac.plugin_path is None: raise + self.device_manager.set_current_library_uuid(db.library_id) if show_gui and self.gui_debug is not None: info_dialog(self, _('Debug mode'), '

    ' + @@ -461,6 +462,7 @@ class Main(MainWindow, MainWindowMixin, DeviceMixin, EmailMixin, # {{{ self.memory_view.reset() self.card_a_view.reset() self.card_b_view.reset() + self.device_manager.set_current_library_uuid(db.library_id) def set_window_title(self): diff --git a/src/calibre/gui2/viewer/main.py b/src/calibre/gui2/viewer/main.py index de0f83a5b2..964616ab48 100644 --- a/src/calibre/gui2/viewer/main.py +++ b/src/calibre/gui2/viewer/main.py @@ -17,17 +17,19 @@ from calibre.gui2.viewer.bookmarkmanager import BookmarkManager from calibre.gui2.widgets import ProgressIndicator from calibre.gui2.main_window import MainWindow from calibre.gui2 import Application, ORG_NAME, APP_UID, choose_files, \ - info_dialog, error_dialog, open_url, available_height, gprefs + info_dialog, error_dialog, open_url, available_height from calibre.ebooks.oeb.iterator import EbookIterator from calibre.ebooks import DRMError from calibre.constants import islinux, isfreebsd, isosx, filesystem_encoding -from calibre.utils.config import Config, StringConfig, dynamic +from calibre.utils.config import Config, StringConfig, JSONConfig from calibre.gui2.search_box import SearchBox2 from calibre.ebooks.metadata import MetaInformation from calibre.customize.ui import available_input_formats from calibre.gui2.viewer.dictionary import Lookup from calibre import as_unicode, force_unicode, isbytestring +vprefs = JSONConfig('viewer') + class TOCItem(QStandardItem): def __init__(self, toc): @@ -303,7 +305,7 @@ class EbookViewer(MainWindow, Ui_EbookViewer): m = self.open_history_menu m.clear() count = 0 - for path in gprefs.get('viewer_open_history', []): + for path in vprefs.get('viewer_open_history', []): if count > 9: break if os.path.exists(path): @@ -315,17 +317,17 @@ class EbookViewer(MainWindow, Ui_EbookViewer): return MainWindow.closeEvent(self, e) def save_state(self): - state = str(self.saveState(self.STATE_VERSION)) - dynamic['viewer_toolbar_state'] = state - dynamic.set('viewer_window_geometry', self.saveGeometry()) + state = bytearray(self.saveState(self.STATE_VERSION)) + vprefs['viewer_toolbar_state'] = state + vprefs.set('viewer_window_geometry', bytearray(self.saveGeometry())) if self.current_book_has_toc: - dynamic.set('viewer_toc_isvisible', bool(self.toc.isVisible())) + vprefs.set('viewer_toc_isvisible', bool(self.toc.isVisible())) if self.toc.isVisible(): - dynamic.set('viewer_splitter_state', + vprefs.set('viewer_splitter_state', bytearray(self.splitter.saveState())) def restore_state(self): - state = dynamic.get('viewer_toolbar_state', None) + state = vprefs.get('viewer_toolbar_state', None) if state is not None: try: state = QByteArray(state) @@ -676,13 +678,13 @@ class EbookViewer(MainWindow, Ui_EbookViewer): self.action_table_of_contents.setChecked(False) if isbytestring(pathtoebook): pathtoebook = force_unicode(pathtoebook, filesystem_encoding) - vh = gprefs.get('viewer_open_history', []) + vh = vprefs.get('viewer_open_history', []) try: vh.remove(pathtoebook) except: pass vh.insert(0, pathtoebook) - gprefs.set('viewer_open_history', vh[:50]) + vprefs.set('viewer_open_history', vh[:50]) self.build_recent_menu() self.action_table_of_contents.setDisabled(not self.iterator.toc) @@ -739,13 +741,13 @@ class EbookViewer(MainWindow, Ui_EbookViewer): c = config().parse() self.splitter.setSizes([1, 300]) if c.remember_window_size: - wg = dynamic.get('viewer_window_geometry', None) + wg = vprefs.get('viewer_window_geometry', None) if wg is not None: self.restoreGeometry(wg) - ss = dynamic.get('viewer_splitter_state', None) + ss = vprefs.get('viewer_splitter_state', None) if ss is not None: self.splitter.restoreState(ss) - self.show_toc_on_open = dynamic.get('viewer_toc_isvisible', False) + self.show_toc_on_open = vprefs.get('viewer_toc_isvisible', False) av = available_height() - 30 if self.height() > av: self.resize(self.width(), av) diff --git a/src/calibre/gui2/widgets.py b/src/calibre/gui2/widgets.py index f6c4cce3ef..8ebf9c2c21 100644 --- a/src/calibre/gui2/widgets.py +++ b/src/calibre/gui2/widgets.py @@ -3,7 +3,7 @@ __copyright__ = '2008, Kovid Goyal ' ''' Miscellaneous widgets used in the GUI ''' -import re, os, traceback +import re, traceback from PyQt4.Qt import QIcon, QFont, QLabel, QListWidget, QAction, \ QListWidgetItem, QTextCharFormat, QApplication, \ @@ -11,9 +11,9 @@ from PyQt4.Qt import QIcon, QFont, QLabel, QListWidget, QAction, \ QPixmap, QSplitterHandle, QToolButton, \ QAbstractListModel, QVariant, Qt, SIGNAL, pyqtSignal, \ QRegExp, QSettings, QSize, QSplitter, \ - QPainter, QLineEdit, QComboBox, QPen, \ + QPainter, QLineEdit, QComboBox, QPen, QGraphicsScene, \ QMenu, QStringListModel, QCompleter, QStringList, \ - QTimer, QRect + QTimer, QRect, QFontDatabase, QGraphicsView from calibre.gui2 import NONE, error_dialog, pixmap_to_data, gprefs from calibre.gui2.filename_pattern_ui import Ui_Form @@ -22,6 +22,8 @@ from calibre.ebooks import BOOK_EXTENSIONS from calibre.ebooks.metadata.meta import metadata_from_filename from calibre.utils.config import prefs, XMLConfig, tweaks from calibre.gui2.progress_indicator import ProgressIndicator as _ProgressIndicator +from calibre.gui2.dnd import dnd_has_image, dnd_get_image, dnd_get_files, \ + IMAGE_EXTENSIONS, dnd_has_extension, DownloadDialog history = XMLConfig('history') @@ -141,36 +143,35 @@ class FilenamePattern(QWidget, Ui_Form): return pat -IMAGE_EXTENSIONS = ['jpg', 'jpeg', 'gif', 'png', 'bmp'] - class FormatList(QListWidget): DROPABBLE_EXTENSIONS = BOOK_EXTENSIONS formats_dropped = pyqtSignal(object, object) delete_format = pyqtSignal() - @classmethod - def paths_from_event(cls, event): - ''' - Accept a drop event and return a list of paths that can be read from - and represent files with extensions. - ''' - if event.mimeData().hasFormat('text/uri-list'): - urls = [unicode(u.toLocalFile()) for u in event.mimeData().urls()] - urls = [u for u in urls if os.path.splitext(u)[1] and os.access(u, os.R_OK)] - return [u for u in urls if os.path.splitext(u)[1][1:].lower() in cls.DROPABBLE_EXTENSIONS] - def dragEnterEvent(self, event): - if int(event.possibleActions() & Qt.CopyAction) + \ - int(event.possibleActions() & Qt.MoveAction) == 0: - return - paths = self.paths_from_event(event) - if paths: + md = event.mimeData() + if dnd_has_extension(md, self.DROPABBLE_EXTENSIONS): event.acceptProposedAction() def dropEvent(self, event): - paths = self.paths_from_event(event) event.setDropAction(Qt.CopyAction) - self.formats_dropped.emit(event, paths) + md = event.mimeData() + # Now look for ebook files + urls, filenames = dnd_get_files(md, self.DROPABBLE_EXTENSIONS) + if not urls: + # Nothing found + return + + if not filenames: + # Local files + self.formats_dropped.emit(event, urls) + else: + # Remote files, use the first file + d = DownloadDialog(urls[0], filenames[0], self) + d.start_download() + if d.err is None: + self.formats_dropped.emit(event, [d.fpath]) + def dragMoveEvent(self, event): event.acceptProposedAction() @@ -181,8 +182,81 @@ class FormatList(QListWidget): else: return QListWidget.keyPressEvent(self, event) +class ImageDropMixin(object): # {{{ + ''' + Adds support for dropping images onto widgets and a context menu for + copy/pasting images. + ''' + DROPABBLE_EXTENSIONS = IMAGE_EXTENSIONS -class ImageView(QWidget): + def __init__(self): + self.setAcceptDrops(True) + + def dragEnterEvent(self, event): + md = event.mimeData() + if dnd_has_extension(md, self.DROPABBLE_EXTENSIONS) or \ + dnd_has_image(md): + event.acceptProposedAction() + + def dropEvent(self, event): + event.setDropAction(Qt.CopyAction) + md = event.mimeData() + + x, y = dnd_get_image(md) + if x is not None: + # We have an image, set cover + event.accept() + if y is None: + # Local image + self.handle_image_drop(x) + else: + # Remote files, use the first file + d = DownloadDialog(x, y, self) + d.start_download() + if d.err is None: + pmap = QPixmap() + pmap.loadFromData(open(d.fpath, 'rb').read()) + if not pmap.isNull(): + self.handle_image_drop(pmap) + + def handle_image_drop(self, pmap): + self.set_pixmap(pmap) + self.cover_changed.emit(pixmap_to_data(pmap)) + + def dragMoveEvent(self, event): + event.acceptProposedAction() + + def get_pixmap(self): + return self.pixmap() + + def set_pixmap(self, pmap): + self.setPixmap(pmap) + + def contextMenuEvent(self, ev): + cm = QMenu(self) + paste = cm.addAction(_('Paste Cover')) + copy = cm.addAction(_('Copy Cover')) + if not QApplication.instance().clipboard().mimeData().hasImage(): + paste.setEnabled(False) + copy.triggered.connect(self.copy_to_clipboard) + paste.triggered.connect(self.paste_from_clipboard) + cm.exec_(ev.globalPos()) + + def copy_to_clipboard(self): + QApplication.instance().clipboard().setPixmap(self.get_pixmap()) + + def paste_from_clipboard(self): + cb = QApplication.instance().clipboard() + pmap = cb.pixmap() + if pmap.isNull() and cb.supportsSelection(): + pmap = cb.pixmap(cb.Selection) + if not pmap.isNull(): + self.set_pixmap(pmap) + self.cover_changed.emit( + pixmap_to_data(pmap)) +# }}} + +class ImageView(QWidget, ImageDropMixin): BORDER_WIDTH = 1 cover_changed = pyqtSignal(object) @@ -191,47 +265,9 @@ class ImageView(QWidget): QWidget.__init__(self, parent) self._pixmap = QPixmap(self) self.setMinimumSize(QSize(150, 200)) - self.setAcceptDrops(True) + ImageDropMixin.__init__(self) self.draw_border = True - # Drag 'n drop {{{ - DROPABBLE_EXTENSIONS = IMAGE_EXTENSIONS - - @classmethod - def paths_from_event(cls, event): - ''' - Accept a drop event and return a list of paths that can be read from - and represent files with extensions. - ''' - if event.mimeData().hasFormat('text/uri-list'): - urls = [unicode(u.toLocalFile()) for u in event.mimeData().urls()] - urls = [u for u in urls if os.path.splitext(u)[1] and os.access(u, os.R_OK)] - return [u for u in urls if os.path.splitext(u)[1][1:].lower() in cls.DROPABBLE_EXTENSIONS] - - def dragEnterEvent(self, event): - if int(event.possibleActions() & Qt.CopyAction) + \ - int(event.possibleActions() & Qt.MoveAction) == 0: - return - paths = self.paths_from_event(event) - if paths: - event.acceptProposedAction() - - def dropEvent(self, event): - paths = self.paths_from_event(event) - event.setDropAction(Qt.CopyAction) - for path in paths: - pmap = QPixmap() - pmap.load(path) - if not pmap.isNull(): - self.setPixmap(pmap) - event.accept() - self.cover_changed.emit(open(path, 'rb').read()) - break - - def dragMoveEvent(self, event): - event.acceptProposedAction() - # }}} - def setPixmap(self, pixmap): if not isinstance(pixmap, QPixmap): raise TypeError('Must use a QPixmap') @@ -272,34 +308,23 @@ class ImageView(QWidget): p.drawRect(target) p.end() +class CoverView(QGraphicsView, ImageDropMixin): - # Clipboard copy/paste # {{{ - def contextMenuEvent(self, ev): - cm = QMenu(self) - copy = cm.addAction(_('Copy Image')) - paste = cm.addAction(_('Paste Image')) - if not QApplication.instance().clipboard().mimeData().hasImage(): - paste.setEnabled(False) - copy.triggered.connect(self.copy_to_clipboard) - paste.triggered.connect(self.paste_from_clipboard) - cm.exec_(ev.globalPos()) - - def copy_to_clipboard(self): - QApplication.instance().clipboard().setPixmap(self.pixmap()) - - def paste_from_clipboard(self): - cb = QApplication.instance().clipboard() - pmap = cb.pixmap() - if pmap.isNull() and cb.supportsSelection(): - pmap = cb.pixmap(cb.Selection) - if not pmap.isNull(): - self.setPixmap(pmap) - self.cover_changed.emit( - pixmap_to_data(pmap)) - # }}} + cover_changed = pyqtSignal(object) + def __init__(self, *args, **kwargs): + QGraphicsView.__init__(self, *args, **kwargs) + ImageDropMixin.__init__(self) + def get_pixmap(self): + for item in self.scene().items(): + if hasattr(item, 'pixmap'): + return item.pixmap() + def set_pixmap(self, pmap): + self.scene = QGraphicsScene() + self.scene.addPixmap(pmap) + self.setScene(self.scene) class FontFamilyModel(QAbstractListModel): @@ -312,6 +337,9 @@ class FontFamilyModel(QAbstractListModel): self.families = [] print 'WARNING: Could not load fonts' traceback.print_exc() + # Restrict to Qt families as Qt tends to crash + qt_families = set([unicode(x) for x in QFontDatabase().families()]) + self.families = list(qt_families.intersection(set(self.families))) self.families.sort() self.families[:0] = [_('None')] diff --git a/src/calibre/gui2/wizard/__init__.py b/src/calibre/gui2/wizard/__init__.py index 5f9f1828fa..c629b10b5d 100644 --- a/src/calibre/gui2/wizard/__init__.py +++ b/src/calibre/gui2/wizard/__init__.py @@ -51,7 +51,7 @@ class Device(object): @classmethod def set_output_format(cls): if cls.output_format: - prefs.set('output_format', cls.output_format) + prefs.set('output_format', cls.output_format.lower()) @classmethod def commit(cls): diff --git a/src/calibre/library/caches.py b/src/calibre/library/caches.py index 0335c1d280..97ddaeb51a 100644 --- a/src/calibre/library/caches.py +++ b/src/calibre/library/caches.py @@ -7,7 +7,7 @@ __copyright__ = '2010, Kovid Goyal ' __docformat__ = 'restructuredtext en' import re, itertools, time, traceback -from itertools import repeat +from itertools import repeat, izip, imap from datetime import timedelta from threading import Thread @@ -121,11 +121,16 @@ CONTAINS_MATCH = 0 EQUALS_MATCH = 1 REGEXP_MATCH = 2 def _match(query, value, matchkind): + if query.startswith('..'): + query = query[1:] + prefix_match_ok = False + else: + prefix_match_ok = True for t in value: t = icu_lower(t) try: ### ignore regexp exceptions, required because search-ahead tries before typing is finished if (matchkind == EQUALS_MATCH): - if query[0] == '.': + if prefix_match_ok and query[0] == '.': if t.startswith(query[1:]): ql = len(query) - 1 if (len(t) == ql) or (t[ql:ql+1] == '.'): @@ -139,6 +144,22 @@ def _match(query, value, matchkind): pass return False +def force_to_bool(val): + if isinstance(val, (str, unicode)): + try: + val = icu_lower(val) + if not val: + val = None + elif val in [_('yes'), _('checked'), 'true']: + val = True + elif val in [_('no'), _('unchecked'), 'false']: + val = False + else: + val = bool(int(val)) + except: + val = None + return val + class CacheRow(list): # {{{ def __init__(self, db, composites, val): @@ -189,6 +210,7 @@ class ResultCache(SearchQueryParser): # {{{ self.first_sort = True self.search_restriction = '' self.search_restriction_book_count = 0 + self.marked_ids_dict = {} self.field_metadata = field_metadata self.all_search_locations = field_metadata.get_search_terms() SearchQueryParser.__init__(self, self.all_search_locations, optimize=True) @@ -297,14 +319,20 @@ class ResultCache(SearchQueryParser): # {{{ for id_ in candidates: item = self._data[id_] if item is None: continue - if item[loc] is None or item[loc] <= UNDEFINED_DATE: + v = item[loc] + if isinstance(v, (str, unicode)): + v = parse_date(v) + if v is None or v <= UNDEFINED_DATE: matches.add(item[0]) return matches if query == 'true': for id_ in candidates: item = self._data[id_] if item is None: continue - if item[loc] is not None and item[loc] > UNDEFINED_DATE: + v = item[loc] + if isinstance(v, (str, unicode)): + v = parse_date(v) + if v is not None and v > UNDEFINED_DATE: matches.add(item[0]) return matches @@ -344,7 +372,10 @@ class ResultCache(SearchQueryParser): # {{{ for id_ in candidates: item = self._data[id_] if item is None or item[loc] is None: continue - if relop(item[loc], qd, field_count): + v = item[loc] + if isinstance(v, (str, unicode)): + v = parse_date(v) + if relop(v, qd, field_count): matches.add(item[0]) return matches @@ -385,7 +416,7 @@ class ResultCache(SearchQueryParser): # {{{ elif dt == 'rating': cast = (lambda x: int (x)) adjust = lambda x: x/2 - elif dt == 'float': + elif dt in ('float', 'composite'): cast = lambda x : float (x) adjust = lambda x: x else: # count operation @@ -408,19 +439,22 @@ class ResultCache(SearchQueryParser): # {{{ item = self._data[id_] if item is None: continue - v = val_func(item) + try: + v = cast(val_func(item)) + except: + v = 0 if not v: - i = 0 + v = 0 else: - i = adjust(v) - if relop(i, q): + v = adjust(v) + if relop(v, q): matches.add(item[0]) return matches def get_user_category_matches(self, location, query, candidates): - res = set([]) + matches = set([]) if self.db_prefs is None or len(query) < 2: - return res + return matches user_cats = self.db_prefs.get('user_categories', []) c = set(candidates) @@ -435,10 +469,104 @@ class ResultCache(SearchQueryParser): # {{{ for (item, category, ign) in user_cats[key]: s = self.get_matches(category, '=' + item, candidates=c) c -= s - res |= s + matches |= s if query == 'false': - return candidates - res - return res + return candidates - matches + return matches + + def get_keypair_matches(self, location, query, candidates): + matches = set([]) + if query.find(':') >= 0: + q = [q.strip() for q in query.split(':')] + if len(q) != 2: + raise ParseException(query, len(query), + 'Invalid query format for colon-separated search', self) + (keyq, valq) = q + keyq_mkind, keyq = self._matchkind(keyq) + valq_mkind, valq = self._matchkind(valq) + else: + keyq = keyq_mkind = '' + valq_mkind, valq = self._matchkind(query) + + loc = self.field_metadata[location]['rec_index'] + split_char = self.field_metadata[location]['is_multiple'] + for id_ in candidates: + item = self._data[id_] + if item is None: + continue + + if item[loc] is None: + if valq == 'false': + matches.add(id_) + continue + + pairs = [p.strip() for p in item[loc].split(split_char)] + for pair in pairs: + parts = pair.split(':') + if len(parts) != 2: + continue + k = parts[:1] + v = parts[1:] + if keyq and not _match(keyq, k, keyq_mkind): + continue + if valq: + if valq == 'true': + if not v: + continue + elif valq == 'false': + if v: + continue + elif not _match(valq, v, valq_mkind): + continue + matches.add(id_) + return matches + + def _matchkind(self, query): + matchkind = CONTAINS_MATCH + if (len(query) > 1): + if query.startswith('\\'): + query = query[1:] + elif query.startswith('='): + matchkind = EQUALS_MATCH + query = query[1:] + elif query.startswith('~'): + matchkind = REGEXP_MATCH + query = query[1:] + + if matchkind != REGEXP_MATCH: + # leave case in regexps because it can be significant e.g. \S \W \D + query = icu_lower(query) + return matchkind, query + + def get_bool_matches(self, location, query, candidates): + bools_are_tristate = tweaks['bool_custom_columns_are_tristate'] != 'no' + loc = self.field_metadata[location]['rec_index'] + matches = set() + query = icu_lower(query) + for id_ in candidates: + item = self._data[id_] + if item is None: + continue + + val = force_to_bool(item[loc]) + if not bools_are_tristate: + if val is None or not val: # item is None or set to false + if query in [_('no'), _('unchecked'), '_no', 'false']: + matches.add(item[0]) + else: # item is explicitly set to true + if query in [_('yes'), _('checked'), '_yes', 'true']: + matches.add(item[0]) + else: + if val is None: + if query in [_('empty'), _('blank'), '_empty', 'false']: + matches.add(item[0]) + elif not val: # is not None and false + if query in [_('no'), _('unchecked'), '_no', 'true']: + matches.add(item[0]) + else: # item is not None and true + if query in [_('yes'), _('checked'), '_yes', 'true']: + matches.add(item[0]) + return matches def get_matches(self, location, query, candidates=None, allow_recursion=True): @@ -455,6 +583,7 @@ class ResultCache(SearchQueryParser): # {{{ if query and query.strip(): # get metadata key associated with the search term. Eliminates # dealing with plurals and other aliases + original_location = location location = self.field_metadata.search_term_to_field_key(icu_lower(location.strip())) # grouped search terms if isinstance(location, list): @@ -489,13 +618,20 @@ class ResultCache(SearchQueryParser): # {{{ if location in self.field_metadata: fm = self.field_metadata[location] # take care of dates special case - if fm['datatype'] == 'datetime': + if fm['datatype'] == 'datetime' or \ + (fm['datatype'] == 'composite' and + fm['display'].get('composite_sort', '') == 'date'): return self.get_dates_matches(location, query.lower(), candidates) # take care of numbers special case - if fm['datatype'] in ('rating', 'int', 'float'): + if fm['datatype'] in ('rating', 'int', 'float') or \ + (fm['datatype'] == 'composite' and + fm['display'].get('composite_sort', '') == 'number'): return self.get_numeric_matches(location, query.lower(), candidates) + if fm['datatype'] == 'bool': + return self.get_bool_matches(location, query, candidates) + # take care of the 'count' operator for is_multiples if fm['is_multiple'] and \ len(query) > 1 and query.startswith('#') and \ @@ -505,24 +641,20 @@ class ResultCache(SearchQueryParser): # {{{ return self.get_numeric_matches(location, query[1:], candidates, val_func=vf) + # special case: colon-separated fields such as identifiers. isbn + # is a special case within the case + if fm.get('is_csp', False): + if location == 'identifiers' and original_location == 'isbn': + return self.get_keypair_matches('identifiers', + '=isbn:'+query, candidates) + return self.get_keypair_matches(location, query, candidates) + # check for user categories if len(location) >= 2 and location.startswith('@'): return self.get_user_category_matches(location[1:], query.lower(), candidates) # everything else, or 'all' matches - matchkind = CONTAINS_MATCH - if (len(query) > 1): - if query.startswith('\\'): - query = query[1:] - elif query.startswith('='): - matchkind = EQUALS_MATCH - query = query[1:] - elif query.startswith('~'): - matchkind = REGEXP_MATCH - query = query[1:] - if matchkind != REGEXP_MATCH: - # leave case in regexps because it can be significant e.g. \S \W \D - query = icu_lower(query) + matchkind, query = self._matchkind(query) if not isinstance(query, unicode): query = query.decode('utf-8') @@ -553,9 +685,6 @@ class ResultCache(SearchQueryParser): # {{{ for i, loc in enumerate(location): location[i] = db_col[loc] - # get the tweak here so that the string lookup and compare aren't in the loop - bools_are_tristate = tweaks['bool_custom_columns_are_tristate'] != 'no' - for loc in location: # location is now an array of field indices if loc == db_col['authors']: ### DB stores authors with commas changed to bars, so change query @@ -567,35 +696,15 @@ class ResultCache(SearchQueryParser): # {{{ item = self._data[id_] if item is None: continue - if col_datatype[loc] == 'bool': # complexity caused by the two-/three-value tweak - v = item[loc] - if not bools_are_tristate: - if v is None or not v: # item is None or set to false - if q in [_('no'), _('unchecked'), 'false']: - matches.add(item[0]) - else: # item is explicitly set to true - if q in [_('yes'), _('checked'), 'true']: - matches.add(item[0]) - else: - if v is None: - if q in [_('empty'), _('blank'), 'false']: - matches.add(item[0]) - elif not v: # is not None and false - if q in [_('no'), _('unchecked'), 'true']: - matches.add(item[0]) - else: # item is not None and true - if q in [_('yes'), _('checked'), 'true']: - matches.add(item[0]) - continue - if not item[loc]: - if q == 'false': + if q == 'false' and matchkind == CONTAINS_MATCH: matches.add(item[0]) continue # item is empty. No possible matches below - if q == 'false': # Field has something in it, so a false query does not match + if q == 'false'and matchkind == CONTAINS_MATCH: + # Field has something in it, so a false query does not match continue - if q == 'true': + if q == 'true' and matchkind == CONTAINS_MATCH: if isinstance(item[loc], basestring): if item[loc].strip() == '': continue @@ -670,6 +779,36 @@ class ResultCache(SearchQueryParser): # {{{ def get_search_restriction_book_count(self): return self.search_restriction_book_count + def set_marked_ids(self, id_dict): + ''' + ids in id_dict are "marked". They can be searched for by + using the search term ``marked:true``. Pass in an empty dictionary or + set to clear marked ids. + + :param id_dict: Either a dictionary mapping ids to values or a set + of ids. In the latter case, the value is set to 'true' for all ids. If + a mapping is provided, then the search can be used to search for + particular values: ``marked:value`` + ''' + if not hasattr(id_dict, 'items'): + # Simple list. Make it a dict of string 'true' + self.marked_ids_dict = dict.fromkeys(id_dict, u'true') + else: + # Ensure that all the items in the dict are text + self.marked_ids_dict = dict(izip(id_dict.iterkeys(), imap(unicode, + id_dict.itervalues()))) + + # Set the values in the cache + marked_col = self.FIELD_MAP['marked'] + for r in self.iterall(): + r[marked_col] = None + + for id_, val in self.marked_ids_dict.iteritems(): + try: + self._data[id_][marked_col] = val + except: + pass + # }}} def remove(self, id): @@ -719,6 +858,7 @@ class ResultCache(SearchQueryParser): # {{{ self._data[id] = CacheRow(db, self.composites, db.conn.get('SELECT * from meta2 WHERE id=?', (id,))[0]) self._data[id].append(db.book_on_device_string(id)) + self._data[id].append(self.marked_ids_dict.get(id, None)) except IndexError: return None try: @@ -735,6 +875,7 @@ class ResultCache(SearchQueryParser): # {{{ self._data[id] = CacheRow(db, self.composites, db.conn.get('SELECT * from meta2 WHERE id=?', (id,))[0]) self._data[id].append(db.book_on_device_string(id)) + self._data[id].append(self.marked_ids_dict.get(id, None)) self._map[0:0] = ids self._map_filtered[0:0] = ids @@ -759,6 +900,15 @@ class ResultCache(SearchQueryParser): # {{{ for item in self._data: if item is not None: item.append(db.book_on_device_string(item[0])) + item.append(None) + + marked_col = self.FIELD_MAP['marked'] + for id_,val in self.marked_ids_dict.iteritems(): + try: + self._data[id_][marked_col] = val + except: + pass + self._map = [i[0] for i in self._data if i is not None] if field is not None: self.sort(field, ascending) @@ -827,6 +977,23 @@ class SortKeyGenerator(object): for name, fm in self.entries: dt = fm['datatype'] val = record[fm['rec_index']] + if dt == 'composite': + sb = fm['display'].get('composite_sort', 'text') + if sb == 'date': + try: + val = parse_date(val) + dt = 'datetime' + except: + pass + elif sb == 'number': + try: + val = float(val) + except: + val = 0.0 + dt = 'float' + elif sb == 'bool': + val = force_to_bool(val) + dt = 'bool' if dt == 'datetime': if val is None: diff --git a/src/calibre/library/catalog.py b/src/calibre/library/catalog.py index f3640af4f0..ffa08eaed2 100644 --- a/src/calibre/library/catalog.py +++ b/src/calibre/library/catalog.py @@ -5103,6 +5103,19 @@ Author '{0}': recommendations.append(('book_producer',opts.output_profile, OptionRecommendation.HIGH)) + # If cover exists, use it + try: + search_text = 'title:"%s" author:%s' % ( + opts.catalog_title.replace('"', '\\"'), 'calibre') + matches = db.search(search_text, return_matches=True) + if matches: + cpath = db.cover(matches[0], index_is_id=True, as_path=True) + if cpath and os.path.exists(cpath): + recommendations.append(('cover', cpath, + OptionRecommendation.HIGH)) + except: + pass + # Run ebook-convert from calibre.ebooks.conversion.plumber import Plumber plumber = Plumber(os.path.join(catalog.catalogPath, diff --git a/src/calibre/library/cli.py b/src/calibre/library/cli.py index e93be187f9..f062aecc26 100644 --- a/src/calibre/library/cli.py +++ b/src/calibre/library/cli.py @@ -20,7 +20,8 @@ from calibre.utils.date import isoformat FIELDS = set(['title', 'authors', 'author_sort', 'publisher', 'rating', 'timestamp', 'size', 'tags', 'comments', 'series', 'series_index', - 'formats', 'isbn', 'uuid', 'pubdate', 'cover']) + 'formats', 'isbn', 'uuid', 'pubdate', 'cover', 'last_modified', + 'identifiers']) def send_message(msg=''): prints('Notifying calibre of the change') diff --git a/src/calibre/library/custom_columns.py b/src/calibre/library/custom_columns.py index 358daf9de6..dec55f2b02 100644 --- a/src/calibre/library/custom_columns.py +++ b/src/calibre/library/custom_columns.py @@ -188,7 +188,7 @@ class CustomColumns(object): table=tn, column='value', datatype=v['datatype'], colnum=v['num'], name=v['name'], display=v['display'], is_multiple=is_m, is_category=is_category, - is_editable=v['editable']) + is_editable=v['editable'], is_csp=False) def get_custom(self, idx, label=None, num=None, index_is_id=False): if label is not None: diff --git a/src/calibre/library/database2.py b/src/calibre/library/database2.py index bf3e9c8a14..b506e7e82d 100644 --- a/src/calibre/library/database2.py +++ b/src/calibre/library/database2.py @@ -6,7 +6,8 @@ __docformat__ = 'restructuredtext en' ''' The database used to store ebook metadata ''' -import os, sys, shutil, cStringIO, glob, time, functools, traceback, re, json +import os, sys, shutil, cStringIO, glob, time, functools, traceback, re, \ + json, uuid import threading, random from itertools import repeat from math import ceil @@ -46,13 +47,15 @@ copyfile = os.link if hasattr(os, 'link') else shutil.copyfile class Tag(object): def __init__(self, name, id=None, count=0, state=0, avg=0, sort=None, - tooltip=None, icon=None, category=None, id_set=None): - self.name = name + tooltip=None, icon=None, category=None, id_set=None, + is_editable = True, is_searchable=True): + self.name = self.original_name = name self.id = id self.count = count self.state = state self.is_hierarchical = False - self.is_editable = True + self.is_editable = is_editable + self.is_searchable = is_searchable self.id_set = id_set self.avg_rating = avg/2.0 if avg is not None else 0 self.sort = sort @@ -94,6 +97,31 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): return property(doc=doc, fget=fget, fset=fset) + @dynamic_property + def library_id(self): + doc = ('The UUID for this library. As long as the user only operates' + ' on libraries with calibre, it will be unique') + + def fget(self): + if self._library_id_ is None: + ans = self.conn.get('SELECT uuid FROM library_id', all=False) + if ans is None: + ans = str(uuid.uuid4()) + self.library_id = ans + else: + self._library_id_ = ans + return self._library_id_ + + def fset(self, val): + self._library_id_ = unicode(val) + self.conn.executescript(''' + DELETE FROM library_id; + INSERT INTO library_id (uuid) VALUES ("%s"); + '''%self._library_id_) + self.conn.commit() + + return property(doc=doc, fget=fget, fset=fset) + def connect(self): if 'win32' in sys.platform and len(self.library_path) + 4*self.PATH_LIMIT + 10 > 259: raise ValueError('Path to library too long. Must be less than %d characters.'%(259-4*self.PATH_LIMIT-10)) @@ -120,6 +148,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): def __init__(self, library_path, row_factory=False, default_prefs=None, read_only=False): self.field_metadata = FieldMetadata() + self._library_id_ = None # Create the lock to be used to guard access to the metadata writer # queues. This must be an RLock, not a Lock self.dirtied_lock = threading.RLock() @@ -148,6 +177,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): self.is_case_sensitive = not iswindows and not isosx and \ not os.path.exists(self.dbpath.replace('metadata.db', 'MeTAdAtA.dB')) SchemaUpgrade.__init__(self) + # Guarantee that the library_id is set + self.library_id # if we are to copy the prefs and structure from some other DB, then # we need to do it before we call initialize_dynamic @@ -293,14 +324,14 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): 'sort', 'author_sort', '(SELECT group_concat(format) FROM data WHERE data.book=books.id) formats', - 'isbn', 'path', - 'lccn', 'pubdate', - 'flags', 'uuid', 'has_cover', - ('au_map', 'authors', 'author', 'aum_sortconcat(link.id, authors.name, authors.sort)') + ('au_map', 'authors', 'author', + 'aum_sortconcat(link.id, authors.name, authors.sort)'), + 'last_modified', + '(SELECT identifiers_concat(type, val) FROM identifiers WHERE identifiers.book=books.id) identifiers', ] lines = [] for col in columns: @@ -318,8 +349,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): self.FIELD_MAP = {'id':0, 'title':1, 'authors':2, 'timestamp':3, 'size':4, 'rating':5, 'tags':6, 'comments':7, 'series':8, 'publisher':9, 'series_index':10, 'sort':11, 'author_sort':12, - 'formats':13, 'isbn':14, 'path':15, 'lccn':16, 'pubdate':17, - 'flags':18, 'uuid':19, 'cover':20, 'au_map':21} + 'formats':13, 'path':14, 'pubdate':15, 'uuid':16, 'cover':17, + 'au_map':18, 'last_modified':19, 'identifiers':20} for k,v in self.FIELD_MAP.iteritems(): self.field_metadata.set_field_record_index(k, v, prefer_custom=False) @@ -343,6 +374,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): self.FIELD_MAP['ondevice'] = base = base+1 self.field_metadata.set_field_record_index('ondevice', base, prefer_custom=False) + self.FIELD_MAP['marked'] = base = base+1 + self.field_metadata.set_field_record_index('marked', base, prefer_custom=False) script = ''' DROP VIEW IF EXISTS meta2; @@ -390,12 +423,18 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): self.row = self.data.row self.has_id = self.data.has_id self.count = self.data.count + self.set_marked_ids = self.data.set_marked_ids - for prop in ('author_sort', 'authors', 'comment', 'comments', 'isbn', - 'publisher', 'rating', 'series', 'series_index', 'tags', - 'title', 'timestamp', 'uuid', 'pubdate', 'ondevice'): + for prop in ( + 'author_sort', 'authors', 'comment', 'comments', + 'publisher', 'rating', 'series', 'series_index', 'tags', + 'title', 'timestamp', 'uuid', 'pubdate', 'ondevice', + 'metadata_last_modified', + ): + fm = {'comment':'comments', 'metadata_last_modified': + 'last_modified'}.get(prop, prop) setattr(self, prop, functools.partial(self.get_property, - loc=self.FIELD_MAP['comments' if prop == 'comment' else prop])) + loc=self.FIELD_MAP[fm])) setattr(self, 'title_sort', functools.partial(self.get_property, loc=self.FIELD_MAP['sort'])) @@ -681,8 +720,20 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): if commit: self.conn.commit() + def update_last_modified(self, book_ids, commit=False, now=None): + if now is None: + now = nowf() + if book_ids: + self.conn.executemany( + 'UPDATE books SET last_modified=? WHERE id=?', + [(now, book) for book in book_ids]) + for book_id in book_ids: + self.data.set(book_id, self.FIELD_MAP['last_modified'], now, row_is_id=True) + if commit: + self.conn.commit() + def dirtied(self, book_ids, commit=True): - changed = False + self.update_last_modified(book_ids) for book in book_ids: with self.dirtied_lock: # print 'dirtied: check id', book @@ -691,21 +742,18 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): self.dirtied_sequence += 1 continue # print 'book not already dirty' - try: - self.conn.execute( - 'INSERT INTO metadata_dirtied (book) VALUES (?)', - (book,)) - changed = True - except IntegrityError: - # Already in table - pass + + self.conn.execute( + 'INSERT OR IGNORE INTO metadata_dirtied (book) VALUES (?)', + (book,)) self.dirtied_cache[book] = self.dirtied_sequence self.dirtied_sequence += 1 + # If the commit doesn't happen, then the DB table will be wrong. This # could lead to a problem because on restart, we won't put the book back # into the dirtied_cache. We deal with this by writing the dirtied_cache # back to the table on GUI exit. Not perfect, but probably OK - if commit and changed: + if book_ids and commit: self.conn.commit() def get_a_dirtied_book(self): @@ -790,6 +838,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): mi.pubdate = row[fm['pubdate']] mi.uuid = row[fm['uuid']] mi.title_sort = row[fm['sort']] + mi.last_modified = row[fm['last_modified']] formats = row[fm['formats']] if not formats: formats = None @@ -803,8 +852,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): if mi.series: mi.series_index = row[fm['series_index']] mi.rating = row[fm['rating']] - mi.isbn = row[fm['isbn']] id = idx if index_is_id else self.id(idx) + mi.set_identifiers(self.get_identifiers(id, index_is_id=True)) mi.application_id = id mi.id = id for key, meta in self.field_metadata.custom_iteritems(): @@ -911,10 +960,14 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): except (IOError, OSError): time.sleep(0.2) save_cover_data_to(data, path) - self.conn.execute('UPDATE books SET has_cover=1 WHERE id=?', (id,)) + now = nowf() + self.conn.execute( + 'UPDATE books SET has_cover=1,last_modified=? WHERE id=?', + (now, id)) if commit: self.conn.commit() self.data.set(id, self.FIELD_MAP['cover'], True, row_is_id=True) + self.data.set(id, self.FIELD_MAP['last_modified'], now, row_is_id=True) if notify: self.notify('cover', [id]) @@ -923,8 +976,12 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): def set_has_cover(self, id, val): dval = 1 if val else 0 - self.conn.execute('UPDATE books SET has_cover=? WHERE id=?', (dval, id,)) + now = nowf() + self.conn.execute( + 'UPDATE books SET has_cover=?,last_modified=? WHERE id=?', + (dval, now, id)) self.data.set(id, self.FIELD_MAP['cover'], val, row_is_id=True) + self.data.set(id, self.FIELD_MAP['last_modified'], now, row_is_id=True) def book_on_device(self, id): if callable(self.book_on_device_func): @@ -1135,12 +1192,6 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): self.clean_custom() self.conn.commit() - def get_recipes(self): - return self.conn.get('SELECT id, script FROM feeds') - - def get_recipe(self, id): - return self.conn.get('SELECT script FROM feeds WHERE id=?', (id,), all=False) - def get_books_for_category(self, category, id_): ans = set([]) @@ -1195,7 +1246,11 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): i += 1 else: new_cats['.'.join(comps)] = user_cats[k] - self.prefs.set('user_categories', new_cats) + try: + if new_cats != user_cats: + self.prefs.set('user_categories', new_cats) + except: + pass return new_cats def get_categories(self, sort='name', ids=None, icon_map=None): @@ -1218,7 +1273,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): for category in tb_cats.keys(): cat = tb_cats[category] if not cat['is_category'] or cat['kind'] in ['user', 'search'] \ - or category in ['news', 'formats']: + or category in ['news', 'formats'] or cat.get('is_csp', + False): continue # Get the ids for the item values if not cat['is_custom']: @@ -1388,10 +1444,11 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): reverse=True items.sort(key=kf, reverse=reverse) + is_editable = category not in ['news', 'rating'] categories[category] = [tag_class(formatter(r.n), count=r.c, id=r.id, avg=avgr(r), sort=r.s, icon=icon, tooltip=tooltip, category=category, - id_set=r.id_set) + id_set=r.id_set, is_editable=is_editable) for r in items] #print 'end phase "tags list":', time.clock() - last, 'seconds' @@ -1428,7 +1485,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): all=False) if count > 0: categories['formats'].append(Tag(fmt, count=count, icon=icon, - category='formats')) + category='formats', is_editable=False)) if sort == 'popularity': categories['formats'].sort(key=lambda x: x.count, reverse=True) @@ -1436,6 +1493,35 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): # No need for ICU here. categories['formats'].sort(key = lambda x:x.name) + # Now do identifiers. This works like formats + categories['identifiers'] = [] + icon = None + if icon_map and 'identifiers' in icon_map: + icon = icon_map['identifiers'] + for ident in self.conn.get('SELECT DISTINCT type FROM identifiers'): + ident = ident[0] + if ids is not None: + count = self.conn.get('''SELECT COUNT(book) + FROM identifiers + WHERE type="%s" AND + books_list_filter(book)'''%ident, + all=False) + else: + count = self.conn.get('''SELECT COUNT(id) + FROM identifiers + WHERE type="%s"'''%ident, + all=False) + if count > 0: + categories['identifiers'].append(Tag(ident, count=count, icon=icon, + category='identifiers', + is_editable=False)) + + if sort == 'popularity': + categories['identifiers'].sort(key=lambda x: x.count, reverse=True) + else: # no ratings exist to sort on + # No need for ICU here. + categories['identifiers'].sort(key = lambda x:x.name) + #### Now do the user-defined categories. #### user_categories = dict.copy(self.clean_user_categories()) @@ -1487,7 +1573,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): icon = icon_map['search'] for srch in saved_searches().names(): items.append(Tag(srch, tooltip=saved_searches().lookup(srch), - sort=srch, icon=icon, category='search')) + sort=srch, icon=icon, category='search', + is_editable=False)) if len(items): if icon_map is not None: icon_map['search'] = icon_map['search'] @@ -1500,25 +1587,30 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): ############# End get_categories - def tags_older_than(self, tag, delta): + def tags_older_than(self, tag, delta, must_have_tag=None): ''' Return the ids of all books having the tag ``tag`` that are older than than the specified time. tag comparison is case insensitive. :param delta: A timedelta object or None. If None, then all ids with the tag are returned. + :param must_have_tag: If not None the list of matches will be + restricted to books that have this tag ''' tag = tag.lower().strip() + mht = must_have_tag.lower().strip() if must_have_tag else None now = nowf() tindex = self.FIELD_MAP['timestamp'] gindex = self.FIELD_MAP['tags'] + iindex = self.FIELD_MAP['id'] for r in self.data._data: if r is not None: if delta is None or (now - r[tindex]) > delta: tags = r[gindex] - if tags and tag in [x.strip() for x in - tags.lower().split(',')]: - yield r[self.FIELD_MAP['id']] + if tags: + tags = [x.strip() for x in tags.lower().split(',')] + if tag in tags and (mht is None or mht in tags): + yield r[iindex] def get_next_series_num_for(self, series): series_id = self.conn.get('SELECT id from series WHERE name=?', @@ -1643,8 +1735,6 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): doit(self.set_tags, id, mi.tags, notify=False, commit=False) if mi.comments: doit(self.set_comment, id, mi.comments, notify=False, commit=False) - if mi.isbn and mi.isbn.strip(): - doit(self.set_isbn, id, mi.isbn, notify=False, commit=False) if mi.series_index: doit(self.set_series_index, id, mi.series_index, notify=False, commit=False) @@ -1654,6 +1744,15 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): doit(self.set_timestamp, id, mi.timestamp, notify=False, commit=False) + mi_idents = mi.get_identifiers() + if mi_idents: + identifiers = self.get_identifiers(id, index_is_id=True) + for key, val in mi_idents.iteritems(): + if val and val.strip(): # Don't delete an existing identifier + identifiers[icu_lower(key)] = val + self.set_identifiers(id, identifiers, notify=False, commit=False) + + user_mi = mi.get_all_user_metadata(make_copy=False) for key in user_mi.iterkeys(): if key in self.field_metadata and \ @@ -2432,14 +2531,88 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): if notify: self.notify('metadata', [id]) - def set_isbn(self, id, isbn, notify=True, commit=True): - self.conn.execute('UPDATE books SET isbn=? WHERE id=?', (isbn, id)) - self.dirtied([id], commit=False) + def isbn(self, idx, index_is_id=False): + row = self.data._data[idx] if index_is_id else self.data[idx] + if row is not None: + raw = row[self.FIELD_MAP['identifiers']] + if raw: + for x in raw.split(','): + if x.startswith('isbn:'): + return x[5:].strip() + + def get_identifiers(self, idx, index_is_id=False): + ans = {} + row = self.data._data[idx] if index_is_id else self.data[idx] + if row is not None: + raw = row[self.FIELD_MAP['identifiers']] + if raw: + for x in raw.split(','): + key, _, val = x.partition(':') + key, val = key.strip(), val.strip() + if key and val: + ans[key] = val + + return ans + + def get_all_identifier_types(self): + idents = self.conn.get('SELECT DISTINCT type FROM identifiers') + return [ident[0] for ident in idents] + + def _clean_identifier(self, typ, val): + typ = icu_lower(typ).strip().replace(':', '').replace(',', '') + val = val.strip().replace(',', '|').replace(':', '|') + return typ, val + + def set_identifier(self, id_, typ, val, notify=True, commit=True): + 'If val is empty, deletes identifier of type typ' + typ, val = self._clean_identifier(typ, val) + identifiers = self.get_identifiers(id_, index_is_id=True) + if not typ: + return + changed = False + if not val and typ in identifiers: + identifiers.pop(typ) + changed = True + self.conn.execute( + 'DELETE from identifiers WHERE book=? AND type=?', + (id_, typ)) + if val and identifiers.get(typ, None) != val: + changed = True + identifiers[typ] = val + self.conn.execute( + 'INSERT OR REPLACE INTO identifiers (book, type, val) VALUES (?, ?, ?)', + (id_, typ, val)) + if changed: + raw = ','.join(['%s:%s'%(k, v) for k, v in + identifiers.iteritems()]) + self.data.set(id_, self.FIELD_MAP['identifiers'], raw, + row_is_id=True) + if commit: + self.conn.commit() + if notify: + self.notify('metadata', [id_]) + + def set_identifiers(self, id_, identifiers, notify=True, commit=True): + cleaned = {} + for typ, val in identifiers.iteritems(): + typ, val = self._clean_identifier(typ, val) + if val: + cleaned[typ] = val + self.conn.execute('DELETE FROM identifiers WHERE book=?', (id_,)) + self.conn.executemany( + 'INSERT INTO identifiers (book, type, val) VALUES (?, ?, ?)', + [(id_, k, v) for k, v in cleaned.iteritems()]) + raw = ','.join(['%s:%s'%(k, v) for k, v in + cleaned.iteritems()]) + self.data.set(id_, self.FIELD_MAP['identifiers'], raw, + row_is_id=True) if commit: self.conn.commit() - self.data.set(id, self.FIELD_MAP['isbn'], isbn, row_is_id=True) if notify: - self.notify('metadata', [id]) + self.notify('metadata', [id_]) + + def set_isbn(self, id_, isbn, notify=True, commit=True): + self.set_identifier(id_, 'isbn', isbn, notify=notify, commit=commit) def add_catalog(self, path, title): format = os.path.splitext(path)[1][1:].lower() @@ -2737,7 +2910,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): prefix = self.library_path FIELDS = set(['title', 'authors', 'author_sort', 'publisher', 'rating', 'timestamp', 'size', 'tags', 'comments', 'series', 'series_index', - 'isbn', 'uuid', 'pubdate']) + 'uuid', 'pubdate', 'last_modified', 'identifiers']) for x in self.custom_column_num_map: FIELDS.add(x) data = [] @@ -2752,6 +2925,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns): data.append(x) x['id'] = db_id x['formats'] = [] + isbn = self.isbn(db_id, index_is_id=True) + x['isbn'] = isbn if isbn else '' if not x['authors']: x['authors'] = _('Unknown') x['authors'] = [i.replace('|', ',') for i in x['authors'].split(',')] @@ -2944,8 +3119,4 @@ books_series_link feeds s = self.conn.get('''SELECT book FROM books_plugin_data WHERE name=?''', (name,)) return [x[0] for x in s] - def get_custom_recipes(self): - for id, title, script in self.conn.get('SELECT id,title,script FROM feeds'): - yield id, title, script - diff --git a/src/calibre/library/field_metadata.py b/src/calibre/library/field_metadata.py index aff2803452..b8180f9f39 100644 --- a/src/calibre/library/field_metadata.py +++ b/src/calibre/library/field_metadata.py @@ -16,7 +16,8 @@ class TagsIcons(dict): ''' category_icons = ['authors', 'series', 'formats', 'publisher', 'rating', - 'news', 'tags', 'custom:', 'user:', 'search',] + 'news', 'tags', 'custom:', 'user:', 'search', + 'identifiers'] def __init__(self, icon_dict): for a in self.category_icons: if a not in icon_dict: @@ -24,16 +25,17 @@ class TagsIcons(dict): self[a] = icon_dict[a] category_icon_map = { - 'authors' : 'user_profile.png', - 'series' : 'series.png', - 'formats' : 'book.png', - 'publisher' : 'publisher.png', - 'rating' : 'rating.png', - 'news' : 'news.png', - 'tags' : 'tags.png', - 'custom:' : 'column.png', - 'user:' : 'tb_folder.png', - 'search' : 'search.png' + 'authors' : 'user_profile.png', + 'series' : 'series.png', + 'formats' : 'book.png', + 'publisher' : 'publisher.png', + 'rating' : 'rating.png', + 'news' : 'news.png', + 'tags' : 'tags.png', + 'custom:' : 'column.png', + 'user:' : 'tb_folder.png', + 'search' : 'search.png', + 'identifiers': 'id_card.png' } @@ -80,6 +82,8 @@ class FieldMetadata(dict): rec_index: the index of the field in the db metadata record. + is_csp: field contains colon-separated pairs. Must also be text, is_multiple + ''' VALID_DATA_TYPES = frozenset([None, 'rating', 'text', 'comments', 'datetime', @@ -98,7 +102,8 @@ class FieldMetadata(dict): 'name':_('Authors'), 'search_terms':['authors', 'author'], 'is_custom':False, - 'is_category':True}), + 'is_category':True, + 'is_csp': False}), ('series', {'table':'series', 'column':'name', 'link_column':'series', @@ -109,7 +114,8 @@ class FieldMetadata(dict): 'name':_('Series'), 'search_terms':['series'], 'is_custom':False, - 'is_category':True}), + 'is_category':True, + 'is_csp': False}), ('formats', {'table':None, 'column':None, 'datatype':'text', @@ -118,7 +124,8 @@ class FieldMetadata(dict): 'name':_('Formats'), 'search_terms':['formats', 'format'], 'is_custom':False, - 'is_category':True}), + 'is_category':True, + 'is_csp': False}), ('publisher', {'table':'publishers', 'column':'name', 'link_column':'publisher', @@ -129,7 +136,8 @@ class FieldMetadata(dict): 'name':_('Publishers'), 'search_terms':['publisher'], 'is_custom':False, - 'is_category':True}), + 'is_category':True, + 'is_csp': False}), ('rating', {'table':'ratings', 'column':'rating', 'link_column':'rating', @@ -140,7 +148,8 @@ class FieldMetadata(dict): 'name':_('Ratings'), 'search_terms':['rating'], 'is_custom':False, - 'is_category':True}), + 'is_category':True, + 'is_csp': False}), ('news', {'table':'news', 'column':'name', 'category_sort':'name', @@ -150,7 +159,8 @@ class FieldMetadata(dict): 'name':_('News'), 'search_terms':[], 'is_custom':False, - 'is_category':True}), + 'is_category':True, + 'is_csp': False}), ('tags', {'table':'tags', 'column':'name', 'link_column': 'tag', @@ -161,7 +171,18 @@ class FieldMetadata(dict): 'name':_('Tags'), 'search_terms':['tags', 'tag'], 'is_custom':False, - 'is_category':True}), + 'is_category':True, + 'is_csp': False}), + ('identifiers', {'table':None, + 'column':None, + 'datatype':'text', + 'is_multiple':',', + 'kind':'field', + 'name':_('Identifiers'), + 'search_terms':['identifiers', 'identifier', 'isbn'], + 'is_custom':False, + 'is_category':True, + 'is_csp': True}), ('author_sort',{'table':None, 'column':None, 'datatype':'text', @@ -170,7 +191,8 @@ class FieldMetadata(dict): 'name':None, 'search_terms':['author_sort'], 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), ('au_map', {'table':None, 'column':None, 'datatype':'text', @@ -179,7 +201,8 @@ class FieldMetadata(dict): 'name':None, 'search_terms':[], 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), ('comments', {'table':None, 'column':None, 'datatype':'text', @@ -187,7 +210,9 @@ class FieldMetadata(dict): 'kind':'field', 'name':_('Comments'), 'search_terms':['comments', 'comment'], - 'is_custom':False, 'is_category':False}), + 'is_custom':False, + 'is_category':False, + 'is_csp': False}), ('cover', {'table':None, 'column':None, 'datatype':'int', @@ -196,16 +221,8 @@ class FieldMetadata(dict): 'name':None, 'search_terms':['cover'], 'is_custom':False, - 'is_category':False}), - ('flags', {'table':None, - 'column':None, - 'datatype':'text', - 'is_multiple':None, - 'kind':'field', - 'name':None, - 'search_terms':[], - 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), ('id', {'table':None, 'column':None, 'datatype':'int', @@ -214,25 +231,18 @@ class FieldMetadata(dict): 'name':None, 'search_terms':[], 'is_custom':False, - 'is_category':False}), - ('isbn', {'table':None, + 'is_category':False, + 'is_csp': False}), + ('last_modified', {'table':None, 'column':None, - 'datatype':'text', + 'datatype':'datetime', 'is_multiple':None, 'kind':'field', - 'name':None, - 'search_terms':['isbn'], + 'name':_('Date'), + 'search_terms':['last_modified'], 'is_custom':False, - 'is_category':False}), - ('lccn', {'table':None, - 'column':None, - 'datatype':'text', - 'is_multiple':None, - 'kind':'field', - 'name':None, - 'search_terms':[], - 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), ('ondevice', {'table':None, 'column':None, 'datatype':'text', @@ -241,7 +251,8 @@ class FieldMetadata(dict): 'name':_('On Device'), 'search_terms':['ondevice'], 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), ('path', {'table':None, 'column':None, 'datatype':'text', @@ -250,7 +261,8 @@ class FieldMetadata(dict): 'name':None, 'search_terms':[], 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), ('pubdate', {'table':None, 'column':None, 'datatype':'datetime', @@ -259,7 +271,18 @@ class FieldMetadata(dict): 'name':_('Published'), 'search_terms':['pubdate'], 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), + ('marked', {'table':None, + 'column':None, + 'datatype':'text', + 'is_multiple':None, + 'kind':'field', + 'name': None, + 'search_terms':['marked'], + 'is_custom':False, + 'is_category':False, + 'is_csp': False}), ('series_index',{'table':None, 'column':None, 'datatype':'float', @@ -268,7 +291,8 @@ class FieldMetadata(dict): 'name':None, 'search_terms':['series_index'], 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), ('sort', {'table':None, 'column':None, 'datatype':'text', @@ -277,7 +301,8 @@ class FieldMetadata(dict): 'name':_('Title Sort'), 'search_terms':['title_sort'], 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), ('size', {'table':None, 'column':None, 'datatype':'float', @@ -286,7 +311,8 @@ class FieldMetadata(dict): 'name':_('Size (MB)'), 'search_terms':['size'], 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), ('timestamp', {'table':None, 'column':None, 'datatype':'datetime', @@ -295,7 +321,8 @@ class FieldMetadata(dict): 'name':_('Date'), 'search_terms':['date'], 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), ('title', {'table':None, 'column':None, 'datatype':'text', @@ -304,7 +331,8 @@ class FieldMetadata(dict): 'name':_('Title'), 'search_terms':['title'], 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), ('uuid', {'table':None, 'column':None, 'datatype':'text', @@ -313,7 +341,8 @@ class FieldMetadata(dict): 'name':None, 'search_terms':[], 'is_custom':False, - 'is_category':False}), + 'is_category':False, + 'is_csp': False}), ] # }}} @@ -335,7 +364,8 @@ class FieldMetadata(dict): self._tb_cats[k]['display'] = {} self._tb_cats[k]['is_editable'] = True self._add_search_terms_to_map(k, v['search_terms']) - self._tb_cats['timestamp']['display'] = { + for x in ('timestamp', 'last_modified'): + self._tb_cats[x]['display'] = { 'date_format': tweaks['gui_timestamp_display_format']} self._tb_cats['pubdate']['display'] = { 'date_format': tweaks['gui_pubdate_display_format']} @@ -441,7 +471,8 @@ class FieldMetadata(dict): return l def add_custom_field(self, label, table, column, datatype, colnum, name, - display, is_editable, is_multiple, is_category): + display, is_editable, is_multiple, is_category, + is_csp=False): key = self.custom_field_prefix + label if key in self._tb_cats: raise ValueError('Duplicate custom field [%s]'%(label)) @@ -454,7 +485,7 @@ class FieldMetadata(dict): 'colnum':colnum, 'display':display, 'is_custom':True, 'is_category':is_category, 'link_column':'value','category_sort':'value', - 'is_editable': is_editable,} + 'is_csp' : is_csp, 'is_editable': is_editable,} self._add_search_terms_to_map(key, [key]) self.custom_label_to_key_map[label] = key if datatype == 'series': @@ -466,7 +497,7 @@ class FieldMetadata(dict): 'colnum':None, 'display':{}, 'is_custom':False, 'is_category':False, 'link_column':None, 'category_sort':None, - 'is_editable': False,} + 'is_editable': False, 'is_csp': False} self._add_search_terms_to_map(key, [key]) self.custom_label_to_key_map[label+'_index'] = key @@ -515,7 +546,7 @@ class FieldMetadata(dict): 'datatype':None, 'is_multiple':None, 'kind':'user', 'name':name, 'search_terms':st, 'is_custom':False, - 'is_category':True} + 'is_category':True, 'is_csp': False} self._add_search_terms_to_map(label, st) def add_search_category(self, label, name): @@ -524,8 +555,8 @@ class FieldMetadata(dict): self._tb_cats[label] = {'table':None, 'column':None, 'datatype':None, 'is_multiple':None, 'kind':'search', 'name':name, - 'search_terms':[], 'is_custom':False, - 'is_category':True} + 'search_terms':[], 'is_custom':False, + 'is_category':True, 'is_csp': False} def set_field_record_index(self, label, index, prefer_custom=False): if prefer_custom: diff --git a/src/calibre/library/prefs.py b/src/calibre/library/prefs.py index 233c717897..4ef1dcb35a 100644 --- a/src/calibre/library/prefs.py +++ b/src/calibre/library/prefs.py @@ -49,8 +49,7 @@ class DBPrefs(dict): if self.disable_setting: return raw = self.to_raw(val) - self.db.conn.execute('DELETE FROM preferences WHERE key=?', (key,)) - self.db.conn.execute('INSERT INTO preferences (key,val) VALUES (?,?)', (key, + self.db.conn.execute('INSERT OR REPLACE INTO preferences (key,val) VALUES (?,?)', (key, raw)) self.db.conn.commit() dict.__setitem__(self, key, val) diff --git a/src/calibre/library/restore.py b/src/calibre/library/restore.py index 76f3c0333d..e03edd449a 100644 --- a/src/calibre/library/restore.py +++ b/src/calibre/library/restore.py @@ -13,6 +13,7 @@ from calibre.ptempfile import TemporaryDirectory from calibre.ebooks.metadata.opf2 import OPF from calibre.library.database2 import LibraryDatabase2 from calibre.constants import filesystem_encoding +from calibre.utils.date import utcfromtimestamp from calibre import isbytestring NON_EBOOK_EXTENSIONS = frozenset([ @@ -211,8 +212,8 @@ class Restore(Thread): force_id=book['id']) if book['mi'].uuid: db.set_uuid(book['id'], book['mi'].uuid, commit=False, notify=False) - db.conn.execute('UPDATE books SET path=? WHERE id=?', (book['path'], - book['id'])) + db.conn.execute('UPDATE books SET path=?,last_modified=? WHERE id=?', (book['path'], + utcfromtimestamp(book['timestamp']), book['id'])) for fmt, size, name in book['formats']: db.conn.execute(''' diff --git a/src/calibre/library/save_to_disk.py b/src/calibre/library/save_to_disk.py index de586048b7..96c42e6e0e 100644 --- a/src/calibre/library/save_to_disk.py +++ b/src/calibre/library/save_to_disk.py @@ -12,13 +12,13 @@ from calibre.constants import DEBUG from calibre.utils.config import Config, StringConfig, tweaks from calibre.utils.formatter import TemplateFormatter from calibre.utils.filenames import shorten_components_to, supports_long_names, \ - ascii_filename, sanitize_file_name + ascii_filename from calibre.ebooks.metadata.opf2 import metadata_to_opf from calibre.ebooks.metadata.meta import set_metadata -from calibre.constants import preferred_encoding, filesystem_encoding +from calibre.constants import preferred_encoding from calibre.ebooks.metadata import fmt_sidx from calibre.ebooks.metadata import title_sort -from calibre import strftime, prints +from calibre import strftime, prints, sanitize_file_name_unicode plugboard_any_device_value = 'any device' plugboard_any_format_value = 'any format' @@ -197,12 +197,10 @@ def get_components(template, mi, id, timefmt='%b %Y', length=250, format_args[key] = '' components = SafeFormat().safe_format(template, format_args, 'G_C-EXCEPTION!', mi) - components = [x.strip() for x in components.split('/') if x.strip()] + components = [x.strip() for x in components.split('/')] components = [sanitize_func(x) for x in components if x] if not components: components = [str(id)] - components = [x.encode(filesystem_encoding, 'replace') if isinstance(x, - unicode) else x for x in components] if to_lowercase: components = [x.lower() for x in components] if replace_whitespace: @@ -247,7 +245,7 @@ def do_save_book_to_disk(id_, mi, cover, plugboards, return True, id_, mi.title components = get_components(opts.template, mi, id_, opts.timefmt, length, - ascii_filename if opts.asciiize else sanitize_file_name, + ascii_filename if opts.asciiize else sanitize_file_name_unicode, to_lowercase=opts.to_lowercase, replace_whitespace=opts.replace_whitespace) base_path = os.path.join(root, *components) @@ -329,8 +327,6 @@ def do_save_book_to_disk(id_, mi, cover, plugboards, def _sanitize_args(root, opts): if opts is None: opts = config().parse() - if isinstance(root, unicode): - root = root.encode(filesystem_encoding) root = os.path.abspath(root) opts.template = preprocess_template(opts.template) diff --git a/src/calibre/library/schema_upgrades.py b/src/calibre/library/schema_upgrades.py index 0b7a3f5350..3fc9a2368a 100644 --- a/src/calibre/library/schema_upgrades.py +++ b/src/calibre/library/schema_upgrades.py @@ -8,6 +8,8 @@ __docformat__ = 'restructuredtext en' import os +from calibre.utils.date import isoformat, DEFAULT_DATE + class SchemaUpgrade(object): def __init__(self): @@ -468,4 +470,134 @@ class SchemaUpgrade(object): ''' self.conn.executescript(script) + def upgrade_version_18(self): + ''' + Add a library UUID. + Add an identifiers table. + Add a languages table. + Add a last_modified column. + NOTE: You cannot downgrade after this update, if you do + any changes you make to book isbns will be lost. + ''' + script = ''' + DROP TABLE IF EXISTS library_id; + CREATE TABLE library_id ( id INTEGER PRIMARY KEY, + uuid TEXT NOT NULL, + UNIQUE(uuid) + ); + + DROP TABLE IF EXISTS identifiers; + CREATE TABLE identifiers ( id INTEGER PRIMARY KEY, + book INTEGER NON NULL, + type TEXT NON NULL DEFAULT "isbn" COLLATE NOCASE, + val TEXT NON NULL COLLATE NOCASE, + UNIQUE(book, type) + ); + + DROP TABLE IF EXISTS languages; + CREATE TABLE languages ( id INTEGER PRIMARY KEY, + lang_code TEXT NON NULL COLLATE NOCASE, + UNIQUE(lang_code) + ); + + DROP TABLE IF EXISTS books_languages_link; + CREATE TABLE books_languages_link ( id INTEGER PRIMARY KEY, + book INTEGER NOT NULL, + lang_code INTEGER NOT NULL, + item_order INTEGER NOT NULL DEFAULT 0, + UNIQUE(book, lang_code) + ); + + DROP TRIGGER IF EXISTS fkc_delete_on_languages; + CREATE TRIGGER fkc_delete_on_languages + BEFORE DELETE ON languages + BEGIN + SELECT CASE + WHEN (SELECT COUNT(id) FROM books_languages_link WHERE lang_code=OLD.id) > 0 + THEN RAISE(ABORT, 'Foreign key violation: language is still referenced') + END; + END; + + DROP TRIGGER IF EXISTS fkc_delete_on_languages_link; + CREATE TRIGGER fkc_delete_on_languages_link + BEFORE INSERT ON books_languages_link + BEGIN + SELECT CASE + WHEN (SELECT id from books WHERE id=NEW.book) IS NULL + THEN RAISE(ABORT, 'Foreign key violation: book not in books') + WHEN (SELECT id from languages WHERE id=NEW.lang_code) IS NULL + THEN RAISE(ABORT, 'Foreign key violation: lang_code not in languages') + END; + END; + + DROP TRIGGER IF EXISTS fkc_update_books_languages_link_a; + CREATE TRIGGER fkc_update_books_languages_link_a + BEFORE UPDATE OF book ON books_languages_link + BEGIN + SELECT CASE + WHEN (SELECT id from books WHERE id=NEW.book) IS NULL + THEN RAISE(ABORT, 'Foreign key violation: book not in books') + END; + END; + DROP TRIGGER IF EXISTS fkc_update_books_languages_link_b; + CREATE TRIGGER fkc_update_books_languages_link_b + BEFORE UPDATE OF lang_code ON books_languages_link + BEGIN + SELECT CASE + WHEN (SELECT id from languages WHERE id=NEW.lang_code) IS NULL + THEN RAISE(ABORT, 'Foreign key violation: lang_code not in languages') + END; + END; + + DROP INDEX IF EXISTS books_languages_link_aidx; + CREATE INDEX books_languages_link_aidx ON books_languages_link (lang_code); + DROP INDEX IF EXISTS books_languages_link_bidx; + CREATE INDEX books_languages_link_bidx ON books_languages_link (book); + DROP INDEX IF EXISTS languages_idx; + CREATE INDEX languages_idx ON languages (lang_code COLLATE NOCASE); + + DROP TRIGGER IF EXISTS books_delete_trg; + CREATE TRIGGER books_delete_trg + AFTER DELETE ON books + BEGIN + DELETE FROM books_authors_link WHERE book=OLD.id; + DELETE FROM books_publishers_link WHERE book=OLD.id; + DELETE FROM books_ratings_link WHERE book=OLD.id; + DELETE FROM books_series_link WHERE book=OLD.id; + DELETE FROM books_tags_link WHERE book=OLD.id; + DELETE FROM books_languages_link WHERE book=OLD.id; + DELETE FROM data WHERE book=OLD.id; + DELETE FROM comments WHERE book=OLD.id; + DELETE FROM conversion_options WHERE book=OLD.id; + DELETE FROM books_plugin_data WHERE book=OLD.id; + DELETE FROM identifiers WHERE book=OLD.id; + END; + + INSERT INTO identifiers (book, val) SELECT id,isbn FROM books WHERE isbn; + + ALTER TABLE books ADD COLUMN last_modified TIMESTAMP NOT NULL DEFAULT "%s"; + + '''%isoformat(DEFAULT_DATE, sep=' ') + # Sqlite does not support non constant default values in alter + # statements + self.conn.executescript(script) + + def upgrade_version_19(self): + recipes = self.conn.get('SELECT id,title,script FROM feeds') + if recipes: + from calibre.web.feeds.recipes import custom_recipes, \ + custom_recipe_filename + bdir = os.path.dirname(custom_recipes.file_path) + for id_, title, script in recipes: + existing = frozenset(map(int, custom_recipes.iterkeys())) + if id_ in existing: + id_ = max(existing) + 1000 + id_ = str(id_) + fname = custom_recipe_filename(id_, title) + custom_recipes[id_] = (title, fname) + if isinstance(script, unicode): + script = script.encode('utf-8') + with open(os.path.join(bdir, fname), 'wb') as f: + f.write(script) + diff --git a/src/calibre/library/server/browse.py b/src/calibre/library/server/browse.py index 7dfedcb6ff..97bfc30f14 100644 --- a/src/calibre/library/server/browse.py +++ b/src/calibre/library/server/browse.py @@ -346,7 +346,7 @@ class BrowseServer(object): for category in sorted(categories, key=lambda x: sort_key(getter(x))): if len(categories[category]) == 0: continue - if category == 'formats': + if category in ('formats', 'identifiers'): continue meta = category_meta.get(category, None) if meta is None: @@ -666,7 +666,7 @@ class BrowseServer(object): if add_category_links: added_key = False fm = mi.metadata_for_field(key) - if val and fm and fm['is_category'] and \ + if val and fm and fm['is_category'] and not fm['is_csp'] and\ key != 'formats' and fm['datatype'] not in ['rating']: categories = mi.get(key) if isinstance(categories, basestring): diff --git a/src/calibre/library/server/opds.py b/src/calibre/library/server/opds.py index 72a802eaa9..e7fdffbbbb 100644 --- a/src/calibre/library/server/opds.py +++ b/src/calibre/library/server/opds.py @@ -580,7 +580,7 @@ class OPDSServer(object): for category in sorted(categories, key=lambda x: sort_key(getter(x))): if len(categories[category]) == 0: continue - if category == 'formats': + if category in ('formats', 'identifiers'): continue meta = category_meta.get(category, None) if meta is None: diff --git a/src/calibre/library/server/xml.py b/src/calibre/library/server/xml.py index efbceb9771..14955dc541 100644 --- a/src/calibre/library/server/xml.py +++ b/src/calibre/library/server/xml.py @@ -89,13 +89,16 @@ class XMLServer(object): for x in ('id', 'title', 'sort', 'author_sort', 'rating', 'size'): kwargs[x] = serialize(record[FM[x]]) - for x in ('isbn', 'formats', 'series', 'tags', 'publisher', - 'comments'): + for x in ('formats', 'series', 'tags', 'publisher', + 'comments', 'identifiers'): y = record[FM[x]] if x == 'tags': y = format_tag_string(y, ',', ignore_max=True) kwargs[x] = serialize(y) if y else '' + isbn = self.db.isbn(record[FM['id']], index_is_id=True) + kwargs['isbn'] = serialize(isbn if isbn else '') + kwargs['safe_title'] = ascii_filename(kwargs['title']) c = kwargs.pop('comments') diff --git a/src/calibre/library/sqlite.py b/src/calibre/library/sqlite.py index 622d6b8459..2075ab5880 100644 --- a/src/calibre/library/sqlite.py +++ b/src/calibre/library/sqlite.py @@ -8,6 +8,7 @@ Wrapper for multi-threaded access to a single sqlite database connection. Serial all calls. ''' import sqlite3 as sqlite, traceback, time, uuid, sys, os +import repr as reprlib from sqlite3 import IntegrityError, OperationalError from threading import Thread from Queue import Queue @@ -16,18 +17,54 @@ from datetime import datetime from functools import partial from calibre.ebooks.metadata import title_sort, author_to_author_sort -from calibre.utils.date import parse_date, isoformat +from calibre.utils.date import parse_date, isoformat, local_tz from calibre import isbytestring, force_unicode -from calibre.constants import iswindows, DEBUG +from calibre.constants import iswindows, DEBUG, plugins from calibre.utils.icu import strcmp +from calibre import prints + +from dateutil.tz import tzoffset global_lock = RLock() -def convert_timestamp(val): +_c_speedup = plugins['speedup'][0] + +def _c_convert_timestamp(val): + if not val: + return None + try: + ret = _c_speedup.parse_date(val.strip()) + except: + ret = None + if ret is None: + return parse_date(val, as_utc=False) + year, month, day, hour, minutes, seconds, tzsecs = ret + return datetime(year, month, day, hour, minutes, seconds, + tzinfo=tzoffset(None, tzsecs)).astimezone(local_tz) + +def _py_convert_timestamp(val): if val: + tzsecs = 0 + try: + sign = {'+':1, '-':-1}.get(val[-6], None) + if sign is not None: + tzsecs = 60*((int(val[-5:-3])*60 + int(val[-2:])) * sign) + year = int(val[0:4]) + month = int(val[5:7]) + day = int(val[8:10]) + hour = int(val[11:13]) + min = int(val[14:16]) + sec = int(val[17:19]) + return datetime(year, month, day, hour, min, sec, + tzinfo=tzoffset(None, tzsecs)) + except: + pass return parse_date(val, as_utc=False) return None +convert_timestamp = _py_convert_timestamp if _c_speedup is None else \ + _c_convert_timestamp + def adapt_datetime(dt): return isoformat(dt, sep=' ') @@ -87,6 +124,18 @@ class SortedConcatenate(object): class SafeSortedConcatenate(SortedConcatenate): sep = '|' +class IdentifiersConcat(object): + '''String concatenation aggregator for the identifiers map''' + def __init__(self): + self.ans = [] + + def step(self, key, val): + self.ans.append(u'%s:%s'%(key, val)) + + def finalize(self): + return ','.join(self.ans) + + class AumSortedConcatenate(object): '''String concatenation aggregator for the author sort map''' def __init__(self): @@ -170,13 +219,13 @@ class DBThread(Thread): detect_types=sqlite.PARSE_DECLTYPES|sqlite.PARSE_COLNAMES) self.conn.execute('pragma cache_size=5000') encoding = self.conn.execute('pragma encoding').fetchone()[0] - c_ext_loaded = load_c_extensions(self.conn) + self.conn.create_aggregate('sortconcat', 2, SortedConcatenate) + self.conn.create_aggregate('sort_concat', 2, SafeSortedConcatenate) + self.conn.create_aggregate('identifiers_concat', 2, IdentifiersConcat) + load_c_extensions(self.conn) self.conn.row_factory = sqlite.Row if self.row_factory else lambda cursor, row : list(row) self.conn.create_aggregate('concat', 1, Concatenate) self.conn.create_aggregate('aum_sortconcat', 3, AumSortedConcatenate) - if not c_ext_loaded: - self.conn.create_aggregate('sortconcat', 2, SortedConcatenate) - self.conn.create_aggregate('sort_concat', 2, SafeSortedConcatenate) self.conn.create_collation('PYNOCASE', partial(pynocase, encoding=encoding)) self.conn.create_function('title_sort', 1, title_sort) @@ -208,17 +257,21 @@ class DBThread(Thread): except Exception, err: ok, res = False, (err, traceback.format_exc()) else: - func = getattr(self.conn, func) + bfunc = getattr(self.conn, func) try: for i in range(3): try: - ok, res = True, func(*args, **kwargs) + ok, res = True, bfunc(*args, **kwargs) break except OperationalError, err: # Retry if unable to open db file - if 'unable to open' not in str(err) or i == 2: + e = str(err) + if 'unable to open' not in e or i == 2: + if 'unable to open' in e: + prints('Unable to open database for func', + func, reprlib.repr(args), + reprlib.repr(kwargs)) raise - traceback.print_exc() time.sleep(0.5) except Exception, err: ok, res = False, (err, traceback.format_exc()) diff --git a/src/calibre/library/sqlite_custom.c b/src/calibre/library/sqlite_custom.c index 650c474c2c..dee17c79d4 100644 --- a/src/calibre/library/sqlite_custom.c +++ b/src/calibre/library/sqlite_custom.c @@ -77,6 +77,7 @@ static void sort_concat_free(SortConcatList *list) { free(list->vals[i]->val); free(list->vals[i]); } + free(list->vals); } static int sort_concat_cmp(const void *a_, const void *b_) { @@ -142,11 +143,102 @@ static void sort_concat_finalize2(sqlite3_context *context) { // }}} +// identifiers_concat {{{ + +typedef struct { + char *val; + size_t length; +} IdentifiersConcatItem; + +typedef struct { + IdentifiersConcatItem **vals; + size_t count; + size_t length; +} IdentifiersConcatList; + +static void identifiers_concat_step(sqlite3_context *context, int argc, sqlite3_value **argv) { + const char *key, *val; + size_t len = 0; + IdentifiersConcatList *list; + + assert(argc == 2); + + list = (IdentifiersConcatList*) sqlite3_aggregate_context(context, sizeof(*list)); + if (list == NULL) return; + + if (list->vals == NULL) { + list->vals = (IdentifiersConcatItem**)calloc(100, sizeof(IdentifiersConcatItem*)); + if (list->vals == NULL) return; + list->length = 100; + list->count = 0; + } + + if (list->count == list->length) { + list->vals = (IdentifiersConcatItem**)realloc(list->vals, list->length + 100); + if (list->vals == NULL) return; + list->length = list->length + 100; + } + + list->vals[list->count] = (IdentifiersConcatItem*)calloc(1, sizeof(IdentifiersConcatItem)); + if (list->vals[list->count] == NULL) return; + + key = (char*) sqlite3_value_text(argv[0]); + val = (char*) sqlite3_value_text(argv[1]); + if (key == NULL || val == NULL) {return;} + len = strlen(key) + strlen(val) + 1; + + list->vals[list->count]->val = (char*)calloc(len+1, sizeof(char)); + if (list->vals[list->count]->val == NULL) return; + snprintf(list->vals[list->count]->val, len+1, "%s:%s", key, val); + list->vals[list->count]->length = len; + + list->count = list->count + 1; + +} + + +static void identifiers_concat_finalize(sqlite3_context *context) { + IdentifiersConcatList *list; + IdentifiersConcatItem *item; + char *ans, *pos; + size_t sz = 0, i; + + list = (IdentifiersConcatList*) sqlite3_aggregate_context(context, sizeof(*list)); + if (list == NULL || list->vals == NULL || list->count < 1) return; + + for (i = 0; i < list->count; i++) { + sz += list->vals[i]->length; + } + sz += list->count; // Space for commas + ans = (char*)calloc(sz+2, sizeof(char)); + if (ans == NULL) return; + + pos = ans; + + for (i = 0; i < list->count; i++) { + item = list->vals[i]; + if (item == NULL || item->val == NULL) continue; + memcpy(pos, item->val, item->length); + pos += item->length; + *pos = ','; + pos += 1; + free(item->val); + free(item); + } + *(pos-1) = 0; // Remove trailing comma + sqlite3_result_text(context, ans, -1, SQLITE_TRANSIENT); + free(ans); + free(list->vals); +} + +// }}} + MYEXPORT int sqlite3_extension_init( sqlite3 *db, char **pzErrMsg, const sqlite3_api_routines *pApi){ SQLITE_EXTENSION_INIT2(pApi); sqlite3_create_function(db, "sortconcat", 2, SQLITE_UTF8, NULL, NULL, sort_concat_step, sort_concat_finalize); sqlite3_create_function(db, "sort_concat", 2, SQLITE_UTF8, NULL, NULL, sort_concat_step, sort_concat_finalize2); + sqlite3_create_function(db, "identifiers_concat", 2, SQLITE_UTF8, NULL, NULL, identifiers_concat_step, identifiers_concat_finalize); return 0; } diff --git a/src/calibre/manual/faq.rst b/src/calibre/manual/faq.rst index 8a78815751..a3d4332fd0 100644 --- a/src/calibre/manual/faq.rst +++ b/src/calibre/manual/faq.rst @@ -327,10 +327,24 @@ Now coming to author name sorting: * When recalculating the author sort values for books, |app| uses the author sort values for each individual author. Therefore, ensure that the individual author sort values are correct before recalculating the books' author sort values. * You can control whether the Tag Browser display authors using their names or their sort values by setting the :guilabel:`categories_use_field_for_author_name` tweak in Preferences->Tweaks -With all this flexibility, it is possible to have |app| manage your author names however you like. For example, one common request is to have |app| display author names LN, FN. To do this first set the ``author_sort_copy_method`` to ``copy``. Then change all author names to LN, FN via the Manage authors dialog. Then have |app| recalculate author sort values for both authors and books as described above. - Note that you can set an individual author's sort value to whatever you want using :guilabel:`Manage authors`. This is useful when dealing with names that |app| will not get right, such as complex multi-part names like Miguel de Cervantes Saavedra or when dealing with Asian names like Sun Tzu. +With all this flexibility, it is possible to have |app| manage your author names however you like. For example, one common request is to have |app| display author names LN, FN. To do this, and if the note below does not apply to you, then: + * Set the ``author_sort_copy_method`` tweak to ``copy`` as described above. + * Restart calibre. Do not change any book metadata before doing the remaining steps. + * Change all author names to LN, FN using the Manage authors dialog. + * After you have changed all the authors, press the `Recalculate all author sort values` button. + * Press OK, at which point |app| will change the authors in all your books. This can take a while. + +.. note:: + + When changing from FN LN to LN, FN, it is often the case that the values in author_sort are already in LN, FN format. If this is your case, then do the following: + * set the ``author_sort_copy_method`` tweak to ``copy`` as described above. + * restart calibre. Do not change any book metadata before doing the remaining steps. + * open the Manage authors dialog. Press the ``copy all author sort values to author`` button. + * Check through the authors to be sure you are happy. You can still press Cancel to abandon the changes. Once you press OK, there is no undo. + * Press OK, at which point |app| will change the authors in all your books. This can take a while. + Why doesn't |app| let me store books in my own directory structure? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -350,7 +364,7 @@ Why doesn't |app| have a column for foo? |app| is designed to have columns for the most frequently and widely used fields. In addition, you can add any columns you like. Columns can be added via :guilabel:`Preferences->Interface->Add your own columns`. Watch the tutorial `UI Power tips `_ to learn how to create your own columns. -You can also create "virtual columns" that contain combinations of the metadata from other columns. In the add column dialog choose the option "Column from other columns" and in the template enter the other column names. For example to create a virtual column containing formats or ISBN, enter ``{formats}`` for formats or ``{isbn}`` for ISBN. For more details, see :ref:`templatelangcalibre`. +You can also create "virtual columns" that contain combinations of the metadata from other columns. In the add column dialog use the :guilabel:`Quick create` links to easily create columns to show the book ISBN, formats or the time the book was last modified. For more details, see :ref:`templatelangcalibre`. Can I have a column showing the formats or the ISBN? diff --git a/src/calibre/manual/gui.rst b/src/calibre/manual/gui.rst index 210bd0569e..fff18a7333 100644 --- a/src/calibre/manual/gui.rst +++ b/src/calibre/manual/gui.rst @@ -338,7 +338,7 @@ You can build advanced search queries easily using the :guilabel:`Advanced Searc clicking the button |sbi|. Available fields for searching are: ``tag, title, author, publisher, series, series_index, rating, cover, -comments, format, isbn, date, pubdate, search, size`` and custom columns. If a device is plugged in, the +comments, format, identifiers, date, pubdate, search, size`` and custom columns. If a device is plugged in, the ``ondevice`` field becomes available. To find the search name for a custom column, hover your mouse over the column header. @@ -385,6 +385,21 @@ with undefined values in the column. Searching for ``true`` will find all books values in the column. Searching for ``yes`` or ``checked`` will find all books with ``Yes`` in the column. Searching for ``no`` or ``unchecked`` will find all books with ``No`` in the column. +Hierarchical items (e.g. A.B.C) use an extended syntax to match initial parts of the hierarchy. This is done by adding a period between the exact match indicator (=) and the text. For example, the query ``tags:=.A`` will find the tags `A` and `A.B`, but will not find the tags `AA` or `AA.B`. The query ``tags:=.A.B`` will find the tags `A.B` and `A.C`, but not the tag `A`. + +Identifiers (e.g., isbn, doi, lccn etc) also use an extended syntax. First, note that an identifier has the form ``key:value``, as in ``isbn:123456789``. The extended syntax permits you to specify independently which key and value to search for. Both the key and the value parts of the query can use `equality`, `contains`, or `regular expression` matches. Examples: + + * ``identifiers:true`` will find books with any identifier. + * ``identifiers:false`` will find books with no identifier. + * ``identifiers:123`` will search for books with any key having a value containing `123`. + * ``identifiers:=123456789`` will search for books with any key having a value equal to `123456789`. + * ``identifiers:=isbn:`` and ``identifiers:isbn:true`` will find books with a key equal to isbn having any value + * ``identifiers:=isbn:false`` will find books with no key equal to isbn. + * ``identifiers:=isbn:123`` will find books with a key equal to isbn having a value containing `123`. + * ``identifiers:=isbn:=123456789`` will find books with a key equal to isbn having a value equal to `123456789`. + * ``identifiers:i:1`` will find books with a key containing an `i` having a value containing a `1`. + + .. |sbi| image:: images/search_button.png :align: middle @@ -482,6 +497,8 @@ Calibre has several keyboard shortcuts to save you time and mouse movement. Thes - Edit the metadata of the currently selected field in the book list. * - :kbd:`A` - Add Books + * - :kbd:`Shift+A` + - Add Formats to the selected books * - :kbd:`C` - Convert selected Books * - :kbd:`D` diff --git a/src/calibre/manual/template_lang.rst b/src/calibre/manual/template_lang.rst index f63a7c4e95..3438f266b5 100644 --- a/src/calibre/manual/template_lang.rst +++ b/src/calibre/manual/template_lang.rst @@ -30,7 +30,7 @@ You can use all the various metadata fields available in calibre in a template, In addition to the column based fields, you also can use:: {formats} - A list of formats available in the calibre library for a book - {isbn} - The ISBN number of the book + {identifiers:select(isbn)} - The ISBN number of the book If a particular book does not have a particular piece of metadata, the field in the template is automatically removed for that book. Consider, for example:: @@ -95,7 +95,7 @@ Advanced features Using templates in custom columns ---------------------------------- -There are sometimes cases where you want to display metadata that |app| does not normally display, or to display data in a way different from how |app| normally does. For example, you might want to display the ISBN, a field that |app| does not display. You can use custom columns for this by creating a column with the type 'column built from other columns' (hereafter called composite columns), and entering a template. Result: |app| will display a column showing the result of evaluating that template. To display the ISBN, create the column and enter ``{isbn}`` into the template box. To display a column containing the values of two series custom columns separated by a comma, use ``{#series1:||,}{#series2}``. +There are sometimes cases where you want to display metadata that |app| does not normally display, or to display data in a way different from how |app| normally does. For example, you might want to display the ISBN, a field that |app| does not display. You can use custom columns for this by creating a column with the type 'column built from other columns' (hereafter called composite columns), and entering a template. Result: |app| will display a column showing the result of evaluating that template. To display the ISBN, create the column and enter ``{identifiers:select(isbn)}`` into the template box. To display a column containing the values of two series custom columns separated by a comma, use ``{#series1:||,}{#series2}``. Composite columns can use any template option, including formatting. @@ -122,10 +122,11 @@ The functions available are: * ``count(separator)`` -- interprets the value as a list of items separated by `separator`, returning the number of items in the list. Most lists use a comma as the separator, but authors uses an ampersand. Examples: `{tags:count(,)}`, `{authors:count(&)}` * ``ifempty(text)`` -- if the field is not empty, return the value of the field. Otherwise return `text`. * ``list_item(index, separator)`` -- interpret the value as a list of items separated by `separator`, returning the `index`th item. The first item is number zero. The last item can be returned using `list_item(-1,separator)`. If the item is not in the list, then the empty value is returned. The separator has the same meaning as in the `count` function. - * ``lookup(pattern, field, pattern, field, ..., else_field)`` -- like switch, except the arguments are field (metadata) names, not text. The value of the appropriate field will be fetched and used. Note that because composite columns are fields, you can use this function in one composite field to use the value of some other composite field. This is extremely useful when constructing variable save paths (more later). * ``re(pattern, replacement)`` -- return the field after applying the regular expression. All instances of `pattern` are replaced with `replacement`. As in all of |app|, these are python-compatible regular expressions. * ``shorten(left chars, middle text, right chars)`` -- Return a shortened version of the field, consisting of `left chars` characters from the beginning of the field, followed by `middle text`, followed by `right chars` characters from the end of the string. `Left chars` and `right chars` must be integers. For example, assume the title of the book is `Ancient English Laws in the Times of Ivanhoe`, and you want it to fit in a space of at most 15 characters. If you use ``{title:shorten(9,-,5)}``, the result will be `Ancient E-nhoe`. If the field's length is less than ``left chars`` + ``right chars`` + the length of ``middle text``, then the field will be used intact. For example, the title `The Dome` would not be changed. * ``switch(pattern, value, pattern, value, ..., else_value)`` -- for each ``pattern, value`` pair, checks if the field matches the regular expression ``pattern`` and if so, returns that ``value``. If no ``pattern`` matches, then ``else_value`` is returned. You can have as many ``pattern, value`` pairs as you want. + * ``lookup(pattern, field, pattern, field, ..., else_field)`` -- like switch, except the arguments are field (metadata) names, not text. The value of the appropriate field will be fetched and used. Note that because composite columns are fields, you can use this function in one composite field to use the value of some other composite field. This is extremely useful when constructing variable save paths (more later). + * ``select(key)`` -- interpret the field as a comma-separated list of items, with the items being of the form "id:value". Find the pair with the id equal to key, and return the corresponding value. This function is particularly useful for extracting a value such as an isbn from the set of identifiers for a book. * ``test(text if not empty, text if empty)`` -- return `text if not empty` if the field is not empty, otherwise return `text if empty`. @@ -141,7 +142,9 @@ Note that you can use the prefix and suffix as well. If you want the number to a Using functions in templates - template program mode ---------------------------------------------------- -The template language program mode differs from single-function mode in that it permits you to write template expressions that refer to other metadata fields, modify values, and do arithmetic. It is a reasonably complete programming language. +The template language program mode differs from single-function mode in that it permits you to write template expressions that refer to other metadata fields, modify values, and do arithmetic. It is a reasonably complete programming language. + +You can use the functions documented above in template program mode. See below for details. Beginning with an example, assume that you want your template to show the series for a book if it has one, otherwise show the value of a custom field #genre. You cannot do this in the basic language because you cannot make reference to another metadata field within a template expression. In program mode, you can. The following expression works:: @@ -203,7 +206,7 @@ For various values of series_index, the program returns: * series_index == 2, result = ``prefix 2->eq suffix`` * series_index == 3, result = ``prefix 3->gt suffix`` -All the functions listed under single-function mode can be used in program mode, noting that unlike the functions described below you must supply a first parameter providing the value the function is to act upon. +**All the functions listed under single-function mode can be used in program mode**. To do so, you must supply the value that the function is to act upon as the first parameter, in addition to the parameters documented above. For example, in program mode the parameters of the `test` function are ``test(x, text_if_not_empty, text_if_empty)``. The `x` parameter, which is the value to be tested, will almost always be a variable or a function call, often `field()`. The following functions are available in addition to those described in single-function mode. Remember from the example above that the single-function mode functions require an additional first parameter specifying the field to operate on. With the exception of the ``id`` parameter of assign, all parameters can be statements (sequences of expressions): @@ -212,9 +215,23 @@ The following functions are available in addition to those described in single-f * ``cmp(x, y, lt, eq, gt)`` -- compares x and y after converting both to numbers. Returns ``lt`` if x < y. Returns ``eq`` if x == y. Otherwise returns ``gt``. * ``divide(x, y)`` -- returns x / y. Throws an exception if either x or y are not numbers. * ``field(name)`` -- returns the metadata field named by ``name``. + * ``format_date(x, date_format)`` -- format_date(val, format_string) -- format the value, which must be a date field, using the format_string, returning a string. The formatting codes are:: + + d : the day as number without a leading zero (1 to 31) + dd : the day as number with a leading zero (01 to 31) ' + ddd : the abbreviated localized day name (e.g. "Mon" to "Sun"). ' + dddd : the long localized day name (e.g. "Monday" to "Sunday"). ' + M : the month as number without a leading zero (1 to 12). ' + MM : the month as number with a leading zero (01 to 12) ' + MMM : the abbreviated localized month name (e.g. "Jan" to "Dec"). ' + MMMM : the long localized month name (e.g. "January" to "December"). ' + yy : the year as two digit number (00 to 99). ' + yyyy : the year as four digit number.' + * ``eval(string)`` -- evaluates the string as a program, passing the local variables (those ``assign`` ed to). This permits using the template processor to construct complex results from local variables. * ``multiply(x, y)`` -- returns x * y. Throws an exception if either x or y are not numbers. * ``print(a, b, ...)`` -- prints the arguments to standard output. Unless you start calibre from the command line (``calibre-debug -g``), the output will go to a black hole. + * ``raw_field(name)`` -- returns the metadata field named by name without applying any formatting. * ``strcat(a, b, ...)`` -- can take any number of arguments. Returns a string formed by concatenating all the arguments. * ``strcmp(x, y, lt, eq, gt)`` -- does a case-insensitive comparison x and y as strings. Returns ``lt`` if x < y. Returns ``eq`` if x == y. Otherwise returns ``gt``. * ``substr(str, start, end)`` -- returns the ``start``'th through the ``end``'th characters of ``str``. The first character in ``str`` is the zero'th character. If end is negative, then it indicates that many characters counting from the right. If end is zero, then it indicates the last character. For example, ``substr('12345', 1, 0)`` returns ``'2345'``, and ``substr('12345', 1, -1)`` returns ``'234'``. diff --git a/src/calibre/manual/templates/layout.html b/src/calibre/manual/templates/layout.html index b427482947..8f35a9a6c5 100644 --- a/src/calibre/manual/templates/layout.html +++ b/src/calibre/manual/templates/layout.html @@ -1,23 +1,6 @@ {% extends "!layout.html" %} {% block extrahead %} - {% if not embedded %} - - {% endif %} -