Cutlery/calibre
1
0
Fork 0
mirror of https://github.com/kovidgoyal/calibre.git synced 2025-07-09 03:04:10 -04:00
Code Issues Packages Projects Releases Wiki Activity

0.9.25

Browse Source
This commit is contained in:
GRiker 2013-03-29 11:33:33 -07:00
commit b995eb7fb9
155 changed files with 52025 additions and 33940 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

44
Changelog.yaml
View File

@ -1,3 +1,4 @@
# vim:fileencoding=UTF-8:ts=2:sw=2:sta:et:sts=2:ai
# Each release can have new features and bug fixes. Each of which # Each release can have new features and bug fixes. Each of which
# must have a title and can optionally have linked tickets and a description. # must have a title and can optionally have linked tickets and a description.
# In addition they can have a type field which defaults to minor, but should be major # In addition they can have a type field which defaults to minor, but should be major
@ -19,6 +20,49 @@
# new recipes: # new recipes:
# - title: # - title:
- version: 0.9.25
date: 2013-03-29
new features:
- title: "Automatic adding: When checking for duplicates is enabled, use the same duplicates found dialog as is used during manual adding."
tickets: [1160914]
- title: "ToC Editor: Allow searching to find a location quickly when browsing through the book to select a location for a ToC item"
- title: "ToC Editor: Add a button to quickly flatten the entire table of contents"
- title: "Conversion: When converting a single book to EPUB or AZW3, add an option to automatically launch the Table of Contents editor after the conversion completes. Found under the Table of Contents section of the conversion dialog."
bug fixes:
- title: "calibredb: Nicer error messages when user provides invalid input"
tickets: [1160452,1160631]
- title: "News download: Always use the .jpg extension for jpeg images as apparently Moon+ Reader cannot handle .jpeg"
- title: "Fix Book Details popup keyboard navigation doesn't work on a Mac"
tickets: [1159610]
- title: "Fix a regression that caused the case of the book files to not be changed when changing the case of the title/author on case insensitive filesystems"
improved recipes:
- RTE news
- Various Polish news sources
- Psychology Today
- Foreign Affairs
- History Today
- Harpers Magazine (printed edition)
- Business Week Magazine
- The Hindu
- Irish Times
- Le Devoir
new recipes:
- title: Fortune Magazine
author: Rick Shang
- title: Eclipse Online
author: Jim DeVona
- version: 0.9.24 - version: 0.9.24
date: 2013-03-22 date: 2013-03-22

14
manual/conversion.rst
View File

@ -434,6 +434,18 @@ a number of older formats either do not support a metadata based Table of Conten
documents do not have one. In these cases, the options in this section can help you automatically documents do not have one. In these cases, the options in this section can help you automatically
generate a Table of Contents in the converted ebook, based on the actual content in the input document. generate a Table of Contents in the converted ebook, based on the actual content in the input document.
.. note:: Using these options can be a little challenging to get exactly right.
If you prefer creating/editing the Table of Contents by hand, convert to
the EPUB or AZW3 formats and select the checkbox at the bottom of the
screen that says
:guilabel:`Manually fine-tune the Table of Contents after conversion`.
This will launch the ToC Editor tool after the conversion. It allows you to
create entries in the Table of Contents by simply clicking the place in the
book where you want the entry to point. You can also use the ToC Editor by
itself, without doing a conversion. Go to :guilabel:`Preferences->Toolbars`
and add the ToC Editor to the main toolbar. Then just select the book you
want to edit and click the ToC Editor button.
The first option is :guilabel:`Force use of auto-generated Table of Contents`. By checking this option The first option is :guilabel:`Force use of auto-generated Table of Contents`. By checking this option
you can have |app| override any Table of Contents found in the metadata of the input document with the you can have |app| override any Table of Contents found in the metadata of the input document with the
auto generated one. auto generated one.
@ -456,7 +468,7 @@ For example, to remove all entries titles "Next" or "Previous" use::
Next|Previous Next|Previous
Finally, the :guilabel:`Level 1,2,3 TOC` options allow you to create a sophisticated multi-level Table of Contents. The :guilabel:`Level 1,2,3 TOC` options allow you to create a sophisticated multi-level Table of Contents.
They are XPath expressions that match tags in the intermediate XHTML produced by the conversion pipeline. See the They are XPath expressions that match tags in the intermediate XHTML produced by the conversion pipeline. See the
:ref:`conversion-introduction` for how to get access to this XHTML. Also read the :ref:`xpath-tutorial`, to learn :ref:`conversion-introduction` for how to get access to this XHTML. Also read the :ref:`xpath-tutorial`, to learn
how to construct XPath expressions. Next to each option is a button that launches a wizard to help with the creation how to construct XPath expressions. Next to each option is a button that launches a wizard to help with the creation

41
manual/faq.rst
View File

@ -87,7 +87,9 @@ this bug.
How do I convert a collection of HTML files in a specific order? How do I convert a collection of HTML files in a specific order?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to convert a collection of HTML files in a specific oder, you have to create a table of contents file. That is, another HTML file that contains links to all the other files in the desired order. Such a file looks like:: In order to convert a collection of HTML files in a specific oder, you have to
create a table of contents file. That is, another HTML file that contains links
to all the other files in the desired order. Such a file looks like::
<html> <html>
<body> <body>
@ -102,19 +104,36 @@ In order to convert a collection of HTML files in a specific oder, you have to c
</body> </body>
</html> </html>
Then just add this HTML file to the GUI and use the convert button to create your ebook. Then, just add this HTML file to the GUI and use the convert button to create
your ebook. You can use the option in the Table of Contents section in the
conversion dialog to control how the Table of Contents is generated.
.. note:: By default, when adding HTML files, |app| follows links in the files in *depth first* order. This means that if file A.html links to B.html and C.html and D.html, but B.html also links to D.html, then the files will be in the order A.html, B.html, D.html, C.html. If instead you want the order to be A.html, B.html, C.html, D.html then you must tell |app| to add your files in *breadth first* order. Do this by going to Preferences->Plugins and customizing the HTML to ZIP plugin. .. note:: By default, when adding HTML files, |app| follows links in the files
in *depth first* order. This means that if file A.html links to B.html and
C.html and D.html, but B.html also links to D.html, then the files will be
in the order A.html, B.html, D.html, C.html. If instead you want the order
to be A.html, B.html, C.html, D.html then you must tell |app| to add your
files in *breadth first* order. Do this by going to Preferences->Plugins
and customizing the HTML to ZIP plugin.
The EPUB I produced with |app| is not valid? The EPUB I produced with |app| is not valid?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|app| does not guarantee that an EPUB produced by it is valid. The only guarantee it makes is that if you feed it valid XHTML 1.1 + CSS 2.1 it will output a valid EPUB. |app| is designed for ebook consumers, not producers. It tries hard to ensure that EPUBs it produces actually work as intended on a wide variety of devices, a goal that is incompatible with producing valid EPUBs, and one that is far more important to the vast majority of its users. If you need a tool that always produces valid EPUBs, |app| is not for you. |app| does not guarantee that an EPUB produced by it is valid. The only
guarantee it makes is that if you feed it valid XHTML 1.1 + CSS 2.1 it will
output a valid EPUB. |app| is designed for ebook consumers, not producers. It
tries hard to ensure that EPUBs it produces actually work as intended on a wide
variety of devices, a goal that is incompatible with producing valid EPUBs, and
one that is far more important to the vast majority of its users. If you need a
tool that always produces valid EPUBs, |app| is not for you.
How do I use some of the advanced features of the conversion tools? How do I use some of the advanced features of the conversion tools?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can get help on any individual feature of the converters by mousing over it in the GUI or running ``ebook-convert dummy.html .epub -h`` at a terminal. A good place to start is to look at the following demo files that demonstrate some of the advanced features: You can get help on any individual feature of the converters by mousing over
* `html-demo.zip <http://calibre-ebook.com/downloads/html-demo.zip>`_ it in the GUI or running ``ebook-convert dummy.html .epub -h`` at a terminal.
A good place to start is to look at the following demo file that demonstrates
some of the advanced features
`html-demo.zip <http://calibre-ebook.com/downloads/html-demo.zip>`_
Device Integration Device Integration
@ -126,11 +145,11 @@ Device Integration
What devices does |app| support? What devices does |app| support?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|app| can directly connect to all the major (and most of the minor) ebook reading devices, |app| can directly connect to all the major (and most of the minor) ebook
smarthphones, tablets, etc. reading devices, smarthphones, tablets, etc. In addition, using the
In addition, using the :guilabel:`Connect to folder` function you can use it with any ebook reader that exports itself as a USB disk. :guilabel:`Connect to folder` function you can use it with any ebook reader
You can even connect to Apple devices (via iTunes), using the :guilabel:`Connect to iTunes` that exports itself as a USB disk. You can even connect to Apple devices (via
function. iTunes), using the :guilabel:`Connect to iTunes` function.
.. _devsupport: .. _devsupport:

12
recipes/astroflesz.recipe
View File

@ -2,12 +2,12 @@
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class Astroflesz(BasicNewsRecipe): class Astroflesz(BasicNewsRecipe):
title = u'Astroflesz' title = u'Astroflesz'
oldest_article = 7 oldest_article = 7
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'astroflesz.pl - to portal poświęcony astronomii. Informuje zarówno o aktualnych wydarzeniach i odkryciach naukowych, jak również zapowiada ciekawe zjawiska astronomiczne' description = u'astroflesz.pl - to portal poświęcony astronomii. Informuje zarówno o aktualnych wydarzeniach i odkryciach naukowych, jak również zapowiada ciekawe zjawiska astronomiczne'
category = 'astronomy' category = 'astronomy'
language = 'pl' language = 'pl'
cover_url = 'http://www.astroflesz.pl/templates/astroflesz/images/logo/logo.png' cover_url = 'http://www.astroflesz.pl/templates/astroflesz/images/logo/logo.png'
ignore_duplicate_articles = {'title', 'url'} ignore_duplicate_articles = {'title', 'url'}
max_articles_per_feed = 100 max_articles_per_feed = 100
@ -17,7 +17,7 @@ class Astroflesz(BasicNewsRecipe):
keep_only_tags = [dict(id="k2Container")] keep_only_tags = [dict(id="k2Container")]
remove_tags_after = dict(name='div', attrs={'class':'itemLinks'}) remove_tags_after = dict(name='div', attrs={'class':'itemLinks'})
remove_tags = [dict(name='div', attrs={'class':['itemLinks', 'itemToolbar', 'itemRatingBlock']})] remove_tags = [dict(name='div', attrs={'class':['itemLinks', 'itemToolbar', 'itemRatingBlock']})]
feeds = [(u'Wszystkie', u'http://astroflesz.pl/?format=feed')] feeds = [(u'Wszystkie', u'http://astroflesz.pl/?format=feed')]
def postprocess_html(self, soup, first_fetch): def postprocess_html(self, soup, first_fetch):
t = soup.find(attrs={'class':'itemIntroText'}) t = soup.find(attrs={'class':'itemIntroText'})

13
recipes/badania_net.recipe
View File

@ -1,17 +1,20 @@
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
import re
class BadaniaNet(BasicNewsRecipe): class BadaniaNet(BasicNewsRecipe):
title = u'badania.net' title = u'badania.net'
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'chcesz wiedzieć więcej?' description = u'chcesz wiedzieć więcej?'
category = 'science' category = 'science'
language = 'pl' language = 'pl'
cover_url = 'http://badania.net/wp-content/badanianet_green_transparent.png' cover_url = 'http://badania.net/wp-content/badanianet_green_transparent.png'
extra_css = '.alignleft {float:left; margin-right:5px;} .alignright {float:right; margin-left:5px;}'
oldest_article = 7 oldest_article = 7
max_articles_per_feed = 100 max_articles_per_feed = 100
no_stylesheets = True no_stylesheets = True
preprocess_regexps = [(re.compile(r"<h4>Tekst sponsoruje</h4>", re.IGNORECASE), lambda m: ''),]
remove_empty_feeds = True remove_empty_feeds = True
use_embedded_content = False use_embedded_content = False
remove_tags = [dict(attrs={'class':['omc-flex-category', 'omc-comment-count', 'omc-single-tags']})] remove_tags = [dict(attrs={'class':['omc-flex-category', 'omc-comment-count', 'omc-single-tags']})]
remove_tags_after = dict(attrs={'class':'omc-single-tags'}) remove_tags_after = dict(attrs={'class':'omc-single-tags'})
keep_only_tags = [dict(id='omc-full-article')] keep_only_tags = [dict(id='omc-full-article')]
feeds = [(u'Psychologia', u'http://badania.net/category/psychologia/feed/'), (u'Technologie', u'http://badania.net/category/technologie/feed/'), (u'Biologia', u'http://badania.net/category/biologia/feed/'), (u'Chemia', u'http://badania.net/category/chemia/feed/'), (u'Zdrowie', u'http://badania.net/category/zdrowie/'), (u'Seks', u'http://badania.net/category/psychologia-ewolucyjna-tematyka-seks/feed/')] feeds = [(u'Psychologia', u'http://badania.net/category/psychologia/feed/'), (u'Technologie', u'http://badania.net/category/technologie/feed/'), (u'Biologia', u'http://badania.net/category/biologia/feed/'), (u'Chemia', u'http://badania.net/category/chemia/feed/'), (u'Zdrowie', u'http://badania.net/category/zdrowie/'), (u'Seks', u'http://badania.net/category/psychologia-ewolucyjna-tematyka-seks/feed/')]

9
recipes/bwmagazine2.recipe
View File

@ -11,8 +11,8 @@ class BusinessWeekMagazine(BasicNewsRecipe):
category = 'news' category = 'news'
encoding = 'UTF-8' encoding = 'UTF-8'
keep_only_tags = [ keep_only_tags = [
dict(name='div', attrs={'id':'article_body_container'}), dict(name='div', attrs={'id':'article_body_container'}),
] ]
remove_tags = [dict(name='ui'),dict(name='li'),dict(name='div', attrs={'id':['share-email']})] remove_tags = [dict(name='ui'),dict(name='li'),dict(name='div', attrs={'id':['share-email']})]
no_javascript = True no_javascript = True
no_stylesheets = True no_stylesheets = True
@ -25,6 +25,7 @@ class BusinessWeekMagazine(BasicNewsRecipe):
#Find date #Find date
mag=soup.find('h2',text='Magazine') mag=soup.find('h2',text='Magazine')
self.log(mag)
dates=self.tag_to_string(mag.findNext('h3')) dates=self.tag_to_string(mag.findNext('h3'))
self.timefmt = u' [%s]'%dates self.timefmt = u' [%s]'%dates
@ -32,7 +33,7 @@ class BusinessWeekMagazine(BasicNewsRecipe):
div0 = soup.find ('div', attrs={'class':'column left'}) div0 = soup.find ('div', attrs={'class':'column left'})
section_title = '' section_title = ''
feeds = OrderedDict() feeds = OrderedDict()
for div in div0.findAll('h4'): for div in div0.findAll(['h4','h5']):
articles = [] articles = []
section_title = self.tag_to_string(div.findPrevious('h3')).strip() section_title = self.tag_to_string(div.findPrevious('h3')).strip()
title=self.tag_to_string(div.a).strip() title=self.tag_to_string(div.a).strip()
@ -48,7 +49,7 @@ class BusinessWeekMagazine(BasicNewsRecipe):
feeds[section_title] += articles feeds[section_title] += articles
div1 = soup.find ('div', attrs={'class':'column center'}) div1 = soup.find ('div', attrs={'class':'column center'})
section_title = '' section_title = ''
for div in div1.findAll('h5'): for div in div1.findAll(['h4','h5']):
articles = [] articles = []
desc=self.tag_to_string(div.findNext('p')).strip() desc=self.tag_to_string(div.findNext('p')).strip()
section_title = self.tag_to_string(div.findPrevious('h3')).strip() section_title = self.tag_to_string(div.findPrevious('h3')).strip()

38
recipes/eclipseonline.recipe Normal file
View File

@ -0,0 +1,38 @@
from calibre.web.feeds.news import BasicNewsRecipe
class EclipseOnline(BasicNewsRecipe):
#
# oldest_article specifies the maximum age, in days, of posts to retrieve.
# The default of 32 is intended to work well with a "days of month = 1"
# recipe schedule to download "monthly issues" of Eclipse Online.
# Increase this value to include additional posts. However, the RSS feed
# currently only includes the 10 most recent posts, so that's the max.
#
oldest_article = 32
title = u'Eclipse Online'
description = u'"Where strange and wonderful things happen, where reality is eclipsed for a little while with something magical and new." Eclipse Online is edited by Jonathan Strahan and published online by Night Shade Books. http://www.nightshadebooks.com/category/eclipse/'
publication_type = 'magazine'
language = 'en'
__author__ = u'Jim DeVona'
__version__ = '1.0'
# For now, use this Eclipse Online logo as the ebook cover image.
# (Disable the cover_url line to let Calibre generate a default cover, including date.)
cover_url = 'http://www.nightshadebooks.com/wp-content/uploads/2012/10/Eclipse-Logo.jpg'
# Extract the "post" div containing the story (minus redundant metadata) from each page.
keep_only_tags = [dict(name='div', attrs={'class':lambda x: x and 'post' in x})]
remove_tags = [dict(name='span', attrs={'class': ['post-author', 'post-category', 'small']})]
# Nice plain markup (like Eclipse's) works best for most e-readers.
# Disregard any special styling rules, but center illustrations.
auto_cleanup = False
no_stylesheets = True
remove_attributes = ['style', 'align']
extra_css = '.wp-caption {text-align: center;} .wp-caption-text {font-size: small; font-style: italic;}'
# Tell Calibre where to look for article links. It will proceed to retrieve
# these posts and format them into an ebook according to the above rules.
feeds = ['http://www.nightshadebooks.com/category/eclipse/feed/']

54
recipes/film_org_pl.recipe
View File

@ -1,20 +1,54 @@
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai # vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import Comment
import re import re
class FilmOrgPl(BasicNewsRecipe): class FilmOrgPl(BasicNewsRecipe):
title = u'Film.org.pl' title = u'Film.org.pl'
__author__ = 'fenuks' __author__ = 'fenuks'
description = u"Recenzje, analizy, artykuły, rankingi - wszystko o filmie dla miłośników kina. Opisy efektów specjalnych, wersji reżyserskich, remake'ów, sequeli. No i forum filmowe. Jedne z największych w Polsce." description = u"Recenzje, analizy, artykuły, rankingi - wszystko o filmie dla miłośników kina. Opisy efektów specjalnych, wersji reżyserskich, remake'ów, sequeli. No i forum filmowe. Jedne z największych w Polsce."
category = 'film' category = 'film'
language = 'pl' language = 'pl'
extra_css = '.alignright {float:right; margin-left:5px;} .alignleft {float:left; margin-right:5px;}' extra_css = '.alignright {float:right; margin-left:5px;} .alignleft {float:left; margin-right:5px;} .recenzja-title {font-size: 150%; margin-top: 5px; margin-bottom: 5px;}'
cover_url = 'http://film.org.pl/wp-content/themes/KMF/images/logo_kmf10.png' cover_url = 'http://film.org.pl/wp-content/themes/KMF/images/logo_kmf10.png'
ignore_duplicate_articles = {'title', 'url'} ignore_duplicate_articles = {'title', 'url'}
oldest_article = 7 oldest_article = 7
max_articles_per_feed = 100 max_articles_per_feed = 100
no_stylesheets = True no_stylesheets = True
remove_javascript = True
remove_empty_feeds = True remove_empty_feeds = True
use_embedded_content = True use_embedded_content = False
preprocess_regexps = [(re.compile(ur'<h3>Przeczytaj także:</h3>.*', re.IGNORECASE|re.DOTALL), lambda m: '</body>'), (re.compile(ur'<div>Artykuł</div>', re.IGNORECASE), lambda m: ''), (re.compile(ur'<div>Ludzie filmu</div>', re.IGNORECASE), lambda m: '')] remove_attributes = ['style']
remove_tags = [dict(name='img', attrs={'alt':['Ludzie filmu', u'Artykuł']})] preprocess_regexps = [(re.compile(ur'<h3>Przeczytaj także:</h3>.*', re.IGNORECASE|re.DOTALL), lambda m: '</body>'), (re.compile(ur'</?center>', re.IGNORECASE|re.DOTALL), lambda m: ''), (re.compile(ur'<div>Artykuł</div>', re.IGNORECASE), lambda m: ''), (re.compile(ur'<div>Ludzie filmu</div>', re.IGNORECASE), lambda m: ''), (re.compile(ur'(<br ?/?>\s*?){2,}', re.IGNORECASE|re.DOTALL), lambda m: '')]
feeds = [(u'Recenzje', u'http://film.org.pl/r/recenzje/feed/'), (u'Artyku\u0142', u'http://film.org.pl/a/artykul/feed/'), (u'Analiza', u'http://film.org.pl/a/analiza/feed/'), (u'Ranking', u'http://film.org.pl/a/ranking/feed/'), (u'Blog', u'http://film.org.pl/kmf/blog/feed/'), (u'Ludzie', u'http://film.org.pl/a/ludzie/feed/'), (u'Seriale', u'http://film.org.pl/a/seriale/feed/'), (u'Oceanarium', u'http://film.org.pl/a/ocenarium/feed/'), (u'VHS', u'http://film.org.pl/a/vhs-a/feed/')] keep_only_tags = [dict(name=['h11', 'h16', 'h17']), dict(attrs={'class':'editor'})]
remove_tags_after = dict(id='comments')
remove_tags = [dict(name=['link', 'meta', 'style']), dict(name='img', attrs={'alt':['Ludzie filmu', u'Artykuł']}), dict(id='comments'), dict(attrs={'style':'border: 0pt none ; margin: 0pt; padding: 0pt;'}), dict(name='p', attrs={'class':'rating'}), dict(attrs={'layout':'button_count'})]
feeds = [(u'Recenzje', u'http://film.org.pl/r/recenzje/feed/'), (u'Artyku\u0142', u'http://film.org.pl/a/artykul/feed/'), (u'Analiza', u'http://film.org.pl/a/analiza/feed/'), (u'Ranking', u'http://film.org.pl/a/ranking/feed/'), (u'Blog', u'http://film.org.pl/kmf/blog/feed/'), (u'Ludzie', u'http://film.org.pl/a/ludzie/feed/'), (u'Seriale', u'http://film.org.pl/a/seriale/feed/'), (u'Oceanarium', u'http://film.org.pl/a/ocenarium/feed/'), (u'VHS', u'http://film.org.pl/a/vhs-a/feed/')]
def append_page(self, soup, appendtag):
tag = soup.find('div', attrs={'class': 'pagelink'})
if tag:
for nexturl in tag.findAll('a'):
url = nexturl['href']
soup2 = self.index_to_soup(url)
pagetext = soup2.find(attrs={'class': 'editor'})
comments = pagetext.findAll(text=lambda text:isinstance(text, Comment))
for comment in comments:
comment.extract()
pos = len(appendtag.contents)
appendtag.insert(pos, pagetext)
for r in appendtag.findAll(attrs={'class': 'pagelink'}):
r.extract()
for r in appendtag.findAll(attrs={'id': 'comments'}):
r.extract()
for r in appendtag.findAll(attrs={'style':'border: 0pt none ; margin: 0pt; padding: 0pt;'}):
r.extract()
for r in appendtag.findAll(attrs={'layout':'button_count'}):
r.extract()
def preprocess_html(self, soup):
for c in soup.findAll('h11'):
c.name = 'h1'
self.append_page(soup, soup.body)
for r in soup.findAll('br'):
r.extract()
return soup

26
recipes/foreignaffairs.recipe
View File

@ -1,6 +1,5 @@
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
import re import re
from calibre.ptempfile import PersistentTemporaryFile
class ForeignAffairsRecipe(BasicNewsRecipe): class ForeignAffairsRecipe(BasicNewsRecipe):
''' there are three modifications: ''' there are three modifications:
@ -45,7 +44,6 @@ class ForeignAffairsRecipe(BasicNewsRecipe):
'publisher': publisher} 'publisher': publisher}
temp_files = [] temp_files = []
articles_are_obfuscated = True
def get_cover_url(self): def get_cover_url(self):
soup = self.index_to_soup(self.FRONTPAGE) soup = self.index_to_soup(self.FRONTPAGE)
@ -53,20 +51,6 @@ class ForeignAffairsRecipe(BasicNewsRecipe):
img_url = div.find('img')['src'] img_url = div.find('img')['src']
return self.INDEX + img_url return self.INDEX + img_url
def get_obfuscated_article(self, url):
br = self.get_browser()
br.open(url)
response = br.follow_link(url_regex = r'/print/[0-9]+', nr = 0)
html = response.read()
self.temp_files.append(PersistentTemporaryFile('_fa.html'))
self.temp_files[-1].write(html)
self.temp_files[-1].close()
return self.temp_files[-1].name
def parse_index(self): def parse_index(self):
answer = [] answer = []
@ -89,10 +73,10 @@ class ForeignAffairsRecipe(BasicNewsRecipe):
if div.find('a') is not None: if div.find('a') is not None:
originalauthor=self.tag_to_string(div.findNext('div', attrs = {'class':'views-field-field-article-book-nid'}).div.a) originalauthor=self.tag_to_string(div.findNext('div', attrs = {'class':'views-field-field-article-book-nid'}).div.a)
title=subsectiontitle+': '+self.tag_to_string(div.span.a)+' by '+originalauthor title=subsectiontitle+': '+self.tag_to_string(div.span.a)+' by '+originalauthor
url=self.INDEX+div.span.a['href'] url=self.INDEX+self.index_to_soup(self.INDEX+div.span.a['href']).find('a', attrs={'class':'fa_addthis_print'})['href']
atr=div.findNext('div', attrs = {'class': 'views-field-field-article-display-authors-value'}) atr=div.findNext('div', attrs = {'class': 'views-field-field-article-display-authors-value'})
if atr is not None: if atr is not None:
author=self.tag_to_string(atr.span.a) author=self.tag_to_string(atr.span)
else: else:
author='' author=''
desc=div.findNext('span', attrs = {'class': 'views-field-field-article-summary-value'}) desc=div.findNext('span', attrs = {'class': 'views-field-field-article-summary-value'})
@ -106,10 +90,10 @@ class ForeignAffairsRecipe(BasicNewsRecipe):
for div in sec.findAll('div', attrs = {'class': 'views-field-title'}): for div in sec.findAll('div', attrs = {'class': 'views-field-title'}):
if div.find('a') is not None: if div.find('a') is not None:
title=self.tag_to_string(div.span.a) title=self.tag_to_string(div.span.a)
url=self.INDEX+div.span.a['href'] url=self.INDEX+self.index_to_soup(self.INDEX+div.span.a['href']).find('a', attrs={'class':'fa_addthis_print'})['href']
atr=div.findNext('div', attrs = {'class': 'views-field-field-article-display-authors-value'}) atr=div.findNext('div', attrs = {'class': 'views-field-field-article-display-authors-value'})
if atr is not None: if atr is not None:
author=self.tag_to_string(atr.span.a) author=self.tag_to_string(atr.span)
else: else:
author='' author=''
desc=div.findNext('span', attrs = {'class': 'views-field-field-article-summary-value'}) desc=div.findNext('span', attrs = {'class': 'views-field-field-article-summary-value'})
@ -119,7 +103,7 @@ class ForeignAffairsRecipe(BasicNewsRecipe):
description='' description=''
articles.append({'title':title, 'date':None, 'url':url, 'description':description, 'author':author}) articles.append({'title':title, 'date':None, 'url':url, 'description':description, 'author':author})
if articles: if articles:
answer.append((section, articles)) answer.append((section, articles))
return answer return answer
def preprocess_html(self, soup): def preprocess_html(self, soup):

75
recipes/fortune_magazine.recipe Normal file
View File

@ -0,0 +1,75 @@
from calibre.web.feeds.recipes import BasicNewsRecipe
from collections import OrderedDict
class Fortune(BasicNewsRecipe):
title = 'Fortune Magazine'
__author__ = 'Rick Shang'
description = 'FORTUNE is a global business magazine that has been revered in its content and credibility since 1930. FORTUNE covers the entire field of business, including specific companies and business trends, prominent business leaders, and new ideas shaping the global marketplace.'
language = 'en'
category = 'news'
encoding = 'UTF-8'
keep_only_tags = [dict(attrs={'id':['storycontent']})]
remove_tags = [dict(attrs={'class':['hed_side','socialMediaToolbarContainer']})]
no_javascript = True
no_stylesheets = True
needs_subscription = True
def get_browser(self):
br = BasicNewsRecipe.get_browser(self)
br.open('http://money.cnn.com/2013/03/21/smallbusiness/legal-marijuana-startups.pr.fortune/index.html')
br.select_form(name="paywall-form")
br['email'] = self.username
br['password'] = self.password
br.submit()
return br
def parse_index(self):
articles = []
soup0 = self.index_to_soup('http://money.cnn.com/magazines/fortune/')
#Go to the latestissue
soup = self.index_to_soup(soup0.find('div',attrs={'class':'latestissue'}).find('a',href=True)['href'])
#Find cover & date
cover_item = soup.find('div', attrs={'id':'cover-story'})
cover = cover_item.find('img',src=True)
self.cover_url = cover['src']
date = self.tag_to_string(cover_item.find('div', attrs={'class':'tocDate'})).strip()
self.timefmt = u' [%s]'%date
feeds = OrderedDict()
section_title = ''
#checkout the cover story
articles = []
coverstory=soup.find('div', attrs={'class':'cnnHeadline'})
title=self.tag_to_string(coverstory.a).strip()
url=coverstory.a['href']
desc=self.tag_to_string(coverstory.findNext('p', attrs={'class':'cnnBlurbTxt'}))
articles.append({'title':title, 'url':url, 'description':desc, 'date':''})
feeds['Cover Story'] = []
feeds['Cover Story'] += articles
for post in soup.findAll('div', attrs={'class':'cnnheader'}):
section_title = self.tag_to_string(post).strip()
articles = []
ul=post.findNext('ul')
for link in ul.findAll('li'):
links=link.find('h2')
title=self.tag_to_string(links.a).strip()
url=links.a['href']
desc=self.tag_to_string(link.find('p', attrs={'class':'cnnBlurbTxt'}))
articles.append({'title':title, 'url':url, 'description':desc, 'date':''})
if articles:
if section_title not in feeds:
feeds[section_title] = []
feeds[section_title] += articles
ans = [(key, val) for key, val in feeds.iteritems()]
return ans

26
recipes/gofin_pl.recipe Normal file
View File

@ -0,0 +1,26 @@
#!/usr/bin/env python
__license__ = 'GPL v3'
__author__ = 'teepel <teepel44@gmail.com>'
'''
gofin.pl
'''
from calibre.web.feeds.news import BasicNewsRecipe
class gofin(BasicNewsRecipe):
title = u'Gofin'
__author__ = 'teepel <teepel44@gmail.com>'
language = 'pl'
description =u'Portal Podatkowo-Księgowy'
INDEX='http://gofin.pl'
oldest_article = 7
max_articles_per_feed = 100
remove_empty_feeds= True
simultaneous_downloads = 5
remove_javascript=True
no_stylesheets=True
auto_cleanup = True
feeds = [(u'Podatki', u'http://www.rss.gofin.pl/podatki.xml'), (u'Prawo Pracy', u'http://www.rss.gofin.pl/prawopracy.xml'), (u'Rachunkowo\u015b\u0107', u'http://www.rss.gofin.pl/rachunkowosc.xml'), (u'Sk\u0142adki, zasi\u0142ki, emerytury', u'http://www.rss.gofin.pl/zasilki.xml'),(u'Firma', u'http://www.rss.gofin.pl/firma.xml'), (u'Prawnik radzi', u'http://www.rss.gofin.pl/prawnikradzi.xml')]

18
recipes/gram_pl.recipe
View File

@ -2,22 +2,22 @@ from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup from calibre.ebooks.BeautifulSoup import BeautifulSoup
class Gram_pl(BasicNewsRecipe): class Gram_pl(BasicNewsRecipe):
title = u'Gram.pl' title = u'Gram.pl'
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'Serwis społecznościowy o grach: recenzje, newsy, zapowiedzi, encyklopedia gier, forum. Gry PC, PS3, X360, PS Vita, sprzęt dla graczy.' description = u'Serwis społecznościowy o grach: recenzje, newsy, zapowiedzi, encyklopedia gier, forum. Gry PC, PS3, X360, PS Vita, sprzęt dla graczy.'
category = 'games' category = 'games'
language = 'pl' language = 'pl'
oldest_article = 8 oldest_article = 8
index='http://www.gram.pl' index='http://www.gram.pl'
max_articles_per_feed = 100 max_articles_per_feed = 100
ignore_duplicate_articles = {'title', 'url'} ignore_duplicate_articles = {'title', 'url'}
no_stylesheets= True no_stylesheets= True
remove_empty_feeds = True remove_empty_feeds = True
#extra_css = 'h2 {font-style: italic; font-size:20px;} .picbox div {float: left;}' #extra_css = 'h2 {font-style: italic; font-size:20px;} .picbox div {float: left;}'
cover_url=u'http://www.gram.pl/www/01/img/grampl_zima.png' cover_url=u'http://www.gram.pl/www/01/img/grampl_zima.png'
keep_only_tags= [dict(id='articleModule')] keep_only_tags= [dict(id='articleModule')]
remove_tags = [dict(attrs={'class':['breadCrump', 'dymek', 'articleFooter', 'twitter-share-button']})] remove_tags = [dict(attrs={'class':['breadCrump', 'dymek', 'articleFooter', 'twitter-share-button']}), dict(name='aside')]
feeds = [(u'Informacje', u'http://www.gram.pl/feed_news.asp'), feeds = [(u'Informacje', u'http://www.gram.pl/feed_news.asp'),
(u'Publikacje', u'http://www.gram.pl/feed_news.asp?type=articles') (u'Publikacje', u'http://www.gram.pl/feed_news.asp?type=articles')
] ]
@ -46,4 +46,4 @@ class Gram_pl(BasicNewsRecipe):
tag=soup.find(name='span', attrs={'class':'platforma'}) tag=soup.find(name='span', attrs={'class':'platforma'})
if tag: if tag:
tag.name = 'p' tag.name = 'p'
return soup return soup

45
recipes/harpers_full.recipe
View File

@ -1,5 +1,5 @@
__license__ = 'GPL v3' __license__ = 'GPL v3'
__copyright__ = '2008-2013, Darko Miletic <darko.miletic at gmail.com>' __copyright__ = '2008-2012, Darko Miletic <darko.miletic at gmail.com>'
''' '''
harpers.org - paid subscription/ printed issue articles harpers.org - paid subscription/ printed issue articles
This recipe only get's article's published in text format This recipe only get's article's published in text format
@ -14,7 +14,7 @@ from calibre import strftime
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class Harpers_full(BasicNewsRecipe): class Harpers_full(BasicNewsRecipe):
title = "Harper's Magazine - articles from printed edition" title = "Harper's Magazine - Printed Edition"
__author__ = 'Darko Miletic' __author__ = 'Darko Miletic'
description = "Harper's Magazine, the oldest general-interest monthly in America, explores the issues that drive our national conversation, through long-form narrative journalism and essays, and such celebrated features as the iconic Harper's Index." description = "Harper's Magazine, the oldest general-interest monthly in America, explores the issues that drive our national conversation, through long-form narrative journalism and essays, and such celebrated features as the iconic Harper's Index."
publisher = "Harpers's" publisher = "Harpers's"
@ -29,6 +29,7 @@ class Harpers_full(BasicNewsRecipe):
needs_subscription = 'optional' needs_subscription = 'optional'
masthead_url = 'http://harpers.org/wp-content/themes/harpers/images/pheader.gif' masthead_url = 'http://harpers.org/wp-content/themes/harpers/images/pheader.gif'
publication_type = 'magazine' publication_type = 'magazine'
INDEX = ''
LOGIN = 'http://harpers.org/wp-content/themes/harpers/ajax_login.php' LOGIN = 'http://harpers.org/wp-content/themes/harpers/ajax_login.php'
extra_css = """ extra_css = """
body{font-family: adobe-caslon-pro,serif} body{font-family: adobe-caslon-pro,serif}
@ -65,41 +66,37 @@ class Harpers_full(BasicNewsRecipe):
def parse_index(self): def parse_index(self):
#find current issue #find current issue
soup = self.index_to_soup('http://harpers.org/') soup = self.index_to_soup('http://harpers.org/')
currentIssue=soup.find('div',attrs={'class':'mainNavi'}).find('li',attrs={'class':'curentIssue'}) currentIssue=soup.find('div',attrs={'class':'mainNavi'}).find('li',attrs={'class':'curentIssue'})
currentIssue_url=self.tag_to_string(currentIssue.a['href']) currentIssue_url=self.tag_to_string(currentIssue.a['href'])
self.log(currentIssue_url)
#go to the current issue #go to the current issue
soup1 = self.index_to_soup(currentIssue_url) soup1 = self.index_to_soup(currentIssue_url)
currentIssue_title = self.tag_to_string(soup1.head.title.string) date = re.split('\s\|\s',self.tag_to_string(soup1.head.title.string))[0]
date = re.split('\s\|\s',currentIssue_title)[0]
self.timefmt = u' [%s]'%date self.timefmt = u' [%s]'%date
#get cover #get cover
coverurl='http://harpers.org/wp-content/themes/harpers/ajax_microfiche.php?img=harpers-'+re.split('harpers.org/',currentIssue_url)[1]+'gif/0001.gif' self.cover_url = soup1.find('div', attrs = {'class':'picture_hp'}).find('img', src=True)['src']
soup2 = self.index_to_soup(coverurl)
self.cover_url = self.tag_to_string(soup2.find('img')['src'])
self.log(self.cover_url)
articles = [] articles = []
count = 0 count = 0
for item in soup1.findAll('div', attrs={'class':'articleData'}): for item in soup1.findAll('div', attrs={'class':'articleData'}):
text_links = item.findAll('h2') text_links = item.findAll('h2')
if text_links: for text_link in text_links:
for text_link in text_links: if count == 0:
if count == 0: count = 1
count = 1 else:
else: url = text_link.a['href']
url = text_link.a['href'] title = text_link.a.contents[0]
title = self.tag_to_string(text_link.a) date = strftime(' %B %Y')
date = strftime(' %B %Y') articles.append({
articles.append({ 'title' :title
'title' :title ,'date' :date
,'date' :date ,'url' :url
,'url' :url ,'description':''
,'description':'' })
}) return [(soup1.head.title.string, articles)]
return [(currentIssue_title, articles)]
def print_version(self, url): def print_version(self, url):
return url + '?single=1' return url + '?single=1'

59
recipes/hindu.recipe
View File

@ -2,7 +2,6 @@ from __future__ import with_statement
__license__ = 'GPL 3' __license__ = 'GPL 3'
__copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>' __copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
import time
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class TheHindu(BasicNewsRecipe): class TheHindu(BasicNewsRecipe):
@ -14,44 +13,42 @@ class TheHindu(BasicNewsRecipe):
max_articles_per_feed = 100 max_articles_per_feed = 100
no_stylesheets = True no_stylesheets = True
keep_only_tags = [dict(id='content')] auto_cleanup = True
remove_tags = [dict(attrs={'class':['article-links', 'breadcr']}),
dict(id=['email-section', 'right-column', 'printfooter', 'topover',
'slidebox', 'th_footer'])]
extra_css = '.photo-caption { font-size: smaller }' extra_css = '.photo-caption { font-size: smaller }'
def preprocess_raw_html(self, raw, url):
return raw.replace('<body><p>', '<p>').replace('</p></body>', '</p>')
def postprocess_html(self, soup, first_fetch):
for t in soup.findAll(['table', 'tr', 'td','center']):
t.name = 'div'
return soup
def parse_index(self): def parse_index(self):
today = time.strftime('%Y-%m-%d') soup = self.index_to_soup('http://www.thehindu.com/todays-paper/')
soup = self.index_to_soup( div = soup.find('div', attrs={'id':'left-column'})
'http://www.thehindu.com/todays-paper/tp-index/?date=' + today) soup.find(id='subnav-tpbar').extract()
div = soup.find(id='left-column')
feeds = []
current_section = None current_section = None
current_articles = [] current_articles = []
for x in div.findAll(['h3', 'div']): feeds = []
if current_section and x.get('class', '') == 'tpaper': for x in div.findAll(['a', 'span']):
a = x.find('a', href=True) if x.name == 'span' and x['class'] == 's-link':
if a is not None: # Section heading found
title = self.tag_to_string(a) if current_articles and current_section:
self.log('\tFound article:', title)
current_articles.append({'url':a['href']+'?css=print',
'title':title, 'date': '',
'description':''})
if x.name == 'h3':
if current_section and current_articles:
feeds.append((current_section, current_articles)) feeds.append((current_section, current_articles))
current_section = self.tag_to_string(x) current_section = self.tag_to_string(x)
self.log('Found section:', current_section)
current_articles = [] current_articles = []
self.log('\tFound section:', current_section)
elif x.name == 'a':
title = self.tag_to_string(x)
url = x.get('href', False)
if not url or not title:
continue
self.log('\t\tFound article:', title)
self.log('\t\t\t', url)
current_articles.append({'title': title, 'url':url,
'description':'', 'date':''})
if current_articles and current_section:
feeds.append((current_section, current_articles))
return feeds return feeds

31
recipes/historia_pl.recipe
View File

@ -1,27 +1,22 @@
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class Historia_org_pl(BasicNewsRecipe): class Historia_org_pl(BasicNewsRecipe):
title = u'Historia.org.pl' title = u'Historia.org.pl'
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'Artykuły dotyczące historii w układzie epok i tematów, forum. Najlepsza strona historii. Matura z historii i egzamin gimnazjalny z historii.' description = u'Artykuły dotyczące historii w układzie epok i tematów, forum. Najlepsza strona historii. Matura z historii i egzamin gimnazjalny z historii.'
cover_url = 'http://lh3.googleusercontent.com/_QeRQus12wGg/TOvHsZ2GN7I/AAAAAAAAD_o/LY1JZDnq7ro/logo5.jpg' cover_url = 'http://lh3.googleusercontent.com/_QeRQus12wGg/TOvHsZ2GN7I/AAAAAAAAD_o/LY1JZDnq7ro/logo5.jpg'
category = 'history' category = 'history'
language = 'pl' language = 'pl'
oldest_article = 8 oldest_article = 8
extra_css = 'img {float: left; margin-right: 10px;} .alignleft {float: left; margin-right: 10px;}'
remove_empty_feeds= True remove_empty_feeds= True
no_stylesheets = True no_stylesheets = True
use_embedded_content = True use_embedded_content = True
max_articles_per_feed = 100 max_articles_per_feed = 100
ignore_duplicate_articles = {'title', 'url'} ignore_duplicate_articles = {'title', 'url'}
feeds = [(u'Wszystkie', u'http://historia.org.pl/feed/'),
(u'Wiadomości', u'http://historia.org.pl/Kategoria/wiadomosci/feed/'),
feeds = [(u'Wszystkie', u'http://historia.org.pl/feed/'), (u'Publikacje', u'http://historia.org.pl/Kategoria/artykuly/feed/'),
(u'Wiadomości', u'http://historia.org.pl/Kategoria/wiadomosci/feed/'), (u'Publicystyka', u'http://historia.org.pl/Kategoria/publicystyka/feed/'),
(u'Publikacje', u'http://historia.org.pl/Kategoria/artykuly/feed/'), (u'Recenzje', u'http://historia.org.pl/Kategoria/recenzje/feed/'),
(u'Publicystyka', u'http://historia.org.pl/Kategoria/publicystyka/feed/'), (u'Projekty', u'http://historia.org.pl/Kategoria/projekty/feed/'),]
(u'Recenzje', u'http://historia.org.pl/Kategoria/recenzje/feed/'),
(u'Projekty', u'http://historia.org.pl/Kategoria/projekty/feed/'),]
def print_version(self, url):
return url + '?tmpl=component&print=1&layout=default&page='

9
recipes/history_today.recipe
View File

@ -1,6 +1,6 @@
import re
from calibre.web.feeds.recipes import BasicNewsRecipe
from collections import OrderedDict from collections import OrderedDict
import re
from calibre.web.feeds.news import BasicNewsRecipe
class HistoryToday(BasicNewsRecipe): class HistoryToday(BasicNewsRecipe):
@ -19,7 +19,6 @@ class HistoryToday(BasicNewsRecipe):
needs_subscription = True needs_subscription = True
def get_browser(self): def get_browser(self):
br = BasicNewsRecipe.get_browser(self) br = BasicNewsRecipe.get_browser(self)
if self.username is not None and self.password is not None: if self.username is not None and self.password is not None:
@ -46,8 +45,9 @@ class HistoryToday(BasicNewsRecipe):
#Go to issue #Go to issue
soup = self.index_to_soup('http://www.historytoday.com/contents') soup = self.index_to_soup('http://www.historytoday.com/contents')
cover = soup.find('div',attrs={'id':'content-area'}).find('img')['src'] cover = soup.find('div',attrs={'id':'content-area'}).find('img', attrs={'src':re.compile('.*cover.*')})['src']
self.cover_url=cover self.cover_url=cover
self.log(self.cover_url)
#Go to the main body #Go to the main body
@ -84,4 +84,3 @@ class HistoryToday(BasicNewsRecipe):
def cleanup(self): def cleanup(self):
self.browser.open('http://www.historytoday.com/logout') self.browser.open('http://www.historytoday.com/logout')

BIN
recipes/icons/gofin_pl.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 618 B

BIN
recipes/icons/histmag.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 537 B

BIN
recipes/icons/kdefamily_pl.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 857 B

BIN
recipes/icons/km_blog.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 532 B

BIN
recipes/icons/optyczne_pl.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 697 B

BIN
recipes/icons/sport_pl.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 627 B

23
recipes/infra_pl.recipe
View File

@ -1,21 +1,20 @@
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class INFRA(BasicNewsRecipe): class INFRA(BasicNewsRecipe):
title = u'INFRA' title = u'INFRA'
oldest_article = 7 oldest_article = 7
max_articles_per_feed = 100 max_articles_per_feed = 100
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'Serwis Informacyjny INFRA - UFO, Zjawiska Paranormalne, Duchy, Tajemnice świata.' description = u'Serwis Informacyjny INFRA - UFO, Zjawiska Paranormalne, Duchy, Tajemnice świata.'
cover_url = 'http://npn.nazwa.pl/templates/ja_teline_ii/images/logo.jpg' cover_url = 'http://i.imgur.com/j7hJT.jpg'
category = 'UFO' category = 'UFO'
index='http://infra.org.pl' index='http://infra.org.pl'
language = 'pl' language = 'pl'
max_articles_per_feed = 100 max_articles_per_feed = 100
no_stylesheers=True remove_attrs = ['style']
remove_tags_before=dict(name='h2', attrs={'class':'contentheading'}) no_stylesheets = True
remove_tags_after=dict(attrs={'class':'pagenav'}) keep_only_tags = [dict(id='ja-current-content')]
remove_tags=[dict(attrs={'class':'pagenav'})] feeds = [(u'Najnowsze wiadomo\u015bci', u'http://www.infra.org.pl/rss')]
feeds = [(u'Najnowsze wiadomo\u015bci', u'http://www.infra.org.pl/rss')]
def preprocess_html(self, soup): def preprocess_html(self, soup):
for item in soup.findAll(style=True): for item in soup.findAll(style=True):
@ -23,4 +22,4 @@ class INFRA(BasicNewsRecipe):
for a in soup('a'): for a in soup('a'):
if a.has_key('href') and 'http://' not in a['href'] and 'https://' not in a['href']: if a.has_key('href') and 'http://' not in a['href'] and 'https://' not in a['href']:
a['href']=self.index + a['href'] a['href']=self.index + a['href']
return soup return soup

49
recipes/interia_fakty.recipe
View File

@ -1,23 +1,24 @@
#!/usr/bin/env python #!/usr/bin/env python
__license__ = 'GPL v3' __license__ = 'GPL v3'
__copyright__ = u'2010, Tomasz Dlugosz <tomek3d@gmail.com>' __copyright__ = u'2010-2013, Tomasz Dlugosz <tomek3d@gmail.com>'
''' '''
fakty.interia.pl fakty.interia.pl
''' '''
import re
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class InteriaFakty(BasicNewsRecipe): class InteriaFakty(BasicNewsRecipe):
title = u'Interia.pl - Fakty' title = u'Interia.pl - Fakty'
description = u'Fakty ze strony interia.pl' description = u'Fakty ze strony interia.pl'
language = 'pl' language = 'pl'
oldest_article = 7 oldest_article = 1
__author__ = u'Tomasz D\u0142ugosz' __author__ = u'Tomasz D\u0142ugosz'
simultaneous_downloads = 2
no_stylesheets = True no_stylesheets = True
remove_javascript = True remove_javascript = True
max_articles_per_feed = 100 remove_empty_feeds= True
use_embedded_content = False
ignore_duplicate_articles = {'title', 'url'}
feeds = [(u'Kraj', u'http://kanaly.rss.interia.pl/kraj.xml'), feeds = [(u'Kraj', u'http://kanaly.rss.interia.pl/kraj.xml'),
(u'\u015awiat', u'http://kanaly.rss.interia.pl/swiat.xml'), (u'\u015awiat', u'http://kanaly.rss.interia.pl/swiat.xml'),
@ -26,14 +27,36 @@ class InteriaFakty(BasicNewsRecipe):
(u'Wywiady', u'http://kanaly.rss.interia.pl/wywiady.xml'), (u'Wywiady', u'http://kanaly.rss.interia.pl/wywiady.xml'),
(u'Ciekawostki', u'http://kanaly.rss.interia.pl/ciekawostki.xml')] (u'Ciekawostki', u'http://kanaly.rss.interia.pl/ciekawostki.xml')]
keep_only_tags = [dict(name='div', attrs={'id':'article'})] keep_only_tags = [
dict(name='h1'),
dict(name='div', attrs={'class': ['lead textContent', 'text textContent', 'source']})]
remove_tags = [ remove_tags = [dict(name='div', attrs={'class':['embed embedAd', 'REMOVE', 'boxHeader']})]
dict(name='div', attrs={'class':'box fontSizeSwitch'}),
dict(name='div', attrs={'class':'clear'}), preprocess_regexps = [
dict(name='div', attrs={'class':'embed embedLeft articleEmbedArticleList articleEmbedArticleListTitle'}), (re.compile(i[0], re.IGNORECASE | re.DOTALL), i[1]) for i in
dict(name='span', attrs={'class':'keywords'})] [
(r'embed embed(Left|Right|Center) articleEmbed(Audio|Wideo articleEmbedVideo|ArticleFull|ArticleTitle|ArticleListTitle|AlbumHorizontal)">', lambda match: 'REMOVE">'),
(r'</div> <div class="source">', lambda match: ''),
(r'<p><a href="http://forum.interia.pl.*?</a></p>', lambda match: '')
]
]
def get_article_url(self, article):
link = article.get('link', None)
if link and 'galerie' not in link and link.split('/')[-1]=="story01.htm":
link=link.split('/')[-2]
encoding = {'0B': '.', '0C': '/', '0A': '0', '0F': '=', '0G': '&',
'0D': '?', '0E': '-', '0H': ',', '0I': '_', '0N': '.com', '0L': 'http://'}
for k, v in encoding.iteritems():
link = link.replace(k, v)
return link
def print_version(self, url):
chunks = url.split(',')
return chunks[0] + '/podglad-wydruku'+ ',' + ','.join(chunks[1:])
extra_css = ''' extra_css = '''
h2 { font-size: 1.2em; } h1 { font-size:130% }
''' div.info { font-style:italic; font-size:70%}
'''

74
recipes/interia_sport.recipe
View File

@ -1,7 +1,7 @@
#!/usr/bin/env python #!/usr/bin/env python
__license__ = 'GPL v3' __license__ = 'GPL v3'
__copyright__ = u'2010, Tomasz Dlugosz <tomek3d@gmail.com>' __copyright__ = u'2010-2013, Tomasz Dlugosz <tomek3d@gmail.com>'
''' '''
sport.interia.pl sport.interia.pl
''' '''
@ -13,61 +13,51 @@ class InteriaSport(BasicNewsRecipe):
title = u'Interia.pl - Sport' title = u'Interia.pl - Sport'
description = u'Sport ze strony interia.pl' description = u'Sport ze strony interia.pl'
language = 'pl' language = 'pl'
oldest_article = 7 oldest_article = 1
__author__ = u'Tomasz D\u0142ugosz' __author__ = u'Tomasz D\u0142ugosz'
simultaneous_downloads = 3
no_stylesheets = True no_stylesheets = True
remove_javascript = True remove_javascript = True
max_articles_per_feed = 100 remove_empty_feeds= True
use_embedded_content = False
ignore_duplicate_articles = {'title', 'url'}
feeds = [(u'Wydarzenia sportowe', u'http://kanaly.rss.interia.pl/sport.xml'), feeds = [(u'Wydarzenia sportowe', u'http://kanaly.rss.interia.pl/sport.xml'),
(u'Pi\u0142ka no\u017cna', u'http://kanaly.rss.interia.pl/pilka_nozna.xml'), (u'Pi\u0142ka no\u017cna', u'http://kanaly.rss.interia.pl/pilka_nozna.xml'),
(u'Siatk\xf3wka', u'http://kanaly.rss.interia.pl/siatkowka.xml'),
(u'Koszyk\xf3wka', u'http://kanaly.rss.interia.pl/koszykowka.xml'), (u'Koszyk\xf3wka', u'http://kanaly.rss.interia.pl/koszykowka.xml'),
(u'NBA', u'http://kanaly.rss.interia.pl/nba.xml'),
(u'Kolarstwo', u'http://kanaly.rss.interia.pl/kolarstwo.xml'),
(u'\u017bu\u017cel', u'http://kanaly.rss.interia.pl/zuzel.xml'),
(u'Tenis', u'http://kanaly.rss.interia.pl/tenis.xml')] (u'Tenis', u'http://kanaly.rss.interia.pl/tenis.xml')]
keep_only_tags = [dict(name='div', attrs={'id':'article'})] keep_only_tags = [
dict(name='h1'),
dict(name='div', attrs={'class': ['lead textContent', 'text textContent', 'source']})]
remove_tags = [dict(name='div', attrs={'class':'object gallery'}), remove_tags = [dict(name='div', attrs={'class':['embed embedAd', 'REMOVE', 'boxHeader']})]
dict(name='div', attrs={'class':'box fontSizeSwitch'})]
extra_css = '''
.articleDate {
font-size: 0.5em;
color: black;
}
.articleFoto {
display: block;
font-family: sans;
font-size: 0.5em;
text-indent: 0
color: black;
}
.articleText {
display: block;
margin-bottom: 1em;
margin-left: 0;
margin-right: 0;
margin-top: 1em
color: black;
}
.articleLead {
font-size: 1.2em;
}
'''
preprocess_regexps = [ preprocess_regexps = [
(re.compile(i[0], re.IGNORECASE | re.DOTALL), i[1]) for i in (re.compile(i[0], re.IGNORECASE | re.DOTALL), i[1]) for i in
[ [
(r'<p><a href.*?</a></p>', lambda match: ''), (r'<p><a href.*?</a></p>', lambda match: ''),
# FIXME (r'<p>(<i>)?<b>(ZOBACZ|CZYTAJ) T.*?</div>', lambda match: '</div>'),
#(r'(<div id="newsAddContent">)(.*?)(<a href=".*">)(.*?)(</a>)', lambda match: '\1\2\4'), (r'embed embed(Left|Right|Center) articleEmbed(Audio|Wideo articleEmbedVideo|ArticleFull|ArticleTitle|ArticleListTitle|AlbumHorizontal)">', lambda match: 'REMOVE">'),
(r'<p>(<i>)?<b>(ZOBACZ|CZYTAJ) T.*?</div>', lambda match: '</div>') (r'</div> <div class="source">', lambda match: ''),
(r'<p><a href="http://forum.interia.pl.*?</a></p>', lambda match: '')
] ]
] ]
def get_article_url(self, article):
link = article.get('link', None)
if link and 'galerie' not in link and link.split('/')[-1]=="story01.htm":
link=link.split('/')[-2]
encoding = {'0B': '.', '0C': '/', '0A': '0', '0F': '=', '0G': '&',
'0D': '?', '0E': '-', '0H': ',', '0I': '_', '0N': '.com', '0L': 'http://'}
for k, v in encoding.iteritems():
link = link.replace(k, v)
return link
def print_version(self, url):
chunks = url.split(',')
return chunks[0] + '/podglad-wydruku'+ ',' + ','.join(chunks[1:])
extra_css = '''
h1 { font-size:130% }
div.info { font-style:italic; font-size:70%}
'''

85
recipes/irish_times.recipe
View File

@ -1,65 +1,62 @@
__license__ = 'GPL v3' __license__ = 'GPL v3'
__copyright__ = "2008, Derry FitzGerald. 2009 Modified by Ray Kinsella and David O'Callaghan, 2011 Modified by Phil Burns" __copyright__ = "2008, Derry FitzGerald. 2009 Modified by Ray Kinsella and David O'Callaghan, 2011 Modified by Phil Burns, 2013 Tom Scholl"
''' '''
irishtimes.com irishtimes.com
''' '''
import re import urlparse, re
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ptempfile import PersistentTemporaryFile
class IrishTimes(BasicNewsRecipe): class IrishTimes(BasicNewsRecipe):
title = u'The Irish Times' title = u'The Irish Times'
encoding = 'ISO-8859-15' __author__ = "Derry FitzGerald, Ray Kinsella, David O'Callaghan and Phil Burns, Tom Scholl"
__author__ = "Derry FitzGerald, Ray Kinsella, David O'Callaghan and Phil Burns"
language = 'en_IE' language = 'en_IE'
timefmt = ' (%A, %B %d, %Y)'
masthead_url = 'http://www.irishtimes.com/assets/images/generic/website/logo_theirishtimes.png'
encoding = 'utf-8'
oldest_article = 1.0 oldest_article = 1.0
max_articles_per_feed = 100 max_articles_per_feed = 100
remove_empty_feeds = True
no_stylesheets = True no_stylesheets = True
simultaneous_downloads= 5 temp_files = []
articles_are_obfuscated = True
r = re.compile('.*(?P<url>http:\/\/(www.irishtimes.com)|(rss.feedsportal.com\/c)\/.*\.html?).*')
remove_tags = [dict(name='div', attrs={'class':'footer'})]
extra_css = 'p, div { margin: 0pt; border: 0pt; text-indent: 0.5em } .headline {font-size: large;} \n .fact { padding-top: 10pt }'
feeds = [ feeds = [
('Frontpage', 'http://www.irishtimes.com/feeds/rss/newspaper/index.rss'), ('News', 'http://www.irishtimes.com/cmlink/the-irish-times-news-1.1319192'),
('Ireland', 'http://www.irishtimes.com/feeds/rss/newspaper/ireland.rss'), ('World', 'http://www.irishtimes.com/cmlink/irishtimesworldfeed-1.1321046'),
('World', 'http://www.irishtimes.com/feeds/rss/newspaper/world.rss'), ('Politics', 'http://www.irishtimes.com/cmlink/irish-times-politics-rss-1.1315953'),
('Finance', 'http://www.irishtimes.com/feeds/rss/newspaper/finance.rss'), ('Business', 'http://www.irishtimes.com/cmlink/the-irish-times-business-1.1319195'),
('Features', 'http://www.irishtimes.com/feeds/rss/newspaper/features.rss'), ('Culture', 'http://www.irishtimes.com/cmlink/the-irish-times-culture-1.1319213'),
('Sport', 'http://www.irishtimes.com/feeds/rss/newspaper/sport.rss'), ('Sport', 'http://www.irishtimes.com/cmlink/the-irish-times-sport-1.1319194'),
('Opinion', 'http://www.irishtimes.com/feeds/rss/newspaper/opinion.rss'), ('Debate', 'http://www.irishtimes.com/cmlink/debate-1.1319211'),
('Letters', 'http://www.irishtimes.com/feeds/rss/newspaper/letters.rss'), ('Life & Style', 'http://www.irishtimes.com/cmlink/the-irish-times-life-style-1.1319214'),
('Magazine', 'http://www.irishtimes.com/feeds/rss/newspaper/magazine.rss'),
('Health', 'http://www.irishtimes.com/feeds/rss/newspaper/health.rss'),
('Education & Parenting', 'http://www.irishtimes.com/feeds/rss/newspaper/education.rss'),
('Motors', 'http://www.irishtimes.com/feeds/rss/newspaper/motors.rss'),
('An Teanga Bheo', 'http://www.irishtimes.com/feeds/rss/newspaper/anteangabheo.rss'),
('Commercial Property', 'http://www.irishtimes.com/feeds/rss/newspaper/commercialproperty.rss'),
('Science Today', 'http://www.irishtimes.com/feeds/rss/newspaper/sciencetoday.rss'),
('Property', 'http://www.irishtimes.com/feeds/rss/newspaper/property.rss'),
('The Tickets', 'http://www.irishtimes.com/feeds/rss/newspaper/theticket.rss'),
('Weekend', 'http://www.irishtimes.com/feeds/rss/newspaper/weekend.rss'),
('News features', 'http://www.irishtimes.com/feeds/rss/newspaper/newsfeatures.rss'),
('Obituaries', 'http://www.irishtimes.com/feeds/rss/newspaper/obituaries.rss'),
] ]
def print_version(self, url): def get_obfuscated_article(self, url):
if url.count('rss.feedsportal.com'): # Insert a pic from the original url, but use content from the print url
#u = url.replace('0Bhtml/story01.htm','_pf0Bhtml/story01.htm') pic = None
u = url.find('irishtimes') pics = self.index_to_soup(url)
u = 'http://www.irishtimes.com' + url[u + 12:] div = pics.find('div', {'class' : re.compile('image-carousel')})
u = u.replace('0C', '/') if div:
u = u.replace('A', '') pic = div.img
u = u.replace('0Bhtml/story01.htm', '_pf.html') if pic:
else: try:
u = url.replace('.html','_pf.html') pic['src'] = urlparse.urljoin(url, pic['src'])
return u pic.extract()
except:
pic = None
content = self.index_to_soup(url + '?mode=print&ot=example.AjaxPageLayout.ot')
if pic:
content.p.insert(0, pic)
self.temp_files.append(PersistentTemporaryFile('_fa.html'))
self.temp_files[-1].write(content.prettify())
self.temp_files[-1].close()
return self.temp_files[-1].name
def get_article_url(self, article):
return article.link

14
recipes/kdefamily_pl.recipe
View File

@ -1,14 +1,16 @@
import re
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class KDEFamilyPl(BasicNewsRecipe): class KDEFamilyPl(BasicNewsRecipe):
title = u'KDEFamily.pl' title = u'KDEFamily.pl'
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'KDE w Polsce' description = u'KDE w Polsce'
category = 'open source, KDE' category = 'open source, KDE'
language = 'pl' language = 'pl'
cover_url = 'http://www.mykde.home.pl/kdefamily/wp-content/uploads/2012/07/logotype-e1341585198616.jpg' cover_url = 'http://www.mykde.home.pl/kdefamily/wp-content/uploads/2012/07/logotype-e1341585198616.jpg'
oldest_article = 7 oldest_article = 7
max_articles_per_feed = 100 max_articles_per_feed = 100
preprocess_regexps = [(re.compile(r"Podobne wpisy.*", re.IGNORECASE|re.DOTALL), lambda m: '')]
no_stylesheets = True no_stylesheets = True
use_embedded_content = True use_embedded_content = True
feeds = [(u'Wszystko', u'http://kdefamily.pl/feed/')] feeds = [(u'Wszystko', u'http://kdefamily.pl/feed/')]

36
recipes/km_blog.recipe Normal file
View File

@ -0,0 +1,36 @@
__license__ = 'GPL v3'
__author__ = 'teepel <teepel44@gmail.com>, Artur Stachecki <artur.stachecki@gmail.com>'
'''
korwin-mikke.pl/blog
'''
from calibre.web.feeds.news import BasicNewsRecipe
class km_blog(BasicNewsRecipe):
title = u'Korwin-Mikke Blog'
__author__ = 'teepel <teepel44@gmail.com>'
language = 'pl'
description ='Wiadomości z bloga korwin-mikke.pl/blog'
INDEX='http://korwin-mikke.pl/blog'
remove_empty_feeds= True
oldest_article = 7
max_articles_per_feed = 100
remove_javascript=True
no_stylesheets=True
remove_empty_feeds = True
feeds = [(u'blog', u'http://korwin-mikke.pl/blog/rss')]
keep_only_tags =[]
#this line should show title of the article, but it doesnt work
keep_only_tags.append(dict(name = 'div', attrs = {'class' : 'posts view'}))
keep_only_tags.append(dict(name = 'div', attrs = {'class' : 'text'}))
keep_only_tags.append(dict(name = 'h1'))
remove_tags =[]
remove_tags.append(dict(name = 'p', attrs = {'class' : 'float_right'}))
remove_tags.append(dict(name = 'p', attrs = {'class' : 'date'}))
remove_tags_after=[(dict(name = 'div', attrs = {'class': 'text'}))]

23
recipes/konflikty_zbrojne.recipe
View File

@ -3,10 +3,10 @@ from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup from calibre.ebooks.BeautifulSoup import BeautifulSoup
class Konflikty(BasicNewsRecipe): class Konflikty(BasicNewsRecipe):
title = u'Konflikty Zbrojne' title = u'Konflikty Zbrojne'
__author__ = 'fenuks' __author__ = 'fenuks'
cover_url = 'http://www.konflikty.pl/images/tapety_logo.jpg' cover_url = 'http://www.konflikty.pl/images/tapety_logo.jpg'
language = 'pl' language = 'pl'
description = u'Zbiór ciekawych artykułów historycznych, militarnych oraz recenzji książek, gier i filmów. Najświeższe informacje o lotnictwie, wojskach lądowych i polityce.' description = u'Zbiór ciekawych artykułów historycznych, militarnych oraz recenzji książek, gier i filmów. Najświeższe informacje o lotnictwie, wojskach lądowych i polityce.'
category='military, history' category='military, history'
oldest_article = 7 oldest_article = 7
@ -14,19 +14,20 @@ class Konflikty(BasicNewsRecipe):
no_stylesheets = True no_stylesheets = True
keep_only_tags=[dict(attrs={'class':['title1', 'image']}), dict(id='body')] keep_only_tags=[dict(attrs={'class':['title1', 'image']}), dict(id='body')]
feeds = [(u'Aktualności', u'http://www.konflikty.pl/rss_aktualnosci_10.xml'), feeds = [(u'Aktualności', u'http://www.konflikty.pl/rss_aktualnosci_10.xml'),
(u'Historia', u'http://www.konflikty.pl/rss_historia_10.xml'), (u'Historia', u'http://www.konflikty.pl/rss_historia_10.xml'),
(u'Militaria', u'http://www.konflikty.pl/rss_militaria_10.xml'), (u'Militaria', u'http://www.konflikty.pl/rss_militaria_10.xml'),
(u'Relacje', u'http://www.konflikty.pl/rss_relacje_10.xml'), (u'Relacje', u'http://www.konflikty.pl/rss_relacje_10.xml'),
(u'Recenzje', u'http://www.konflikty.pl/rss_recenzje_10.xml'), (u'Recenzje', u'http://www.konflikty.pl/rss_recenzje_10.xml'),
(u'Teksty źródłowe', u'http://www.konflikty.pl/rss_tekstyzrodlowe_10.xml')] (u'Teksty źródłowe', u'http://www.konflikty.pl/rss_tekstyzrodlowe_10.xml')]
def preprocess_html(self, soup): def preprocess_html(self, soup):
for item in soup.findAll(style=True): for item in soup.findAll(style=True):
del item['style'] del item['style']
for image in soup.findAll(name='a', attrs={'class':'image'}): for image in soup.findAll(name='a', attrs={'class':'image'}):
image['style'] = 'width: 210px; float: left; margin-right:5px;'
if image.img and image.img.has_key('alt'): if image.img and image.img.has_key('alt'):
image.name='div' image.name='div'
pos = len(image.contents) pos = len(image.contents)
image.insert(pos, BeautifulSoup('<p style="font-style:italic;">'+image.img['alt']+'</p>')) image.insert(pos, BeautifulSoup('<p style="font-style:italic;">'+image.img['alt']+'</p>'))
return soup return soup

21
recipes/kosmonauta_pl.recipe
View File

@ -2,12 +2,13 @@
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class Kosmonauta(BasicNewsRecipe): class Kosmonauta(BasicNewsRecipe):
title = u'Kosmonauta.net' title = u'Kosmonauta.net'
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'polskojęzyczny portal w całości dedykowany misjom kosmicznym i badaniom kosmosu.' description = u'polskojęzyczny portal w całości dedykowany misjom kosmicznym i badaniom kosmosu.'
category = 'astronomy' category = 'astronomy'
language = 'pl' language = 'pl'
cover_url = 'http://bi.gazeta.pl/im/4/10393/z10393414X,Kosmonauta-net.jpg' cover_url = 'http://bi.gazeta.pl/im/4/10393/z10393414X,Kosmonauta-net.jpg'
extra_css = '.thumbnail {float:left;margin-right:5px;}'
no_stylesheets = True no_stylesheets = True
INDEX = 'http://www.kosmonauta.net' INDEX = 'http://www.kosmonauta.net'
oldest_article = 7 oldest_article = 7
@ -16,9 +17,12 @@ class Kosmonauta(BasicNewsRecipe):
remove_attributes = ['style'] remove_attributes = ['style']
max_articles_per_feed = 100 max_articles_per_feed = 100
keep_only_tags = [dict(name='div', attrs={'class':'item-page'})] keep_only_tags = [dict(name='div', attrs={'class':'item-page'})]
remove_tags = [dict(attrs={'class':['article-tools clearfix', 'cedtag', 'nav clearfix', 'jwDisqusForm']})] remove_tags = [dict(attrs={'class':['article-tools clearfix', 'cedtag', 'nav clearfix', 'jwDisqusForm']}), dict(attrs={'alt':['Poprzednia strona', 'Następna strona']})]
remove_tags_after = dict(name='div', attrs={'class':'cedtag'}) remove_tags_after = dict(name='div', attrs={'class':'cedtag'})
feeds = [(u'Kosmonauta.net', u'http://www.kosmonauta.net/?format=feed&type=atom')] feeds = [(u'Kosmonauta.net', u'http://www.kosmonauta.net/?format=feed&type=atom')]
def print_version(self, url):
return url + '?tmpl=component&print=1&layout=default&page='
def preprocess_html(self, soup): def preprocess_html(self, soup):
for a in soup.findAll(name='a'): for a in soup.findAll(name='a'):
@ -26,5 +30,4 @@ class Kosmonauta(BasicNewsRecipe):
href = a['href'] href = a['href']
if not href.startswith('http'): if not href.startswith('http'):
a['href'] = self.INDEX + href a['href'] = self.INDEX + href
return soup return soup

46
recipes/ledevoir.recipe
View File

@ -2,7 +2,7 @@ __license__ = 'GPL v3'
__author__ = 'Lorenzo Vigentini and Olivier Daigle' __author__ = 'Lorenzo Vigentini and Olivier Daigle'
__copyright__ = '2012, Lorenzo Vigentini <l.vigentini at gmail.com>, Olivier Daigle <odaigle _at nuvucameras __dot__ com>' __copyright__ = '2012, Lorenzo Vigentini <l.vigentini at gmail.com>, Olivier Daigle <odaigle _at nuvucameras __dot__ com>'
__version__ = 'v1.01' __version__ = 'v1.01'
__date__ = '22, December 2012' __date__ = '17, March 2013'
__description__ = 'Canadian Paper ' __description__ = 'Canadian Paper '
''' '''
@ -28,10 +28,14 @@ class ledevoir(BasicNewsRecipe):
oldest_article = 1 oldest_article = 1
max_articles_per_feed = 200 max_articles_per_feed = 200
min_articles_per_feed = 0
use_embedded_content = False use_embedded_content = False
recursion = 10 recursion = 10
needs_subscription = 'optional' needs_subscription = 'optional'
compress_news_images = True
compress_news_images_auto_size = 4
filterDuplicates = False filterDuplicates = False
url_list = [] url_list = []
@ -66,16 +70,16 @@ class ledevoir(BasicNewsRecipe):
feeds = [ feeds = [
(u'A la une', 'http://www.ledevoir.com/rss/manchettes.xml'), (u'A la une', 'http://www.ledevoir.com/rss/manchettes.xml'),
# (u'Édition complete', 'http://feeds2.feedburner.com/fluxdudevoir'), (u'Édition complete', 'http://feeds2.feedburner.com/fluxdudevoir'),
# (u'Opinions', 'http://www.ledevoir.com/rss/opinions.xml'), (u'Opinions', 'http://www.ledevoir.com/rss/opinions.xml'),
# (u'Chroniques', 'http://www.ledevoir.com/rss/chroniques.xml'), (u'Chroniques', 'http://www.ledevoir.com/rss/chroniques.xml'),
# (u'Politique', 'http://www.ledevoir.com/rss/section/politique.xml?id=51'), (u'Politique', 'http://www.ledevoir.com/rss/section/politique.xml?id=51'),
# (u'International', 'http://www.ledevoir.com/rss/section/international.xml?id=76'), (u'International', 'http://www.ledevoir.com/rss/section/international.xml?id=76'),
# (u'Culture', 'http://www.ledevoir.com/rss/section/culture.xml?id=48'), (u'Culture', 'http://www.ledevoir.com/rss/section/culture.xml?id=48'),
# (u'Environnement', 'http://www.ledevoir.com/rss/section/environnement.xml?id=78'), (u'Environnement', 'http://www.ledevoir.com/rss/section/environnement.xml?id=78'),
# (u'Societe', 'http://www.ledevoir.com/rss/section/societe.xml?id=52'), (u'Societe', 'http://www.ledevoir.com/rss/section/societe.xml?id=52'),
# (u'Economie', 'http://www.ledevoir.com/rss/section/economie.xml?id=49'), (u'Economie', 'http://www.ledevoir.com/rss/section/economie.xml?id=49'),
# (u'Sports', 'http://www.ledevoir.com/rss/section/sports.xml?id=85'), (u'Sports', 'http://www.ledevoir.com/rss/section/sports.xml?id=85'),
(u'Art de vivre', 'http://www.ledevoir.com/rss/section/art-de-vivre.xml?id=50') (u'Art de vivre', 'http://www.ledevoir.com/rss/section/art-de-vivre.xml?id=50')
] ]
@ -113,3 +117,23 @@ class ledevoir(BasicNewsRecipe):
self.url_list.append(url) self.url_list.append(url)
return url return url
'''
def postprocess_html(self, soup, first):
#process all the images. assumes that the new html has the correct path
if first == 0:
return soup
for tag in soup.findAll(lambda tag: tag.name.lower()=='img' and tag.has_key('src')):
iurl = tag['src']
img = Image()
img.open(iurl)
# width, height = img.size
# print 'img is: ', iurl, 'width is: ', width, 'height is: ', height
if img < 0:
raise RuntimeError('Out of memory')
img.set_compression_quality(30)
img.save(iurl)
return soup
'''

29
recipes/mlody_technik_pl.recipe
View File

@ -2,13 +2,14 @@
import re import re
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class Mlody_technik(BasicNewsRecipe): class Mlody_technik(BasicNewsRecipe):
title = u'Młody technik' title = u'Młody technik'
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'Młody technik' description = u'Młody technik'
category = 'science' category = 'science'
language = 'pl' language = 'pl'
#cover_url = 'http://science-everywhere.pl/wp-content/uploads/2011/10/mt12.jpg' #cover_url = 'http://science-everywhere.pl/wp-content/uploads/2011/10/mt12.jpg'
no_stylesheets = True no_stylesheets = True
extra_css = 'img.alignleft {float: left; margin-right: 5px;}'
preprocess_regexps = [(re.compile(r"<h4>Podobne</h4>", re.IGNORECASE), lambda m: '')] preprocess_regexps = [(re.compile(r"<h4>Podobne</h4>", re.IGNORECASE), lambda m: '')]
oldest_article = 7 oldest_article = 7
max_articles_per_feed = 100 max_articles_per_feed = 100
@ -17,18 +18,18 @@ class Mlody_technik(BasicNewsRecipe):
keep_only_tags = [dict(id='content')] keep_only_tags = [dict(id='content')]
remove_tags = [dict(attrs={'class':'st-related-posts'})] remove_tags = [dict(attrs={'class':'st-related-posts'})]
remove_tags_after = dict(attrs={'class':'entry-content clearfix'}) remove_tags_after = dict(attrs={'class':'entry-content clearfix'})
feeds = [(u'Wszystko', u'http://www.mt.com.pl/feed'), feeds = [(u'Wszystko', u'http://www.mt.com.pl/feed'),
#(u'MT NEWS 24/7', u'http://www.mt.com.pl/kategoria/mt-newsy-24-7/feed'), #(u'MT NEWS 24/7', u'http://www.mt.com.pl/kategoria/mt-newsy-24-7/feed'),
(u'Info zoom', u'http://www.mt.com.pl/kategoria/info-zoom/feed'), (u'Info zoom', u'http://www.mt.com.pl/kategoria/info-zoom/feed'),
(u'm.technik', u'http://www.mt.com.pl/kategoria/m-technik/feed'), (u'm.technik', u'http://www.mt.com.pl/kategoria/m-technik/feed'),
(u'Szkoła', u'http://www.mt.com.pl/kategoria/szkola-2/feed'), (u'Szkoła', u'http://www.mt.com.pl/kategoria/szkola-2/feed'),
(u'Na Warsztacie', u'http://www.mt.com.pl/kategoria/na-warsztacie/feed'), (u'Na Warsztacie', u'http://www.mt.com.pl/kategoria/na-warsztacie/feed'),
(u'Z pasji do...', u'http://www.mt.com.pl/kategoria/z-pasji-do/feed'), (u'Z pasji do...', u'http://www.mt.com.pl/kategoria/z-pasji-do/feed'),
(u'MT testuje', u'http://www.mt.com.pl/kategoria/mt-testuje/feed')] (u'MT testuje', u'http://www.mt.com.pl/kategoria/mt-testuje/feed')]
def get_cover_url(self): def get_cover_url(self):
soup = self.index_to_soup('http://www.mt.com.pl/') soup = self.index_to_soup('http://www.mt.com.pl/')
tag = soup.find(attrs={'class':'xoxo'}) tag = soup.find(attrs={'class':'xoxo'})
if tag: if tag:
self.cover_url = tag.find('img')['src'] self.cover_url = tag.find('img')['src']
return getattr(self, 'cover_url', self.cover_url) return getattr(self, 'cover_url', self.cover_url)

18
recipes/nauka_w_polsce.recipe
View File

@ -1,16 +1,18 @@
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
import re import re
class NaukawPolsce(BasicNewsRecipe): class NaukawPolsce(BasicNewsRecipe):
title = u'Nauka w Polsce' title = u'Nauka w Polsce'
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'Serwis Nauka w Polsce ma za zadanie popularyzację polskiej nauki. Można na nim znaleźć wiadomości takie jak: osiągnięcia polskich naukowców, wydarzenia na polskich uczelniach, osiągnięcia studentów, konkursy dla badaczy, staże i stypendia naukowe, wydarzenia w polskiej nauce, kalendarium wydarzeń w nauce, materiały wideo o nauce.' description = u'Serwis Nauka w Polsce ma za zadanie popularyzację polskiej nauki. Można na nim znaleźć wiadomości takie jak: osiągnięcia polskich naukowców, wydarzenia na polskich uczelniach, osiągnięcia studentów, konkursy dla badaczy, staże i stypendia naukowe, wydarzenia w polskiej nauce, kalendarium wydarzeń w nauce, materiały wideo o nauce.'
category = 'science' category = 'science'
language = 'pl' language = 'pl'
cover_url = 'http://www.naukawpolsce.pap.pl/Themes/Pap/images/logo-pl.gif' cover_url = 'http://www.naukawpolsce.pap.pl/Themes/Pap/images/logo-pl.gif'
oldest_article = 7 oldest_article = 7
max_articles_per_feed = 100 max_articles_per_feed = 100
no_stylesheets = True no_stylesheets = True
remove_empty_feeds = True remove_empty_feeds = True
extra_css = '.miniaturka {float: left; margin-right: 5px; max-width: 350px;} .miniaturka-dol-strony {display: inline-block; margin: 0 15px; width: 120px;}'
ignore_duplicate_articles = {'title', 'url'}
index = 'http://www.naukawpolsce.pl' index = 'http://www.naukawpolsce.pl'
keep_only_tags = [dict(name='div', attrs={'class':'margines wiadomosc'})] keep_only_tags = [dict(name='div', attrs={'class':'margines wiadomosc'})]
remove_tags = [dict(name='div', attrs={'class':'tagi'})] remove_tags = [dict(name='div', attrs={'class':'tagi'})]
@ -23,8 +25,8 @@ class NaukawPolsce(BasicNewsRecipe):
url = self.index + i.h1.a['href'] url = self.index + i.h1.a['href']
date = '' #i.span.string date = '' #i.span.string
articles.append({'title' : title, articles.append({'title' : title,
'url' : url, 'url' : url,
'date' : date, 'date' : date,
'description' : '' 'description' : ''
}) })
return articles return articles
@ -44,4 +46,4 @@ class NaukawPolsce(BasicNewsRecipe):
def preprocess_html(self, soup): def preprocess_html(self, soup):
for p in soup.findAll(name='p', text=re.compile('&nbsp;')): for p in soup.findAll(name='p', text=re.compile('&nbsp;')):
p.extract() p.extract()
return soup return soup

16
recipes/niebezpiecznik.recipe
View File

@ -1,17 +1,19 @@
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class Niebezpiecznik_pl(BasicNewsRecipe): class Niebezpiecznik_pl(BasicNewsRecipe):
title = u'Niebezpiecznik.pl' title = u'Niebezpiecznik.pl'
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'Niebezpiecznik.pl – o bezpieczeństwie i nie...' description = u'Niebezpiecznik.pl – o bezpieczeństwie i nie...'
category = 'hacking, IT' category = 'hacking, IT'
language = 'pl' language = 'pl'
oldest_article = 8 oldest_article = 8
extra_css = '.entry {margin-top: 25px;}'
remove_attrs = ['style']
max_articles_per_feed = 100 max_articles_per_feed = 100
no_stylesheets = True no_stylesheets = True
remove_empty_feeds = True remove_empty_feeds = True
cover_url = u'http://userlogos.org/files/logos/Karmody/niebezpiecznik_01.png' cover_url = u'http://userlogos.org/files/logos/Karmody/niebezpiecznik_01.png'
remove_tags = [dict(name='div', attrs={'class':['sociable']}), dict(name='h4'), dict(attrs={'class':'similar-posts'})] remove_tags = [dict(name='div', attrs={'class':['sociable']}), dict(name='h4'), dict(attrs={'class':'similar-posts'})]
keep_only_tags = [dict(name='div', attrs={'class':['title', 'entry']})] keep_only_tags = [dict(name='div', attrs={'class':['title', 'entry']})]
feeds = [(u'Wiadomości', u'http://feeds.feedburner.com/niebezpiecznik/'), feeds = [(u'Wiadomości', u'http://feeds.feedburner.com/niebezpiecznik/'),
('Blog', 'http://feeds.feedburner.com/niebezpiecznik/linkblog/')] ('Blog', 'http://feeds.feedburner.com/niebezpiecznik/linkblog/')]

41
recipes/optyczne_pl.recipe Normal file
View File

@ -0,0 +1,41 @@
#!/usr/bin/env python
__license__ = 'GPL v3'
from calibre.web.feeds.news import BasicNewsRecipe
class OptyczneRecipe(BasicNewsRecipe):
__author__ = u'Artur Stachecki <artur.stachecki@gmail.com>'
language = 'pl'
title = u'optyczne.pl'
category = u'News'
description = u'Najlepsze testy obiektywów, testy aparatów cyfrowych i testy lornetek w sieci!'
cover_url=''
remove_empty_feeds= True
no_stylesheets=True
oldest_article = 7
max_articles_per_feed = 100000
recursions = 0
no_stylesheets = True
remove_javascript = True
keep_only_tags =[]
keep_only_tags.append(dict(name = 'div', attrs = {'class' : 'news'}))
remove_tags =[]
remove_tags.append(dict(name = 'div', attrs = {'class' : 'center'}))
remove_tags.append(dict(name = 'div', attrs = {'class' : 'news_foto'}))
remove_tags.append(dict(name = 'div', attrs = {'align' : 'right'}))
extra_css = '''
body {font-family: Arial,Helvetica,sans-serif;}
h1{text-align: left;}
h2{font-size: medium; font-weight: bold;}
p.lead {font-weight: bold; text-align: left;}
.authordate {font-size: small; color: #696969;}
.fot{font-size: x-small; color: #666666;}
'''
feeds = [
('Aktualnosci', 'http://www.optyczne.pl/rss.xml'),
]

15
recipes/osworld_pl.recipe
View File

@ -1,11 +1,12 @@
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class OSWorld(BasicNewsRecipe): class OSWorld(BasicNewsRecipe):
title = u'OSWorld.pl' title = u'OSWorld.pl'
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'OSWorld.pl to serwis internetowy, dzięki któremu poznasz czym naprawdę jest Open Source. Serwis poświęcony jest wolnemu oprogramowaniu jak linux mint, centos czy ubunty. Znajdziecie u nasz artykuły, unity oraz informacje o certyfikatach CACert. OSWorld to mały świat wielkich systemów!' description = u'OSWorld.pl to serwis internetowy, dzięki któremu poznasz czym naprawdę jest Open Source. Serwis poświęcony jest wolnemu oprogramowaniu jak linux mint, centos czy ubunty. Znajdziecie u nasz artykuły, unity oraz informacje o certyfikatach CACert. OSWorld to mały świat wielkich systemów!'
category = 'OS, IT, open source, Linux' category = 'OS, IT, open source, Linux'
language = 'pl' language = 'pl'
cover_url = 'http://osworld.pl/wp-content/uploads/osworld-kwadrat-128x111.png' cover_url = 'http://osworld.pl/wp-content/uploads/osworld-kwadrat-128x111.png'
extra_css = 'img.alignleft {float: left; margin-right: 5px;}'
oldest_article = 7 oldest_article = 7
max_articles_per_feed = 100 max_articles_per_feed = 100
no_stylesheets = True no_stylesheets = True
@ -14,7 +15,7 @@ class OSWorld(BasicNewsRecipe):
keep_only_tags = [dict(id=['dzial', 'posts'])] keep_only_tags = [dict(id=['dzial', 'posts'])]
remove_tags = [dict(attrs={'class':'post-comments'})] remove_tags = [dict(attrs={'class':'post-comments'})]
remove_tags_after = dict(attrs={'class':'entry clr'}) remove_tags_after = dict(attrs={'class':'entry clr'})
feeds = [(u'Artyku\u0142y', u'http://osworld.pl/category/artykuly/feed/'), (u'Nowe wersje', u'http://osworld.pl/category/nowe-wersje/feed/')] feeds = [(u'Artyku\u0142y', u'http://osworld.pl/category/artykuly/feed/'), (u'Nowe wersje', u'http://osworld.pl/category/nowe-wersje/feed/')]
def append_page(self, soup, appendtag): def append_page(self, soup, appendtag):
tag = appendtag.find(attrs={'id':'paginacja'}) tag = appendtag.find(attrs={'id':'paginacja'})
@ -30,4 +31,4 @@ class OSWorld(BasicNewsRecipe):
def preprocess_html(self, soup): def preprocess_html(self, soup):
self.append_page(soup, soup.body) self.append_page(soup, soup.body)
return soup return soup

13
recipes/pc_centre_pl.recipe
View File

@ -1,20 +1,21 @@
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class PC_Centre(BasicNewsRecipe): class PC_Centre(BasicNewsRecipe):
title = u'PC Centre' title = u'PC Centre'
oldest_article = 7 oldest_article = 7
max_articles_per_feed = 100 max_articles_per_feed = 100
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'Portal komputerowy, a w nim: testy sprzętu komputerowego, recenzje gier i oprogramowania. a także opisy produktów związanych z komputerami.' description = u'Portal komputerowy, a w nim: testy sprzętu komputerowego, recenzje gier i oprogramowania. a także opisy produktów związanych z komputerami.'
category = 'IT' category = 'IT'
language = 'pl' language = 'pl'
masthead_url= 'http://pccentre.pl/views/images/logo.gif' masthead_url= 'http://pccentre.pl/views/images/logo.gif'
cover_url= 'http://pccentre.pl/views/images/logo.gif' cover_url= 'http://pccentre.pl/views/images/logo.gif'
no_stylesheets = True no_stylesheets = True
remove_empty_feeds = True remove_empty_feeds = True
ignore_duplicate_articles = {'title', 'url'}
#keep_only_tags= [dict(id='content')] #keep_only_tags= [dict(id='content')]
#remove_tags=[dict(attrs={'class':['ikony r', 'list_of_content', 'dot accordion']}), dict(id='comments')] #remove_tags=[dict(attrs={'class':['ikony r', 'list_of_content', 'dot accordion']}), dict(id='comments')]
remove_tags=[dict(attrs={'class':'logo_print'})] remove_tags=[dict(attrs={'class':'logo_print'})]
feeds = [(u'Aktualno\u015bci', u'http://pccentre.pl/backend.php'), (u'Publikacje', u'http://pccentre.pl/backend.php?mode=a'), (u'Sprz\u0119t komputerowy', u'http://pccentre.pl/backend.php?mode=n&section=2'), (u'Oprogramowanie', u'http://pccentre.pl/backend.php?mode=n&section=3'), (u'Gry komputerowe i konsole', u'http://pccentre.pl/backend.php?mode=n&section=4'), (u'Internet', u'http://pccentre.pl/backend.php?mode=n&section=7'), (u'Bezpiecze\u0144stwo', u'http://pccentre.pl/backend.php?mode=n&section=5'), (u'Multimedia', u'http://pccentre.pl/backend.php?mode=n&section=6'), (u'Biznes', u'http://pccentre.pl/backend.php?mode=n&section=9')] feeds = [(u'Aktualno\u015bci', u'http://pccentre.pl/backend.php'), (u'Publikacje', u'http://pccentre.pl/backend.php?mode=a'), (u'Sprz\u0119t komputerowy', u'http://pccentre.pl/backend.php?mode=n&section=2'), (u'Oprogramowanie', u'http://pccentre.pl/backend.php?mode=n&section=3'), (u'Gry komputerowe i konsole', u'http://pccentre.pl/backend.php?mode=n&section=4'), (u'Internet', u'http://pccentre.pl/backend.php?mode=n&section=7'), (u'Bezpiecze\u0144stwo', u'http://pccentre.pl/backend.php?mode=n&section=5'), (u'Multimedia', u'http://pccentre.pl/backend.php?mode=n&section=6'), (u'Biznes', u'http://pccentre.pl/backend.php?mode=n&section=9')]
def print_version(self, url): def print_version(self, url):
return url.replace('show', 'print') return url.replace('show', 'print')

13
recipes/psych.recipe
View File

@ -67,12 +67,13 @@ class PsychologyToday(BasicNewsRecipe):
title = title + u' (%s)'%author title = title + u' (%s)'%author
article_page= self.index_to_soup('http://www.psychologytoday.com'+post.find('a', href=True)['href']) article_page= self.index_to_soup('http://www.psychologytoday.com'+post.find('a', href=True)['href'])
print_page=article_page.find('li', attrs={'class':'print_html first'}) print_page=article_page.find('li', attrs={'class':'print_html first'})
url='http://www.psychologytoday.com'+print_page.find('a',href=True)['href'] if print_page is not None:
desc = self.tag_to_string(post.find('div', attrs={'class':'collection-node-description'})).strip() url='http://www.psychologytoday.com'+print_page.find('a',href=True)['href']
self.log('Found article:', title) desc = self.tag_to_string(post.find('div', attrs={'class':'collection-node-description'})).strip()
self.log('\t', url) self.log('Found article:', title)
self.log('\t', desc) self.log('\t', url)
articles.append({'title':title, 'url':url, 'date':'','description':desc}) self.log('\t', desc)
articles.append({'title':title, 'url':url, 'date':'','description':desc})
return [('Current Issue', articles)] return [('Current Issue', articles)]

4
recipes/publico.recipe
View File

@ -23,8 +23,8 @@ class PublicoPT(BasicNewsRecipe):
remove_empty_feeds = True remove_empty_feeds = True
extra_css = ' body{font-family: Arial,Helvetica,sans-serif } img{margin-bottom: 0.4em} ' extra_css = ' body{font-family: Arial,Helvetica,sans-serif } img{margin-bottom: 0.4em} '
keep_only_tags = [dict(attrs={'class':['content-noticia-title','artigoHeader','ECOSFERA_MANCHETE','noticia','textoPrincipal','ECOSFERA_texto_01']})] keep_only_tags = [dict(attrs={'class':['hentry article single']})]
remove_tags = [dict(attrs={'class':['options','subcoluna']})] remove_tags = [dict(attrs={'class':['entry-options entry-options-above group','entry-options entry-options-below group', 'module tag-list']})]
feeds = [ feeds = [
(u'Geral', u'http://feeds.feedburner.com/publicoRSS'), (u'Geral', u'http://feeds.feedburner.com/publicoRSS'),

6
recipes/rte.recipe
View File

@ -6,10 +6,12 @@ class RTE(BasicNewsRecipe):
max_articles_per_feed = 100 max_articles_per_feed = 100
__author__ = u'Robin Phillips' __author__ = u'Robin Phillips'
language = 'en_IE' language = 'en_IE'
auto_cleanup=True
auto_cleanup_keep = '//figure[@class="photography gal642 single"]'
remove_tags = [dict(attrs={'class':['topAd','botad','previousNextItem','headline','footerLinks','footernav']})] remove_tags = [dict(attrs={'class':['topAd','botad','previousNextItem','headline','footerLinks','footernav']})]
feeds = [(u'News', u'http://www.rte.ie/rss/news.xml'), (u'Sport', u'http://www.rte.ie/rss/sport.xml'), (u'Soccer', u'http://www.rte.ie/rss/soccer.xml'), (u'GAA', u'http://www.rte.ie/rss/gaa.xml'), (u'Rugby', u'http://www.rte.ie/rss/rugby.xml'), (u'Racing', u'http://www.rte.ie/rss/racing.xml'), (u'Business', u'http://www.rte.ie/rss/business.xml'), (u'Entertainment', u'http://www.rte.ie/rss/entertainment.xml')] feeds = [(u'News', u'http://www.rte.ie/rss/news.xml'), (u'Sport', u'http://www.rte.ie/rss/sport.xml'), (u'Soccer', u'http://www.rte.ie/rss/soccer.xml'), (u'GAA', u'http://www.rte.ie/rss/gaa.xml'), (u'Rugby', u'http://www.rte.ie/rss/rugby.xml'), (u'Racing', u'http://www.rte.ie/rss/racing.xml'), (u'Business', u'http://www.rte.ie/rss/business.xml'), (u'Entertainment', u'http://www.rte.ie/rss/entertainment.xml')]
def print_version(self, url): #def print_version(self, url):
return url.replace('http://www', 'http://m') #return url.replace('http://www', 'http://m')

71
recipes/sport_pl.recipe Normal file
View File

@ -0,0 +1,71 @@
#!/usr/bin/env python
__license__ = 'GPL v3'
__copyright__ = 'teepel 2012'
'''
sport.pl
'''
from calibre.web.feeds.news import BasicNewsRecipe
class sport_pl(BasicNewsRecipe):
title = 'Sport.pl'
__author__ = 'teepel <teepel44@gmail.com>'
language = 'pl'
description =u'Największy portal sportowy w Polsce. Wiadomości sportowe z najważniejszych wydarzeń, relacje i wyniki meczów na żywo.'
masthead_url='http://press.gazeta.pl/file/mediakit/154509/c8/sportpl.jpg'
oldest_article = 1
max_articles_per_feed = 100
remove_javascript=True
no_stylesheets=True
remove_empty_feeds = True
keep_only_tags =[]
keep_only_tags.append(dict(name = 'div', attrs = {'id' : 'article'}))
remove_tags =[]
remove_tags.append(dict(name = 'a', attrs = {'href' : 'www.gazeta.pl'}))
feeds = [
(u'Wszystkie wiadomości', u'http://rss.gazeta.pl/pub/rss/sport.xml'),
(u'Piłka nożna', u'http://www.sport.pl/pub/rss/sport/pilka_nozna.htm'),
(u'F1', u'http://www.sport.pl/pub/rss/sportf1.htm'),
(u'Tenis', u'http://serwisy.gazeta.pl/pub/rss/tenis.htm'),
(u'Siatkówka', u'http://gazeta.pl.feedsportal.com/c/32739/f/611628/index.rss'),
(u'Koszykówka', u'http://gazeta.pl.feedsportal.com/c/32739/f/611647/index.rss'),
(u'Piłka ręczna', u'http://gazeta.pl.feedsportal.com/c/32739/f/611635/index.rss'),
(u'Inne sporty', u'http://gazeta.pl.feedsportal.com/c/32739/f/611649/index.rss'),
]
def parse_feeds(self):
feeds = BasicNewsRecipe.parse_feeds(self)
for feed in feeds:
for article in feed.articles[:]:
if '[ZDJĘCIA]' in article.title:
article.title = article.title.replace('[ZDJĘCIA]','')
elif '[WIDEO]' in article.title:
article.title = article.title.replace('[WIDEO]','')
return feeds
def print_version(self, url):
if 'feedsportal' in url:
segment = url.split('/')
urlPart = segment[-2]
urlPart = urlPart.replace('0L0Ssport0Bpl0C','')
urlPart = urlPart.replace('0C10H','/')
urlPart = urlPart.replace('0H',',')
urlPart = urlPart.replace('0I','_')
urlPart = urlPart.replace('A','')
segment1 = urlPart.split('/')
seg1 = segment1[0]
seg2 = segment1[1]
segment2 = seg2.split(',')
part = segment2[0] + ',' + segment2[1]
return 'http://www.sport.pl/' + seg1 + '/2029020,' + part + '.html'
else:
segment = url.split('/')
part2 = segment[-2]
part1 = segment[-1]
segment2 = part1.split(',')
part = segment2[1] + ',' + segment2[2]
return 'http://www.sport.pl/' + part2 + '/2029020,' + part + '.html'

57
recipes/stopklatka.recipe
View File

File diff suppressed because one or more lines are too long

20
recipes/tablety_pl.recipe
View File

@ -1,18 +1,20 @@
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
import re import re
class Tablety_pl(BasicNewsRecipe): class Tablety_pl(BasicNewsRecipe):
title = u'Tablety.pl' title = u'Tablety.pl'
__author__ = 'fenuks' __author__ = 'fenuks'
description = u'Tablety, gry i aplikacje na tablety.' description = u'Tablety, gry i aplikacje na tablety.'
masthead_url= 'http://www.tablety.pl/wp-content/themes/kolektyw/img/logo.png' masthead_url= 'http://www.tablety.pl/wp-content/themes/kolektyw/img/logo.png'
cover_url = 'http://www.tablety.pl/wp-content/themes/kolektyw/img/logo.png' cover_url = 'http://www.tablety.pl/wp-content/themes/kolektyw/img/logo.png'
category = 'IT' category = 'IT'
language = 'pl' language = 'pl'
use_embedded_content=True use_embedded_content = False
no_stylesheets = True
oldest_article = 8 oldest_article = 8
max_articles_per_feed = 100 max_articles_per_feed = 100
preprocess_regexps = [(re.compile(ur'<p><strong>Przeczytaj także.*?</a></strong></p>', re.DOTALL), lambda match: ''), (re.compile(ur'<p><strong>Przeczytaj koniecznie.*?</a></strong></p>', re.DOTALL), lambda match: '')] preprocess_regexps = [(re.compile(ur'<p><strong>Przeczytaj także.*?</a></strong></p>', re.DOTALL), lambda match: ''), (re.compile(ur'<p><strong>Przeczytaj koniecznie.*?</a></strong></p>', re.DOTALL), lambda match: '')]
keep_only_tags = [dict(id='news_block')]
#remove_tags_before=dict(name="h1", attrs={'class':'entry-title'}) #remove_tags_before=dict(name="h1", attrs={'class':'entry-title'})
#remove_tags_after=dict(name="footer", attrs={'class':'entry-footer clearfix'}) #remove_tags_after=dict(name="footer", attrs={'class':'entry-footer clearfix'})
#remove_tags=[dict(name='footer', attrs={'class':'entry-footer clearfix'}), dict(name='div', attrs={'class':'entry-comment-counter'})] remove_tags=[dict(attrs={'class':['comments_icon', 'wp-polls', 'entry-comments']})]
feeds = [(u'Najnowsze posty', u'http://www.tablety.pl/feed/')] feeds = [(u'Najnowsze posty', u'http://www.tablety.pl/feed/')]

26
recipes/trystero.recipe Normal file
View File

@ -0,0 +1,26 @@
#!/usr/bin/env python
__license__ = 'GPL v3'
__copyright__ = u'2013, Tomasz Dlugosz <tomek3d@gmail.com>'
'''
trystero.pl
'''
from calibre.web.feeds.news import BasicNewsRecipe
class trystero(BasicNewsRecipe):
title = 'Trystero'
__author__ = u'Tomasz D\u0142ugosz'
language = 'pl'
description =u'Trystero.pl jest niezależnym blogiem finansowym. Publikowane na nim teksty dotyczą rynku kapitałowego, ekonomii, gospodarki i życia społecznego – w takiej mniej więcej kolejności.'
oldest_article = 7
remove_javascript=True
no_stylesheets=True
feeds = [(u'Newsy', u'http://www.trystero.pl/feed')]
keep_only_tags = [
dict(name='h1'),
dict(name='div', attrs={'class': ['post-content']})]

17
recipes/wirtualnemedia_pl.recipe
View File

@ -1,21 +1,22 @@
from calibre.web.feeds.news import BasicNewsRecipe from calibre.web.feeds.news import BasicNewsRecipe
class WirtualneMedia(BasicNewsRecipe): class WirtualneMedia(BasicNewsRecipe):
title = u'wirtualnemedia.pl' title = u'wirtualnemedia.pl'
oldest_article = 7 oldest_article = 7
max_articles_per_feed = 100 max_articles_per_feed = 100
no_stylesheets = True no_stylesheets = True
use_embedded_content = False use_embedded_content = False
remove_empty_feeds = True remove_empty_feeds = True
__author__ = 'fenuks' __author__ = 'fenuks'
extra_css = '.thumbnail {float:left; max-width:150px; margin-right:5px;}' extra_css = '.thumbnail {float:left; max-width:150px; margin-right:5px;}'
description = u'Portal o mediach, reklamie, internecie, PR, telekomunikacji - nr 1 w Polsce - WirtualneMedia.pl - wiadomości z pierwszej ręki.' description = u'Portal o mediach, reklamie, internecie, PR, telekomunikacji - nr 1 w Polsce - WirtualneMedia.pl - wiadomości z pierwszej ręki.'
category = 'internet' category = 'internet'
language = 'pl' language = 'pl'
ignore_duplicate_articles = {'title', 'url'}
masthead_url= 'http://i.wp.pl/a/f/jpeg/8654/wirtualnemedia.jpeg' masthead_url= 'http://i.wp.pl/a/f/jpeg/8654/wirtualnemedia.jpeg'
cover_url= 'http://static.wirtualnemedia.pl/img/logo_wirtualnemedia_newsletter.gif' cover_url= 'http://static.wirtualnemedia.pl/img/logo_wirtualnemedia_newsletter.gif'
remove_tags=[dict(id=['header', 'footer'])] remove_tags=[dict(id=['header', 'footer'])]
feeds = [(u'Gospodarka', u'http://www.wirtualnemedia.pl/rss/wm_gospodarka.xml'), feeds = [(u'Gospodarka', u'http://www.wirtualnemedia.pl/rss/wm_gospodarka.xml'),
(u'Internet', u'http://www.wirtualnemedia.pl/rss/wm_internet.xml'), (u'Internet', u'http://www.wirtualnemedia.pl/rss/wm_internet.xml'),
(u'Kultura', u'http://www.wirtualnemedia.pl/rss/wm_kulturarozrywka.xml'), (u'Kultura', u'http://www.wirtualnemedia.pl/rss/wm_kulturarozrywka.xml'),
(u'Badania', u'http://www.wirtualnemedia.pl/rss/wm_marketing.xml'), (u'Badania', u'http://www.wirtualnemedia.pl/rss/wm_marketing.xml'),
@ -24,8 +25,6 @@ class WirtualneMedia(BasicNewsRecipe):
(u'Reklama', u'http://www.wirtualnemedia.pl/rss/wm_reklama.xml'), (u'Reklama', u'http://www.wirtualnemedia.pl/rss/wm_reklama.xml'),
(u'PR', u'http://www.wirtualnemedia.pl/rss/wm_relations.xml'), (u'PR', u'http://www.wirtualnemedia.pl/rss/wm_relations.xml'),
(u'Technologie', u'http://www.wirtualnemedia.pl/rss/wm_telekomunikacja.xml'), (u'Technologie', u'http://www.wirtualnemedia.pl/rss/wm_telekomunikacja.xml'),
(u'Telewizja', u'http://www.wirtualnemedia.pl/rss/wm_telewizja_rss.xml') (u'Telewizja', u'http://www.wirtualnemedia.pl/rss/wm_telewizja_rss.xml')]
]
def print_version(self, url): def print_version(self, url):
return url.replace('artykul', 'print') return url.replace('artykul', 'print')

1552
setup/iso_639/ca.po
View File

File diff suppressed because it is too large Load Diff

10
setup/iso_639/ru.po
View File

@ -13,14 +13,14 @@ msgstr ""
"Report-Msgid-Bugs-To: Debian iso-codes team <pkg-isocodes-" "Report-Msgid-Bugs-To: Debian iso-codes team <pkg-isocodes-"
"devel@lists.alioth.debian.org>\n" "devel@lists.alioth.debian.org>\n"
"POT-Creation-Date: 2011-11-25 14:01+0000\n" "POT-Creation-Date: 2011-11-25 14:01+0000\n"
"PO-Revision-Date: 2013-02-21 23:51+0000\n" "PO-Revision-Date: 2013-03-23 10:17+0000\n"
"Last-Translator: Глория Хрусталёва <gloriya@hushmail.com>\n" "Last-Translator: Глория Хрусталёва <gloriya@hushmail.com>\n"
"Language-Team: Russian <debian-l10n-russian@lists.debian.org>\n" "Language-Team: Russian <debian-l10n-russian@lists.debian.org>\n"
"MIME-Version: 1.0\n" "MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n" "Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n" "Content-Transfer-Encoding: 8bit\n"
"X-Launchpad-Export-Date: 2013-02-23 05:19+0000\n" "X-Launchpad-Export-Date: 2013-03-24 04:45+0000\n"
"X-Generator: Launchpad (build 16506)\n" "X-Generator: Launchpad (build 16540)\n"
"Language: ru\n" "Language: ru\n"
#. name for aaa #. name for aaa
@ -5381,7 +5381,7 @@ msgstr ""
#. name for cof #. name for cof
msgid "Colorado" msgid "Colorado"
msgstr "" msgstr "Колорадо"
#. name for cog #. name for cog
msgid "Chong" msgid "Chong"
@ -5505,7 +5505,7 @@ msgstr ""
#. name for cqu #. name for cqu
msgid "Quechua; Chilean" msgid "Quechua; Chilean"
msgstr "" msgstr "Кечуа; Чилийский"
#. name for cra #. name for cra
msgid "Chara" msgid "Chara"

2
src/calibre/constants.py
View File

@ -4,7 +4,7 @@ __license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net' __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
__docformat__ = 'restructuredtext en' __docformat__ = 'restructuredtext en'
__appname__ = u'calibre' __appname__ = u'calibre'
numeric_version = (0, 9, 24) numeric_version = (0, 9, 25)
__version__ = u'.'.join(map(unicode, numeric_version)) __version__ = u'.'.join(map(unicode, numeric_version))
__author__ = u"Kovid Goyal <kovid@kovidgoyal.net>" __author__ = u"Kovid Goyal <kovid@kovidgoyal.net>"

171
src/calibre/db/backend.py
View File

@ -23,9 +23,11 @@ from calibre.ebooks.metadata import title_sort, author_to_author_sort
from calibre.utils.icu import sort_key from calibre.utils.icu import sort_key
from calibre.utils.config import to_json, from_json, prefs, tweaks from calibre.utils.config import to_json, from_json, prefs, tweaks
from calibre.utils.date import utcfromtimestamp, parse_date from calibre.utils.date import utcfromtimestamp, parse_date
from calibre.utils.filenames import (is_case_sensitive, samefile, hardlink_file) from calibre.utils.filenames import (is_case_sensitive, samefile, hardlink_file, ascii_filename,
WindowsAtomicFolderMove)
from calibre.utils.recycle_bin import delete_tree
from calibre.db.tables import (OneToOneTable, ManyToOneTable, ManyToManyTable, from calibre.db.tables import (OneToOneTable, ManyToOneTable, ManyToManyTable,
SizeTable, FormatsTable, AuthorsTable, IdentifiersTable, SizeTable, FormatsTable, AuthorsTable, IdentifiersTable, PathTable,
CompositeTable, LanguagesTable) CompositeTable, LanguagesTable)
# }}} # }}}
@ -672,7 +674,7 @@ class DB(object):
if col == 'cover' else col) if col == 'cover' else col)
if not metadata['column']: if not metadata['column']:
metadata['column'] = col metadata['column'] = col
tables[col] = OneToOneTable(col, metadata) tables[col] = (PathTable if col == 'path' else OneToOneTable)(col, metadata)
for col in ('series', 'publisher', 'rating'): for col in ('series', 'publisher', 'rating'):
tables[col] = ManyToOneTable(col, self.field_metadata[col].copy()) tables[col] = ManyToOneTable(col, self.field_metadata[col].copy())
@ -778,6 +780,44 @@ class DB(object):
self.user_version = 1 self.user_version = 1
# }}} # }}}
def normpath(self, path):
path = os.path.abspath(os.path.realpath(path))
if not self.is_case_sensitive:
path = os.path.normcase(path).lower()
return path
def rmtree(self, path, permanent=False):
if not self.normpath(self.library_path).startswith(self.normpath(path)):
delete_tree(path, permanent=permanent)
def construct_path_name(self, book_id, title, author):
'''
Construct the directory name for this book based on its metadata.
'''
author = ascii_filename(author
)[:self.PATH_LIMIT].decode('ascii', 'replace')
title = ascii_filename(title
)[:self.PATH_LIMIT].decode('ascii', 'replace')
while author[-1] in (' ', '.'):
author = author[:-1]
if not author:
author = ascii_filename(_('Unknown')).decode(
'ascii', 'replace')
return '%s/%s (%d)'%(author, title, book_id)
def construct_file_name(self, book_id, title, author):
'''
Construct the file name for this book based on its metadata.
'''
author = ascii_filename(author
)[:self.PATH_LIMIT].decode('ascii', 'replace')
title = ascii_filename(title
)[:self.PATH_LIMIT].decode('ascii', 'replace')
name = title + ' - ' + author
while name.endswith('.'):
name = name[:-1]
return name
# Database layer API {{{ # Database layer API {{{
def custom_table_names(self, num): def custom_table_names(self, num):
@ -865,7 +905,7 @@ class DB(object):
return self.format_abspath(book_id, fmt, fname, path) is not None return self.format_abspath(book_id, fmt, fname, path) is not None
def copy_cover_to(self, path, dest, windows_atomic_move=None, use_hardlink=False): def copy_cover_to(self, path, dest, windows_atomic_move=None, use_hardlink=False):
path = os.path.join(self.library_path, path, 'cover.jpg') path = os.path.abspath(os.path.join(self.library_path, path, 'cover.jpg'))
if windows_atomic_move is not None: if windows_atomic_move is not None:
if not isinstance(dest, basestring): if not isinstance(dest, basestring):
raise Exception("Error, you must pass the dest as a path when" raise Exception("Error, you must pass the dest as a path when"
@ -907,24 +947,125 @@ class DB(object):
if not isinstance(dest, basestring): if not isinstance(dest, basestring):
raise Exception("Error, you must pass the dest as a path when" raise Exception("Error, you must pass the dest as a path when"
" using windows_atomic_move") " using windows_atomic_move")
if dest and not samefile(dest, path): if dest:
windows_atomic_move.copy_path_to(path, dest) if samefile(dest, path):
# Ensure that the file has the same case as dest
try:
if path != dest:
os.rename(path, dest)
except:
pass # Nothing too catastrophic happened, the cases mismatch, that's all
else:
windows_atomic_move.copy_path_to(path, dest)
else: else:
if hasattr(dest, 'write'): if hasattr(dest, 'write'):
with lopen(path, 'rb') as f: with lopen(path, 'rb') as f:
shutil.copyfileobj(f, dest) shutil.copyfileobj(f, dest)
if hasattr(dest, 'flush'): if hasattr(dest, 'flush'):
dest.flush() dest.flush()
elif dest and not samefile(dest, path): elif dest:
if use_hardlink: if samefile(dest, path):
try: if not self.is_case_sensitive and path != dest:
hardlink_file(path, dest) # Ensure that the file has the same case as dest
return True try:
except: os.rename(path, dest)
pass except:
with lopen(path, 'rb') as f, lopen(dest, 'wb') as d: pass # Nothing too catastrophic happened, the cases mismatch, that's all
shutil.copyfileobj(f, d) else:
if use_hardlink:
try:
hardlink_file(path, dest)
return True
except:
pass
with lopen(path, 'rb') as f, lopen(dest, 'wb') as d:
shutil.copyfileobj(f, d)
return True return True
def windows_check_if_files_in_use(self, paths):
'''
Raises an EACCES IOError if any of the files in the folder of book_id
are opened in another program on windows.
'''
if iswindows:
for path in paths:
spath = os.path.join(self.library_path, *path.split('/'))
wam = None
if os.path.exists(spath):
try:
wam = WindowsAtomicFolderMove(spath)
finally:
if wam is not None:
wam.close_handles()
def update_path(self, book_id, title, author, path_field, formats_field):
path = self.construct_path_name(book_id, title, author)
current_path = path_field.for_book(book_id)
formats = formats_field.for_book(book_id, default_value=())
fname = self.construct_file_name(book_id, title, author)
# Check if the metadata used to construct paths has changed
changed = False
for fmt in formats:
name = formats_field.format_fname(book_id, fmt)
if name and name != fname:
changed = True
break
if path == current_path and not changed:
return
spath = os.path.join(self.library_path, *current_path.split('/'))
tpath = os.path.join(self.library_path, *path.split('/'))
source_ok = current_path and os.path.exists(spath)
wam = WindowsAtomicFolderMove(spath) if iswindows and source_ok else None
try:
if not os.path.exists(tpath):
os.makedirs(tpath)
if source_ok: # Migrate existing files
dest = os.path.join(tpath, 'cover.jpg')
self.copy_cover_to(current_path, dest,
windows_atomic_move=wam, use_hardlink=True)
for fmt in formats:
dest = os.path.join(tpath, fname+'.'+fmt.lower())
self.copy_format_to(book_id, fmt, formats_field.format_fname(book_id, fmt), current_path,
dest, windows_atomic_move=wam, use_hardlink=True)
# Update db to reflect new file locations
for fmt in formats:
formats_field.table.set_fname(book_id, fmt, fname, self)
path_field.table.set_path(book_id, path, self)
# Delete not needed directories
if source_ok:
if os.path.exists(spath) and not samefile(spath, tpath):
if wam is not None:
wam.delete_originals()
self.rmtree(spath, permanent=True)
parent = os.path.dirname(spath)
if len(os.listdir(parent)) == 0:
self.rmtree(parent, permanent=True)
finally:
if wam is not None:
wam.close_handles()
curpath = self.library_path
c1, c2 = current_path.split('/'), path.split('/')
if not self.is_case_sensitive and len(c1) == len(c2):
# On case-insensitive systems, title and author renames that only
# change case don't cause any changes to the directories in the file
# system. This can lead to having the directory names not match the
# title/author, which leads to trouble when libraries are copied to
# a case-sensitive system. The following code attempts to fix this
# by checking each segment. If they are different because of case,
# then rename the segment. Note that the code above correctly
# handles files in the directories, so no need to do them here.
for oldseg, newseg in zip(c1, c2):
if oldseg.lower() == newseg.lower() and oldseg != newseg:
try:
os.rename(os.path.join(curpath, oldseg),
os.path.join(curpath, newseg))
except:
break # Fail silently since nothing catastrophic has happened
curpath = os.path.join(curpath, newseg)
# }}} # }}}

48
src/calibre/db/cache.py
View File

@ -12,6 +12,7 @@ from io import BytesIO
from collections import defaultdict from collections import defaultdict
from functools import wraps, partial from functools import wraps, partial
from calibre.constants import iswindows
from calibre.db import SPOOL_SIZE from calibre.db import SPOOL_SIZE
from calibre.db.categories import get_categories from calibre.db.categories import get_categories
from calibre.db.locking import create_locks, RecordLock from calibre.db.locking import create_locks, RecordLock
@ -219,6 +220,8 @@ class Cache(object):
field.series_field = self.fields['series'] field.series_field = self.fields['series']
elif name == 'authors': elif name == 'authors':
field.author_sort_field = self.fields['author_sort'] field.author_sort_field = self.fields['author_sort']
elif name == 'title':
field.title_sort_field = self.fields['sort']
@read_api @read_api
def field_for(self, name, book_id, default_value=None): def field_for(self, name, book_id, default_value=None):
@ -619,11 +622,12 @@ class Cache(object):
@write_api @write_api
def set_field(self, name, book_id_to_val_map, allow_case_change=True): def set_field(self, name, book_id_to_val_map, allow_case_change=True):
# TODO: Specialize title/authors to also update path
# TODO: Handle updating caches used by composite fields
# TODO: Ensure the sort fields are updated for title/author/series?
f = self.fields[name] f = self.fields[name]
is_series = f.metadata['datatype'] == 'series' is_series = f.metadata['datatype'] == 'series'
update_path = name in {'title', 'authors'}
if update_path and iswindows:
paths = (x for x in (self._field_for('path', book_id) for book_id in book_id_to_val_map) if x)
self.backend.windows_check_if_files_in_use(paths)
if is_series: if is_series:
bimap, simap = {}, {} bimap, simap = {}, {}
@ -646,11 +650,31 @@ class Cache(object):
sf = self.fields[f.name+'_index'] sf = self.fields[f.name+'_index']
dirtied |= sf.writer.set_books(simap, self.backend, allow_case_change=False) dirtied |= sf.writer.set_books(simap, self.backend, allow_case_change=False)
if dirtied and self.composites:
for name in self.composites:
self.fields[name].pop_cache(dirtied)
if dirtied and update_path:
self._update_path(dirtied, mark_as_dirtied=False)
# TODO: Mark these as dirtied so that the opf is regenerated
return dirtied return dirtied
@write_api
def update_path(self, book_ids, mark_as_dirtied=True):
for book_id in book_ids:
title = self._field_for('title', book_id, default_value=_('Unknown'))
author = self._field_for('authors', book_id, default_value=(_('Unknown'),))[0]
self.backend.update_path(book_id, title, author, self.fields['path'], self.fields['formats'])
if mark_as_dirtied:
pass
# TODO: Mark these books as dirtied so that metadata.opf is
# re-created
# }}} # }}}
class SortKey(object): class SortKey(object): # {{{
def __init__(self, fields, sort_keys, book_id): def __init__(self, fields, sort_keys, book_id):
self.orders = tuple(1 if f[1] else -1 for f in fields) self.orders = tuple(1 if f[1] else -1 for f in fields)
@ -662,19 +686,5 @@ class SortKey(object):
if ans != 0: if ans != 0:
return ans * order return ans * order
return 0 return 0
# Testing {{{
def test(library_path):
from calibre.db.backend import DB
backend = DB(library_path)
cache = Cache(backend)
cache.init()
print ('All book ids:', cache.all_book_ids())
if __name__ == '__main__':
from calibre.utils.config import prefs
test(prefs['library_path'])
# }}} # }}}

7
src/calibre/db/fields.py
View File

@ -167,9 +167,10 @@ class CompositeField(OneToOneField):
with self._lock: with self._lock:
self._render_cache = {} self._render_cache = {}
def pop_cache(self, book_id): def pop_cache(self, book_ids):
with self._lock: with self._lock:
self._render_cache.pop(book_id, None) for book_id in book_ids:
self._render_cache.pop(book_id, None)
def get_value_with_cache(self, book_id, get_metadata): def get_value_with_cache(self, book_id, get_metadata):
with self._lock: with self._lock:
@ -177,6 +178,8 @@ class CompositeField(OneToOneField):
if ans is None: if ans is None:
mi = get_metadata(book_id) mi = get_metadata(book_id)
ans = mi.get('#'+self.metadata['label']) ans = mi.get('#'+self.metadata['label'])
with self._lock:
self._render_cache[book_id] = ans
return ans return ans
def sort_keys_for_books(self, get_metadata, lang_map, all_book_ids): def sort_keys_for_books(self, get_metadata, lang_map, all_book_ids):

20
src/calibre/db/tables.py
View File

@ -13,7 +13,6 @@ from dateutil.tz import tzoffset
from calibre.constants import plugins from calibre.constants import plugins
from calibre.utils.date import parse_date, local_tz, UNDEFINED_DATE from calibre.utils.date import parse_date, local_tz, UNDEFINED_DATE
from calibre.utils.localization import lang_map
from calibre.ebooks.metadata import author_to_author_sort from calibre.ebooks.metadata import author_to_author_sort
_c_speedup = plugins['speedup'][0] _c_speedup = plugins['speedup'][0]
@ -83,6 +82,13 @@ class OneToOneTable(Table):
self.metadata['column'], self.metadata['table'])): self.metadata['column'], self.metadata['table'])):
self.book_col_map[row[0]] = self.unserialize(row[1]) self.book_col_map[row[0]] = self.unserialize(row[1])
class PathTable(OneToOneTable):
def set_path(self, book_id, path, db):
self.book_col_map[book_id] = path
db.conn.execute('UPDATE books SET path=? WHERE id=?',
(path, book_id))
class SizeTable(OneToOneTable): class SizeTable(OneToOneTable):
def read(self, db): def read(self, db):
@ -144,7 +150,7 @@ class ManyToManyTable(ManyToOneTable):
''' '''
table_type = MANY_MANY table_type = MANY_MANY
selectq = 'SELECT book, {0} FROM {1}' selectq = 'SELECT book, {0} FROM {1} ORDER BY id'
def read_maps(self, db): def read_maps(self, db):
for row in db.conn.execute( for row in db.conn.execute(
@ -161,8 +167,6 @@ class ManyToManyTable(ManyToOneTable):
class AuthorsTable(ManyToManyTable): class AuthorsTable(ManyToManyTable):
selectq = 'SELECT book, {0} FROM {1} ORDER BY id'
def read_id_maps(self, db): def read_id_maps(self, db):
self.alink_map = {} self.alink_map = {}
self.asort_map = {} self.asort_map = {}
@ -196,6 +200,11 @@ class FormatsTable(ManyToManyTable):
for key in tuple(self.book_col_map.iterkeys()): for key in tuple(self.book_col_map.iterkeys()):
self.book_col_map[key] = tuple(sorted(self.book_col_map[key])) self.book_col_map[key] = tuple(sorted(self.book_col_map[key]))
def set_fname(self, book_id, fmt, fname, db):
self.fname_map[book_id][fmt] = fname
db.conn.execute('UPDATE data SET name=? WHERE book=? AND format=?',
(fname, book_id, fmt))
class IdentifiersTable(ManyToManyTable): class IdentifiersTable(ManyToManyTable):
def read_id_maps(self, db): def read_id_maps(self, db):
@ -215,6 +224,3 @@ class LanguagesTable(ManyToManyTable):
def read_id_maps(self, db): def read_id_maps(self, db):
ManyToManyTable.read_id_maps(self, db) ManyToManyTable.read_id_maps(self, db)
lm = lang_map()
self.lang_name_map = {x:lm.get(x, x) for x in self.id_map.itervalues()}

10
src/calibre/db/tests/base.py
View File

@ -7,7 +7,7 @@ __license__ = 'GPL v3'
__copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>' __copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en' __docformat__ = 'restructuredtext en'
import unittest, os, shutil, tempfile, atexit import unittest, os, shutil, tempfile, atexit, gc
from functools import partial from functools import partial
from io import BytesIO from io import BytesIO
from future_builtins import map from future_builtins import map
@ -21,6 +21,7 @@ class BaseTest(unittest.TestCase):
self.create_db(self.library_path) self.create_db(self.library_path)
def tearDown(self): def tearDown(self):
gc.collect(), gc.collect()
shutil.rmtree(self.library_path) shutil.rmtree(self.library_path)
def create_db(self, library_path): def create_db(self, library_path):
@ -36,6 +37,7 @@ class BaseTest(unittest.TestCase):
db.add_format(1, 'FMT1', BytesIO(b'book1fmt1'), index_is_id=True) db.add_format(1, 'FMT1', BytesIO(b'book1fmt1'), index_is_id=True)
db.add_format(1, 'FMT2', BytesIO(b'book1fmt2'), index_is_id=True) db.add_format(1, 'FMT2', BytesIO(b'book1fmt2'), index_is_id=True)
db.add_format(2, 'FMT1', BytesIO(b'book2fmt1'), index_is_id=True) db.add_format(2, 'FMT1', BytesIO(b'book2fmt1'), index_is_id=True)
db.conn.close()
return dest return dest
def init_cache(self, library_path): def init_cache(self, library_path):
@ -65,6 +67,10 @@ class BaseTest(unittest.TestCase):
shutil.copytree(library_path, dest) shutil.copytree(library_path, dest)
return dest return dest
@property
def cloned_library(self):
return self.clone_library(self.library_path)
def compare_metadata(self, mi1, mi2): def compare_metadata(self, mi1, mi2):
allfk1 = mi1.all_field_keys() allfk1 = mi1.all_field_keys()
allfk2 = mi2.all_field_keys() allfk2 = mi2.all_field_keys()
@ -79,6 +85,8 @@ class BaseTest(unittest.TestCase):
attr1, attr2 = getattr(mi1, attr), getattr(mi2, attr) attr1, attr2 = getattr(mi1, attr), getattr(mi2, attr)
if attr == 'formats': if attr == 'formats':
attr1, attr2 = map(lambda x:tuple(x) if x else (), (attr1, attr2)) attr1, attr2 = map(lambda x:tuple(x) if x else (), (attr1, attr2))
if isinstance(attr1, (tuple, list)) and 'authors' not in attr and 'languages' not in attr:
attr1, attr2 = set(attr1), set(attr2)
self.assertEqual(attr1, attr2, self.assertEqual(attr1, attr2,
'%s not the same: %r != %r'%(attr, attr1, attr2)) '%s not the same: %r != %r'%(attr, attr1, attr2))
if attr.startswith('#'): if attr.startswith('#'):

82
src/calibre/db/tests/filesystem.py Normal file
View File

@ -0,0 +1,82 @@
#!/usr/bin/env python
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai
from __future__ import (unicode_literals, division, absolute_import,
print_function)
__license__ = 'GPL v3'
__copyright__ = '2013, Kovid Goyal <kovid at kovidgoyal.net>'
__docformat__ = 'restructuredtext en'
import unittest, os
from io import BytesIO
from calibre.constants import iswindows
from calibre.db.tests.base import BaseTest
class FilesystemTest(BaseTest):
def get_filesystem_data(self, cache, book_id):
fmts = cache.field_for('formats', book_id)
ans = {}
for fmt in fmts:
buf = BytesIO()
if cache.copy_format_to(book_id, fmt, buf):
ans[fmt] = buf.getvalue()
buf = BytesIO()
if cache.copy_cover_to(book_id, buf):
ans['cover'] = buf.getvalue()
return ans
def test_metadata_move(self):
'Test the moving of files when title/author change'
cl = self.cloned_library
cache = self.init_cache(cl)
ae, af, sf = self.assertEqual, self.assertFalse, cache.set_field
# Test that changing metadata on a book with no formats/cover works
ae(sf('title', {3:'moved1'}), set([3]))
ae(sf('authors', {3:'moved1'}), set([3]))
ae(sf('title', {3:'Moved1'}), set([3]))
ae(sf('authors', {3:'Moved1'}), set([3]))
ae(cache.field_for('title', 3), 'Moved1')
ae(cache.field_for('authors', 3), ('Moved1',))
# Now try with a book that has covers and formats
orig_data = self.get_filesystem_data(cache, 1)
orig_fpath = cache.format_abspath(1, 'FMT1')
ae(sf('title', {1:'moved'}), set([1]))
ae(sf('authors', {1:'moved'}), set([1]))
ae(sf('title', {1:'Moved'}), set([1]))
ae(sf('authors', {1:'Moved'}), set([1]))
ae(cache.field_for('title', 1), 'Moved')
ae(cache.field_for('authors', 1), ('Moved',))
cache2 = self.init_cache(cl)
for c in (cache, cache2):
data = self.get_filesystem_data(c, 1)
ae(set(orig_data.iterkeys()), set(data.iterkeys()))
ae(orig_data, data, 'Filesystem data does not match')
ae(c.field_for('path', 1), 'Moved/Moved (1)')
ae(c.field_for('path', 3), 'Moved1/Moved1 (3)')
fpath = c.format_abspath(1, 'FMT1').replace(os.sep, '/').split('/')
ae(fpath[-3:], ['Moved', 'Moved (1)', 'Moved - Moved.fmt1'])
af(os.path.exists(os.path.dirname(orig_fpath)), 'Original book folder still exists')
# Check that the filesystem reflects fpath (especially on
# case-insensitive systems).
for x in range(1, 4):
base = os.sep.join(fpath[:-x])
part = fpath[-x:][0]
self.assertIn(part, os.listdir(base))
@unittest.skipUnless(iswindows, 'Windows only')
def test_windows_atomic_move(self):
'Test book file open in another process when changing metadata'
cl = self.cloned_library
cache = self.init_cache(cl)
fpath = cache.format_abspath(1, 'FMT1')
f = open(fpath, 'rb')
with self.assertRaises(IOError):
cache.set_field('title', {1:'Moved'})
f.close()
self.assertNotEqual(cache.field_for('title', 1), 'Moved', 'Title was changed despite file lock')

23
src/calibre/db/tests/main.py Normal file
View File

@ -0,0 +1,23 @@
#!/usr/bin/env python
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai
from __future__ import (unicode_literals, division, absolute_import,
print_function)
__license__ = 'GPL v3'
__copyright__ = '2013, Kovid Goyal <kovid at kovidgoyal.net>'
__docformat__ = 'restructuredtext en'
import unittest, os, argparse
def find_tests():
return unittest.defaultTestLoader.discover(os.path.dirname(os.path.abspath(__file__)), pattern='*.py')
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('name', nargs='?', default=None, help='The name of the test to run, for e.g. writing.WritingTest.many_many_basic')
args = parser.parse_args()
if args.name:
unittest.TextTestRunner(verbosity=4).run(unittest.defaultTestLoader.loadTestsFromName(args.name))
else:
unittest.TextTestRunner(verbosity=4).run(find_tests())

18
src/calibre/db/tests/reading.py
View File

@ -7,7 +7,7 @@ __license__ = 'GPL v3'
__copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>' __copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en' __docformat__ = 'restructuredtext en'
import unittest, datetime import datetime
from calibre.utils.date import utc_tz from calibre.utils.date import utc_tz
from calibre.db.tests.base import BaseTest from calibre.db.tests.base import BaseTest
@ -115,6 +115,8 @@ class ReadingTest(BaseTest):
for book_id, test in tests.iteritems(): for book_id, test in tests.iteritems():
for field, expected_val in test.iteritems(): for field, expected_val in test.iteritems():
val = cache.field_for(field, book_id) val = cache.field_for(field, book_id)
if isinstance(val, tuple) and 'authors' not in field and 'languages' not in field:
val, expected_val = set(val), set(expected_val)
self.assertEqual(expected_val, val, self.assertEqual(expected_val, val,
'Book id: %d Field: %s failed: %r != %r'%( 'Book id: %d Field: %s failed: %r != %r'%(
book_id, field, expected_val, val)) book_id, field, expected_val, val))
@ -173,6 +175,7 @@ class ReadingTest(BaseTest):
mi.format_metadata = dict(mi.format_metadata) mi.format_metadata = dict(mi.format_metadata)
if mi.formats: if mi.formats:
mi.formats = tuple(mi.formats) mi.formats = tuple(mi.formats)
old.conn.close()
old = None old = None
cache = self.init_cache(self.library_path) cache = self.init_cache(self.library_path)
@ -189,6 +192,7 @@ class ReadingTest(BaseTest):
from calibre.library.database2 import LibraryDatabase2 from calibre.library.database2 import LibraryDatabase2
old = LibraryDatabase2(self.library_path) old = LibraryDatabase2(self.library_path)
covers = {i: old.cover(i, index_is_id=True) for i in old.all_ids()} covers = {i: old.cover(i, index_is_id=True) for i in old.all_ids()}
old.conn.close()
old = None old = None
cache = self.init_cache(self.library_path) cache = self.init_cache(self.library_path)
for book_id, cdata in covers.iteritems(): for book_id, cdata in covers.iteritems():
@ -247,6 +251,7 @@ class ReadingTest(BaseTest):
'#formats:fmt1', '#formats:fmt2', '#formats:fmt1 and #formats:fmt2', '#formats:fmt1', '#formats:fmt2', '#formats:fmt1 and #formats:fmt2',
)} )}
old.conn.close()
old = None old = None
cache = self.init_cache(self.library_path) cache = self.init_cache(self.library_path)
@ -263,6 +268,7 @@ class ReadingTest(BaseTest):
from calibre.library.database2 import LibraryDatabase2 from calibre.library.database2 import LibraryDatabase2
old = LibraryDatabase2(self.library_path) old = LibraryDatabase2(self.library_path)
old_categories = old.get_categories() old_categories = old.get_categories()
old.conn.close()
cache = self.init_cache(self.library_path) cache = self.init_cache(self.library_path)
new_categories = cache.get_categories() new_categories = cache.get_categories()
self.assertEqual(set(old_categories), set(new_categories), self.assertEqual(set(old_categories), set(new_categories),
@ -305,6 +311,7 @@ class ReadingTest(BaseTest):
i, index_is_id=True) else set() for i in ids} i, index_is_id=True) else set() for i in ids}
formats = {i:{f:old.format(i, f, index_is_id=True) for f in fmts} for formats = {i:{f:old.format(i, f, index_is_id=True) for f in fmts} for
i, fmts in lf.iteritems()} i, fmts in lf.iteritems()}
old.conn.close()
old = None old = None
cache = self.init_cache(self.library_path) cache = self.init_cache(self.library_path)
for book_id, fmts in lf.iteritems(): for book_id, fmts in lf.iteritems():
@ -328,12 +335,3 @@ class ReadingTest(BaseTest):
# }}} # }}}
def tests():
return unittest.TestLoader().loadTestsFromTestCase(ReadingTest)
def run():
unittest.TextTestRunner(verbosity=2).run(tests())
if __name__ == '__main__':
run()

70
src/calibre/db/tests/writing.py
View File

@ -7,19 +7,15 @@ __license__ = 'GPL v3'
__copyright__ = '2013, Kovid Goyal <kovid at kovidgoyal.net>' __copyright__ = '2013, Kovid Goyal <kovid at kovidgoyal.net>'
__docformat__ = 'restructuredtext en' __docformat__ = 'restructuredtext en'
import unittest
from collections import namedtuple from collections import namedtuple
from functools import partial from functools import partial
from calibre.ebooks.metadata import author_to_author_sort
from calibre.utils.date import UNDEFINED_DATE from calibre.utils.date import UNDEFINED_DATE
from calibre.db.tests.base import BaseTest from calibre.db.tests.base import BaseTest
class WritingTest(BaseTest): class WritingTest(BaseTest):
@property
def cloned_library(self):
return self.clone_library(self.library_path)
def create_getter(self, name, getter=None): def create_getter(self, name, getter=None):
if getter is None: if getter is None:
if name.endswith('_index'): if name.endswith('_index'):
@ -214,7 +210,7 @@ class WritingTest(BaseTest):
{1, 2}) {1, 2})
for name in ('tags', '#tags'): for name in ('tags', '#tags'):
f = cache.fields[name] f = cache.fields[name]
af(sf(name, {1:('tag one', 'News')}, allow_case_change=False)) af(sf(name, {1:('News', 'tag one')}, allow_case_change=False))
ae(sf(name, {1:'tag one, News'}), {1, 2}) ae(sf(name, {1:'tag one, News'}), {1, 2})
ae(sf(name, {3:('tag two', 'sep,sep2')}), {2, 3}) ae(sf(name, {3:('tag two', 'sep,sep2')}), {2, 3})
ae(len(f.table.id_map), 4) ae(len(f.table.id_map), 4)
@ -225,7 +221,7 @@ class WritingTest(BaseTest):
ae(len(c.fields[name].table.id_map), 3) ae(len(c.fields[name].table.id_map), 3)
ae(len(c.fields[name].table.id_map), 3) ae(len(c.fields[name].table.id_map), 3)
ae(c.field_for(name, 1), ()) ae(c.field_for(name, 1), ())
ae(c.field_for(name, 2), ('tag one', 'tag two')) ae(c.field_for(name, 2), ('tag two', 'tag one'))
del cache2 del cache2
# Authors # Authors
@ -244,27 +240,55 @@ class WritingTest(BaseTest):
ae(c.field_for(name, 3), ('Kovid Goyal', 'Divok Layog')) ae(c.field_for(name, 3), ('Kovid Goyal', 'Divok Layog'))
ae(c.field_for(name, 2), ('An, Author',)) ae(c.field_for(name, 2), ('An, Author',))
ae(c.field_for(name, 1), ('Unknown',) if name=='authors' else ()) ae(c.field_for(name, 1), ('Unknown',) if name=='authors' else ())
ae(c.field_for('author_sort', 1), 'Unknown') if name == 'authors':
ae(c.field_for('author_sort', 2), 'An, Author') ae(c.field_for('author_sort', 1), author_to_author_sort('Unknown'))
ae(c.field_for('author_sort', 3), 'Goyal, Kovid & Layog, Divok') ae(c.field_for('author_sort', 2), author_to_author_sort('An, Author'))
ae(c.field_for('author_sort', 3), author_to_author_sort('Kovid Goyal') + ' & ' + author_to_author_sort('Divok Layog'))
del cache2 del cache2
ae(cache.set_field('authors', {1:'KoviD GoyaL'}), {1, 3}) ae(cache.set_field('authors', {1:'KoviD GoyaL'}), {1, 3})
ae(cache.field_for('author_sort', 1), 'GoyaL, KoviD') ae(cache.field_for('author_sort', 1), 'GoyaL, KoviD')
ae(cache.field_for('author_sort', 3), 'GoyaL, KoviD & Layog, Divok') ae(cache.field_for('author_sort', 3), 'GoyaL, KoviD & Layog, Divok')
# TODO: identifiers, languages # Languages
f = cache.fields['languages']
ae(f.table.id_map, {1: 'eng', 2: 'deu'})
ae(sf('languages', {1:''}), set([1]))
ae(cache.field_for('languages', 1), ())
ae(sf('languages', {2:('und',)}), set([2]))
af(f.table.id_map)
ae(sf('languages', {1:'eng,fra,deu', 2:'es,Dutch', 3:'English'}), {1, 2, 3})
ae(cache.field_for('languages', 1), ('eng', 'fra', 'deu'))
ae(cache.field_for('languages', 2), ('spa', 'nld'))
ae(cache.field_for('languages', 3), ('eng',))
ae(sf('languages', {3:None}), set([3]))
ae(cache.field_for('languages', 3), ())
ae(sf('languages', {1:'deu,fra,eng'}), set([1]), 'Changing order failed')
ae(sf('languages', {2:'deu,eng,eng'}), set([2]))
cache2 = self.init_cache(cl)
for c in (cache, cache2):
ae(cache.field_for('languages', 1), ('deu', 'fra', 'eng'))
ae(cache.field_for('languages', 2), ('deu', 'eng'))
del cache2
# Identifiers
f = cache.fields['identifiers']
ae(sf('identifiers', {3: 'one:1,two:2'}), set([3]))
ae(sf('identifiers', {2:None}), set([2]))
ae(sf('identifiers', {1: {'test':'1', 'two':'2'}}), set([1]))
cache2 = self.init_cache(cl)
for c in (cache, cache2):
ae(c.field_for('identifiers', 3), {'one':'1', 'two':'2'})
ae(c.field_for('identifiers', 2), {})
ae(c.field_for('identifiers', 1), {'test':'1', 'two':'2'})
del cache2
# Test setting of title sort
ae(sf('title', {1:'The Moose', 2:'Cat'}), {1, 2})
cache2 = self.init_cache(cl)
for c in (cache, cache2):
ae(c.field_for('sort', 1), 'Moose, The')
ae(c.field_for('sort', 2), 'Cat')
# }}} # }}}
def tests():
tl = unittest.TestLoader()
# return tl.loadTestsFromName('writing.WritingTest.test_many_many_basic')
return tl.loadTestsFromTestCase(WritingTest)
def run():
unittest.TextTestRunner(verbosity=2).run(tests())
if __name__ == '__main__':
run()

62
src/calibre/db/write.py
View File

@ -12,9 +12,10 @@ from functools import partial
from datetime import datetime from datetime import datetime
from calibre.constants import preferred_encoding, ispy3 from calibre.constants import preferred_encoding, ispy3
from calibre.ebooks.metadata import author_to_author_sort from calibre.ebooks.metadata import author_to_author_sort, title_sort
from calibre.utils.date import (parse_only_date, parse_date, UNDEFINED_DATE, from calibre.utils.date import (parse_only_date, parse_date, UNDEFINED_DATE,
isoformat) isoformat)
from calibre.utils.localization import canonicalize_lang
from calibre.utils.icu import strcmp from calibre.utils.icu import strcmp
if ispy3: if ispy3:
@ -96,6 +97,30 @@ def adapt_bool(x):
x = bool(int(x)) x = bool(int(x))
return x if x is None else bool(x) return x if x is None else bool(x)
def adapt_languages(to_tuple, x):
ans = []
for lang in to_tuple(x):
lc = canonicalize_lang(lang)
if not lc or lc in ans or lc in ('und', 'zxx', 'mis', 'mul'):
continue
ans.append(lc)
return tuple(ans)
def clean_identifier(typ, val):
typ = icu_lower(typ).strip().replace(':', '').replace(',', '')
val = val.strip().replace(',', '|').replace(':', '|')
return typ, val
def adapt_identifiers(to_tuple, x):
if not isinstance(x, dict):
x = {k:v for k, v in (y.partition(':')[0::2] for y in to_tuple(x))}
ans = {}
for k, v in x.iteritems():
k, v = clean_identifier(k, v)
if k and v:
ans[k] = v
return ans
def get_adapter(name, metadata): def get_adapter(name, metadata):
dt = metadata['datatype'] dt = metadata['datatype']
if dt == 'text': if dt == 'text':
@ -133,6 +158,10 @@ def get_adapter(name, metadata):
return lambda x: ans(x) or UNDEFINED_DATE return lambda x: ans(x) or UNDEFINED_DATE
if name == 'series_index': if name == 'series_index':
return lambda x: 1.0 if ans(x) is None else ans(x) return lambda x: 1.0 if ans(x) is None else ans(x)
if name == 'languages':
return partial(adapt_languages, ans)
if name == 'identifiers':
return partial(adapt_identifiers, ans)
return ans return ans
# }}} # }}}
@ -145,6 +174,10 @@ def one_one_in_books(book_id_val_map, db, field, *args):
db.conn.executemany( db.conn.executemany(
'UPDATE books SET %s=? WHERE id=?'%field.metadata['column'], sequence) 'UPDATE books SET %s=? WHERE id=?'%field.metadata['column'], sequence)
field.table.book_col_map.update(book_id_val_map) field.table.book_col_map.update(book_id_val_map)
if field.name == 'title':
# Set the title sort field
field.title_sort_field.writer.set_books(
{k:title_sort(v) for k, v in book_id_val_map.iteritems()}, db)
return set(book_id_val_map) return set(book_id_val_map)
def one_one_in_other(book_id_val_map, db, field, *args): def one_one_in_other(book_id_val_map, db, field, *args):
@ -384,6 +417,31 @@ def many_many(book_id_val_map, db, field, allow_case_change, *args):
# }}} # }}}
def identifiers(book_id_val_map, db, field, *args): # {{{
table = field.table
updates = set()
for book_id, identifiers in book_id_val_map.iteritems():
if book_id not in table.book_col_map:
table.book_col_map[book_id] = {}
current_ids = table.book_col_map[book_id]
remove_keys = set(current_ids) - set(identifiers)
for key in remove_keys:
table.col_book_map.get(key, set()).discard(book_id)
current_ids.pop(key, None)
current_ids.update(identifiers)
for key, val in identifiers.iteritems():
if key not in table.col_book_map:
table.col_book_map[key] = set()
table.col_book_map[key].add(book_id)
updates.add((book_id, key, val))
db.conn.executemany('DELETE FROM identifiers WHERE book=?',
((x,) for x in book_id_val_map))
if updates:
db.conn.executemany('INSERT OR REPLACE INTO identifiers (book, type, val) VALUES (?, ?, ?)',
tuple(updates))
return set(book_id_val_map)
# }}}
def dummy(book_id_val_map, *args): def dummy(book_id_val_map, *args):
return set() return set()
@ -400,6 +458,8 @@ class Writer(object):
self.set_books_func = dummy self.set_books_func = dummy
elif self.name[0] == '#' and self.name.endswith('_index'): elif self.name[0] == '#' and self.name.endswith('_index'):
self.set_books_func = custom_series_index self.set_books_func = custom_series_index
elif self.name == 'identifiers':
self.set_books_func = identifiers
elif field.is_many_many: elif field.is_many_many:
self.set_books_func = many_many self.set_books_func = many_many
elif field.is_many: elif field.is_many:

11
src/calibre/gui2/actions/convert.py
View File

@ -167,8 +167,8 @@ class ConvertAction(InterfaceAction):
def queue_convert_jobs(self, jobs, changed, bad, rows, previous, def queue_convert_jobs(self, jobs, changed, bad, rows, previous,
converted_func, extra_job_args=[], rows_are_ids=False): converted_func, extra_job_args=[], rows_are_ids=False):
for func, args, desc, fmt, id, temp_files in jobs: for func, args, desc, fmt, id, temp_files in jobs:
func, _, same_fmt = func.partition(':') func, _, parts = func.partition(':')
same_fmt = same_fmt == 'same_fmt' parts = {x for x in parts.split(';')}
input_file = args[0] input_file = args[0]
input_fmt = os.path.splitext(input_file)[1] input_fmt = os.path.splitext(input_file)[1]
core_usage = 1 core_usage = 1
@ -182,7 +182,8 @@ class ConvertAction(InterfaceAction):
job = self.gui.job_manager.run_job(Dispatcher(converted_func), job = self.gui.job_manager.run_job(Dispatcher(converted_func),
func, args=args, description=desc, func, args=args, description=desc,
core_usage=core_usage) core_usage=core_usage)
job.conversion_of_same_fmt = same_fmt job.conversion_of_same_fmt = 'same_fmt' in parts
job.manually_fine_tune_toc = 'manually_fine_tune_toc' in parts
args = [temp_files, fmt, id]+extra_job_args args = [temp_files, fmt, id]+extra_job_args
self.conversion_jobs[job] = tuple(args) self.conversion_jobs[job] = tuple(args)
@ -223,6 +224,7 @@ class ConvertAction(InterfaceAction):
self.gui.job_exception(job) self.gui.job_exception(job)
return return
same_fmt = getattr(job, 'conversion_of_same_fmt', False) same_fmt = getattr(job, 'conversion_of_same_fmt', False)
manually_fine_tune_toc = getattr(job, 'manually_fine_tune_toc', False)
fmtf = temp_files[-1].name fmtf = temp_files[-1].name
if os.stat(fmtf).st_size < 1: if os.stat(fmtf).st_size < 1:
raise Exception(_('Empty output file, ' raise Exception(_('Empty output file, '
@ -248,4 +250,7 @@ class ConvertAction(InterfaceAction):
current = self.gui.library_view.currentIndex() current = self.gui.library_view.currentIndex()
if current.isValid(): if current.isValid():
self.gui.library_view.model().current_changed(current, QModelIndex()) self.gui.library_view.model().current_changed(current, QModelIndex())
if manually_fine_tune_toc:
self.gui.iactions['Edit ToC'].do_one(book_id, fmt.upper())

28
src/calibre/gui2/auto_add.py
View File

@ -15,7 +15,8 @@ from PyQt4.Qt import (QFileSystemWatcher, QObject, Qt, pyqtSignal, QTimer)
from calibre import prints from calibre import prints
from calibre.ptempfile import PersistentTemporaryDirectory from calibre.ptempfile import PersistentTemporaryDirectory
from calibre.ebooks import BOOK_EXTENSIONS from calibre.ebooks import BOOK_EXTENSIONS
from calibre.gui2 import question_dialog, gprefs from calibre.gui2 import gprefs
from calibre.gui2.dialogs.duplicates import DuplicatesQuestion
AUTO_ADDED = frozenset(BOOK_EXTENSIONS) - {'pdr', 'mbp', 'tan'} AUTO_ADDED = frozenset(BOOK_EXTENSIONS) - {'pdr', 'mbp', 'tan'}
@ -218,17 +219,20 @@ class AutoAdder(QObject):
paths.extend(p) paths.extend(p)
formats.extend(f) formats.extend(f)
metadata.extend(mis) metadata.extend(mis)
files = [_('%(title)s by %(author)s')%dict(title=mi.title, dups = [(mi, mi.cover, [p]) for mi, p in zip(metadata, paths)]
author=mi.format_field('authors')[1]) for mi in metadata] d = DuplicatesQuestion(m.db, dups, parent=gui)
if question_dialog(self.parent(), _('Duplicates found!'), dups = tuple(d.duplicates)
_('Books with the same title as the following already ' if dups:
'exist in the database. Add them anyway?'), paths, formats, metadata = [], [], []
'\n'.join(files)): for mi, cover, book_paths in dups:
dups, ids = m.add_books(paths, formats, metadata, paths.extend(book_paths)
add_duplicates=True, return_ids=True) formats.extend([p.rpartition('.')[-1] for p in book_paths])
added_ids |= set(ids) metadata.extend([mi for i in book_paths])
num = len(ids) ids = m.add_books(paths, formats, metadata,
count += num add_duplicates=True, return_ids=True)[1]
added_ids |= set(ids)
num = len(ids)
count += num
for tdir in data.itervalues(): for tdir in data.itervalues():
try: try:

1
src/calibre/gui2/convert/bulk.py
View File

@ -88,6 +88,7 @@ class BulkConfig(Config):
ps = widget_factory(PageSetupWidget) ps = widget_factory(PageSetupWidget)
sd = widget_factory(StructureDetectionWidget) sd = widget_factory(StructureDetectionWidget)
toc = widget_factory(TOCWidget) toc = widget_factory(TOCWidget)
toc.manually_fine_tune_toc.hide()
output_widget = self.plumber.output_plugin.gui_configuration_widget( output_widget = self.plumber.output_plugin.gui_configuration_widget(
self.stack, self.plumber.get_option_by_name, self.stack, self.plumber.get_option_by_name,

8
src/calibre/gui2/convert/single.py
View File

@ -165,6 +165,12 @@ class Config(ResizableDialog, Ui_Dialog):
def output_format(self): def output_format(self):
return unicode(self.output_formats.currentText()).lower() return unicode(self.output_formats.currentText()).lower()
@property
def manually_fine_tune_toc(self):
for i in xrange(self.stack.count()):
w = self.stack.widget(i)
if hasattr(w, 'manually_fine_tune_toc'):
return w.manually_fine_tune_toc.isChecked()
def setup_pipeline(self, *args): def setup_pipeline(self, *args):
oidx = self.groups.currentIndex().row() oidx = self.groups.currentIndex().row()
@ -191,6 +197,8 @@ class Config(ResizableDialog, Ui_Dialog):
ps = widget_factory(PageSetupWidget) ps = widget_factory(PageSetupWidget)
sd = widget_factory(StructureDetectionWidget) sd = widget_factory(StructureDetectionWidget)
toc = widget_factory(TOCWidget) toc = widget_factory(TOCWidget)
from calibre.gui2.actions.toc_edit import SUPPORTED
toc.manually_fine_tune_toc.setVisible(output_format.upper() in SUPPORTED)
debug = widget_factory(DebugWidget) debug = widget_factory(DebugWidget)
output_widget = self.plumber.output_plugin.gui_configuration_widget( output_widget = self.plumber.output_plugin.gui_configuration_widget(

111
src/calibre/gui2/convert/toc.ui
View File

@ -6,22 +6,32 @@
<rect> <rect>
<x>0</x> <x>0</x>
<y>0</y> <y>0</y>
<width>436</width> <width>596</width>
<height>382</height> <height>493</height>
</rect> </rect>
</property> </property>
<property name="windowTitle"> <property name="windowTitle">
<string>Form</string> <string>Form</string>
</property> </property>
<layout class="QGridLayout" name="gridLayout"> <layout class="QGridLayout" name="gridLayout">
<item row="7" column="1">
<widget class="QSpinBox" name="opt_toc_threshold"/>
</item>
<item row="1" column="0" colspan="2"> <item row="1" column="0" colspan="2">
<widget class="QCheckBox" name="opt_use_auto_toc">
<property name="text">
<string>&amp;Force use of auto-generated Table of Contents</string>
</property>
</widget>
</item>
<item row="2" column="0" colspan="2">
<widget class="QCheckBox" name="opt_no_chapters_in_toc"> <widget class="QCheckBox" name="opt_no_chapters_in_toc">
<property name="text"> <property name="text">
<string>Do not add &amp;detected chapters to the Table of Contents</string> <string>Do not add &amp;detected chapters to the Table of Contents</string>
</property> </property>
</widget> </widget>
</item> </item>
<item row="3" column="0"> <item row="6" column="0">
<widget class="QLabel" name="label_10"> <widget class="QLabel" name="label_10">
<property name="text"> <property name="text">
<string>Number of &amp;links to add to Table of Contents</string> <string>Number of &amp;links to add to Table of Contents</string>
@ -31,34 +41,7 @@
</property> </property>
</widget> </widget>
</item> </item>
<item row="3" column="1"> <item row="8" column="0">
<widget class="QSpinBox" name="opt_max_toc_links">
<property name="maximum">
<number>10000</number>
</property>
</widget>
</item>
<item row="4" column="0">
<widget class="QLabel" name="label_16">
<property name="text">
<string>Chapter &amp;threshold</string>
</property>
<property name="buddy">
<cstring>opt_toc_threshold</cstring>
</property>
</widget>
</item>
<item row="4" column="1">
<widget class="QSpinBox" name="opt_toc_threshold"/>
</item>
<item row="0" column="0" colspan="2">
<widget class="QCheckBox" name="opt_use_auto_toc">
<property name="text">
<string>&amp;Force use of auto-generated Table of Contents</string>
</property>
</widget>
</item>
<item row="5" column="0">
<widget class="QLabel" name="label"> <widget class="QLabel" name="label">
<property name="text"> <property name="text">
<string>TOC &amp;Filter:</string> <string>TOC &amp;Filter:</string>
@ -68,19 +51,27 @@
</property> </property>
</widget> </widget>
</item> </item>
<item row="5" column="1"> <item row="11" column="0" colspan="2">
<widget class="QLineEdit" name="opt_toc_filter"/>
</item>
<item row="6" column="0" colspan="2">
<widget class="XPathEdit" name="opt_level1_toc" native="true"/>
</item>
<item row="7" column="0" colspan="2">
<widget class="XPathEdit" name="opt_level2_toc" native="true"/>
</item>
<item row="8" column="0" colspan="2">
<widget class="XPathEdit" name="opt_level3_toc" native="true"/> <widget class="XPathEdit" name="opt_level3_toc" native="true"/>
</item> </item>
<item row="9" column="0"> <item row="6" column="1">
<widget class="QSpinBox" name="opt_max_toc_links">
<property name="maximum">
<number>10000</number>
</property>
</widget>
</item>
<item row="7" column="0">
<widget class="QLabel" name="label_16">
<property name="text">
<string>Chapter &amp;threshold</string>
</property>
<property name="buddy">
<cstring>opt_toc_threshold</cstring>
</property>
</widget>
</item>
<item row="13" column="0">
<spacer name="verticalSpacer"> <spacer name="verticalSpacer">
<property name="orientation"> <property name="orientation">
<enum>Qt::Vertical</enum> <enum>Qt::Vertical</enum>
@ -93,13 +84,47 @@
</property> </property>
</spacer> </spacer>
</item> </item>
<item row="2" column="0" colspan="2"> <item row="3" column="0" colspan="2">
<widget class="QCheckBox" name="opt_duplicate_links_in_toc"> <widget class="QCheckBox" name="opt_duplicate_links_in_toc">
<property name="text"> <property name="text">
<string>Allow &amp;duplicate links when creating the Table of Contents</string> <string>Allow &amp;duplicate links when creating the Table of Contents</string>
</property> </property>
</widget> </widget>
</item> </item>
<item row="10" column="0" colspan="2">
<widget class="XPathEdit" name="opt_level2_toc" native="true"/>
</item>
<item row="8" column="1">
<widget class="QLineEdit" name="opt_toc_filter"/>
</item>
<item row="9" column="0" colspan="2">
<widget class="XPathEdit" name="opt_level1_toc" native="true"/>
</item>
<item row="0" column="0" colspan="2">
<widget class="QLabel" name="label_2">
<property name="text">
<string>&lt;a href=&quot;http://manual.calibre-ebook.com/conversion.html#table-of-contents&quot;&gt;Help with using these options to generate a Table of Contents&lt;/a&gt;</string>
</property>
<property name="wordWrap">
<bool>true</bool>
</property>
<property name="openExternalLinks">
<bool>true</bool>
</property>
</widget>
</item>
<item row="12" column="0" colspan="2">
<widget class="QCheckBox" name="manually_fine_tune_toc">
<property name="toolTip">
<string>This option will cause calibre to popup the Table of Contents Editor tool,
which will allow you to manually edit the Table of Contents, to fix any errors
caused by automatic generation.</string>
</property>
<property name="text">
<string>&amp;Manually fine-tune the ToC after conversion is completed</string>
</property>
</widget>
</item>
</layout> </layout>
</widget> </widget>
<customwidgets> <customwidgets>

10
src/calibre/gui2/dialogs/book_info.py
View File

@ -5,7 +5,7 @@ __docformat__ = 'restructuredtext en'
from PyQt4.Qt import (QCoreApplication, SIGNAL, QModelIndex, QTimer, Qt, from PyQt4.Qt import (QCoreApplication, SIGNAL, QModelIndex, QTimer, Qt,
QDialog, QPixmap, QIcon, QSize, QPalette) QDialog, QPixmap, QIcon, QSize, QPalette, QShortcut, QKeySequence)
from calibre.gui2.dialogs.book_info_ui import Ui_BookInfo from calibre.gui2.dialogs.book_info_ui import Ui_BookInfo
from calibre.gui2 import dynamic from calibre.gui2 import dynamic
@ -43,6 +43,14 @@ class BookInfo(QDialog, Ui_BookInfo):
self.fit_cover.stateChanged.connect(self.toggle_cover_fit) self.fit_cover.stateChanged.connect(self.toggle_cover_fit)
self.cover.resizeEvent = self.cover_view_resized self.cover.resizeEvent = self.cover_view_resized
self.cover.cover_changed.connect(self.cover_changed) self.cover.cover_changed.connect(self.cover_changed)
self.ns = QShortcut(QKeySequence('Alt+Right'), self)
self.ns.activated.connect(self.next)
self.ps = QShortcut(QKeySequence('Alt+Left'), self)
self.ps.activated.connect(self.previous)
self.next_button.setToolTip(_('Next [%s]')%
unicode(self.ns.key().toString(QKeySequence.NativeText)))
self.previous_button.setToolTip(_('Previous [%s]')%
unicode(self.ps.key().toString(QKeySequence.NativeText)))
desktop = QCoreApplication.instance().desktop() desktop = QCoreApplication.instance().desktop()
screen_height = desktop.availableGeometry().height() - 100 screen_height = desktop.availableGeometry().height() - 100

2
src/calibre/gui2/email.py
View File

@ -160,7 +160,7 @@ def email_news(mi, remove, get_fmts, done, job_manager):
return sent_mails return sent_mails
plugboard_email_value = 'email' plugboard_email_value = 'email'
plugboard_email_formats = ['epub', 'mobi'] plugboard_email_formats = ['epub', 'mobi', 'azw3']
class EmailMixin(object): # {{{ class EmailMixin(object): # {{{

2
src/calibre/gui2/preferences/conversion.py
View File

@ -61,6 +61,8 @@ class Base(ConfigWidgetBase, Ui_Form):
for w in widgets: for w in widgets:
w.changed_signal.connect(self.changed_signal) w.changed_signal.connect(self.changed_signal)
self.stack.addWidget(w) self.stack.addWidget(w)
if isinstance(w, TOCWidget):
w.manually_fine_tune_toc.hide()
self.list.currentChanged = self.category_current_changed self.list.currentChanged = self.category_current_changed
self.list.setCurrentIndex(self.model.index(0)) self.list.setCurrentIndex(self.model.index(0))

59
src/calibre/gui2/toc/location.py
View File

@ -11,10 +11,11 @@ from base64 import b64encode
from PyQt4.Qt import (QWidget, QGridLayout, QListWidget, QSize, Qt, QUrl, from PyQt4.Qt import (QWidget, QGridLayout, QListWidget, QSize, Qt, QUrl,
pyqtSlot, pyqtSignal, QVBoxLayout, QFrame, QLabel, pyqtSlot, pyqtSignal, QVBoxLayout, QFrame, QLabel,
QLineEdit, QTimer) QLineEdit, QTimer, QPushButton, QIcon)
from PyQt4.QtWebKit import QWebView, QWebPage, QWebElement from PyQt4.QtWebKit import QWebView, QWebPage, QWebElement
from calibre.ebooks.oeb.display.webview import load_html from calibre.ebooks.oeb.display.webview import load_html
from calibre.gui2 import error_dialog, question_dialog
from calibre.utils.logging import default_log from calibre.utils.logging import default_log
class Page(QWebPage): # {{{ class Page(QWebPage): # {{{
@ -115,16 +116,26 @@ class ItemEdit(QWidget):
self.dest_list = dl = QListWidget(self) self.dest_list = dl = QListWidget(self)
dl.setMinimumWidth(250) dl.setMinimumWidth(250)
dl.currentItemChanged.connect(self.current_changed) dl.currentItemChanged.connect(self.current_changed)
l.addWidget(dl, 1, 0) l.addWidget(dl, 1, 0, 2, 1)
self.view = WebView(self) self.view = WebView(self)
self.view.elem_clicked.connect(self.elem_clicked) self.view.elem_clicked.connect(self.elem_clicked)
l.addWidget(self.view, 1, 1) l.addWidget(self.view, 1, 1, 1, 3)
self.f = f = QFrame() self.f = f = QFrame()
f.setFrameShape(f.StyledPanel) f.setFrameShape(f.StyledPanel)
f.setMinimumWidth(250) f.setMinimumWidth(250)
l.addWidget(f, 1, 2) l.addWidget(f, 1, 4, 2, 1)
self.search_text = s = QLineEdit(self)
s.setPlaceholderText(_('Search for text...'))
l.addWidget(s, 2, 1, 1, 1)
self.ns_button = b = QPushButton(QIcon(I('arrow-down.png')), _('Find &next'), self)
b.clicked.connect(self.find_next)
l.addWidget(b, 2, 2, 1, 1)
self.ps_button = b = QPushButton(QIcon(I('arrow-up.png')), _('Find &previous'), self)
l.addWidget(b, 2, 3, 1, 1)
b.clicked.connect(self.find_previous)
l.setRowStretch(1, 10)
l = f.l = QVBoxLayout() l = f.l = QVBoxLayout()
f.setLayout(l) f.setLayout(l)
@ -156,6 +167,42 @@ class ItemEdit(QWidget):
l.addStretch() l.addStretch()
def keyPressEvent(self, ev):
if ev.key() in (Qt.Key_Return, Qt.Key_Enter) and self.search_text.hasFocus():
# Prevent pressing enter in the search box from triggering the dialog's accept() method
ev.accept()
return
return super(ItemEdit, self).keyPressEvent(ev)
def find(self, forwards=True):
text = unicode(self.search_text.text()).strip()
flags = QWebPage.FindFlags(0) if forwards else QWebPage.FindBackward
d = self.dest_list
if d.count() == 1:
flags |= QWebPage.FindWrapsAroundDocument
if not self.view.findText(text, flags) and text:
if d.count() == 1:
return error_dialog(self, _('No match found'),
_('No match found for: %s')%text, show=True)
delta = 1 if forwards else -1
current = unicode(d.currentItem().data(Qt.DisplayRole).toString())
next_index = (d.currentRow() + delta)%d.count()
next = unicode(d.item(next_index).data(Qt.DisplayRole).toString())
msg = '<p>'+_('No matches for %(text)s found in the current file [%(current)s].'
' Do you want to search in the %(which)s file [%(next)s]?')
msg = msg%dict(text=text, current=current, next=next,
which=_('next') if forwards else _('previous'))
if question_dialog(self, _('No match found'), msg):
self.pending_search = self.find_next if forwards else self.find_previous
d.setCurrentRow(next_index)
def find_next(self):
return self.find()
def find_previous(self):
return self.find(forwards=False)
def load(self, container): def load(self, container):
self.container = container self.container = container
spine_names = [container.abspath_to_name(p) for p in spine_names = [container.abspath_to_name(p) for p in
@ -175,6 +222,10 @@ class ItemEdit(QWidget):
self.view.load_js() self.view.load_js()
self.dest_label.setText(self.base_msg + '<br>' + _('File:') + ' ' + self.dest_label.setText(self.base_msg + '<br>' + _('File:') + ' ' +
name + '<br>' + _('Top of the file')) name + '<br>' + _('Top of the file'))
if hasattr(self, 'pending_search'):
f = self.pending_search
del self.pending_search
f()
def __call__(self, item, where): def __call__(self, item, where):
self.current_item, self.current_where = item, where self.current_item, self.current_where = item, where

32
src/calibre/gui2/toc/main.py
View File

@ -126,6 +126,7 @@ class ItemView(QFrame): # {{{
go_to_root = pyqtSignal() go_to_root = pyqtSignal()
create_from_xpath = pyqtSignal(object) create_from_xpath = pyqtSignal(object)
create_from_links = pyqtSignal() create_from_links = pyqtSignal()
flatten_toc = pyqtSignal()
def __init__(self, parent): def __init__(self, parent):
QFrame.__init__(self, parent) QFrame.__init__(self, parent)
@ -189,6 +190,13 @@ class ItemView(QFrame): # {{{
))) )))
l.addWidget(b) l.addWidget(b)
self.fal = b = QPushButton(_('Flatten the ToC'))
b.clicked.connect(self.flatten_toc)
b.setToolTip(textwrap.fill(_(
'Flatten the Table of Contents, putting all entries at the top level'
)))
l.addWidget(b)
l.addStretch() l.addStretch()
self.w1 = la = QLabel(_('<b>WARNING:</b> calibre only supports the ' self.w1 = la = QLabel(_('<b>WARNING:</b> calibre only supports the '
@ -388,6 +396,7 @@ class TOCView(QWidget): # {{{
i.create_from_xpath.connect(self.create_from_xpath) i.create_from_xpath.connect(self.create_from_xpath)
i.create_from_links.connect(self.create_from_links) i.create_from_links.connect(self.create_from_links)
i.flatten_item.connect(self.flatten_item) i.flatten_item.connect(self.flatten_item)
i.flatten_toc.connect(self.flatten_toc)
i.go_to_root.connect(self.go_to_root) i.go_to_root.connect(self.go_to_root)
l.addWidget(i, 0, 4, col, 1) l.addWidget(i, 0, 4, col, 1)
@ -413,8 +422,29 @@ class TOCView(QWidget): # {{{
p = item.parent() or self.root p = item.parent() or self.root
p.removeChild(item) p.removeChild(item)
def iteritems(self, parent=None):
if parent is None:
parent = self.root
for i in xrange(parent.childCount()):
child = parent.child(i)
yield child
for gc in self.iteritems(parent=child):
yield gc
def flatten_toc(self):
found = True
while found:
found = False
for item in self.iteritems():
if item.childCount() > 0:
self._flatten_item(item)
found = True
break
def flatten_item(self): def flatten_item(self):
item = self.tocw.currentItem() self._flatten_item(self.tocw.currentItem())
def _flatten_item(self, item):
if item is not None: if item is not None:
p = item.parent() or self.root p = item.parent() or self.root
idx = p.indexOfChild(item) idx = p.indexOfChild(item)

7
src/calibre/gui2/tools.py
View File

@ -82,8 +82,13 @@ def convert_single_ebook(parent, db, book_ids, auto_conversion=False, # {{{
args = [in_file.name, out_file.name, recs] args = [in_file.name, out_file.name, recs]
temp_files.append(out_file) temp_files.append(out_file)
func = 'gui_convert_override' func = 'gui_convert_override'
parts = []
if not auto_conversion and d.manually_fine_tune_toc:
parts.append('manually_fine_tune_toc')
if same_fmt: if same_fmt:
func += ':same_fmt' parts.append('same_fmt')
if parts:
func += ':%s'%(';'.join(parts))
jobs.append((func, args, desc, d.output_format.upper(), book_id, temp_files)) jobs.append((func, args, desc, d.output_format.upper(), book_id, temp_files))
changed = True changed = True

15
src/calibre/library/cli.py
View File

@ -592,6 +592,9 @@ def command_set_metadata(args, dbpath):
print >>sys.stderr, _('You must specify either a field or an opf file') print >>sys.stderr, _('You must specify either a field or an opf file')
return 1 return 1
book_id = int(args[1]) book_id = int(args[1])
if book_id not in db.all_ids():
prints(_('No book with id: %s in the database')%book_id, file=sys.stderr)
raise SystemExit(1)
if len(args) > 2: if len(args) > 2:
opf = args[2] opf = args[2]
@ -870,6 +873,9 @@ def parse_series_string(db, label, value):
return val, s_index return val, s_index
def do_set_custom(db, col, id_, val, append): def do_set_custom(db, col, id_, val, append):
if id_ not in db.all_ids():
prints(_('No book with id: %s in the database')%id_, file=sys.stderr)
raise SystemExit(1)
if db.custom_column_label_map[col]['datatype'] == 'series': if db.custom_column_label_map[col]['datatype'] == 'series':
val, s_index = parse_series_string(db, col, val) val, s_index = parse_series_string(db, col, val)
db.set_custom(id_, val, extra=s_index, label=col, append=append) db.set_custom(id_, val, extra=s_index, label=col, append=append)
@ -941,11 +947,16 @@ def command_custom_columns(args, dbpath):
def do_remove_custom_column(db, label, force): def do_remove_custom_column(db, label, force):
if not force: if not force:
q = raw_input(_('You will lose all data in the column: %r.' q = raw_input(_('You will lose all data in the column: %s.'
' Are you sure (y/n)? ')%label) ' Are you sure (y/n)? ')%label)
if q.lower().strip() != _('y'): if q.lower().strip() != _('y'):
return return
db.delete_custom_column(label=label) try:
db.delete_custom_column(label=label)
except KeyError:
prints(_('No column named %s found. You must use column labels, not titles.'
' Use calibredb custom_columns to get a list of labels.')%label, file=sys.stderr)
raise SystemExit(1)
prints('Column %r removed.'%label) prints('Column %r removed.'%label)
def remove_custom_column_option_parser(): def remove_custom_column_option_parser():

38
src/calibre/library/database2.py
View File

@ -1343,23 +1343,39 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
if not isinstance(dest, basestring): if not isinstance(dest, basestring):
raise Exception("Error, you must pass the dest as a path when" raise Exception("Error, you must pass the dest as a path when"
" using windows_atomic_move") " using windows_atomic_move")
if dest and not samefile(dest, path): if dest:
windows_atomic_move.copy_path_to(path, dest) if samefile(path, dest):
# Ensure that the file has the same case as dest
try:
if path != dest:
os.rename(path, dest)
except:
pass # Nothing too catastrophic happened, the cases mismatch, that's all
else:
windows_atomic_move.copy_path_to(path, dest)
else: else:
if hasattr(dest, 'write'): if hasattr(dest, 'write'):
with lopen(path, 'rb') as f: with lopen(path, 'rb') as f:
shutil.copyfileobj(f, dest) shutil.copyfileobj(f, dest)
if hasattr(dest, 'flush'): if hasattr(dest, 'flush'):
dest.flush() dest.flush()
elif dest and not samefile(dest, path): elif dest:
if use_hardlink: if samefile(dest, path):
try: if not self.is_case_sensitive and path != dest:
hardlink_file(path, dest) # Ensure that the file has the same case as dest
return try:
except: os.rename(path, dest)
pass except:
with lopen(path, 'rb') as f, lopen(dest, 'wb') as d: pass # Nothing too catastrophic happened, the cases mismatch, that's all
shutil.copyfileobj(f, d) else:
if use_hardlink:
try:
hardlink_file(path, dest)
return
except:
pass
with lopen(path, 'rb') as f, lopen(dest, 'wb') as d:
shutil.copyfileobj(f, d)
def copy_cover_to(self, index, dest, index_is_id=False, def copy_cover_to(self, index, dest, index_is_id=False,
windows_atomic_move=None, use_hardlink=False): windows_atomic_move=None, use_hardlink=False):

989
src/calibre/translations/af.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/ar.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/ast.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/az.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/ber.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/bg.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/bn.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/br.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/bs.po
View File

File diff suppressed because it is too large Load Diff

1023
src/calibre/translations/ca.po
View File

File diff suppressed because it is too large Load Diff

474
src/calibre/translations/calibre.pot
View File

File diff suppressed because it is too large Load Diff

981
src/calibre/translations/cs.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/cy.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/da.po
View File

File diff suppressed because it is too large Load Diff

1023
src/calibre/translations/de.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/el.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/en_AU.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/en_CA.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/en_GB.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/eo.po
View File

File diff suppressed because it is too large Load Diff

1161
src/calibre/translations/es.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/et.po
View File

File diff suppressed because it is too large Load Diff

981
src/calibre/translations/eu.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/fa.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/fi.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/fo.po
View File

File diff suppressed because it is too large Load Diff

999
src/calibre/translations/fr.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/fr_CA.po
View File

File diff suppressed because it is too large Load Diff

979
src/calibre/translations/fur.po
View File

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More