Cutlery/calibre
1
0
Fork 0
mirror of https://github.com/kovidgoyal/calibre.git synced 2025-07-09 03:04:10 -04:00
Code Issues Packages Projects Releases Wiki Activity

Sync to pluginize

Browse Source
This commit is contained in:
John Schember 2009-05-15 06:28:37 -04:00
commit b722903a1a
90 changed files with 3285 additions and 2862 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

363
COPYRIGHT Normal file
View File

@ -0,0 +1,363 @@
Files: *
Copyright: Copyright (C) 2008 Kovid Goyal <kovid@kovidgoyal.net>
License: GPL-3
The full text of the GPL is distributed as in
/usr/share/common-licenses/GPL-3 on Debian systems.
Files: src/calibre/ebooks/BeautifulSoup.py
Copyright: Copyright (c) 2004-2007, Leonard Richardson
License: BSD
The full text of the BSD license is distributed as in
/usr/share/common-licenses/BSD on Debian systems.
Files: src/calibre/ebooks/chardet/*
Copyright: Copyright (C) 1998-2001 Netscape Communications Corporation
License: LGPL-2.1+
The full text of the LGPL is distributed as in
/usr/share/common-licenses/LGPL-2.1 on Debian systems.
Files: src/calibre/ebooks/hyphenate.py
Copyright: Copyright (C) 1990, 2004, 2005 Gerard D.C. Kuiken.
License: other
Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved.
Files: /src/cherrypy/*
Copyright: Copyright (c) 2004-2007, CherryPy Team (team@cherrypy.org)
Copyright: Copyright (C) 2005, Tiago Cogumbreiro <cogumbreiro@users.sf.net>
License: BSD
The full text of the BSD license is distributed as in
/usr/share/common-licenses/BSD on Debian systems.
Files: src/odf/*
Copyright: Copyright (C) 2006-2008 Søren Roug, European Environment Agency
License: LGPL2.1+
The full text of the LGPL is distributed as in
/usr/share/common-licenses/LGPL-2.1 on Debian systems.
Files: src/odf/teletype.py
Files: src/odf/easyliststyle.py
Copyright: Copyright (C) 2008, J. David Eisenberg
License: GPL2+
The full text of the GPL is distributed as in
/usr/share/common-licenses/GPL-2 on Debian systems.
Files: src/pyPdf/*
Copyright: Copyright (c) 2006, Mathieu Fenniak
Copyright: Copyright (c) 2007, Ashish Kulkarni <kulkarni.ashish@gmail.com>
License: BSD
The full text of the BSD license is distributed as in
/usr/share/common-licenses/BSD on Debian systems.
Files: src/calibre/utils/genshi/*
Copyright: Copyright (C) 2006-2008 Edgewall Software
License: BSD
The full text of the BSD license is distributed as in
/usr/share/common-licenses/BSD on Debian systems.
Files: src/calibre/utils/lzx/*
Copyright: Copyright (C) 2002, Matthew T. Russotto
Copyright: Copyright (C) 2008, Marshall T. Vandegrift <llasram@gmail.com>
Copyright: Copyright (C) 2006-2008, Alexander Chemeris
License: LGPL-2.1
The full text of the LGPL is distributed as in
/usr/share/common-licenses/LGPL-2.1 on Debian systems.
Files: src/calibre/utils/lzx/msstdint.h
Copyright: Copyright (C) 2006-2008, Alexander Chemeris
License: BSD
The full text of the BSD license is distributed as in
/usr/share/common-licenses/BSD on Debian systems.
Files: src/calibre/utils/pyparsing.py
Copyright: Copyright (c) 2003-2008, Paul T. McGuire
License: MIT
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Files: src/calibre/utils/PythonMagickWand.py
Copyright: (c) 2007 - Achim Domma - domma@procoders.net
License: MIT
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
Files: src/calibre/utils/msdes/d3des.h:
Files: src/calibre/utils/msdes/des.c:
Copyright: Copyright (C) 1988,1989,1990,1991,1992, Richard Outerbridge
License: Other
THIS SOFTWARE PLACED IN THE PUBLIC DOMAIN BY THE AUTHOUR
Files: src/calibre/utils/msdes/msdesmodule.c
Copyright: Copyright (C) 2008, Marshall T. Vandegrift <llasram@gmail.com>
License: GPL-3
The full text of the GPL is distributed as in
/usr/share/common-licenses/GPL-3 on Debian systems.
Files: src/calibre/utils/msdes/spr.h
Copyright: Copyright (C) 2002, Dan A. Jackson
License: GPL2+
The full text of the GPL is distributed as in
/usr/share/common-licenses/GPL-2 on Debian systems.
Files: src/calibre/gui2/pictureflow/*
Copyright: (C) Copyright 2007 Trolltech ASA
License: BSD
The full text of the BSD license is distributed as in
/usr/share/common-licenses/BSD on Debian systems.
Files: src/calibre/ebooks/lit/*
Copyright: 2008, Marshall T. Vandegrift <llasram@gmail.com>
License: GPL-3
The full text of the GPL is distributed as in
/usr/share/common-licenses/GPL-3 on Debian systems.
Files: src/calibre/ebooks/lrf/*
Copyright: 2008, Anatoly Shipitsin <norguhtar at gmail.com>
Copyright: copyright 2002 Paul Henry Tremblay
Copyright: Copyright (C) 2008 B.Scott Wxby [bswxby]
Copyright: Copyright (C) 2007 David Chen SonyReader<at>DaveChen<dot>org
Copyright: Copyright (c) 2007 Mike Higgins (Falstaff)
License: GPL-3
The full text of the GPL is distributed as in
/usr/share/common-licenses/GPL-3 on Debian systems.
Files: src/calibre/ebooks/BeautifulSoup.py
Copyright: Copyright (c) 2004-2007, Leonard Richardson
License: BSD
The full text of the BSD license is distributed as in
/usr/share/common-licenses/BSD on Debian systems.
Files: src/calibre/ebooks/rtf2xml/*
Copyright: copyright 2002 Paul Henry Tremblay
License: GPL
The full text of the GPL is distributed as in
/usr/share/common-licenses/GPL on Debian systems.
Files: src/calibre/web/feeds/feedparser.py
Copyright: Copyright (c) 2002-2006, Mark Pilgrim
License: BSD
The full text of the BSD license is distributed as in
/usr/share/common-licenses/BSD on Debian systems.
Files: src/calibre/web/feeds/recipes/*
Copyright: 2008, Darko Miletic <darko.miletic at gmail.com>
Copyright: 2008, Mathieu Godlewski <mathieu at godlewski.fr>
Copyright: Copyright (C) 2008 B.Scott Wxby [bswxby]
Copyright: Copyright (C) 2007 David Chen SonyReader<at>DaveChen<dot>org
Copyright: 2008, Derry FitzGerald
License: GPL-3
The full text of the GPL is distributed as in
/usr/share/common-licenses/GPL-3 on Debian systems.
Files: src/calibre/ebooks/metadata/*
Copyright: 2008, Ashish Kulkarni <kulkarni.ashish@gmail.com>
Copyright: Copyright (C) 2006 Søren Roug, European Environment Agency
License: GPL-3
The full text of the GPL is distributed as in
/usr/share/common-licenses/GPL-3 on Debian systems.
Files: src/encutils/__init__.py
Copyright: 2005-2008: Christof Hoeke
License: LGPL-3+, CC-BY-3.0
The full text of the LGPL is distributed as in
/usr/share/common-licenses/LGPL-3 on Debian systems.
Files: src/calibre/translations/*
Copyright: Copyright (C) 2007, Kovid Goyal
Copyright: Copyright (C) 2008, Rosetta Contributors and Canonical Ltd.
License: GPL-3
The full text of the GPL is distributed as in
/usr/share/common-licenses/GPL-3 on Debian systems.
Files: src/calibre/gui2/viewer/jquery.js
Files: src/calibre/gui2/viewer/jquery_scrollTo.js
Files: src/calibre/library/static/date.js
Copyright: Copyright (C) 2008, John Resig (jquery.com)
Copyright: Copyright (C) 2007-2008, Ariel Flesler - aflesler@gmail.com | http://flesler.blogspot.com
Copyright: Copyright (C) 2006-2007, Coolite Inc. (http://www.coolite.com/)
License: MIT
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Files: src/calibre/ebooks/lrf/fonts/liberation/*
Copyright: Copyright (C) 2007, Red Hat, Inc. All rights reserved.
License: Other
Copyright (C) 2007, Red Hat, Inc. All rights reserved.
LIBERATION is a trademark of Red Hat, Inc.
This agreement governs the use of the Software and any updates to the Software,
regardless of the delivery mechanism. Subject to the following terms, Red Hat, Inc.
("Red Hat") grants to the user ("Client") a license to this work pursuant to
the GNU General Public License v.2 with the exceptions set forth below and such
other terms as our set forth in this End User License Agreement.
1. The Software and License Exception. LIBERATION font software (the "Software")
consists of TrueType-OpenType formatted font software for rendering LIBERATION
typefaces in sans serif, serif, and monospaced character styles. You are licensed
to use, modify, copy, and distribute the Software pursuant to the GNU General
Public License v.2 with the following exceptions:
(a) As a special exception, if you create a document which uses this font, and
embed this font or unaltered portions of this font into the document, this
font does not by itself cause the resulting document to be covered by the
GNU General Public License. This exception does not however invalidate any
other reasons why the document might be covered by the GNU General Public
License. If you modify this font, you may extend this exception to your
version of the font, but you are not obligated to do so. If you do not
wish to do so, delete this exception statement from your version.
(b) As a further exception, any distribution of the object code of the Software
in a physical product must provide you the right to access and modify the
source code for the Software and to reinstall that modified version of the
Software in object code form on the same physical product on which you
received it.
2. Intellectual Property Rights. The Software and each of its components, including
the source code, documentation, appearance, structure and organization are owned
by Red Hat and others and are protected under copyright and other laws. Title to
the Software and any component, or to any copy, modification, or merged portion
shall remain with the aforementioned, subject to the applicable license.
The "LIBERATION" trademark is a trademark of Red Hat, Inc. in the U.S. and other
countries. This agreement does not permit Client to distribute modified versions
of the Software using Red Hat's trademarks. If Client makes a redistribution of
a modified version of the Software, then Client must modify the files names to
remove any reference to the Red Hat trademarks and must not use the Red Hat
trademarks in any way to reference or promote the modified Software.
3. Limited Warranty. To the maximum extent permitted under applicable law, the
Software is provided and licensed "as is" without warranty of any kind,
expressed or implied, including the implied warranties of merchantability,
non-infringement or fitness for a particular purpose. Red Hat does not warrant
that the functions contained in the Software will meet Client's requirements or
that the operation of the Software will be entirely error free or appear precisely
as described in the accompanying documentation.
4. Limitation of Remedies and Liability. To the maximum extent permitted by applicable
law, Red Hat or any Red Hat authorized dealer will not be liable to Client for any
incidental or consequential damages, including lost profits or lost savings arising
out of the use or inability to use the Software, even if Red Hat or such dealer has
been advised of the possibility of such damages.
5. General. If any provision of this agreement is held to be unenforceable, that shall
not affect the enforceability of the remaining provisions. This agreement shall be
governed by the laws of the State of North Carolina and of the United States, without
regard to any conflict of laws provisions, except that the United Nations Convention
on the International Sale of Goods shall not apply.
Files: installer/cx_Freeze/*
Copyright: Copyright © 2007-2008, Colt Engineering, Edmonton, Alberta, Canada.
Copyright: Copyright © 2001-2006, Computronix (Canada) Ltd., Edmonton, Alberta, Canada.
License: other
All rights reserved.
NOTE: this license is derived from the Python Software Foundation License
which can be found at http://www.python.org/psf/license
License for cx_Freeze 4.0.1
---------------------------
1. This LICENSE AGREEMENT is between the copyright holders and the Individual
or Organization ("Licensee") accessing and otherwise using cx_Freeze
software in source or binary form and its associated documentation.
2. Subject to the terms and conditions of this License Agreement, the
copyright holders hereby grant Licensee a nonexclusive, royalty-free,
world-wide license to reproduce, analyze, test, perform and/or display
publicly, prepare derivative works, distribute, and otherwise use cx_Freeze
alone or in any derivative version, provided, however, that this License
Agreement and this notice of copyright are retained in cx_Freeze alone or in
any derivative version prepared by Licensee.
3. In the event Licensee prepares a derivative work that is based on or
incorporates cx_Freeze or any part thereof, and wants to make the derivative
work available to others as provided herein, then Licensee hereby agrees to
include in any such work a brief summary of the changes made to cx_Freeze.
4. The copyright holders are making cx_Freeze available to Licensee on an
"AS IS" basis. THE COPYRIGHT HOLDERS MAKE NO REPRESENTATIONS OR WARRANTIES,
EXPRESS OR IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, THE COPYRIGHT
HOLDERS MAKE NO AND DISCLAIM ANY REPRESENTATION OR WARRANTY OF
MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF
CX_FREEZE WILL NOT INFRINGE ANY THIRD PARTY RIGHTS.
5. THE COPYRIGHT HOLDERS SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF
CX_FREEZE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING CX_FREEZE, OR ANY
DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
6. This License Agreement will automatically terminate upon a material breach
of its terms and conditions.
7. Nothing in this License Agreement shall be deemed to create any relationship
of agency, partnership, or joint venture between the copyright holders and
Licensee. This License Agreement does not grant permission to use
copyright holder's trademarks or trade name in a trademark sense to endorse
or promote products or services of Licensee, or any third party.
8. By copying, installing or otherwise using cx_Freeze, Licensee agrees to be
bound by the terms and conditions of this License Agreement.
Computronix® is a registered trademark of Computronix (Canada) Ltd.
BSD License (for all the BSD licensed code indicated above)
-----------------------------------------------------------
Copyright (c) <YEAR>, <OWNER>
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
* Neither the name of the <ORGANIZATION> nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

6
setup.py
View File

@ -65,10 +65,10 @@ if __name__ == '__main__':
'/Users/kovid/podofo/include/podofo'
podofo_lib = '/usr/lib' if islinux else r'C:\podofo' if iswindows else \
'/Users/kovid/podofo/lib'
if os.path.exists(os.path.join(podofo_inc, 'PdfString.h')):
if os.path.exists(os.path.join(podofo_inc, 'podofo.h')):
eca = ['/EHsc'] if iswindows else []
optional.append(PyQtExtension('calibre.plugins.podofo', [],
['src/calibre/utils/podofo/podofo.sip'],
optional.append(Extension('calibre.plugins.podofo',
sources=['src/calibre/utils/podofo/podofo.cpp'],
libraries=['podofo'], extra_compile_args=eca,
library_dirs=[os.environ.get('PODOFO_LIB_DIR', podofo_lib)],
include_dirs=\

9
src/calibre/__init__.py
View File

@ -2,7 +2,7 @@
__license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'
import sys, os, re, logging, time, subprocess, mimetypes, \
import sys, os, re, logging, time, mimetypes, \
__builtin__, warnings, multiprocessing
__builtin__.__dict__['dynamic_property'] = lambda(func): func(None)
from htmlentitydefs import name2codepoint
@ -91,11 +91,16 @@ def prints(*args, **kwargs):
file = kwargs.get('file', sys.stdout)
sep = kwargs.get('sep', ' ')
end = kwargs.get('end', '\n')
enc = preferred_encoding
if 'CALIBRE_WORKER' in os.environ:
enc = 'utf-8'
for i, arg in enumerate(args):
if isinstance(arg, unicode):
arg = arg.encode(preferred_encoding)
arg = arg.encode(enc)
if not isinstance(arg, str):
arg = str(arg)
if not isinstance(arg, unicode):
arg = arg.decode(preferred_encoding, 'replace').encode(enc)
file.write(arg)
if i != len(args)-1:
file.write(sep)

2
src/calibre/devices/interface.py
View File

@ -231,7 +231,7 @@ class DevicePlugin(Plugin):
def settings(cls):
'''
Should return an opts object. The opts object should have one attribute
`formats` whihc is an ordered list of formats for the device.
`format_map` which is an ordered list of formats for the device.
'''
raise NotImplementedError()

2
src/calibre/devices/kindle/driver.py
View File

@ -53,7 +53,7 @@ class KINDLE(USBMS):
@classmethod
def metadata_from_path(cls, path):
from calibre.devices.usbms.driver import metadata_from_formats
from calibre.ebooks.metadata.meta import metadata_from_formats
mi = metadata_from_formats([path])
if mi.title == _('Unknown') or ('-asin' in mi.title and '-type' in mi.title):
match = cls.WIRELESS_FILE_NAME_PATTERN.match(os.path.basename(path))

2
src/calibre/devices/prs500/books.py
View File

@ -116,7 +116,7 @@ class Book(object):
self.authors.encode('utf-8') + " at " + self.path.encode('utf-8')
def fix_ids(media, cache):
def fix_ids(media, cache, *args):
'''
Adjust ids in cache to correspond with media.
'''

5
src/calibre/devices/prs500/driver.py
View File

@ -47,6 +47,7 @@ from calibre.devices.prs500.prstypes import *
from calibre.devices.errors import *
from calibre.devices.prs500.books import BookList, fix_ids
from calibre import __author__, __appname__
from calibre.devices.usbms.deviceconfig import DeviceConfig
# Protocol versions this driver has been tested with
KNOWN_USB_PROTOCOL_VERSIONS = [0x3030303030303130L]
@ -76,7 +77,7 @@ class File(object):
return self.name
class PRS500(DevicePlugin):
class PRS500(DeviceConfig, DevicePlugin):
"""
Implements the backend for communication with the SONY Reader.
@ -624,6 +625,8 @@ class PRS500(DevicePlugin):
data_type=FreeSpaceAnswer, \
command_number=FreeSpaceQuery.NUMBER)[0]
data.append( pkt.free )
data = [x for x in data if x != 0]
data.append(0)
return data
def _exists(self, path):

2
src/calibre/devices/prs505/books.py
View File

@ -147,7 +147,7 @@ class BookList(_BookList):
nodes = self.root_element.childNodes
for i, book in enumerate(nodes):
if report_progress:
self.report_progress((i+1) / float(len(nodes)), _('Getting list of books on device...'))
report_progress((i+1) / float(len(nodes)), _('Getting list of books on device...'))
if hasattr(book, 'tagName') and book.tagName.endswith('text'):
tags = [i.getAttribute('title') for i in self.get_playlists(book.getAttribute('id'))]
self.append(Book(book, mountpath, tags, prefix=self.prefix))

3
src/calibre/devices/scanner.py
View File

@ -61,7 +61,8 @@ class DeviceScanner(object):
vendor_ids = device.VENDOR_ID if hasattr(device.VENDOR_ID, '__len__') else [device.VENDOR_ID]
product_ids = device.PRODUCT_ID if hasattr(device.PRODUCT_ID, '__len__') else [device.PRODUCT_ID]
if iswindows:
for vendor_id, product_id in zip(vendor_ids, product_ids):
for vendor_id in vendor_ids:
for product_id in product_ids:
vid, pid = 'vid_%4.4x'%vendor_id, 'pid_%4.4x'%product_id
vidd, pidd = 'vid_%i'%vendor_id, 'pid_%i'%product_id
for device_id in self.devices:

3
src/calibre/devices/usbms/deviceconfig.py
View File

@ -12,7 +12,8 @@ class DeviceConfig(object):
@classmethod
def _config(cls):
c = Config('device_drivers_%s' % cls.__class__.__name__, _('settings for device drivers'))
klass = cls if isinstance(cls, type) else cls.__class__
c = Config('device_drivers_%s' % klass.__name__, _('settings for device drivers'))
c.add_opt('format_map', default=cls.FORMATS, help=cls.HELP_MESSAGE)
return c

57
src/calibre/ebooks/comic/input.py
View File

@ -7,12 +7,14 @@ __docformat__ = 'restructuredtext en'
Based on ideas from comiclrf created by FangornUK.
'''
import os, shutil, traceback, textwrap
import os, shutil, traceback, textwrap, time
from Queue import Empty
from calibre.customize.conversion import InputFormatPlugin, OptionRecommendation
from calibre import extract, CurrentDir
from calibre import extract, CurrentDir, prints
from calibre.ptempfile import PersistentTemporaryDirectory
from calibre.parallel import Server, ParallelJob
from calibre.utils.ipc.server import Server
from calibre.utils.ipc.job import ParallelJob
def extract_comic(path_to_comic_file):
'''
@ -47,8 +49,8 @@ def find_pages(dir, sort_on_mtime=False, verbose=False):
pages.sort(cmp=comparator)
if verbose:
print 'Found comic pages...'
print '\t'+'\n\t'.join([os.path.basename(p) for p in pages])
prints('Found comic pages...')
prints('\t'+'\n\t'.join([os.path.basename(p) for p in pages]))
return pages
class PageProcessor(list):
@ -181,7 +183,7 @@ class PageProcessor(list):
p.DestroyPixelWand(pw)
p.DestroyMagickWand(wand)
def render_pages(tasks, dest, opts, notification=None):
def render_pages(tasks, dest, opts, notification=lambda x, y: x):
'''
Entry point for the job server.
'''
@ -197,30 +199,23 @@ def render_pages(tasks, dest, opts, notification=None):
msg = _('Failed %s')%path
if opts.verbose:
msg += '\n' + traceback.format_exc()
if notification is not None:
prints(msg)
notification(0.5, msg)
return pages, failures
class JobManager(object):
'''
Simple job manager responsible for keeping track of overall progress.
'''
class Progress(object):
def __init__(self, total, update):
self.total = total
self.update = update
self.done = 0
self.add_job = lambda j: j
self.output = lambda j: j
self.start_work = lambda j: j
self.job_done = lambda j: j
def status_update(self, job):
def __call__(self, percent, msg=''):
self.done += 1
#msg = msg%os.path.basename(job.args[0])
self.update(float(self.done)/self.total, job.msg)
self.update(float(self.done)/self.total, msg)
def process_pages(pages, opts, update, tdir):
'''
@ -229,22 +224,38 @@ def process_pages(pages, opts, update, tdir):
from calibre.utils.PythonMagickWand import ImageMagick
ImageMagick
job_manager = JobManager(len(pages), update)
progress = Progress(len(pages), update)
server = Server()
jobs = []
tasks = [(p, os.path.join(tdir, os.path.basename(p))) for p in pages]
tasks = server.split(pages)
for task in tasks:
jobs.append(ParallelJob('render_pages', lambda s:s, job_manager=job_manager,
jobs.append(ParallelJob('render_pages', '', progress,
args=[task, tdir, opts]))
server.add_job(jobs[-1])
server.wait()
server.killall()
while True:
time.sleep(1)
running = False
for job in jobs:
while True:
try:
x = job.notifications.get_nowait()
progress(*x)
except Empty:
break
job.update()
if not job.is_finished:
running = True
if not running:
break
server.close()
ans, failures = [], []
for job in jobs:
if job.result is None:
raise Exception(_('Failed to process comic: %s\n\n%s')%(job.exception, job.traceback))
if job.failed:
raw_input()
raise Exception(_('Failed to process comic: \n\n%s')%
job.log_file.read())
pages, failures_ = job.result
ans += pages
failures += failures_

96
src/calibre/ebooks/conversion/config.py Normal file
View File

@ -0,0 +1,96 @@
#!/usr/bin/env python
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
from __future__ import with_statement
__license__ = 'GPL v3'
__copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'
import os
from calibre.utils.config import config_dir
from calibre.utils.lock import ExclusiveFile
from calibre import sanitize_file_name
from calibre.customize.conversion import OptionRecommendation
config_dir = os.path.join(config_dir, 'conversion')
if not os.path.exists(config_dir):
os.makedirs(config_dir)
def name_to_path(name):
return os.path.join(config_dir, sanitize_file_name(name)+'.py')
def save_defaults(name, recs):
path = name_to_path(name)
raw = str(recs)
with open(path, 'wb'):
pass
with ExclusiveFile(path) as f:
f.write(raw)
def load_defaults(name):
path = name_to_path(name)
if not os.path.exists(path):
open(path, 'wb').close()
with ExclusiveFile(path) as f:
raw = f.read()
r = GuiRecommendations()
if raw:
r.from_string(raw)
return r
def save_specifics(db, book_id, recs):
raw = str(recs)
db.set_conversion_options(book_id, 'PIPE', raw)
def load_specifics(db, book_id):
raw = db.conversion_options(book_id, 'PIPE')
r = GuiRecommendations()
if raw:
r.from_string(raw)
return r
class GuiRecommendations(dict):
def __new__(cls, *args):
dict.__new__(cls)
obj = super(GuiRecommendations, cls).__new__(cls, *args)
obj.disabled_options = set([])
return obj
def to_recommendations(self, level=OptionRecommendation.LOW):
ans = []
for key, val in self.items():
ans.append((key, val, level))
return ans
def __str__(self):
ans = ['{']
for key, val in self.items():
ans.append('\t'+repr(key)+' : '+repr(val)+',')
ans.append('}')
return '\n'.join(ans)
def from_string(self, raw):
try:
d = eval(raw)
except SyntaxError:
d = None
if d:
self.update(d)
def merge_recommendations(self, get_option, level, options,
only_existing=False):
for name in options:
if only_existing and name not in self:
continue
opt = get_option(name)
if opt is None: continue
if opt.level == OptionRecommendation.HIGH:
self[name] = opt.recommended_value
self.disabled_options.add(name)
elif opt.level > level or name not in self:
self[name] = opt.recommended_value

1
src/calibre/ebooks/conversion/plumber.py
View File

@ -668,6 +668,7 @@ OptionRecommendation(name='list_recipes',
self.output_plugin.convert(self.oeb, self.output, self.input_plugin,
self.opts, self.log)
self.ui_reporter(1.)
self.log(self.output_fmt.upper(), 'output written to', self.output)
def create_oebbook(log, path_or_stream, opts, input_plugin, reader=None):
'''

14
src/calibre/ebooks/lit/output.py
View File

@ -7,7 +7,8 @@ __copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'
from calibre.customize.conversion import OutputFormatPlugin
from calibre.customize.conversion import OutputFormatPlugin, \
OptionRecommendation
class LITOutput(OutputFormatPlugin):
@ -15,12 +16,23 @@ class LITOutput(OutputFormatPlugin):
author = 'Marshall T. Vandegrift'
file_type = 'lit'
recommendations = set([
('dont_split_on_page_breaks', False, OptionRecommendation.HIGH),
])
def convert(self, oeb, output_path, input_plugin, opts, log):
self.log, self.opts, self.oeb = log, opts, oeb
from calibre.ebooks.oeb.transforms.manglecase import CaseMangler
from calibre.ebooks.oeb.transforms.rasterize import SVGRasterizer
from calibre.ebooks.oeb.transforms.htmltoc import HTMLTOCAdder
from calibre.ebooks.lit.writer import LitWriter
from calibre.ebooks.oeb.transforms.split import Split
split = Split(not self.opts.dont_split_on_page_breaks,
max_flow_size=0
)
split(self.oeb, self.opts)
tocadder = HTMLTOCAdder()
tocadder(oeb, opts)
mangler = CaseMangler()

7
src/calibre/ebooks/metadata/__init__.py
View File

@ -241,7 +241,12 @@ class MetaInformation(object):
self.tags += mi.tags
self.tags = list(set(self.tags))
if getattr(mi, 'cover_data', None) and mi.cover_data[0] is not None:
if getattr(mi, 'cover_data', False):
other_cover = mi.cover_data[-1]
self_cover = self.cover_data[-1] if self.cover_data else ''
if not self_cover: self_cover = ''
if not other_cover: other_cover = ''
if len(other_cover) > len(self_cover):
self.cover_data = mi.cover_data
my_comments = getattr(self, 'comments', '')

8
src/calibre/ebooks/metadata/meta.py
View File

@ -28,6 +28,14 @@ def path_to_ext(path):
return os.path.splitext(path)[1][1:].lower()
def metadata_from_formats(formats):
try:
return _metadata_from_formats(formats)
except:
mi = metadata_from_filename(formats[0])
if not mi.authors:
mi.authors = [_('Unknown')]
def _metadata_from_formats(formats):
mi = MetaInformation(None, None)
formats.sort(cmp=lambda x,y: cmp(METADATA_PRIORITIES[path_to_ext(x)],
METADATA_PRIORITIES[path_to_ext(y)]))

12
src/calibre/ebooks/metadata/mobi.py
View File

@ -9,12 +9,9 @@ __copyright__ = '2009, Kovid Goyal kovid@kovidgoyal.net and ' \
'Marshall T. Vandegrift <llasram@gmail.com>'
__docformat__ = 'restructuredtext en'
import sys
import os
from struct import pack, unpack
from cStringIO import StringIO
from calibre.ebooks.mobi import MobiError
from calibre.ebooks.mobi.reader import get_metadata
from calibre.ebooks.mobi.writer import rescale_image, MAX_THUMB_DIMEN
from calibre.ebooks.mobi.langcodes import iana2mobi
@ -116,8 +113,13 @@ class MetadataUpdater(object):
def update(self, mi):
recs = []
from calibre.ebooks.mobi.from_any import config
if mi.author_sort and config().parse().prefer_author_sort:
try:
from calibre.ebooks.conversion.config import load_defaults
prefs = load_defaults('mobi_output')
pas = prefs.get('prefer_author_sort', False)
except:
pas = False
if mi.author_sort and pas:
authors = mi.author_sort
recs.append((100, authors.encode(self.codec, 'replace')))
elif mi.authors:

2
src/calibre/ebooks/metadata/opf2.py
View File

@ -387,6 +387,8 @@ class MetadataField(object):
ans = self.formatter(ans)
except:
return None
if hasattr(ans, 'strip'):
ans = ans.strip()
return ans
def __get__(self, obj, type=None):

6
src/calibre/ebooks/metadata/pdf.py
View File

@ -18,13 +18,13 @@ except:
from calibre.ebooks.metadata import MetaInformation, authors_to_string
from calibre.utils.pdftk import set_metadata as pdftk_set_metadata
from calibre.utils.podofo import get_metadata as podofo_get_metadata, \
set_metadata as podofo_set_metadata
set_metadata as podofo_set_metadata, Unavailable
def get_metadata(stream, extract_cover=True):
try:
mi = podofo_get_metadata(stream)
except:
except Unavailable:
mi = get_metadata_pypdf(stream)
stream.seek(0)
@ -43,7 +43,7 @@ def set_metadata(stream, mi):
stream.seek(0)
try:
return podofo_set_metadata(stream, mi)
except:
except Unavailable:
pass
try:
return pdftk_set_metadata(stream, mi)

122
src/calibre/ebooks/metadata/worker.py Normal file
View File

@ -0,0 +1,122 @@
#!/usr/bin/env python
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
from __future__ import with_statement
__license__ = 'GPL v3'
__copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'
from threading import Thread
from Queue import Empty
import os, time, sys
from calibre.utils.ipc.job import ParallelJob
from calibre.utils.ipc.server import Server
from calibre.ptempfile import PersistentTemporaryDirectory
from calibre import prints
def debug(*args):
prints(*args)
sys.stdout.flush()
def read_metadata_(task, tdir, notification=lambda x,y:x):
from calibre.ebooks.metadata.meta import metadata_from_formats
from calibre.ebooks.metadata.opf2 import OPFCreator
for x in task:
id, formats = x
if isinstance(formats, basestring): formats = [formats]
mi = metadata_from_formats(formats)
mi.cover = None
cdata = None
if mi.cover_data:
cdata = mi.cover_data[-1]
mi.cover_data = None
opf = OPFCreator(tdir, mi)
with open(os.path.join(tdir, '%s.opf'%id), 'wb') as f:
opf.render(f)
if cdata:
with open(os.path.join(tdir, str(id)), 'wb') as f:
f.write(cdata)
notification(0.5, id)
class Progress(object):
def __init__(self, result_queue, tdir):
self.result_queue = result_queue
self.tdir = tdir
def __call__(self, id):
cover = os.path.join(self.tdir, str(id))
if not os.path.exists(cover): cover = None
self.result_queue.put((id, os.path.join(self.tdir, '%s.opf'%id), cover))
class ReadMetadata(Thread):
def __init__(self, tasks, result_queue):
self.tasks, self.result_queue = tasks, result_queue
self.canceled = False
Thread.__init__(self)
self.daemon = True
self.tdir = PersistentTemporaryDirectory('_rm_worker')
def run(self):
jobs, ids = set([]), set([])
for t in self.tasks:
for b in t:
ids.add(b[0])
progress = Progress(self.result_queue, self.tdir)
server = Server()
for i, task in enumerate(self.tasks):
job = ParallelJob('read_metadata',
'Read metadata (%d of %d)'%(i, len(self.tasks)),
lambda x,y:x, args=[task, self.tdir])
jobs.add(job)
server.add_job(job)
while not self.canceled:
time.sleep(0.2)
running = False
for job in jobs:
while True:
try:
id = job.notifications.get_nowait()[-1]
if id in ids:
progress(id)
ids.remove(id)
except Empty:
break
job.update()
if not job.is_finished:
running = True
if not running:
break
server.close()
time.sleep(1)
if self.canceled:
return
for id in ids:
print 11111111, id
progress(id)
for job in jobs:
if job.failed:
prints(job.details)
if os.path.exists(job.log_path):
os.remove(job.log_path)
def read_metadata(paths, result_queue, chunk=50):
tasks = []
pos = 0
while pos < len(paths):
tasks.append(paths[pos:pos+chunk])
pos += chunk
t = ReadMetadata(tasks, result_queue)
t.start()
return t

13
src/calibre/ebooks/oeb/base.py
View File

@ -96,8 +96,12 @@ def iterlinks(root):
for el in root.iter():
attribs = el.attrib
try:
tag = el.tag
except UnicodeDecodeError:
continue
if el.tag == XHTML('object'):
if tag == XHTML('object'):
codebase = None
## <object> tags have attributes that are relative to
## codebase
@ -122,7 +126,7 @@ def iterlinks(root):
yield (el, attr, attribs[attr], 0)
if el.tag == XHTML('style') and el.text:
if tag == XHTML('style') and el.text:
for match in _css_url_re.finditer(el.text):
yield (el, None, match.group(1), match.start(1))
for match in _css_import_re.finditer(el.text):
@ -801,6 +805,11 @@ class Manifest(object):
self.oeb.logger.warn(
'File %r missing <body/> element' % self.href)
etree.SubElement(data, XHTML('body'))
# Remove microsoft office markup
r = [x for x in data.iterdescendants(etree.Element) if 'microsoft-com' in x.tag]
for x in r:
x.tag = XHTML('span')
return data
def _parse_css(self, data):

2
src/calibre/ebooks/oeb/stylizer.py
View File

@ -18,7 +18,7 @@ from xml.dom import SyntaxErr as CSSSyntaxError
import cssutils
from cssutils.css import CSSStyleRule, CSSPageRule, CSSStyleDeclaration, \
CSSValueList, cssproperties
from cssutils.profiles import profiles as cssprofiles
from cssutils import profile as cssprofiles
from lxml import etree
from lxml.cssselect import css_to_xpath, ExpressionError, SelectorSyntaxError
from calibre.ebooks.oeb.base import XHTML, XHTML_NS, CSS_MIME, OEB_STYLES

8
src/calibre/ebooks/oeb/transforms/structure.py
View File

@ -12,7 +12,13 @@ from lxml import etree
from urlparse import urlparse
from calibre.ebooks.oeb.base import XPNSMAP, TOC, XHTML
XPath = lambda x: etree.XPath(x, namespaces=XPNSMAP)
from calibre.ebooks import ConversionError
def XPath(x):
try:
return etree.XPath(x, namespaces=XPNSMAP)
except etree.XPathSyntaxError:
raise ConversionError(
'The syntax of the XPath expression %s is invalid.' % repr(x))
class DetectStructure(object):

10
src/calibre/ebooks/pdf/pdftohtml.py
View File

@ -6,7 +6,7 @@ __copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>, ' \
'2009, John Schember <john@nachtimwald.com>'
__docformat__ = 'restructuredtext en'
import errno, os, re, sys, subprocess
import errno, os, sys, subprocess
from functools import partial
from calibre.ebooks import ConversionError, DRMError
@ -33,7 +33,7 @@ def pdftohtml(pdf_path):
if isinstance(pdf_path, unicode):
pdf_path = pdf_path.encode(sys.getfilesystemencoding())
if not os.access(pdf_path, os.R_OK):
raise ConversionError, 'Cannot read from ' + pdf_path
raise ConversionError('Cannot read from ' + pdf_path)
with TemporaryDirectory('_pdftohtml') as tdir:
index = os.path.join(tdir, 'index.html')
@ -47,7 +47,7 @@ def pdftohtml(pdf_path):
p = popen(cmd, stderr=subprocess.PIPE)
except OSError, err:
if err.errno == 2:
raise ConversionError(_('Could not find pdftohtml, check it is in your PATH'), True)
raise ConversionError(_('Could not find pdftohtml, check it is in your PATH'))
else:
raise
@ -63,13 +63,13 @@ def pdftohtml(pdf_path):
if ret != 0:
err = p.stderr.read()
raise ConversionError, err
raise ConversionError(err)
if not os.path.exists(index) or os.stat(index).st_size < 100:
raise DRMError()
with open(index, 'rb') as i:
raw = i.read()
if not '<br' in raw[:4000]:
raise ConversionError(os.path.basename(pdf_path) + _(' is an image based PDF. Only conversion of text based PDFs is supported.'), True)
raise ConversionError(os.path.basename(pdf_path) + _(' is an image based PDF. Only conversion of text based PDFs is supported.'))
return '<!-- created by calibre\'s pdftohtml -->\n' + raw

2
src/calibre/gui2/__init__.py
View File

@ -122,7 +122,7 @@ def question_dialog(parent, title, msg, det_msg=''):
parent)
d.setDetailedText(det_msg)
d.setIconPixmap(QPixmap(':/images/dialog_information.svg'))
return d
return d.exec_() == QMessageBox.Yes
def info_dialog(parent, title, msg, det_msg='', show=False):
d = QMessageBox(QMessageBox.Information, title, msg, QMessageBox.NoButton,

343
src/calibre/gui2/add.py
View File

@ -2,238 +2,163 @@
UI for adding books to the database
'''
import os
from Queue import Queue, Empty
from PyQt4.Qt import QThread, SIGNAL, QMutex, QWaitCondition, Qt
from PyQt4.Qt import QThread, SIGNAL, QObject, QTimer, Qt
from calibre.gui2.dialogs.progress import ProgressDialog
from calibre.gui2 import question_dialog
from calibre.ebooks.metadata.opf2 import OPF
from calibre.ebooks.metadata import MetaInformation
from calibre.constants import preferred_encoding
from calibre.gui2 import warning_dialog
class Add(QThread):
class RecursiveFind(QThread):
def __init__(self):
QThread.__init__(self)
self._lock = QMutex()
self._waiting = QWaitCondition()
def is_canceled(self):
if self.pd.canceled:
self.canceled = True
return self.canceled
def wait_for_condition(self):
self._lock.lock()
self._waiting.wait(self._lock)
self._lock.unlock()
def wake_up(self):
self._waiting.wakeAll()
class AddFiles(Add):
def __init__(self, paths, default_thumbnail, get_metadata, db=None):
Add.__init__(self)
self.paths = paths
self.get_metadata = get_metadata
self.default_thumbnail = default_thumbnail
def __init__(self, parent, db, root, single):
QThread.__init__(self, parent)
self.db = db
self.formats, self.metadata, self.names, self.infos = [], [], [], []
self.duplicates = []
self.number_of_books_added = 0
self.connect(self.get_metadata,
SIGNAL('metadata(PyQt_PyObject, PyQt_PyObject)'),
self.metadata_delivered)
def metadata_delivered(self, id, mi):
if self.is_canceled():
self.wake_up()
return
if not mi.title:
mi.title = os.path.splitext(self.names[id])[0]
mi.title = mi.title if isinstance(mi.title, unicode) else \
mi.title.decode(preferred_encoding, 'replace')
self.metadata.append(mi)
self.infos.append({'title':mi.title,
'authors':', '.join(mi.authors),
'cover':self.default_thumbnail, 'tags':[]})
if self.db is not None:
duplicates, num = self.db.add_books(self.paths[id:id+1],
self.formats[id:id+1], [mi],
add_duplicates=False)
self.number_of_books_added += num
if duplicates:
if not self.duplicates:
self.duplicates = [[], [], [], []]
for i in range(4):
self.duplicates[i] += duplicates[i]
self.emit(SIGNAL('processed(PyQt_PyObject,PyQt_PyObject)'),
mi.title, id)
self.wake_up()
def create_progress_dialog(self, title, msg, parent):
self._parent = parent
self.pd = ProgressDialog(title, msg, -1, len(self.paths)-1, parent)
self.connect(self, SIGNAL('processed(PyQt_PyObject,PyQt_PyObject)'),
self.update_progress_dialog)
self.pd.setModal(True)
self.pd.show()
self.connect(self, SIGNAL('finished()'), self.pd.hide)
return self.pd
def update_progress_dialog(self, title, count):
self.pd.set_value(count)
if self.db is not None:
self.pd.set_msg(_('Added %s to library')%title)
else:
self.pd.set_msg(_('Read metadata from ')+title)
self.path = root
self.single_book_per_directory = single
self.canceled = False
def run(self):
try:
self.canceled = False
for c, book in enumerate(self.paths):
if self.pd.canceled:
self.canceled = True
break
format = os.path.splitext(book)[1]
format = format[1:] if format else None
stream = open(book, 'rb')
self.formats.append(format)
self.names.append(os.path.basename(book))
self.get_metadata(c, stream, stream_type=format,
use_libprs_metadata=True)
self.wait_for_condition()
finally:
self.disconnect(self.get_metadata,
SIGNAL('metadata(PyQt_PyObject, PyQt_PyObject)'),
self.metadata_delivered)
self.get_metadata = None
def process_duplicates(self):
if self.duplicates:
files = ''
for mi in self.duplicates[2]:
files += mi.title+'\n'
d = warning_dialog(_('Duplicates found!'),
_('Books with the same title as the following already '
'exist in the database. Add them anyway?'),
files, parent=self._parent)
if d.exec_() == d.Accepted:
num = self.db.add_books(*self.duplicates,
**dict(add_duplicates=True))[1]
self.number_of_books_added += num
class AddRecursive(Add):
def __init__(self, path, db, get_metadata, single_book_per_directory, parent):
self.path = path
self.db = db
self.get_metadata = get_metadata
self.single_book_per_directory = single_book_per_directory
self.duplicates, self.books, self.metadata = [], [], []
self.number_of_books_added = 0
self.canceled = False
Add.__init__(self)
self.connect(self.get_metadata,
SIGNAL('metadataf(PyQt_PyObject, PyQt_PyObject)'),
self.metadata_delivered, Qt.QueuedConnection)
self.connect(self, SIGNAL('searching_done()'), self.searching_done,
Qt.QueuedConnection)
self._parent = parent
self.pd = ProgressDialog(_('Adding books recursively...'),
_('Searching for books in all sub-directories...'),
0, 0, parent)
self.connect(self, SIGNAL('processed(PyQt_PyObject,PyQt_PyObject)'),
self.update_progress_dialog)
self.connect(self, SIGNAL('update(PyQt_PyObject)'), self.pd.set_msg,
Qt.QueuedConnection)
self.connect(self, SIGNAL('pupdate(PyQt_PyObject)'), self.pd.set_value,
Qt.QueuedConnection)
self.pd.setModal(True)
self.pd.show()
self.connect(self, SIGNAL('finished()'), self.pd.hide)
def update_progress_dialog(self, title, count):
self.pd.set_value(count)
if title:
self.pd.set_msg(_('Read metadata from ')+title)
def metadata_delivered(self, id, mi):
if self.is_canceled():
self.wake_up()
return
self.emit(SIGNAL('processed(PyQt_PyObject,PyQt_PyObject)'),
mi.title, id)
self.metadata.append((mi if mi.title else None, self.books[id]))
if len(self.metadata) >= len(self.books):
self.metadata = [x for x in self.metadata if x[0] is not None]
self.pd.set_min(-1)
self.pd.set_max(len(self.metadata)-1)
self.pd.set_value(-1)
self.pd.set_msg(_('Adding books to database...'))
self.wake_up()
def searching_done(self):
self.pd.set_min(-1)
self.pd.set_max(len(self.books)-1)
self.pd.set_value(-1)
self.pd.set_msg(_('Reading metadata...'))
def run(self):
try:
root = os.path.abspath(self.path)
self.books = []
for dirpath in os.walk(root):
if self.is_canceled():
if self.canceled:
return
self.emit(SIGNAL('update(PyQt_PyObject)'),
_('Searching in')+' '+dirpath[0])
self.books += list(self.db.find_books_in_directory(dirpath[0],
self.single_book_per_directory))
self.books = [formats for formats in self.books if formats]
# Reset progress bar
self.emit(SIGNAL('searching_done()'))
for c, formats in enumerate(self.books):
self.get_metadata.from_formats(c, formats)
self.wait_for_condition()
if not self.canceled:
self.emit(SIGNAL('found(PyQt_PyObject)'), self.books)
# Add books to database
for c, x in enumerate(self.metadata):
mi, formats = x
if self.is_canceled():
break
if self.db.has_book(mi):
self.duplicates.append((mi, formats))
else:
self.db.import_book(mi, formats, notify=False)
class Adder(QObject):
def __init__(self, parent, db, callback):
QObject.__init__(self, parent)
self.pd = ProgressDialog(_('Add books'), parent=parent)
self.db = db
self.pd.setModal(True)
self.pd.show()
self._parent = parent
self.number_of_books_added = 0
self.rfind = self.worker = self.timer = None
self.callback = callback
self.callback_called = False
self.infos, self.paths, self.names = [], [], []
self.connect(self.pd, SIGNAL('canceled()'), self.canceled)
def add_recursive(self, root, single=True):
self.path = root
self.pd.set_msg(_('Searching for books in all sub-directories...'))
self.pd.set_min(0)
self.pd.set_max(0)
self.pd.value = 0
self.rfind = RecursiveFind(self, self.db, root, single)
self.connect(self.rfind, SIGNAL('update(PyQt_PyObject)'),
self.pd.set_msg, Qt.QueuedConnection)
self.connect(self.rfind, SIGNAL('found(PyQt_PyObject)'),
self.add, Qt.QueuedConnection)
self.rfind.start()
def add(self, books):
books = [[b] if isinstance(b, basestring) else b for b in books]
self.rfind = None
from calibre.ebooks.metadata.worker import read_metadata
self.rq = Queue()
tasks = []
self.ids = {}
self.nmap = {}
self.duplicates = []
for i, b in enumerate(books):
tasks.append((i, b))
self.ids[i] = b
self.nmap[i] = os.path.basename(b[0])
self.worker = read_metadata(tasks, self.rq)
self.pd.set_min(0)
self.pd.set_max(len(self.ids))
self.pd.value = 0
self.timer = QTimer(self)
self.connect(self.timer, SIGNAL('timeout()'), self.update)
self.timer.start(200)
def add_formats(self, id, formats):
for path in formats:
fmt = os.path.splitext(path)[-1].replace('.', '').upper()
self.db.add_format(id, fmt, open(path, 'rb'), index_is_id=True,
notify=False)
def canceled(self):
if self.rfind is not None:
self.rfind.cenceled = True
if self.timer is not None:
self.timer.stop()
if self.worker is not None:
self.worker.canceled = True
self.pd.hide()
if not self.callback_called:
self.callback(self.paths, self.names, self.infos)
self.callback_called = True
def update(self):
if not self.ids:
self.timer.stop()
self.process_duplicates()
self.pd.hide()
if not self.callback_called:
self.callback(self.paths, self.names, self.infos)
self.callback_called = True
return
try:
id, opf, cover = self.rq.get_nowait()
except Empty:
return
self.pd.value += 1
formats = self.ids.pop(id)
mi = MetaInformation(OPF(opf))
name = self.nmap.pop(id)
if not mi.title:
mi.title = os.path.splitext(name)[0]
mi.title = mi.title if isinstance(mi.title, unicode) else \
mi.title.decode(preferred_encoding, 'replace')
self.pd.set_msg(_('Added')+' '+mi.title)
if self.db is not None:
if cover:
cover = open(cover, 'rb').read()
id = self.db.create_book_entry(mi, cover=cover, add_duplicates=False)
self.number_of_books_added += 1
self.emit(SIGNAL('pupdate(PyQt_PyObject)'), c)
finally:
self.disconnect(self.get_metadata,
SIGNAL('metadataf(PyQt_PyObject, PyQt_PyObject)'),
self.metadata_delivered)
self.get_metadata = None
if id is None:
self.duplicates.append((mi, cover, formats))
else:
self.add_formats(id, formats)
else:
self.names.append(name)
self.paths.append(formats[0])
self.infos.append({'title':mi.title,
'authors':', '.join(mi.authors),
'cover':None,
'tags':mi.tags if mi.tags else []})
def process_duplicates(self):
if self.duplicates:
files = ''
for mi in self.duplicates:
title = mi[0].title
if not isinstance(title, unicode):
title = title.decode(preferred_encoding, 'replace')
files += title+'\n'
d = warning_dialog(_('Duplicates found!'),
if not self.duplicates:
return
files = [x[0].title for x in self.duplicates]
if question_dialog(self._parent, _('Duplicates found!'),
_('Books with the same title as the following already '
'exist in the database. Add them anyway?'),
files, parent=self._parent)
if d.exec_() == d.Accepted:
for mi, formats in self.duplicates:
self.db.import_book(mi, formats, notify=False)
'\n'.join(files)):
for mi, cover, formats in self.duplicates:
id = self.db.create_book_entry(mi, cover=cover,
add_duplicates=True)
self.add_formats(id, formats)
self.number_of_books_added += 1

91
src/calibre/gui2/convert/__init__.py
View File

@ -6,96 +6,13 @@ __license__ = 'GPL v3'
__copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'
import os
from PyQt4.Qt import QWidget, QSpinBox, QDoubleSpinBox, QLineEdit, QTextEdit, \
QCheckBox, QComboBox, Qt, QIcon, SIGNAL
from calibre.customize.conversion import OptionRecommendation
from calibre.utils.config import config_dir
from calibre.utils.lock import ExclusiveFile
from calibre import sanitize_file_name
config_dir = os.path.join(config_dir, 'conversion')
if not os.path.exists(config_dir):
os.makedirs(config_dir)
def name_to_path(name):
return os.path.join(config_dir, sanitize_file_name(name)+'.py')
def save_defaults(name, recs):
path = name_to_path(name)
raw = str(recs)
with open(path, 'wb'):
pass
with ExclusiveFile(path) as f:
f.write(raw)
save_defaults_ = save_defaults
def load_defaults(name):
path = name_to_path(name)
if not os.path.exists(path):
open(path, 'wb').close()
with ExclusiveFile(path) as f:
raw = f.read()
r = GuiRecommendations()
if raw:
r.from_string(raw)
return r
def save_specifics(db, book_id, recs):
raw = str(recs)
db.set_conversion_options(book_id, 'PIPE', raw)
def load_specifics(db, book_id):
raw = db.conversion_options(book_id, 'PIPE')
r = GuiRecommendations()
if raw:
r.from_string(raw)
return r
class GuiRecommendations(dict):
def __new__(cls, *args):
dict.__new__(cls)
obj = super(GuiRecommendations, cls).__new__(cls, *args)
obj.disabled_options = set([])
return obj
def to_recommendations(self, level=OptionRecommendation.LOW):
ans = []
for key, val in self.items():
ans.append((key, val, level))
return ans
def __str__(self):
ans = ['{']
for key, val in self.items():
ans.append('\t'+repr(key)+' : '+repr(val)+',')
ans.append('}')
return '\n'.join(ans)
def from_string(self, raw):
try:
d = eval(raw)
except SyntaxError:
d = None
if d:
self.update(d)
def merge_recommendations(self, get_option, level, options,
only_existing=False):
for name in options:
if only_existing and name not in self:
continue
opt = get_option(name)
if opt is None: continue
if opt.level == OptionRecommendation.HIGH:
self[name] = opt.recommended_value
self.disabled_options.add(name)
elif opt.level > level or name not in self:
self[name] = opt.recommended_value
from calibre.ebooks.conversion.config import load_defaults, \
save_defaults as save_defaults_, \
load_specifics, GuiRecommendations
class Widget(QWidget):
@ -176,7 +93,7 @@ class Widget(QWidget):
elif isinstance(g, QCheckBox):
return bool(g.isChecked())
elif isinstance(g, XPathEdit):
return g.xpath
return g.xpath if g.xpath else None
else:
raise Exception('Can\'t get value from %s'%type(g))

4
src/calibre/gui2/convert/single.py
View File

@ -11,7 +11,7 @@ import sys, cPickle
from PyQt4.Qt import QString, SIGNAL, QAbstractListModel, Qt, QVariant, QFont
from calibre.gui2 import ResizableDialog, NONE
from calibre.gui2.convert import GuiRecommendations, save_specifics, \
from calibre.ebooks.conversion.config import GuiRecommendations, save_specifics, \
load_specifics
from calibre.gui2.convert.single_ui import Ui_Dialog
from calibre.gui2.convert.metadata import MetadataWidget
@ -189,6 +189,8 @@ class Config(ResizableDialog, Ui_Dialog):
def setup_input_output_formats(self, db, book_id, preferred_input_format,
preferred_output_format):
if preferred_output_format:
preferred_output_format = preferred_output_format.lower()
available_formats = db.formats(book_id, index_is_id=True)
if not available_formats:
available_formats = ''

64
src/calibre/gui2/device.py
View File

@ -1,7 +1,7 @@
from __future__ import with_statement
__license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
import os, traceback, Queue, time, socket
import os, traceback, Queue, time, socket, cStringIO
from threading import Thread, RLock
from itertools import repeat
from functools import partial
@ -15,7 +15,7 @@ from calibre.customize.ui import available_input_formats, available_output_forma
from calibre.devices.interface import DevicePlugin
from calibre.constants import iswindows
from calibre.gui2.dialogs.choose_format import ChooseFormatDialog
from calibre.parallel import Job
from calibre.utils.ipc.job import BaseJob
from calibre.devices.scanner import DeviceScanner
from calibre.gui2 import config, error_dialog, Dispatcher, dynamic, \
pixmap_to_data, warning_dialog, \
@ -27,22 +27,46 @@ from calibre.devices.errors import FreeSpaceError
from calibre.utils.smtp import compose_mail, sendmail, extract_email_address, \
config as email_config
class DeviceJob(Job):
class DeviceJob(BaseJob):
def __init__(self, func, *args, **kwargs):
Job.__init__(self, *args, **kwargs)
def __init__(self, func, done, job_manager, args=[], kwargs={},
description=''):
BaseJob.__init__(self, description, done=done)
self.func = func
self.args, self.kwargs = args, kwargs
self.exception = None
self.job_manager = job_manager
self._details = _('No details available.')
def start_work(self):
self.start_time = time.time()
self.job_manager.changed_queue.put(self)
def job_done(self):
self.duration = time.time() - self.start_time
self.percent = 1
self.job_manager.changed_queue.put(self)
def report_progress(self, percent, msg=''):
self.notifications.put((percent, msg))
self.job_manager.changed_queue.put(self)
def run(self):
self.start_work()
try:
self.result = self.func(*self.args, **self.kwargs)
except (Exception, SystemExit), err:
self.failed = True
self._details = unicode(err) + '\n\n' + \
traceback.format_exc()
self.exception = err
self.traceback = traceback.format_exc()
finally:
self.job_done()
@property
def log_file(self):
return cStringIO.StringIO(self._details.encode('utf-8'))
class DeviceManager(Thread):
@ -113,7 +137,7 @@ class DeviceManager(Thread):
job = self.next()
if job is not None:
self.current_job = job
self.device.set_progress_reporter(job.update_status)
self.device.set_progress_reporter(job.report_progress)
self.current_job.run()
self.current_job = None
else:
@ -206,7 +230,6 @@ class DeviceManager(Thread):
def _view_book(self, path, target):
f = open(target, 'wb')
print self.device
self.device.get_file(path, f)
f.close()
return target
@ -355,12 +378,12 @@ class DeviceMenu(QMenu):
if action.dest == 'main:':
action.setEnabled(True)
elif action.dest == 'carda:0':
if card_prefix[0] != None:
if card_prefix and card_prefix[0] != None:
action.setEnabled(True)
else:
action.setEnabled(False)
elif action.dest == 'cardb:0':
if card_prefix[1] != None:
if card_prefix and card_prefix[1] != None:
action.setEnabled(True)
else:
action.setEnabled(False)
@ -463,7 +486,8 @@ class DeviceGUI(object):
fmts = [x.strip().lower() for x in fmts.split(',')]
self.send_by_mail(to, fmts, delete)
def send_by_mail(self, to, fmts, delete_from_library, send_ids=None, do_auto_convert=True):
def send_by_mail(self, to, fmts, delete_from_library, send_ids=None,
do_auto_convert=True, specific_format=None):
ids = [self.library_view.model().id(r) for r in self.library_view.selectionModel().selectedRows()] if send_ids is None else send_ids
if not ids or len(ids) == 0:
return
@ -537,10 +561,10 @@ class DeviceGUI(object):
bad += auto
else:
autos = [self.library_view.model().db.title(id, index_is_id=True) for id in auto]
autos = '\n'.join('<li>%s</li>'%(i,) for i in autos)
d = info_dialog(self, _('No suitable formats'),
_('Auto converting the following books before uploading to the device:<br><ul>%s</ul>')%(autos,))
d.exec_()
autos = '\n'.join('%s'%i for i in autos)
info_dialog(self, _('No suitable formats'),
_('Auto converting the following books before uploading to '
'the device:'), det_msg=autos, show=True)
self.auto_convert_mail(to, delete_from_library, auto, format)
if bad:
@ -724,10 +748,10 @@ class DeviceGUI(object):
bad += auto
else:
autos = [self.library_view.model().db.title(id, index_is_id=True) for id in auto]
autos = '\n'.join('<li>%s</li>'%(i,) for i in autos)
d = info_dialog(self, _('No suitable formats'),
_('Auto converting the following books before uploading to the device:<br><ul>%s</ul>')%(autos,))
d.exec_()
autos = '\n'.join('%s'%i for i in autos)
info_dialog(self, _('No suitable formats'),
_('Auto converting the following books before uploading to '
'the device:'), det_msg=autos, show=True)
self.auto_convert(auto, on_card, format)
if bad:
@ -750,7 +774,7 @@ class DeviceGUI(object):
'''
Called once metadata has been uploaded.
'''
if job.exception is not None:
if job.failed:
self.device_job_exception(job)
return
cp, fs = job.result

66
src/calibre/gui2/dialogs/jobs.py
View File

@ -1,66 +0,0 @@
__license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
'''Display active jobs'''
from PyQt4.QtCore import Qt, QObject, SIGNAL, QSize, QString, QTimer
from PyQt4.QtGui import QDialog, QAbstractItemDelegate, QStyleOptionProgressBarV2, \
QApplication, QStyle
from calibre.gui2.dialogs.jobs_ui import Ui_JobsDialog
from calibre import __appname__
class ProgressBarDelegate(QAbstractItemDelegate):
def sizeHint(self, option, index):
return QSize(120, 30)
def paint(self, painter, option, index):
opts = QStyleOptionProgressBarV2()
opts.rect = option.rect
opts.minimum = 1
opts.maximum = 100
opts.textVisible = True
percent, ok = index.model().data(index, Qt.DisplayRole).toInt()
if not ok:
percent = 0
opts.progress = percent
opts.text = QString(_('Unavailable') if percent == 0 else '%d%%'%percent)
QApplication.style().drawControl(QStyle.CE_ProgressBar, opts, painter)
class JobsDialog(QDialog, Ui_JobsDialog):
def __init__(self, window, model):
QDialog.__init__(self, window)
Ui_JobsDialog.__init__(self)
self.setupUi(self)
self.jobs_view.setModel(model)
self.model = model
self.setWindowModality(Qt.NonModal)
self.setWindowTitle(__appname__ + _(' - Jobs'))
QObject.connect(self.jobs_view.model(), SIGNAL('modelReset()'),
self.jobs_view.resizeColumnsToContents)
QObject.connect(self.kill_button, SIGNAL('clicked()'),
self.kill_job)
QObject.connect(self, SIGNAL('kill_job(int, PyQt_PyObject)'),
self.jobs_view.model().kill_job)
self.pb_delegate = ProgressBarDelegate(self)
self.jobs_view.setItemDelegateForColumn(2, self.pb_delegate)
self.running_time_timer = QTimer(self)
self.connect(self.running_time_timer, SIGNAL('timeout()'), self.update_running_time)
self.running_time_timer.start(1000)
def update_running_time(self, *args):
try:
self.model.running_time_updated()
except: # Raises random exceptions on OS X
pass
def kill_job(self):
for index in self.jobs_view.selectedIndexes():
row = index.row()
self.model.kill_job(row, self)
return
def closeEvent(self, e):
self.jobs_view.write_settings()
e.accept()

43
src/calibre/gui2/dialogs/jobs.ui
View File

@ -1,7 +1,8 @@
<ui version="4.0" >
<?xml version="1.0" encoding="UTF-8"?>
<ui version="4.0">
<class>JobsDialog</class>
<widget class="QDialog" name="JobsDialog" >
<property name="geometry" >
<widget class="QDialog" name="JobsDialog">
<property name="geometry">
<rect>
<x>0</x>
<y>0</y>
@ -9,31 +10,32 @@
<height>542</height>
</rect>
</property>
<property name="windowTitle" >
<property name="windowTitle">
<string>Active Jobs</string>
</property>
<property name="windowIcon" >
<iconset resource="../images.qrc" >:/images/jobs.svg</iconset>
<property name="windowIcon">
<iconset resource="../images.qrc">
<normaloff>:/images/jobs.svg</normaloff>:/images/jobs.svg</iconset>
</property>
<layout class="QVBoxLayout" >
<layout class="QVBoxLayout">
<item>
<widget class="JobsView" name="jobs_view" >
<property name="contextMenuPolicy" >
<widget class="JobsView" name="jobs_view">
<property name="contextMenuPolicy">
<enum>Qt::NoContextMenu</enum>
</property>
<property name="editTriggers" >
<property name="editTriggers">
<set>QAbstractItemView::NoEditTriggers</set>
</property>
<property name="alternatingRowColors" >
<property name="alternatingRowColors">
<bool>true</bool>
</property>
<property name="selectionMode" >
<property name="selectionMode">
<enum>QAbstractItemView::SingleSelection</enum>
</property>
<property name="selectionBehavior" >
<property name="selectionBehavior">
<enum>QAbstractItemView::SelectRows</enum>
</property>
<property name="iconSize" >
<property name="iconSize">
<size>
<width>32</width>
<height>32</height>
@ -42,12 +44,19 @@
</widget>
</item>
<item>
<widget class="QPushButton" name="kill_button" >
<property name="text" >
<widget class="QPushButton" name="kill_button">
<property name="text">
<string>&amp;Stop selected job</string>
</property>
</widget>
</item>
<item>
<widget class="QPushButton" name="details_button">
<property name="text">
<string>Show job &amp;details</string>
</property>
</widget>
</item>
</layout>
</widget>
<customwidgets>
@ -58,7 +67,7 @@
</customwidget>
</customwidgets>
<resources>
<include location="../images.qrc" />
<include location="../images.qrc"/>
</resources>
<connections/>
</ui>

11
src/calibre/gui2/dialogs/progress.py
View File

@ -31,6 +31,16 @@ class ProgressDialog(QDialog, Ui_Dialog):
def set_value(self, val):
self.bar.setValue(val)
@dynamic_property
def value(self):
def fset(self, val):
return self.bar.setValue(val)
def fget(self):
return self.bar.value()
return property(fget=fget, fset=fset)
def set_min(self, min):
self.bar.setMinimum(min)
@ -41,6 +51,7 @@ class ProgressDialog(QDialog, Ui_Dialog):
self.canceled = True
self.button_box.setDisabled(True)
self.title.setText(_('Aborting...'))
self.emit(SIGNAL('canceled()'))
def keyPressEvent(self, ev):
if ev.key() == Qt.Key_Escape:

8
src/calibre/gui2/dialogs/tag_editor.py
View File

@ -1,7 +1,7 @@
__license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
from PyQt4.QtCore import SIGNAL, Qt
from PyQt4.QtGui import QDialog, QMessageBox
from PyQt4.QtGui import QDialog
from calibre.gui2.dialogs.tag_editor_ui import Ui_TagEditor
from calibre.gui2 import qstring_to_unicode
@ -59,9 +59,9 @@ class TagEditor(QDialog, Ui_TagEditor):
deletes.append(item)
if confirms:
ct = ', '.join([qstring_to_unicode(item.text()) for item in confirms])
d = question_dialog(self, 'Are your sure?',
'<p>The following tags are used by one or more books. Are you certain you want to delete them?<br>'+ct)
if d.exec_() == QMessageBox.Yes:
if question_dialog(self, _('Are your sure?'),
'<p>'+_('The following tags are used by one or more books. '
'Are you certain you want to delete them?')+'<br>'+ct):
deletes += confirms
for item in deletes:

22
src/calibre/gui2/dialogs/user_profiles.py
View File

@ -3,7 +3,7 @@ __copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
import time, os
from PyQt4.QtCore import SIGNAL, QUrl
from PyQt4.QtGui import QMessageBox, QDesktopServices
from PyQt4.QtGui import QDesktopServices
from calibre.web.feeds.recipes import compile_recipe
from calibre.web.feeds.news import AutomaticNewsRecipe
@ -86,7 +86,7 @@ class UserProfiles(ResizableDialog, Ui_Dialog):
self.source_code.setPlainText('')
else:
self.source_code.setPlainText(src)
#self.highlighter = PythonHighlighter(self.source_code.document())
self.highlighter = PythonHighlighter(self.source_code.document())
self.stacks.setCurrentIndex(1)
self.toggle_mode_button.setText(_('Switch to Basic mode'))
@ -175,9 +175,9 @@ class %(classname)s(%(base_class)s):
try:
self.available_profiles.add_item(title, (title, profile), replace=False)
except ValueError:
d = question_dialog(self, _('Replace recipe?'),
_('A custom recipe named %s already exists. Do you want to replace it?')%title)
if d.exec_() == QMessageBox.Yes:
if question_dialog(self, _('Replace recipe?'),
_('A custom recipe named %s already exists. Do you want to '
'replace it?')%title):
self.available_profiles.add_item(title, (title, profile), replace=True)
else:
return
@ -207,9 +207,9 @@ class %(classname)s(%(base_class)s):
try:
self.available_profiles.add_item(title, (title, r.text), replace=False)
except ValueError:
d = question_dialog(self, _('Replace recipe?'),
_('A custom recipe named %s already exists. Do you want to replace it?')%title)
if d.exec_() == QMessageBox.Yes:
if question_dialog(self, _('Replace recipe?'),
_('A custom recipe named %s already exists. Do you '
'want to replace it?')%title):
self.available_profiles.add_item(title, (title, r.text), replace=True)
else:
return
@ -231,9 +231,9 @@ class %(classname)s(%(base_class)s):
try:
self.available_profiles.add_item(title, (title, src), replace=False)
except ValueError:
d = question_dialog(self, _('Replace recipe?'),
_('A custom recipe named %s already exists. Do you want to replace it?')%title)
if d.exec_() == QMessageBox.Yes:
if question_dialog(self, _('Replace recipe?'),
_('A custom recipe named %s already exists. Do you want to '
'replace it?')%title):
self.available_profiles.add_item(title, (title, src), replace=True)
else:
return

BIN
src/calibre/gui2/images/news/hrt.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 606 B

BIN
src/calibre/gui2/images/news/rts.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 458 B

253
src/calibre/gui2/jobs.py Normal file
View File

@ -0,0 +1,253 @@
#!/usr/bin/env python
__license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
__docformat__ = 'restructuredtext en'
'''
Job management.
'''
from Queue import Empty, Queue
from PyQt4.Qt import QAbstractTableModel, QVariant, QModelIndex, Qt, \
QTimer, SIGNAL, QIcon, QDialog, QAbstractItemDelegate, QApplication, \
QSize, QStyleOptionProgressBarV2, QString, QStyle
from calibre.utils.ipc.server import Server
from calibre.utils.ipc.job import ParallelJob
from calibre.gui2 import Dispatcher, error_dialog, NONE
from calibre.gui2.device import DeviceJob
from calibre.gui2.dialogs.jobs_ui import Ui_JobsDialog
from calibre import __appname__
class JobManager(QAbstractTableModel):
def __init__(self):
QAbstractTableModel.__init__(self)
self.wait_icon = QVariant(QIcon(':/images/jobs.svg'))
self.running_icon = QVariant(QIcon(':/images/exec.svg'))
self.error_icon = QVariant(QIcon(':/images/dialog_error.svg'))
self.done_icon = QVariant(QIcon(':/images/ok.svg'))
self.jobs = []
self.add_job = Dispatcher(self._add_job)
self.server = Server()
self.changed_queue = Queue()
self.timer = QTimer(self)
self.connect(self.timer, SIGNAL('timeout()'), self.update,
Qt.QueuedConnection)
self.timer.start(1000)
def columnCount(self, parent=QModelIndex()):
return 4
def rowCount(self, parent=QModelIndex()):
return len(self.jobs)
def headerData(self, section, orientation, role):
if role != Qt.DisplayRole:
return NONE
if orientation == Qt.Horizontal:
if section == 0: text = _('Job')
elif section == 1: text = _('Status')
elif section == 2: text = _('Progress')
elif section == 3: text = _('Running time')
return QVariant(text)
else:
return QVariant(section+1)
def data(self, index, role):
try:
if role not in (Qt.DisplayRole, Qt.DecorationRole):
return NONE
row, col = index.row(), index.column()
job = self.jobs[row]
if role == Qt.DisplayRole:
if col == 0:
desc = job.description
if not desc:
desc = _('Unknown job')
return QVariant(desc)
if col == 1:
return QVariant(job.status_text)
if col == 2:
return QVariant(job.percent)
if col == 3:
rtime = job.running_time
if rtime is None:
return NONE
return QVariant('%dm %ds'%(int(rtime)//60, int(rtime)%60))
if role == Qt.DecorationRole and col == 0:
state = job.run_state
if state == job.WAITING:
return self.wait_icon
if state == job.RUNNING:
return self.running_icon
if job.killed or job.failed:
return self.error_icon
return self.done_icon
except:
import traceback
traceback.print_exc()
return NONE
def update(self):
try:
self._update()
except BaseException:
import traceback
traceback.print_exc()
def _update(self):
# Update running time
rows = set([])
for i, j in enumerate(self.jobs):
if j.run_state == j.RUNNING:
idx = self.index(i, 3)
self.emit(SIGNAL('dataChanged(QModelIndex,QModelIndex)'),
idx, idx)
# Update parallel jobs
jobs = set([])
while True:
try:
jobs.add(self.server.changed_jobs_queue.get_nowait())
except Empty:
break
while True:
try:
jobs.add(self.changed_queue.get_nowait())
except Empty:
break
if jobs:
needs_reset = False
for job in jobs:
orig_state = job.run_state
job.update()
if orig_state != job.run_state:
needs_reset = True
if needs_reset:
self.jobs.sort()
self.reset()
if job.is_finished:
self.emit(SIGNAL('job_done(int)'), len(self.unfinished_jobs()))
else:
for job in jobs:
idx = self.jobs.index(job)
self.emit(SIGNAL('dataChanged(QModelIndex,QModelIndex)'),
self.index(idx, 0), self.index(idx, 3))
def _add_job(self, job):
self.emit(SIGNAL('layoutAboutToBeChanged()'))
self.jobs.append(job)
self.jobs.sort()
self.emit(SIGNAL('job_added(int)'), len(self.unfinished_jobs()))
self.emit(SIGNAL('layoutChanged()'))
def done_jobs(self):
return [j for j in self.jobs if j.is_finished]
def unfinished_jobs(self):
return [j for j in self.jobs if not j.is_finished]
def row_to_job(self, row):
return self.jobs[row]
def has_device_jobs(self):
for job in self.jobs:
if job.is_running and isinstance(job, DeviceJob):
return True
return False
def has_jobs(self):
for job in self.jobs:
if job.is_running:
return True
return False
def run_job(self, done, name, args=[], kwargs={},
description=''):
job = ParallelJob(name, description, done, args=args, kwargs=kwargs)
self.add_job(job)
self.server.add_job(job)
return job
def launch_gui_app(self, name, args=[], kwargs={}, description=''):
job = ParallelJob(name, description, lambda x: x,
args=args, kwargs=kwargs)
self.server.run_job(job, gui=True, redirect_output=False)
def kill_job(self, row, view):
job = self.jobs[row]
if isinstance(job, DeviceJob):
return error_dialog(view, _('Cannot kill job'),
_('Cannot kill jobs that communicate with the device')).exec_()
if job.duration is not None:
return error_dialog(view, _('Cannot kill job'),
_('Job has already run')).exec_()
self.server.kill_job(job)
def terminate_all_jobs(self):
self.server.killall()
class ProgressBarDelegate(QAbstractItemDelegate):
def sizeHint(self, option, index):
return QSize(120, 30)
def paint(self, painter, option, index):
opts = QStyleOptionProgressBarV2()
opts.rect = option.rect
opts.minimum = 1
opts.maximum = 100
opts.textVisible = True
percent, ok = index.model().data(index, Qt.DisplayRole).toInt()
if not ok:
percent = 0
opts.progress = percent
opts.text = QString(_('Unavailable') if percent == 0 else '%d%%'%percent)
QApplication.style().drawControl(QStyle.CE_ProgressBar, opts, painter)
class JobsDialog(QDialog, Ui_JobsDialog):
def __init__(self, window, model):
QDialog.__init__(self, window)
Ui_JobsDialog.__init__(self)
self.setupUi(self)
self.jobs_view.setModel(model)
self.model = model
self.setWindowModality(Qt.NonModal)
self.setWindowTitle(__appname__ + _(' - Jobs'))
self.connect(self.jobs_view.model(), SIGNAL('modelReset()'),
self.jobs_view.resizeColumnsToContents)
self.connect(self.kill_button, SIGNAL('clicked()'),
self.kill_job)
self.connect(self.details_button, SIGNAL('clicked()'),
self.show_details)
self.connect(self, SIGNAL('kill_job(int, PyQt_PyObject)'),
self.jobs_view.model().kill_job)
self.pb_delegate = ProgressBarDelegate(self)
self.jobs_view.setItemDelegateForColumn(2, self.pb_delegate)
def kill_job(self):
for index in self.jobs_view.selectedIndexes():
row = index.row()
self.model.kill_job(row, self)
return
def show_details(self):
for index in self.jobs_view.selectedIndexes():
self.jobs_view.show_details(index)
return
def closeEvent(self, e):
self.jobs_view.write_settings()
e.accept()

203
src/calibre/gui2/jobs2.py
View File

@ -1,203 +0,0 @@
#!/usr/bin/env python
__license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
__docformat__ = 'restructuredtext en'
'''
Job management.
'''
import time
from PyQt4.QtCore import QAbstractTableModel, QVariant, QModelIndex, Qt, SIGNAL
from PyQt4.QtGui import QIcon, QDialog
from calibre.parallel import ParallelJob, Server
from calibre.gui2 import Dispatcher, error_dialog
from calibre.gui2.device import DeviceJob
from calibre.gui2.dialogs.job_view_ui import Ui_Dialog
NONE = QVariant()
class JobManager(QAbstractTableModel):
def __init__(self):
QAbstractTableModel.__init__(self)
self.wait_icon = QVariant(QIcon(':/images/jobs.svg'))
self.running_icon = QVariant(QIcon(':/images/exec.svg'))
self.error_icon = QVariant(QIcon(':/images/dialog_error.svg'))
self.done_icon = QVariant(QIcon(':/images/ok.svg'))
self.jobs = []
self.server = Server()
self.add_job = Dispatcher(self._add_job)
self.status_update = Dispatcher(self._status_update)
self.start_work = Dispatcher(self._start_work)
self.job_done = Dispatcher(self._job_done)
def columnCount(self, parent=QModelIndex()):
return 4
def rowCount(self, parent=QModelIndex()):
return len(self.jobs)
def headerData(self, section, orientation, role):
if role != Qt.DisplayRole:
return NONE
if orientation == Qt.Horizontal:
if section == 0: text = _("Job")
elif section == 1: text = _("Status")
elif section == 2: text = _("Progress")
elif section == 3: text = _('Running time')
return QVariant(text)
else:
return QVariant(section+1)
def data(self, index, role):
try:
if role not in (Qt.DisplayRole, Qt.DecorationRole):
return NONE
row, col = index.row(), index.column()
job = self.jobs[row]
if role == Qt.DisplayRole:
if col == 0:
desc = job.description
if not desc:
desc = _('Unknown job')
return QVariant(desc)
if col == 1:
status = job.status()
if status == 'DONE':
st = _('Finished')
elif status == 'ERROR':
st = _('Error')
elif status == 'WAITING':
st = _('Waiting')
else:
st = _('Working')
return QVariant(st)
if col == 2:
pc = job.percent
if pc <=0:
percent = 0
else:
percent = int(100*pc)
return QVariant(percent)
if col == 3:
if job.start_time is None:
return NONE
rtime = job.running_time if job.running_time is not None else \
time.time() - job.start_time
return QVariant('%dm %ds'%(int(rtime)//60, int(rtime)%60))
if role == Qt.DecorationRole and col == 0:
status = job.status()
if status == 'WAITING':
return self.wait_icon
if status == 'WORKING':
return self.running_icon
if status == 'ERROR':
return self.error_icon
if status == 'DONE':
return self.done_icon
except:
import traceback
traceback.print_exc()
return NONE
def _add_job(self, job):
self.emit(SIGNAL('layoutAboutToBeChanged()'))
self.jobs.append(job)
self.jobs.sort()
self.emit(SIGNAL('job_added(int)'), self.rowCount())
self.emit(SIGNAL('layoutChanged()'))
def done_jobs(self):
return [j for j in self.jobs if j.status() in ['DONE', 'ERROR']]
def row_to_job(self, row):
return self.jobs[row]
def _start_work(self, job):
self.emit(SIGNAL('layoutAboutToBeChanged()'))
self.jobs.sort()
self.emit(SIGNAL('layoutChanged()'))
def _job_done(self, job):
self.emit(SIGNAL('layoutAboutToBeChanged()'))
self.jobs.sort()
self.emit(SIGNAL('job_done(int)'), len(self.jobs) - len(self.done_jobs()))
self.emit(SIGNAL('layoutChanged()'))
def _status_update(self, job):
try:
row = self.jobs.index(job)
except ValueError: # Job has been stopped
return
self.emit(SIGNAL('dataChanged(QModelIndex, QModelIndex)'),
self.index(row, 0), self.index(row, 3))
def running_time_updated(self, *args):
for job in self.jobs:
if not job.is_running:
continue
row = self.jobs.index(job)
self.emit(SIGNAL('dataChanged(QModelIndex, QModelIndex)'),
self.index(row, 3), self.index(row, 3))
def has_device_jobs(self):
for job in self.jobs:
if job.is_running and isinstance(job, DeviceJob):
return True
return False
def has_jobs(self):
for job in self.jobs:
if job.is_running:
return True
return False
def run_job(self, done, func, args=[], kwargs={},
description=None):
job = ParallelJob(func, done, self, args=args, kwargs=kwargs,
description=description)
self.server.add_job(job)
return job
def output(self, job):
self.emit(SIGNAL('output_received()'))
def kill_job(self, row, view):
job = self.jobs[row]
if isinstance(job, DeviceJob):
error_dialog(view, _('Cannot kill job'),
_('Cannot kill jobs that communicate with the device')).exec_()
return
if job.has_run:
error_dialog(view, _('Cannot kill job'),
_('Job has already run')).exec_()
return
if not job.is_running:
self.jobs.remove(job)
self.reset()
return
self.server.kill(job)
def terminate_all_jobs(self):
pass
class DetailView(QDialog, Ui_Dialog):
def __init__(self, parent, job):
QDialog.__init__(self, parent)
self.setupUi(self)
self.setWindowTitle(job.description)
self.job = job
self.update()
def update(self):
self.log.setPlainText(self.job.console_text())
vbar = self.log.verticalScrollBar()
vbar.setValue(vbar.maximum())

7
src/calibre/gui2/library.py
View File

@ -727,9 +727,6 @@ class BooksView(TableView):
def set_editable(self, editable):
self._model.set_editable(editable)
def set_editable(self, editable):
self._model.set_editable(editable)
def connect_to_search_box(self, sb):
QObject.connect(sb, SIGNAL('search(PyQt_PyObject, PyQt_PyObject)'),
self._model.search)
@ -1024,10 +1021,6 @@ class DeviceBooksModel(BooksModel):
self.editable = editable
def set_editable(self, editable):
self.editable = editable
class SearchBox(QLineEdit):
INTERVAL = 1000 #: Time to wait before emitting search signal

172
src/calibre/gui2/main.py
View File

@ -13,7 +13,7 @@ from PyQt4.Qt import Qt, SIGNAL, QObject, QCoreApplication, QUrl, QTimer, \
from PyQt4.QtSvg import QSvgRenderer
from calibre import __version__, __appname__, islinux, sanitize_file_name, \
iswindows, isosx
iswindows, isosx, prints
from calibre.ptempfile import PersistentTemporaryFile
from calibre.utils.config import prefs, dynamic
from calibre.gui2 import APP_UID, warning_dialog, choose_files, error_dialog, \
@ -32,10 +32,9 @@ from calibre.gui2.main_window import MainWindow, option_parser as _option_parser
from calibre.gui2.main_ui import Ui_MainWindow
from calibre.gui2.device import DeviceManager, DeviceMenu, DeviceGUI, Emailer
from calibre.gui2.status import StatusBar
from calibre.gui2.jobs2 import JobManager
from calibre.gui2.jobs import JobManager, JobsDialog
from calibre.gui2.dialogs.metadata_single import MetadataSingleDialog
from calibre.gui2.dialogs.metadata_bulk import MetadataBulkDialog
from calibre.gui2.dialogs.jobs import JobsDialog
from calibre.gui2.tools import convert_single_ebook, convert_bulk_ebook, \
fetch_scheduled_recipe
from calibre.gui2.dialogs.config import ConfigDialog
@ -44,7 +43,6 @@ from calibre.gui2.dialogs.choose_format import ChooseFormatDialog
from calibre.gui2.dialogs.book_info import BookInfo
from calibre.ebooks import BOOK_EXTENSIONS
from calibre.library.database2 import LibraryDatabase2, CoverCache
from calibre.parallel import JobKilled
from calibre.gui2.dialogs.confirm_delete import confirm
class SaveMenu(QMenu):
@ -202,17 +200,22 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
self.delete_books)
QObject.connect(self.action_edit, SIGNAL("triggered(bool)"),
self.edit_metadata)
self.__em1__ = partial(self.edit_metadata, bulk=False)
QObject.connect(md.actions()[0], SIGNAL('triggered(bool)'),
partial(self.edit_metadata, bulk=False))
self.__em1__)
self.__em2__ = partial(self.edit_metadata, bulk=True)
QObject.connect(md.actions()[2], SIGNAL('triggered(bool)'),
partial(self.edit_metadata, bulk=True))
self.__em2__)
self.__em3__ = partial(self.download_metadata, covers=True)
QObject.connect(md.actions()[4], SIGNAL('triggered(bool)'),
partial(self.download_metadata, covers=True))
self.__em3__)
self.__em4__ = partial(self.download_metadata, covers=False)
QObject.connect(md.actions()[5], SIGNAL('triggered(bool)'),
partial(self.download_metadata, covers=False))
self.__em4__)
self.__em5__ = partial(self.download_metadata, covers=True,
set_metadata=False)
QObject.connect(md.actions()[6], SIGNAL('triggered(bool)'),
partial(self.download_metadata, covers=True,
set_metadata=False))
self.__em5__)
@ -626,9 +629,8 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
'''
Called once device information has been read.
'''
if job.exception is not None:
self.device_job_exception(job)
return
if job.failed:
return self.device_job_exception(job)
info, cp, fs = job.result
self.location_view.model().update_devices(cp, fs)
self.device_info = _('Connected ')+info[0]
@ -641,7 +643,7 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
'''
Called once metadata has been read for all books on the device.
'''
if job.exception is not None:
if job.failed:
if isinstance(job.exception, ExpatError):
error_dialog(self, _('Device database corrupted'),
_('''
@ -679,21 +681,11 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
'Select root folder')
if not root:
return
from calibre.gui2.add import AddRecursive
self._add_recursive_thread = AddRecursive(root,
self.library_view.model().db, self.get_metadata,
single, self)
self.connect(self._add_recursive_thread, SIGNAL('finished()'),
self._recursive_files_added)
self._add_recursive_thread.start()
def _recursive_files_added(self):
self._add_recursive_thread.process_duplicates()
if self._add_recursive_thread.number_of_books_added > 0:
self.library_view.model().resort(reset=False)
self.library_view.model().research()
self.library_view.model().count_changed()
self._add_recursive_thread = None
from calibre.gui2.add import Adder
self._adder = Adder(self,
self.library_view.model().db,
Dispatcher(self._files_added))
self._adder.add_recursive(root, single)
def add_recursive_single(self, checked):
'''
@ -734,10 +726,10 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
(_('LRF Books'), ['lrf']),
(_('HTML Books'), ['htm', 'html', 'xhtm', 'xhtml']),
(_('LIT Books'), ['lit']),
(_('MOBI Books'), ['mobi', 'prc']),
(_('MOBI Books'), ['mobi', 'prc', 'azw']),
(_('Text books'), ['txt', 'rtf']),
(_('PDF Books'), ['pdf']),
(_('Comics'), ['cbz', 'cbr']),
(_('Comics'), ['cbz', 'cbr', 'cbc']),
(_('Archives'), ['zip', 'rar']),
])
if not books:
@ -748,40 +740,30 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
def _add_books(self, paths, to_device, on_card=None):
if on_card is None:
on_card = self.stack.currentIndex() == 2
on_card = self.stack.currentIndex() >= 2
if not paths:
return
from calibre.gui2.add import AddFiles
self._add_files_thread = AddFiles(paths, self.default_thumbnail,
self.get_metadata,
None if to_device else \
self.library_view.model().db
)
self._add_files_thread.send_to_device = to_device
self._add_files_thread.on_card = on_card
self._add_files_thread.create_progress_dialog(_('Adding books...'),
_('Reading metadata...'), self)
self.connect(self._add_files_thread, SIGNAL('finished()'),
self._files_added)
self._add_files_thread.start()
from calibre.gui2.add import Adder
self.__adder_func = partial(self._files_added, on_card=on_card)
self._adder = Adder(self,
None if to_device else self.library_view.model().db,
Dispatcher(self.__adder_func))
self._adder.add(paths)
def _files_added(self):
t = self._add_files_thread
self._add_files_thread = None
if not t.canceled:
if t.send_to_device:
self.upload_books(t.paths,
list(map(sanitize_file_name, t.names)),
t.infos, on_card=t.on_card)
def _files_added(self, paths=[], names=[], infos=[], on_card=False):
if paths:
self.upload_books(paths,
list(map(sanitize_file_name, names)),
infos, on_card=on_card)
self.status_bar.showMessage(
_('Uploading books to device.'), 2000)
else:
t.process_duplicates()
if t.number_of_books_added > 0:
self.library_view.model().books_added(t.number_of_books_added)
if self._adder.number_of_books_added > 0:
self.library_view.model().books_added(self._adder.number_of_books_added)
if hasattr(self, 'db_images'):
self.db_images.reset()
self._adder = None
############################################################################
@ -823,8 +805,8 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
Called once deletion is done on the device
'''
for view in (self.memory_view, self.card_a_view, self.card_b_view):
view.model().deletion_done(job, bool(job.exception))
if job.exception is not None:
view.model().deletion_done(job, job.failed)
if job.failed:
self.device_job_exception(job)
return
@ -993,9 +975,8 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
progress.hide()
def books_saved(self, job):
if job.exception is not None:
self.device_job_exception(job)
return
if job.failed:
return self.device_job_exception(job)
############################################################################
@ -1013,9 +994,8 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
def scheduled_recipe_fetched(self, job):
temp_files, fmt, recipe, callback = self.conversion_jobs.pop(job)
pt = temp_files[0]
if job.exception is not None:
self.job_exception(job)
return
if job.failed:
return self.job_exception(job)
id = self.library_view.model().add_news(pt.name, recipe)
self.library_view.model().reset()
sync = dynamic.get('news_to_be_synced', set([]))
@ -1047,7 +1027,7 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
current = self.library_view.currentIndex()
self.library_view.model().current_changed(current, previous)
def auto_convert_mail(to, delete_from_library, book_ids, format):
def auto_convert_mail(self, to, delete_from_library, book_ids, format):
previous = self.library_view.currentIndex()
rows = [x.row() for x in \
self.library_view.selectionModel().selectedRows()]
@ -1057,7 +1037,8 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
if id not in bad:
job = self.job_manager.run_job(Dispatcher(self.book_auto_converted_mail),
func, args=args, description=desc)
self.conversion_jobs[job] = (temp_files, fmt, id, delete_from_library)
self.conversion_jobs[job] = (temp_files, fmt, id,
delete_from_library, to)
if changed:
self.library_view.model().refresh_rows(rows)
@ -1115,9 +1096,8 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
def book_auto_converted(self, job):
temp_files, fmt, book_id, on_card = self.conversion_jobs.pop(job)
try:
if job.exception is not None:
self.job_exception(job)
return
if job.failed:
return self.job_exception(job)
data = open(temp_files[0].name, 'rb')
self.library_view.model().db.add_format(book_id, fmt, data, index_is_id=True)
data.close()
@ -1137,9 +1117,9 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
self.sync_to_device(on_card, False, specific_format=fmt, send_ids=[book_id], do_auto_convert=False)
def book_auto_converted_mail(self, job):
temp_files, fmt, book_id, delete_from_library = self.conversion_jobs.pop(job)
temp_files, fmt, book_id, delete_from_library, to = self.conversion_jobs.pop(job)
try:
if job.exception is not None:
if job.failed:
self.job_exception(job)
return
data = open(temp_files[0].name, 'rb')
@ -1163,7 +1143,7 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
def book_converted(self, job):
temp_files, fmt, book_id = self.conversion_jobs.pop(job)
try:
if job.exception is not None:
if job.failed:
self.job_exception(job)
return
data = open(temp_files[-1].name, 'rb')
@ -1192,7 +1172,7 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
self._view_file(fmt_path)
def book_downloaded_for_viewing(self, job):
if job.exception:
if job.failed:
self.device_job_exception(job)
return
self._view_file(job.result)
@ -1206,12 +1186,11 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
args.append('--raise-window')
if name is not None:
args.append(name)
self.job_manager.server.run_free_job(viewer,
kwdargs=dict(args=args))
self.job_manager.launch_gui_app(viewer,
kwargs=dict(args=args))
else:
QDesktopServices.openUrl(QUrl.fromLocalFile(name))#launch(name)
time.sleep(5) # User feedback
time.sleep(2) # User feedback
finally:
self.unsetCursor()
@ -1436,7 +1415,7 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
'''
try:
if 'Could not read 32 bytes on the control bus.' in \
unicode(job.exception):
unicode(job.details):
error_dialog(self, _('Error talking to device'),
_('There was a temporary error talking to the '
'device. Please unplug and reconnect the device '
@ -1445,16 +1424,16 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
except:
pass
try:
print >>sys.stderr, job.console_text()
prints(job.details, file=sys.stderr)
except:
pass
if not self.device_error_dialog.isVisible():
self.device_error_dialog.set_message(job.gui_text())
self.device_error_dialog.setDetailedText(job.details)
self.device_error_dialog.show()
def job_exception(self, job):
try:
if job.exception[0] == 'DRMError':
if 'calibre.ebooks.DRMError' in job.details:
error_dialog(self, _('Conversion Error'),
_('<p>Could not convert: %s<p>It is a '
'<a href="%s">DRM</a>ed book. You must first remove the '
@ -1464,23 +1443,15 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
return
except:
pass
only_msg = getattr(job.exception, 'only_msg', False)
if job.killed:
return
try:
print job.console_text()
prints(job.details, file=sys.stderr)
except:
pass
if only_msg:
try:
exc = unicode(job.exception)
except:
exc = repr(job.exception)
error_dialog(self, _('Conversion Error'), exc).exec_()
return
if isinstance(job.exception, JobKilled):
return
error_dialog(self, _('Conversion Error'),
_('Failed to process')+': '+unicode(job.description),
det_msg=job.console_text()).exec_()
_('<b>Failed</b>')+': '+unicode(job.description),
det_msg=job.details).exec_()
def initialize_database(self):
@ -1581,8 +1552,8 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
if self.job_manager.has_device_jobs():
msg = '<p>'+__appname__ + \
_(''' is communicating with the device!<br>
'Quitting may cause corruption on the device.<br>
'Are you sure you want to quit?''')+'</p>'
Quitting may cause corruption on the device.<br>
Are you sure you want to quit?''')+'</p>'
d = QMessageBox(QMessageBox.Warning, _('WARNING: Active jobs'), msg,
QMessageBox.Yes|QMessageBox.No, self)
@ -1596,7 +1567,7 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
def shutdown(self, write_settings=True):
if write_settings:
self.write_settings()
self.job_manager.terminate_all_jobs()
self.job_manager.server.close()
self.device_manager.keep_going = False
self.cover_cache.stop()
self.hide()
@ -1647,12 +1618,11 @@ class Main(MainWindow, Ui_MainWindow, DeviceGUI):
self.vanity.update()
if config.get('new_version_notification') and \
dynamic.get('update to version %s'%version, True):
d = question_dialog(self, _('Update available'),
if question_dialog(self, _('Update available'),
_('%s has been updated to version %s. '
'See the <a href="http://calibre.kovidgoyal.net/wiki/'
'Changelog">new features</a>. Visit the download pa'
'ge?')%(__appname__, version))
if d.exec_() == QMessageBox.Yes:
'ge?')%(__appname__, version)):
url = 'http://calibre.kovidgoyal.net/download_'+\
('windows' if iswindows else 'osx' if isosx else 'linux')
QDesktopServices.openUrl(QUrl(url))

5
src/calibre/gui2/main_window.py
View File

@ -8,6 +8,7 @@ from PyQt4.Qt import QMainWindow, QString, Qt, QFont, QCoreApplication, SIGNAL,\
from calibre.gui2.dialogs.conversion_error import ConversionErrorDialog
from calibre.utils.config import OptionParser
from calibre.gui2 import error_dialog
from calibre import prints
def option_parser(usage='''\
Usage: %prog [options]
@ -79,8 +80,8 @@ class MainWindow(QMainWindow):
sio = StringIO.StringIO()
traceback.print_exception(type, value, tb, file=sio)
fe = sio.getvalue()
print >>sys.stderr, fe
msg = unicode(str(value), 'utf8', 'replace')
prints(fe, file=sys.stderr)
msg = '<b>%s</b>:'%type.__name__ + unicode(str(value), 'utf8', 'replace')
error_dialog(self, _('ERROR: Unhandled exception'), msg, det_msg=fe,
show=True)
except:

7
src/calibre/gui2/status.py
View File

@ -219,19 +219,18 @@ class StatusBar(QStatusBar):
jobs = self.movie_button.jobs
src = qstring_to_unicode(jobs.text())
num = self.jobs()
nnum = num + 1
text = src.replace(str(num), str(nnum))
jobs.setText(text)
if self.movie_button.movie.state() == QMovie.Paused:
self.movie_button.movie.setPaused(False)
def job_done(self, running):
def job_done(self, nnum):
jobs = self.movie_button.jobs
src = qstring_to_unicode(jobs.text())
num = self.jobs()
text = src.replace(str(num), str(running))
text = src.replace(str(num), str(nnum))
jobs.setText(text)
if running == 0:
if nnum == 0:
self.no_more_jobs()
def no_more_jobs(self):

51
src/calibre/gui2/widgets.py
View File

@ -4,16 +4,16 @@ __copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
Miscellaneous widgets used in the GUI
'''
import re, os, traceback
from PyQt4.QtGui import QListView, QIcon, QFont, QLabel, QListWidget, \
from PyQt4.Qt import QListView, QIcon, QFont, QLabel, QListWidget, \
QListWidgetItem, QTextCharFormat, QApplication, \
QSyntaxHighlighter, QCursor, QColor, QWidget, QDialog, \
QPixmap, QMovie, QPalette
from PyQt4.QtCore import QAbstractListModel, QVariant, Qt, SIGNAL, \
QSyntaxHighlighter, QCursor, QColor, QWidget, \
QPixmap, QMovie, QPalette, QTimer, QDialog, \
QAbstractListModel, QVariant, Qt, SIGNAL, \
QRegExp, QSettings, QSize, QModelIndex
from calibre.gui2.jobs2 import DetailView
from calibre.gui2 import human_readable, NONE, TableView, \
qstring_to_unicode, error_dialog
from calibre.gui2.dialogs.job_view_ui import Ui_Dialog
from calibre.gui2.filename_pattern_ui import Ui_Form
from calibre import fit_image
from calibre.utils.fontconfig import find_font_families
@ -176,8 +176,8 @@ class LocationModel(QAbstractListModel):
_('Click to see the list of books on storage card B in your reader')
]
def rowCount(self, parent):
return 1 + sum([1 for i in self.free if i >= 0])
def rowCount(self, *args):
return 1 + len([i for i in self.free if i >= 0])
def data(self, index, role):
row = index.row()
@ -249,6 +249,31 @@ class LocationView(QListView):
if 0 <= row and row <= 3:
self.model().location_changed(row)
class DetailView(QDialog, Ui_Dialog):
def __init__(self, parent, job):
QDialog.__init__(self, parent)
self.setupUi(self)
self.setWindowTitle(job.description)
self.job = job
self.next_pos = 0
self.update()
self.timer = QTimer(self)
self.connect(self.timer, SIGNAL('timeout()'), self.update)
self.timer.start(1000)
def update(self):
f = self.job.log_file
f.seek(self.next_pos)
more = f.read()
self.next_pos = f.tell()
if more:
self.log.appendPlainText(more.decode('utf-8', 'replace'))
vbar = self.log.verticalScrollBar()
vbar.setValue(vbar.maximum())
class JobsView(TableView):
def __init__(self, parent):
@ -259,8 +284,8 @@ class JobsView(TableView):
row = index.row()
job = self.model().row_to_job(row)
d = DetailView(self, job)
self.connect(self.model(), SIGNAL('output_received()'), d.update)
d.exec_()
d.timer.stop()
class FontFamilyModel(QAbstractListModel):
@ -539,12 +564,12 @@ class PythonHighlighter(QSyntaxHighlighter):
return
for regex, format in PythonHighlighter.Rules:
i = text.indexOf(regex)
i = regex.indexIn(text)
while i >= 0:
length = regex.matchedLength()
self.setFormat(i, length,
PythonHighlighter.Formats[format])
i = text.indexOf(regex, i + length)
i = regex.indexIn(text, i + length)
# Slow but good quality highlighting for comments. For more
# speed, comment this out and add the following to __init__:
@ -569,12 +594,12 @@ class PythonHighlighter(QSyntaxHighlighter):
self.setCurrentBlockState(NORMAL)
if text.indexOf(self.stringRe) != -1:
if self.stringRe.indexIn(text) != -1:
return
# This is fooled by triple quotes inside single quoted strings
for i, state in ((text.indexOf(self.tripleSingleRe),
for i, state in ((self.tripleSingleRe.indexIn(text),
TRIPLESINGLE),
(text.indexOf(self.tripleDoubleRe),
(self.tripleDoubleRe.indexIn(text),
TRIPLEDOUBLE)):
if self.previousBlockState() == state:
if i == -1:

22
src/calibre/library/database2.py
View File

@ -1183,6 +1183,28 @@ class LibraryDatabase2(LibraryDatabase):
path = path_or_stream
return run_plugins_on_import(path, format)
def create_book_entry(self, mi, cover=None, add_duplicates=True):
if not add_duplicates and self.has_book(mi):
return None
series_index = 1 if mi.series_index is None else mi.series_index
aus = mi.author_sort if mi.author_sort else ', '.join(mi.authors)
title = mi.title
if isinstance(aus, str):
aus = aus.decode(preferred_encoding, 'replace')
if isinstance(title, str):
title = title.decode(preferred_encoding)
obj = self.conn.execute('INSERT INTO books(title, series_index, author_sort) VALUES (?, ?, ?)',
(title, series_index, aus))
id = obj.lastrowid
self.data.books_added([id], self.conn)
self.set_path(id, True)
self.conn.commit()
self.set_metadata(id, mi)
if cover:
self.set_cover(id, cover)
return id
def add_books(self, paths, formats, metadata, uris=[], add_duplicates=True):
'''
Add a book to the database. The result cache is not updated.

2
src/calibre/linux.py
View File

@ -29,7 +29,7 @@ entry_points = {
'calibre-debug = calibre.debug:main',
'calibredb = calibre.library.cli:main',
'calibre-fontconfig = calibre.utils.fontconfig:main',
'calibre-parallel = calibre.parallel:main',
'calibre-parallel = calibre.utils.ipc.worker:main',
'calibre-customize = calibre.customize.ui:main',
'calibre-complete = calibre.utils.complete:main',
'pdfmanipulate = calibre.ebooks.pdf.manipulate.cli:main',

7
src/calibre/manual/custom.py
View File

@ -4,9 +4,10 @@
__license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
import sys, os, inspect, re
from sphinx.builder import StandaloneHTMLBuilder, bold
from sphinx.builder import StandaloneHTMLBuilder
from sphinx.util import rpartition
from sphinx.ext.autodoc import get_module_charset, prepare_docstring
from sphinx.util.console import bold
from sphinx.ext.autodoc import prepare_docstring
from docutils.statemachine import ViewList
from docutils import nodes
@ -181,7 +182,7 @@ def auto_member(dirname, arguments, options, content, lineno,
docstring = '\n'.join(comment_lines)
if module is not None and docstring is not None:
docstring = docstring.decode(get_module_charset(mod))
docstring = docstring.decode('utf-8')
result = ViewList()
result.append('.. attribute:: %s.%s'%(cls.__name__, obj), '<autodoc>')

40
src/calibre/manual/faq.rst
View File

@ -17,39 +17,11 @@ E-book Format Conversion
What formats does |app| support conversion to/from?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|app| supports the conversion of the following formats:
|app| supports the conversion of many input formats to many output formats.
It can convert every input format in the following list, to every output format.
+----------------------------+------------------------------------------------------------------+
| | **Output formats** |
| +------------------+-----------------------+-----------------------+
| | EPUB | LRF | MOBI |
+===================+========+==================+=======================+=======================+
| | MOBI | ✔ | ✔ | ✔ |
| | | | | |
| | LIT | ✔ | ✔ | ✔ |
| | | | | |
| | PRC** | ✔ | ✔ | ✔ |
| | | | | |
| | EPUB | ✔ | ✔ | ✔ |
| | | | | |
| | ODT | ✔ | ✔ | ✔ |
| | | | | |
| | FB2 | ✔ | ✔ | ✔ |
| | | | | |
| | HTML | ✔ | ✔ | ✔ |
| | | | | |
| **Input formats** | CBR | ✔ | ✔ | ✔ |
| | | | | |
| | CBZ | ✔ | ✔ | ✔ |
| | | | | |
| | RTF | ✔ | ✔ | ✔ |
| | | | | |
| | TXT | ✔ | ✔ | ✔ |
| | | | | |
| | PDF | ✔ | ✔ | ✔ |
| | | | | |
| | LRS | | ✔ | |
+-------------------+--------+------------------+-----------------------+-----------------------+
*Input Formats:* CBZ, CBR, CBC, EPUB, FB2, HTML, LIT, MOBI, ODT, PDF, PRC**, RTF, TXT
*Output Formats:* EPUB, FB2, OEB, LIT, LRF, MOBI, PDB, PDF, TXT
** PRC is a generic format, |app| supports PRC files with TextRead and MOBIBook headers
@ -64,7 +36,7 @@ The PDF conversion tries to extract the text and images from the PDF file and co
are also represented as vector diagrams, thus they cannot be extracted.
How do I convert a collection of HTML files in a specific order?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to convert a collection of HTML files in a specific oder, you have to create a table of contents file. That is, another HTML file that contains links to all the other files in the desired order. Such a file looks like::
<html>
@ -105,7 +77,7 @@ Device Integration
What devices does |app| support?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
At the moment |app| has full support for the SONY PRS 500/505/700, Cybook Gen 3, Amazon Kindle 1/2, Netronix EB600, Ectaco Jetbook and the iPhone. In addition, using the :guilabel:`Save to disk` function you can use it with any ebook reader that exports itself as a USB disk.
At the moment |app| has full support for the SONY PRS 500/505/700, Cybook Gen 3, Amazon Kindle 1/2, Netronix EB600, Ectaco Jetbook, BeBook/BeBook Mini and the iPhone. In addition, using the :guilabel:`Save to disk` function you can use it with any ebook reader that exports itself as a USB disk.
I used |app| to transfer some books to my reader, and now the SONY software hangs every time I connect the reader?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

42
src/calibre/manual/news.rst
View File

@ -257,38 +257,26 @@ The final new feature is the :meth:`calibre.web.feeds.news.BasicNewsRecipe.prepr
Tips for developing new recipes
---------------------------------
The best way to develop new recipes is to use the command line interface. Create the recipe using your favorite python editor and save it to a file say :file:`myrecipe.py`. You can download content using this recipe with the command::
The best way to develop new recipes is to use the command line interface. Create the recipe using your favorite python editor and save it to a file say :file:`myrecipe.recipe`. The `.recipe` extension is required. You can download content using this recipe with the command::
feeds2disk --debug --test myrecipe.py
ebook-convert myrecipe.recipe output_dir --test -vv
The :command:`feeds2disk` will download all the webpages and save them to the current directory. The :option:`--debug` makes feeds2disk spit out a lot of information about what it is doing. The :option:`--test` makes it download only a couple of articles from at most two feeds.
The :command:`ebook-convert` will download all the webpages and save them to the directory :file:`output_dir`, creating it if necessary. The :option:`-vv` makes ebook-convert spit out a lot of information about what it is doing. The :option:`--test` makes it download only a couple of articles from at most two feeds.
Once the download is complete, you can look at the downloaded :term:`HTML` by opening the file :file:`index.html` in a browser. Once you're satisfied that the download and preprocessing is happening correctly, you can generate an LRF ebook with the command::
Once the download is complete, you can look at the downloaded :term:`HTML` by opening the file :file:`index.html` in a browser. Once you're satisfied that the download and preprocessing is happening correctly, you can generate ebooks in different formats as shown below::
html2lrf --use-spine --page-break-before "$" index.html
If the generated :term:`LRF` looks good, you can finally, run::
feeds2lrf myrecipe.py
to see the final :term:`LRF` format e-book generated from your recipe. If you're satisfied with your recipe, consider attaching it to `the wiki <http://calibre.kovidgoyal.net/wiki/UserRecipes>`_, so that others can use it as well. If you feel there is enough demand to justify its inclusion into the set of built-in recipes, add a comment to the ticket http://calibre.kovidgoyal.net/ticket/405
ebook-convert myrecipe.recipe myrecipe.epub
ebook-convert myrecipe.recipe myrecipe.mobi
...
If you just want to quickly test a couple of feeds, you can use the :option:`--feeds` option::
feeds2disk --feeds "['http://feeds.newsweek.com/newsweek/TopNews', 'http://feeds.newsweek.com/headlines/politics']"
If you're satisfied with your recipe, consider attaching it to `the wiki <http://calibre.kovidgoyal.net/wiki/UserRecipes>`_, so that others can use it as well. If you feel there is enough demand to justify its inclusion into the set of built-in recipes, add a comment to the ticket http://calibre.kovidgoyal.net/ticket/405
.. seealso::
:ref:`feeds2disk`
The command line interfce for downloading content from the internet
:ref:`feeds2lrf`
The command line interface for downloading content fro the internet and converting it into a :term:`LRF` e-book.
:ref:`html2lrf`
The command line interface for converting :term:`HTML` into a :term:`LRF` e-book.
:ref:`ebook-convert`
The command line interface for all ebook conversion.
Further reading
@ -305,16 +293,6 @@ To learn more about writing advanced recipes using some of the facilities, avail
`Built-in recipes <http://bazaar.launchpad.net/~kovid/calibre/trunk/files/head:/src/calibre/web/feeds/recipes/>`_
The source code for the built-in recipes that come with |app|
Migrating old style profiles to recipes
----------------------------------------
In earlier versions of |app| there was a similar, if less powerful, framework for fetching news based on *Profiles*. If you have a profile that you would like to migrate to a recipe, the basic technique is simple, as they are very similar (on the surface). Common changes you have to make include:
* Replace ``DefaultProfile`` with ``BasicNewsRecipe``
* Remove ``max_recursions``
* If the server you're downloading from doesn't like multiple connects, set ``simultaneous_downloads = 1``.
API documentation
--------------------

2
src/calibre/manual/news_recipe.rst
View File

@ -54,6 +54,8 @@ Customizing e-book download
.. automember:: BasicNewsRecipe.timefmt
.. automember:: BasicNewsRecipe.conversion_options
.. automember:: BasicNewsRecipe.feeds
.. automember:: BasicNewsRecipe.no_stylesheets

980
src/calibre/parallel.py
View File

@ -1,980 +0,0 @@
from __future__ import with_statement
__license__ = 'GPL v3'
__copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
__docformat__ = 'restructuredtext en'
'''
Used to run jobs in parallel in separate processes. Features output streaming,
support for progress notification as well as job killing. The worker processes
are controlled via a simple protocol run over sockets. The control happens
mainly in two class, :class:`Server` and :class:`Overseer`. The worker is
encapsulated in the function :function:`worker`. Every worker process
has the environment variable :envvar:`CALIBRE_WORKER` defined.
The worker control protocol has two modes of operation. In the first mode, the
worker process listens for commands from the controller process. The controller
process can either hand off a job to the worker or tell the worker to die.
Once a job is handed off to the worker, the protocol enters the second mode, where
the controller listens for messages from the worker. The worker can send progress updates
as well as console output (i.e. text that would normally have been written to stdout
or stderr by the job). Once the job completes (or raises an exception) the worker
returns the result (or exception) to the controller and the protocol reverts to the first mode.
In the second mode, the controller can also send the worker STOP messages, in which case
the worker interrupts the job and dies. The sending of progress and console output messages
is buffered and asynchronous to prevent the job from being IO bound.
'''
import sys, os, gc, cPickle, traceback, cStringIO, time, signal, \
subprocess, socket, collections, binascii, re, thread, tempfile, atexit
from select import select
from threading import RLock, Thread, Event
from math import ceil
from calibre.ptempfile import PersistentTemporaryFile
from calibre import iswindows, detect_ncpus, isosx, preferred_encoding
from calibre.utils.config import prefs
DEBUG = False
#: A mapping from job names to functions that perform the jobs
PARALLEL_FUNCS = {
'lrfviewer' :
('calibre.gui2.lrf_renderer.main', 'main', {}, None),
'ebook-viewer' :
('calibre.gui2.viewer.main', 'main', {}, None),
'render_pages' :
('calibre.ebooks.comic.input', 'render_pages', {}, 'notification'),
'ebook-convert' :
('calibre.ebooks.conversion.cli', 'main', {}, None),
'gui_convert' :
('calibre.gui2.convert.gui_conversion', 'gui_convert', {}, 'notification'),
}
isfrozen = hasattr(sys, 'frozen')
isworker = False
win32event = __import__('win32event') if iswindows else None
win32process = __import__('win32process') if iswindows else None
msvcrt = __import__('msvcrt') if iswindows else None
SOCKET_TYPE = socket.AF_UNIX if not iswindows else socket.AF_INET
class WorkerStatus(object):
'''
A platform independent class to control child processes. Provides the
methods:
.. method:: WorkerStatus.is_alive()
Return True is the child process is alive (i.e. it hasn't exited and returned a return code).
.. method:: WorkerStatus.returncode()
Wait for the child process to exit and return its return code (blocks until child returns).
.. method:: WorkerStatus.kill()
Forcibly terminates child process using operating system specific semantics.
'''
def __init__(self, obj):
'''
`obj`: On windows a process handle, on unix a subprocess.Popen object.
'''
self.obj = obj
self.win32process = win32process # Needed if kill is called during shutdown of interpreter
self.os = os
self.signal = signal
ext = 'windows' if iswindows else 'unix'
for func in ('is_alive', 'returncode', 'kill'):
setattr(self, func, getattr(self, func+'_'+ext))
def is_alive_unix(self):
return self.obj.poll() == None
def returncode_unix(self):
return self.obj.wait()
def kill_unix(self):
os.kill(self.obj.pid, self.signal.SIGKILL)
def is_alive_windows(self):
return win32event.WaitForSingleObject(self.obj, 0) != win32event.WAIT_OBJECT_0
def returncode_windows(self):
return win32process.GetExitCodeProcess(self.obj)
def kill_windows(self, returncode=-1):
self.win32process.TerminateProcess(self.obj, returncode)
class WorkerMother(object):
'''
Platform independent object for launching child processes. All processes
have the environment variable :envvar:`CALIBRE_WORKER` set.
..method:: WorkerMother.spawn_free_spirit(arg)
Launch a non monitored process with argument `arg`.
..method:: WorkerMother.spawn_worker(arg)
Launch a monitored and controllable process with argument `arg`.
'''
def __init__(self):
ext = 'windows' if iswindows else 'osx' if isosx else 'linux'
self.os = os # Needed incase cleanup called when interpreter is shutting down
self.env = {}
if iswindows:
self.executable = os.path.join(os.path.dirname(sys.executable),
'calibre-parallel.exe' if isfrozen else 'Scripts\\calibre-parallel.exe')
elif isosx:
self.executable = self.gui_executable = sys.executable
self.prefix = ''
if isfrozen:
fd = os.path.realpath(getattr(sys, 'frameworks_dir'))
contents = os.path.dirname(fd)
self.gui_executable = os.path.join(contents, 'MacOS',
os.path.basename(sys.executable))
contents = os.path.join(contents, 'console.app', 'Contents')
exe = os.path.basename(sys.executable)
if 'python' not in exe:
exe = 'python'
self.executable = os.path.join(contents, 'MacOS', exe)
resources = os.path.join(contents, 'Resources')
fd = os.path.join(contents, 'Frameworks')
sp = os.path.join(resources, 'lib', 'python'+sys.version[:3], 'site-packages.zip')
self.prefix += 'import sys; sys.frameworks_dir = "%s"; sys.frozen = "macosx_app"; '%fd
self.prefix += 'sys.path.insert(0, %s); '%repr(sp)
if fd not in os.environ['PATH']:
self.env['PATH'] = os.environ['PATH']+':'+fd
self.env['PYTHONHOME'] = resources
self.env['MAGICK_HOME'] = os.path.join(fd, 'ImageMagick')
self.env['DYLD_LIBRARY_PATH'] = os.path.join(fd, 'ImageMagick', 'lib')
else:
self.executable = os.path.join(getattr(sys, 'frozen_path'), 'calibre-parallel') \
if isfrozen else 'calibre-parallel'
if isfrozen:
self.env['LD_LIBRARY_PATH'] = getattr(sys, 'frozen_path') + ':' + os.environ.get('LD_LIBRARY_PATH', '')
self.spawn_worker_windows = lambda arg : self.spawn_free_spirit_windows(arg, type='worker')
self.spawn_worker_linux = lambda arg : self.spawn_free_spirit_linux(arg, type='worker')
self.spawn_worker_osx = lambda arg : self.spawn_free_spirit_osx(arg, type='worker')
for func in ('spawn_free_spirit', 'spawn_worker'):
setattr(self, func, getattr(self, func+'_'+ext))
def cleanup_child_windows(self, child, name=None, fd=None):
try:
child.kill()
except:
pass
try:
if fd is not None:
self.os.close(fd)
except:
pass
try:
if name is not None and os.path.exists(name):
self.os.unlink(name)
except:
pass
def cleanup_child_linux(self, child):
try:
child.kill()
except:
pass
def get_env(self):
env = dict(os.environ)
env['CALIBRE_WORKER'] = '1'
env['ORIGWD'] = os.path.abspath(os.getcwd())
if hasattr(self, 'env'):
env.update(self.env)
return env
def spawn_free_spirit_osx(self, arg, type='free_spirit'):
script = ('from calibre.parallel import main; '
'main(args=["calibre-parallel", %s]);')%repr(arg)
exe = self.gui_executable if type == 'free_spirit' else self.executable
cmdline = [exe, '-c', self.prefix+script]
child = WorkerStatus(subprocess.Popen(cmdline, env=self.get_env()))
atexit.register(self.cleanup_child_linux, child)
return child
def spawn_free_spirit_linux(self, arg, type='free_spirit'):
cmdline = [self.executable, arg]
child = WorkerStatus(subprocess.Popen(cmdline,
env=self.get_env(), cwd=getattr(sys, 'frozen_path', None)))
atexit.register(self.cleanup_child_linux, child)
return child
def spawn_free_spirit_windows(self, arg, type='free_spirit'):
priority = {'high':win32process.HIGH_PRIORITY_CLASS, 'normal':win32process.NORMAL_PRIORITY_CLASS,
'low':win32process.IDLE_PRIORITY_CLASS}[prefs['worker_process_priority']]
fd, name = tempfile.mkstemp('.log', 'calibre_'+type+'_')
handle = msvcrt.get_osfhandle(fd)
si = win32process.STARTUPINFO()
si.hStdOutput = handle
si.hStdError = handle
cmdline = self.executable + ' ' + str(arg)
hProcess = \
win32process.CreateProcess(
None, # Application Name
cmdline, # Command line
None, # processAttributes
None, # threadAttributes
1, # bInheritHandles
win32process.CREATE_NO_WINDOW|priority, # Dont want ugly console popping up
self.get_env(), # New environment
None, # Current directory
si
)[0]
child = WorkerStatus(hProcess)
atexit.register(self.cleanup_child_windows, child, name, fd)
return child
mother = WorkerMother()
_comm_lock = RLock()
def write(socket, msg, timeout=5):
'''
Write a message on socket. If `msg` is unicode, it is encoded in utf-8.
Raises a `RuntimeError` if the socket is not ready for writing or the writing fails.
`msg` is broken into chunks of size 4096 and sent. The :function:`read` function
automatically re-assembles the chunks into whole message.
'''
if isworker:
_comm_lock.acquire()
try:
if isinstance(msg, unicode):
msg = msg.encode('utf-8')
if DEBUG:
print >>sys.__stdout__, 'write(%s):'%('worker' if isworker else 'overseer'), repr(msg)
length = None
while len(msg) > 0:
if length is None:
length = len(msg)
chunk = ('%-12d'%length) + msg[:4096-12]
msg = msg[4096-12:]
else:
chunk, msg = msg[:4096], msg[4096:]
w = select([], [socket], [], timeout)[1]
if not w:
raise RuntimeError('Write to socket timed out')
if socket.sendall(chunk) is not None:
raise RuntimeError('Failed to write chunk to socket')
finally:
if isworker:
_comm_lock.release()
def read(socket, timeout=5):
'''
Read a message from `socket`. The message must have been sent with the :function:`write`
function. Raises a `RuntimeError` if the message is corrupted. Can return an
empty string.
'''
if isworker:
_comm_lock.acquire()
try:
buf = cStringIO.StringIO()
length = None
while select([socket],[],[],timeout)[0]:
msg = socket.recv(4096)
if not msg:
break
if length is None:
try:
length, msg = int(msg[:12]), msg[12:]
except ValueError:
if DEBUG:
print >>sys.__stdout__, 'read(%s):'%('worker' if isworker else 'overseer'), 'no length in', msg
return ''
buf.write(msg)
if buf.tell() >= length:
break
if not length:
if DEBUG:
print >>sys.__stdout__, 'read(%s):'%('worker' if isworker else 'overseer'), 'nothing'
return ''
msg = buf.getvalue()[:length]
if len(msg) < length:
raise RuntimeError('Corrupted packet received')
if DEBUG:
print >>sys.__stdout__, 'read(%s):'%('worker' if isworker else 'overseer'), repr(msg)
return msg
finally:
if isworker:
_comm_lock.release()
class RepeatingTimer(Thread):
'''
Calls a specified function repeatedly at a specified interval. Runs in a
daemon thread (i.e. the interpreter can exit while it is still running).
Call :meth:`start()` to start it.
'''
def repeat(self):
while True:
self.event.wait(self.interval)
if self.event.isSet():
break
self.action()
def __init__(self, interval, func, name):
self.event = Event()
self.interval = interval
self.action = func
Thread.__init__(self, target=self.repeat, name=name)
self.setDaemon(True)
class ControlError(Exception):
pass
class Overseer(object):
'''
Responsible for controlling worker processes. The main interface is the
methods, :meth:`initialize_job`, :meth:`control`.
'''
KILL_RESULT = 'Server: job killed by user|||#@#$%&*)*(*$#$%#$@&'
INTERVAL = 0.1
def __init__(self, server, port, timeout=5):
self.worker_status = mother.spawn_worker('127.0.0.1:'+str(port))
self.socket = server.accept()[0]
# Needed if terminate called when interpreter is shutting down
self.os = os
self.signal = signal
self.on_probation = False
self.terminated = False
self.working = False
self.timeout = timeout
self.last_job_time = time.time()
self._stop = False
if not select([self.socket], [], [], 120)[0]:
raise RuntimeError(_('Could not launch worker process.'))
ID = self.read().split(':')
if ID[0] != 'CALIBRE_WORKER':
raise RuntimeError('Impostor')
self.worker_pid = int(ID[1])
self.write('OK')
if self.read() != 'WAITING':
raise RuntimeError('Worker sulking')
def terminate(self):
'Kill worker process.'
self.terminated = True
try:
if self.socket:
self.write('STOP:')
time.sleep(1)
self.socket.shutdown(socket.SHUT_RDWR)
except:
pass
if iswindows:
win32api = __import__('win32api')
try:
handle = win32api.OpenProcess(1, False, self.worker_pid)
win32api.TerminateProcess(handle, -1)
except:
pass
else:
try:
try:
self.os.kill(self.worker_pid, self.signal.SIGKILL)
time.sleep(0.5)
finally:
self.worker_status.kill()
except:
pass
def write(self, msg, timeout=None):
write(self.socket, msg, timeout=self.timeout if timeout is None else timeout)
def read(self, timeout=None):
return read(self.socket, timeout=self.timeout if timeout is None else timeout)
def __eq__(self, other):
return hasattr(other, 'process') and hasattr(other, 'worker_pid') and self.worker_pid == other.worker_pid
def is_viable(self):
if self.terminated:
return False
return self.worker_status.is_alive()
def select(self, timeout=0):
return select([self.socket], [self.socket], [self.socket], timeout)
def initialize_job(self, job):
'''
Sends `job` to worker process. Can raise `ControlError` if worker process
does not respond appropriately. In this case, this Overseer is useless
and should be discarded.
`job`: An instance of :class:`Job`.
'''
self.working = True
self.write('JOB:'+cPickle.dumps((job.func, job.args, job.kwargs), -1))
msg = self.read()
if msg != 'OK':
raise ControlError('Failed to initialize job on worker %d:%s'%(self.worker_pid, msg))
self.job = job
self.last_report = time.time()
job.start_work()
def control(self):
'''
Listens for messages from the worker process and dispatches them
appropriately. If the worker process dies unexpectedly, returns a result
of None with a ControlError indicating the worker died.
Returns a :class:`Result` instance or None, if the worker is still working.
'''
if select([self.socket],[],[],0)[0]:
msg = self.read()
if msg:
self.on_probation = False
self.last_report = time.time()
else:
if self.on_probation:
self.terminate()
self.job.result = None
self.job.exception = ControlError('Worker process died unexpectedly')
return
else:
self.on_probation = True
return
word, msg = msg.partition(':')[0], msg.partition(':')[-1]
if word == 'PING':
self.write('OK')
return
elif word == 'RESULT':
self.write('OK')
self.job.result = cPickle.loads(msg)
return True
elif word == 'OUTPUT':
self.write('OK')
try:
self.job.output(''.join(cPickle.loads(msg)))
except:
self.job.output('Bad output message: '+ repr(msg))
elif word == 'PROGRESS':
self.write('OK')
percent = None
try:
percent, msg = cPickle.loads(msg)[-1]
except:
print 'Bad progress update:', repr(msg)
if percent is not None:
self.job.update_status(percent, msg)
elif word == 'ERROR':
self.write('OK')
exception, tb = cPickle.loads(msg)
self.job.output(u'%s\n%s'%(exception, tb))
self.job.exception, self.job.traceback = exception, tb
return True
else:
self.terminate()
self.job.exception = ControlError('Worker sent invalid msg: %s'%repr(msg))
return
if not self.worker_status.is_alive() or time.time() - self.last_report > 380:
self.terminate()
self.job.exception = ControlError('Worker process died unexpectedly')
return
class JobKilled(Exception):
pass
class Job(object):
def __init__(self, job_done, job_manager=None,
args=[], kwargs={}, description=None):
self.args = args
self.kwargs = kwargs
self._job_done = job_done
self.job_manager = job_manager
self.is_running = False
self.has_run = False
self.percent = -1
self.msg = None
self.description = description
self.start_time = None
self.running_time = None
self.result = self.exception = self.traceback = self.log = None
def __cmp__(self, other):
sstatus, ostatus = self.status(), other.status()
if sstatus == ostatus or (self.has_run and other.has_run):
if self.start_time == other.start_time:
return cmp(id(self), id(other))
return cmp(self.start_time, other.start_time)
if sstatus == 'WORKING':
return -1
if ostatus == 'WORKING':
return 1
if sstatus == 'WAITING':
return -1
if ostatus == 'WAITING':
return 1
def job_done(self):
self.is_running, self.has_run = False, True
self.running_time = (time.time() - self.start_time) if \
self.start_time is not None else 0
if self.job_manager is not None:
self.job_manager.job_done(self)
self._job_done(self)
def start_work(self):
self.is_running = True
self.has_run = False
self.start_time = time.time()
if self.job_manager is not None:
self.job_manager.start_work(self)
def update_status(self, percent, msg=None):
self.percent = percent
self.msg = msg
if self.job_manager is not None:
try:
self.job_manager.status_update(self)
except:
traceback.print_exc()
def status(self):
if self.is_running:
return 'WORKING'
if not self.has_run:
return 'WAITING'
if self.has_run:
if self.exception is None:
return 'DONE'
return 'ERROR'
def console_text(self):
ans = [u'Job: ']
if self.description:
ans[0] += self.description
if self.exception is not None:
header = unicode(self.exception.__class__.__name__) if \
hasattr(self.exception, '__class__') else u'Error'
header = u'**%s**'%header
header += u': '
try:
header += unicode(self.exception)
except:
header += unicode(repr(self.exception))
ans.append(header)
if self.traceback:
ans.append(u'**Traceback**:')
ans.extend(self.traceback.split('\n'))
if self.log:
if isinstance(self.log, str):
self.log = unicode(self.log, 'utf-8', 'replace')
ans.append(self.log)
return (u'\n'.join(ans)).encode('utf-8')
def gui_text(self):
ans = [u'Job: ']
if self.description:
if not isinstance(self.description, unicode):
self.description = self.description.decode('utf-8', 'replace')
ans[0] += u'**%s**'%self.description
if self.exception is not None:
header = unicode(self.exception.__class__.__name__) if \
hasattr(self.exception, '__class__') else u'Error'
header = u'**%s**'%header
header += u': '
try:
header += unicode(self.exception)
except:
header += unicode(repr(self.exception))
ans.append(header)
if self.traceback:
ans.append(u'**Traceback**:')
ans.extend(self.traceback.split('\n'))
if self.log:
ans.append(u'**Log**:')
if isinstance(self.log, str):
self.log = unicode(self.log, 'utf-8', 'replace')
ans.extend(self.log.split('\n'))
ans = [x.decode(preferred_encoding, 'replace') if isinstance(x, str) else x for x in ans]
return u'<br>'.join(ans)
class ParallelJob(Job):
def __init__(self, func, *args, **kwargs):
Job.__init__(self, *args, **kwargs)
self.func = func
self.done = self.job_done
def output(self, msg):
if not self.log:
self.log = u''
if not isinstance(msg, unicode):
msg = msg.decode('utf-8', 'replace')
if msg:
self.log += msg
if self.job_manager is not None:
self.job_manager.output(self)
def remove_ipc_socket(path):
os = __import__('os')
if os.path.exists(path):
os.unlink(path)
class Server(Thread):
KILL_RESULT = Overseer.KILL_RESULT
START_PORT = 10013
PID = os.getpid()
def __init__(self, number_of_workers=detect_ncpus()):
Thread.__init__(self)
self.setDaemon(True)
self.server_socket = socket.socket(SOCKET_TYPE, socket.SOCK_STREAM)
self.port = tempfile.mktemp(prefix='calibre_server')+'_%d_'%self.PID if not iswindows else self.START_PORT
while True:
try:
address = ('localhost', self.port) if iswindows else self.port
self.server_socket.bind(address)
break
except socket.error:
self.port += (1 if iswindows else '1')
if not iswindows:
atexit.register(remove_ipc_socket, self.port)
self.server_socket.listen(5)
self.number_of_workers = number_of_workers
self.pool, self.jobs, self.working = [], collections.deque(), []
atexit.register(self.killall)
atexit.register(self.close)
self.job_lock = RLock()
self.overseer_lock = RLock()
self.working_lock = RLock()
self.result_lock = RLock()
self.pool_lock = RLock()
self.start()
def split(self, tasks):
'''
Split a list into a list of sub lists, with the number of sub lists being
no more than the number of workers this server supports. Each sublist contains
two tuples of the form (i, x) where x is an element fro the original list
and i is the index of the element x in the original list.
'''
ans, count, pos = [], 0, 0
delta = int(ceil(len(tasks)/float(self.number_of_workers)))
while count < len(tasks):
section = []
for t in tasks[pos:pos+delta]:
section.append((count, t))
count += 1
ans.append(section)
pos += delta
return ans
def close(self):
try:
self.server_socket.shutdown(socket.SHUT_RDWR)
except:
pass
def add_job(self, job):
with self.job_lock:
self.jobs.append(job)
if job.job_manager is not None:
job.job_manager.add_job(job)
def poll(self):
'''
Return True if the server has either working or queued jobs
'''
with self.job_lock:
with self.working_lock:
return len(self.jobs) + len(self.working) > 0
def wait(self, sleep=1):
'''
Wait until job queue is empty
'''
while self.poll():
time.sleep(sleep)
def run(self):
while True:
job = None
with self.job_lock:
if len(self.jobs) > 0 and len(self.working) < self.number_of_workers:
job = self.jobs.popleft()
with self.pool_lock:
o = None
while self.pool:
o = self.pool.pop()
try:
o.initialize_job(job)
break
except:
o.terminate()
if o is None:
o = Overseer(self.server_socket, self.port)
try:
o.initialize_job(job)
except Exception, err:
o.terminate()
job.exception = err
job.traceback = traceback.format_exc()
job.done()
o = None
if o and o.is_viable():
with self.working_lock:
self.working.append(o)
with self.working_lock:
done = []
for o in self.working:
try:
if o.control() is not None or o.job.exception is not None:
o.job.done()
done.append(o)
except Exception, err:
o.job.exception = err
o.job.traceback = traceback.format_exc()
o.terminate()
o.job.done()
done.append(o)
for o in done:
self.working.remove(o)
if o and o.is_viable():
with self.pool_lock:
self.pool.append(o)
try:
time.sleep(1)
except:
return
def killall(self):
with self.pool_lock:
map(lambda x: x.terminate(), self.pool)
self.pool = []
def kill(self, job):
with self.working_lock:
pop = None
for o in self.working:
if o.job == job or o == job:
try:
o.terminate()
except: pass
o.job.exception = JobKilled(_('Job stopped by user'))
try:
o.job.done()
except: pass
pop = o
break
if pop is not None:
self.working.remove(pop)
def run_free_job(self, func, args=[], kwdargs={}):
pt = PersistentTemporaryFile('.pickle', '_IPC_')
pt.write(cPickle.dumps((func, args, kwdargs)))
pt.close()
mother.spawn_free_spirit(binascii.hexlify(pt.name))
##########################################################################################
##################################### CLIENT CODE #####################################
##########################################################################################
class BufferedSender(object):
def __init__(self, socket):
self.socket = socket
self.wbuf, self.pbuf = [], []
self.wlock, self.plock = RLock(), RLock()
self.last_report = None
self.timer = RepeatingTimer(0.5, self.send, 'BufferedSender')
self.timer.start()
def write(self, msg):
if not isinstance(msg, basestring):
msg = unicode(msg)
with self.wlock:
self.wbuf.append(msg)
def send(self):
if callable(select) and select([self.socket], [], [], 0)[0]:
msg = read(self.socket)
if msg == 'PING:':
write(self.socket, 'OK')
elif msg:
self.socket.shutdown(socket.SHUT_RDWR)
thread.interrupt_main()
time.sleep(1)
raise SystemExit
if not select([], [self.socket], [], 30)[1]:
print >>sys.__stderr__, 'Cannot pipe to overseer'
return
reported = False
with self.wlock:
if self.wbuf:
msg = cPickle.dumps(self.wbuf, -1)
self.wbuf = []
write(self.socket, 'OUTPUT:'+msg)
read(self.socket, 10)
reported = True
with self.plock:
if self.pbuf:
msg = cPickle.dumps(self.pbuf, -1)
self.pbuf = []
write(self.socket, 'PROGRESS:'+msg)
read(self.socket, 10)
reported = True
if self.last_report is not None:
if reported:
self.last_report = time.time()
elif time.time() - self.last_report > 60:
write(self.socket, 'PING:')
read(self.socket, 10)
self.last_report = time.time()
def notify(self, percent, msg=''):
with self.plock:
self.pbuf.append((percent, msg))
def flush(self):
pass
def get_func(name):
module, func, kwdargs, notification = PARALLEL_FUNCS[name]
module = __import__(module, fromlist=[1])
func = getattr(module, func)
return func, kwdargs, notification
_atexit = collections.deque()
def myatexit(func, *args, **kwargs):
_atexit.append((func, args, kwargs))
def work(client_socket, func, args, kwdargs):
sys.stdout.last_report = time.time()
orig = atexit.register
atexit.register = myatexit
try:
func, kargs, notification = get_func(func)
if notification is not None and hasattr(sys.stdout, 'notify'):
kargs[notification] = sys.stdout.notify
kargs.update(kwdargs)
res = func(*args, **kargs)
if hasattr(sys.stdout, 'send'):
sys.stdout.send()
return res
finally:
atexit.register = orig
sys.stdout.last_report = None
while True:
try:
func, args, kwargs = _atexit.pop()
except IndexError:
break
try:
func(*args, **kwargs)
except (Exception, SystemExit):
continue
time.sleep(5) # Give any in progress BufferedSend time to complete
def worker(host, port):
client_socket = socket.socket(SOCKET_TYPE, socket.SOCK_STREAM)
address = (host, port) if iswindows else port
client_socket.connect(address)
write(client_socket, 'CALIBRE_WORKER:%d'%os.getpid())
msg = read(client_socket, timeout=10)
if msg != 'OK':
return 1
write(client_socket, 'WAITING')
sys.stdout = BufferedSender(client_socket)
sys.stderr = sys.stdout
while True:
if not select([client_socket], [], [], 60)[0]:
time.sleep(1)
continue
msg = read(client_socket, timeout=60)
if msg.startswith('JOB:'):
func, args, kwdargs = cPickle.loads(msg[4:])
write(client_socket, 'OK')
try:
result = work(client_socket, func, args, kwdargs)
write(client_socket, 'RESULT:'+ cPickle.dumps(result))
except BaseException, err:
exception = (err.__class__.__name__, unicode(str(err), 'utf-8', 'replace'))
tb = unicode(traceback.format_exc(), 'utf-8', 'replace')
msg = 'ERROR:'+cPickle.dumps((exception, tb),-1)
write(client_socket, msg)
res = read(client_socket, 10)
if res != 'OK':
break
gc.collect()
elif msg == 'PING:':
write(client_socket, 'OK')
elif msg == 'STOP:':
client_socket.shutdown(socket.SHUT_RDWR)
return 0
elif not msg:
time.sleep(1)
else:
print >>sys.__stderr__, 'Invalid protocols message', msg
return 1
def free_spirit(path):
func, args, kwdargs = cPickle.load(open(path, 'rb'))
try:
os.unlink(path)
except:
pass
func, kargs = get_func(func)[:2]
kargs.update(kwdargs)
func(*args, **kargs)
def main(args=sys.argv):
global isworker
isworker = True
args = args[1].split(':')
if len(args) == 1:
free_spirit(binascii.unhexlify(re.sub(r'[^a-f0-9A-F]', '', args[0])))
else:
worker(args[0].replace("'", ''), int(args[1]) if iswindows else args[1])
return 0
if __name__ == '__main__':
sys.exit(main())

2
src/calibre/trac/plugins/download.py
View File

@ -20,7 +20,7 @@ DEPENDENCIES = [
('BeautifulSoup', '3.0.5', 'beautifulsoup', 'python-beautifulsoup', 'python-BeautifulSoup'),
('dnspython', '1.6.0', 'dnspython', 'dnspython', 'dnspython', 'dnspython'),
('poppler', '0.10.5', 'poppler', 'poppler', 'poppler', 'poppler'),
('pdftk', '1.12', 'pdftk', 'pdftk', 'pdftk', 'pdftk'),
('podofo', '0.7', 'podofo', 'podofo', 'podofo', 'podofo'),
]

2
src/calibre/trac/plugins/templates/linux.html
View File

@ -49,7 +49,7 @@
</p>
<p>
${app} is available in the software repositories of the following
linux distributions:
supported linux distributions:
<table id="install_info">
<col width="150" /><col width="*" />
<tr>

137
src/calibre/utils/ipc/job.py Normal file
View File

@ -0,0 +1,137 @@
#!/usr/bin/env python
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
from __future__ import with_statement
__license__ = 'GPL v3'
__copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'
_count = 0
import time, cStringIO
from Queue import Queue, Empty
class BaseJob(object):
WAITING = 0
RUNNING = 1
FINISHED = 2
def __init__(self, description, done=lambda x: x):
global _count
_count += 1
self.id = _count
self.description = description
self.done = done
self.done2 = None
self.killed = False
self.failed = False
self.start_time = None
self.result = None
self.duration = None
self.log_path = None
self.notifications = Queue()
self._run_state = self.WAITING
self.percent = 0
self._message = None
self._status_text = _('Waiting...')
self._done_called = False
def update(self):
if self.duration is not None:
self._run_state = self.FINISHED
self.percent = 100
if self.killed:
self._status_text = _('Stopped')
else:
self._status_text = _('Error') if self.failed else _('Finished')
if not self._done_called:
self._done_called = True
try:
self.done(self)
except:
pass
try:
if callable(self.done2):
self.done2(self)
except:
pass
elif self.start_time is not None:
self._run_state = self.RUNNING
self._status_text = _('Working...')
while True:
try:
self.percent, self._message = self.notifications.get_nowait()
self.percent *= 100.
except Empty:
break
@property
def status_text(self):
if self._run_state == self.FINISHED or not self._message:
return self._status_text
return self._message
@property
def run_state(self):
return self._run_state
@property
def running_time(self):
if self.duration is not None:
return self.duration
if self.start_time is not None:
return time.time() - self.start_time
return None
@property
def is_finished(self):
return self._run_state == self.FINISHED
@property
def is_started(self):
return self._run_state != self.WAITING
@property
def is_running(self):
return self.is_started and not self.is_finished
def __cmp__(self, other):
if self.is_finished == other.is_finished:
if self.start_time is None:
if other.start_time is None: # Both waiting
return cmp(other.id, self.id)
else:
return 1
else:
if other.start_time is None:
return -1
else: # Both running
return cmp(other.start_time, self.start_time)
else:
return 1 if self.is_finished else -1
return 0
@property
def log_file(self):
if self.log_path:
return open(self.log_path, 'rb')
return cStringIO.StringIO(_('No details available.'))
@property
def details(self):
return self.log_file.read().decode('utf-8')
class ParallelJob(BaseJob):
def __init__(self, name, description, done, args=[], kwargs={}):
self.name, self.args, self.kwargs = name, args, kwargs
BaseJob.__init__(self, description, done)

3
src/calibre/utils/ipc/launch.py
View File

@ -70,7 +70,7 @@ class Worker(object):
@property
def is_alive(self):
return hasattr(self, 'child') and self.child.poll() is not None
return hasattr(self, 'child') and self.child.poll() is None
@property
def returncode(self):
@ -144,6 +144,7 @@ class Worker(object):
self.child = subprocess.Popen(cmd, **args)
self.log_path = ret
return ret

236
src/calibre/utils/ipc/server.py
View File

@ -6,5 +6,241 @@ __license__ = 'GPL v3'
__copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'
import os, cPickle, time, tempfile
from math import ceil
from threading import Thread, RLock
from Queue import Queue, Empty
from multiprocessing.connection import Listener
from multiprocessing import cpu_count
from collections import deque
from binascii import hexlify
from calibre.utils.ipc.launch import Worker
from calibre.utils.ipc.worker import PARALLEL_FUNCS
_counter = 0
class ConnectedWorker(Thread):
def __init__(self, worker, conn, rfile):
Thread.__init__(self)
self.daemon = True
self.conn = conn
self.worker = worker
self.notifications = Queue()
self._returncode = 'dummy'
self.killed = False
self.log_path = worker.log_path
self.rfile = rfile
def start_job(self, job):
notification = PARALLEL_FUNCS[job.name][-1] is not None
self.conn.send((job.name, job.args, job.kwargs))
if notification:
self.start()
else:
self.conn.close()
self.job = job
def run(self):
while True:
try:
x = self.conn.recv()
self.notifications.put(x)
except BaseException:
break
try:
self.conn.close()
except BaseException:
pass
def kill(self):
self.killed = True
try:
self.worker.kill()
except BaseException:
pass
@property
def is_alive(self):
return not self.killed and self.worker.is_alive
@property
def returncode(self):
if self._returncode != 'dummy':
return self._returncode
r = self.worker.returncode
if self.killed and r is None:
self._returncode = 1
return 1
if r is not None:
self._returncode = r
return r
class Server(Thread):
def __init__(self, notify_on_job_done=lambda x: x, pool_size=None):
Thread.__init__(self)
self.daemon = True
global _counter
self.id = _counter+1
_counter += 1
self.pool_size = cpu_count() if pool_size is None else pool_size
self.notify_on_job_done = notify_on_job_done
self.auth_key = os.urandom(32)
self.listener = Listener(authkey=self.auth_key, backlog=4)
self.add_jobs_queue, self.changed_jobs_queue = Queue(), Queue()
self.kill_queue = Queue()
self.waiting_jobs, self.processing_jobs = deque(), deque()
self.pool, self.workers = deque(), deque()
self.launched_worker_count = 0
self._worker_launch_lock = RLock()
self.start()
def launch_worker(self, gui=False, redirect_output=None):
with self._worker_launch_lock:
self.launched_worker_count += 1
id = self.launched_worker_count
rfile = os.path.join(tempfile.gettempdir(),
'calibre_ipc_result_%d_%d.pickle'%(self.id, id))
env = {
'CALIBRE_WORKER_ADDRESS' :
hexlify(cPickle.dumps(self.listener.address, -1)),
'CALIBRE_WORKER_KEY' : hexlify(self.auth_key),
'CALIBRE_WORKER_RESULT' : hexlify(rfile),
}
w = Worker(env, gui=gui)
if redirect_output is None:
redirect_output = not gui
w(redirect_output=redirect_output)
conn = self.listener.accept()
if conn is None:
raise Exception('Failed to launch worker process')
return ConnectedWorker(w, conn, rfile)
def add_job(self, job):
job.done2 = self.notify_on_job_done
self.add_jobs_queue.put(job)
def run_job(self, job, gui=True, redirect_output=False):
w = self.launch_worker(gui=gui, redirect_output=redirect_output)
w.start_job(job)
def run(self):
while True:
try:
job = self.add_jobs_queue.get(True, 0.2)
if job is None:
break
self.waiting_jobs.append(job)
except Empty:
pass
for worker in self.workers:
while True:
try:
n = worker.notifications.get_nowait()
worker.job.notifications.put(n)
self.changed_jobs_queue.put(job)
except Empty:
break
for worker in [w for w in self.workers if not w.is_alive]:
self.workers.remove(worker)
job = worker.job
if worker.returncode != 0:
job.failed = True
job.returncode = worker.returncode
elif os.path.exists(worker.rfile):
job.result = cPickle.load(open(worker.rfile, 'rb'))
os.remove(worker.rfile)
job.duration = time.time() - job.start_time
self.changed_jobs_queue.put(job)
if len(self.pool) + len(self.workers) < self.pool_size:
try:
self.pool.append(self.launch_worker())
except Exception, err:
pass
if len(self.pool) > 0 and len(self.waiting_jobs) > 0:
job = self.waiting_jobs.pop()
worker = self.pool.pop()
job.start_time = time.time()
worker.start_job(job)
self.workers.append(worker)
job.log_path = worker.log_path
self.changed_jobs_queue.put(job)
while True:
try:
j = self.kill_queue.get_nowait()
self._kill_job(j)
except Empty:
break
def kill_job(self, job):
self.kill_queue.put(job)
def killall(self):
for job in self.workers:
self.kill_queue.put(job)
def _kill_job(self, job):
if job.start_time is None: return
for worker in self.workers:
if job is worker.job:
worker.kill()
job.killed = True
break
def split(self, tasks):
'''
Split a list into a list of sub lists, with the number of sub lists being
no more than the number of workers this server supports. Each sublist contains
two tuples of the form (i, x) where x is an element from the original list
and i is the index of the element x in the original list.
'''
ans, count, pos = [], 0, 0
delta = int(ceil(len(tasks)/float(self.pool_size)))
while count < len(tasks):
section = []
for t in tasks[pos:pos+delta]:
section.append((count, t))
count += 1
ans.append(section)
pos += delta
return ans
def close(self):
try:
self.add_jobs_queue.put(None)
self.listener.close()
except:
pass
time.sleep(0.2)
for worker in self.workers:
try:
worker.kill()
except:
pass
for worker in self.pool:
try:
worker.kill()
except:
pass
def __enter__(self):
return self
def __exit__(self, *args):
self.close()

25
src/calibre/utils/ipc/worker.py
View File

@ -6,11 +6,12 @@ __license__ = 'GPL v3'
__copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'
import os, cPickle
import os, cPickle, sys
from multiprocessing.connection import Client
from threading import Thread
from queue import Queue
from Queue import Queue
from contextlib import closing
from binascii import unhexlify
PARALLEL_FUNCS = {
'lrfviewer' :
@ -24,13 +25,16 @@ PARALLEL_FUNCS = {
'gui_convert' :
('calibre.gui2.convert.gui_conversion', 'gui_convert', 'notification'),
'read_metadata' :
('calibre.ebooks.metadata.worker', 'read_metadata_', 'notification'),
}
class Progress(Thread):
def __init__(self, conn):
self.daemon = True
Thread.__init__(self)
self.daemon = True
self.conn = conn
self.queue = Queue()
@ -56,23 +60,30 @@ def get_func(name):
return func, notification
def main():
address = cPickle.loads(os.environ['CALIBRE_WORKER_ADDRESS'])
key = os.environ['CALIBRE_WORKER_KEY']
address = cPickle.loads(unhexlify(os.environ['CALIBRE_WORKER_ADDRESS']))
key = unhexlify(os.environ['CALIBRE_WORKER_KEY'])
resultf = unhexlify(os.environ['CALIBRE_WORKER_RESULT'])
with closing(Client(address, authkey=key)) as conn:
name, args, kwargs = conn.recv()
#print (name, args, kwargs)
#sys.stdout.flush()
func, notification = get_func(name)
notifier = Progress(conn)
if notification:
kwargs[notification] = notifier
notifier.start()
func(*args, **kwargs)
result = func(*args, **kwargs)
if result is not None:
cPickle.dump(result, open(resultf, 'wb'), -1)
notifier.queue.put(None)
sys.stdout.flush()
sys.stderr.flush()
return 0
if __name__ == '__main__':
raise SystemExit(main())
sys.exit(main())

26
src/calibre/utils/podofo/__init__.py
View File

@ -21,17 +21,16 @@ def get_metadata(stream):
raise Unavailable(podofo_err)
raw = stream.read()
stream.seek(0)
p = podofo.PdfMemDocument()
p.Load(raw, len(raw))
info = p.GetInfo()
title = info.GetTitle().decode('utf-8').strip()
p = podofo.PDFDoc()
p.load(raw)
title = p.title
if not title:
title = getattr(stream, 'name', _('Unknown'))
title = os.path.splitext(os.path.basename(title))[0]
author = info.GetAuthor().decode('utf-8').strip()
author = p.author
authors = string_to_authors(author) if author else [_('Unknown')]
mi = MetaInformation(title, authors)
creator = info.GetCreator().decode('utf-8').strip()
creator = p.creator
if creator:
mi.book_producer = creator
return mi
@ -47,31 +46,28 @@ def set_metadata(stream, mi):
if not podofo:
raise Unavailable(podofo_err)
raw = stream.read()
p = podofo.PdfMemDocument()
p.Load(raw, len(raw))
info = p.GetInfo()
p = podofo.PDFDoc()
p.load(raw)
title = prep(mi.title)
touched = False
if title:
info.SetTitle(title)
p.title = title
touched = True
author = prep(authors_to_string(mi.authors))
if author:
print repr(author)
info.SetAuthor(author)
p.author = author
touched = True
bkp = prep(mi.book_producer)
if bkp:
info.SetCreator(bkp)
p.creator = bkp
touched = True
if touched:
p.SetInfo(info)
from calibre.ptempfile import TemporaryFile
with TemporaryFile('_pdf_set_metadata.pdf') as f:
p.Write(f)
p.save(f)
raw = open(f, 'rb').read()
stream.seek(0)
stream.truncate()

330
src/calibre/utils/podofo/podofo.cpp Normal file
View File

@ -0,0 +1,330 @@
#define UNICODE
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#define USING_SHARED_PODOFO
#include <podofo.h>
using namespace PoDoFo;
#include <string.h>
class podofo_pdfmem_wrapper : public PdfMemDocument {
public:
inline void set_info(PdfInfo *i) { this->SetInfo(i); }
};
typedef struct {
PyObject_HEAD
/* Type-specific fields go here. */
podofo_pdfmem_wrapper *doc;
} podofo_PDFDoc;
extern "C" {
static void
podofo_PDFDoc_dealloc(podofo_PDFDoc* self)
{
if (self->doc != NULL) delete self->doc;
self->ob_type->tp_free((PyObject*)self);
}
static PyObject *
podofo_PDFDoc_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
podofo_PDFDoc *self;
self = (podofo_PDFDoc *)type->tp_alloc(type, 0);
if (self != NULL) {
self->doc = new podofo_pdfmem_wrapper();
if (self->doc == NULL) { Py_DECREF(self); return NULL; }
}
return (PyObject *)self;
}
static PyObject *
podofo_PDFDoc_load(podofo_PDFDoc *self, PyObject *args, PyObject *kwargs) {
char *buffer; Py_ssize_t size;
if (PyArg_ParseTuple(args, "s#", &buffer, &size)) {
try {
self->doc->Load(buffer, size);
} catch(const PdfError & err) {
PyErr_SetString(PyExc_ValueError, PdfError::ErrorMessage(err.GetError()));
return NULL;
}
} else return NULL;
Py_INCREF(Py_None);
return Py_None;
}
static PyObject *
podofo_PDFDoc_save(podofo_PDFDoc *self, PyObject *args, PyObject *kwargs) {
char *buffer;
if (PyArg_ParseTuple(args, "s", &buffer)) {
try {
self->doc->Write(buffer);
} catch(const PdfError & err) {
PyErr_SetString(PyExc_ValueError, PdfError::ErrorMessage(err.GetError()));
return NULL;
}
} else return NULL;
Py_INCREF(Py_None);
return Py_None;
}
static PyObject *
podofo_convert_pdfstring(const PdfString &s) {
std::string raw = s.GetStringUtf8();
return PyString_FromStringAndSize(raw.c_str(), raw.length());
}
static PdfString *
podofo_convert_pystring(PyObject *py) {
Py_UNICODE* u = PyUnicode_AS_UNICODE(py);
PyObject *u8 = PyUnicode_EncodeUTF8(u, PyUnicode_GET_SIZE(py), "replace");
if (u8 == NULL) { PyErr_NoMemory(); return NULL; }
pdf_utf8 *s8 = (pdf_utf8 *)PyString_AS_STRING(u8);
PdfString *ans = new PdfString(s8);
Py_DECREF(u8);
if (ans == NULL) PyErr_NoMemory();
return ans;
}
static PyObject *
podofo_PDFDoc_getter(podofo_PDFDoc *self, int field)
{
PyObject *ans;
PdfString s;
PdfInfo *info = self->doc->GetInfo();
if (info == NULL) {
PyErr_SetString(PyExc_Exception, "You must first load a PDF Document");
return NULL;
}
switch (field) {
case 0:
s = info->GetTitle(); break;
case 1:
s = info->GetAuthor(); break;
case 2:
s = info->GetSubject(); break;
case 3:
s = info->GetKeywords(); break;
case 4:
s = info->GetCreator(); break;
case 5:
s = info->GetProducer(); break;
default:
PyErr_SetString(PyExc_Exception, "Bad field");
return NULL;
}
ans = podofo_convert_pdfstring(s);
if (ans == NULL) {PyErr_NoMemory(); return NULL;}
PyObject *uans = PyUnicode_FromEncodedObject(ans, "utf-8", "replace");
Py_DECREF(ans);
if (uans == NULL) {return NULL;}
Py_INCREF(uans);
return uans;
}
static int
podofo_PDFDoc_setter(podofo_PDFDoc *self, PyObject *val, int field) {
if (val == NULL || !PyUnicode_Check(val)) {
PyErr_SetString(PyExc_ValueError, "Must use unicode objects to set metadata");
return -1;
}
PdfInfo *info = new PdfInfo(*self->doc->GetInfo());
if (info == NULL) {
PyErr_SetString(PyExc_Exception, "You must first load a PDF Document");
return -1;
}
PdfString *s = podofo_convert_pystring(val);
if (s == NULL) return -1;
switch (field) {
case 0:
info->SetTitle(*s); break;
case 1:
info->SetAuthor(*s); break;
case 2:
info->SetSubject(*s); break;
case 3:
info->SetKeywords(*s); break;
case 4:
info->SetCreator(*s); break;
case 5:
info->SetProducer(*s); break;
default:
PyErr_SetString(PyExc_Exception, "Bad field");
return -1;
}
self->doc->set_info(info);
return 0;
}
static PyObject *
podofo_PDFDoc_title_getter(podofo_PDFDoc *self, void *closure) {
return podofo_PDFDoc_getter(self, 0);
}
static PyObject *
podofo_PDFDoc_author_getter(podofo_PDFDoc *self, void *closure) {
return podofo_PDFDoc_getter(self, 1);
}
static PyObject *
podofo_PDFDoc_subject_getter(podofo_PDFDoc *self, void *closure) {
return podofo_PDFDoc_getter(self, 2);
}
static PyObject *
podofo_PDFDoc_keywords_getter(podofo_PDFDoc *self, void *closure) {
return podofo_PDFDoc_getter(self, 3);
}
static PyObject *
podofo_PDFDoc_creator_getter(podofo_PDFDoc *self, void *closure) {
return podofo_PDFDoc_getter(self, 4);
}
static PyObject *
podofo_PDFDoc_producer_getter(podofo_PDFDoc *self, void *closure) {
return podofo_PDFDoc_getter(self, 5);
}
static int
podofo_PDFDoc_title_setter(podofo_PDFDoc *self, PyObject *val, void *closure) {
return podofo_PDFDoc_setter(self, val, 0);
}
static int
podofo_PDFDoc_author_setter(podofo_PDFDoc *self, PyObject *val, void *closure) {
return podofo_PDFDoc_setter(self, val, 1);
}
static int
podofo_PDFDoc_subject_setter(podofo_PDFDoc *self, PyObject *val, void *closure) {
return podofo_PDFDoc_setter(self, val, 2);
}
static int
podofo_PDFDoc_keywords_setter(podofo_PDFDoc *self, PyObject *val, void *closure) {
return podofo_PDFDoc_setter(self, val, 3);
}
static int
podofo_PDFDoc_creator_setter(podofo_PDFDoc *self, PyObject *val, void *closure) {
return podofo_PDFDoc_setter(self, val, 4);
}
static int
podofo_PDFDoc_producer_setter(podofo_PDFDoc *self, PyObject *val, void *closure) {
return podofo_PDFDoc_setter(self, val, 5);
}
} /* extern "C" */
static PyMethodDef podofo_PDFDoc_methods[] = {
{"load", (PyCFunction)podofo_PDFDoc_load, METH_VARARGS,
"Load a PDF document from a byte buffer (string)"
},
{"save", (PyCFunction)podofo_PDFDoc_save, METH_VARARGS,
"Save the PDF document to a path on disk"
},
{NULL} /* Sentinel */
};
static PyGetSetDef podofo_PDFDoc_getseters[] = {
{"title",
(getter)podofo_PDFDoc_title_getter, (setter)podofo_PDFDoc_title_setter,
"Document title",
NULL},
{"author",
(getter)podofo_PDFDoc_author_getter, (setter)podofo_PDFDoc_author_setter,
"Document author",
NULL},
{"subject",
(getter)podofo_PDFDoc_subject_getter, (setter)podofo_PDFDoc_subject_setter,
"Document subject",
NULL},
{"keywords",
(getter)podofo_PDFDoc_keywords_getter, (setter)podofo_PDFDoc_keywords_setter,
"Document keywords",
NULL},
{"creator",
(getter)podofo_PDFDoc_creator_getter, (setter)podofo_PDFDoc_creator_setter,
"Document creator",
NULL},
{"producer",
(getter)podofo_PDFDoc_producer_getter, (setter)podofo_PDFDoc_producer_setter,
"Document producer",
NULL},
{NULL} /* Sentinel */
};
static PyTypeObject podofo_PDFDocType = {
PyObject_HEAD_INIT(NULL)
0, /*ob_size*/
"podofo.PDFDoc", /*tp_name*/
sizeof(podofo_PDFDoc), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor)podofo_PDFDoc_dealloc, /*tp_dealloc*/
0, /*tp_print*/
0, /*tp_getattr*/
0, /*tp_setattr*/
0, /*tp_compare*/
0, /*tp_repr*/
0, /*tp_as_number*/
0, /*tp_as_sequence*/
0, /*tp_as_mapping*/
0, /*tp_hash */
0, /*tp_call*/
0, /*tp_str*/
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
Py_TPFLAGS_DEFAULT, /*tp_flags*/
"PDF Documents", /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
podofo_PDFDoc_methods, /* tp_methods */
0, /* tp_members */
podofo_PDFDoc_getseters, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
0, /* tp_init */
0, /* tp_alloc */
podofo_PDFDoc_new, /* tp_new */
};
static PyMethodDef podofo_methods[] = {
{NULL} /* Sentinel */
};
extern "C" {
PyMODINIT_FUNC
initpodofo(void)
{
PyObject* m;
if (PyType_Ready(&podofo_PDFDocType) < 0)
return;
m = Py_InitModule3("podofo", podofo_methods,
"Wrapper for the PoDoFo pDF library");
Py_INCREF(&podofo_PDFDocType);
PyModule_AddObject(m, "PDFDoc", (PyObject *)&podofo_PDFDocType);
}
}

128
src/calibre/utils/podofo/podofo.sip
View File

@ -1,128 +0,0 @@
%Module podofo 0
%MappedType PdfString
{
%TypeHeaderCode
#define USING_SHARED_PODOFO
#include <PdfString.h>
using namespace PoDoFo;
%End
%ConvertFromTypeCode
if (sipCpp -> IsValid()) {
std::string raw = sipCpp->GetStringUtf8();
return PyString_FromStringAndSize(raw.c_str(), raw.length());
} else return PyString_FromString("");
%End
%ConvertToTypeCode
if (sipIsErr == NULL) {
if (sipIsErr == NULL)
return (PyUnicode_Check(sipPy) || PyString_Check(sipPy));
}
if (sipPy == Py_None) {
*sipCppPtr = NULL;
return 0;
}
if (PyString_Check(sipPy)) {
*sipCppPtr = new PdfString((pdf_utf8 *)PyString_AS_STRING(sipPy));
return sipGetState(sipTransferObj);
}
if (PyUnicode_Check(sipPy)) {
Py_UNICODE* u = PyUnicode_AS_UNICODE(sipPy);
PyObject *u8 = PyUnicode_EncodeUTF8(u, PyUnicode_GET_SIZE(sipPy), "replace");
pdf_utf8 *s8 = (pdf_utf8 *)PyString_AS_STRING(u8);
*sipCppPtr = new PdfString(s8);
return sipGetState(sipTransferObj);
}
*sipCppPtr = (PdfString *)sipForceConvertTo_PdfString(sipPy,sipIsErr);
return 1;
%End
};
class PdfObject {
%TypeHeaderCode
#define USING_SHARED_PODOFO
#include <PdfObject.h>
using namespace PoDoFo;
%End
public:
PdfObject();
};
class PdfInfo {
%TypeHeaderCode
#define USING_SHARED_PODOFO
#include <PdfInfo.h>
using namespace PoDoFo;
%End
public:
PdfInfo(PdfObject *);
PdfString GetAuthor() const;
PdfString GetSubject() const;
PdfString GetTitle() const;
PdfString GetKeywords() const;
PdfString GetCreator() const;
PdfString GetProducer() const;
void SetAuthor(PdfString &);
void SetSubject(PdfString &);
void SetTitle(PdfString &);
void SetKeywords(PdfString &);
void SetCreator(PdfString &);
void SetProducer(PdfString &);
};
class PdfOutputDevice {
%TypeHeaderCode
#define USING_SHARED_PODOFO
#include <PdfOutputDevice.h>
using namespace PoDoFo;
%End
public:
PdfOutputDevice(char *, long);
unsigned long GetLength();
unsigned long Tell();
void Flush();
};
class PdfMemDocument {
%TypeHeaderCode
#define USING_SHARED_PODOFO
#include <PdfMemDocument.h>
using namespace PoDoFo;
%End
public:
PdfMemDocument();
void Load(const char *filename);
void Load(const char *buffer, long size);
void Write(const char *filename);
PdfInfo *GetInfo() const;
protected:
void SetInfo(PdfInfo * /TransferThis/);
private:
PdfMemDocument(PdfMemDocument &);
};
%Exception PoDoFo::PdfError /PyName=PdfError/
{
%TypeHeaderCode
#define USING_SHARED_PODOFO
#include <PdfError.h>
%End
%RaiseCode
const char *detail = sipExceptionRef.what();
SIP_BLOCK_THREADS
PyErr_SetString(sipException_PoDoFo_PdfError, detail);
SIP_UNBLOCK_THREADS
%End
};

2
src/calibre/web/feeds/input.py
View File

@ -57,6 +57,8 @@ class RecipeInput(InputFormatPlugin):
ro = recipe(opts, log, self.report_progress)
ro.download()
for key, val in recipe.conversion_options.items():
setattr(opts, key, val)
opts.output_profile.flow_size = 0

18
src/calibre/web/feeds/news.py
View File

@ -156,13 +156,17 @@ class BasicNewsRecipe(Recipe):
#: :attr:`BasicNewsRecipe.filter_regexps` should be defined.
filter_regexps = []
#: List of options to pass to html2lrf, to customize generation of LRF ebooks.
html2lrf_options = []
#: Options to pass to html2epub to customize generation of EPUB ebooks.
html2epub_options = ''
#: Options to pass to oeb2mobi to customize generation of MOBI ebooks.
oeb2mobi_options = ''
#: Recipe specific options to control the conversion of the downloaded
#: content into an e-book. These will override any user or plugin specified
#: values, so only use if absolutely necessary. For example::
#: conversion_options = {
#: 'base_font_size' : 16,
#: 'tags' : 'mytag1,mytag2',
#: 'title' : 'My Title',
#: 'linearize_tables' : True,
#: }
#:
conversion_options = {}
#: List of tags to be removed. Specified tags are removed from downloaded HTML.
#: A tag is specified as a dictionary of the form::

2
src/calibre/web/feeds/recipes/__init__.py
View File

@ -42,7 +42,7 @@ recipe_modules = ['recipe_' + r for r in (
'moneynews', 'der_standard', 'diepresse', 'nzz_ger', 'hna',
'seattle_times', 'scott_hanselman', 'coding_horror', 'twitchfilms',
'stackoverflow', 'telepolis_artikel', 'zaobao', 'usnews',
'straitstimes',
'straitstimes', 'index_hu', 'pcworld_hu', 'hrt', 'rts',
)]
import re, imp, inspect, time, os

2
src/calibre/web/feeds/recipes/recipe_barrons.py
View File

@ -22,7 +22,7 @@ class Barrons(BasicNewsRecipe):
use_embedded_content = False
no_stylesheets = False
match_regexps = ['http://online.barrons.com/.*?html\?mod=.*?|file:.*']
html2lrf_options = [('--ignore-tables'),('--base-font-size=10')]
conversion_options = {'linearize_tables': True}
##delay = 1
## Don't grab articles more than 7 days old

28
src/calibre/web/feeds/recipes/recipe_blic.py
View File

@ -16,12 +16,14 @@ class Blic(BasicNewsRecipe):
description = 'Blic.co.yu online verzija najtiraznije novine u Srbiji donosi najnovije vesti iz Srbije i sveta, komentare, politicke analize, poslovne i ekonomske vesti, vesti iz regiona, intervjue, informacije iz kulture, reportaze, pokriva sve sportske dogadjaje, detaljan tv program, nagradne igre, zabavu, fenomenalni Blic strip, dnevni horoskop, arhivu svih dogadjaja'
publisher = 'RINGIER d.o.o.'
category = 'news, politics, Serbia'
delay = 1
oldest_article = 2
max_articles_per_feed = 100
remove_javascript = True
no_stylesheets = True
use_embedded_content = False
language = _('Serbian')
lang = 'sr-Latn-RS'
extra_css = '@font-face {font-family: "serif1";src:url(res:///opt/sony/ebook/FONT/tt0011m_.ttf)} @font-face {font-family: "sans1";src:url(res:///opt/sony/ebook/FONT/tt0003m_.ttf)} body{font-family: serif1, serif} .article_description{font-family: sans1, sans-serif} '
html2lrf_options = [
@ -45,26 +47,14 @@ class Blic(BasicNewsRecipe):
start_url, question, rest_url = url.partition('?')
return u'http://www.blic.rs/_print.php?' + rest_url
def cleanup_image_tags(self,soup):
for item in soup.findAll('img'):
for attrib in ['height','width','border','align']:
if item.has_key(attrib):
del item[attrib]
oldParent = item.parent
myIndex = oldParent.contents.index(item)
item.extract()
divtag = Tag(soup,'div')
brtag = Tag(soup,'br')
oldParent.insert(myIndex,divtag)
divtag.append(item)
divtag.append(brtag)
return soup
def preprocess_html(self, soup):
mtag = '<meta http-equiv="Content-Language" content="sr-Latn-RS"/>'
soup.head.insert(0,mtag)
mlang = Tag(soup,'meta',[("http-equiv","Content-Language"),("content",self.lang)])
soup.head.insert(0,mlang)
for item in soup.findAll(style=True):
del item['style']
return self.cleanup_image_tags(soup)
return self.adeify_images(soup)
def get_article_url(self, article):
raw = article.get('link', None)
return raw.replace('.co.yu','.rs')

66
src/calibre/web/feeds/recipes/recipe_hrt.py Normal file
View File

@ -0,0 +1,66 @@
#!/usr/bin/env python
__license__ = 'GPL v3'
__copyright__ = '2009, Darko Miletic <darko.miletic at gmail.com>'
'''
www.hrt.hr
'''
import re
from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup, Tag
class HRT(BasicNewsRecipe):
title = 'HRT: Vesti'
__author__ = 'Darko Miletic'
description = 'News from Croatia'
publisher = 'HRT'
category = 'news, politics, Croatia, HRT'
no_stylesheets = True
encoding = 'utf-8'
use_embedded_content = False
language = _("Croatian")
lang = 'hr-HR'
extra_css = '@font-face {font-family: "serif1";src:url(res:///opt/sony/ebook/FONT/tt0011m_.ttf)} body{font-family: serif1, serif} .article_description{font-family: serif1, serif}'
html2lrf_options = [
'--comment', description
, '--category', category
, '--publisher', publisher
]
html2epub_options = 'publisher="' + publisher + '"\ncomments="' + description + '"\ntags="' + category + '"\noverride_css=" p {text-indent: 0em; margin-top: 0em; margin-bottom: 0.5em} img {margin-top: 0em; margin-bottom: 0.4em}"'
preprocess_regexps = [(re.compile(u'\u0110'), lambda match: u'\u00D0')]
keep_only_tags = [dict(name='div', attrs={'class':'bigVijest'})]
remove_tags = [dict(name=['object','link','embed'])]
remove_tags_after = dict(name='div', attrs={'class':'nsAuthor'})
feeds = [
(u'Vijesti' , u'http://www.hrt.hr/?id=316&type=100&rss=vijesti' )
,(u'Sport' , u'http://www.hrt.hr/?id=316&type=100&rss=sport' )
,(u'Zabava' , u'http://www.hrt.hr/?id=316&type=100&rss=zabava' )
,(u'Filmovi i serije' , u'http://www.hrt.hr/?id=316&type=100&rss=filmovi' )
,(u'Dokumentarni program', u'http://www.hrt.hr/?id=316&type=100&rss=dokumentarci')
,(u'Glazba' , u'http://www.hrt.hr/?id=316&type=100&rss=glazba' )
,(u'Kultura' , u'http://www.hrt.hr/?id=316&type=100&rss=kultura' )
,(u'Mladi' , u'http://www.hrt.hr/?id=316&type=100&rss=mladi' )
,(u'Manjine' , u'http://www.hrt.hr/?id=316&type=100&rss=manjine' )
,(u'Radio' , u'http://www.hrt.hr/?id=316&type=100&rss=radio' )
]
def preprocess_html(self, soup):
soup.html['xml:lang'] = self.lang
soup.html['lang'] = self.lang
mlang = Tag(soup,'meta',[("http-equiv","Content-Language"),("content",self.lang)])
mcharset = Tag(soup,'meta',[("http-equiv","Content-Type"),("content","text/html; charset=UTF-8")])
soup.head.insert(0,mlang)
soup.head.insert(1,mcharset)
for item in soup.findAll(style=True):
del item['style']
return self.adeify_images(soup)

20
src/calibre/web/feeds/recipes/recipe_index_hu.py Normal file
View File

@ -0,0 +1,20 @@
from calibre.web.feeds.news import BasicNewsRecipe
class Index(BasicNewsRecipe):
title = u'INDEX.HU'
oldest_article = 3
max_articles_per_feed = 50
language = _('Hungarian')
__author__ = 'Ezmegaz'
feeds = [(u'ALL', u'http://index.hu/24ora/rss/'),
(u'BELF\xd6LD', u'http://index.hu/belfold/rss/default/'),
(u'K\xdcLF\xd6LD', u'http://index.hu/kulfold/rss/default/'),
(u'BULV\xc1R', u'http://index.hu/bulvar/rss/default/'),
(u'GAZDAS\xc1G', u'http://index.hu/gazdasag/rss/default/'),
(u'TECH', u'http://index.hu/tech/rss/main/'),
(u'KULT\xdaRA', u'http://index.hu/kultur/rss/main/'),
(u'TUDOM\xc1NY', u'http://index.hu/tudomany/rss/main/'),
(u'V\xc9LEM\xc9NY', u'http://index.hu/velemeny/rss/default/')]

14
src/calibre/web/feeds/recipes/recipe_nin.py
View File

@ -8,12 +8,13 @@ nin.co.rs
import re, urllib
from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup, Tag
class Nin(BasicNewsRecipe):
title = 'NIN online'
__author__ = 'Darko Miletic'
description = 'Nedeljne informativne novine'
publisher = 'NIN'
publisher = 'NIN D.O.O.'
category = 'news, politics, Serbia'
no_stylesheets = True
oldest_article = 15
@ -28,9 +29,9 @@ class Nin(BasicNewsRecipe):
remove_javascript = True
use_embedded_content = False
language = _('Serbian')
lang = 'sr-RS'
lang = 'sr-Latn-RS'
direction = 'ltr'
extra_css = '@font-face {font-family: "serif1";src:url(res:///opt/sony/ebook/FONT/tt0011m_.ttf)} @font-face {font-family: "sans1";src:url(res:///opt/sony/ebook/FONT/tt0003m_.ttf)} body{text-align: justify; font-family: serif1, serif} .article_description{font-family: sans1, sans-serif}'
extra_css = '@font-face {font-family: "serif1";src:url(res:///opt/sony/ebook/FONT/tt0011m_.ttf)} @font-face {font-family: "sans1";src:url(res:///opt/sony/ebook/FONT/tt0003m_.ttf)} body{font-family: serif1, serif} .article_description{font-family: sans1, sans-serif} .artTitle{font-size: x-large; font-weight: bold} .columnhead{font-size: small; font-weight: bold}'
html2lrf_options = [
'--comment' , description
@ -70,9 +71,10 @@ class Nin(BasicNewsRecipe):
def preprocess_html(self, soup):
soup.html['lang'] = self.lang
soup.html['dir' ] = self.direction
mtag = '<meta http-equiv="Content-Language" content="' + self.lang + '"/>'
mtag += '\n<meta http-equiv="Content-Type" content="text/html; charset=' + self.encoding + '"/>'
soup.head.insert(0,mtag)
mlang = Tag(soup,'meta',[("http-equiv","Content-Language"),("content",self.lang)])
mcharset = Tag(soup,'meta',[("http-equiv","Content-Type"),("content","text/html; charset=UTF-8")])
soup.head.insert(0,mlang)
soup.head.insert(1,mcharset)
for item in soup.findAll(style=True):
del item['style']
return soup

22
src/calibre/web/feeds/recipes/recipe_pcworld_hu.py Normal file
View File

@ -0,0 +1,22 @@
#!/usr/bin/env python
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
from __future__ import with_statement
__license__ = 'GPL v3'
__copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
__docformat__ = 'restructuredtext en'
from calibre.web.feeds.news import BasicNewsRecipe
class Index(BasicNewsRecipe):
title = u'PCWORLD.HU'
oldest_article = 3
max_articles_per_feed = 50
language = _('Hungarian')
__author__ = 'Ezmegaz'
feeds = [(u'H\xedrek', u'http://pcworld.hu/rss/rss.xml'), (u'Hardver h\xedrek', u'http://www.pcworld.hu/rss/rss_hardverhirek.xml'), (u'Szoftver h\xedrek', u'http://www.pcworld.hu/rss/rss_szoftverhirek.xml'), (u'Hardver cikkek', u'http://www.pcworld.hu/rss/rss_hardvercikkek.xml'), (u'Szoftver cikkek', u'http://www.pcworld.hu/rss/rss_szoftvercikkek.xml'), (u'Mobil h\xedrek', u'http://www.pcworld.hu/rss/rss_mobil.xml'), (u'\xdczleti h\xedrek', u'http://www.pcworld.hu/rss/rss_uzlet.xml'), (u'Let\xf6lt\xe9sek', u'http://www.pcworld.hu/rss/rss_letoltes.xml'), (u'PC World TV', u'http://tv.pcworld.hu/rss/rss_hun_pcw.xml'), (u'Tudta-e...?', u'http://pcworld.hu/rss/rss_tudtae.xml')]

18
src/calibre/web/feeds/recipes/recipe_pobjeda.py
View File

@ -10,6 +10,7 @@ pobjeda.co.me
import re
from calibre import strftime
from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup, Tag
class Pobjeda(BasicNewsRecipe):
title = 'Pobjeda Online'
@ -22,12 +23,13 @@ class Pobjeda(BasicNewsRecipe):
encoding = 'utf8'
remove_javascript = True
use_embedded_content = False
language = _('Serbian')
lang = 'sr-Latn-Me'
INDEX = u'http://www.pobjeda.co.me'
extra_css = '@font-face {font-family: "serif1";src:url(res:///opt/sony/ebook/FONT/tt0011m_.ttf)} body{text-align: justify; font-family: serif1, serif} .article_description{font-family: serif1, serif}'
extra_css = '@font-face {font-family: "serif1";src:url(res:///opt/sony/ebook/FONT/tt0011m_.ttf)} body{font-family: serif1, serif} .article_description{font-family: serif1, serif}'
html2lrf_options = [
'--comment', description
, '--base-font-size', '10'
, '--category', category
, '--publisher', publisher
]
@ -59,11 +61,13 @@ class Pobjeda(BasicNewsRecipe):
]
def preprocess_html(self, soup):
soup.html['xml:lang'] = 'sr-Latn-ME'
soup.html['lang'] = 'sr-Latn-ME'
mtag = '<meta http-equiv="Content-Language" content="sr-Latn-ME"/>'
soup.head.insert(0,mtag)
return soup
soup.html['xml:lang'] = self.lang
soup.html['lang'] = self.lang
mlang = Tag(soup,'meta',[("http-equiv","Content-Language"),("content",self.lang)])
mcharset = Tag(soup,'meta',[("http-equiv","Content-Type"),("content","text/html; charset=UTF-8")])
soup.head.insert(0,mlang)
soup.head.insert(1,mcharset)
return self.adeify_images(soup)
def get_cover_url(self):
cover_url = None

60
src/calibre/web/feeds/recipes/recipe_rts.py Normal file
View File

@ -0,0 +1,60 @@
#!/usr/bin/env python
__license__ = 'GPL v3'
__copyright__ = '2009, Darko Miletic <darko.miletic at gmail.com>'
'''
www.rts.rs
'''
import re
from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup, Tag
class RTS(BasicNewsRecipe):
title = 'RTS: Vesti'
__author__ = 'Darko Miletic'
description = 'News from Serbia'
publisher = 'RTS'
category = 'news, politics, Serbia, RTS'
no_stylesheets = True
encoding = 'utf-8'
use_embedded_content = True
language = _("Serbian")
lang = 'sr-Latn-RS'
extra_css = '@font-face {font-family: "serif1";src:url(res:///opt/sony/ebook/FONT/tt0011m_.ttf)} body{font-family: serif1, serif} .article_description{font-family: serif1, serif}'
html2lrf_options = [
'--comment', description
, '--category', category
, '--publisher', publisher
]
html2epub_options = 'publisher="' + publisher + '"\ncomments="' + description + '"\ntags="' + category + '"\noverride_css=" p {text-indent: 0em; margin-top: 0em; margin-bottom: 0.5em} img {margin-top: 0em; margin-bottom: 0.4em}"'
preprocess_regexps = [(re.compile(u'\u0110'), lambda match: u'\u00D0')]
feeds = [
(u'Vesti' , u'http://www.rts.rs/page/stories/sr/rss.html' )
,(u'Srbija' , u'http://www.rts.rs/page/stories/sr/rss/9/Srbija.html' )
,(u'Region' , u'http://www.rts.rs/page/stories/sr/rss/11/Region.html' )
,(u'Svet' , u'http://www.rts.rs/page/stories/sr/rss/10/Svet.html' )
,(u'Hronika' , u'http://www.rts.rs/page/stories/sr/rss/135/Hronika.html' )
,(u'Drustvo' , u'http://www.rts.rs/page/stories/sr/rss/125/Dru%C5%A1tvo.html')
,(u'Ekonomija' , u'http://www.rts.rs/page/stories/sr/rss/13/Ekonomija.html' )
,(u'Nauka' , u'http://www.rts.rs/page/stories/sr/rss/14/Nauka.html' )
,(u'Kultura' , u'http://www.rts.rs/page/stories/sr/rss/16/Kultura.html' )
,(u'Zanimljivosti' , u'http://www.rts.rs/page/stories/sr/rss/15/Zanimljivosti.html')
,(u'Sport' , u'http://www.rts.rs/page/sport/sr/rss.html' )
]
def preprocess_html(self, soup):
soup.html['xml:lang'] = self.lang
soup.html['lang'] = self.lang
mlang = Tag(soup,'meta',[("http-equiv","Content-Language"),("content",self.lang)])
mcharset = Tag(soup,'meta',[("http-equiv","Content-Type"),("content","text/html; charset=UTF-8")])
soup.head.insert(0,mlang)
soup.head.insert(1,mcharset)
return self.adeify_images(soup)

51
src/calibre/web/feeds/recipes/recipe_st_petersburg_times.py
View File

@ -1,39 +1,48 @@
#!/usr/bin/env python
__license__ = 'GPL v3'
__copyright__ = '2008, Darko Miletic <darko.miletic at gmail.com>'
__copyright__ = '2009, Darko Miletic <darko.miletic at gmail.com>'
'''
sptimes.ru
'''
from calibre import strftime
from calibre.web.feeds.news import BasicNewsRecipe
class PetersburgTimes(BasicNewsRecipe):
title = u'The St. Petersburg Times'
title = 'The St. Petersburg Times'
__author__ = 'Darko Miletic'
description = 'News from Russia'
oldest_article = 7
publisher = 'sptimes.ru'
category = 'news, politics, Russia'
max_articles_per_feed = 100
no_stylesheets = True
remove_javascript = True
encoding = 'cp1251'
use_embedded_content = False
language = _('English')
INDEX = 'http://www.sptimes.ru'
def parse_index(self):
articles = []
soup = self.index_to_soup(self.INDEX)
html2lrf_options = [
'--comment', description
, '--category', category
, '--publisher', publisher
, '--ignore-tables'
]
html2epub_options = 'publisher="' + publisher + '"\ncomments="' + description + '"\ntags="' + category + '"\nlinearize_tables=True'
remove_tags = [dict(name=['object','link','embed'])]
feeds = [(u'Headlines', u'http://sptimes.ru/headlines.php' )]
def preprocess_html(self, soup):
return self.adeify_images(soup)
def get_article_url(self, article):
raw = article.get('guid', None)
return raw
def print_version(self, url):
start_url, question, article_id = url.rpartition('/')
return u'http://www.sptimes.ru/index.php?action_id=100&story_id=' + article_id
for item in soup.findAll('a', attrs={'class':'story_link_o'}):
if item.has_key('href'):
url = self.INDEX + item['href'].replace('action_id=2','action_id=100')
title = self.tag_to_string(item)
c_date = strftime('%A, %d %B, %Y')
description = ''
articles.append({
'title':title,
'date':c_date,
'url':url,
'description':description
})
return [(soup.head.title.string, articles)]

111
src/calibre/web/feeds/recipes/recipe_sueddeutsche.py
View File

@ -4,13 +4,12 @@ __copyright__ = '2008, Kovid Goyal <kovid at kovidgoyal.net>'
'''
Fetch sueddeutsche.
'''
from calibre.web.feeds.news import BasicNewsRecipe
class Sueddeutsche(BasicNewsRecipe):
title = u'Sueddeutsche'
title = u'S\xc3\xbcddeutsche'
description = 'News from Germany'
__author__ = 'Oliver Niesner'
use_embedded_content = False
@ -19,56 +18,74 @@ class Sueddeutsche(BasicNewsRecipe):
oldest_article = 7
max_articles_per_feed = 50
no_stylesheets = True
encoding = 'latin1'
remove_tags_after = [dict(name='div', attrs={'class':'artikelBox navigatorBox'})]
#dict(name='table', attrs={'class':'bgf2f2f2 absatz print100'})]
encoding = 'iso-8859-15'
remove_javascript = True
remove_tags = [dict(name='div', attrs={'class':'bannerSuperBanner'}),
dict(name='div', attrs={'class':'bannerSky'}),
dict(name='div', attrs={'class':'footerLinks'}),
dict(name='div', attrs={'class':'seitenanfang'}),
dict(name='td', attrs={'class':'mar5'}),
dict(name='a', attrs={'class':'top'}),
dict(name='table', attrs={'class':'pageAktiv'}),
dict(name='table', attrs={'class':'xartable'}),
dict(name='table', attrs={'class':'wpnavi'}),
dict(name='table', attrs={'class':'bgcontent absatz'}),
dict(name='table', attrs={'class':'footer'}),
dict(name='table', attrs={'class':'artikelBox'}),
dict(name='table', attrs={'class':'kommentare'}),
dict(name='table', attrs={'class':'pageBoxBot'}),
dict(name='div', attrs={'class':'artikelBox navigatorBox'}),
dict(name='div', attrs={'class':'similar-article-box'}),
dict(name='div', attrs={'class':'videoBigHack'}),
remove_tags_after = [dict(name='p', attrs={'class':'mttt'}),
dict(name='p', attrs={'class':'artikelFliestext'})]
remove_tags = [dict(name='span', attrs={'class':'r10000000'}),
dict(name='td', attrs={'class':'artikelDruckenRight'}),
dict(name='td', attrs={'class':'stoerBSbgUnten'}),
dict(name='li', attrs={'class':'first'}),
dict(name='li', attrs={'class':'bookmark closed'}),
dict(name='li', attrs={'class':'print'}),
dict(name='li', attrs={'class':'mail'}),
dict(name='li', attrs={'class':'last'}),
dict(name='li', attrs={'class':'tiefethemen'}),
dict(name='li', attrs={'class':'prev'}),
dict(name='ul', attrs={'class':'activities'}),
dict(name='li', attrs={'class':'next'}),
dict(name='span', attrs={'class':'hidePrint'}),
dict(id='headerLBox'),
dict(id='bookmarklist1'),
dict(id='bookmarklist2'),
dict(id='rechteSpalte'),
dict(id='newsticker-list-small'),
dict(id='ntop5'),
dict(id='ntop5send'),
dict(id='ntop5commented'),
dict(id='nnav-bgheader'),
dict(id='nnav-headerteaser'),
dict(name='td', attrs={'class':'bgc4c4c4'}),
dict(name='div', attrs={'class':'footerCopy padleft5'}),
dict(name='div', attrs={'class':'articleDistractor'}),
dict(name='div', attrs={'class':'footerLinks'}),
dict(name='div', attrs={'class':'nnav-headimagebottom'}),
dict(name='div', attrs={'class':'nnavlink'}),
dict(name='div', attrs={'class':'nnavlinkhome'}),
dict(name='div', attrs={'class':'SpecialGrafik'}),
dict(name='div', attrs={'class':'similar-article-box'}),
dict(name='div', attrs={'class':'tiefethemen'}),
dict(name='table', attrs={'class':'footer'}),
dict(name='ul', attrs={'class':'breadcrumb'}),
dict(name='a', attrs={'class':'List'}),
dict(name='span', attrs={'class':'icVers'}),
dict(id='nnav-head'),
dict(id='nnav-top'),
dict(id='nnav-logodiv'),
dict(id='nnav-logo'),
dict(id='nnav-logodiv'),
dict(id='nnav-bottom'),
dict(id='nnav-headimagebottom'),
dict(id='headerLBox'),
dict(id='logout'),
dict(id='nnav-headerteaser'),
dict(id='nnav-oly'),
dict(id='readcomment')]
feeds = [ (u'Sueddeutsche', u'http://www.sueddeutsche.de/app/service/rss/alles/rss.xml') ]
dict(id='bookmarklist1'),
dict(id='bookmarklist2'),
dict(id='navlist-personnames'),
dict(id='artikelfoot'),
dict(id='nnav-bgheader'),
dict(id='rechteSpalte'),
dict(id=''),
dict(name='td', attrs={'class':'artikelDruckenCenter'})]
#feeds = [(u'Topthemen', u'http://suche.sueddeutsche.de/query/politik/-docdatetime/drilldown/%C2%A7documenttype%3AArtikel?output=rss')]
feeds = [(u'Wissen', u'http://suche.sueddeutsche.de/query/wissen/nav/%C2%A7ressort%3AWissen/sort/-docdatetime?output=rss'),
(u'Politik', u'http://suche.sueddeutsche.de/query/politik/nav/%C2%A7ressort%3APolitik/sort/-docdatetime?output=rss'),
(u'Wirtschaft', u'http://suche.sueddeutsche.de/query/wirtschaft/nav/%C2%A7ressort%3AWirtschaft/sort/-docdatetime?output=rss'),
(u'Finanzen', u'http://suche.sueddeutsche.de/query/finanzen/nav/%C2%A7ressort%3AGeld/sort/-docdatetime?output=rss'),
(u'Kultur', u'http://suche.sueddeutsche.de/query/kultur/nav/%C2%A7ressort%3AKultur/sort/-docdatetime?output=rss'),
(u'Sport', u'http://suche.sueddeutsche.de/query/sport/nav/%C2%A7ressort%3ASport/sort/-docdatetime?output=rss'),
(u'Bayern', u'http://suche.sueddeutsche.de/query/bayern/nav/%C2%A7ressort%3ABayern/sort/-docdatetime?output=rss'),
(u'Panorama', u'http://suche.sueddeutsche.de/query/panorama/sort/-docdatetime?output=rss'),
(u'Leben&Stil', u'http://suche.sueddeutsche.de/query/stil/nav/%C2%A7ressort%3A%22Leben%20%26%20Stil%22/sort/-docdatetime?output=rss'),
(u'Gesundheit', u'http://suche.sueddeutsche.de/query/gesundheit/nav/%C2%A7ressort%3AGesundheit/sort/-docdatetime?output=rss'),
(u'Auto&Reise', u'http://suche.sueddeutsche.de/query/automobil/nav/%C2%A7ressort%3A%22Auto%20%26%20Mobil%22/sort/-docdatetime?output=rss'),
(u'Computer', u'http://suche.sueddeutsche.de/query/computer/nav/%C2%A7ressort%3AComputer/sort/-docdatetime?output=rss'),
(u'Job&Karriere', u'http://suche.sueddeutsche.de/query/job/nav/%C2%A7ressort%3A%22Job%20%26%20Karriere%22/sort/-docdatetime?output=rss'),
(u'Reise', u'http://suche.sueddeutsche.de/query/reise/nav/%C2%A7ressort%3AReise/sort/-docdatetime?output=rss')]
def print_version(self, url):
return url.replace('/text/', '/text/print.html')

29
src/calibre/web/feeds/recipes/recipe_vijesti.py
View File

@ -9,6 +9,7 @@ vijesti.me
import re
from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup, Tag
class Vijesti(BasicNewsRecipe):
title = 'Vijesti'
@ -16,8 +17,8 @@ class Vijesti(BasicNewsRecipe):
description = 'News from Montenegro'
publisher = 'Daily Press Vijesti'
category = 'news, politics, Montenegro'
oldest_article = 1
max_articles_per_feed = 100
oldest_article = 2
max_articles_per_feed = 150
no_stylesheets = True
remove_javascript = True
encoding = 'cp1250'
@ -25,7 +26,8 @@ class Vijesti(BasicNewsRecipe):
remove_javascript = True
use_embedded_content = False
language = _('Serbian')
extra_css = '@font-face {font-family: "serif1";src:url(res:///opt/sony/ebook/FONT/tt0011m_.ttf)} @font-face {font-family: "sans1";src:url(res:///opt/sony/ebook/FONT/tt0003m_.ttf)} body{text-align: justify; font-family: serif1, serif} .article_description{font-family: sans1, sans-serif}'
lang ='sr-Latn-Me'
extra_css = '@font-face {font-family: "serif1";src:url(res:///opt/sony/ebook/FONT/tt0011m_.ttf)} @font-face {font-family: "sans1";src:url(res:///opt/sony/ebook/FONT/tt0003m_.ttf)} body{font-family: serif1, serif} .article_description{font-family: sans1, sans-serif}'
html2lrf_options = [
'--comment', description
@ -44,12 +46,15 @@ class Vijesti(BasicNewsRecipe):
feeds = [(u'Sve vijesti', u'http://www.vijesti.me/rss.php' )]
def preprocess_html(self, soup):
soup.html['xml:lang'] = 'sr-Latn-ME'
soup.html['lang'] = 'sr-Latn-ME'
mtag = '<meta http-equiv="Content-Language" content="sr-Latn-ME"/>'
soup.head.insert(0,mtag)
for item in soup.findAll('img'):
if item.has_key('align'):
del item['align']
item.insert(0,'<br /><br />')
return soup
soup.html['xml:lang'] = self.lang
soup.html['lang'] = self.lang
mlang = Tag(soup,'meta',[("http-equiv","Content-Language"),("content",self.lang)])
mcharset = Tag(soup,'meta',[("http-equiv","Content-Type"),("content","text/html; charset=UTF-8")])
soup.head.insert(0,mlang)
soup.head.insert(1,mcharset)
return self.adeify_images(soup)
def get_article_url(self, article):
raw = article.get('link', None)
return raw.replace('.cg.yu','.me')

10
src/calibre/web/feeds/recipes/recipe_vreme.py
View File

@ -9,6 +9,7 @@ vreme.com
import re
from calibre import strftime
from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup, Tag
class Vreme(BasicNewsRecipe):
title = 'Vreme'
@ -27,7 +28,7 @@ class Vreme(BasicNewsRecipe):
language = _('Serbian')
lang = 'sr-Latn-RS'
direction = 'ltr'
extra_css = '@font-face {font-family: "serif1";src:url(res:///opt/sony/ebook/FONT/tt0011m_.ttf)} body{text-align: justify; font-family: serif1, serif} .article_description{font-family: serif1, serif}'
extra_css = '@font-face {font-family: "serif1";src:url(res:///opt/sony/ebook/FONT/tt0011m_.ttf)} body{font-family: serif1, serif} .article_description{font-family: serif1, serif} .heading1{font-size: x-large; font-weight: bold} .heading2{font-size: large; font-weight: bold} .toc-heading{font-size: small}'
html2lrf_options = [
'--comment' , description
@ -89,9 +90,10 @@ class Vreme(BasicNewsRecipe):
del item['size']
soup.html['lang'] = self.lang
soup.html['dir' ] = self.direction
mtag = '<meta http-equiv="Content-Language" content="' + self.lang + '"/>'
mtag += '\n<meta http-equiv="Content-Type" content="text/html; charset=' + self.encoding + '"/>'
soup.head.insert(0,mtag)
mlang = Tag(soup,'meta',[("http-equiv","Content-Language"),("content",self.lang)])
mcharset = Tag(soup,'meta',[("http-equiv","Content-Type"),("content","text/html; charset=UTF-8")])
soup.head.insert(0,mlang)
soup.head.insert(1,mcharset)
return soup
def get_cover_url(self):

3
src/calibre/web/feeds/recipes/recipe_winsupersite.py
View File

@ -13,8 +13,7 @@ class Winsupersite(BasicNewsRecipe):
no_stylesheets = True
use_embedded_content = False
remove_javascript = True
html2lrf_options = ['--ignore-tables']
html2epub_options = 'linearize_tables = True'
conversion_options = {'linearize_tables' : True}
remove_tags_before = dict(name='h1')
preprocess_regexps = [
(re.compile(r'<p>--Paul Thurrott.*</body>', re.DOTALL|re.IGNORECASE),

9
src/cssutils/__init__.py
View File

@ -70,11 +70,11 @@ Usage may be::
__all__ = ['css', 'stylesheets', 'CSSParser', 'CSSSerializer']
__docformat__ = 'restructuredtext'
__author__ = 'Christof Hoeke with contributions by Walter Doerwald'
__date__ = '$LastChangedDate:: 2009-02-16 12:05:02 -0800 #$:'
__date__ = '$LastChangedDate:: 2009-05-09 13:59:54 -0700 #$:'
VERSION = '0.9.6a1'
VERSION = '0.9.6a5'
__version__ = '%s $Id: __init__.py 1669 2009-02-16 20:05:02Z cthedot $' % VERSION
__version__ = '%s $Id: __init__.py 1747 2009-05-09 20:59:54Z cthedot $' % VERSION
import codec
import xml.dom
@ -92,6 +92,9 @@ from parse import CSSParser
from serialize import CSSSerializer
ser = CSSSerializer()
from profiles import Profiles
profile = Profiles(log=log)
# used by Selector defining namespace prefix '*'
_ANYNS = -1

13
src/cssutils/css/cssmediarule.py
View File

@ -1,7 +1,7 @@
"""CSSMediaRule implements DOM Level 2 CSS CSSMediaRule."""
__all__ = ['CSSMediaRule']
__docformat__ = 'restructuredtext'
__version__ = '$Id: cssmediarule.py 1641 2009-01-13 21:05:37Z cthedot $'
__version__ = '$Id: cssmediarule.py 1743 2009-05-09 20:33:15Z cthedot $'
import cssrule
import cssutils
@ -131,8 +131,15 @@ class CSSMediaRule(cssrule.CSSRule):
mediaendonly=True,
separateEnd=True)
nonetoken = self._nexttoken(tokenizer, None)
if (u'}' != self._tokenvalue(braceOrEOF) and
'EOF' != self._type(braceOrEOF)):
if 'EOF' == self._type(braceOrEOF):
# HACK!!!
# TODO: Not complete, add EOF to rule and } to @media
cssrulestokens.append(braceOrEOF)
braceOrEOF = ('CHAR', '}', 0, 0)
self._log.debug(u'CSSMediaRule: Incomplete, adding "}".',
token=braceOrEOF, neverraise=True)
if u'}' != self._tokenvalue(braceOrEOF):
self._log.error(u'CSSMediaRule: No "}" found.',
token=braceOrEOF)
elif nonetoken:

4
src/cssutils/css/cssstyledeclaration.py
View File

@ -51,7 +51,7 @@ TODO:
"""
__all__ = ['CSSStyleDeclaration', 'Property']
__docformat__ = 'restructuredtext'
__version__ = '$Id: cssstyledeclaration.py 1658 2009-02-07 18:24:40Z cthedot $'
__version__ = '$Id: cssstyledeclaration.py 1710 2009-04-18 15:46:20Z cthedot $'
from cssproperties import CSS2Properties
from property import Property
@ -613,7 +613,7 @@ class CSSStyleDeclaration(CSS2Properties, cssutils.util.Base2):
except IndexError:
return u''
length = property(lambda self: len(self.__nnames()),
length = property(lambda self: len(list(self.__nnames())),
doc="(DOM) The number of distinct properties that have been explicitly "
"in this declaration block. The range of valid indices is 0 to "
"length-1 inclusive. These are properties with a different ``name`` "

9
src/cssutils/css/cssvalue.py
View File

@ -7,10 +7,9 @@
"""
__all__ = ['CSSValue', 'CSSPrimitiveValue', 'CSSValueList', 'RGBColor']
__docformat__ = 'restructuredtext'
__version__ = '$Id: cssvalue.py 1638 2009-01-13 20:39:33Z cthedot $'
__version__ = '$Id: cssvalue.py 1684 2009-03-01 18:26:21Z cthedot $'
from cssutils.prodparser import *
from cssutils.profiles import profiles
import cssutils
import cssutils.helper
import re
@ -121,7 +120,8 @@ class CSSValue(cssutils.util._NewBase):
# special case IE only expression
Prod(name='expression',
match=lambda t, v: t == self._prods.FUNCTION and
cssutils.helper.normalize(v) == 'expression(',
cssutils.helper.normalize(v) in (u'expression(',
u'alpha('),
nextSor=nextSor,
toSeq=lambda t, tokens: (ExpressionValue.name,
ExpressionValue(cssutils.helper.pushtoken(t,
@ -968,7 +968,8 @@ class RGBColor(CSSPrimitiveValue):
class ExpressionValue(CSSFunction):
"""Special IE only CSSFunction which may contain *anything*."""
"""Special IE only CSSFunction which may contain *anything*.
Used for expressions and ``alpha(opacity=100)`` currently"""
name = u'Expression (IE only)'
def _productiondefinition(self):

116
src/cssutils/css/property.py
View File

@ -1,10 +1,9 @@
"""Property is a single CSS property in a CSSStyleDeclaration."""
__all__ = ['Property']
__docformat__ = 'restructuredtext'
__version__ = '$Id: property.py 1664 2009-02-07 22:47:09Z cthedot $'
__version__ = '$Id: property.py 1685 2009-03-01 18:26:48Z cthedot $'
from cssutils.helper import Deprecated
from cssutils.profiles import profiles
from cssvalue import CSSValue
import cssutils
import xml.dom
@ -67,6 +66,7 @@ class Property(cssutils.util.Base):
self._mediaQuery = _mediaQuery
self._parent = _parent
self.__nametoken = None
self._name = u''
self._literalname = u''
if name:
@ -193,6 +193,7 @@ class Property(cssutils.util.Base):
# define a token for error logging
if isinstance(name, list):
token = name[0]
self.__nametoken = token
else:
token = None
@ -208,9 +209,9 @@ class Property(cssutils.util.Base):
self.seqs[0] = newseq
# # validate
if self._name not in profiles.knownnames:
if self._name not in cssutils.profile.knownNames:
# self.valid = False
self._log.warn(u'Property: Unknown Property.',
self._log.warn(u'Property: Unknown Property name.',
token=token, neverraise=True)
else:
pass
@ -362,42 +363,101 @@ class Property(cssutils.util.Base):
literalpriority = property(lambda self: self._literalpriority,
doc="Readonly literal (not normalized) priority of this property")
def validate(self, profile=None):
"""Validate value against `profile`.
def validate(self, profiles=None):
"""Validate value against `profiles`.
:param profile:
A profile name used for validating. If no `profile` is given
``Property.profiles
:param profiles:
A list of profile names used for validating. If no `profiles`
is given ``cssutils.profile.defaultProfiles`` is used
For each of the following cases a message is reported:
- INVALID (so the property is known but not valid)
``ERROR Property: Invalid value for "{PROFILE-1[/PROFILE-2...]"
property: ...``
- VALID but not in given profiles or defaultProfiles
``WARNING Property: Not valid for profile "{PROFILE-X}" but valid
"{PROFILE-Y}" property: ...``
- VALID in current profile
``DEBUG Found valid "{PROFILE-1[/PROFILE-2...]" property...``
- UNKNOWN property
``WARNING Unknown Property name...`` is issued
so for example::
cssutils.log.setLevel(logging.DEBUG)
parser = cssutils.CSSParser()
s = parser.parseString('''body {
unknown-property: x;
color: 4;
color: rgba(1,2,3,4);
color: red
}''')
# Log output:
WARNING Property: Unknown Property name. [2:9: unknown-property]
ERROR Property: Invalid value for "CSS Color Module Level 3/CSS Level 2.1" property: 4 [3:9: color]
DEBUG Property: Found valid "CSS Color Module Level 3" value: rgba(1, 2, 3, 4) [4:9: color]
DEBUG Property: Found valid "CSS Level 2.1" value: red [5:9: color]
and when setting an explicit default profile::
cssutils.profile.defaultProfiles = cssutils.profile.CSS_LEVEL_2
s = parser.parseString('''body {
unknown-property: x;
color: 4;
color: rgba(1,2,3,4);
color: red
}''')
# Log output:
WARNING Property: Unknown Property name. [2:9: unknown-property]
ERROR Property: Invalid value for "CSS Color Module Level 3/CSS Level 2.1" property: 4 [3:9: color]
WARNING Property: Not valid for profile "CSS Level 2.1" but valid "CSS Color Module Level 3" value: rgba(1, 2, 3, 4) [4:9: color]
DEBUG Property: Found valid "CSS Level 2.1" value: red [5:9: color]
"""
valid = False
if self.name and self.value:
if profile is None:
usedprofile = cssutils.profiles.defaultprofile
else:
usedprofile = profile
if self.name in profiles.knownnames:
valid, validprofiles = profiles.validateWithProfile(self.name,
if self.name in cssutils.profile.knownNames:
# add valid, matching, validprofiles...
valid, matching, validprofiles = \
cssutils.profile.validateWithProfile(self.name,
self.value,
usedprofile)
profiles)
if not valid:
self._log.error(u'Property: Invalid value for "%s" property: %s: %s'
% (u'/'.join(validprofiles),
self.name,
self.value),
neverraise=True)
elif valid and (usedprofile and usedprofile not in validprofiles):
self._log.warn(u'Property: Not valid for profile "%s": %s: %s'
% (usedprofile, self.name, self.value),
self._log.error(u'Property: Invalid value for '
u'"%s" property: %s'
% (u'/'.join(validprofiles), self.value),
token=self.__nametoken,
neverraise=True)
if valid:
self._log.info(u'Property: Found valid "%s" property: %s: %s'
% (u'/'.join(validprofiles),
self.name,
# TODO: remove logic to profiles!
elif valid and not matching:#(profiles and profiles not in validprofiles):
if not profiles:
notvalidprofiles = u'/'.join(cssutils.profile.defaultProfiles)
else:
notvalidprofiles = profiles
self._log.warn(u'Property: Not valid for profile "%s" '
u'but valid "%s" value: %s '
% (notvalidprofiles, u'/'.join(validprofiles),
self.value),
token = self.__nametoken,
neverraise=True)
valid = False
elif valid:
self._log.debug(u'Property: Found valid "%s" value: %s'
% (u'/'.join(validprofiles), self.value),
token = self.__nametoken,
neverraise=True)
if self._priority not in (u'', u'important'):

13
src/cssutils/css/selector.py
View File

@ -7,7 +7,7 @@ TODO
"""
__all__ = ['Selector']
__docformat__ = 'restructuredtext'
__version__ = '$Id: selector.py 1638 2009-01-13 20:39:33Z cthedot $'
__version__ = '$Id: selector.py 1741 2009-05-09 18:20:20Z cthedot $'
from cssutils.util import _SimpleNamespaces
import cssutils
@ -701,6 +701,14 @@ class Selector(cssutils.util.Base2):
u'Selector: Unexpected negation.', token=token)
return expected
def _atkeyword(expected, seq, token, tokenizer=None):
"invalidates selector"
new['wellformed'] = False
self._log.error(
u'Selector: Unexpected ATKEYWORD.', token=token)
return expected
# expected: only|not or mediatype, mediatype, feature, and
newseq = self._tempSeq()
@ -727,7 +735,8 @@ class Selector(cssutils.util.Base2):
'INCLUDES': _attcombinator,
'S': _S,
'COMMENT': _COMMENT})
'COMMENT': _COMMENT,
'ATKEYWORD': _atkeyword})
wellformed = wellformed and new['wellformed']
# post condition

6
src/cssutils/cssproductions.py
View File

@ -12,14 +12,14 @@ open issues
"""
__all__ = ['CSSProductions', 'MACROS', 'PRODUCTIONS']
__docformat__ = 'restructuredtext'
__version__ = '$Id: cssproductions.py 1537 2008-12-03 14:37:10Z cthedot $'
__version__ = '$Id: cssproductions.py 1738 2009-05-02 13:03:28Z cthedot $'
# a complete list of css3 macros
MACROS = {
'nonascii': r'[^\0-\177]',
'unicode': r'\\[0-9a-f]{1,6}(?:{nl}|{s})?',
# 'escape': r'{unicode}|\\[ -~\200-\4177777]',
'escape': r'{unicode}|\\[ -~\200-\777]',
#'escape': r'{unicode}|\\[ -~\200-\777]',
'escape': r'{unicode}|\\[^\n\r\f0-9a-f]',
'nmstart': r'[_a-zA-Z]|{nonascii}|{escape}',
'nmchar': r'[-_a-zA-Z0-9]|{nonascii}|{escape}',
'string1': r'"([^\n\r\f\\"]|\\{nl}|{escape})*"',

6
src/cssutils/errorhandler.py
View File

@ -16,7 +16,7 @@ log
"""
__all__ = ['ErrorHandler']
__docformat__ = 'restructuredtext'
__version__ = '$Id: errorhandler.py 1560 2008-12-14 16:13:16Z cthedot $'
__version__ = '$Id: errorhandler.py 1728 2009-05-01 20:35:25Z cthedot $'
from helper import Deprecated
import logging
@ -89,8 +89,8 @@ class _ErrorHandler(object):
elif issubclass(error, xml.dom.DOMException):
error.line = line
error.col = col
raise error(msg)
else:
# raise error(msg, line, col)
# else:
raise error(msg)
else:
self._logcall(msg)

5
src/cssutils/helper.py
View File

@ -68,6 +68,9 @@ def string(value):
u'\f', u'\\c ').replace(
u'"', u'\\"')
if value.endswith(u'\\'):
value = value[:-1] + u'\\\\'
return u'"%s"' % value
def stringvalue(string):
@ -77,7 +80,7 @@ def stringvalue(string):
``'a \'string'`` => ``a 'string``
"""
return string.replace('\\'+string[0], string[0])[1:-1]
return string.replace(u'\\'+string[0], string[0])[1:-1]
_match_forbidden_in_uri = re.compile(ur'''.*?[\(\)\s\;,'"]''', re.U).match
def uri(value):

634
src/cssutils/profiles.py
View File

@ -1,41 +1,340 @@
"""CSS profiles.
css2 is based on cssvalues
contributed by Kevin D. Smith, thanks!
"cssvalues" is used as a property validator.
it is an importable object that contains a dictionary of compiled regular
expressions. The keys of this dictionary are all of the valid CSS property
names. The values are compiled regular expressions that can be used to
validate the values for that property. (Actually, the values are references
to the 'match' method of a compiled regular expression, so that they are
simply called like functions.)
Profiles is based on code by Kevin D. Smith, orginally used as cssvalues,
thanks!
"""
__all__ = ['profiles']
__all__ = ['Profiles']
__docformat__ = 'restructuredtext'
__version__ = '$Id: cssproperties.py 1116 2008-03-05 13:52:23Z cthedot $'
import cssutils
import re
class NoSuchProfileException(Exception):
"""Raised if no profile with given name is found"""
pass
class Profiles(object):
"""
All profiles used for validation. ``cssutils.profile`` is a
preset object of this class and used by all properties for validation.
Predefined profiles are (use
:meth:`~cssutils.profiles.Profiles.propertiesByProfile` to
get a list of defined properties):
:attr:`~cssutils.profiles.Profiles.CSS_LEVEL_2`
Properties defined by CSS2.1
:attr:`~cssutils.profiles.Profiles.CSS3_COLOR`
CSS 3 color properties
:attr:`~cssutils.profiles.Profiles.CSS3_BOX`
Currently overflow related properties only
:attr:`~cssutils.profiles.Profiles.CSS3_PAGED_MEDIA`
As defined at http://www.w3.org/TR/css3-page/ (at 090307)
Predefined macros are:
:attr:`~cssutils.profiles.Profiles._TOKEN_MACROS`
Macros containing the token values as defined to CSS2
:attr:`~cssutils.profiles.Profiles._MACROS`
Additional general macros.
If you want to redefine any of these macros do this in your custom
macros.
"""
CSS_LEVEL_2 = 'CSS Level 2.1'
CSS3_COLOR = CSS_COLOR_LEVEL_3 = 'CSS Color Module Level 3'
CSS3_BOX = CSS_BOX_LEVEL_3 = 'CSS Box Module Level 3'
CSS3_PAGED_MEDIA = 'CSS3 Paged Media Module'
_TOKEN_MACROS = {
'ident': r'[-]?{nmstart}{nmchar}*',
'name': r'{nmchar}+',
'nmstart': r'[_a-z]|{nonascii}|{escape}',
'nonascii': r'[^\0-\177]',
'unicode': r'\\[0-9a-f]{1,6}(\r\n|[ \n\r\t\f])?',
'escape': r'{unicode}|\\[ -~\200-\777]',
# 'escape': r'{unicode}|\\[ -~\200-\4177777]',
'int': r'[-]?\d+',
'nmchar': r'[\w-]|{nonascii}|{escape}',
'num': r'[-]?\d+|[-]?\d*\.\d+',
'number': r'{num}',
'string': r'{string1}|{string2}',
'string1': r'"(\\\"|[^\"])*"',
'uri': r'url\({w}({string}|(\\\)|[^\)])+){w}\)',
'string2': r"'(\\\'|[^\'])*'",
'nl': r'\n|\r\n|\r|\f',
'w': r'\s*',
}
_MACROS = {
'hexcolor': r'#[0-9a-f]{3}|#[0-9a-f]{6}',
'rgbcolor': r'rgb\({w}{int}{w},{w}{int}{w},{w}{int}{w}\)|rgb\({w}{num}%{w},{w}{num}%{w},{w}{num}%{w}\)',
'namedcolor': r'(transparent|orange|maroon|red|orange|yellow|olive|purple|fuchsia|white|lime|green|navy|blue|aqua|teal|black|silver|gray)',
'uicolor': r'(ActiveBorder|ActiveCaption|AppWorkspace|Background|ButtonFace|ButtonHighlight|ButtonShadow|ButtonText|CaptionText|GrayText|Highlight|HighlightText|InactiveBorder|InactiveCaption|InactiveCaptionText|InfoBackground|InfoText|Menu|MenuText|Scrollbar|ThreeDDarkShadow|ThreeDFace|ThreeDHighlight|ThreeDLightShadow|ThreeDShadow|Window|WindowFrame|WindowText)',
'color': r'{namedcolor}|{hexcolor}|{rgbcolor}|{uicolor}',
#'color': r'(maroon|red|orange|yellow|olive|purple|fuchsia|white|lime|green|navy|blue|aqua|teal|black|silver|gray|ActiveBorder|ActiveCaption|AppWorkspace|Background|ButtonFace|ButtonHighlight|ButtonShadow|ButtonText|CaptionText|GrayText|Highlight|HighlightText|InactiveBorder|InactiveCaption|InactiveCaptionText|InfoBackground|InfoText|Menu|MenuText|Scrollbar|ThreeDDarkShadow|ThreeDFace|ThreeDHighlight|ThreeDLightShadow|ThreeDShadow|Window|WindowFrame|WindowText)|#[0-9a-f]{3}|#[0-9a-f]{6}|rgb\({w}{int}{w},{w}{int}{w},{w}{int}{w}\)|rgb\({w}{num}%{w},{w}{num}%{w},{w}{num}%{w}\)',
'integer': r'{int}',
'length': r'0|{num}(em|ex|px|in|cm|mm|pt|pc)',
'angle': r'0|{num}(deg|grad|rad)',
'time': r'0|{num}m?s',
'frequency': r'0|{num}k?Hz',
'percentage': r'{num}%',
}
def __init__(self, log=None):
"""A few known profiles are predefined."""
self._log = log
self._profileNames = [] # to keep order, REFACTOR!
self._profiles = {}
self._defaultProfiles = None
self.addProfile(self.CSS_LEVEL_2,
properties[self.CSS_LEVEL_2],
macros[self.CSS_LEVEL_2])
self.addProfile(self.CSS3_BOX,
properties[self.CSS3_BOX],
macros[self.CSS3_BOX])
self.addProfile(self.CSS3_COLOR,
properties[self.CSS3_COLOR],
macros[self.CSS3_COLOR])
self.addProfile(self.CSS3_PAGED_MEDIA,
properties[self.CSS3_PAGED_MEDIA],
macros[self.CSS3_PAGED_MEDIA])
self.__update_knownNames()
def _expand_macros(self, dictionary, macros):
"""Expand macros in token dictionary"""
def macro_value(m):
return '(?:%s)' % macros[m.groupdict()['macro']]
for key, value in dictionary.items():
if not hasattr(value, '__call__'):
while re.search(r'{[a-z][a-z0-9-]*}', value):
value = re.sub(r'{(?P<macro>[a-z][a-z0-9-]*)}',
macro_value, value)
dictionary[key] = value
return dictionary
def _compile_regexes(self, dictionary):
"""Compile all regular expressions into callable objects"""
for key, value in dictionary.items():
if not hasattr(value, '__call__'):
value = re.compile('^(?:%s)$' % value, re.I).match
dictionary[key] = value
return dictionary
def __update_knownNames(self):
self._knownNames = []
for properties in self._profiles.values():
self._knownNames.extend(properties.keys())
def _getDefaultProfiles(self):
"If not explicitly set same as Profiles.profiles but in reverse order."
if not self._defaultProfiles:
return self.profiles#list(reversed(self.profiles))
else:
return self._defaultProfiles
def _setDefaultProfiles(self, profiles):
"profiles may be a single or a list of profile names"
if isinstance(profiles, basestring):
self._defaultProfiles = (profiles,)
else:
self._defaultProfiles = profiles
defaultProfiles = property(_getDefaultProfiles,
_setDefaultProfiles,
doc=u"Names of profiles to use for validation."
u"To use e.g. the CSS2 profile set "
u"``cssutils.profile.defaultProfiles = "
u"cssutils.profile.CSS_LEVEL_2``")
profiles = property(lambda self: self._profileNames,
doc=u'Names of all profiles in order as defined.')
knownNames = property(lambda self: self._knownNames,
doc="All known property names of all profiles.")
def addProfile(self, profile, properties, macros=None):
"""Add a new profile with name `profile` (e.g. 'CSS level 2')
and the given `properties`.
:param profile:
the new `profile`'s name
:param properties:
a dictionary of ``{ property-name: propery-value }`` items where
property-value is a regex which may use macros defined in given
``macros`` or the standard macros Profiles.tokens and
Profiles.generalvalues.
``propery-value`` may also be a function which takes a single
argument which is the value to validate and which should return
True or False.
Any exceptions which may be raised during this custom validation
are reported or raised as all other cssutils exceptions depending
on cssutils.log.raiseExceptions which e.g during parsing normally
is False so the exceptions would be logged only.
:param macros:
may be used in the given properties definitions. There are some
predefined basic macros which may always be used in
:attr:`Profiles._TOKEN_MACROS` and :attr:`Profiles._MACROS`.
"""
if not macros:
macros = {}
m = Profiles._TOKEN_MACROS.copy()
m.update(Profiles._MACROS)
m.update(macros)
properties = self._expand_macros(properties, m)
self._profileNames.append(profile)
self._profiles[profile] = self._compile_regexes(properties)
self.__update_knownNames()
def removeProfile(self, profile=None, all=False):
"""Remove `profile` or remove `all` profiles.
:param profile:
profile name to remove
:param all:
if ``True`` removes all profiles to start with a clean state
:exceptions:
- :exc:`cssutils.profiles.NoSuchProfileException`:
If given `profile` cannot be found.
"""
if all:
self._profiles.clear()
del self._profileNames[:]
else:
try:
del self._profiles[profile]
del self._profileNames[self._profileNames.index(profile)]
except KeyError:
raise NoSuchProfileException(u'No profile %r.' % profile)
self.__update_knownNames()
def propertiesByProfile(self, profiles=None):
"""Generator: Yield property names, if no `profiles` is given all
profile's properties are used.
:param profiles:
a single profile name or a list of names.
"""
if not profiles:
profiles = self.profiles
elif isinstance(profiles, basestring):
profiles = (profiles, )
try:
for profile in sorted(profiles):
for name in sorted(self._profiles[profile].keys()):
yield name
except KeyError, e:
raise NoSuchProfileException(e)
def validate(self, name, value):
"""Check if `value` is valid for given property `name` using **any**
profile.
:param name:
a property name
:param value:
a CSS value (string)
:returns:
if the `value` is valid for the given property `name` in any
profile
"""
for profile in self.profiles:
if name in self._profiles[profile]:
try:
# custom validation errors are caught
r = bool(self._profiles[profile][name](value))
except Exception, e:
self._log.error(e, error=Exception)
return False
if r:
return r
return False
def validateWithProfile(self, name, value, profiles=None):
"""Check if `value` is valid for given property `name` returning
``(valid, profile)``.
:param name:
a property name
:param value:
a CSS value (string)
:param profiles:
internal parameter used by Property.validate only
:returns:
``valid, matching, profiles`` where ``valid`` is if the `value`
is valid for the given property `name` in any profile,
``matching==True`` if it is valid in the given `profiles`
and ``profiles`` the profile names for which the value is valid
(or ``[]`` if not valid at all)
Example::
>>> cssutils.profile.defaultProfiles = cssutils.profile.CSS_LEVEL_2
>>> print cssutils.profile.validateWithProfile('color', 'rgba(1,1,1,1)')
(True, False, Profiles.CSS3_COLOR)
"""
if name not in self.knownNames:
return False, False, []
else:
if not profiles:
profiles = self.defaultProfiles
elif isinstance(profiles, basestring):
profiles = (profiles, )
for profilename in profiles:
# check given profiles
if name in self._profiles[profilename]:
validate = self._profiles[profilename][name]
try:
if validate(value):
return True, True, [profilename]
except Exception, e:
self._log.error(e, error=Exception)
for profilename in (p for p in self._profileNames
if p not in profiles):
# check remaining profiles as well
if name in self._profiles[profilename]:
validate = self._profiles[profilename][name]
try:
if validate(value):
return True, False, [profilename]
except Exception, e:
self._log.error(e, error=Exception)
names = []
for profilename, properties in self._profiles.items():
# return profile to which name belongs
if name in properties.keys():
names.append(profilename)
names.sort()
return False, False, names
properties = {}
macros = {}
"""
Define some regular expression fragments that will be used as
macros within the CSS property value regular expressions.
"""
css2macros = {
macros[Profiles.CSS_LEVEL_2] = {
'border-style': 'none|hidden|dotted|dashed|solid|double|groove|ridge|inset|outset',
'border-color': '{color}',
'border-width': '{length}|thin|medium|thick',
'background-color': r'{color}|transparent|inherit',
'background-image': r'{uri}|none|inherit',
'background-position': r'({percentage}|{length})(\s*({percentage}|{length}))?|((top|center|bottom)\s*(left|center|right))|((left|center|right)\s*(top|center|bottom))|inherit',
#'background-position': r'({percentage}|{length})(\s*({percentage}|{length}))?|((top|center|bottom)\s*(left|center|right)?)|((left|center|right)\s*(top|center|bottom)?)|inherit',
'background-position': r'({percentage}|{length}|left|center|right)(\s*({percentage}|{length}|top|center|bottom))?|((top|center|bottom)\s*(left|center|right)?)|((left|center|right)\s*(top|center|bottom)?)|inherit',
'background-repeat': r'repeat|repeat-x|repeat-y|no-repeat|inherit',
'background-attachment': r'scroll|fixed|inherit',
'shape': r'rect\(({w}({length}|auto}){w},){3}{w}({length}|auto){w}\)',
'counter': r'counter\({w}{identifier}{w}(?:,{w}{list-style-type}{w})?\)',
'identifier': r'{ident}',
@ -72,7 +371,7 @@ css2macros = {
"""
Define the regular expressions for validation all CSS values
"""
properties['css2'] = {
properties[Profiles.CSS_LEVEL_2] = {
'azimuth': r'{angle}|(behind\s+)?(left-side|far-left|left|center-left|center|center-right|right|far-right|right-side)(\s+behind)?|behind|leftwards|rightwards|inherit',
'background-attachment': r'{background-attachment}',
'background-color': r'{background-color}',
@ -108,7 +407,7 @@ properties['css2'] = {
'clear': r'none|left|right|both|inherit',
'clip': r'{shape}|auto|inherit',
'color': r'{color}|inherit',
'content': r'normal|{content}(\s+{content})*|inherit',
'content': r'none|normal|{content}(\s+{content})*|inherit',
'counter-increment': r'({identifier}(\s+{integer})?)(\s+({identifier}(\s+{integer})))*|none|inherit',
'counter-reset': r'({identifier}(\s+{integer})?)(\s+({identifier}(\s+{integer})))*|none|inherit',
'cue-after': r'{uri}|none|inherit',
@ -191,288 +490,47 @@ properties['css2'] = {
'z-index': r'auto|{integer}|inherit',
}
# CSS Box Module Level 3
macros[Profiles.CSS3_BOX] = {
'overflow': macros[Profiles.CSS_LEVEL_2]['overflow']
}
properties[Profiles.CSS3_BOX] = {
'overflow': '{overflow}\s?{overflow}?|inherit',
'overflow-x': '{overflow}|inherit',
'overflow-y': '{overflow}|inherit'
}
# CSS Color Module Level 3
css3colormacros = {
macros[Profiles.CSS3_COLOR] = {
# orange and transparent in CSS 2.1
'namedcolor': r'(currentcolor|transparent|orange|black|green|silver|lime|gray|olive|white|yellow|maroon|navy|red|blue|purple|teal|fuchsia|aqua)',
'namedcolor': r'(currentcolor|transparent|aqua|black|blue|fuchsia|gray|green|lime|maroon|navy|olive|orange|purple|red|silver|teal|white|yellow)',
# orange?
'rgbacolor': r'rgba\({w}{int}{w},{w}{int}{w},{w}{int}{w},{w}{int}{w}\)|rgba\({w}{num}%{w},{w}{num}%{w},{w}{num}%{w},{w}{num}{w}\)',
'hslcolor': r'hsl\({w}{int}{w},{w}{num}%{w},{w}{num}%{w}\)|hsla\({w}{int}{w},{w}{num}%{w},{w}{num}%{w},{w}{num}{w}\)',
'x11color': r'aliceblue|antiquewhite|aqua|aquamarine|azure|beige|bisque|black|blanchedalmond|blue|blueviolet|brown|burlywood|cadetblue|chartreuse|chocolate|coral|cornflowerblue|cornsilk|crimson|cyan|darkblue|darkcyan|darkgoldenrod|darkgray|darkgreen|darkgrey|darkkhaki|darkmagenta|darkolivegreen|darkorange|darkorchid|darkred|darksalmon|darkseagreen|darkslateblue|darkslategray|darkslategrey|darkturquoise|darkviolet|deeppink|deepskyblue|dimgray|dimgrey|dodgerblue|firebrick|floralwhite|forestgreen|fuchsia|gainsboro|ghostwhite|gold|goldenrod|gray|green|greenyellow|grey|honeydew|hotpink|indianred|indigo|ivory|khaki|lavender|lavenderblush|lawngreen|lemonchiffon|lightblue|lightcoral|lightcyan|lightgoldenrodyellow|lightgray|lightgreen|lightgrey|lightpink|lightsalmon|lightseagreen|lightskyblue|lightslategray|lightslategrey|lightsteelblue|lightyellow|lime|limegreen|linen|magenta|maroon|mediumaquamarine|mediumblue|mediumorchid|mediumpurple|mediumseagreen|mediumslateblue|mediumspringgreen|mediumturquoise|mediumvioletred|midnightblue|mintcream|mistyrose|moccasin|navajowhite|navy|oldlace|olive|olivedrab|orange|orangered|orchid|palegoldenrod|palegreen|paleturquoise|palevioletred|papayawhip|peachpuff|peru|pink|plum|powderblue|purple|red|rosybrown|royalblue|saddlebrown|salmon|sandybrown|seagreen|seashell|sienna|silver|skyblue|slateblue|slategray|slategrey|snow|springgreen|steelblue|tan|teal|thistle|tomato|turquoise|violet|wheat|white|whitesmoke|yellow|yellowgreen',
'uicolor': r'(ActiveBorder|ActiveCaption|AppWorkspace|Background|ButtonFace|ButtonHighlight|ButtonShadow|ButtonText|CaptionText|GrayText|Highlight|HighlightText|InactiveBorder|InactiveCaption|InactiveCaptionText|InfoBackground|InfoText|Menu|MenuText|Scrollbar|ThreeDDarkShadow|ThreeDFace|ThreeDHighlight|ThreeDLightShadow|ThreeDShadow|Window|WindowFrame|WindowText)',
}
properties['css3color'] = {
properties[Profiles.CSS3_COLOR] = {
'color': r'{namedcolor}|{hexcolor}|{rgbcolor}|{rgbacolor}|{hslcolor}|inherit',
'opacity': r'{num}|inherit'
}
# CSS Box Module Level 3
properties['css3box'] = {
'overflow': '{overflow}\s?{overflow}?',
'overflow-x': '{overflow}',
'overflow-y': '{overflow}'
# CSS3 Paged Media
macros[Profiles.CSS3_PAGED_MEDIA] = {
'pagesize': 'a5|a4|a3|b5|b4|letter|legal|ledger',
'pagebreak': 'auto|always|avoid|left|right'
}
properties[Profiles.CSS3_PAGED_MEDIA] = {
'fit': 'fill|hidden|meet|slice',
'fit-position': r'auto|(({percentage}|{length})(\s*({percentage}|{length}))?|((top|center|bottom)\s*(left|center|right)?)|((left|center|right)\s*(top|center|bottom)?))',
'image-orientation': 'auto|{angle}',
'orphans': r'{integer}|inherit',
'page': 'auto|{ident}',
'page-break-before': '{pagebreak}|inherit',
'page-break-after': '{pagebreak}|inherit',
'page-break-inside': 'auto|avoid|inherit',
'size': '({length}{w}){1,2}|auto|{pagesize}{w}(?:portrait|landscape)',
'widows': r'{integer}|inherit'
}
class NoSuchProfileException(Exception):
"""Raised if no profile with given name is found"""
pass
class Profiles(object):
"""
All profiles used for validation. ``cssutils.profiles.profiles`` is a
preset object of this class and used by all properties for validation.
Predefined profiles are (use
:meth:`~cssutils.profiles.Profiles.propertiesByProfile` to
get a list of defined properties):
:attr:`~cssutils.profiles.Profiles.Profiles.CSS_LEVEL_2`
Properties defined by CSS2.1
:attr:`~cssutils.profiles.Profiles.Profiles.CSS_COLOR_LEVEL_3`
CSS 3 color properties
:attr:`~cssutils.profiles.Profiles.Profiles.CSS_BOX_LEVEL_3`
Currently overflow related properties only
"""
CSS_LEVEL_2 = 'CSS Level 2.1'
CSS_COLOR_LEVEL_3 = 'CSS Color Module Level 3'
CSS_BOX_LEVEL_3 = 'CSS Box Module Level 3'
basicmacros = {
'ident': r'[-]?{nmstart}{nmchar}*',
'name': r'{nmchar}+',
'nmstart': r'[_a-z]|{nonascii}|{escape}',
'nonascii': r'[^\0-\177]',
'unicode': r'\\[0-9a-f]{1,6}(\r\n|[ \n\r\t\f])?',
'escape': r'{unicode}|\\[ -~\200-\777]',
# 'escape': r'{unicode}|\\[ -~\200-\4177777]',
'int': r'[-]?\d+',
'nmchar': r'[\w-]|{nonascii}|{escape}',
'num': r'[-]?\d+|[-]?\d*\.\d+',
'number': r'{num}',
'string': r'{string1}|{string2}',
'string1': r'"(\\\"|[^\"])*"',
'uri': r'url\({w}({string}|(\\\)|[^\)])+){w}\)',
'string2': r"'(\\\'|[^\'])*'",
'nl': r'\n|\r\n|\r|\f',
'w': r'\s*',
}
generalmacros = {
'hexcolor': r'#[0-9a-f]{3}|#[0-9a-f]{6}',
'rgbcolor': r'rgb\({w}{int}{w},{w}{int}{w},{w}{int}{w}\)|rgb\({w}{num}%{w},{w}{num}%{w},{w}{num}%{w}\)',
'namedcolor': r'(transparent|orange|maroon|red|orange|yellow|olive|purple|fuchsia|white|lime|green|navy|blue|aqua|teal|black|silver|gray)',
'uicolor': r'(ActiveBorder|ActiveCaption|AppWorkspace|Background|ButtonFace|ButtonHighlight|ButtonShadow|ButtonText|CaptionText|GrayText|Highlight|HighlightText|InactiveBorder|InactiveCaption|InactiveCaptionText|InfoBackground|InfoText|Menu|MenuText|Scrollbar|ThreeDDarkShadow|ThreeDFace|ThreeDHighlight|ThreeDLightShadow|ThreeDShadow|Window|WindowFrame|WindowText)',
'color': r'{namedcolor}|{hexcolor}|{rgbcolor}|{uicolor}',
#'color': r'(maroon|red|orange|yellow|olive|purple|fuchsia|white|lime|green|navy|blue|aqua|teal|black|silver|gray|ActiveBorder|ActiveCaption|AppWorkspace|Background|ButtonFace|ButtonHighlight|ButtonShadow|ButtonText|CaptionText|GrayText|Highlight|HighlightText|InactiveBorder|InactiveCaption|InactiveCaptionText|InfoBackground|InfoText|Menu|MenuText|Scrollbar|ThreeDDarkShadow|ThreeDFace|ThreeDHighlight|ThreeDLightShadow|ThreeDShadow|Window|WindowFrame|WindowText)|#[0-9a-f]{3}|#[0-9a-f]{6}|rgb\({w}{int}{w},{w}{int}{w},{w}{int}{w}\)|rgb\({w}{num}%{w},{w}{num}%{w},{w}{num}%{w}\)',
'integer': r'{int}',
'length': r'0|{num}(em|ex|px|in|cm|mm|pt|pc)',
'angle': r'0|{num}(deg|grad|rad)',
'time': r'0|{num}m?s',
'frequency': r'0|{num}k?Hz',
'percentage': r'{num}%',
}
def __init__(self):
"""A few known profiles are predefined."""
self._log = cssutils.log
self._profilenames = [] # to keep order, REFACTOR!
self._profiles = {}
self.addProfile(self.CSS_LEVEL_2, properties['css2'], css2macros)
self.addProfile(self.CSS_COLOR_LEVEL_3, properties['css3color'], css3colormacros)
self.addProfile(self.CSS_BOX_LEVEL_3, properties['css3box'])
self.__update_knownnames()
def _expand_macros(self, dictionary, macros):
"""Expand macros in token dictionary"""
def macro_value(m):
return '(?:%s)' % macros[m.groupdict()['macro']]
for key, value in dictionary.items():
if not hasattr(value, '__call__'):
while re.search(r'{[a-z][a-z0-9-]*}', value):
value = re.sub(r'{(?P<macro>[a-z][a-z0-9-]*)}',
macro_value, value)
dictionary[key] = value
return dictionary
def _compile_regexes(self, dictionary):
"""Compile all regular expressions into callable objects"""
for key, value in dictionary.items():
if not hasattr(value, '__call__'):
value = re.compile('^(?:%s)$' % value, re.I).match
dictionary[key] = value
return dictionary
def __update_knownnames(self):
self._knownnames = []
for properties in self._profiles.values():
self._knownnames.extend(properties.keys())
profiles = property(lambda self: sorted(self._profiles.keys()),
doc=u'Names of all profiles.')
knownnames = property(lambda self: self._knownnames,
doc="All known property names of all profiles.")
def addProfile(self, profile, properties, macros=None):
"""Add a new profile with name `profile` (e.g. 'CSS level 2')
and the given `properties`.
:param profile:
the new `profile`'s name
:param properties:
a dictionary of ``{ property-name: propery-value }`` items where
property-value is a regex which may use macros defined in given
``macros`` or the standard macros Profiles.tokens and
Profiles.generalvalues.
``propery-value`` may also be a function which takes a single
argument which is the value to validate and which should return
True or False.
Any exceptions which may be raised during this custom validation
are reported or raised as all other cssutils exceptions depending
on cssutils.log.raiseExceptions which e.g during parsing normally
is False so the exceptions would be logged only.
:param macros:
may be used in the given properties definitions. There are some
predefined basic macros which may always be used in
:attr:`Profiles.basicmacros` and :attr:`Profiles.generalmacros`.
"""
if not macros:
macros = {}
m = self.basicmacros
m.update(self.generalmacros)
m.update(macros)
properties = self._expand_macros(properties, m)
self._profilenames.append(profile)
self._profiles[profile] = self._compile_regexes(properties)
self.__update_knownnames()
def removeProfile(self, profile=None, all=False):
"""Remove `profile` or remove `all` profiles.
:param profile:
profile name to remove
:param all:
if ``True`` removes all profiles to start with a clean state
:exceptions:
- :exc:`cssutils.profiles.NoSuchProfileException`:
If given `profile` cannot be found.
"""
if all:
self._profiles.clear()
else:
try:
del self._profiles[profile]
except KeyError:
raise NoSuchProfileException(u'No profile %r.' % profile)
self.__update_knownnames()
def propertiesByProfile(self, profiles=None):
"""Generator: Yield property names, if no `profiles` is given all
profile's properties are used.
:param profiles:
a single profile name or a list of names.
"""
if not profiles:
profiles = self.profiles
elif isinstance(profiles, basestring):
profiles = (profiles, )
try:
for profile in sorted(profiles):
for name in sorted(self._profiles[profile].keys()):
yield name
except KeyError, e:
raise NoSuchProfileException(e)
def validate(self, name, value):
"""Check if `value` is valid for given property `name` using **any**
profile.
:param name:
a property name
:param value:
a CSS value (string)
:returns:
if the `value` is valid for the given property `name` in any
profile
"""
for profile in self.profiles:
if name in self._profiles[profile]:
try:
# custom validation errors are caught
r = bool(self._profiles[profile][name](value))
except Exception, e:
self._log.error(e, error=Exception)
return False
if r:
return r
return False
def validateWithProfile(self, name, value, profiles=None):
"""Check if `value` is valid for given property `name` returning
``(valid, profile)``.
:param name:
a property name
:param value:
a CSS value (string)
:returns:
``valid, profiles`` where ``valid`` is if the `value` is valid for
the given property `name` in any profile of given `profiles`
and ``profiles`` the profile names for which the value is valid
(or ``[]`` if not valid at all)
Example: You might expect a valid Profiles.CSS_LEVEL_2 value but
e.g. ``validateWithProfile('color', 'rgba(1,1,1,1)')`` returns
(True, Profiles.CSS_COLOR_LEVEL_3)
"""
if name not in self.knownnames:
return False, []
else:
if not profiles:
profiles = self._profilenames
elif isinstance(profiles, basestring):
profiles = (profiles, )
for profilename in profiles:
# check given profiles
if name in self._profiles[profilename]:
validate = self._profiles[profilename][name]
try:
if validate(value):
return True, [profilename]
except Exception, e:
self._log.error(e, error=Exception)
for profilename in (p for p in self._profilenames if p not in profiles):
# check remaining profiles as well
if name in self._profiles[profilename]:
validate = self._profiles[profilename][name]
try:
if validate(value):
return True, [profilename]
except Exception, e:
self._log.error(e, error=Exception)
names = []
for profilename, properties in self._profiles.items():
# return profile to which name belongs
if name in properties.keys():
names.append(profilename)
names.sort()
return False, names
# used by
profiles = Profiles()
# set for validation to e.g.``Profiles.CSS_LEVEL_2``
defaultprofile = None

16
src/cssutils/serialize.py
View File

@ -3,7 +3,7 @@
"""cssutils serializer"""
__all__ = ['CSSSerializer', 'Preferences']
__docformat__ = 'restructuredtext'
__version__ = '$Id: serialize.py 1606 2009-01-03 20:32:17Z cthedot $'
__version__ = '$Id: serialize.py 1741 2009-05-09 18:20:20Z cthedot $'
import codecs
import cssutils
@ -58,6 +58,9 @@ class Preferences(object):
keepEmptyRules = False
defines if empty rules like e.g. ``a {}`` are kept in the resulting
serialized sheet
keepUnkownAtRules = True
defines if unknown @rules like e.g. ``@three-dee {}`` are kept in the
serialized sheet
keepUsedNamespaceRulesOnly = False
if True only namespace rules which are actually used are kept
@ -82,12 +85,10 @@ class Preferences(object):
spacer = u' '
general spacer, used e.g. by CSSUnknownRule
validOnly = False **DO NOT CHANGE YET**
if True only valid (currently Properties) are kept
validOnly = False
if True only valid (Properties) are output
A Property is valid if it is a known Property with a valid value.
Currently CSS 2.1 values as defined in cssproperties.py would be
valid.
"""
def __init__(self, **initials):
"""Always use named instead of positional parameters."""
@ -118,6 +119,7 @@ class Preferences(object):
self.keepAllProperties = True
self.keepComments = True
self.keepEmptyRules = False
self.keepUnkownAtRules = True
self.keepUsedNamespaceRulesOnly = False
self.lineNumbers = False
self.lineSeparator = u'\n'
@ -139,6 +141,7 @@ class Preferences(object):
self.indent = u''
self.keepComments = False
self.keepEmptyRules = False
self.keepUnkownAtRules = False
self.keepUsedNamespaceRulesOnly = True
self.lineNumbers = False
self.lineSeparator = u''
@ -563,7 +566,7 @@ class CSSSerializer(object):
anything until ";" or "{...}"
+ CSSComments
"""
if rule.wellformed:
if rule.wellformed and self.prefs.keepUnkownAtRules:
out = Out(self)
out.append(rule.atkeyword)
@ -741,6 +744,7 @@ class CSSSerializer(object):
out.append(separator)
elif isinstance(val, cssutils.css.Property):
# PropertySimilarNameList
if val.cssText:
out.append(val.cssText)
if not (self.prefs.omitLastSemicolon and i==len(seq)-1):
out.append(u';')

10
src/cssutils/stylesheets/mediaquery.py
View File

@ -5,7 +5,7 @@ A cssutils implementation, not defined in official DOM.
"""
__all__ = ['MediaQuery']
__docformat__ = 'restructuredtext'
__version__ = '$Id: mediaquery.py 1638 2009-01-13 20:39:33Z cthedot $'
__version__ = '$Id: mediaquery.py 1738 2009-05-02 13:03:28Z cthedot $'
import cssutils
import re
@ -21,8 +21,8 @@ class MediaQuery(cssutils.util.Base):
media_query: [[only | not]? <media_type> [ and <expression> ]*]
| <expression> [ and <expression> ]*
expression: ( <media_feature> [: <value>]? )
media_type: all | aural | braille | handheld | print |
projection | screen | tty | tv | embossed
media_type: all | braille | handheld | print |
projection | speech | screen | tty | tv | embossed
media_feature: width | min-width | max-width
| height | min-height | max-height
| device-width | min-device-width | max-device-width
@ -35,8 +35,8 @@ class MediaQuery(cssutils.util.Base):
| scan | grid
"""
MEDIA_TYPES = [u'all', u'aural', u'braille', u'embossed', u'handheld',
u'print', u'projection', u'screen', u'tty', u'tv']
MEDIA_TYPES = [u'all', u'braille', u'embossed', u'handheld',
u'print', u'projection', u'screen', u'speech', u'tty', u'tv']
# From the HTML spec (see MediaQuery):
# "[...] character that isn't a US ASCII letter [a-zA-Z] (Unicode

3
src/cssutils/util.py
View File

@ -2,7 +2,7 @@
"""
__all__ = []
__docformat__ = 'restructuredtext'
__version__ = '$Id: util.py 1654 2009-02-03 20:16:20Z cthedot $'
__version__ = '$Id: util.py 1743 2009-05-09 20:33:15Z cthedot $'
from helper import normalize
from itertools import ifilter
@ -307,7 +307,6 @@ class Base(_BaseClass):
bracket == parant == 0) and typ in endtypes:
# mediaqueryendonly with STRING
break
if separateEnd:
# TODO: use this method as generator, then this makes sense
if resulttokens:

8
todo
View File

@ -1,16 +1,8 @@
* Refactor web.fetch.simple to use per connection timeouts via the timeout kwarg for mechanize.open
* Refactor IPC code to use communication logic from multiprocessing
* Rationalize books table. Add a pubdate column, remove the uri column (and associated support in add_books) and convert series_index to a float.
* Replace single application stuff with Listener from multiprocessing
* Refactor add books to use a separate process named calibre-worker-add
- Dont use the process for adding a single book
- Use a process pool for speed or multiple process for stability (20 per process?)
* Change mobi metadata setter to use author_sort setting from MOBI output plugin instead of mobi.py
* Fix HTML-to-ZIP plugin