Sync to trunk.

Changelog.yaml

@@ -19,6 +19,69 @@
 #  new recipes:
 #    - title: 
 
+- version: 0.8.15
+  date: 2011-08-19
+
+  new features:
+    - title: "Add a 'languages' metadata field."
+      type: major
+      description: "This is useful if you have a multi-lingual book collection. You can now set one or more languages per book via the Edit Metadata dialog. If you want the languages
+      column to be visible, then go to Preferences->Add your own columns and unhide the languages columns. You can also bulk set the languages on multiple books via the bulk edit metadata dialog. You can also have the languages show up in the book details panel on the right by going to Preferences->Look and Feel->Book details" 
+
+    - title: "Get Books: Add XinXii store."
+    
+    - title: "Metadata download plugin for ozon.ru, enabled only when user selects russian as their language in the welcome wizard."
+
+    - title: "Bambook driver: Allow direct transfer of PDF files to Bambook devices"
+
+    - title: "Driver for Coby MID7015A and Asus EEE Note"
+
+    - title: "Edit metadata dialog: The keyboard shortcut Ctrl+D can now be used to trigger a metadata download. Also show the row number of the book being edited in the titlebar"
+
+    - title: "Add an option to not preserve the date when using the 'Copy to Library' function (found in Preferences->Adding books)"
+  
+  bug fixes:
+    - title: "Linux binary: Use readlink -f rather than readlink -e in the launcher scripts so that they work with recent releases of busybox"
+
+    - title: "When bulk downloading metadata for more than 100 books at a time, automatically split up the download into batches of 100."
+      tickets: [828373]
+
+    - title: "When deleting books from the Kindle also delete 'sidecar' .apnx and .ph1 files as the kindle does not clean them up automatically"
+      tickets: [827684]
+
+    - title: "Fix a subtle bug in the device drivers that caused calibre to lose track of some books on the device if you used author_sort in the send to device template and your books have author sort values that differ only in case."
+      tickets: [825706]
+
+    - title: "Fix scene break character pattern not saved in conversion preferences"
+      tickets: [826038]
+
+    - title: "Keyboard shortcuts: Fix a bug triggered by some third party plugins that made the keyboard preferences unusable in OS X."
+      tickets: [826325]
+
+    - title: "Search box: Fix completion no longer working after using Tag Browser to do a search. Also ensure that completer popup is always hidden when a search is performed."
+
+    - title: "Fix pressing Enter in the search box causes the same search to be executed twice in the plugins and keyboard shortcuts preferences panels"
+
+    - title: "Catalog generation: Fix error creating epub/mobi catalogs on non UTF-8 windows systems when the metadata contained non ASCII characters"
+
+  improved recipes:
+    - Financial Times UK
+    - La Tercera
+    - Folha de Sao Paolo
+    - Metro niews NL 
+    - La Nacion
+    - Juventud Rebelde
+    - Rzeczpospolita Online
+    - Newsweek Polska
+    - CNET news
+
+  new recipes:
+    - title: El Mostrador and The Clinic
+      author: Alex Mitrani
+
+    - title: Patente de Corso
+      author: Oscar Megia Lopez
+
 - version: 0.8.14
   date: 2011-08-12
 

recipes/el_mostrador.recipe

@@ -0,0 +1,40 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class AdvancedUserRecipe1313609361(BasicNewsRecipe):
+    news                  = True
+    title          = u'El Mostrador'
+    __author__            = 'Alex Mitrani'
+    description           = u'Chilean online newspaper'
+    publisher             = u'La Plaza S.A.'
+    category              = 'news, rss'
+    oldest_article = 7
+    max_articles_per_feed = 100
+    summary_length        = 1000
+    language              = 'es_CL'
+    remove_javascript     = True
+    no_stylesheets        = True
+    use_embedded_content  = False
+    remove_empty_feeds    = True
+    masthead_url          = 'http://www.elmostrador.cl/assets/img/logo-elmostrador-m.jpg'
+    remove_tags_before = dict(name='div', attrs={'class':'news-heading cf'})
+    remove_tags_after  = dict(name='div', attrs={'class':'footer-actions cf'})
+    remove_tags = [dict(name='div', attrs={'class':'footer-actions cb cf'})
+	,dict(name='div', attrs={'class':'news-aside fl'})
+	,dict(name='div', attrs={'class':'footer-actions cf'})
+	,dict(name='div', attrs={'class':'user-bar','id':'top'})
+	,dict(name='div', attrs={'class':'indicators'})
+	,dict(name='div', attrs={'id':'header'})
+	]
+
+
+    feeds          = [(u'Temas Destacados'
+	, u'http://www.elmostrador.cl/destacado/feed/')
+	, (u'El D\xeda', u'http://www.elmostrador.cl/dia/feed/')
+	, (u'Pa\xeds', u'http://www.elmostrador.cl/noticias/pais/feed/')
+	, (u'Mundo', u'http://www.elmostrador.cl/noticias/mundo/feed/')
+	, (u'Negocios', u'http://www.elmostrador.cl/noticias/negocios/feed/')
+	, (u'Cultura', u'http://www.elmostrador.cl/noticias/cultura/feed/')
+	, (u'Vida en L\xednea', u'http://www.elmostrador.cl/vida-en-linea/feed/')
+	, (u'Opini\xf3n & Blogs', u'http://www.elmostrador.cl/opinion/feed/')
+	]
+

recipes/financial_times_uk.recipe

@@ -24,6 +24,7 @@ class FinancialTimes(BasicNewsRecipe):
     publication_type      = 'newspaper'
     masthead_url          = 'http://im.media.ft.com/m/img/masthead_main.jpg'
     LOGIN                 = 'https://registration.ft.com/registration/barrier/login'
+    LOGIN2                = 'http://media.ft.com/h/subs3.html'
     INDEX                 = 'http://www.ft.com/uk-edition'
     PREFIX                = 'http://www.ft.com'
 
@@ -39,7 +40,7 @@ class FinancialTimes(BasicNewsRecipe):
         br = BasicNewsRecipe.get_browser()
         br.open(self.INDEX)
         if self.username is not None and self.password is not None:
-            br.open(self.LOGIN)
+            br.open(self.LOGIN2)
             br.select_form(name='loginForm')
             br['username'] = self.username
             br['password'] = self.password

recipes/folhadesaopaulo.recipe

@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 from calibre.web.feeds.news import BasicNewsRecipe
 from datetime import datetime, timedelta
 from calibre.ebooks.BeautifulSoup import Tag,BeautifulSoup
@@ -16,7 +17,7 @@ class FolhaOnline(BasicNewsRecipe):
     news                  = True
 
     title                 = u'Folha de S\xE3o Paulo'
-    __author__            = 'Euler Alves'
+    __author__            = 'Euler Alves and Alex Mitrani'
     description           = u'Brazilian news from Folha de S\xE3o Paulo'
     publisher             = u'Folha de S\xE3o Paulo'
     category              = 'news, rss'
@@ -62,37 +63,50 @@ class FolhaOnline(BasicNewsRecipe):
                         ,dict(name='div',
                             attrs={'class':[
                             'openBox adslibraryArticle'
+	       ,'toolbar'
                             ]})
 
                         ,dict(name='a')
                         ,dict(name='iframe')
                         ,dict(name='link')
                         ,dict(name='script')
+	    ,dict(name='li')
     ]
+    remove_tags_after = 	dict(name='div',attrs={'id':'articleEnd'})
 
     feeds = [
     (u'Em cima da hora', u'http://feeds.folha.uol.com.br/emcimadahora/rss091.xml')
+    ,(u'Cotidiano', u'http://feeds.folha.uol.com.br/folha/cotidiano/rss091.xml')
+    ,(u'Brasil', u'http://feeds.folha.uol.com.br/folha/brasil/rss091.xml')
+    ,(u'Mundo', u'http://feeds.folha.uol.com.br/mundo/rss091.xml')
+    ,(u'Poder', u'http://feeds.folha.uol.com.br/poder/rss091.xml')
+    ,(u'Mercado', u'http://feeds.folha.uol.com.br/folha/dinheiro/rss091.xml')
+    ,(u'Saber', u'http://feeds.folha.uol.com.br/folha/educacao/rss091.xml')
+    ,(u'Tec', u'http://feeds.folha.uol.com.br/folha/informatica/rss091.xml')
+    ,(u'Ilustrada', u'http://feeds.folha.uol.com.br/folha/ilustrada/rss091.xml')
     ,(u'Ambiente', u'http://feeds.folha.uol.com.br/ambiente/rss091.xml')
     ,(u'Bichos', u'http://feeds.folha.uol.com.br/bichos/rss091.xml')
     ,(u'Ci\xEAncia', u'http://feeds.folha.uol.com.br/ciencia/rss091.xml')
-    ,(u'Poder', u'http://feeds.folha.uol.com.br/poder/rss091.xml')
     ,(u'Equil\xEDbrio e Sa\xFAde', u'http://feeds.folha.uol.com.br/equilibrioesaude/rss091.xml')
     ,(u'Turismo', u'http://feeds.folha.uol.com.br/folha/turismo/rss091.xml')
-    ,(u'Mundo', u'http://feeds.folha.uol.com.br/mundo/rss091.xml')
+    ,(u'Esporte', u'http://feeds.folha.uol.com.br/folha/esporte/rss091.xml')
-    ,(u'Pelo Mundo', u'http://feeds.folha.uol.com.br/pelomundo.folha.rssblog.uol.com.br/')
+    ,(u'Zapping', u'http://feeds.folha.uol.com.br/colunas/zapping/rss091.xml')
-    ,(u'Circuito integrado', u'http://feeds.folha.uol.com.br/circuitointegrado.folha.rssblog.uol.com.br/')
+    ,(u'Cida Santos', u'http://feeds.folha.uol.com.br/colunas/cidasantos/rss091.xml')
-    ,(u'Blog do Fred', u'http://feeds.folha.uol.com.br/blogdofred.folha.rssblog.uol.com.br/')
+    ,(u'Clóvis Rossi', u'http://feeds.folha.uol.com.br/colunas/clovisrossi/rss091.xml')
-    ,(u'Maria In\xEAs Dolci', u'http://feeds.folha.uol.com.br/mariainesdolci.folha.blog.uol.com.br/')
+    ,(u'Eliane Cantanhêde', u'http://feeds.folha.uol.com.br/colunas/elianecantanhede/rss091.xml')
-    ,(u'Eduardo Ohata', u'http://feeds.folha.uol.com.br/folha/pensata/eduardoohata/rss091.xml')
+    ,(u'Fernando Canzian', u'http://feeds.folha.uol.com.br/colunas/fernandocanzian/rss091.xml')
-    ,(u'Kennedy Alencar', u'http://feeds.folha.uol.com.br/folha/pensata/kennedyalencar/rss091.xml')
+    ,(u'Gilberto Dimenstein', u'http://feeds.folha.uol.com.br/colunas/gilbertodimenstein/rss091.xml')
-    ,(u'Eliane Catanh\xEAde', u'http://feeds.folha.uol.com.br/folha/pensata/elianecantanhede/rss091.xml')
+    ,(u'Hélio Schwartsman', u'http://feeds.folha.uol.com.br/colunas/helioschwartsman/rss091.xml')
-    ,(u'Fernado Canzian', u'http://feeds.folha.uol.com.br/folha/pensata/fernandocanzian/rss091.xml')
+    ,(u'Humberto Luiz Peron', u'http://feeds.folha.uol.com.br/colunas/futebolnarede/rss091.xml')
-    ,(u'Gilberto Dimenstein', u'http://feeds.folha.uol.com.br/folha/pensata/gilbertodimenstein/rss091.xml')
+    ,(u'João Pereira Coutinho', u'http://feeds.folha.uol.com.br/colunas/joaopereiracoutinho/rss091.xml')
-    ,(u'H\xE9lio Schwartsman', u'http://feeds.folha.uol.com.br/folha/pensata/helioschwartsman/rss091.xml')
+    ,(u'José Antonio Ramalho', u'http://feeds.folha.uol.com.br/colunas/canalaberto/rss091.xml')
-    ,(u'Jo\xE3o Pereira Coutinho', u'http://http://feeds.folha.uol.com.br/folha/pensata/joaopereiracoutinho/rss091.xml')
+    ,(u'Kennedy Alencar', u'http://feeds.folha.uol.com.br/colunas/kennedyalencar/rss091.xml')
-    ,(u'Luiz Caversan', u'http://http://feeds.folha.uol.com.br/folha/pensata/luizcaversan/rss091.xml')
+    ,(u'Luiz Caversan', u'http://feeds.folha.uol.com.br/colunas/luizcaversan/rss091.xml')
-    ,(u'S\xE9rgio Malbergier', u'http://http://feeds.folha.uol.com.br/folha/pensata/sergiomalbergier/rss091.xml')
+    ,(u'Luiz Rivoiro', u'http://feeds.folha.uol.com.br/colunas/paiepai/rss091.xml')
-    ,(u'Valdo Cruz', u'http://http://feeds.folha.uol.com.br/folha/pensata/valdocruz/rss091.xml')
+    ,(u'Marcelo Leite', u'http://feeds.folha.uol.com.br/colunas/marceloleite/rss091.xml')
+    ,(u'Sérgio Malbergier', u'http://feeds.folha.uol.com.br/colunas/sergiomalbergier/rss091.xml')
+    ,(u'Sylvia Colombo', u'http://feeds.folha.uol.com.br/colunas/sylviacolombo/rss091.xml')
+    ,(u'Valdo Cruz', u'http://feeds.folha.uol.com.br/colunas/valdocruz/rss091.xml')
     ]
 
 

recipes/la_tercera.recipe

@@ -7,8 +7,9 @@ latercera.com
 from calibre.web.feeds.news import BasicNewsRecipe
 
 class LaTercera(BasicNewsRecipe):
+    news                  = True
     title                 = 'La Tercera'
-    __author__            = 'Darko Miletic'
+    __author__            = 'Darko Miletic and Alex Mitrani'
     description           = 'El sitio de noticias online de Chile'
     publisher             = 'La Tercera'
     category              = 'news, politics, Chile'
@@ -18,8 +19,8 @@ class LaTercera(BasicNewsRecipe):
     encoding              = 'cp1252'
     use_embedded_content  = False
     remove_empty_feeds    = True
-    language              = 'es'
+    language              = 'es_CL'
- 
+
     conversion_options = {
                           'comment'          : description
                         , 'tags'             : category
@@ -28,28 +29,33 @@ class LaTercera(BasicNewsRecipe):
 						, 'linearize_tables' : True
                         }
 
-    keep_only_tags = [dict(name='div', attrs={'class':['span-16 articulo border','span-16 border','span-16']}) ]
+    keep_only_tags = [
+                    dict(name='h1', attrs={'class':['titularArticulo']})
+	,dict(name='h4', attrs={'class':['bajadaArt']})
+	,dict(name='h5', attrs={'class':['autorArt']})
+                    ,dict(name='div', attrs={'class':['articleContent']})
+                  ]
 
     remove_tags = [
-                     dict(name=['ul','input','base'])
+                    dict(name='div', attrs={'class':['boxCompartir','keywords']})
-                    ,dict(name='div', attrs={'id':['boxComentarios','shim','enviarAmigo']})
+                  ]
-                    ,dict(name='div', attrs={'class':['ad640','span-10 imgSet A','infoRelCol']})
+
-                    ,dict(name='p', attrs={'id':['mensajeError','mensajeEnviandoNoticia','mensajeExito']})
+    remove_tags_after = [
+                    dict(name='div', attrs={'class':['keywords']})
                   ]
 
 
-    feeds = [
+    feeds = [(u'La Tercera', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&ul=1')
-               (u'Noticias de ultima hora', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&ul=1')
+              ,(u'Politica', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=674')
               ,(u'Nacional', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=680')
-              ,(u'Politica', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=674')              
               ,(u'Mundo', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=678')
-              ,(u'Deportes', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=656')
               ,(u'Negocios', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=655')
-              ,(u'Entretenimiento', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=661')
+              ,(u'Santiago', u'http://www.latercera.com/feed/manager?type=rss&sc=TEFURVJDRVJB&citId=9&categoryId=1731')
-              ,(u'Motores', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=665')
               ,(u'Tendencias', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=659')
-              ,(u'Estilo', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=660')
               ,(u'Educacion', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=657')
+              ,(u'Cultura', u'http://www.latercera.com/feed/manager?type=rss&sc=TEFURVJDRVJB&citId=9&categoryId=1453')
+              ,(u'Entretención', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=661')
+              ,(u'Deportes', u'http://www.latercera.com/app/rss?sc=TEFURVJDRVJB&category=656')
             ]
 
     def preprocess_html(self, soup):

recipes/liberation.recipe

@@ -18,21 +18,28 @@ class Liberation(BasicNewsRecipe):
     max_articles_per_feed = 100
     no_stylesheets        = True
     use_embedded_content  = False
-    
+
     html2lrf_options = ['--base-font-size', '10']
 
     keep_only_tags    = [
                            dict(name='h1')
-                          ,dict(name='div', attrs={'class':'articleContent'})
+                          #,dict(name='div', attrs={'class':'object-content text text-item'})
+                          ,dict(name='div', attrs={'class':'article'})
+                          #,dict(name='div', attrs={'class':'articleContent'})
                           ,dict(name='div', attrs={'class':'entry'})
                         ]
+    remove_tags_after = [ dict(name='div',attrs={'class':'toolbox extra_toolbox'}) ]
     remove_tags    = [
                         dict(name='p', attrs={'class':'clear'})
                        ,dict(name='ul', attrs={'class':'floatLeft clear'})
                        ,dict(name='div', attrs={'class':'clear floatRight'})
                        ,dict(name='object')
+                       ,dict(name='div', attrs={'class':'toolbox'})
+                       ,dict(name='div', attrs={'class':'cartridge cartridge-basic-bubble cat-zoneabo'})
+                       #,dict(name='div', attrs={'class':'clear block block-call-items'})
+                       ,dict(name='div', attrs={'class':'block-content'})
                      ]
-    
+
     feeds          = [
                          (u'La une', u'http://www.liberation.fr/rss/laune')
                         ,(u'Monde' , u'http://www.liberation.fr/rss/monde')

recipes/metro_news_nl.recipe

@@ -2,6 +2,9 @@ from calibre.web.feeds.news import BasicNewsRecipe
 
 class AdvancedUserRecipe1306097511(BasicNewsRecipe):
     title = u'Metro Nieuws NL'
+    description = u'Metro Nieuws - NL'
+# Version 1.2, updated cover image to match the changed website.
+# added info date on title
     oldest_article = 2
     max_articles_per_feed = 100
     __author__     = u'DrMerry'
@@ -10,11 +13,11 @@ class AdvancedUserRecipe1306097511(BasicNewsRecipe):
     simultaneous_downloads = 5
     delay          = 1
 #    timefmt        = ' [%A, %d %B, %Y]'
-    timefmt        = ''
+    timefmt        = ' [%A, %d %b %Y]'
     no_stylesheets = True
     remove_javascript = True
     remove_empty_feeds = True
-    cover_url      = 'http://www.readmetro.com/img/en/metroholland/last/1/small.jpg'
+    cover_url      = 'http://www.oldreadmetro.com/img/en/metroholland/last/1/small.jpg'
     remove_empty_feeds = True
     publication_type = 'newspaper'
     remove_tags_before = dict(name='div', attrs={'id':'date'})

recipes/the_clinic_online.recipe

@@ -0,0 +1,27 @@
+from calibre.web.feeds.news import BasicNewsRecipe
+
+class AdvancedUserRecipe1313555075(BasicNewsRecipe):
+    news                  = True
+    title          = u'The Clinic'
+    __author__            = 'Alex Mitrani'
+    description           = u'Online version of Chilean satirical weekly'
+    publisher             = u'The Clinic'
+    category              = 'news, politics, Chile, rss'
+    oldest_article = 7
+    max_articles_per_feed = 100
+    summary_length        = 1000
+    language              = 'es_CL'
+
+    remove_javascript     = True
+    no_stylesheets        = True
+    use_embedded_content  = False
+    remove_empty_feeds    = True
+    masthead_url          = 'http://www.theclinic.cl/wp-content/themes/tc12m/css/ui/mainLogoTC-top.png'
+    remove_tags_before = dict(name='article', attrs={'class':'scope bordered'})
+    remove_tags_after  = dict(name='div', attrs={'id':'commentsSection'})
+    remove_tags = [dict(name='span', attrs={'class':'relTags'})
+		,dict(name='div', attrs={'class':'articleActivity hdcol'})
+		,dict(name='div', attrs={'id':'commentsSection'})
+	]
+
+    feeds          = [(u'The Clinic Online', u'http://www.theclinic.cl/feed/')]

resources/default_tweaks.py

@@ -181,7 +181,7 @@ save_template_title_series_sorting = 'library_order'
 # To disable use the expression: '^$'
 # This expression is designed for articles that are followed by spaces. If you
 # also need to match articles that are followed by other characters, for example L'
-# in French, use: r"^(A\s+|The\s+|An\s+|L')" instead.
+# in French, use: "^(A\s+|The\s+|An\s+|L')" instead.
 # Default: '^(A|The|An)\s+'
 title_sort_articles=r'^(A|The|An)\s+'
 

setup/installer/linux/freeze2.py

@@ -290,7 +290,7 @@ class LinuxFreeze(Command):
 
                 launcher = textwrap.dedent('''\
                 #!/bin/sh
-                path=`readlink -e $0`
+                path=`readlink -f $0`
                 base=`dirname $path`
                 lib=$base/lib
                 export LD_LIBRARY_PATH=$lib:$LD_LIBRARY_PATH

setup/translations.py

@@ -291,6 +291,8 @@ class ISO639(Command):
         by_3t = {}
         m2to3 = {}
         m3to2 = {}
+        m3bto3t = {}
+        nm = {}
         codes2, codes3t, codes3b = set([]), set([]), set([])
         for x in root.xpath('//iso_639_entry'):
             name = x.get('name')
@@ -304,12 +306,19 @@ class ISO639(Command):
                 m3to2[threeb] = m3to2[threet] = two
             by_3b[threeb] = name
             by_3t[threet] = name
+            if threeb != threet:
+                m3bto3t[threeb] = threet
             codes3b.add(x.get('iso_639_2B_code'))
             codes3t.add(x.get('iso_639_2T_code'))
+            base_name = name.lower()
+            nm[base_name] = threet
+            simple_name = base_name.partition(';')[0].strip()
+            if simple_name not in nm:
+                nm[simple_name] = threet
 
         from cPickle import dump
         x = {'by_2':by_2, 'by_3b':by_3b, 'by_3t':by_3t, 'codes2':codes2,
                 'codes3b':codes3b, 'codes3t':codes3t, '2to3':m2to3,
-                '3to2':m3to2}
+                '3to2':m3to2, '3bto3t':m3bto3t, 'name_map':nm}
         dump(x, open(dest, 'wb'), -1)
 

src/calibre/constants.py

@@ -4,7 +4,7 @@ __license__   = 'GPL v3'
 __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 __appname__   = u'calibre'
-numeric_version = (0, 8, 14)
+numeric_version = (0, 8, 15)
 __version__   = u'.'.join(map(unicode, numeric_version))
 __author__    = u"Kovid Goyal <kovid@kovidgoyal.net>"
 

src/calibre/customize/builtins.py

@@ -590,8 +590,9 @@ from calibre.ebooks.metadata.sources.openlibrary import OpenLibrary
 from calibre.ebooks.metadata.sources.isbndb import ISBNDB
 from calibre.ebooks.metadata.sources.overdrive import OverDrive
 from calibre.ebooks.metadata.sources.douban import Douban
+from calibre.ebooks.metadata.sources.ozon import Ozon
 
-plugins += [GoogleBooks, Amazon, OpenLibrary, ISBNDB, OverDrive, Douban]
+plugins += [GoogleBooks, Amazon, OpenLibrary, ISBNDB, OverDrive, Douban, Ozon]
 
 # }}}
 
@@ -1476,6 +1477,14 @@ class StoreWoblinkStore(StoreBase):
     headquarters = 'PL'
     formats = ['EPUB']
 
+class XinXiiStore(StoreBase):
+    name = 'XinXii'
+    description = ''
+    actual_plugin = 'calibre.gui2.store.stores.xinxii_plugin:XinXiiStore'
+    
+    headquarters = 'DE'
+    formats = ['EPUB', 'PDF']
+
 class StoreZixoStore(StoreBase):
     name = 'Zixo'
     author = u'Tomasz Długosz'
@@ -1524,6 +1533,7 @@ plugins += [
     StoreWHSmithUKStore,
     StoreWizardsTowerBooksStore,
     StoreWoblinkStore,
+    XinXiiStore,
     StoreZixoStore
 ]
 

src/calibre/customize/ui.py

@@ -92,7 +92,7 @@ def restore_plugin_state_to_default(plugin_or_name):
     config['enabled_plugins'] = ep
 
 default_disabled_plugins = set([
-    'Overdrive', 'Douban Books',
+    'Overdrive', 'Douban Books', 'OZON.ru',
 ])
 
 def is_disabled(plugin):

src/calibre/db/cache.py

@@ -122,12 +122,17 @@ class Cache(object):
         formats = self._field_for('formats', book_id)
         mi.format_metadata = {}
         if not formats:
-            formats = None
+            good_formats = None
         else:
+            good_formats = []
             for f in formats:
-                mi.format_metadata[f] = self._format_metadata(book_id, f)
+                try:
-            formats = ','.join(formats)
+                    mi.format_metadata[f] = self._format_metadata(book_id, f)
-        mi.formats = formats
+                except:
+                    pass
+                else:
+                    good_formats.append(f)
+        mi.formats = good_formats
         mi.has_cover = _('Yes') if self._field_for('cover', book_id,
                 default_value=False) else ''
         mi.tags = list(self._field_for('tags', book_id, default_value=()))

src/calibre/devices/android/driver.py

@@ -30,6 +30,7 @@ class ANDROID(USBMS):
                        0xca2  : [0x100, 0x0227, 0x0226, 0x222],
                        0xca3  : [0x100, 0x0227, 0x0226, 0x222],
                        0xca4  : [0x100, 0x0227, 0x0226, 0x222],
+                       0xca9  : [0x100, 0x0227, 0x0226, 0x222]
             },
 
             # Eken
@@ -64,6 +65,7 @@ class ANDROID(USBMS):
                        0x6860 : [0x0400],
                        0x6877 : [0x0400],
                        0x689e : [0x0400],
+                       0xdeed : [0x0222],
                      },
 
             # Viewsonic
@@ -132,7 +134,7 @@ class ANDROID(USBMS):
             '7', 'A956', 'A955', 'A43', 'ANDROID_PLATFORM', 'TEGRA_2',
             'MB860', 'MULTI-CARD', 'MID7015A', 'INCREDIBLE', 'A7EB', 'STREAK',
             'MB525', 'ANDROID2.3', 'SGH-I997', 'GT-I5800_CARD', 'MB612',
-            'GT-S5830_CARD', 'GT-S5570_CARD', 'MB870']
+            'GT-S5830_CARD', 'GT-S5570_CARD', 'MB870', 'MID7015A']
     WINDOWS_CARD_A_MEM = ['ANDROID_PHONE', 'GT-I9000_CARD', 'SGH-I897',
             'FILE-STOR_GADGET', 'SGH-T959', 'SAMSUNG_ANDROID', 'GT-P1000_CARD',
             'A70S', 'A101IT', '7', 'INCREDIBLE', 'A7EB', 'SGH-T849_CARD',

src/calibre/devices/bambook/driver.py

@@ -8,7 +8,7 @@ __docformat__ = 'restructuredtext en'
 Device driver for Sanda's Bambook
 '''
 
-import time, os, hashlib
+import time, os, hashlib, shutil
 from itertools import cycle
 from calibre.devices.interface import DevicePlugin
 from calibre.devices.usbms.deviceconfig import DeviceConfig
@@ -31,7 +31,7 @@ class BAMBOOK(DeviceConfig, DevicePlugin):
 
     ip = None
 
-    FORMATS = [ "snb" ]
+    FORMATS = [ "snb", "pdf" ]
     USER_CAN_ADD_NEW_FORMATS = False
     VENDOR_ID = 0x230b
     PRODUCT_ID = 0x0001
@@ -267,14 +267,59 @@ class BAMBOOK(DeviceConfig, DevicePlugin):
             for (i, f) in enumerate(files):
                 self.report_progress((i+1) / float(len(files)), _('Transferring books to device...'))
                 if not hasattr(f, 'read'):
-                    if self.bambook.VerifySNB(f):
+                    # Handle PDF File
-                        guid = self.bambook.SendFile(f, self.get_guid(metadata[i].uuid))
+                    if f[-3:].upper() == "PDF":
-                        if guid:
+                        # Package the PDF file
-                            paths.append(guid)
+                        with TemporaryDirectory() as tdir:
-                        else:
+                            snbcdir = os.path.join(tdir, 'snbc')
-                            print "Send fail"
+                            snbfdir = os.path.join(tdir, 'snbf')
+                            os.mkdir(snbcdir)
+                            os.mkdir(snbfdir)
+
+                            tmpfile = open(os.path.join(snbfdir, 'book.snbf'), 'wb')
+                            tmpfile.write('''<book-snbf version="1.0">
+  <head>
+    <name><![CDATA[''' + metadata[i].title + ''']]></name>
+    <author><![CDATA[''' + ' '.join(metadata[i].authors) + ''']]></author>
+    <language>ZH-CN</language>
+    <rights/>
+    <publisher>calibre</publisher>
+    <generator>''' + __appname__ + ' ' + __version__ + '''</generator>
+    <created/>
+    <abstract></abstract>
+    <cover/>
+  </head>
+</book-snbf>
+''')
+                            tmpfile.close()
+                            tmpfile = open(os.path.join(snbfdir, 'toc.snbf'), 'wb')
+                            tmpfile.write('''<toc-snbf>
+  <head>
+    <chapters>1</chapters>
+  </head>
+  <body>
+    <chapter src="pdf1.pdf"><![CDATA[''' + metadata[i].title + ''']]></chapter>
+  </body>
+</toc-snbf>
+''');
+                            tmpfile.close()
+                            pdf_name = os.path.join(snbcdir, "pdf1.pdf")
+                            shutil.copyfile(f, pdf_name)
+
+                            with TemporaryFile('.snb') as snbfile:
+                                if self.bambook.PackageSNB(snbfile, tdir) and self.bambook.VerifySNB(snbfile):
+                                    guid = self.bambook.SendFile(snbfile, self.get_guid(metadata[i].uuid))
+                        
+                    elif f[-3:].upper() == 'SNB':    
+                        if self.bambook.VerifySNB(f):
+                            guid = self.bambook.SendFile(f, self.get_guid(metadata[i].uuid))
                     else:
                         print "book invalid"
+                    if guid:
+                        paths.append(guid)
+                    else:
+                        print "Send fail"
+
         ret = zip(paths, cycle([on_card]))
         self.report_progress(1.0, _('Transferring books to device...'))
         return ret

src/calibre/devices/misc.py

@@ -252,8 +252,8 @@ class EEEREADER(USBMS):
 
     EBOOK_DIR_MAIN = EBOOK_DIR_CARD_A = 'Book'
 
-    VENDOR_NAME = 'LINUX'
+    VENDOR_NAME = ['LINUX', 'ASUS']
-    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = 'FILE-STOR_GADGET'
+    WINDOWS_MAIN_MEM = WINDOWS_CARD_A_MEM = ['FILE-STOR_GADGET', 'EEE_NOTE']
 
 class ADAM(USBMS):
 

src/calibre/ebooks/fb2/input.py

@@ -100,23 +100,28 @@ class FB2Input(InputFormatPlugin):
             mi.title = _('Unknown')
         if not mi.authors:
             mi.authors = [_('Unknown')]
-        opf = OPFCreator(os.getcwdu(), mi)
+        cpath = None
-        entries = [(f, guess_type(f)[0]) for f in os.listdir('.')]
-        opf.create_manifest(entries)
-        opf.create_spine(['index.xhtml'])
         if mi.cover_data and mi.cover_data[1]:
             with open('fb2_cover_calibre_mi.jpg', 'wb') as f:
                 f.write(mi.cover_data[1])
-            opf.guide.set_cover(os.path.abspath('fb2_cover_calibre_mi.jpg'))
+            cpath = os.path.abspath('fb2_cover_calibre_mi.jpg')
         else:
             for img in doc.xpath('//f:coverpage/f:image', namespaces=NAMESPACES):
                 href = img.get('{%s}href'%XLINK_NS, img.get('href', None))
                 if href is not None:
                     if href.startswith('#'):
                         href = href[1:]
-                    opf.guide.set_cover(os.path.abspath(href))
+                    cpath = os.path.abspath(href)
+                    break
 
-        opf.render(open('metadata.opf', 'wb'))
+        opf = OPFCreator(os.getcwdu(), mi)
+        entries = [(f, guess_type(f)[0]) for f in os.listdir('.')]
+        opf.create_manifest(entries)
+        opf.create_spine(['index.xhtml'])
+        if cpath:
+            opf.guide.set_cover(cpath)
+        with open('metadata.opf', 'wb') as f:
+            opf.render(f)
         return os.path.join(os.getcwd(), 'metadata.opf')
 
     def extract_embedded_content(self, doc):

src/calibre/ebooks/metadata/book/__init__.py

@@ -47,8 +47,7 @@ PUBLICATION_METADATA_FIELDS = frozenset([
     # If None, means book
     'publication_type',
     'uuid',             # A UUID usually of type 4
-    'language',         # the primary language of this book
+    'languages',        # ordered list of languages in this publication
-    'languages',        # ordered list
     'publisher',        # Simple string, no special semantics
     # Absolute path to image file encoded in filesystem_encoding
     'cover',
@@ -109,7 +108,7 @@ STANDARD_METADATA_FIELDS = SOCIAL_METADATA_FIELDS.union(
 # Metadata fields that smart update must do special processing to copy.
 SC_FIELDS_NOT_COPIED =     frozenset(['title', 'title_sort', 'authors',
                                       'author_sort', 'author_sort_map',
-                                      'cover_data', 'tags', 'language',
+                                      'cover_data', 'tags', 'languages',
                                       'identifiers'])
 
 # Metadata fields that smart update should copy only if the source is not None

src/calibre/ebooks/metadata/book/base.py

@@ -102,6 +102,7 @@ class Metadata(object):
         @param other: None or a metadata object
         '''
         _data = copy.deepcopy(NULL_VALUES)
+        _data.pop('language')
         object.__setattr__(self, '_data', _data)
         if other is not None:
             self.smart_update(other)
@@ -136,6 +137,11 @@ class Metadata(object):
         _data = object.__getattribute__(self, '_data')
         if field in TOP_LEVEL_IDENTIFIERS:
             return _data.get('identifiers').get(field, None)
+        if field == 'language':
+            try:
+                return _data.get('languages', [])[0]
+            except:
+                return NULL_VALUES['language']
         if field in STANDARD_METADATA_FIELDS:
             return _data.get(field, None)
         try:
@@ -175,6 +181,11 @@ class Metadata(object):
             if not val:
                 val = copy.copy(NULL_VALUES.get('identifiers', None))
             self.set_identifiers(val)
+        elif field == 'language':
+            langs = []
+            if val and val.lower() != 'und':
+                langs = [val]
+            _data['languages'] = langs
         elif field in STANDARD_METADATA_FIELDS:
             if val is None:
                 val = copy.copy(NULL_VALUES.get(field, None))
@@ -553,9 +564,9 @@ class Metadata(object):
                 for attr in TOP_LEVEL_IDENTIFIERS:
                     copy_not_none(self, other, attr)
 
-        other_lang = getattr(other, 'language', None)
+        other_lang = getattr(other, 'languages', [])
-        if other_lang and other_lang.lower() != 'und':
+        if other_lang and other_lang != ['und']:
-            self.language = other_lang
+            self.languages = list(other_lang)
         if not getattr(self, 'series', None):
             self.series_index = None
 
@@ -706,8 +717,8 @@ class Metadata(object):
             fmt('Tags', u', '.join([unicode(t) for t in self.tags]))
         if self.series:
             fmt('Series', self.series + ' #%s'%self.format_series_index())
-        if not self.is_null('language'):
+        if not self.is_null('languages'):
-            fmt('Language', self.language)
+            fmt('Languages', ', '.join(self.languages))
         if self.rating is not None:
             fmt('Rating', self.rating)
         if self.timestamp is not None:
@@ -743,7 +754,7 @@ class Metadata(object):
         ans += [(_('Tags'), u', '.join([unicode(t) for t in self.tags]))]
         if self.series:
             ans += [(_('Series'), unicode(self.series) + ' #%s'%self.format_series_index())]
-        ans += [(_('Language'), unicode(self.language))]
+        ans += [(_('Languages'), u', '.join(self.languages))]
         if self.timestamp is not None:
             ans += [(_('Timestamp'), unicode(self.timestamp.isoformat(' ')))]
         if self.pubdate is not None:

src/calibre/ebooks/metadata/fb2.py

@@ -11,7 +11,7 @@ from functools import partial
 from base64 import b64decode
 from lxml import etree
 from calibre.utils.date import parse_date
-from calibre import guess_all_extensions, prints, force_unicode
+from calibre import guess_type, guess_all_extensions, prints, force_unicode
 from calibre.ebooks.metadata import MetaInformation, check_isbn
 from calibre.ebooks.chardet import xml_to_unicode
 
@@ -147,6 +147,12 @@ def _parse_cover_data(root, imgid, mi):
     if elm_binary:
         mimetype = elm_binary[0].get('content-type', 'image/jpeg')
         mime_extensions = guess_all_extensions(mimetype)
+
+        if not mime_extensions and mimetype.startswith('image/'):
+            mimetype_fromid = guess_type(imgid)[0]
+            if mimetype_fromid and mimetype_fromid.startswith('image/'):
+                mime_extensions = guess_all_extensions(mimetype_fromid)
+
         if mime_extensions:
             pic_data = elm_binary[0].text
             if pic_data:

src/calibre/ebooks/metadata/opf2.py

@@ -19,7 +19,7 @@ from calibre.ebooks.metadata.toc import TOC
 from calibre.ebooks.metadata import string_to_authors, MetaInformation, check_isbn
 from calibre.ebooks.metadata.book.base import Metadata
 from calibre.utils.date import parse_date, isoformat
-from calibre.utils.localization import get_lang
+from calibre.utils.localization import get_lang, canonicalize_lang
 from calibre import prints, guess_type
 from calibre.utils.cleantext import clean_ascii_chars
 from calibre.utils.config import tweaks
@@ -515,6 +515,7 @@ class OPF(object): # {{{
                             '(re:match(@opf:scheme, "calibre|libprs500", "i") or re:match(@scheme, "calibre|libprs500", "i"))]')
     uuid_id_path    = XPath('descendant::*[re:match(name(), "identifier", "i") and '+
                             '(re:match(@opf:scheme, "uuid", "i") or re:match(@scheme, "uuid", "i"))]')
+    languages_path  = XPath('descendant::*[local-name()="language"]')
 
     manifest_path   = XPath('descendant::*[re:match(name(), "manifest", "i")]/*[re:match(name(), "item", "i")]')
     manifest_ppath  = XPath('descendant::*[re:match(name(), "manifest", "i")]')
@@ -523,7 +524,6 @@ class OPF(object): # {{{
 
     title           = MetadataField('title', formatter=lambda x: re.sub(r'\s+', ' ', x))
     publisher       = MetadataField('publisher')
-    language        = MetadataField('language')
     comments        = MetadataField('description')
     category        = MetadataField('type')
     rights          = MetadataField('rights')
@@ -930,6 +930,44 @@ class OPF(object): # {{{
         return property(fget=fget, fset=fset)
 
 
+    @dynamic_property
+    def language(self):
+
+        def fget(self):
+            ans = self.languages
+            if ans:
+                return ans[0]
+
+        def fset(self, val):
+            self.languages = [val]
+
+        return property(fget=fget, fset=fset)
+
+
+    @dynamic_property
+    def languages(self):
+
+        def fget(self):
+            ans = []
+            for match in self.languages_path(self.metadata):
+                t = self.get_text(match)
+                if t and t.strip():
+                    l = canonicalize_lang(t.strip())
+                    if l:
+                        ans.append(l)
+            return ans
+
+        def fset(self, val):
+            matches = self.languages_path(self.metadata)
+            for x in matches:
+                x.getparent().remove(x)
+
+            for lang in val:
+                l = self.create_metadata_element('language')
+                self.set_text(l, unicode(lang))
+
+        return property(fget=fget, fset=fset)
+
 
     @dynamic_property
     def book_producer(self):
@@ -989,7 +1027,7 @@ class OPF(object): # {{{
             if self.guide is not None:
                 for t in ('cover', 'other.ms-coverimage-standard', 'other.ms-coverimage'):
                     for item in self.guide:
-                        if item.type.lower() == t:
+                        if item.type and item.type.lower() == t:
                             return item.path
             try:
                 return self.guess_cover()
@@ -1052,9 +1090,9 @@ class OPF(object): # {{{
             val = getattr(mi, attr, None)
             if val is not None and val != [] and val != (None, None):
                 setattr(self, attr, val)
-        lang = getattr(mi, 'language', None)
+        langs = getattr(mi, 'languages', [])
-        if lang and lang != 'und':
+        if langs and langs != ['und']:
-            self.language = lang
+            self.languages = langs
         temp = self.to_book_metadata()
         temp.smart_update(mi, replace_metadata=replace_metadata)
         self._user_metadata_ = temp.get_all_user_metadata(True)
@@ -1202,10 +1240,11 @@ class OPFCreator(Metadata):
             dc_attrs={'id':__appname__+'_id'}))
         if getattr(self, 'pubdate', None) is not None:
             a(DC_ELEM('date', self.pubdate.isoformat()))
-        lang = self.language
+        langs = self.languages
-        if not lang or lang.lower() == 'und':
+        if not langs or langs == ['und']:
-            lang = get_lang().replace('_', '-')
+            langs = [get_lang().replace('_', '-').partition('-')[0]]
-        a(DC_ELEM('language', lang))
+        for lang in langs:
+            a(DC_ELEM('language', lang))
         if self.comments:
             a(DC_ELEM('description', self.comments))
         if self.publisher:
@@ -1288,8 +1327,9 @@ def metadata_to_opf(mi, as_string=True):
         mi.book_producer = __appname__ + ' (%s) '%__version__ + \
             '[http://calibre-ebook.com]'
 
-    if not mi.language:
+    if not mi.languages:
-        mi.language = 'UND'
+        lang = get_lang().replace('_', '-').partition('-')[0]
+        mi.languages = [lang]
 
     root = etree.fromstring(textwrap.dedent(
     '''
@@ -1339,8 +1379,10 @@ def metadata_to_opf(mi, as_string=True):
         factory(DC('identifier'), val, scheme=icu_upper(key))
     if mi.rights:
         factory(DC('rights'), mi.rights)
-    factory(DC('language'), mi.language if mi.language and mi.language.lower()
+    for lang in mi.languages:
-            != 'und' else get_lang().replace('_', '-'))
+        if not lang or lang.lower() == 'und':
+            continue
+        factory(DC('language'), lang)
     if mi.tags:
         for tag in mi.tags:
             factory(DC('subject'), tag)

src/calibre/ebooks/metadata/sources/amazon.py

@@ -22,6 +22,7 @@ from calibre.ebooks.chardet import xml_to_unicode
 from calibre.ebooks.metadata.book.base import Metadata
 from calibre.library.comments import sanitize_comments_html
 from calibre.utils.date import parse_date
+from calibre.utils.localization import canonicalize_lang
 
 class Worker(Thread): # Get details {{{
 
@@ -106,10 +107,11 @@ class Worker(Thread): # Get details {{{
             r'([0-9.]+) (out of|von|su|étoiles sur) (\d+)( (stars|Sternen|stelle)){0,1}')
 
         lm = {
-                'en': ('English', 'Englisch'),
+                'eng': ('English', 'Englisch'),
-                'fr': ('French', 'Français'),
+                'fra': ('French', 'Français'),
-                'it': ('Italian', 'Italiano'),
+                'ita': ('Italian', 'Italiano'),
-                'de': ('German', 'Deutsch'),
+                'deu': ('German', 'Deutsch'),
+                'spa': ('Spanish', 'Espa\xf1ol', 'Espaniol'),
                 }
         self.lang_map = {}
         for code, names in lm.iteritems():
@@ -374,8 +376,11 @@ class Worker(Thread): # Get details {{{
     def parse_language(self, pd):
         for x in reversed(pd.xpath(self.language_xpath)):
             if x.tail:
-                ans = x.tail.strip()
+                raw = x.tail.strip()
-                ans = self.lang_map.get(ans, None)
+                ans = self.lang_map.get(raw, None)
+                if ans:
+                    return ans
+                ans = canonicalize_lang(ans)
                 if ans:
                     return ans
 # }}}
@@ -388,7 +393,7 @@ class Amazon(Source):
     capabilities = frozenset(['identify', 'cover'])
     touched_fields = frozenset(['title', 'authors', 'identifier:amazon',
         'identifier:isbn', 'rating', 'comments', 'publisher', 'pubdate',
-        'language'])
+        'languages'])
     has_html_comments = True
     supports_gzip_transfer_encoding = True
 

src/calibre/ebooks/metadata/sources/google.py

@@ -20,6 +20,7 @@ from calibre.ebooks.metadata.book.base import Metadata
 from calibre.ebooks.chardet import xml_to_unicode
 from calibre.utils.date import parse_date, utcnow
 from calibre.utils.cleantext import clean_ascii_chars
+from calibre.utils.localization import canonicalize_lang
 from calibre import as_unicode
 
 NAMESPACES = {
@@ -95,7 +96,9 @@ def to_metadata(browser, log, entry_, timeout): # {{{
         return mi
 
     mi.comments = get_text(extra, description)
-    #mi.language = get_text(extra, language)
+    lang = canonicalize_lang(get_text(extra, language))
+    if lang:
+        mi.language = lang
     mi.publisher = get_text(extra, publisher)
 
     # ISBN
@@ -162,7 +165,7 @@ class GoogleBooks(Source):
     capabilities = frozenset(['identify', 'cover'])
     touched_fields = frozenset(['title', 'authors', 'tags', 'pubdate',
         'comments', 'publisher', 'identifier:isbn', 'rating',
-        'identifier:google']) # language currently disabled
+        'identifier:google', 'languages'])
     supports_gzip_transfer_encoding = True
     cached_cover_url_is_reliable = False
 

src/calibre/ebooks/metadata/sources/identify.py

@@ -484,6 +484,7 @@ def identify(log, abort, # {{{
             'publication dates')
     start_time = time.time()
     results = merge_identify_results(results, log)
+
     log('We have %d merged results, merging took: %.2f seconds' %
             (len(results), time.time() - start_time))
 

src/calibre/ebooks/metadata/sources/overdrive.py

@@ -35,7 +35,7 @@ class OverDrive(Source):
     capabilities = frozenset(['identify', 'cover'])
     touched_fields = frozenset(['title', 'authors', 'tags', 'pubdate',
         'comments', 'publisher', 'identifier:isbn', 'series', 'series_index',
-        'language', 'identifier:overdrive'])
+        'languages', 'identifier:overdrive'])
     has_html_comments = True
     supports_gzip_transfer_encoding = False
     cached_cover_url_is_reliable = True
@@ -421,8 +421,10 @@ class OverDrive(Source):
                 pass
         if lang:
             lang = lang[0].strip().lower()
-            mi.language = {'english':'en', 'french':'fr', 'german':'de',
+            lang = {'english':'eng', 'french':'fra', 'german':'deu',
-                    'spanish':'es'}.get(lang, None)
+                    'spanish':'spa'}.get(lang, None)
+            if lang:
+                mi.language = lang
 
         if ebook_isbn:
             #print "ebook isbn is "+str(ebook_isbn[0])

src/calibre/ebooks/metadata/sources/ozon.py

@@ -0,0 +1,442 @@
+# -*- coding: utf-8 -*-
+from __future__ import (unicode_literals, division, absolute_import, print_function)
+
+__license__ = 'GPL 3'
+__copyright__ = '2011, Roman Mukhin <ramses_ru at hotmail.com>'
+__docformat__ = 'restructuredtext en'
+
+import re
+import urllib2
+import datetime
+from urllib import quote_plus
+from Queue import Queue, Empty
+from lxml import etree, html
+from calibre import as_unicode
+
+from calibre.ebooks.chardet import xml_to_unicode
+
+from calibre.ebooks.metadata import check_isbn
+from calibre.ebooks.metadata.sources.base import Source
+from calibre.ebooks.metadata.book.base import Metadata
+
+class Ozon(Source):
+    name = 'OZON.ru'
+    description = _('Downloads metadata and covers from OZON.ru')
+
+    capabilities = frozenset(['identify', 'cover'])
+
+    touched_fields = frozenset(['title', 'authors', 'identifier:isbn', 'identifier:ozon',
+                               'publisher', 'pubdate', 'comments', 'series', 'rating', 'language'])
+    # Test purpose only, test function does not like when sometimes some filed are empty
+    #touched_fields = frozenset(['title', 'authors', 'identifier:isbn', 'identifier:ozon',
+    #                          'publisher', 'pubdate', 'comments'])
+
+    supports_gzip_transfer_encoding = True
+    has_html_comments = True
+
+    ozon_url = 'http://www.ozon.ru'
+
+    # match any ISBN10/13. From "Regular Expressions Cookbook"
+    isbnPattern = r'(?:ISBN(?:-1[03])?:? )?(?=[-0-9 ]{17}|'\
+             '[-0-9X ]{13}|[0-9X]{10})(?:97[89][- ]?)?[0-9]{1,5}[- ]?'\
+             '(?:[0-9]+[- ]?){2}[0-9X]'
+    isbnRegex = re.compile(isbnPattern)
+
+    def get_book_url(self, identifiers): # {{{
+        ozon_id = identifiers.get('ozon', None)
+        res = None
+        if ozon_id:
+            url = '{}/context/detail/id/{}?partner={}'.format(self.ozon_url, urllib2.quote(ozon_id), _get_affiliateId())
+            res = ('ozon', ozon_id, url)
+        return res
+    # }}}
+
+    def create_query(self, log, title=None, authors=None, identifiers={}): # {{{
+        # div_book -> search only books, ebooks and audio books
+        search_url = self.ozon_url + '/webservice/webservice.asmx/SearchWebService?searchContext=div_book&searchText='
+
+        isbn = _format_isbn(log, identifiers.get('isbn', None))
+        # TODO: format isbn!
+        qItems = set([isbn, title])
+        if authors:
+            qItems |= frozenset(authors)
+        qItems.discard(None)
+        qItems.discard('')
+        qItems = map(_quoteString, qItems)
+
+        q = ' '.join(qItems).strip()
+        log.info(u'search string: ' + q)
+
+        if isinstance(q, unicode):
+            q = q.encode('utf-8')
+        if not q:
+            return None
+
+        search_url += quote_plus(q)
+        log.debug(u'search url: %r'%search_url)
+
+        return search_url
+    # }}}
+
+    def identify(self, log, result_queue, abort, title=None, authors=None, # {{{
+            identifiers={}, timeout=30):
+        if not self.is_configured():
+            return
+        query = self.create_query(log, title=title, authors=authors, identifiers=identifiers)
+        if not query:
+            err = 'Insufficient metadata to construct query'
+            log.error(err)
+            return err
+
+        try:
+            raw = self.browser.open_novisit(query).read()
+
+        except Exception as e:
+            log.exception(u'Failed to make identify query: %r'%query)
+            return as_unicode(e)
+
+        try:
+            parser = etree.XMLParser(recover=True, no_network=True)
+            feed = etree.fromstring(xml_to_unicode(raw, strip_encoding_pats=True, assume_utf8=True)[0], parser=parser)
+            entries = feed.xpath('//*[local-name() = "SearchItems"]')
+            if entries:
+                metadata = self.get_metadata(log, entries, title, authors, identifiers)
+                self.get_all_details(log, metadata, abort, result_queue, identifiers, timeout)
+        except Exception as e:
+            log.exception('Failed to parse identify results')
+            return as_unicode(e)
+
+    # }}}
+
+    def get_metadata(self, log, entries, title, authors, identifiers): # {{{
+        title = unicode(title).upper() if title else ''
+        authors = map(unicode.upper, map(unicode, authors)) if authors else None
+        ozon_id = identifiers.get('ozon', None)
+
+        unk = unicode(_('Unknown')).upper()
+
+        if title == unk:
+            title = None
+
+        if authors == [unk]:
+            authors = None
+
+        def in_authors(authors, miauthors):
+            for author in authors:
+                for miauthor in miauthors:
+                    if author in miauthor: return True
+            return None
+
+        def ensure_metadata_match(mi): # {{{
+            match = True
+            if title:
+                mititle = unicode(mi.title).upper() if mi.title else ''
+                match = title in mititle
+            if match and authors:
+                miauthors = map(unicode.upper, map(unicode, mi.authors)) if mi.authors else []
+                match = in_authors(authors, miauthors)
+
+            if match and ozon_id:
+                mozon_id = mi.identifiers['ozon']
+                match = ozon_id == mozon_id
+
+            return match
+
+        metadata = []
+        for i, entry in enumerate(entries):
+            mi = self.to_metadata(log, entry)
+            mi.source_relevance = i
+            if ensure_metadata_match(mi):
+                metadata.append(mi)
+                # log.debug(u'added metadata %s %s. '%(mi.title, mi.authors))
+            else:
+                log.debug(u'skipped metadata %s %s. (does not match the query)'%(mi.title, mi.authors))
+        return metadata
+    # }}}
+
+    def get_all_details(self, log, metadata, abort, result_queue, identifiers, timeout): # {{{
+        req_isbn = identifiers.get('isbn', None)
+
+        for mi in metadata:
+            if abort.is_set():
+                break
+            try:
+                ozon_id = mi.identifiers['ozon']
+
+                try:
+                    self.get_book_details(log, mi, timeout)
+                except:
+                    log.exception(u'Failed to get details for metadata: %s'%mi.title)
+
+                all_isbns = getattr(mi, 'all_isbns', [])
+                if req_isbn and all_isbns and check_isbn(req_isbn) not in all_isbns:
+                    log.debug(u'skipped, no requested ISBN %s found'%req_isbn)
+                    continue
+
+                for isbn in all_isbns:
+                    self.cache_isbn_to_identifier(isbn, ozon_id)
+
+                if mi.ozon_cover_url:
+                    self.cache_identifier_to_cover_url(ozon_id, mi.ozon_cover_url)
+
+                self.clean_downloaded_metadata(mi)
+                result_queue.put(mi)
+            except:
+                log.exception(u'Failed to get details for metadata: %s'%mi.title)
+    # }}}
+
+    def to_metadata(self, log, entry): # {{{
+        xp_template = 'normalize-space(./*[local-name() = "{0}"]/text())'
+
+        title = entry.xpath(xp_template.format('Name'))
+        author = entry.xpath(xp_template.format('Author'))
+        mi = Metadata(title, author.split(','))
+
+        ozon_id = entry.xpath(xp_template.format('ID'))
+        mi.identifiers = {'ozon':ozon_id}
+
+        mi.comments = entry.xpath(xp_template.format('Annotation'))
+
+        mi.ozon_cover_url = None
+        cover = entry.xpath(xp_template.format('Picture'))
+        if cover:
+            mi.ozon_cover_url = _translateToBigCoverUrl(cover)
+
+        rating = entry.xpath(xp_template.format('ClientRatingValue'))
+        if rating:
+            try:
+                #'rating',     A floating point number between 0 and 10
+                # OZON raion N of 5, calibre of 10, but there is a bug? in identify
+                mi.rating = float(rating)
+            except:
+                pass
+            rating
+        return mi
+    # }}}
+
+    def get_cached_cover_url(self, identifiers): # {{{
+        url = None
+        ozon_id = identifiers.get('ozon', None)
+        if ozon_id is None:
+            isbn = identifiers.get('isbn', None)
+            if isbn is not None:
+                ozon_id = self.cached_isbn_to_identifier(isbn)
+        if ozon_id is not None:
+            url = self.cached_identifier_to_cover_url(ozon_id)
+        return url
+    # }}}
+
+    def download_cover(self, log, result_queue, abort, title=None, authors=None, identifiers={}, timeout=30): # {{{
+        cached_url = self.get_cached_cover_url(identifiers)
+        if cached_url is None:
+            log.debug('No cached cover found, running identify')
+            rq = Queue()
+            self.identify(log, rq, abort, title=title, authors=authors, identifiers=identifiers)
+            if abort.is_set():
+                return
+            results = []
+            while True:
+                try:
+                    results.append(rq.get_nowait())
+                except Empty:
+                    break
+            results.sort(key=self.identify_results_keygen(title=title, authors=authors, identifiers=identifiers))
+            for mi in results:
+                cached_url = self.get_cached_cover_url(mi.identifiers)
+                if cached_url is not None:
+                    break
+
+        if cached_url is None:
+            log.info('No cover found')
+            return
+
+        if abort.is_set():
+            return
+
+        log.debug('Downloading cover from:', cached_url)
+        try:
+            cdata = self.browser.open_novisit(cached_url, timeout=timeout).read()
+            if cdata:
+                result_queue.put((self, cdata))
+        except Exception as e:
+            log.exception(u'Failed to download cover from: %s'%cached_url)
+            return as_unicode(e)
+    # }}}
+
+    def get_book_details(self, log, metadata, timeout): # {{{
+        url = self.get_book_url(metadata.get_identifiers())[2]
+
+        raw = self.browser.open_novisit(url, timeout=timeout).read()
+        doc = html.fromstring(raw)
+
+        # series
+        xpt = u'normalize-space(//div[@class="frame_content"]//div[contains(normalize-space(text()), "Серия:")]//a/@title)'
+        series = doc.xpath(xpt)
+        if series:
+            metadata.series = series
+
+        xpt = u'substring-after(//meta[@name="description"]/@content, "ISBN")'
+        isbn_str = doc.xpath(xpt)
+        if isbn_str:
+            all_isbns = [check_isbn(isbn) for isbn in self.isbnRegex.findall(isbn_str) if check_isbn(isbn)]
+            if all_isbns:
+                metadata.all_isbns = all_isbns
+                metadata.isbn = all_isbns[0]
+
+        xpt = u'//div[@class="frame_content"]//div[contains(normalize-space(text()), "Издатель")]//a[@title="Издательство"]'
+        publishers = doc.xpath(xpt)
+        if publishers:
+            metadata.publisher = publishers[0].text
+
+            xpt = u'string(../text()[contains(., "г.")])'
+            yearIn = publishers[0].xpath(xpt)
+            if yearIn:
+                matcher = re.search(r'\d{4}', yearIn)
+                if matcher:
+                    year = int(matcher.group(0))
+                    # only year is available, so use 1-st of Jan
+                    metadata.pubdate = datetime.datetime(year, 1, 1) #<- failed comparation in identify.py
+                    #metadata.pubdate = datetime(year, 1, 1)
+            xpt = u'substring-after(string(../text()[contains(., "Язык")]), ": ")'
+            displLang = publishers[0].xpath(xpt)
+            lang_code =_translageLanguageToCode(displLang)
+            if lang_code:
+                metadata.language = lang_code
+
+        # overwrite comments from HTML if any
+        # tr/td[contains(.//text(), "От издателя")] -> does not work, why?
+        xpt = u'//div[contains(@class, "detail")]//tr/td//text()[contains(., "От издателя")]'\
+              u'/ancestor::tr[1]/following-sibling::tr[1]/td[contains(./@class, "description")][1]'
+        comment_elem = doc.xpath(xpt)
+        if comment_elem:
+            comments = unicode(etree.tostring(comment_elem[0]))
+            if comments:
+                # cleanup root tag, TODO: remove tags like object/embeded
+                comments = re.sub(r'^<td.+?>|</td>.+?$', u'', comments).strip()
+                if comments:
+                    metadata.comments = comments
+        else:
+            log.debug('No book description found in HTML')
+    # }}}
+
+def _quoteString(str): # {{{
+    return '"' + str + '"' if str and str.find(' ') != -1 else str
+# }}}
+
+# TODO: make customizable
+def _translateToBigCoverUrl(coverUrl): # {{{
+    # http://www.ozon.ru/multimedia/books_covers/small/1002986468.gif
+    # http://www.ozon.ru/multimedia/books_covers/1002986468.jpg
+
+    m = re.match(r'^(.+\/)small\/(.+\.).+$', coverUrl)
+    if m:
+        coverUrl = m.group(1) + m.group(2) + 'jpg'
+    return coverUrl
+# }}}
+
+def _get_affiliateId(): # {{{
+    import random
+
+    aff_id = 'romuk'
+    # Use Kovid's affiliate id 30% of the time.
+    if random.randint(1, 10) in (1, 2, 3):
+        aff_id = 'kovidgoyal'
+    return aff_id
+# }}}
+
+# for now only RUS ISBN are supported
+#http://ru.wikipedia.org/wiki/ISBN_российских_издательств
+isbn_pat = re.compile(r"""
+    ^
+    (\d{3})?            # match GS1 Prefix for ISBN13
+    (5)                 # group identifier for rRussian-speaking countries
+    (                   # begin variable length for Publisher
+        [01]\d{1}|      # 2x
+        [2-6]\d{2}|     # 3x
+        7\d{3}|         # 4x (starting with 7)
+        8[0-4]\d{2}|    # 4x (starting with 8)
+        9[2567]\d{2}|   # 4x (starting with 9)
+        99[26]\d{1}|    # 4x (starting with 99)
+        8[5-9]\d{3}|    # 5x (starting with 8)
+        9[348]\d{3}|    # 5x (starting with 9)
+        900\d{2}|       # 5x (starting with 900)
+        91[0-8]\d{2}|   # 5x (starting with 91)
+        90[1-9]\d{3}|   # 6x (starting with 90)
+        919\d{3}|       # 6x (starting with 919)
+        99[^26]\d{4}    # 7x (starting with 99)
+    )                   # end variable length for Publisher
+    (\d+)               # Title
+    ([\dX])             # Check digit
+    $
+""", re.VERBOSE)
+
+def _format_isbn(log, isbn):  # {{{
+    res = check_isbn(isbn)
+    if res:
+        m = isbn_pat.match(res)
+        if m:
+            res = '-'.join([g for g in m.groups() if g])
+        else:
+            log.error('cannot format isbn %s'%isbn)
+    return res
+# }}}
+
+def _translageLanguageToCode(displayLang): # {{{
+    displayLang = unicode(displayLang).strip() if displayLang else None
+    langTbl = {  None: 'ru',
+                u'Немецкий': 'de',
+                u'Английский': 'en',
+                u'Французский': 'fr',
+                u'Итальянский': 'it',
+                u'Испанский': 'es',
+                u'Китайский': 'zh',
+                u'Японский': 'ja' }
+    return langTbl.get(displayLang, None)
+# }}}
+
+if __name__ == '__main__': # tests {{{
+    # To run these test use: calibre-debug -e src/calibre/ebooks/metadata/sources/ozon.py
+    # comment some touched_fields before run thoses tests
+    from calibre.ebooks.metadata.sources.test import (test_identify_plugin,
+            title_test, authors_test, isbn_test)
+
+
+    test_identify_plugin(Ozon.name,
+        [
+
+            (
+                {'identifiers':{'isbn': '9785916572629'} },
+                [title_test(u'На все четыре стороны', exact=True),
+                 authors_test([u'А. А. Гилл'])]
+            ),
+            (
+                {'identifiers':{}, 'title':u'Der Himmel Kennt Keine Gunstlinge',
+                    'authors':[u'Erich Maria Remarque']},
+                [title_test(u'Der Himmel Kennt Keine Gunstlinge', exact=True),
+                 authors_test([u'Erich Maria Remarque'])]
+            ),
+            (
+                {'identifiers':{ }, 'title':u'Метро 2033',
+                    'authors':[u'Дмитрий Глуховский']},
+                [title_test(u'Метро 2033', exact=False)]
+            ),
+            (
+                {'identifiers':{'isbn': '9785170727209'}, 'title':u'Метро 2033',
+                    'authors':[u'Дмитрий Глуховский']},
+                [title_test(u'Метро 2033', exact=True),
+                    authors_test([u'Дмитрий Глуховский']),
+                    isbn_test('9785170727209')]
+            ),
+            (
+                {'identifiers':{'isbn': '5-699-13613-4'}, 'title':u'Метро 2033',
+                    'authors':[u'Дмитрий Глуховский']},
+                [title_test(u'Метро 2033', exact=True),
+                 authors_test([u'Дмитрий Глуховский'])]
+            ),
+            (
+                {'identifiers':{}, 'title':u'Метро',
+                    'authors':[u'Глуховский']},
+                [title_test(u'Метро', exact=False)]
+            ),
+    ])
+# }}}

src/calibre/ebooks/mobi/debug.py

@@ -1421,7 +1421,7 @@ class MOBIFile(object): # {{{
                 except:
                     pass
             if fmt is not None:
-                self.image_records.append(ImageRecord(i, r, fmt))
+                self.image_records.append(ImageRecord(len(self.image_records)+1, r, fmt))
             else:
                 self.binary_records.append(BinaryRecord(i, r))
 

src/calibre/ebooks/mobi/langcodes.py

@@ -4,6 +4,7 @@ __copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
 __docformat__ = 'restructuredtext en'
 
 from struct import pack
+from calibre.utils.localization import lang_as_iso639_1
 
 lang_codes = {
         }
@@ -314,7 +315,8 @@ def iana2mobi(icode):
         subtags = list(icode.split('-'))
         while len(subtags) > 0:
             lang = subtags.pop(0).lower()
-            if lang in IANA_MOBI:
+            lang = lang_as_iso639_1(lang)
+            if lang and lang in IANA_MOBI:
                 langdict = IANA_MOBI[lang]
                 break
 

src/calibre/ebooks/mobi/utils.py

@@ -314,6 +314,8 @@ def detect_periodical(toc, log=None):
     Detect if the TOC object toc contains a periodical that conforms to the
     structure required by kindlegen to generate a periodical.
     '''
+    if toc.count() < 1 or not toc[0].klass == 'periodical':
+        return False
     for node in toc.iterdescendants():
         if node.depth() == 1 and node.klass != 'article':
             if log is not None:

src/calibre/ebooks/mobi/writer2/indexer.py

@@ -109,20 +109,6 @@ class TAGX(object): # {{{
         list(map(self.add_tag, (11, 0)))
         return self.header(1) + bytes(self.byts)
 
-
-
-class TAGX_BOOK(TAGX):
-    BITMASKS = dict(TAGX.BITMASKS)
-    BITMASKS.update({x:(1 << i) for i, x in enumerate([1, 2, 3, 4, 21, 22, 23])})
-
-    @property
-    def hierarchical_book(self):
-        '''
-        TAGX block for the primary index header of a hierarchical book
-        '''
-        list(map(self.add_tag, (1, 2, 3, 4, 21, 22, 23, 0)))
-        return self.header(1) + bytes(self.byts)
-
     @property
     def flat_book(self):
         '''
@@ -244,17 +230,6 @@ class IndexEntry(object):
         ans = buf.getvalue()
         return ans
 
-class BookIndexEntry(IndexEntry):
-
-    @property
-    def entry_type(self):
-        tagx = TAGX_BOOK()
-        ans = 0
-        for tag in self.tag_nums:
-            ans |= tagx.BITMASKS[tag]
-        return ans
-
-
 class PeriodicalIndexEntry(IndexEntry):
 
     def __init__(self, offset, label_offset, class_offset, depth):
@@ -305,9 +280,7 @@ class TBS(object): # {{{
     def __init__(self, data, is_periodical, first=False, section_map={},
             after_first=False):
         self.section_map = section_map
-        #import pprint
+
-        #pprint.pprint(data)
-        #print()
         if is_periodical:
             # The starting bytes.
             # The value is zero which I think indicates the periodical
@@ -420,6 +393,8 @@ class TBS(object): # {{{
                 first_article = articles[0]
                 last_article = articles[-1]
                 num = len(articles)
+                last_article_ends = (last_article in data['ends'] or
+                        last_article in data['completes'])
 
                 try:
                     next_sec = sections[i+1]
@@ -440,6 +415,19 @@ class TBS(object): # {{{
                 if next_sec is not None:
                     buf.write(encode_tbs(last_article.index-next_sec.index,
                         {0b1000: 0}))
+
+
+                # If a section TOC starts and extends into the next record add
+                # a trailing vwi. We detect this by TBS type==3, processing last
+                # section present in the record, and the last article in that
+                # section either ends or completes and doesn't finish
+                # on the last byte of the record.
+                elif (typ == self.type_011 and last_article_ends and
+                      ((last_article.offset+last_article.size) % RECORD_SIZE > 0)
+                     ):
+                    buf.write(encode_tbs(last_article.index-section.index-1,
+                        {0b1000: 0}))
+
         else:
             buf.write(encode_tbs(spanner.index - parent_section_index,
                 {0b0001: 0}))
@@ -447,7 +435,26 @@ class TBS(object): # {{{
         self.bytestring = buf.getvalue()
 
     def book_tbs(self, data, first):
-        self.bytestring = b''
+        spanner = data['spans']
+        if spanner is not None:
+            self.bytestring = encode_tbs(spanner.index, {0b010: 0, 0b001: 0},
+                    flag_size=3)
+        else:
+            starts, completes, ends = (data['starts'], data['completes'],
+                                        data['ends'])
+            if (not completes and (
+                (len(starts) == 1 and not ends) or (len(ends) == 1 and not
+                    starts))):
+                node = starts[0] if starts else ends[0]
+                self.bytestring = encode_tbs(node.index, {0b010: 0}, flag_size=3)
+            else:
+                nodes = []
+                for x in (starts, completes, ends):
+                    nodes.extend(x)
+                nodes.sort(key=lambda x:x.index)
+                self.bytestring = encode_tbs(nodes[0].index, {0b010:0,
+                    0b100: len(nodes)}, flag_size=3)
+
 # }}}
 
 class Indexer(object): # {{{
@@ -518,6 +525,7 @@ class Indexer(object): # {{{
         for i in indices:
             offsets.append(buf.tell())
             buf.write(i.bytestring)
+
         index_block = align_block(buf.getvalue())
 
         # Write offsets to index entries as an IDXT block
@@ -557,9 +565,7 @@ class Indexer(object): # {{{
             tagx_block = TAGX().secondary
         else:
             tagx_block = (TAGX().periodical if self.is_periodical else
-                            (TAGX_BOOK().hierarchical_book if
+                                TAGX().flat_book)
-                                self.book_has_subchapters else
-                                TAGX_BOOK().flat_book))
         header_length = 192
 
         # Ident 0 - 4
@@ -645,15 +651,13 @@ class Indexer(object): # {{{
     # }}}
 
     def create_book_index(self): # {{{
-        self.book_has_subchapters = False
         indices = []
-        seen, sub_seen = set(), set()
+        seen = set()
         id_offsets = self.serializer.id_offsets
 
-        # Flatten toc to contain only chapters and subchapters
+        # Flatten toc so that chapter to chapter jumps work with all sub
-        # Anything deeper than a subchapter is made into a subchapter
+        # chapter levels as well
-        chapters = []
+        for node in self.oeb.toc.iterdescendants():
-        for node in self.oeb.toc:
             try:
                 offset = id_offsets[node.href]
                 label = self.cncx[node.title]
@@ -666,77 +670,33 @@ class Indexer(object): # {{{
                 continue
             seen.add(offset)
 
-            subchapters = []
+            indices.append(IndexEntry(offset, label))
-            chapters.append((offset, label, subchapters))
 
-            for descendant in node.iterdescendants():
+        indices.sort(key=lambda x:x.offset)
-                try:
-                    offset = id_offsets[descendant.href]
-                    label = self.cncx[descendant.title]
-                except:
-                    self.log.warn('TOC item %s [%s] not found in document'%(
-                        descendant.title, descendant.href))
-                    continue
 
-                if offset in sub_seen:
+        # Set lengths
-                    continue
+        for i, index in enumerate(indices):
-                sub_seen.add(offset)
+            try:
-                subchapters.append((offset, label))
+                next_offset = indices[i+1].offset
+            except:
+                next_offset = self.serializer.body_end_offset
+            index.length = next_offset - index.offset
 
-            subchapters.sort(key=lambda x:x[0])
 
-        chapters.sort(key=lambda x:x[0])
+        # Remove empty indices
+        indices = [x for x in indices if x.length > 0]
 
-        chapters = [(BookIndexEntry(x[0], x[1]), [
+        # Reset lengths in case any were removed
-            BookIndexEntry(y[0], y[1]) for y in x[2]]) for x in chapters]
+        for i, index in enumerate(indices):
+            try:
+                next_offset = indices[i+1].offset
+            except:
+                next_offset = self.serializer.body_end_offset
+            index.length = next_offset - index.offset
 
-        def set_length(indices):
+        # Set index values
-            for i, index in enumerate(indices):
+        for index, x in enumerate(indices):
-                try:
+            x.index = index
-                    next_offset = indices[i+1].offset
-                except:
-                    next_offset = self.serializer.body_end_offset
-                index.length = next_offset - index.offset
-
-        # Set chapter and subchapter lengths
-        set_length([x[0] for x in chapters])
-        for x in chapters:
-            set_length(x[1])
-
-        # Remove empty chapters
-        chapters = [x for x in chapters if x[0].length > 0]
-
-        # Remove invalid subchapters
-        for i, x in enumerate(list(chapters)):
-            chapter, subchapters = x
-            ok_subchapters = []
-            for sc in subchapters:
-                if sc.offset < chapter.next_offset and sc.length > 0:
-                    ok_subchapters.append(sc)
-            chapters[i] = (chapter, ok_subchapters)
-
-        # Reset chapter and subchapter lengths in case any were removed
-        set_length([x[0] for x in chapters])
-        for x in chapters:
-            set_length(x[1])
-
-        # Set index and depth values
-        indices = []
-        for index, x in enumerate(chapters):
-            x[0].index = index
-            indices.append(x[0])
-
-        for chapter, subchapters in chapters:
-            for sc in subchapters:
-                index += 1
-                sc.index = index
-                sc.parent_index = chapter.index
-                indices.append(sc)
-                sc.depth = 1
-                self.book_has_subchapters = True
-            if subchapters:
-                chapter.first_child_index = subchapters[0].index
-                chapter.last_child_index = subchapters[-1].index
 
         return indices
 
@@ -772,9 +732,11 @@ class Indexer(object): # {{{
                 continue
             if offset in seen_sec_offsets:
                 continue
+
             seen_sec_offsets.add(offset)
             section = PeriodicalIndexEntry(offset, label, klass, 1)
             section.parent_index = 0
+
             for art in sec:
                 try:
                     offset = id_offsets[art.href]
@@ -830,6 +792,7 @@ class Indexer(object): # {{{
             for art in articles:
                 i += 1
                 art.index = i
+
                 art.parent_index = sec.index
 
         for sec, normalized_articles in normalized_sections:
@@ -905,6 +868,7 @@ class Indexer(object): # {{{
                     'spans':None, 'offset':offset, 'record_number':i+1}
 
             for index in self.indices:
+
                 if index.offset >= next_offset:
                     # Node starts after current record
                     if index.depth == deepest:

src/calibre/ebooks/mobi/writer2/main.py

@@ -97,6 +97,9 @@ class MobiWriter(object):
     # Indexing {{{
     def generate_index(self):
         self.primary_index_record_idx = None
+        if self.oeb.toc.count() < 1:
+            self.log.warn('No TOC, MOBI index not generated')
+            return
         try:
             self.indexer = Indexer(self.serializer, self.last_text_record_idx,
                     len(self.records[self.last_text_record_idx]),
@@ -147,15 +150,19 @@ class MobiWriter(object):
         oeb.logger.info('Serializing images...')
         self.image_records = []
         self.image_map = {}
+        self.masthead_offset = 0
+        index = 1
 
-        mh_href = self.masthead_offset = None
+        mh_href = None
         if 'masthead' in oeb.guide:
             mh_href = oeb.guide['masthead'].href
+            self.image_records.append(None)
+            index += 1
         elif self.is_periodical:
             # Generate a default masthead
-            data = generate_masthead(unicode(self.oeb.metadata('title')[0]))
+            data = generate_masthead(unicode(self.oeb.metadata['title'][0]))
             self.image_records.append(data)
-            self.masthead_offset = 0
+            index += 1
 
         cover_href = self.cover_offset = self.thumbnail_offset = None
         if (oeb.metadata.cover and
@@ -172,13 +179,16 @@ class MobiWriter(object):
                 oeb.logger.warn('Bad image file %r' % item.href)
                 continue
             else:
-                self.image_map[item.href] = len(self.image_records)
+                if mh_href and item.href == mh_href:
-                self.image_records.append(data)
+                    self.image_records[0] = data
+                    continue
 
-                if item.href == mh_href:
+                self.image_records.append(data)
-                    self.masthead_offset = len(self.image_records) - 1
+                self.image_map[item.href] = index
-                elif item.href == cover_href:
+                index += 1
-                    self.cover_offset = len(self.image_records) - 1
+
+                if cover_href and item.href == cover_href:
+                    self.cover_offset = self.image_map[item.href] - 1
                     try:
                         data = rescale_image(item.data, dimen=MAX_THUMB_DIMEN,
                             maxsizeb=MAX_THUMB_SIZE)
@@ -186,10 +196,14 @@ class MobiWriter(object):
                         oeb.logger.warn('Failed to generate thumbnail')
                     else:
                         self.image_records.append(data)
-                        self.thumbnail_offset = len(self.image_records) - 1
+                        self.thumbnail_offset = index - 1
+                        index += 1
             finally:
                 item.unload_data_from_memory()
 
+        if self.image_records and self.image_records[0] is None:
+            raise ValueError('Failed to find masthead image in manifest')
+
     # }}}
 
     # Text {{{
@@ -197,6 +211,7 @@ class MobiWriter(object):
     def generate_text(self):
         self.oeb.logger.info('Serializing markup content...')
         self.serializer = Serializer(self.oeb, self.image_map,
+                self.is_periodical,
                 write_page_breaks_after_item=self.write_page_breaks_after_item)
         text = self.serializer()
         self.text_length = len(text)

src/calibre/ebooks/mobi/writer2/serializer.py

@@ -7,6 +7,8 @@ __license__   = 'GPL v3'
 __copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'
 
+import re
+
 from calibre.ebooks.oeb.base import (OEB_DOCS, XHTML, XHTML_NS, XML_NS,
         namespace, prefixname, urlnormalize)
 from calibre.ebooks.mobi.mobiml import MBP_NS
@@ -19,7 +21,7 @@ from cStringIO import StringIO
 class Serializer(object):
     NSRMAP = {'': None, XML_NS: 'xml', XHTML_NS: '', MBP_NS: 'mbp'}
 
-    def __init__(self, oeb, images, write_page_breaks_after_item=True):
+    def __init__(self, oeb, images, is_periodical, write_page_breaks_after_item=True):
         '''
         Write all the HTML markup in oeb into a single in memory buffer
         containing a single html document with links replaced by offsets into
@@ -35,8 +37,10 @@ class Serializer(object):
         is written after every element of the spine in ``oeb``.
         '''
         self.oeb = oeb
+        # Map of image hrefs to image index in the MOBI file
         self.images = images
         self.logger = oeb.logger
+        self.is_periodical = is_periodical
         self.write_page_breaks_after_item = write_page_breaks_after_item
 
         # If not None, this is a number pointing to the location at which to
@@ -187,13 +191,63 @@ class Serializer(object):
         moved to the end.
         '''
         buf = self.buf
+
+        def serialize_toc_level(tocref, href=None):
+            # add the provided toc level to the output stream
+            # if href is provided add a link ref to the toc level output (e.g. feed_0/index.html)
+            if href is not None:
+                # resolve the section url in id_offsets
+                buf.write('<mbp:pagebreak/>')
+                self.id_offsets[urlnormalize(href)] = buf.tell()
+
+            if tocref.klass == "periodical":
+                buf.write('<div> <div height="1em"></div>')
+            else:
+                buf.write('<div></div> <div> <h2 height="1em"><font size="+2"><b>'+tocref.title+'</b></font></h2> <div height="1em"></div>')
+
+            buf.write('<ul>')
+
+            for tocitem in tocref.nodes:
+                buf.write('<li><a filepos=')
+                itemhref = tocitem.href
+                if tocref.klass == 'periodical':
+                    # This is a section node.
+                    # For periodical toca, the section urls are like r'feed_\d+/index.html'
+                    # We dont want to point to the start of the first article
+                    # so we change the href.
+                    itemhref = re.sub(r'article_\d+/', '', itemhref)
+                self.href_offsets[itemhref].append(buf.tell())
+                buf.write('0000000000')
+                buf.write(' ><font size="+1" color="blue"><b><u>')
+                buf.write(tocitem.title)
+                buf.write('</u></b></font></a></li>')
+
+            buf.write('</ul><div height="1em"></div></div>')
+
         self.anchor_offset = buf.tell()
         buf.write(b'<body>')
         self.body_start_offset = buf.tell()
+
+        if self.is_periodical:
+            top_toc = self.oeb.toc.nodes[0]
+            serialize_toc_level(top_toc)
+
         spine = [item for item in self.oeb.spine if item.linear]
         spine.extend([item for item in self.oeb.spine if not item.linear])
+
         for item in spine:
+
+            if self.is_periodical and item.is_section_start:
+                for section_toc in top_toc.nodes:
+                    if urlnormalize(item.href) == section_toc.href:
+                        # create section url of the form r'feed_\d+/index.html'
+                        section_url = re.sub(r'article_\d+/', '', section_toc.href)
+                        serialize_toc_level(section_toc, section_url)
+                        section_toc.href = section_url
+                        break
+
             self.serialize_item(item)
+
         self.body_end_offset = buf.tell()
         buf.write(b'</body>')
 

src/calibre/ebooks/oeb/transforms/metadata.py

@@ -61,9 +61,11 @@ def meta_info_to_oeb_metadata(mi, m, log, override_input_metadata=False):
             m.add('identifier', val, scheme=typ.upper())
     if override_input_metadata and not set_isbn:
         m.filter('identifier', lambda x: x.scheme.lower() == 'isbn')
-    if not mi.is_null('language'):
+    if not mi.is_null('languages'):
         m.clear('language')
-        m.add('language', mi.language)
+        for lang in mi.languages:
+            if lang and lang.lower() not in ('und', ''):
+                m.add('language', lang)
     if not mi.is_null('series_index'):
         m.clear('series_index')
         m.add('series_index', mi.format_series_index())

src/calibre/gui2/__init__.py

@@ -94,7 +94,7 @@ gprefs.defaults['book_display_fields'] = [
         ('path', True), ('publisher', False), ('rating', False),
         ('author_sort', False), ('sort', False), ('timestamp', False),
         ('uuid', False), ('comments', True), ('id', False), ('pubdate', False),
-        ('last_modified', False), ('size', False),
+        ('last_modified', False), ('size', False), ('languages', False),
         ]
 gprefs.defaults['default_author_link'] = 'http://en.wikipedia.org/w/index.php?search={author}'
 gprefs.defaults['preserve_date_on_ctl'] = True

src/calibre/gui2/book_details.py

@@ -24,6 +24,7 @@ from calibre.gui2 import (config, open_local_file, open_url, pixmap_to_data,
 from calibre.utils.icu import sort_key
 from calibre.utils.formatter import EvalFormatter
 from calibre.utils.date import is_date_undefined
+from calibre.utils.localization import calibre_langcode_to_name
 
 def render_html(mi, css, vertical, widget, all_fields=False): # {{{
     table = render_data(mi, all_fields=all_fields,
@@ -152,6 +153,12 @@ def render_data(mi, use_roman_numbers=True, all_fields=False):
                     authors.append(aut)
             ans.append((field, u'<td class="title">%s</td><td>%s</td>'%(name,
                 u' & '.join(authors))))
+        elif field == 'languages':
+            if not mi.languages:
+                continue
+            names = filter(None, map(calibre_langcode_to_name, mi.languages))
+            ans.append((field, u'<td class="title">%s</td><td>%s</td>'%(name,
+                u', '.join(names))))
         else:
             val = mi.format_field(field)[-1]
             if val is None:

src/calibre/gui2/dialogs/metadata_bulk.py

@@ -134,7 +134,7 @@ class MyBlockingBusy(QDialog): # {{{
             do_autonumber, do_remove_format, remove_format, do_swap_ta, \
             do_remove_conv, do_auto_author, series, do_series_restart, \
             series_start_value, do_title_case, cover_action, clear_series, \
-            pubdate, adddate, do_title_sort = self.args
+            pubdate, adddate, do_title_sort, languages, clear_languages = self.args
 
 
         # first loop: do author and title. These will commit at the end of each
@@ -238,6 +238,12 @@ class MyBlockingBusy(QDialog): # {{{
 
             if do_remove_conv:
                 self.db.delete_conversion_options(id, 'PIPE', commit=False)
+
+            if clear_languages:
+                self.db.set_languages(id, [], notify=False, commit=False)
+            elif languages:
+                self.db.set_languages(id, languages, notify=False, commit=False)
+
         elif self.current_phase == 3:
             # both of these are fast enough to just do them all
             for w in self.cc_widgets:
@@ -329,6 +335,7 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog):
         geom = gprefs.get('bulk_metadata_window_geometry', None)
         if geom is not None:
             self.restoreGeometry(bytes(geom))
+        self.languages.setEditText('')
         self.exec_()
 
     def save_state(self, *args):
@@ -352,6 +359,7 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog):
             self.do_again = True
             self.accept()
 
+    # S&R {{{
     def prepare_search_and_replace(self):
         self.search_for.initialize('bulk_edit_search_for')
         self.replace_with.initialize('bulk_edit_replace_with')
@@ -796,6 +804,7 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog):
             # permanent. Make sure it really is.
             self.db.commit()
             self.model.refresh_ids(list(books_to_refresh))
+    # }}}
 
     def create_custom_column_editors(self):
         w = self.central_widget.widget(1)
@@ -919,6 +928,8 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog):
         do_auto_author = self.auto_author_sort.isChecked()
         do_title_case = self.change_title_to_title_case.isChecked()
         do_title_sort = self.update_title_sort.isChecked()
+        clear_languages = self.clear_languages.isChecked()
+        languages = self.languages.lang_codes
         pubdate = adddate = None
         if self.apply_pubdate.isChecked():
             pubdate = qt_to_dt(self.pubdate.date())
@@ -937,7 +948,7 @@ class MetadataBulkDialog(ResizableDialog, Ui_MetadataBulkDialog):
                 do_autonumber, do_remove_format, remove_format, do_swap_ta,
                 do_remove_conv, do_auto_author, series, do_series_restart,
                 series_start_value, do_title_case, cover_action, clear_series,
-                pubdate, adddate, do_title_sort)
+                pubdate, adddate, do_title_sort, languages, clear_languages)
 
         bb = MyBlockingBusy(_('Applying changes to %d books.\nPhase {0} {1}%%.')
                 %len(self.ids), args, self.db, self.ids,

src/calibre/gui2/dialogs/metadata_bulk.ui

@@ -443,7 +443,7 @@ from the value in the box</string>
              </property>
             </widget>
            </item>
-           <item row="11" column="0">
+           <item row="13" column="0">
             <widget class="QLabel" name="label_5">
              <property name="text">
               <string>Remove &amp;format:</string>
@@ -453,7 +453,7 @@ from the value in the box</string>
              </property>
             </widget>
            </item>
-           <item row="11" column="1">
+           <item row="13" column="1">
             <widget class="QComboBox" name="remove_format">
              <property name="maximumSize">
               <size>
@@ -463,7 +463,7 @@ from the value in the box</string>
              </property>
             </widget>
            </item>
-           <item row="12" column="0">
+           <item row="14" column="0">
             <spacer name="verticalSpacer">
              <property name="orientation">
               <enum>Qt::Vertical</enum>
@@ -479,7 +479,7 @@ from the value in the box</string>
              </property>
             </spacer>
            </item>
-           <item row="13" column="0" colspan="3">
+           <item row="15" column="0" colspan="3">
             <layout class="QHBoxLayout" name="horizontalLayout_3">
              <item>
               <widget class="QCheckBox" name="change_title_to_title_case">
@@ -529,7 +529,7 @@ Future conversion of these books will use the default settings.</string>
              </item>
             </layout>
            </item>
-           <item row="14" column="0" colspan="3">
+           <item row="16" column="0" colspan="3">
             <widget class="QGroupBox" name="groupBox">
              <property name="title">
               <string>Change &amp;cover</string>
@@ -559,7 +559,7 @@ Future conversion of these books will use the default settings.</string>
              </layout>
             </widget>
            </item>
-           <item row="15" column="0">
+           <item row="17" column="0">
             <spacer name="verticalSpacer_2">
              <property name="orientation">
               <enum>Qt::Vertical</enum>
@@ -572,6 +572,29 @@ Future conversion of these books will use the default settings.</string>
              </property>
             </spacer>
            </item>
+           <item row="11" column="0">
+            <widget class="QLabel" name="label_11">
+             <property name="text">
+              <string>&amp;Languages:</string>
+             </property>
+             <property name="alignment">
+              <set>Qt::AlignRight|Qt::AlignTrailing|Qt::AlignVCenter</set>
+             </property>
+             <property name="buddy">
+              <cstring>languages</cstring>
+             </property>
+            </widget>
+           </item>
+           <item row="11" column="1">
+            <widget class="LanguagesEdit" name="languages"/>
+           </item>
+           <item row="11" column="2">
+            <widget class="QCheckBox" name="clear_languages">
+             <property name="text">
+              <string>Remove &amp;all</string>
+             </property>
+            </widget>
+           </item>
           </layout>
          </widget>
          <widget class="QWidget" name="tab">
@@ -1145,6 +1168,11 @@ not multiple and the destination field is multiple</string>
    <extends>QLineEdit</extends>
    <header>widgets.h</header>
   </customwidget>
+  <customwidget>
+   <class>LanguagesEdit</class>
+   <extends>QComboBox</extends>
+   <header>calibre/gui2/languages.h</header>
+  </customwidget>
  </customwidgets>
  <tabstops>
   <tabstop>authors</tabstop>

src/calibre/gui2/keyboard.py

@@ -114,6 +114,8 @@ class Manager(QObject): # {{{
         custom_keys_map = {un:tuple(keys) for un, keys in self.config.get(
             'map', {}).iteritems()}
         self.keys_map = finalize(self.shortcuts, custom_keys_map=custom_keys_map)
+        #import pprint
+        #pprint.pprint(self.keys_map)
 
 # }}}
 
@@ -372,8 +374,8 @@ class Editor(QFrame): # {{{
                     self.current_keys])
         if not current: current = _('None')
 
-        self.use_default.setText(_('Default: %s [Currently not conflicting: %s]')%
+        self.use_default.setText(_('Default: %(deflt)s [Currently not conflicting: %(curr)s]')%
-                (default, current))
+                dict(deflt=default, curr=current))
 
         if shortcut['set_to_default']:
             self.use_default.setChecked(True)

src/calibre/gui2/languages.py

@@ -0,0 +1,71 @@
+#!/usr/bin/env python
+# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__   = 'GPL v3'
+__copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
+__docformat__ = 'restructuredtext en'
+
+from calibre.gui2.complete import MultiCompleteComboBox
+from calibre.utils.localization import lang_map
+from calibre.utils.icu import sort_key
+
+class LanguagesEdit(MultiCompleteComboBox):
+
+    def __init__(self, parent=None):
+        MultiCompleteComboBox.__init__(self, parent)
+
+        self._lang_map = lang_map()
+        self.names_with_commas = [x for x in self._lang_map.itervalues() if ',' in x]
+        self.comma_map = {k:k.replace(',', '|') for k in self.names_with_commas}
+        self.comma_rmap = {v:k for k, v in self.comma_map.iteritems()}
+        self._rmap = {v:k for k,v in self._lang_map.iteritems()}
+
+        all_items = sorted(self._lang_map.itervalues(),
+            key=sort_key)
+        self.update_items_cache(all_items)
+        for item in all_items:
+            self.addItem(item)
+
+    @property
+    def vals(self):
+        raw = unicode(self.lineEdit().text())
+        for k, v in self.comma_map.iteritems():
+            raw = raw.replace(k, v)
+        parts = [x.strip() for x in raw.split(',')]
+        return [self.comma_rmap.get(x, x) for x in parts]
+
+    @dynamic_property
+    def lang_codes(self):
+
+        def fget(self):
+            vals = self.vals
+            ans = []
+            for name in vals:
+                if name:
+                    code = self._rmap.get(name, None)
+                    if code is not None:
+                        ans.append(code)
+            return ans
+
+        def fset(self, lang_codes):
+            ans = []
+            for lc in lang_codes:
+                name = self._lang_map.get(lc, None)
+                if name is not None:
+                    ans.append(name)
+            self.setEditText(', '.join(ans))
+
+        return property(fget=fget, fset=fset)
+
+    def validate(self):
+        vals = self.vals
+        bad = []
+        for name in vals:
+            if name:
+                code = self._rmap.get(name, None)
+                if code is None:
+                    bad.append(name)
+        return bad
+

src/calibre/gui2/library/delegates.py

@@ -23,6 +23,7 @@ from calibre.utils.formatter import validation_formatter
 from calibre.utils.icu import sort_key
 from calibre.gui2.dialogs.comments_dialog import CommentsDialog
 from calibre.gui2.dialogs.template_dialog import TemplateDialog
+from calibre.gui2.languages import LanguagesEdit
 
 
 class RatingDelegate(QStyledItemDelegate): # {{{
@@ -155,7 +156,7 @@ class TextDelegate(QStyledItemDelegate): # {{{
     def __init__(self, parent):
         '''
         Delegate for text data. If auto_complete_function needs to return a list
-        of text items to auto-complete with. The funciton is None no
+        of text items to auto-complete with. If the function is None no
         auto-complete will be used.
         '''
         QStyledItemDelegate.__init__(self, parent)
@@ -229,6 +230,20 @@ class CompleteDelegate(QStyledItemDelegate): # {{{
             QStyledItemDelegate.setModelData(self, editor, model, index)
 # }}}
 
+class LanguagesDelegate(QStyledItemDelegate): # {{{
+
+    def createEditor(self, parent, option, index):
+        editor = LanguagesEdit(parent)
+        ct = index.data(Qt.DisplayRole).toString()
+        editor.setEditText(ct)
+        editor.lineEdit().selectAll()
+        return editor
+
+    def setModelData(self, editor, model, index):
+        val = ','.join(editor.lang_codes)
+        model.setData(index, QVariant(val), Qt.EditRole)
+# }}}
+
 class CcDateDelegate(QStyledItemDelegate): # {{{
     '''
     Delegate for custom columns dates. Because this delegate stores the

src/calibre/gui2/library/models.py

@@ -25,6 +25,7 @@ from calibre.library.caches import (_match, CONTAINS_MATCH, EQUALS_MATCH,
 from calibre import strftime, isbytestring
 from calibre.constants import filesystem_encoding, DEBUG
 from calibre.gui2.library import DEFAULT_SORT
+from calibre.utils.localization import calibre_langcode_to_name
 
 def human_readable(size, precision=1):
     """ Convert a size in bytes into megabytes """
@@ -64,6 +65,7 @@ class BooksModel(QAbstractTableModel): # {{{
                         'tags'      : _("Tags"),
                         'series'    : ngettext("Series", 'Series', 1),
                         'last_modified' : _('Modified'),
+                        'languages' : _('Languages'),
     }
 
     def __init__(self, parent=None, buffer=40):
@@ -71,7 +73,8 @@ class BooksModel(QAbstractTableModel): # {{{
         self.db = None
         self.book_on_device = None
         self.editable_cols = ['title', 'authors', 'rating', 'publisher',
-                              'tags', 'series', 'timestamp', 'pubdate']
+                              'tags', 'series', 'timestamp', 'pubdate',
+                              'languages']
         self.default_image = default_image()
         self.sorted_on = DEFAULT_SORT
         self.sort_history = [self.sorted_on]
@@ -540,6 +543,13 @@ class BooksModel(QAbstractTableModel): # {{{
             else:
                 return None
 
+        def languages(r, idx=-1):
+            lc = self.db.data[r][idx]
+            if lc:
+                langs = [calibre_langcode_to_name(l.strip()) for l in lc.split(',')]
+                return QVariant(', '.join(langs))
+            return None
+
         def tags(r, idx=-1):
             tags = self.db.data[r][idx]
             if tags:
@@ -641,6 +651,8 @@ class BooksModel(QAbstractTableModel): # {{{
                                 siix=self.db.field_metadata['series_index']['rec_index']),
                    'ondevice' : functools.partial(text_type,
                                 idx=self.db.field_metadata['ondevice']['rec_index'], mult=None),
+                   'languages': functools.partial(languages,
+                                idx=self.db.field_metadata['languages']['rec_index']),
                    }
 
         self.dc_decorator = {
@@ -884,6 +896,9 @@ class BooksModel(QAbstractTableModel): # {{{
                     if val.isNull() or not val.isValid():
                         return False
                     self.db.set_pubdate(id, qt_to_dt(val, as_utc=False))
+                elif column == 'languages':
+                    val = val.split(',')
+                    self.db.set_languages(id, val)
                 else:
                     books_to_refresh |= self.db.set(row, column, val,
                                                     allow_case_change=True)

src/calibre/gui2/library/views.py

@@ -8,14 +8,14 @@ __docformat__ = 'restructuredtext en'
 import os
 from functools import partial
 
-from PyQt4.Qt import QTableView, Qt, QAbstractItemView, QMenu, pyqtSignal, \
+from PyQt4.Qt import (QTableView, Qt, QAbstractItemView, QMenu, pyqtSignal,
-    QModelIndex, QIcon, QItemSelection, QMimeData, QDrag, QApplication, \
+    QModelIndex, QIcon, QItemSelection, QMimeData, QDrag, QApplication,
-    QPoint, QPixmap, QUrl, QImage, QPainter, QColor, QRect
+    QPoint, QPixmap, QUrl, QImage, QPainter, QColor, QRect)
 
-from calibre.gui2.library.delegates import RatingDelegate, PubDateDelegate, \
+from calibre.gui2.library.delegates import (RatingDelegate, PubDateDelegate,
-    TextDelegate, DateDelegate, CompleteDelegate, CcTextDelegate, \
+    TextDelegate, DateDelegate, CompleteDelegate, CcTextDelegate,
-    CcBoolDelegate, CcCommentsDelegate, CcDateDelegate, CcTemplateDelegate, \
+    CcBoolDelegate, CcCommentsDelegate, CcDateDelegate, CcTemplateDelegate,
-    CcEnumDelegate, CcNumberDelegate
+    CcEnumDelegate, CcNumberDelegate, LanguagesDelegate)
 from calibre.gui2.library.models import BooksModel, DeviceBooksModel
 from calibre.utils.config import tweaks, prefs
 from calibre.gui2 import error_dialog, gprefs
@@ -85,6 +85,7 @@ class BooksView(QTableView): # {{{
         self.pubdate_delegate = PubDateDelegate(self)
         self.last_modified_delegate = DateDelegate(self,
                 tweak_name='gui_last_modified_display_format')
+        self.languages_delegate = LanguagesDelegate(self)
         self.tags_delegate = CompleteDelegate(self, ',', 'all_tags')
         self.authors_delegate = CompleteDelegate(self, '&', 'all_author_names', True)
         self.cc_names_delegate = CompleteDelegate(self, '&', 'all_custom', True)
@@ -306,6 +307,7 @@ class BooksView(QTableView): # {{{
         state['hidden_columns'] = [cm[i] for i in  range(h.count())
                 if h.isSectionHidden(i) and cm[i] != 'ondevice']
         state['last_modified_injected'] = True
+        state['languages_injected'] = True
         state['sort_history'] = \
             self.cleanup_sort_history(self.model().sort_history)
         state['column_positions'] = {}
@@ -390,7 +392,7 @@ class BooksView(QTableView): # {{{
 
     def get_default_state(self):
         old_state = {
-                'hidden_columns': ['last_modified'],
+                'hidden_columns': ['last_modified', 'languages'],
                 'sort_history':[DEFAULT_SORT],
                 'column_positions': {},
                 'column_sizes': {},
@@ -399,6 +401,7 @@ class BooksView(QTableView): # {{{
                     'timestamp':'center',
                     'pubdate':'center'},
                 'last_modified_injected': True,
+                'languages_injected': True,
                 }
         h = self.column_header
         cm = self.column_map
@@ -430,11 +433,20 @@ class BooksView(QTableView): # {{{
                     if ans is not None:
                         db.prefs[name] = ans
                 else:
+                    injected = False
                     if not ans.get('last_modified_injected', False):
+                        injected = True
                         ans['last_modified_injected'] = True
                         hc = ans.get('hidden_columns', [])
                         if 'last_modified' not in hc:
                             hc.append('last_modified')
+                    if not ans.get('languages_injected', False):
+                        injected = True
+                        ans['languages_injected'] = True
+                        hc = ans.get('hidden_columns', [])
+                        if 'languages' not in hc:
+                            hc.append('languages')
+                    if injected:
                         db.prefs[name] = ans
         return ans
 
@@ -501,7 +513,7 @@ class BooksView(QTableView): # {{{
         for i in range(self.model().columnCount(None)):
             if self.itemDelegateForColumn(i) in (self.rating_delegate,
                     self.timestamp_delegate, self.pubdate_delegate,
-                    self.last_modified_delegate):
+                    self.last_modified_delegate, self.languages_delegate):
                 self.setItemDelegateForColumn(i, self.itemDelegate())
 
         cm = self.column_map

src/calibre/gui2/metadata/basic_widgets.py

@@ -34,6 +34,7 @@ from calibre.library.comments import comments_to_html
 from calibre.gui2.dialogs.tag_editor import TagEditor
 from calibre.utils.icu import strcmp
 from calibre.ptempfile import PersistentTemporaryFile
+from calibre.gui2.languages import LanguagesEdit as LE
 
 def save_dialog(parent, title, msg, det_msg=''):
     d = QMessageBox(parent)
@@ -1133,6 +1134,43 @@ class TagsEdit(MultiCompleteLineEdit): # {{{
 
 # }}}
 
+class LanguagesEdit(LE): # {{{
+
+    LABEL = _('&Languages:')
+    TOOLTIP = _('A comma separated list of languages for this book')
+
+    def __init__(self, *args, **kwargs):
+        LE.__init__(self, *args, **kwargs)
+        self.setToolTip(self.TOOLTIP)
+
+    @dynamic_property
+    def current_val(self):
+        def fget(self): return self.lang_codes
+        def fset(self, val): self.lang_codes = val
+        return property(fget=fget, fset=fset)
+
+    def initialize(self, db, id_):
+        lc = []
+        langs = db.languages(id_, index_is_id=True)
+        if langs:
+            lc = [x.strip() for x in langs.split(',')]
+        self.current_val = self.original_val = lc
+
+    def commit(self, db, id_):
+        bad = self.validate()
+        if bad:
+            error_dialog(self, _('Unknown language'),
+                    ngettext('The language %s is not recognized',
+                        'The languages %s are not recognized', len(bad))%(
+                            ', '.join(bad)),
+                    show=True)
+            return False
+        cv = self.current_val
+        if cv != self.original_val:
+            db.set_languages(id_, cv)
+        return True
+# }}}
+
 class IdentifiersEdit(QLineEdit): # {{{
     LABEL = _('I&ds:')
     BASE_TT = _('Edit the identifiers for this book. '

src/calibre/gui2/metadata/bulk_download.py

@@ -89,6 +89,15 @@ class ConfirmDialog(QDialog):
         self.identify = False
         self.accept()
 
+def split_jobs(ids, batch_size=100):
+    ans = []
+    ids = list(ids)
+    while ids:
+        jids = ids[:batch_size]
+        ans.append(jids)
+        ids = ids[batch_size:]
+    return ans
+
 def start_download(gui, ids, callback):
     d = ConfirmDialog(ids, gui)
     ret = d.exec_()
@@ -96,11 +105,13 @@ def start_download(gui, ids, callback):
     if ret != d.Accepted:
         return
 
-    job = ThreadedJob('metadata bulk download',
+    for batch in split_jobs(ids):
-            _('Download metadata for %d books')%len(ids),
+        job = ThreadedJob('metadata bulk download',
-            download, (ids, gui.current_db, d.identify, d.covers), {}, callback)
+            _('Download metadata for %d books')%len(batch),
-    gui.job_manager.run_threaded_job(job)
+            download, (batch, gui.current_db, d.identify, d.covers), {}, callback)
+        gui.job_manager.run_threaded_job(job)
     gui.status_bar.show_message(_('Metadata download started'), 3000)
+
 # }}}
 
 def get_job_details(job):

src/calibre/gui2/metadata/single.py

@@ -13,19 +13,21 @@ from functools import partial
 from PyQt4.Qt import (Qt, QVBoxLayout, QHBoxLayout, QWidget, QPushButton,
         QGridLayout, pyqtSignal, QDialogButtonBox, QScrollArea, QFont,
         QTabWidget, QIcon, QToolButton, QSplitter, QGroupBox, QSpacerItem,
-        QSizePolicy, QPalette, QFrame, QSize, QKeySequence, QMenu)
+        QSizePolicy, QPalette, QFrame, QSize, QKeySequence, QMenu, QShortcut)
 
 from calibre.ebooks.metadata import authors_to_string, string_to_authors
 from calibre.gui2 import ResizableDialog, error_dialog, gprefs, pixmap_to_data
 from calibre.gui2.metadata.basic_widgets import (TitleEdit, AuthorsEdit,
     AuthorSortEdit, TitleSortEdit, SeriesEdit, SeriesIndexEdit, IdentifiersEdit,
     RatingEdit, PublisherEdit, TagsEdit, FormatsManager, Cover, CommentsEdit,
-    BuddyLabel, DateEdit, PubdateEdit)
+    BuddyLabel, DateEdit, PubdateEdit, LanguagesEdit)
 from calibre.gui2.metadata.single_download import FullFetch
 from calibre.gui2.custom_column_widgets import populate_metadata_page
 from calibre.utils.config import tweaks
 from calibre.ebooks.metadata.book.base import Metadata
 
+BASE_TITLE = _('Edit Metadata')
+
 class MetadataSingleDialogBase(ResizableDialog):
 
     view_format = pyqtSignal(object, object)
@@ -43,6 +45,16 @@ class MetadataSingleDialogBase(ResizableDialog):
     def setupUi(self, *args): # {{{
         self.resize(990, 650)
 
+        self.download_shortcut = QShortcut(self)
+        self.download_shortcut.setKey(QKeySequence('Ctrl+D',
+            QKeySequence.PortableText))
+        p = self.parent()
+        if hasattr(p, 'keyboard'):
+            kname = u'Interface Action: Edit Metadata (Edit Metadata) : menu action : download'
+            sc = p.keyboard.keys_map.get(kname, None)
+            if sc:
+                self.download_shortcut.setKey(sc[0])
+
         self.button_box = QDialogButtonBox(
                 QDialogButtonBox.Ok|QDialogButtonBox.Cancel, Qt.Horizontal,
                 self)
@@ -77,7 +89,7 @@ class MetadataSingleDialogBase(ResizableDialog):
         ll.addSpacing(10)
 
         self.setWindowIcon(QIcon(I('edit_input.png')))
-        self.setWindowTitle(_('Edit Metadata'))
+        self.setWindowTitle(BASE_TITLE)
 
         self.create_basic_metadata_widgets()
 
@@ -183,6 +195,9 @@ class MetadataSingleDialogBase(ResizableDialog):
         self.publisher = PublisherEdit(self)
         self.basic_metadata_widgets.append(self.publisher)
 
+        self.languages = LanguagesEdit(self)
+        self.basic_metadata_widgets.append(self.languages)
+
         self.timestamp = DateEdit(self)
         self.pubdate = PubdateEdit(self)
         self.basic_metadata_widgets.extend([self.timestamp, self.pubdate])
@@ -190,6 +205,7 @@ class MetadataSingleDialogBase(ResizableDialog):
         self.fetch_metadata_button = QPushButton(
                 _('&Download metadata'), self)
         self.fetch_metadata_button.clicked.connect(self.fetch_metadata)
+        self.download_shortcut.activated.connect(self.fetch_metadata_button.click)
         font = self.fmb_font = QFont()
         font.setBold(True)
         self.fetch_metadata_button.setFont(font)
@@ -264,8 +280,11 @@ class MetadataSingleDialogBase(ResizableDialog):
         title = self.title.current_val
         if len(title) > 50:
             title = title[:50] + u'\u2026'
-        self.setWindowTitle(_('Edit Metadata') + ' - ' +
+        self.setWindowTitle(BASE_TITLE + ' - ' +
-                title)
+                title + ' - ' +
+                _(' [%(num)d of %(tot)d]')%dict(num=
+                    self.current_row+1,
+                tot=len(self.row_list)))
 
     def swap_title_author(self, *args):
         title = self.title.current_val
@@ -351,6 +370,8 @@ class MetadataSingleDialogBase(ResizableDialog):
             self.series.current_val = mi.series
             if mi.series_index is not None:
                 self.series_index.current_val = float(mi.series_index)
+        if not mi.is_null('languages'):
+            self.languages.lang_codes = mi.languages
         if mi.comments and mi.comments.strip():
             self.comments.current_val = mi.comments
 
@@ -610,11 +631,13 @@ class MetadataSingleDialog(MetadataSingleDialogBase): # {{{
         create_row2(5, self.pubdate, self.pubdate.clear_button)
         sto(self.pubdate.clear_button, self.publisher)
         create_row2(6, self.publisher)
+        sto(self.publisher, self.languages)
+        create_row2(7, self.languages)
         self.tabs[0].spc_two = QSpacerItem(10, 10, QSizePolicy.Expanding,
                 QSizePolicy.Expanding)
-        l.addItem(self.tabs[0].spc_two, 8, 0, 1, 3)
+        l.addItem(self.tabs[0].spc_two, 9, 0, 1, 3)
-        l.addWidget(self.fetch_metadata_button, 9, 0, 1, 2)
+        l.addWidget(self.fetch_metadata_button, 10, 0, 1, 2)
-        l.addWidget(self.config_metadata_button, 9, 2, 1, 1)
+        l.addWidget(self.config_metadata_button, 10, 2, 1, 1)
 
         self.tabs[0].gb2 = gb = QGroupBox(_('Co&mments'), self)
         gb.l = l = QVBoxLayout()
@@ -717,16 +740,17 @@ class MetadataSingleDialogAlt1(MetadataSingleDialogBase): # {{{
         create_row(7, self.rating, self.pubdate)
         create_row(8, self.pubdate, self.publisher,
                    button=self.pubdate.clear_button, icon='trash.png')
-        create_row(9, self.publisher, self.timestamp)
+        create_row(9, self.publisher, self.languages)
-        create_row(10, self.timestamp, self.identifiers,
+        create_row(10, self.languages, self.timestamp)
+        create_row(11, self.timestamp, self.identifiers,
                    button=self.timestamp.clear_button, icon='trash.png')
-        create_row(11, self.identifiers, self.comments,
+        create_row(12, self.identifiers, self.comments,
                    button=self.clear_identifiers_button, icon='trash.png')
         sto(self.clear_identifiers_button, self.swap_title_author_button)
         sto(self.swap_title_author_button, self.manage_authors_button)
         sto(self.manage_authors_button, self.paste_isbn_button)
         tl.addItem(QSpacerItem(1, 1, QSizePolicy.Fixed, QSizePolicy.Expanding),
-                   12, 1, 1 ,1)
+                   13, 1, 1 ,1)
 
         w = getattr(self, 'custom_metadata_widgets_parent', None)
         if w is not None:
@@ -852,16 +876,17 @@ class MetadataSingleDialogAlt2(MetadataSingleDialogBase): # {{{
         create_row(7, self.rating, self.pubdate)
         create_row(8, self.pubdate, self.publisher,
                    button=self.pubdate.clear_button, icon='trash.png')
-        create_row(9, self.publisher, self.timestamp)
+        create_row(9, self.publisher, self.languages)
-        create_row(10, self.timestamp, self.identifiers,
+        create_row(10, self.languages, self.timestamp)
+        create_row(11, self.timestamp, self.identifiers,
                    button=self.timestamp.clear_button, icon='trash.png')
-        create_row(11, self.identifiers, self.comments,
+        create_row(12, self.identifiers, self.comments,
                    button=self.clear_identifiers_button, icon='trash.png')
         sto(self.clear_identifiers_button, self.swap_title_author_button)
         sto(self.swap_title_author_button, self.manage_authors_button)
         sto(self.manage_authors_button, self.paste_isbn_button)
         tl.addItem(QSpacerItem(1, 1, QSizePolicy.Fixed, QSizePolicy.Expanding),
-                   12, 1, 1 ,1)
+                   13, 1, 1 ,1)
 
         # Custom metadata in col 1
         w = getattr(self, 'custom_metadata_widgets_parent', None)

src/calibre/gui2/preferences/metadata_sources.py

@@ -161,7 +161,7 @@ class FieldsModel(QAbstractListModel): # {{{
                 'tags' : _('Tags'),
                 'title': _('Title'),
                 'series': _('Series'),
-                'language': _('Language'),
+                'languages': _('Languages'),
         }
         self.overrides = {}
         self.exclude = frozenset(['series_index'])

src/calibre/gui2/store/stores/ozon_ru_plugin.py

@@ -50,6 +50,7 @@ class OzonRUStore(BasicStoreConfig, StorePlugin):
     def search(self, query, max_results=10, timeout=60):
         search_url = self.shop_url + '/webservice/webservice.asmx/SearchWebService?'\
                     'searchText=%s&searchContext=ebook' % urllib2.quote(query)
+        xp_template = 'normalize-space(./*[local-name() = "{0}"]/text())'
                     
         counter = max_results
         br = browser()
@@ -60,17 +61,14 @@ class OzonRUStore(BasicStoreConfig, StorePlugin):
                 if counter <= 0:
                     break
                 counter -= 1
-                        
+
-                xp_template = 'normalize-space(./*[local-name() = "{0}"]/text())'
-                
                 s = SearchResult()
                 s.detail_item = data.xpath(xp_template.format('ID'))
                 s.title = data.xpath(xp_template.format('Name'))
                 s.author = data.xpath(xp_template.format('Author'))
                 s.price = data.xpath(xp_template.format('Price'))
                 s.cover_url = data.xpath(xp_template.format('Picture'))
-                if re.match("^\d+?\.\d+?$", s.price):
+                s.price = format_price_in_RUR(s.price)
-                    s.price = u'{:.2F} руб.'.format(float(s.price))
                 yield s
 
     def get_details(self, search_result, timeout=60):
@@ -97,7 +95,22 @@ class OzonRUStore(BasicStoreConfig, StorePlugin):
                 # unfortunately no direct links to download books (only buy link)
                 # search_result.downloads['BF2'] = self.shop_url + '/order/digitalorder.aspx?id=' + + urllib2.quote(search_result.detail_item)
         return result
-    
+
+def format_price_in_RUR(price):
+    '''
+    Try to format price according ru locale: '12 212,34 руб.'
+    @param price: price in format like 25.99
+    @return: formatted price if possible otherwise original value
+    @rtype: unicode
+    '''
+    if price and re.match("^\d*?\.\d*?$", price):
+        try:
+            price = u'{:,.2F} руб.'.format(float(price))
+            price = price.replace(',', ' ').replace('.', ',', 1)
+        except:
+            pass
+    return price
+
 def _parse_ebook_formats(formatsStr):
     '''
     Creates a list with displayable names of the formats

src/calibre/gui2/store/stores/xinxii_plugin.py

@@ -0,0 +1,81 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import (unicode_literals, division, absolute_import, print_function)
+
+__license__ = 'GPL 3'
+__copyright__ = '2011, John Schember <john@nachtimwald.com>'
+__docformat__ = 'restructuredtext en'
+
+import urllib
+from contextlib import closing
+
+from lxml import etree
+
+from calibre import browser
+from calibre.gui2.store.basic_config import BasicStoreConfig
+from calibre.gui2.store.opensearch_store import OpenSearchOPDSStore
+from calibre.gui2.store.search_result import SearchResult
+
+class XinXiiStore(BasicStoreConfig, OpenSearchOPDSStore):
+
+    open_search_url = 'http://www.xinxii.com/catalog-search/'
+    web_url = 'http://xinxii.com/'
+    
+    # http://www.xinxii.com/catalog/
+
+    def search(self, query, max_results=10, timeout=60):
+        '''
+        XinXii's open search url is:
+        http://www.xinxii.com/catalog-search/query/?keywords={searchTerms}&amp;pw={startPage?}&amp;doc_lang={docLang}&amp;ff={docFormat},{docFormat},{docFormat}
+        
+        This url requires the docLang and docFormat. However, the search itself
+        sent to XinXii does not require them. They can be ignored. We cannot
+        push this into the stanard OpenSearchOPDSStore search because of the
+        required attributes.
+        
+        XinXii doesn't return all info supported by OpenSearchOPDSStore search
+        function so this one is modified to remove parts that are used.
+        '''
+        
+        url = 'http://www.xinxii.com/catalog-search/query/?keywords=' + urllib.quote_plus(query)
+        
+        counter = max_results
+        br = browser()
+        with closing(br.open(url, timeout=timeout)) as f:
+            doc = etree.fromstring(f.read())
+            for data in doc.xpath('//*[local-name() = "entry"]'):
+                if counter <= 0:
+                    break
+            
+                counter -= 1
+    
+                s = SearchResult()
+                
+                s.detail_item = ''.join(data.xpath('./*[local-name() = "id"]/text()')).strip()
+
+                for link in data.xpath('./*[local-name() = "link"]'):
+                    rel = link.get('rel')
+                    href = link.get('href')
+                    type = link.get('type')
+                    
+                    if rel and href and type:
+                        if rel in ('http://opds-spec.org/thumbnail', 'http://opds-spec.org/image/thumbnail'):
+                            s.cover_url = href
+                        if rel == 'alternate':
+                            s.detail_item = href
+                
+                s.formats = 'EPUB, PDF'
+                
+                s.title = ' '.join(data.xpath('./*[local-name() = "title"]//text()')).strip()
+                s.author = ', '.join(data.xpath('./*[local-name() = "author"]//*[local-name() = "name"]//text()')).strip()
+                
+                price_e = data.xpath('.//*[local-name() = "price"][1]')
+                if price_e:
+                    price_e = price_e[0]
+                    currency_code = price_e.get('currencycode', '')
+                    price = ''.join(price_e.xpath('.//text()')).strip()
+                    s.price = currency_code + ' ' + price
+                    s.price = s.price.strip()
+                
+
+                yield s

src/calibre/gui2/wizard/__init__.py

@@ -640,6 +640,7 @@ class LibraryPage(QWizardPage, LibraryUI):
             metadata_plugins = {
                     'zh' : ('Douban Books',),
                     'fr' : ('Nicebooks',),
+                    'ru' : ('OZON.ru',),
             }.get(lang, [])
             from calibre.customize.ui import enable_plugin
             for name in metadata_plugins:

src/calibre/library/caches.py

@@ -15,6 +15,7 @@ from calibre.utils.config import tweaks, prefs
 from calibre.utils.date import parse_date, now, UNDEFINED_DATE
 from calibre.utils.search_query_parser import SearchQueryParser
 from calibre.utils.pyparsing import ParseException
+from calibre.utils.localization import canonicalize_lang
 from calibre.ebooks.metadata import title_sort, author_to_author_sort
 from calibre.ebooks.metadata.opf2 import metadata_to_opf
 from calibre import prints
@@ -721,9 +722,13 @@ class ResultCache(SearchQueryParser): # {{{
                 if loc == db_col['authors']:
                     ### DB stores authors with commas changed to bars, so change query
                     if matchkind == REGEXP_MATCH:
-                        q = query.replace(',', r'\|');
+                        q = query.replace(',', r'\|')
                     else:
-                        q = query.replace(',', '|');
+                        q = query.replace(',', '|')
+                elif loc == db_col['languages']:
+                    q = canonicalize_lang(query)
+                    if q is None:
+                        q = query
                 else:
                     q = query
 

src/calibre/library/database2.py

@@ -39,6 +39,8 @@ from calibre.utils.magick.draw import save_cover_data_to
 from calibre.utils.recycle_bin import delete_file, delete_tree
 from calibre.utils.formatter_functions import load_user_template_functions
 from calibre.db.errors import NoSuchFormat
+from calibre.utils.localization import (canonicalize_lang,
+        calibre_langcode_to_name)
 
 copyfile = os.link if hasattr(os, 'link') else shutil.copyfile
 SPOOL_SIZE = 30*1024*1024
@@ -372,6 +374,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                 'aum_sortconcat(link.id, authors.name, authors.sort, authors.link)'),
             'last_modified',
             '(SELECT identifiers_concat(type, val) FROM identifiers WHERE identifiers.book=books.id) identifiers',
+            ('languages', 'languages', 'lang_code',
+                'sortconcat(link.id, languages.lang_code)'),
             ]
         lines = []
         for col in columns:
@@ -390,7 +394,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
              'size':4, 'rating':5, 'tags':6, 'comments':7, 'series':8,
              'publisher':9, 'series_index':10, 'sort':11, 'author_sort':12,
              'formats':13, 'path':14, 'pubdate':15, 'uuid':16, 'cover':17,
-             'au_map':18, 'last_modified':19, 'identifiers':20}
+             'au_map':18, 'last_modified':19, 'identifiers':20, 'languages':21}
 
         for k,v in self.FIELD_MAP.iteritems():
             self.field_metadata.set_field_record_index(k, v, prefer_custom=False)
@@ -469,7 +473,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                 'author_sort', 'authors', 'comment', 'comments',
                 'publisher', 'rating', 'series', 'series_index', 'tags',
                 'title', 'timestamp', 'uuid', 'pubdate', 'ondevice',
-                'metadata_last_modified',
+                'metadata_last_modified', 'languages',
                 ):
             fm = {'comment':'comments', 'metadata_last_modified':
                     'last_modified'}.get(prop, prop)
@@ -921,15 +925,24 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         formats = row[fm['formats']]
         mi.format_metadata = {}
         if not formats:
-            formats = None
+            good_formats = None
         else:
             formats = formats.split(',')
+            good_formats = []
             for f in formats:
-                mi.format_metadata[f] = self.format_metadata(id, f)
+                try:
-        mi.formats = formats
+                    mi.format_metadata[f] = self.format_metadata(id, f)
+                except:
+                    pass
+                else:
+                    good_formats.append(f)
+        mi.formats = good_formats
         tags = row[fm['tags']]
         if tags:
             mi.tags = [i.strip() for i in tags.split(',')]
+        languages = row[fm['languages']]
+        if languages:
+            mi.languages = [i.strip() for i in languages.split(',')]
         mi.series = row[fm['series']]
         if mi.series:
             mi.series_index = row[fm['series_index']]
@@ -1206,7 +1219,13 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
             except: # If path contains strange characters this throws an exc
                 candidates = []
             if format and candidates and os.path.exists(candidates[0]):
-                shutil.copyfile(candidates[0], fmt_path)
+                try:
+                    shutil.copyfile(candidates[0], fmt_path)
+                except:
+                    # This can happen if candidates[0] or fmt_path is too long,
+                    # which can happen if the user copied the library from a
+                    # non windows machine to a windows machine.
+                    return None
                 return fmt_path
 
     def copy_format_to(self, index, fmt, dest, index_is_id=False):
@@ -1390,7 +1409,8 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                 ('authors', 'authors', 'author'),
                 ('publishers', 'publishers', 'publisher'),
                 ('tags', 'tags', 'tag'),
-                ('series', 'series', 'series')
+                ('series', 'series', 'series'),
+                ('languages', 'languages', 'lang_code'),
                 ]:
             doit(ltable, table, ltable_col)
 
@@ -1507,6 +1527,7 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                         'series'   : self.get_series_with_ids,
                         'publisher': self.get_publishers_with_ids,
                         'tags'     : self.get_tags_with_ids,
+                        'languages': self.get_languages_with_ids,
                         'rating'   : self.get_ratings_with_ids,
                     }
                 func = funcs.get(category, None)
@@ -1521,6 +1542,10 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                 for l in list:
                     (id, val, sort_val) = (l[0], l[1], l[2])
                     tids[category][val] = (id, sort_val)
+            elif category == 'languages':
+                for l in list:
+                    id, val = l[0], calibre_langcode_to_name(l[1])
+                    tids[category][l[1]] = (id, val)
             elif cat['datatype'] == 'series':
                 for l in list:
                     (id, val) = (l[0], l[1])
@@ -1684,6 +1709,10 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
                 # Clean up the authors strings to human-readable form
                 formatter = (lambda x: x.replace('|', ','))
                 items = [v for v in tcategories[category].values() if v.c > 0]
+            elif category == 'languages':
+                # Use a human readable language string
+                formatter = calibre_langcode_to_name
+                items = [v for v in tcategories[category].values() if v.c > 0]
             else:
                 formatter = (lambda x:unicode(x))
                 items = [v for v in tcategories[category].values() if v.c > 0]
@@ -2043,6 +2072,9 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         if should_replace_field('comments'):
             doit(self.set_comment, id, mi.comments, notify=False, commit=False)
 
+        if should_replace_field('languages'):
+            doit(self.set_languages, id, mi.languages, notify=False, commit=False)
+
         # Setting series_index to zero is acceptable
         if mi.series_index is not None:
             doit(self.set_series_index, id, mi.series_index, notify=False,
@@ -2265,6 +2297,37 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
         if notify:
             self.notify('metadata', [id])
 
+    def set_languages(self, book_id, languages, notify=True, commit=True):
+        self.conn.execute(
+            'DELETE FROM books_languages_link WHERE book=?', (book_id,))
+        self.conn.execute('''DELETE FROM languages WHERE (SELECT COUNT(id)
+                                 FROM books_languages_link WHERE
+                                 books_languages_link.lang_code=languages.id) < 1''')
+
+        books_to_refresh = set([book_id])
+        final_languages = []
+        for l in languages:
+            lc = canonicalize_lang(l)
+            if not lc or lc in final_languages or lc in ('und', 'zxx', 'mis',
+                    'mul'):
+                continue
+            final_languages.append(lc)
+            lc_id = self.conn.get('SELECT id FROM languages WHERE lang_code=?',
+                    (lc,), all=False)
+            if lc_id is None:
+                lc_id = self.conn.execute('''INSERT INTO languages(lang_code)
+                                           VALUES (?)''', (lc,)).lastrowid
+            self.conn.execute('''INSERT INTO books_languages_link(book, lang_code)
+                                     VALUES (?,?)''', (book_id, lc_id))
+        self.dirtied(books_to_refresh, commit=False)
+        if commit:
+            self.conn.commit()
+        self.data.set(book_id, self.FIELD_MAP['languages'],
+                u','.join(final_languages), row_is_id=True)
+        if notify:
+            self.notify('metadata', [book_id])
+        return books_to_refresh
+
     def set_timestamp(self, id, dt, notify=True, commit=True):
         if dt:
             self.conn.execute('UPDATE books SET timestamp=? WHERE id=?', (dt, id))
@@ -2363,6 +2426,12 @@ class LibraryDatabase2(LibraryDatabase, SchemaUpgrade, CustomColumns):
             return []
         return result
 
+    def get_languages_with_ids(self):
+        result = self.conn.get('SELECT id,lang_code FROM languages')
+        if not result:
+            return []
+        return result
+
     def rename_tag(self, old_id, new_name):
         # It is possible that new_name is in fact a set of names. Split it on
         # comma to find out. If it is, then rename the first one and append the

src/calibre/library/field_metadata.py

@@ -17,7 +17,7 @@ class TagsIcons(dict):
 
     category_icons = ['authors', 'series', 'formats', 'publisher', 'rating',
                       'news',    'tags',   'custom:', 'user:',     'search',
-                      'identifiers',       'gst']
+                      'identifiers', 'languages', 'gst']
     def __init__(self, icon_dict):
         for a in self.category_icons:
             if a not in icon_dict:
@@ -37,6 +37,7 @@ category_icon_map = {
                     'search'     : 'search.png',
                     'identifiers': 'identifiers.png',
                     'gst'        : 'catalog.png',
+                    'languages'  : 'languages.png',
             }
 
 
@@ -114,6 +115,21 @@ class FieldMetadata(dict):
                            'is_custom':False,
                            'is_category':True,
                            'is_csp': False}),
+            ('languages', {'table':'languages',
+                           'column':'lang_code',
+                           'link_column':'lang_code',
+                           'category_sort':'lang_code',
+                           'datatype':'text',
+                           'is_multiple':{'cache_to_list': ',',
+                                          'ui_to_list': ',',
+                                          'list_to_ui': ', '},
+                           'kind':'field',
+                           'name':_('Languages'),
+                           'search_terms':['languages', 'language'],
+                           'is_custom':False,
+                           'is_category':True,
+                           'is_csp': False}),
+
             ('series',    {'table':'series',
                            'column':'name',
                            'link_column':'series',

src/calibre/manual/faq.rst

@@ -360,7 +360,7 @@ When you first run |app|, it will ask you for a folder in which to store your bo
 
 Metadata about the books is stored in the file ``metadata.db`` at the top level of the library folder This file is is a sqlite database. When backing up your library make sure you copy the entire folder and all its sub-folders.
 
-The library folder and all it's contents make up what is called a *|app| library*. You can have multiple such libraries. To manage the libraries, click the |app| icon on the toolbar. You can create new libraries, remove/rename existing ones and switch between libraries easily.
+The library folder and all it's contents make up what is called a |app| library. You can have multiple such libraries. To manage the libraries, click the |app| icon on the toolbar. You can create new libraries, remove/rename existing ones and switch between libraries easily.
 
 You can copy or move books between different libraries (once you have more than one library setup) by right clicking on a book and selecting the :guilabel:`Copy to library` action. 
 
@@ -438,7 +438,19 @@ Simply copy the |app| library folder from the old to the new computer. You can f
 
 Note that if you are transferring between different types of computers (for example Windows to OS X) then after doing the above you should also click the arrow next to the calibre icon on the tool bar, select Library Maintenance and run the Check Library action. It will warn you about any problems in your library, which you should fix by hand.
 
-.. note:: A |app| library is just a folder which contains all the book files and their metadata. All the emtadata is stored in a single file called metadata.db, in the top level folder. If this file gets corrupted, you may see an empty list of books in |app|. In this case you can ask |app| to restore your books by clicking the arrow next to the |app| icon on the toolbar and selecting Library Maintenance->Restore Library.
+.. note:: A |app| library is just a folder which contains all the book files and their metadata. All the metadata is stored in a single file called metadata.db, in the top level folder. If this file gets corrupted, you may see an empty list of books in |app|. In this case you can ask |app| to restore your books by clicking the arrow next to the |app| icon on the toolbar and selecting Library Maintenance->Restore Library.
+
+The list of books in |app| is blank!
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In order to understand why that happened, you have to understand what a |app| library is. At the most basic level, a |app| library is just a folder. Whenever you add a book to |app|, that book's files are copied into this folder (arranged into sub folders by author and title). Inside the |app| library folder, at the top level, you will see a file called metadata.db. This file is where |app| stores the metadata like title/author/rating/tags etc. for *every* book in your |app| library. The list of books that |app| displays is created by reading the contents of this metadata.db file.
+
+There can be two reasons why |app| is showing a empty list of books:
+
+  * Your |app| library folder changed its location. This can happen if it was on an external disk and the drive letter for that disk changed. Or if you accidentally moved the folder. In this case, |app| cannot find its library and so starts up with an empty library instead. To remedy this, simply click the arrow next to the |app| icon in the |app| toolbar (it will say 0 books underneath it) and select Switch/create library. Click the little blue icon to select the new location of your |app| library and click OK.
+
+  * Your metadata.db file was deleted/corrupted. In this case, you can ask |app| to rebuild the metadata.db from its backups. Click the arrow next to the |app| icon in the |app| toolbar (it will say 0 books underneath it) and select Library maintenance->Restore database. |app| will automatically rebuild metadata.db.
+
 
 Content From The Web
 ---------------------
@@ -446,6 +458,7 @@ Content From The Web
   :depth: 1
   :local:
 
+
 I obtained a recipe for a news site as a .py file from somewhere, how do I use it?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Start the :guilabel:`Add custom news sources` dialog (from the :guilabel:`Fetch news` menu) and click the :guilabel:`Switch to advanced mode` button. Delete everything in the box with the recipe source code and copy paste the contents of your .py file into the box. Click :guilabel:`Add/update recipe`.