Merge branch 'master' of https://github.com/cbhaley/calibre

manual/template_lang.rst

@@ -630,7 +630,7 @@ the value of a custom field #genre. You cannot do this in the :ref:`Single Funct
 
 The example shows several things:
 
-* `TPM` is used if the expression begins with ``:'`` and ends with ``'``. Anything else is assumed to be in :ref:`Single Function Mode <single_mode>`.
+* `TPM` is used if the expression begins with ``:'`` and ends with ``'}``. Anything else is assumed to be in :ref:`Single Function Mode <single_mode>`.
 * the variable ``$`` stands for the field named in the template: the expression is operating upon, ``#series`` in this case.
 * functions must be given all their arguments. There is no default value. For example, the standard built-in functions must be given an additional initial parameter indicating the source field.
 * white space is ignored and can be used anywhere within the expression.
@@ -642,16 +642,56 @@ In `TPM`, using ``{`` and ``}`` characters in string literals can lead to errors
 
 As with `General Program Mode`, for functions documented under :ref:`Single Function Mode <single_mode>` you must supply the value the function is to act upon as the first parameter in addition to the documented parameters. In `TPM` you can use ``$`` to access the value specified by the ``lookup name`` for the template expression.
 
-Stored general program mode templates
+.. _python_mode:
+
+Python Template Mode
+-----------------------------------
+
+Python Template Mode (PTM) lets you write templates using native python and the `calibre API <https://manual.calibre-ebook.com/develop.html#api-documentation-for-various-parts-of-calibre>`_. The database API will be of most use; further discussion is beyond the scope of this manual. PTM templates are faster and can do more complicated operations but you must know how to write code in python using the calibre API.
+
+A PTM template begins with::
+
+ python:
+ def evaluate(book, db, globals, arguments, **kwargs):
+     # book is a calibre metadata object
+     # db is a calibre legacy database object
+     # globals is the template global variable dictionary
+     # arguments is a list of arguments if the template is called by a GPM template, otherwise None
+     # kwargs is a dictionary provided for future use
+
+     # Python code goes here
+	 return 'a string'
+
+You can add the above text to your template using the context menu, usually accessed with a right click. The comments are not significant and can be removed. You must use python indenting.
+
+Here is an example of a PTM template that produces a list of all the authors for a series. The list is stored in a `Column built from other columns, behaves like tags`. It shows in :guilabel:`Book details` and has the `on separate lines` checked (in :guilabel:`Preferences->Look & feel->Book details`). That option requires the list to be comma-separated. To satisfy that requirement the template converts commas in author names to semicolons then builds a comma-separated list of authors. The authors are then sorted, which is why the template uses author_sort.::
+
+    python:
+    def evaluate(book, db, globals, arguments, **kwargs):
+        if book.series is None:
+            return ''
+        ans = set()
+        for id_ in db.search_getting_ids(f'series:"={book.series}"', ''):
+            ans.update([v.strip() for v in db.new_api.field_for('author_sort', id_).split('&')])
+        return ', '.join(v.replace(',', ';') for v in sorted(ans))
+
+The output in :guilabel:`Book details` looks like this:
+
+.. image:: images/python_template_example.png
+    :align: center
+    :alt: E-book conversion dialog
+    :class: half-width-img
+
+Stored templates
 ----------------------------------------
 
-:ref:`General Program Mode <general_mode>` supports saving templates and calling those templates from another template, much like calling stored functions. You save templates using :guilabel:`Preferences->Advanced->Template functions`. More information is provided in that dialog. You call a template the same way you call a function, passing positional arguments if desired. An argument can be any expression. Examples of calling a template, assuming the stored template is named ``foo``:
+Both :ref:`General Program Mode <general_mode>` and :ref:`Python Template Mode <python_mode>` support saving templates and calling those templates from another template, much like calling stored functions. You save templates using :guilabel:`Preferences->Advanced->Template functions`. More information is provided in that dialog. You call a template the same way you call a function, passing positional arguments if desired. An argument can be any expression. Examples of calling a template, assuming the stored template is named ``foo``:
 
 * ``foo()`` -- call the template passing no arguments.
 * ``foo(a, b)`` call the template passing the values of the two variables ``a`` and ``b``.
 * ``foo(if field('series') then field('series_index') else 0 fi)`` -- if the book has a ``series`` then pass the ``series_index``, otherwise pass the value ``0``.
 
-You retrieve the arguments passed in the call to the stored template using the ``arguments`` function. It both declares and initializes local variables, effectively parameters. The variables are positional; they get the value of the parameter given in the call in the same position. If the corresponding parameter is not provided in the call then ``arguments`` assigns that variable the provided default value. If there is no default value then the variable is set to the empty string. For example, the following ``arguments`` function declares 2 variables, ``key``, ``alternate``::
+In GPM you retrieve the arguments passed in the call to the stored template using the ``arguments`` function. It both declares and initializes local variables, effectively parameters. The variables are positional; they get the value of the parameter given in the call in the same position. If the corresponding parameter is not provided in the call then ``arguments`` assigns that variable the provided default value. If there is no default value then the variable is set to the empty string. For example, the following ``arguments`` function declares 2 variables, ``key``, ``alternate``::
 
   arguments(key, alternate='series')
 
@@ -661,6 +701,8 @@ Examples, again assuming the stored template is named ``foo``:
 * ``foo('series', '#genre')`` the variable ``key`` is assigned the value ``'series'`` and the variable ``alternate`` is assigned the value ``'#genre'``.
 * ``foo()`` -- the variable ``key`` is assigned the empty string and the variable ``alternate`` is assigned the value ``'series'``.
 
+In PTM the arguments are passed in the ``arguments`` parameter, which is a list of strings. There isn't any way to specify default values. You must check the length of the ``arguments`` list to be sure that the number of arguments is what you expect.
+
 An easy way to test stored templates is using the ``Template tester`` dialog. For ease of access give it a keyboard shortcut in :guilabel:`Preferences->Advanced->Keyboard shortcuts->Template tester`. Giving the ``Stored templates`` dialog a shortcut will help switching more rapidly between the tester and editing the stored template's source code.
 
 Providing additional information to templates

src/calibre/db/tests/reading.py

@@ -811,3 +811,58 @@ class ReadingTest(BaseTest):
         v = formatter.safe_format('program: book_values("rating", "title:true", ",", 0)', {}, 'TEMPLATE ERROR', mi)
         self.assertEqual(set(v.split(',')), {'4', '6'})
     # }}}
+
+    def test_python_templates(self):  # {{{
+        from calibre.ebooks.metadata.book.formatter import SafeFormat
+        formatter = SafeFormat()
+
+        # need an empty metadata object to pass to the formatter
+        db = self.init_legacy(self.library_path)
+        mi = db.get_metadata(1)
+
+        # test counting books matching a search
+        template = '''python:
+def evaluate(book, db, **kwargs):
+    ids = db.new_api.search("series:true")
+    return str(len(ids))
+'''
+        v = formatter.safe_format(template, {}, 'TEMPLATE ERROR', mi)
+        self.assertEqual(v, '2')
+
+        # test counting books when none match the search
+        template = '''python:
+def evaluate(book, db, **kwargs):
+    ids = db.new_api.search("series:afafaf")
+    return str(len(ids))
+'''
+        v = formatter.safe_format(template, {}, 'TEMPLATE ERROR', mi)
+        self.assertEqual(v, '0')
+
+        # test is_multiple values
+        template = '''python:
+def evaluate(book, db, **kwargs):
+    tags = db.new_api.all_field_names('tags')
+    return ','.join(list(tags))
+'''
+        v = formatter.safe_format(template, {}, 'TEMPLATE ERROR', mi)
+        self.assertEqual(set(v.split(',')), {'Tag One', 'News', 'Tag Two'})
+
+        # test calling a python stored template from a GPM template
+        from calibre.utils.formatter_functions import (
+                load_user_template_functions, unload_user_template_functions)
+        load_user_template_functions('aaaaa',
+                                     [['python_stored_template',
+                                      "",
+                                      0,
+                                      '''python:
+def evaluate(book, db, globals, arguments):
+    tags = set(db.new_api.all_field_names('tags'))
+    tags.add(arguments[0])
+    return ','.join(list(tags))
+'''
+                                      ]], None)
+        v = formatter.safe_format('program: python_stored_template("one argument")', {},
+                                  'TEMPLATE ERROR', mi)
+        unload_user_template_functions('aaaaa')
+        self.assertEqual(set(v.split(',')), {'Tag One', 'News', 'Tag Two', 'one argument'})
+    # }}}

src/calibre/gui2/dialogs/template_dialog.py

@@ -23,7 +23,7 @@ from calibre.gui2.dialogs.template_dialog_ui import Ui_TemplateDialog
 from calibre.library.coloring import (displayable_columns, color_row_key)
 from calibre.utils.config_base import tweaks
 from calibre.utils.date import DEFAULT_DATE
-from calibre.utils.formatter_functions import formatter_functions
+from calibre.utils.formatter_functions import formatter_functions, StoredObjectType
 from calibre.utils.formatter import StopException
 from calibre.utils.icu import sort_key
 from calibre.utils.localization import localize_user_manual_link
@@ -42,33 +42,56 @@ class ParenPosition:
 
 
 class TemplateHighlighter(QSyntaxHighlighter):
+    # Code in this class is liberally borrowed from gui2.widgets.PythonHighlighter
 
     BN_FACTOR = 1000
 
-    KEYWORDS = ["program", 'if', 'then', 'else', 'elif', 'fi', 'for', 'rof',
+    KEYWORDS_GPM = ['if', 'then', 'else', 'elif', 'fi', 'for', 'rof',
                     'separator', 'break', 'continue', 'return', 'in', 'inlist',
                     'def', 'fed', 'limit']
 
+    KEYWORDS_PYTHON = ["and", "as", "assert", "break", "class", "continue", "def",
+                       "del", "elif", "else", "except", "exec", "finally", "for", "from",
+                       "global", "if", "import", "in", "is", "lambda", "not", "or",
+                       "pass", "print", "raise", "return", "try", "while", "with",
+                       "yield"]
+
+    BUILTINS_PYTHON = ["abs", "all", "any", "basestring", "bool", "callable", "chr",
+                       "classmethod", "cmp", "compile", "complex", "delattr", "dict",
+                       "dir", "divmod", "enumerate", "eval", "execfile", "exit", "file",
+                       "filter", "float", "frozenset", "getattr", "globals", "hasattr",
+                       "hex", "id", "int", "isinstance", "issubclass", "iter", "len",
+                       "list", "locals", "long", "map", "max", "min", "object", "oct",
+                       "open", "ord", "pow", "property", "range", "reduce", "repr",
+                       "reversed", "round", "set", "setattr", "slice", "sorted",
+                       "staticmethod", "str", "sum", "super", "tuple", "type", "unichr",
+                       "unicode", "vars", "xrange", "zip"]
+
+    CONSTANTS_PYTHON = ["False", "True", "None", "NotImplemented", "Ellipsis"]
+
     def __init__(self, parent=None, builtin_functions=None):
         super().__init__(parent)
         self.initialize_formats()
-        self.initialize_rules(builtin_functions)
+        self.initialize_rules(builtin_functions, for_python=False)
         self.regenerate_paren_positions()
         self.highlighted_paren = False
 
-    def initialize_rules(self, builtin_functions):
+    def initialize_rules(self, builtin_functions, for_python=False):
+        self.for_python = for_python
         r = []
 
         def a(a, b):
             r.append((re.compile(a), b))
 
+        if not for_python:
             a(
                 r"\b[a-zA-Z]\w*\b(?!\(|\s+\()"
                 r"|\$+#?[a-zA-Z]\w*",
                 "identifier")
 
+            a(r"^program:", "keymode")
             a(
-            "|".join([r"\b%s\b" % keyword for keyword in self.KEYWORDS]),
+                "|".join([r"\b%s\b" % keyword for keyword in self.KEYWORDS_GPM]),
                 "keyword")
 
             a(
@@ -77,14 +100,35 @@ class TemplateHighlighter(QSyntaxHighlighter):
                                                 formatter_functions().get_builtins())]),
                 "builtin")
 
+            a(r"""(?<!:)'[^']*'|"[^"]*\"""", "string")
+        else:
+            a(r"^python:", "keymode")
+
+            a(
+                "|".join([r"\b%s\b" % keyword for keyword in self.KEYWORDS_PYTHON]),
+                "keyword")
+            a(
+                "|".join([r"\b%s\b" % builtin for builtin in self.BUILTINS_PYTHON]),
+                "builtin")
+            a(
+                "|".join([r"\b%s\b" % constant for constant in self.CONSTANTS_PYTHON]),
+                "constant")
+
+            a(r"\bPyQt6\b|\bqt.core\b|\bQt?[A-Z][a-z]\w+\b", "pyqt")
+            a(r"@\w+(\.\w+)?\b", "decorator")
+
+            a(r"""('|").*?\1""", "string")
+            stringRe = r"""((?:"|'){3}).*?\1"""
+            a(stringRe, "string")
+            self.stringRe = re.compile(stringRe)
+            self.tripleSingleRe = re.compile(r"""'''(?!")""")
+            self.tripleDoubleRe = re.compile(r'''"""(?!')''')
         a(
             r"\b[+-]?[0-9]+[lL]?\b"
             r"|\b[+-]?0[xX][0-9A-Fa-f]+[lL]?\b"
             r"|\b[+-]?[0-9]+(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?\b",
             "number")
 
-        a(r"""(?<!:)'[^']*'|"[^"]*\"""", "string")
-
         a(r'\(', "lparen")
         a(r'\)', "rparen")
         self.Rules = tuple(r)
@@ -100,16 +144,23 @@ class TemplateHighlighter(QSyntaxHighlighter):
         config = self.Config = {}
         config["fontfamily"] = font_name
         app_palette = QApplication.instance().palette()
-        for name, color, bold, italic in (
+
+        all_formats = (
+            # name, color, bold, italic
             ("normal", None, False, False),
             ("keyword", app_palette.color(QPalette.ColorRole.Link).name(), True, False),
             ("builtin", app_palette.color(QPalette.ColorRole.Link).name(), False, False),
+            ("constant", app_palette.color(QPalette.ColorRole.Link).name(), False, False),
             ("identifier", None, False, True),
             ("comment", "#007F00", False, True),
             ("string", "#808000", False, False),
             ("number", "#924900", False, False),
+            ("decorator", "#FF8000", False, True),
+            ("pyqt", None, False, False),
             ("lparen", None, True, True),
-                ("rparen", None, True, True)):
+            ("rparen", None, True, True))
+
+        for name, color, bold, italic in all_formats:
             config["%sfontcolor" % name] = color
             config["%sfontbold" % name] = bold
             config["%sfontitalic" % name] = italic
@@ -119,8 +170,7 @@ class TemplateHighlighter(QSyntaxHighlighter):
         base_format.setFontPointSize(config["fontsize"])
 
         self.Formats = {}
-        for name in ("normal", "keyword", "builtin", "comment", "identifier",
-                     "string", "number", "lparen", "rparen"):
+        for name, color, bold, italic in all_formats:
             format_ = QTextCharFormat(base_format)
             color = config["%sfontcolor" % name]
             if color:
@@ -135,6 +185,8 @@ class TemplateHighlighter(QSyntaxHighlighter):
         return self.paren_pos_map.get(dex, None)
 
     def highlightBlock(self, text):
+        NORMAL, TRIPLESINGLE, TRIPLEDOUBLE = range(3)
+
         bn = self.currentBlock().blockNumber()
         textLength = len(text)
 
@@ -145,6 +197,17 @@ class TemplateHighlighter(QSyntaxHighlighter):
         elif text[0] == "#":
             self.setFormat(0, textLength, self.Formats["comment"])
             return
+        elif self.for_python:
+            stack = []
+            for i, c in enumerate(text):
+                if c in ('"', "'"):
+                    if stack and stack[-1] == c:
+                        stack.pop()
+                    else:
+                        stack.append(c)
+                elif c == "#" and len(stack) == 0:
+                    self.setFormat(i, len(text), self.Formats["comment"])
+                    return
 
         for regex, format_ in self.Rules:
             for m in regex.finditer(text):
@@ -153,9 +216,31 @@ class TemplateHighlighter(QSyntaxHighlighter):
                     pp = self.find_paren(bn, i)
                     if pp and pp.highlight:
                         self.setFormat(i, length, self.Formats[format_])
+                elif format_ == 'keymode':
+                    if bn > 0 and i == 0:
+                        continue
+                    self.setFormat(i, length, self.Formats['keyword'])
                 else:
                     self.setFormat(i, length, self.Formats[format_])
 
+        self.setCurrentBlockState(NORMAL)
+
+        if self.for_python and self.stringRe.search(text) is None:
+            # This is fooled by triple quotes inside single quoted strings
+            for m, state in (
+                (self.tripleSingleRe.search(text), TRIPLESINGLE),
+                (self.tripleDoubleRe.search(text), TRIPLEDOUBLE)
+            ):
+                i = -1 if m is None else m.start()
+                if self.previousBlockState() == state:
+                    if i == -1:
+                        i = len(text)
+                        self.setCurrentBlockState(state)
+                    self.setFormat(0, i + 3, self.Formats["string"])
+                elif i > -1:
+                    self.setCurrentBlockState(state)
+                    self.setFormat(i, len(text), self.Formats["string"])
+
         if self.generate_paren_positions:
             t = str(text)
             i = 0
@@ -327,6 +412,7 @@ class TemplateDialog(QDialog, Ui_TemplateDialog):
         self.set_mi(mi, fm)
 
         self.last_text = ''
+        self.highlighting_gpm = True
         self.highlighter = TemplateHighlighter(self.textbox.document(), builtin_functions=self.builtins)
         self.textbox.cursorPositionChanged.connect(self.text_cursor_changed)
         self.textbox.textChanged.connect(self.textbox_changed)
@@ -494,9 +580,12 @@ class TemplateDialog(QDialog, Ui_TemplateDialog):
             ca.setIcon(QIcon.ic('ok.png'))
         ca.triggered.connect(partial(self.set_word_wrap, not word_wrapping))
         m.addSeparator()
-        ca = m.addAction(_('Load template from the Template tester'))
-        ca.triggered.connect(self.load_last_template_text)
+        ca = m.addAction(_('Add python template definition text'))
+        ca.triggered.connect(self.add_python_template_header_text)
         m.addSeparator()
+        ca = m.addAction(_('Load template from the Template tester'))
+        m.addSeparator()
+        ca.triggered.connect(self.load_last_template_text)
         ca = m.addAction(_('Load template from file'))
         ca.setIcon(QIcon.ic('document_open.png'))
         ca.triggered.connect(self.load_template_from_file)
@@ -505,6 +594,19 @@ class TemplateDialog(QDialog, Ui_TemplateDialog):
         ca.triggered.connect(self.save_template)
         m.exec(self.textbox.mapToGlobal(point))
 
+    def add_python_template_header_text(self):
+        self.textbox.setPlainText('python:\n'
+                                  'def evaluate(book, db, globals, arguments, **kwargs):\n'
+                                  '\t# book is a calibre metadata object\n'
+                                  '\t# db is a calibre legacy database object\n'
+                                  '\t# globals is the template global variable dictionary\n'
+                                  '\t# arguments is a list of arguments if the template is '
+                                        'called by a GPM template, otherwise None\n'
+                                  '\t# kwargs is a dictionary provided for future use'
+                                  '\n\n\t# Python code goes here\n'
+                                  "\treturn 'a string'" +
+                                  self.textbox.toPlainText())
+
     def set_word_wrap(self, to_what):
         gprefs['gpm_template_editor_word_wrap_mode'] = to_what
         self.textbox.setWordWrapMode(QTextOption.WrapMode.WordWrap if to_what else QTextOption.WrapMode.NoWrap)
@@ -673,6 +775,16 @@ class TemplateDialog(QDialog, Ui_TemplateDialog):
 
     def textbox_changed(self):
         cur_text = str(self.textbox.toPlainText())
+        if cur_text.startswith('python:'):
+            if self.highlighting_gpm == True:
+                self.highlighter.initialize_rules(self.builtins, True)
+                self.highlighting_gpm = False
+                self.break_box.setChecked(False)
+                self.break_box.setEnabled(False)
+        elif not self.highlighting_gpm:
+            self.highlighter.initialize_rules(self.builtins, False)
+            self.highlighting_gpm = True
+            self.break_box.setEnabled(True)
         if self.last_text != cur_text:
             self.last_text = cur_text
             self.highlighter.regenerate_paren_positions()
@@ -707,14 +819,15 @@ class TemplateDialog(QDialog, Ui_TemplateDialog):
                                               pos_in_block)
 
     def function_type_string(self, name, longform=True):
-        if self.all_functions[name].is_python:
+        if self.all_functions[name].object_type is StoredObjectType.PythonFunction:
             if name in self.builtins:
                 return (_('Built-in template function') if longform else
                             _('Built-in function'))
             return (_('User defined Python template function') if longform else
                             _('User function'))
-        else:
-            return (_('Stored user defined template') if longform else _('Stored template'))
+        elif self.all_functions[name].object_type is StoredObjectType.StoredPythonTemplate:
+            return (_('Stored user defined python template') if longform else _('Stored template'))
+        return (_('Stored user defined GPM template') if longform else _('Stored template'))
 
     def function_changed(self, toWhat):
         name = str(self.function.itemData(toWhat))

src/calibre/gui2/dialogs/template_dialog.ui

@@ -344,8 +344,10 @@ you the value as well as all the local variables&lt;/p&gt;</string>
           <widget class="CodeEditor" name="textbox">
            <property name="toolTip">
             <string>&lt;p&gt;The text of the template program goes in this box.
-            Don't forget that a General Program Mode template must begin with
-            the word "program:".&lt;/p&gt;</string>
+            A General Program Mode template must begin with the word "program:".
+            A python template must begin with the word "python:" followed by a
+            function definition line. There is a context menu item you can use
+            to enter the first lines of a python template.&lt;/p&gt;</string>
            </property>
            <property name="sizePolicy">
             <sizepolicy hsizetype="MinimumExpanding" vsizetype="MinimumExpanding">

src/calibre/gui2/preferences/template_functions.py

@@ -13,7 +13,8 @@ from calibre.gui2.preferences.template_functions_ui import Ui_Form
 from calibre.gui2.widgets import PythonHighlighter
 from calibre.utils.formatter_functions import (
     compile_user_function, compile_user_template_functions, formatter_functions,
-    function_pref_is_python, function_pref_name, load_user_template_functions
+    function_object_type, function_pref_name, load_user_template_functions,
+    StoredObjectType
 )
 from polyglot.builtins import iteritems
 
@@ -48,9 +49,8 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         <li><b>your parameters</b>: you must supply one or more formal
         parameters. The number must match the arg count box, unless arg count is
         -1 (variable number or arguments), in which case the last argument must
-        be *args. At least one argument is required, and is usually the value of
-        the field being operated upon. Note that when writing in basic template
-        mode, the user does not provide this first argument. Instead it is
+        be *args. Note that when a function is called in basic template
+        mode at least one argument is always passed. It is
         supplied by the formatter.</li>
         </ul></p>
         <p>
@@ -87,10 +87,12 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         in template processing. You use a stored template in another template as
         if it were a template function, for example 'some_name(arg1, arg2...)'.</p>
 
-        <p>Stored templates must use General Program Mode -- they must begin with
-        the text '{0}'. You retrieve arguments passed to a stored template using
-        the '{1}()' template function, as in '{1}(var1, var2, ...)'. The passed
-        arguments are copied to the named variables.</p>
+        <p>Stored templates must use either General Program Mode -- they must
+        either begin with the text '{0}' or be {1}. You retrieve arguments
+        passed to a GPM stored template using the '{2}()' template function, as
+        in '{2}(var1, var2, ...)'. The passed arguments are copied to the named
+        variables. Arguments passed to a python template are in the '{2}'
+        parameter. Arguments are always strings.</p>
 
         <p>For example, this stored template checks if any items are in a
         list, returning '1' if any are found and '' if not.</p>
@@ -112,7 +114,7 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         See the template language tutorial for more information.</p>
         </p>
         ''')
-        self.st_textBrowser.setHtml(help_text.format('program:', 'arguments'))
+        self.st_textBrowser.setHtml(help_text.format('program:', 'python templates', 'arguments'))
         self.st_textBrowser.adjustSize()
         self.st_show_hide_help_button.clicked.connect(self.st_show_hide_help)
         self.st_textBrowser_height = self.st_textBrowser.height()
@@ -150,14 +152,14 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
             self.builtin_source_dict = {}
 
         self.funcs = {k:v for k,v in formatter_functions().get_functions().items()
-                                if v.is_python}
+                                if v.object_type is StoredObjectType.PythonFunction}
 
         self.builtins = formatter_functions().get_builtins_and_aliases()
 
         self.st_funcs = {}
         try:
             for v in self.db.prefs.get('user_template_functions', []):
-                if not function_pref_is_python(v):
+                if function_object_type(v) is not StoredObjectType.PythonFunction:
                     self.st_funcs.update({function_pref_name(v):compile_user_function(*v)})
         except:
             if question_dialog(self, _('Template functions'),
@@ -281,34 +283,53 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
             error_dialog(self.gui, _('Template functions'),
                          _('Function not defined'), show=True)
 
-    def create_button_clicked(self, use_name=None):
-        self.changed_signal.emit()
-        name = use_name if use_name else str(self.function_name.currentText())
-        name = name.split(' -- ')[0]
+    def check_errors_before_save(self, name, for_replace=False):
+        # Returns True if there is an error
         if not name:
             error_dialog(self.gui, _('Template functions'),
                          _('Name cannot be empty'), show=True)
-            return
-        if name in self.funcs:
+            return True
+        if not for_replace and name in self.funcs:
             error_dialog(self.gui, _('Template functions'),
                          _('Name %s already used')%(name,), show=True)
-            return
+            return True
         if name in {function_pref_name(v) for v in
                         self.db.prefs.get('user_template_functions', [])
-                        if not function_pref_is_python(v)}:
+                        if function_object_type(v) is not StoredObjectType.PythonFunction}:
             error_dialog(self.gui, _('Template functions'),
                          _('The name {} is already used for stored template').format(name), show=True)
-            return
+            return True
         if self.argument_count.value() == 0:
-            box = warning_dialog(self.gui, _('Template functions'),
-                         _('Argument count should be -1 or greater than zero. '
-                           'Setting it to zero means that this function cannot '
-                           'be used in single function mode.'), det_msg='',
-                         show=False, show_copy_button=False)
-            box.bb.setStandardButtons(box.bb.standardButtons() | QDialogButtonBox.StandardButton.Cancel)
-            box.det_msg_toggle.setVisible(False)
-            if not box.exec():
+            if not question_dialog(self.gui, _('Template functions'),
+                         _('Setting argument count to to zero means that this '
+                           'function cannot be used in single function mode. '
+                           'Is this OK?'),
+                         det_msg='',
+                         show_copy_button=False,
+                         default_yes=False,
+                         skip_dialog_name='template_functions_zero_args_warning',
+                         skip_dialog_msg='Ask this question again',
+                         yes_text=_('Save the function'),
+                         no_text=_('Cancel the save')):
+                print('cancelled')
+                return True
+        try:
+            prog = str(self.program.toPlainText())
+            cls = compile_user_function(name, str(self.documentation.toPlainText()),
+                                        self.argument_count.value(), prog)
+        except:
+            error_dialog(self.gui, _('Template functions'),
+                         _('Exception while compiling function'), show=True,
+                         det_msg=traceback.format_exc())
+            return True
+        return False
+
+    def create_button_clicked(self, use_name=None, need_error_checks=True):
+        name = use_name if use_name else str(self.function_name.currentText())
+        name = name.split(' -- ')[0]
+        if need_error_checks and self.check_errors_before_save(name, for_replace=False):
             return
+        self.changed_signal.emit()
         try:
             prog = str(self.program.toPlainText())
             cls = compile_user_function(name, str(self.documentation.toPlainText()),
@@ -364,8 +385,10 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
 
     def replace_button_clicked(self):
         name = str(self.function_name.itemData(self.function_name.currentIndex()))
+        if self.check_errors_before_save(name, for_replace=True):
+            return
         self.delete_button_clicked()
-        self.create_button_clicked(use_name=name)
+        self.create_button_clicked(use_name=name, need_error_checks=False)
 
     def refresh_gui(self, gui):
         pass
@@ -428,14 +451,14 @@ class ConfigWidget(ConfigWidgetBase, Ui_Form):
         self.changed_signal.emit()
         name = use_name if use_name else str(self.te_name.currentText())
         for k,v in formatter_functions().get_functions().items():
-            if k == name and v.is_python:
+            if k == name and v.object_type is StoredObjectType.PythonFunction:
                 error_dialog(self.gui, _('Stored templates'),
-                         _('The name {} is already used for template function').format(name), show=True)
+                         _('The name {} is already used by a template function').format(name), show=True)
         try:
             prog = str(self.te_textbox.toPlainText())
-            if not prog.startswith('program:'):
+            if not prog.startswith(('program:', 'python:')):
                 error_dialog(self.gui, _('Stored templates'),
-                         _('The stored template must begin with "program:"'), show=True)
+                     _("The stored template must begin with '{0}' or '{1}'").format('program:', 'python:'), show=True)
 
             cls = compile_user_function(name, str(self.template_editor.new_doc.toPlainText()),
                                         0, prog)

src/calibre/utils/formatter.py

@@ -11,12 +11,14 @@ __docformat__ = 'restructuredtext en'
 import re, string, traceback, numbers
 from functools import partial
 from math import modf
+from sys import exc_info
 
 from calibre import prints
 from calibre.constants import DEBUG
 from calibre.ebooks.metadata.book.base import field_metadata
 from calibre.utils.config import tweaks
-from calibre.utils.formatter_functions import formatter_functions
+from calibre.utils.formatter_functions import (
+    formatter_functions, get_database, function_object_type, StoredObjectType)
 from calibre.utils.icu import strcmp
 from polyglot.builtins import error_message
 
@@ -137,7 +139,8 @@ class StoredTemplateCallNode(Node):
     def __init__(self, line_number, name, function, expression_list):
         Node.__init__(self, line_number, 'call template: ' + name + '()')
         self.node_type = self.NODE_CALL_STORED_TEMPLATE
-        self.function = function
+        self.name = name
+        self.function = function # instance of the definition class
         self.expression_list = expression_list
 
 
@@ -579,16 +582,20 @@ class _Parser:
         return LocalFunctionCallNode(self.line_number, name, arguments)
 
     def call_expression(self, name, arguments):
-        subprog = self.funcs[name].cached_parse_tree
-        if subprog is None:
+        compiled_func = self.funcs[name].cached_compiled_text
+        if compiled_func is None:
             text = self.funcs[name].program_text
-            if not text.startswith('program:'):
-                self.error(_("A stored template must begin with '{0}'").format('program:'))
+            if function_object_type(text) is StoredObjectType.StoredGPMTemplate:
                 text = text[len('program:'):]
-            subprog = _Parser().program(self.parent, self.funcs,
+                compiled_func = _Parser().program(self.parent, self.funcs,
                                             self.parent.lex_scanner.scan(text))
-            self.funcs[name].cached_parse_tree = subprog
-        return StoredTemplateCallNode(self.line_number, name, subprog, arguments)
+            elif function_object_type(text) is StoredObjectType.StoredPythonTemplate:
+                text = text[len('python:'):]
+                compiled_func = self.parent.compile_python_template(text)
+            else:
+                self.error(_("A stored template must begin with '{0}' or {1}").format('program:', 'python:'))
+            self.funcs[name].cached_compiled_text = compiled_func
+        return StoredTemplateCallNode(self.line_number, name, self.funcs[name], arguments)
 
     def top_expr(self):
         return self.or_expr()
@@ -775,7 +782,7 @@ class _Parser:
             if id_ in self.local_functions:
                 return self.local_call_expression(id_, arguments)
             # Check for calling a stored template
-            if id_ in self.func_names and not self.funcs[id_].is_python:
+            if id_ in self.func_names and self.funcs[id_].object_type is not StoredObjectType.PythonFunction:
                 return self.call_expression(id_, arguments)
             # We must have a reference to a formatter function. Check if
             # the right number of arguments were supplied
@@ -846,7 +853,8 @@ class _Interpreter:
 
         try:
             if is_call:
-                ret =  self.do_node_stored_template_call(StoredTemplateCallNode(1, prog, None), args=args)
+                # prog is an instance of the function definition class
+                ret =  self.do_node_stored_template_call(StoredTemplateCallNode(1, prog.name, prog, None), args=args)
             else:
                 ret = self.expression_list(prog)
         except ReturnExecuted as e:
@@ -1014,7 +1022,10 @@ class _Interpreter:
         else:
             saved_line_number = None
         try:
-            val = self.expression_list(prog.function)
+            if function_object_type(prog.function.program_text) is StoredObjectType.StoredGPMTemplate:
+                val = self.expression_list(prog.function.cached_compiled_text)
+            else:
+                val = self.parent._run_python_template(prog.function.cached_compiled_text, args)
         except ReturnExecuted as e:
             val = e.get_value()
         self.override_line_number = saved_line_number
@@ -1526,14 +1537,62 @@ class TemplateFormatter(string.Formatter):
 
     def _eval_sfm_call(self, template_name, args, global_vars):
         func = self.funcs[template_name]
-        tree = func.cached_parse_tree
-        if tree is None:
-            tree = self.gpm_parser.program(self, self.funcs,
+        compiled_text = func.cached_compiled_text
+        if func.object_type is StoredObjectType.StoredGPMTemplate:
+            if compiled_text is None:
+                compiled_text = self.gpm_parser.program(self, self.funcs,
                                self.lex_scanner.scan(func.program_text[len('program:'):]))
-            func.cached_parse_tree = tree
-        return self.gpm_interpreter.program(self.funcs, self, tree, None,
+                func.cached_compiled_text = compiled_text
+            return self.gpm_interpreter.program(self.funcs, self, func, None,
                                                 is_call=True, args=args,
                                                 global_vars=global_vars)
+        elif function_object_type(func) is StoredObjectType.StoredPythonTemplate:
+            if compiled_text is None:
+                compiled_text = self.compile_python_template(func.program_text[len('python:'):])
+                func.cached_compiled_text = compiled_text
+            print(args)
+            return self._run_python_template(compiled_text, args)
+
+    def _eval_python_template(self, template, column_name):
+        if column_name is not None and self.template_cache is not None:
+            func = self.template_cache.get(column_name + '::python', None)
+            if not func:
+                func = self.compile_python_template(template)
+                self.template_cache[column_name + '::python'] = func
+        else:
+            func = self.compile_python_template(template)
+        return self._run_python_template(func, arguments=None)
+
+    def _run_python_template(self, compiled_template, arguments):
+        try:
+            return compiled_template(book=self.book,
+                                     db=get_database(self.book, get_database(self.book, None)),
+                                     globals=self.global_vars,
+                                     arguments=arguments)
+        except Exception as e:
+            ss = traceback.extract_tb(exc_info()[2])[-1]
+            raise ValueError(_('Error in function {0} on line {1} : {2} - {3}').format(
+                            ss.name, ss.lineno, type(e).__name__, str(e)))
+
+    def compile_python_template(self, template):
+        def replace_func(mo):
+            return mo.group().replace('\t', '    ')
+
+        prog ='\n'.join([re.sub(r'^\t*', replace_func, line)
+                                           for line in template.splitlines()])
+        locals_ = {}
+        if DEBUG and tweaks.get('enable_template_debug_printing', False):
+            print(prog)
+        try:
+            exec(prog, locals_)
+            func = locals_['evaluate']
+            return func
+        except SyntaxError as e:
+            raise(ValueError(
+                _('Syntax error on line {0} column {1}: text {2}').format(e.lineno, e.offset, e.text)))
+        except KeyError:
+            raise(ValueError(_("The {0} function is not defined in the template").format('evaluate')))
+
     # ################# Override parent classes methods #####################
 
     def get_value(self, key, args, kwargs):
@@ -1587,7 +1646,7 @@ class TemplateFormatter(string.Formatter):
                     else:
                         args = self.arg_parser.scan(fmt[p+1:])[0]
                         args = [self.backslash_comma_to_comma.sub(',', a) for a in args]
-                    if not func.is_python:
+                    if func.object_type is not StoredObjectType.PythonFunction:
                         args.insert(0, val)
                         val = self._eval_sfm_call(fname, args, self.global_vars)
                     else:
@@ -1615,6 +1674,8 @@ class TemplateFormatter(string.Formatter):
         if fmt.startswith('program:'):
             ans = self._eval_program(kwargs.get('$', None), fmt[8:],
                                      self.column_name, global_vars, break_reporter)
+        elif fmt.startswith('python:'):
+            ans = self._eval_python_template(fmt[7:], self.column_name)
         else:
             ans = self.vformat(fmt, args, kwargs)
             if self.strip_results:

src/calibre/utils/formatter_functions.py

@@ -14,6 +14,7 @@ __docformat__ = 'restructuredtext en'
 import inspect, re, traceback, numbers
 from contextlib import suppress
 from datetime import datetime, timedelta
+from enum import Enum
 from functools import partial
 from math import trunc, floor, ceil, modf
 
@@ -28,6 +29,12 @@ from calibre.utils.localization import calibre_langcode_to_name, canonicalize_la
 from polyglot.builtins import iteritems, itervalues
 
 
+class StoredObjectType(Enum):
+    PythonFunction = 1
+    StoredGPMTemplate = 2
+    StoredPythonTemplate = 3
+
+
 class FormatterFunctions:
 
     error_function_body = ('def evaluate(self, formatter, kwargs, mi, locals):\n'
@@ -123,6 +130,39 @@ def formatter_functions():
     return _ff
 
 
+def only_in_gui_error(name):
+    raise ValueError(_('The function {} can be used only in the GUI').format(name))
+
+
+def get_database(mi, name):
+    proxy = mi.get('_proxy_metadata', None)
+    if proxy is None:
+        if name is not None:
+            only_in_gui_error(name)
+        return None
+    wr = proxy.get('_db', None)
+    if wr is None:
+        if name is not None:
+            raise ValueError(_('In function {}: The database has been closed').format(name))
+        return None
+    cache = wr()
+    if cache is None:
+        if name is not None:
+            raise ValueError(_('In function {}: The database has been closed').format(name))
+        return None
+    wr = getattr(cache, 'library_database_instance', None)
+    if wr is None:
+        if name is not None:
+            only_in_gui_error()
+        return None
+    db = wr()
+    if db is None:
+        if name is not None:
+            raise ValueError(_('In function {}: The database has been closed').format(name))
+        return None
+    return db
+
+
 class FormatterFunction:
 
     doc = _('No documentation provided')
@@ -130,7 +170,7 @@ class FormatterFunction:
     category = 'Unknown'
     arg_count = 0
     aliases = []
-    is_python = True
+    object_type = StoredObjectType.PythonFunction
 
     def evaluate(self, formatter, kwargs, mi, locals, *args):
         raise NotImplementedError()
@@ -145,25 +185,10 @@ class FormatterFunction:
             return str(ret)
 
     def only_in_gui_error(self):
-        raise ValueError(_('The function {} can be used only in the GUI').format(self.name))
+        only_in_gui_error(self.name)
 
     def get_database(self, mi):
-        proxy = mi.get('_proxy_metadata', None)
-        if proxy is None:
-            self.only_in_gui_error()
-        wr = proxy.get('_db', None)
-        if wr is None:
-            raise ValueError(_('In function {}: The database has been closed').format(self.name))
-        cache = wr()
-        if cache is None:
-            raise ValueError(_('In function {}: The database has been closed').format(self.name))
-        wr = getattr(cache, 'library_database_instance', None)
-        if wr is None:
-            self.only_in_gui_error()
-        db = wr()
-        if db is None:
-            raise ValueError(_('In function {}: The database has been closed').format(self.name))
-        return db
+        return get_database(mi, self.name)
 
 
 class BuiltinFormatterFunction(FormatterFunction):
@@ -2368,13 +2393,17 @@ _formatter_builtins = [
 
 class FormatterUserFunction(FormatterFunction):
 
-    def __init__(self, name, doc, arg_count, program_text, is_python):
-        self.is_python = is_python
+    def __init__(self, name, doc, arg_count, program_text, object_type):
+        self.object_type = object_type
         self.name = name
         self.doc = doc
         self.arg_count = arg_count
         self.program_text = program_text
-        self.cached_parse_tree = None
+        self.cached_compiled_text = None
+        # Keep this for external code compatibility. Set it to True if we have a
+        # python template function, otherwise false. This might break something
+        # if the code depends on stored templates being in GPM.
+        self.is_python = True if object_type is StoredObjectType.PythonFunction else False
 
     def to_pref(self):
         return [self.name, self.doc, self.arg_count, self.program_text]
@@ -2383,13 +2412,20 @@ class FormatterUserFunction(FormatterFunction):
 tabs = re.compile(r'^\t*')
 
 
-def function_pref_is_python(pref):
-    if isinstance(pref, list):
-        pref = pref[3]
-    if pref.startswith('def'):
-        return True
-    if pref.startswith('program'):
-        return False
+def function_object_type(thing):
+    # 'thing' can be a preference instance, program text, or an already-compiled function
+    if isinstance(thing, FormatterUserFunction):
+        return thing.object_type
+    if isinstance(thing, list):
+        text = thing[3]
+    else:
+        text = thing
+    if text.startswith('def'):
+        return StoredObjectType.PythonFunction
+    if text.startswith('program'):
+        return StoredObjectType.StoredGPMTemplate
+    if text.startswith('python'):
+        return StoredObjectType.StoredPythonTemplate
     raise ValueError('Unknown program type in formatter function pref')
 
 
@@ -2398,8 +2434,9 @@ def function_pref_name(pref):
 
 
 def compile_user_function(name, doc, arg_count, eval_func):
-    if not function_pref_is_python(eval_func):
-        return FormatterUserFunction(name, doc, arg_count, eval_func, False)
+    typ = function_object_type(eval_func)
+    if typ is not StoredObjectType.PythonFunction:
+        return FormatterUserFunction(name, doc, arg_count, eval_func, typ)
 
     def replace_func(mo):
         return mo.group().replace('\t', '    ')
@@ -2415,7 +2452,7 @@ class UserFunction(FormatterUserFunction):
     if DEBUG and tweaks.get('enable_template_debug_printing', False):
         print(prog)
     exec(prog, locals_)
-    cls = locals_['UserFunction'](name, doc, arg_count, eval_func, True)
+    cls = locals_['UserFunction'](name, doc, arg_count, eval_func, typ)
     return cls
 
 
@@ -2432,7 +2469,7 @@ def compile_user_template_functions(funcs):
             # then white space differences don't cause them to compare differently
 
             cls = compile_user_function(*func)
-            cls.is_python = function_pref_is_python(func)
+            cls.object_type = function_object_type(func)
             compiled_funcs[cls.name] = cls
         except Exception:
             try: