Improvements to the template processor. Document new features.

src/calibre/gui2/dialogs/template_dialog.py

@@ -19,7 +19,7 @@ class TemplateDialog(QDialog, Ui_TemplateDialog):
 
         if text is not None:
             self.textbox.setPlainText(text)
-        self.textbox.setTabChangesFocus(True)
+        self.textbox.setTabStopWidth(50)
         self.buttonBox.button(QDialogButtonBox.Ok).setText(_('&OK'))
         self.buttonBox.button(QDialogButtonBox.Cancel).setText(_('&Cancel'))
 

src/calibre/gui2/dialogs/template_dialog.ui

@@ -6,7 +6,7 @@
    <rect>
     <x>0</x>
     <y>0</y>
-    <width>336</width>
+    <width>500</width>
     <height>235</height>
    </rect>
   </property>

src/calibre/manual/template_lang.rst

@@ -137,8 +137,8 @@ Note that you can use the prefix and suffix as well. If you want the number to a
     {#myint:0>3s:ifempty(0)|[|]}
     
 
-Using functions in templates - program mode
+Using functions in templates - template program mode
--------------------------------------------
+----------------------------------------------------
 
 The template language program mode differs from single-function mode in that it permits you to write template expressions that refer to other metadata fields, modify values, and do arithmetic. It is a reasonably complete programming language.
 
@@ -161,10 +161,13 @@ The syntax of the language is shown by the following grammar::
     constant   ::= " string " | ' string ' | number
     identifier ::= sequence of letters or ``_`` characters
     function   ::= identifier ( statement [ , statement ]* )
-    expression ::= identifier | constant | function
+    expression ::= identifier | constant | function | assignment
+    assignment ::= identifier '=' expression
     statement  ::= expression [ ; expression ]*
     program    ::= statement
 
+Comments are lines with a '#' character at the beginning of the line.
+
 An ``expression`` always has a value, either the value of the constant, the value contained in the identifier, or the value returned by a function. The value of a ``statement`` is the value of the last expression in the sequence of statements. As such, the value of the program (statement)::
 
     1; 2; 'foobar'; 3
@@ -208,13 +211,102 @@ The following functions are available in addition to those described in single-f
     * ``cmp(x, y, lt, eq, gt)`` -- compares x and y after converting both to numbers. Returns ``lt`` if x < y. Returns ``eq`` if x == y. Otherwise returns ``gt``.
     * ``divide(x, y)`` -- returns x / y. Throws an exception if either x or y are not numbers.
     * ``field(name)`` -- returns the metadata field named by ``name``.
+    * ``eval(string)`` -- evaluates the string as a program, passing the local variables (those `assign`ed to). This permits using the template processor to construct complex results from local variables.
     * ``multiply(x, y)`` -- returns x * y. Throws an exception if either x or y are not numbers.
+    * ``print(a, b, ...)`` -- prints the arguments to standard output. Unless you start calibre from the command line (``calibre-debug -g``), the output will go to a black hole.
     * ``strcat(a, b, ...)`` -- can take any number of arguments. Returns a string formed by concatenating all the arguments.
     * ``strcmp(x, y, lt, eq, gt)`` -- does a case-insensitive comparison x and y as strings. Returns ``lt`` if x < y. Returns ``eq`` if x == y. Otherwise returns ``gt``.
     * ``substr(str, start, end)`` -- returns the ``start``'th through the ``end``'th characters of ``str``. The first character in ``str`` is the zero'th character. If end is negative, then it indicates that many characters counting from the right. If end is zero, then it indicates the last character. For example, ``substr('12345', 1, 0)`` returns ``'2345'``, and ``substr('12345', 1, -1)`` returns ``'234'``.
     * ``subtract(x, y)`` -- returns x - y. Throws an exception if either x or y are not numbers.
     * ``template(x)`` -- evaluates x as a template. The evaluation is done in its own context, meaning that variables are not shared between the caller and the template evaluation. Because the `{` and `}` characters are special, you must use `[[` for the `{` character and `]]` for the '}' character; they are converted automatically. For example, ``template('[[title_sort]]') will evaluate the template ``{title_sort}`` and return its value.
 
+Using general program mode
+-----------------------------------
+
+For more complicated template programs, it is sometimes easier to avoid template syntax (all the `{` and `}` characters), instead writing a more classical-looking program. You can do this in |app| by beginning the template with `program:`. In this case, no template processing is done. The special variable `$` is not set. It is up to your program to produce the correct results.
+
+One advantage of `program:` mode is that the brackets are no longer special. For example, it is not necessary to use `[[` and `]]` when using the `template()` function.
+
+The following example is a `program:` mode implementation of a recipe on the MobileRead forum: "Put series into the title, using either initials or a shortened form. Strip leading articles from the series name (any)." For example, for the book The Two Towers in the Lord of the Rings series, the recipe gives `LotR [02] The Two Towers`. Using standard templates, the recipe requires three custom columns and a plugboard, as explained in the following:
+
+The solution requires creating three composite columns. The first column is used to remove the leading articles. The second is used to compute the 'shorten' form. The third is to compute the 'initials' form. Once you have these columns, the plugboard selects between them. You can hide any or all of the three columns on the library view.
+
+    First column:
+    Name: #stripped_series. 
+    Template: {series:re(^(A|The|An)\s+,)||}
+
+    Second column (the shortened form):
+    Name: #shortened. 
+    Template: {#stripped_series:shorten(4,-,4)}
+
+    Third column (the initials form):
+    Name: #initials. 
+    Template: {#stripped_series:re(([^\s])[^\s]+(\s|$),\1)}
+
+    Plugboard expression:
+    Template:{#stripped_series:lookup(.\s,#initials,.,#shortened,series)}{series_index:0>2.0f| [|] }{title}
+    Destination field: title
+
+    This set of fields and plugboard produces:
+    Series: The Lord of the Rings
+    Series index: 2
+    Title: The Two Towers
+    Output: LotR [02] The Two Towers
+
+    Series: Dahak
+    Series index: 1
+    Title: Mutineers Moon
+    Output: Dahak [01] Mutineers Moon
+
+    Series: Berserkers
+    Series Index: 4
+    Title: Berserker Throne
+    Output: Bers-kers [04] Berserker Throne
+
+    Series: Meg Langslow Mysteries
+    Series Index: 3
+    Title: Revenge of the Wrought-Iron Flamingos
+    Output: MLM [03] Revenge of the Wrought-Iron Flamingos
+
+The following program produces the same results as the original recipe, using only one custom column to hold the results of a program that computes the special title value.
+
+    Custom column: 
+    Name: #special_title
+    Template: (the following with all leading spaces removed)
+        program:
+        #	compute the equivalent of the composit fields and store them in local variables
+            stripped = re(field('series'), '^(A|The|An)\s+', '');
+            shortened = shorten(stripped, 4, '-' ,4);
+            initials = re(stripped, '[^\w]*(\w?)[^\s]+(\s|$)', '\1');
+
+        #	Format the series index. Ends up as empty if there is no series index.
+        #	Note that leading and trailing spaces will be removed by the formatter,
+        # 	so we cannot add them here. We will do that in the strcat below.
+        #	Also note that because we are in 'program' mode, we can freely use
+        #	curly brackets in strings, something we cannot do in template mode.
+            s_index = template('{series_index:0>2.0f}');
+
+        #	print(stripped, shortened, initials, s_index);
+
+        #	Now concatenate all the bits together. The switch picks between 
+        # 	initials and shortened, depending on whether there is a space
+        #	in stripped. We then add the brackets around s_index if it is
+        #	not empty. Finally, add the title. As this is the last function in
+        # 	the program, its value will be returned.
+            strcat(
+                switch(	stripped, 
+                        '.\s', initials, 
+                        '.', shortened,
+                        field('series')),
+                test(s_index, strcat(' [', s_index, '] '), ''),
+                field('title'));
+
+    Plugboard expression:
+    Template:{#special_title}
+    Destination field: title
+
+It would be possible to do the above with no custom columns by putting the program into the template box of the plugboard. However, to do so, all comments must be removed because the plugboard text box does not support multi-line editing. It is debatable whether the gain of not having the custom column is worth the vast increase in difficulty caused by the program being one giant line.
+
 Special notes for save/send templates
 -------------------------------------
 

src/calibre/utils/formatter.py

@@ -66,6 +66,10 @@ class _Parser(object):
         template = template.replace('[[', '{').replace(']]', '}')
         return eval_formatter.safe_format(template, self.variables, 'EVAL', None)
 
+    def _print(self, *args):
+        print args
+        return None
+
     local_functions = {
             'add'      : (2, partial(_math, op='+')),
             'assign'   : (2, _assign),
@@ -74,6 +78,7 @@ class _Parser(object):
             'eval'     : (1, _eval),
             'field'    : (1, lambda s, x: s.parent.get_value(x, [], s.parent.kwargs)),
             'multiply' : (2, partial(_math, op='*')),
+            'print'    : (-1, _print),
             'strcat'   : (-1, _concat),
             'strcmp'   : (5, _strcmp),
             'substr'   : (3, lambda s, x, y, z: x[int(y): len(x) if int(z) == 0 else int(z)]),
@@ -143,12 +148,18 @@ class _Parser(object):
             if not self.token_op_is_a(';'):
                 return val
             self.consume()
+            if self.token_is_eof():
+                return val
 
     def expr(self):
         if self.token_is_id():
             # We have an identifier. Determine if it is a function
             id = self.token()
             if not self.token_op_is_a('('):
+                if self.token_op_is_a('='):
+                    # classic assignment statement
+                    self.consume()
+                    return self._assign(id, self.expr())
                 return self.variables.get(id, _('unknown id ') + id)
             # We have a function.
             # Check if it is a known one. We do this here so error reporting is
@@ -339,6 +350,7 @@ class TemplateFormatter(string.Formatter):
                 (r'\w+',                lambda x,t: (2, t)),
                 (r'".*?((?<!\\)")',     lambda x,t: (3, t[1:-1])),
                 (r'\'.*?((?<!\\)\')',   lambda x,t: (3, t[1:-1])),
+                (r'\n#.*?(?=\n)',       None),
                 (r'\s',                 None)
         ])
 
@@ -422,10 +434,10 @@ class TemplateFormatter(string.Formatter):
         self.kwargs = kwargs
         self.book = book
         self.composite_values = {}
+        try:
             if fmt.startswith('program:'):
                 ans = self._eval_program(None, fmt[8:])
             else:
-            try:
                 ans = self.vformat(fmt, [], kwargs).strip()
         except Exception, e:
             if DEBUG: