.. include:: global.rst

.. _conversion:

E-book Conversion
===================

|app| has a conversion system that is designed to be very easy to use. Normally, you just
add a book to |app|, click convert and |app| will try hard to generate output that is as
close as possible to the input. However, |app| accepts a very large number of input formats,
not all of which are as suitable as others for conversion to e-books. In the case of
such input formats, or if you just want greater control over the conversion system,
|app| has a lot of options to fine tune the conversion process. Note however that |app|'s
conversion system is not a substitute for a full blown e-book editor. To edit e-books, I
would recommend first converting them to EPUB using |app| and then using a dedicated EPUB editor,
like `Sigil <http://code.google.com/p/sigil/>`_ to get the book into perfect shape. You can then
use the edited EPUB as input for conversion into other formats in |app|.

This document will refer mainly to the conversion settings as found in the conversion dialog,
pictured below. All these settings are also available via command line interface to conversion,
documented at :ref:`ebook-convert`. In |app|, you can obtain help on any individual setting by holding your
mouse over it, a tooltip will appear describing the setting.

.. image:: images/conv_dialog.png
    :align: center
    :alt: E-book conversion dialog
    :scale: 50 

.. contents:: Contents
  :depth: 1
  :local: 

.. _conversion-introduction:

Introduction
-------------

.. |dbgi| image:: images/debug.png
    :align: middle
    :alt: Debug icon

The first thing to understand about the conversion system is that it is designed as a pipeline.
Schematically, it looks like this:

.. image:: images/pipeline.png
    :align: center
    :alt: The conversion pipeline

The input format is first converted to XHTML by the appropriate *Input Plugin*.
This HTML is then *transformed*. In the last step, the processed XHTML is converted
to the specified output format by the appropriate *Output Plugin*. The results
of the conversion can vary greatly, based on the input format. Some formats
convert much better than others. A list of the best source formats for conversion
is available :ref:`here <best-source-formats>`.

The transforms that act on the XHTML output are where all the work happens. There are various
transforms, for example, to insert book metadata as a page at the start of the book,
to detect chapter headings and automatically create a Table of Contents, to proportionally
adjust font sizes, et cetera. It is important to remeber that all the transforms act on the
XHTML output by the *Input Plugin*, not on the input file itself. So, for example, if you ask |app|
to convert an RTF file to EPUB, it will first be converted to XHTML internally,
the various transforms will be applied to the XHTML and then the *Output Plugin* will
create the EPUB file, automatically generating all metadata, Table of Contents, et cetera.

You can see this process in action by using the debug option |dbgi|. Just specify the path to
a directory for the debug output. During conversion, |app| will place the XHTML generated by
the various stages of the conversion pipeline in different sub-directories.
The four sub-directories are:

.. table:: Stages of the conversion pipeline

    ==========    =============
    Directory     Description
    ==========    =============
    input         This contains the HTML output by the Input Plugin. Use this to debug the Input Plugin.
    parsed        The result of pre-processing and converting to XHTML the output from the Input Plugin. Use to debug structure detection. 
    structure     Post structure detection, but before CSS flattening and font size conversion. Use to debug font size conversion and CSS transforms.
    processed     Just before the e-book is passed to the output plugin. Use to debug the Output Plugin.
    ==========    =============

If you want to edit the input document a little before having |app| convert it, the best thing to 
do is edit the files in the :file:`input` sub-directory, then zip it up, and use the zip file as the 
input format for subsequent conversions. To do this use the :guilabel:`Edit meta information` dialog
to add the zip file as a format for the book and then, in the top left corner of the conversion dialog,
select ZIP as the input format.

This document will deal mainly with the various transforms that operate on the intermediate XHTML
and how to control them. At the end are some tips specific to each Input/Output format.

Look & Feel
-------------

.. contents:: Contents
  :depth: 1
  :local:

This group of options controls various aspects of the look and feel of the converted e-book. 

.. _font-size-rescaling:

Font size rescaling
~~~~~~~~~~~~~~~~~~~~~~~

One of the nicest features of the e-reading experience is the ability to easily adjust font sizes to
suit individual needs and lighting conditions. |app| has sophisticated algorithms to ensure that
all the books it outputs have a consistent font sizes, no matter what font sizes are specified
in the input document.

The base font size of a document is the most common font size in that document,
i.e., the size of the bulk of text in that document. When you specify a
:guilabel:`Base font size`, |app| automatically rescales all font sizes in the document
proportionately, so that the most common font size becomes the specified base font size and other
font sizes are rescaled appropriately. By choosing a larger base font size, you can make the fonts
in the document larger and vice versa. When you set the base font size, for best results, you should
also set the font size key.

Normally, |app| will automatically choose a base font size appropriate to the Output Profile you
have chosen (see :ref:`page-setup`). However, you can override this here in case the default is
not suitable for you.

The :guilabel:`Font size key` option lets you control how non-base font sizes are rescaled. 
The font rescaling algorithm works using a font size key, which is simply a comma-separated
list of font sizes. The font size key tells |app| how many "steps" bigger or smaller a given font
size should be compared to the base font size. The idea is that there should be a limited number
of font sizes in a document. For example, one size for the body text, a couple of sizes for
different levels of headings and a couple of sizes for super/sub scripts and footnotes. The
font size key allows |app| to compartmentalize the font sizes in the input documents into
separate "bins" corresponding to the different logical font sizes. 

Let's illustrate with an example. 
Suppose the source document we are converting was produced by someone with excellent
eyesight and has a base font size of 8pt. That means the bulk of the text in the document is sized
at 8pts, while headings are somewhat larger (say 10 and 12pt) and footnotes somewhat smaller at 6pt. 
Now if we use the following settings::

    Base font size : 12pt
    Font size key  : 7, 8, 10, 12, 14, 16, 18, 20

The output document will have a base font size of 12pt, headings of 14 and 16pt and footnotes of 8pt.
Now suppose we want to make the largest heading size stand out more and make the footnotes a
little larger as well. To achieve this, the font key should be changed to::

    New font size key : 7, 9, 12, 14, 18, 20, 22

The largest headings will now become 18pt, while the footnotes will become 9pt. You can
play with these settings to try and figure out what would be optimum for you by using the
font rescaling wizard, which can be accessed by clicking the little button next to the
:guilabel:`Font size key` setting.

All the font size rescaling in the conversion can also be disabled here, if you would
like to preserve the font sizes in the input document. 

A related setting is :guilabel:`Line height`. Line height controls the vertical height of
lines. By default, (a line height of 0), no manipulation of line heights is performed. If
you specify a non-default value, line heights will be set in all locations that don't specify their
own line heights. However, this is something of a blunt weapon and should be used sparingly.
If you want to adjust the line heights for some section of the input, it's better to use
the :ref:`extra-css`.

Paragraph spacing
~~~~~~~~~~~~~~~~~~~

Normally, paragraphs in XHTML are rendered with a blank line between them and no leading text
indent. |app| has a couple of options to control this. :guilabel:`Remove spacing between paragraphs`
forcefully ensure that all paragraphs have no inter paragraph spacing. It also sets the text
indent to 1.5em (can be changed) to mark the start of every paragraph.
:guilabel:`Insert blank line` does the
opposite, guaranteeing that there is exactly one blank line between each pair of paragraphs. 
Both these options are very comprehensive, removing spacing, or inserting it for *all* paragraphs
(technically <p> and <div> tags). This is so that you can just set the option and be sure that
it performs as advertised, irrespective of how messy the input file is. The one exception is
when the input file uses hard line breaks to implement inter-paragraph spacing.

If you want to remove the spacing between all paragraphs, except a select few, don't use these
options. Instead add the following CSS code to :ref:`extra-css`::

    p, div { margin: 0pt; border: 0pt; text-indent: 1.5em }
    .spacious { margin-bottom: 1em; text-indent: 0pt; }

Then, in your source document, mark the paragraphs that need spacing with `class="spacious"`.
If your input document is not in HTML, use the Debug option, described in the Introduction to get HTML
(use the :file:`input` sub-directory).


.. _extra-css:

Extra CSS
~~~~~~~~~~

This option allows you to specify arbitrary CSS that will be applied to all HTML files in the
input. This CSS is applied with very high priority and so should override most CSS present in
the input document itself. You can use this setting to fine tune the presentation/layout of your
document. For example, if you want all paragraphs of class `endnote` to be right aligned, just 
add::
    
    .endnote { text-align: right }

or if you want to change the indentation of all paragraphs::

    p { text-indent: 5mm; }

:guilabel:`Extra CSS` is a very powerful option, but you do need an understanding of how CSS works
to use it to its full potential.

Miscellaneous
~~~~~~~~~~~~~~

There are a few more options in this section.

:guilabel:`No text justification`
    Normally, if the output format supports it, |app| will force the output e-book
    to have *justified* text (i.e., a smooth right margin). This option will turn
    off this behavior, in which case whatever justification is specified in the input document
    will be used instead.

:guilabel:`Linearize tables`
    Some badly designed documents use tables to control the layout of text on the page.
    When converted these documents often have text that runs off the page and other artifacts.
    This option will extract the content from the tables and present it in a linear fashion.
    Note that this option linearizes *all* tables, so only use it if you are sure the
    input document does not use tables for legitimate purposes, like presenting tabular information.

:guilabel:`Transliterate unicode characters`
    Transliterate unicode characters to an ASCII representation. Use with care because this will
    replace unicode characters with ASCII. For instance it will replace "Михаил Горбачёв"
    with "Mikhail Gorbachiov". Also, note that in cases where there are multiple representations
    of a character (characters shared by Chinese and Japanese for instance) the representation used
    by the largest number of people will be used (Chinese in the previous example).
    This option is mainly useful if you are going to view the e-book on a device that does not
    have support for unicode.

:guilabel:`Input character encoding`
    Older documents sometimes don't specify their character encoding. When converted, this can
    result in non-English characters or special characters like smart quotes being corrupted. 
    |app| tries to auto-detect the character encoding of the source document, but it does not'
    always succeed. You can force it to assume a particular character encoding by using this setting.
    `cp1252` is a common encoding for documents produced using windows software. You should also read
    :ref:`char-encoding-faq` for more on encoding issues.
    

.. _page-setup:

Page Setup
-------------

The Page Setup options are for controlling screen layout, like margins and screen sizes. There are
options to setup page margins, which will be used by the Output Plugin, if the selected Output Format
supports page margins. In addition, you should choose an Input profile and an Output profile. Both sets
of profiles basically deal with how to interpret measurements in the input/output documents, screen sizes
and default font rescaling keys. 

If you know that the file you are converting was intended to be used on a particular device/software platform,
choose the corresponding input profile, otherwise just choose the default input profile. If you know the files
you are producing are meant for a particular device type, choose the corresponding Output profile. In particular, for MOBI Output files, you should choose the Kindle, for LIT the Microsoft Reader and for EPUB the Sony Reader. In the case of EPUB, the Sony Reader profile will result in EPUB files that will work everywhere. However, it has some side effects, like inserting artificial section breaks to keep internal components below the size threshold, needed for SONY devices. In particular for the iPhone/Android phones, choose the SONY output profile. If you know your EPUB files will not be read on a SONY or similar device, use the default output profile. If you are producing MOBI files that are not intended for the Kindle, choose the Mobipocket books output profile.

The Output profile also controls the screen size. This will cause, for example, images to be auto-resized to be fit to the screen in some output formats. So choose a profile of a device that has a screen size similar to your device.

.. _structure-detection:

Structure Detection
---------------------

Structure detection involves |app| trying its best to detect structural elements in the input document, when they are not properly specified. For example, chapters, page breaks, headers, footers, etc. As you can imagine, this process varies widely from book to book. Fortunately, |app| has very powerful options to control this. With power comes complexity, but if once you take the time to learn the complexity, you will find it well worth the effort.

Chapters and page breaks
~~~~~~~~~~~~~~~~~~~~~~~~~~~

|app| has two sets of options for :guilabel:`chapter detection` and :guilabel:`inserting page breaks`. This can sometimes be slightly confusing, as by default,
|app| will insert page breaks before detected chapters as well as the locations detected by the page breaks option.
The reason for this is that there are often location where page breaks should be inserted that are not chapter boundaries.
Also, detected chapters can be optionally inserted into the auto generated Table of Contents. 

|app| uses *XPath*, a powerful language to allow the user to specify chapter boundaries/page breaks. XPath can seem a little daunting
to use at first, fortunately, there is a :ref:`XPath tutorial <xpath-tutorial>` in the User Manual. Remember that Structure Detection
operates on the intermediate XHTML produced by the conversion pipeline. Use the debug option described in the
:ref:`conversion-introduction` to figure out the appropriate settings for your book. There is also a button for a XPath wizard
to help with the generation of simple XPath expressions.

By default, |app| uses the following expression for chapter detection::

    //*[((name()='h1' or name()='h2') and re:test(., 'chapter|book|section|part\s+', 'i')) or @class = 'chapter']

This expression is rather complex, because it tries to handle a number of common cases simulataneously. What it means
is that |app| will assume chapters start at either `<h1>` or `<h2>` tags that have any of the words 
`(chapter, book, section or part)` in them or that have the `class="chapter"` attribute. 

A related option is :guilabel:`Chapter mark`, which allows you to control what |app| does when it detects a chapter. By default,
it will insert a page break before the chapter. You can have it insert a ruled line instead of, or in addition to the page break.
You can also have it do nothing.

The default setting for detecting page breaks is::

    //*[name()='h1' or name()='h2']

which means that |app| will insert page breaks before every `<h1>` and `<h2>` tag by default.

.. note::
    
    The default expressions may change depending on the input format you are converting.

Removing headers and footers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

These options are useful primarily for conversion of PDF documents. Often, the conversion leaves
behind page headers and footers in the text. These options use regular expressions to try and detect
the headers and footers and remove them. Remember that they operate on the intermediate XHTML produced
by the conversion pipeline. There is also a wizard to help you customize the regular expressions for
your document.

The header and footer regular expressions are used in conjunction with the remove header and footer options.
If the remove option is not enabled the regular expression will not be applied to remove the matched text.
The removal works by using a python regular expression. All matched text is simply removed from
the document. You can learn more about regular expressions and their syntax at
http://docs.python.org/library/re.html.

Miscellaneous
~~~~~~~~~~~~~~

There are a few more options in this section.

:guilabel:`Insert metadata as page at start of book`
    One of the great things about |app| is that it allows you to maintain very complete metadata
    about all of your books, for example, a rating, tags, comments, etc. This option will create
    a single page with all this metadata and insert it into the converted e-book, typically just
    after the cover. Think of it as a way to create your own customised book jacket.

:guilabel:`Remove first image`
    Sometimes, the source document you are converting includes the cover as part of the book, instead
    of as a separate cover. If you also specify a cover in |app|, then the converted book will have
    two covers. This option will simply remove the first image from the source document, thereby
    ensuring that the converted book has only one cover, the one specified in |app|.
    
Table of Contents
------------------

When the input document has a Table of Contents in its metadata, |app| will just use that. However,
a number of older formats either do not support a metadata based Table of Contents, or individual
documents do not have one. In these cases, the options in this section can help you automatically
generate a Table of Contents in the converted e-book, based on the actual content in the input document.

The first option is :guilabel:`Force use of auto-generated Table of Contents`. By checking this option
you can have |app| override any Table of Contents found in the metadata of the input document with the
auto generated one. 

The default way that the creation of the auto generated Table of Contents works is that, |app| will first try
to add any detected chapters to the generated table of contents. You can learn how to customize the detection of chapters
in the :ref:`structure-detection` section above. If you do not want to include detected chapters in the generated
table of contents, check the :guilabel:`Do not add detected chapters` option.

If less than the :guilabel:`Chapter threshold` number of chapters were detected, |app| will then add any hyperlinks
it finds in the input document to the Table of Contents. This often works well many input documents include a
hyperlinked Table of Contents right at the start. The :guilabel:`Number fo links` option can be used to control
this behavior. If set to zero, no links are added. If set to a number greater than zero, at most that number of links
is added.

|app| will automatically filter duplicates from the generated Table of Contents. However, if there are some additional
undesirable entries, you can filter them using the :guilabel:`TOC Filter` option. This is a regular expression that
will match the title of entries in the generated table of contents. Whenever a match is found, it will be removed.
For example, to remove all entries titles "Next" or "Previous" use::

    Next|Previous

Finally, the :guilabel:`Level 1,2,3 TOC` options allow you to create a sophisticated multi-level Table of Contents.
They are XPath expressions that match tags in the intermediate XHTML produced by the conversion pipeline. See the 
:ref:`conversion-introduction` for how to get access to this XHTML. Also read the :ref:`xpath-tutorial`, to learn
how to construct XPath expressions. Next to each option is a button that launches a wizard to help with the creation
of basic XPath expressions. The following simple example illustrates how to use these options.

Suppose you have an input document taht results in XHTML that look like this:

.. code-block:: html

    <html xmlns="http://www.w3.org/1999/xhtml">
        <head>
            <title>Sample document</title>
        </head>
        <body>
            <h1>Chapter 1</h1>
            ...
            <h2>Section 1.1</h2>
            ...
            <h2>Section 1.2</h2>
            ...
            <h1>Chapter 2</h1>
            ...
            <h2>Section 2.1</h2>
            ...
        </body>
    </html>

Then, we set the options as::

    Level 1 TOC : //h:h1
    Level 2 TOC : //h:h2

This will result in an automatically generated two level Table of Contents that looks like::

    Chapter 1
        Section 1.1
        Section 1.2
    Chapter 2
        Section 2.1


.. warning::

    Not all output formats support a multi level Table of Contents. You should first try with EPUB Output. If that
    works, then try your format of choice. 

How options are set/saved for Conversion
-------------------------------------------

There are two places where conversion options can be set in |app|. The first is in Preferences->Conversion. These
settings are the defaults for the conversion options. Whenever you try to convert a new book, the settings set here
will be used by default.

You can also change settings in the conversion dialog for each book conversion. When you convert a book, |app| remembers the
settings you used for that book, so that if you convert it again, the saved settings for the individual book will take
precedence over the defaults set in Preferences. You can restore the individual settings to defaults by using the Restore to defaults
button in the individual book conversion dialog.

When you Bulk Convert a set of books, settings are taken in the following order:

    * From the defaults set in Preferences->Conversion
    * From the saved conversion settings for each book being converted (if any). This can be turned off by the option in the top left corner of the Bulk Conversion dialog.
    * From the settings set in the Bulk conversion dialog

Note that the final settings for each book in a Bulk Conversion will be saved and re-used if the book is converted again. Since the
highest priority in Bulk Conversion is given to the settings in the Bulk Conversion dialog, these will override any book specific
settings. So you should only bulk convert books together that need similar settings. The exceptions are metadata and input format specific 
settings. Since the Bulk Conversion dialog does not have settings for these two categories, they will be taken from book specific
settings (if any) or the defaults. 

.. note::

    You can see the actual settings used during any conversion by clicking the rotating icon in the lower right corner 
    and then double clicking the individual conversion job. This will bring up a conversion log
    that will contain the actual settings used, near the top. 

Format specific tips
----------------------

Here you will find tips specific to the conversion of particular formats. Options specific to particular
format, whether input or output are available in the conversion dialog under their own section, for example
`TXT Input` or `EPUB Output`.

Convert Microsoft Word documents
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

|app| does not directly convert .doc files from Microsoft Word. However, in Word, you can save the document
as HTML and then convert the resulting HTML file with |app|. When saving as HTML, be sure to use the
"Save as Web Page, Filtered" option as this will produce clean HTML that will convert well.

There is a Word macro package that can automate the conversion of Word documents using |app|. It also makes
generating the Table of Contents much simpler. It is called BookCreator and is available for free
`here <http://www.mobileread.com/forums/showthread.php?t=28313>`_.

Convert TXT documents
~~~~~~~~~~~~~~~~~~~~~~

TXT documents have no well defined way to specify formatting like bold, italics, etc, or document structure like paragraphs, headings, sections and so on.
Since TXT documents provide no way to explicitly mark parts of
the text, by default |app| only groups lines in the input document into paragraphs. The default is to assume one or
more blank lines are a paragraph boundary::

    This is the first.
    
    This is the
    second paragraph.

TXT input supports a number of options to differentiate how paragraphs are detected.

    :guilabel:`Treat each line as a paragraph`
        Assumes that every line is a paragraph::

            This is the first.
            This is the second.
            This is the third.
        
    :guilabel:`Assume print formatting`
        Assumes that every paragraph starts with an indent (either a tab or 2+ spaces). Paragraphs end when
        the next line that starts with an indent is reached::

            This is the
            first.
            This is the second.
            
            This is the
            third.

    :guilabel:`Process using markdown`
        |app| also supports running TXT input though a transformation preprocessor known as markdown. Markdown
        allows for basic formatting to be added to TXT documents, such as bold, italics, section headings, tables,
        loists, a Table of Contents, etc. Marking chapter headings with a leading # and setting the chapter XPath detection
        expression to "//h:h1" is the easiest way to have a proper table of contents generated from a TXT document.
        You can learn more about the markdown syntax at http://daringfireball.net/projects/markdown/syntax.


Convert PDF documents
~~~~~~~~~~~~~~~~~~~~~~~~~~~

PDF documents are one of the worst formats to convert from. They are a fixed page size and text placement format.
Meaning, it is very difficult to determine where one paragraph ends and another begins. |app| will try to unwrap
paragraphs using a configurable, :guilabel:`Line Un-Wrapping Factor`. This is a scale used to determine the length
at which a line should be unwrapped. Valid values are a decimal
between 0 and 1. The default is 0.5, this is the median line length. Lower this value to include more
text in the unwrapping. Increase to include less.

Also, they often have headers and footers as part of the document that will become included with the text.
Use the options to remove headers and footers to mitigate this issue. If the headers and footers are not
removed from the text it can throw off the paragraph unwrapping.

Some limitations of PDF input is complex, multi-column, and image based documents are not supported.
Extraction of vector images and tables from within the document is also not supported.

