Add support for remote debugging with pdb

manual/develop.rst

@@ -271,6 +271,22 @@ Python is a
 dynamically typed language with excellent facilities for introspection. Kovid wrote the core |app| code without once
 using a debugger. There are many strategies to debug |app| code:
 
+Using print statements
+^^^^^^^^^^^^^^^^^^^^^^^
+
+This is Kovid's favorite way to debug. Simply insert print statements at points of interest and run your program in the
+terminal. For example, you can start the GUI from the terminal as::
+
+    calibre-debug -g
+
+Similarly, you can start the ebook-viewer as::
+
+    calibre-debug -w /path/to/file/to/be/viewed
+
+The ebook-editor can be started as::
+
+    calibre-debug -t /path/to/be/edited
+
 Using an interactive python interpreter
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -284,23 +300,51 @@ locally defined variables (variables in the local scope). The interactive prompt
 for object properties and you can use the various Python facilities for introspection, such as
 :func:`dir`, :func:`type`, :func:`repr`, etc.
 
-Using print statements
+Using the python debugger as a remote debugger
-^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-This is Kovid's favorite way to debug. Simply insert print statements at points of interest and run your program in the
+You can use the builtin python debugger (pdb) as a remote debugger from the
-terminal. For example, you can start the GUI from the terminal as::
+command line. First, start the remote debugger at the point in the calibre code
+you are interested in, like this::
 
-    calibre-debug -g
+    from calibre.rpdb import set_trace
+    set_trace()
 
-Similarly, you can start the ebook-viewer as::
+Then run calibre, either as normal, or using one of the calibre-debug commands
+described in the previous section. Once the above point in the code is reached,
+calibre will freeze, waiting for the debugger to connect.
 
-    calibre-debug -w /path/to/file/to/be/viewed
+Now open a terminal or command prompt and use the following command to start
+the debugging session::
 
-Using the debugger in PyDev
+    calibre-debug -c "from calibre.rpdb import cli; cli()"
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-It is possible to get the debugger in PyDev working with the |app| development environment,
+You can read about how to use the python debugger in the `python stdlib docs
-see the `forum thread <http://www.mobileread.com/forums/showthread.php?t=143208>`_.
+for the pdb module <https://docs.python.org/2/library/pdb.html#debugger-commands>`_.
+
+.. note::
+    By default, the remote debugger will try to connect on port 4444. You can
+    change it, by passing the port parameter to both the set_trace() and the
+    cli() functions above, like this: ``set_trace(port=1234)`` and
+    ``cli(port=1234)``.
+
+.. note:: 
+    The python debugger cannot handle multiple threads, so you have to
+    call set_starace once per thread, each time with a different port number.
+
+Using the debugger in your favorite python IDE
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+It is possible to use the builtin debugger in your favorite python IDE, if it
+supports remote debugging. The first step is to add the |app| src checkout to
+the ``PYTHONPATH`` in your IDE. In other words, the directory you set as
+``CALIBRE_DEVELOP_FROM`` above, must also be in the ``PYTHONPATH`` of your IDE.
+
+Then place the IDE's remote debugger module into the :file:`src` subdirectory
+of the |app| source code checkout. Add whatever code is needed to launch the
+remote debugger to |app| at the point of interest, for example in the main
+function. Then run |app| as normal. Your IDE should now be able to connect to
+the remote debugger running inside |app|.
 
 Executing arbitrary scripts in the |app| python environment
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

src/calibre/rpdb.py

@@ -0,0 +1,125 @@
+#!/usr/bin/env python
+# vim:fileencoding=utf-8
+from __future__ import (unicode_literals, division, absolute_import,
+                        print_function)
+
+__license__ = 'GPL v3'
+__copyright__ = '2014, Kovid Goyal <kovid at kovidgoyal.net>'
+
+import pdb, socket, inspect, sys, select
+
+from calibre import prints
+from calibre.utils.ipc import eintr_retry_call
+
+PROMPT = b'(debug) '
+MSG = b'\x00\x01\x02'
+
+class RemotePdb(pdb.Pdb):
+
+    def __init__(self, addr="127.0.0.1", port=4444, skip=None):
+        try:
+            prints("pdb is running on %s:%d" % (addr, port), file=sys.stderr)
+        except IOError:
+            pass
+
+        # Open a reusable socket to allow for reloads
+        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)
+        self.sock.bind((addr, port))
+        self.sock.listen(1)
+        clientsocket, address = self.sock.accept()
+        self.handle = clientsocket.makefile('rw')
+        pdb.Pdb.__init__(self, completekey='tab', stdin=self.handle, stdout=self.handle, skip=skip)
+        self.prompt = PROMPT
+
+    def send_message(self, *args, **kwargs):
+        kwargs['file'] = self.handle
+        self.handle.write(MSG)
+        prints(*args, **kwargs)
+        self.handle.write(PROMPT)
+        self.handle.flush()
+
+    def ask_question(self, query):
+        self.send_message(query, end='')
+        return self.handle.readline()
+
+    def end_session(self, *args):
+        self.clear_all_breaks()
+        self.reset()
+        del self.handle
+        try:
+            self.sock.shutdown(socket.SHUT_RDWR)
+            self.sock.close()
+        except socket.error:
+            pass
+        return pdb.Pdb.do_continue(self, None)
+
+    def do_clear(self, arg):
+        if not arg:
+            ans = self.ask_question("Clear all breaks? [y/n]: ")
+            if ans.strip().lower() in {b'y', b'yes'}:
+                self.clear_all_breaks()
+                self.send_message('All breaks cleared')
+            return
+        return pdb.Pdb.do_clear(self, arg)
+    do_cl = do_clear
+
+    def do_continue(self, arg):
+        if not self.breaks:
+            ans = self.ask_question(
+                'There are no breakpoints set. Continuing will terminate this debug session. Are you sure? [y/n]: ')
+            if ans.strip().lower() in {b'y', b'yes'}:
+                return self.end_session()
+            return
+        return pdb.Pdb.do_continue(self, arg)
+    do_c = do_cont = do_continue
+
+    do_EOF = do_quit = do_exit = do_q = end_session
+
+def set_trace(port=4444, skip=None):
+    frame = inspect.currentframe().f_back
+
+    try:
+        debugger = RemotePdb(port=port, skip=skip)
+        debugger.set_trace(frame)
+    except KeyboardInterrupt:
+        prints('Debugging aborted by keyboard interrupt')
+    except Exception:
+        prints('Failed to run debugger')
+        import traceback
+        traceback.print_exc()
+
+def cli(port=4444):
+    prints('Connecting to remote process on port %d...' % port)
+    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    sock.settimeout(120)
+    sock.connect(('127.0.0.1', port))
+    prints('Connected to remote process')
+    sock.setblocking(True)
+    try:
+        while True:
+            recvd = b''
+            while not recvd.endswith(PROMPT) or select.select([sock], [], [], 0) == ([sock], [], []):
+                buf = eintr_retry_call(sock.recv, 16 * 1024)
+                if not buf:
+                    return
+                recvd += buf
+            if recvd:
+                if recvd.startswith(MSG):
+                    recvd = recvd[len(MSG):-len(PROMPT)]
+                sys.stdout.write(recvd)
+            buf = []
+            raw = b''
+            try:
+                while not raw.endswith(b'\n'):
+                    raw += sys.stdin.read(1)
+                    if not raw:  # EOF (Ctrl+D)
+                        raw = b'quit\n'
+                        break
+                eintr_retry_call(sock.send, raw)
+            except KeyboardInterrupt:
+                eintr_retry_call(sock.send, b'quit\n')
+                continue
+    except KeyboardInterrupt:
+        pass
+