Homemade debugger

Artem Malyshev

@proofit404

I'm a proud member of the

Django Channels

maintenance team

Jimmy's story

Wow, Python is so cool!

I'll write my next project in Python!

Lol, are you using print statements for debugging?!

if only I could write debugger myself...

Chapter 1

Inside a Python interpreter

Source code

def foo(a, b):
    return a + b

if __name__ == '__main__':
    print(foo(1, 2))

Bytecode

# python -m dis simple.py
   0 LOAD_CONST        (<code object foo>)
  ...
  16 LOAD_NAME         (print)
  18 LOAD_NAME         (foo)
  20 LOAD_CONST        (1)
  22 LOAD_CONST        (2)
  24 CALL_FUNCTION
  26 CALL_FUNCTION

Frame evaluation

_PyEval_EvalFrame(PyFrameObject *f, int throwflag) {
     switch (opcode) {
          TARGET(CALL_FUNCTION) { /* <- */
               PyObject **sp, *res;
               res = call_function(&sp, oparg, NULL); /* <- */
               stack_pointer = sp;
               PUSH(res);
          }
     }
}

Python 1.4 Changelog

Trace function

_PyEval_EvalFrame(PyFrameObject *f, int throwflag) {
     switch (opcode) {
          TARGET(CALL_FUNCTION) {
               PyObject **sp, *res;
               res = call_function(&sp, oparg, NULL);
               call_trace(tracefunc, frame);
               JUMPTO(frame->f_lasti);
               PUSH(res);
          }
    }
}

Settrace event table

'call'Before a function is executed.
'line'Before a line is executed.
'return'Before a function returns.
'exception'After an exception occurs.

Chapter 2

Pdb implementation

Setup debugger

def foo(a, b):
    return a + b

if __name__ == '__main__':
    import mydbg; mydbg.set_trace()
    print(foo(1, 2))

Naive usage

import sys                                   # python simple.py
                                                    call
def set_trace():                           ['f_back', 'f_code', 'f_locals']
    sys.settrace(dispatch)

def dispatch(frame, event, arg):
    print(event)
    print(dir(frame))

Nested calls

import sys                                   # python simple.py
                                                    call
def set_trace():                           ['f_back', 'f_code', 'f_locals']
    sys.settrace(dispatch)               line
                                                    ['f_back', 'f_code', 'f_locals']
def dispatch(frame, event, arg):
    print(event)
    print(dir(frame))
    return dispatch

Print lines of code

import sys                                       # python simple.py
import linecache                               def foo(a, b):
                                                        return a + b
def dispatch(frame, event, arg):       return a + b
    line = linecache.getline(
        frame.f_code.co_filename,
        frame.f_lineno,
    )
    print(line)
    return dispatch

Interactive prompt

        frame.f_lineno,                      # python simple.py
    )                                                def foo(a, b):
    print(line)                                  (Mydbg) foo
    cmd = True                               <function foo at 0x7f153b>
    while cmd:                                  (Mydbg) b
        cmd = input('(Mydbg) ')           2
        run_cmd(cmd, frame)             (Mydbg)
    return dispatch                         return a + b
                                                     (Mydbg)
def run_cmd(cmd, frame):
    if cmd in frame.f_locals:
        print(frame.f_locals[cmd])

Commands

next, step in, contunue

up, down, where

expression eval, post mortem

Chapter 3

Threads and Multiprocessing

Things are way more interesting in the world of concurrency

Problems

Install tracer for all threads

Acquire the GIL

Reinstall tracer after fork

Stdin is not available

Chapter 4

Coroutines and asyncio

Generator based concurrency

(Mydbg) list
   1     def view():
  2           mydbg.set_trace()
  3  ->     cache = yield access_cache()
  4           data = yield access_db()
  5
(Mydbg) next
--Return--
(Mydbg) # Hello event loop internals :'(

Coroutines based concurrency

(Mydbg) list
   1     async def view():
  2           mydbg.set_trace()
  3  ->     cache = await access_cache()
  4           data = await access_db()
  5
(Mydbg) next
-> data = await access_db()
(Mydbg)

Chapter 5

Smart Tracing

What if you don't even know where to start

Something goes wrong

File "django/db/backends/utils.py", line 65, in execute
  return self.cursor.execute(sql, params)
File "django/db/utils.py", line 94, in __exit__
  six.reraise(dj_exc_type, dj_exc_value, traceback)
File "django/utils/six.py", line 685, in reraise
  raise value.with_traceback(tb)
File "django/db/backends/utils.py", line 63, in execute
  return self.cursor.execute(sql)
django.db.utils.DatabaseError: server closed the connection
      This probably means the server terminated abnormally
      before or while processing the request.

Let's track it down

from hunter import trace, Q
import mydbg

trace(
    Q(
        module="django.db.utils",
        function="close",
        kind="call",
        action=mydbg.set_trace,
    )
)

Chapter 6

Many years later...

A lot of cool tools were developed

ipdb, pdb++, winpdb

BeeWare bugjar

JetBrains PyCharm debugger

GDB python extensions

10x engineer path

The End