KGDB Tutorial

How to set up KGDB :
                                   The steps mentioned here are with reference to 2.6.32 kernel version.The main reason is that is KGDB is merged into linux main line from 2.6.26-RC5 kernel. For kernel version < 2.6.26-RC5, we have to get KGDB patch from http://kgdb.linsyssoft.com/kernel.htm and apply them to kernel.

Few terminology which we gonna use :

Target Machine :
                       The machine on which the source code to be debugged is present. This is the machine which is to be monitored via a remote machine using gdb on remote machine. We need the kernel configured with  KGDB and boot kernel with kgdb parameters set on this machine.

Host Machine :
                     The machine which is used to monitor the target computer. This machine requires gdb and the uncompressed kernel image of target machine.

Configure the Target Machine :
                     Get into the kernel source code.
                     $ cd /usr/src/linux/
                     $ vim .config

Enable the KGDB options

File : /usr/src/linux/.config

                   CONFIG_HAVE_ARCH_KGDB=y

                   CONFIG_KGDB=y

                   CONFIG_KGDB_SERIAL_CONSOLE=y

Recompile the kernel
                 
                    $ make menuconfig || make oldconfig
                    $ make
                    $ make modules
                    $ make modules_install
                    $ make install

After kernel recompilation, transfer the uncompressed image of kernel i.e. vmlinux to the host
machine. Before doing scp you should boot host machine. Copy kernel image at /boot/ directory.

                    $ cd /usr/src/linux/
                    $ scp vmlinux  /boot/
                 

Edit the grub entry of target machine, add kgdb parameters

File: /etc/grub.conf

                 

  title Scientific Linux (2.6.32-131.17.1-lustre)
         root (hd0,0)      

         kernel /boot/vmlinuz-2.6.32-131.17.1-lustre ro root=UUID=85149a02-7fb9-4eb2-a780-c2abe3e4b0a1 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us  kgdbwait kgdboc=ttyS1,115200 rhgb quiet

        initrd /boot/initramfs-2.6.32-131.17.1-lustre.imgNG=en_US.UTF-8

Working with KGDB:
             Boot target machine: Target machine should wait for host machine to connect.

Configuring named pipe to connect two VM's over serial port :
         On host machine :
                      Named pipe configuring should  look something like below :

         On target machine :
                      Named pipe configuring should  look something like below :

    
 Boot host machine : Connect target machine after boot process is over.
               On host Machine :
                       $ cd /boot/
                       $ gdb ./vmlinux
                                 Pass kernel image file of target machine.
                       $ gdb> target remote /dev/ttyS1
                                 This will establish communication between target and host machine.
                                 Both machine are connected via serial port and device name is /dev/ttyS1.
                                 It may vary. So, before working on kgdb you can verify which serial port is working
                                 You can use "minicom" (a linux tool) to check which serial port is active. 
                       $ gdb> cont   
                                 This will continue the boot process of target machine.
             On target machine :
                                $  echo "g"  >  /proc/sysrq-trigger
                                This will regain gdb session on the host machine. And now we can set
                                 breakpoints and continue debugging.
                                This will hang the session on your target machine, continuing gdb session on host
                                will invoke the session on target machine. If you are using putty to connect with target
                                machine, you will have to restart the session.
        Adding symbols for loadable modules :
                On target machine :
                                 Load the modules and copy symbols of various segments of our program
                                 $ insmod  your_module
                                                or
                                 $ modprobe  your_module
                                 $cd /sys/module/your_module/sections
                                 $ cat .text
                                 $ cat .bss
                                 $ cat .data
                                 $ cat .rodata
               On host machine :
                                  $ gdb> add-symbol-file  module_name_with complete_path  \
                                              text_segment_address \
                                             -s .bss    bss_segment_address \
                                             -s .rodata   rodata_segment_address \
                                             -s .data    data_segment_address
                                  Now we can set break points even in  our modules.
                                  $ cont
                                  Continuing gdb session will revoke target machine session.
                                  We can perform various action on target machine inorder to hit the breakpoint on
                                  machine.
                                  
                   
        Note:
                   Use gdb 6.8 , gdb 7 has some issue while working with kgdb.
                   While working with kgdb, if you are not able to set break points in the kernel code,
                   check for the  CONFIG_DEBUG_RODATA in the .config file.  
                    Disable CONFIG_DEBUG_RODATA and recompile your kernel and then try again.
                 
                    If  you are not able to step in to the function, try using new gdb version 7.5 on both machine.