	                      IBM DYNAMIC PROBES
	                      ==================
Contents
--------
 1. Introduction
 2. How to install
 3. How to use
 4. Security Considerations
 5. Known Bugs
 6. Restrictions in the current release
 7. License
 8. Version
 9. Reporting Bugs
10. Contributing
11. Authors

1. Introduction
---------------
Dynamic Probes is a generic and pervasive debugging facility that will operate
under the most extreme software conditions such as debugging a deep rooted
operating system problem in a live environment, for example in the page-manager
of the kernel or perhaps a problem that will not re-create easily in either a
lab or production environment. For such inaccessible problem scenarios
Dynamic Probes not only offers a technique for gathering diagnostic
information but has a high probability of successful outcome without the need
to build custom modules for debugging purposes. At this time, Dprobes is 
available on Intel 32bit (i386), zSeries 32bit (s390), zSeries 64bit (s390x),
PowerPC 32bit (ppc), and PowerPC 64bit (ppc64) platforms.

The dprobe facility can be used to insert software probes dynamically into
executing code modules. When a probe is fired, a user written probe-handler
is executed. The probe-handler is a program written in an assembly-like language,
based on the Reverse Polish Notation. Instructions are provided to enable the
probe-handler to access all the hardware registers, system data structures and
memory.

This is equivalent to inserting a breakpoint, monitoring various context
specific values, dumping out certain data areas when the breakpoint is hit.
So, in short, it is a dynamic debug session that runs without disrupting
the code being debugged and with only a very small runtime performance hit.
The dprobes facility is particularly useful for diagnosing problems that show
up rarely in frequently hit areas of code. Considering the kind of access
Dynamic Probes gives to a probe-handler, it could be thought of as a dynamically
loadable, automated kernel debugger.

At the heart of the mechanism is an interpreter which executes a
"probe program" when a probe fires. A probe program is registered per probe.
Probes may be registered before a process is created or a module is loaded.

Another interesting aspect of the Dynamic Probes facility is that the probes
are effective globally and they interoperate well with other debuggers. This
means that probes placed in an executable module will be applied on all
instances of that executable, under the context of all processes.
For eg.:
- if probes are placed in an executable program, they will be active in all
  instances of that program
- if probes are placed in a shared library, they will be active under the
  context of all processes using that shared library
- the probes will be effective even on those instances of the processes that
  may be running under the control of a debugger

Some of the unique aspects of the Dynamic Probes facility are:

- Probes can be placed almost anywhere
- Probes can be placed in any executable code, including the kernel, even in
  interrupt handlers, kernel modules etc.
- Read access to all the hardware registers and write access to most of them
- Read/write access to any area in the virtual address space that is currently
  resident in physical memory
- Probes placed on an executable program or shared library are active globally
  under the context of all processes executing it
- Probes can be placed on programs that are being run under a debugger
- DProbes can be essentially used as an enabler of other debugging facilities
  like kernel debugger, trace facility and crash dump. Kernel debugger and
  crash dump facility can be invoked from DProbes by using exit n rpn 
  instruction. Data collected from DProbes can be directed to trace buffer.
- DProbes also allows the probing storage accesses, implemented using the
  debug registers available on Intel x86 processors. These probes, called the
  watchpoint probes, can be specified to be fired on a specific type of memory
  access (execute|write|readwrite|io). 

2. How to install
------------------
You can download Dynamic Probes kernel patch and application from:
http://oss.software.ibm.com/developerworks/opensource/linux/projects/dprobes.

For using the Dynamic Probes facility, download the dprobes tarball for your
version of the kernel, which contains kernel patches in patches/ directory. 
You need to apply the core patch and then apply the patch for your 
architecture. Assuming you unzipped it in /usr/src, apply as below for i386:
	$ cd /usr/src/linux
	$ patch -p1 < /usr/src/dprobes-3.x.x/patches/dprobes-vx.x.x-core-x.patch
	$ patch -p1 < /usr/src/dprobes-3.x.x/patches/dprobes-vx.x.x-i386-x.patch
	$ make *config

On S/390 you need to apply a few base patches to a stock kernel.org 2.4.17 kernel
before you apply the dprobes patches. The required base patches are:

linux-2.4.17-s390
linux-2.4.17-s390-1-lcs
linux-2.4.17-s390-2
linux-2.4.17-s390-kerntypes

available at:
http://www10.software.ibm.com/developerworks/opensource/linux390/current2_4_17.shtml

Assuming you expanded the S/390 patches in /usr/src/kernpatch and the dprobes 
package in /usr/src you would apply all the patches as follows:
	$ cd /usr/src/linux
	$ patch -p1 < ../kernpatch/linux-2.4.17-s390.diff 
	$ patch -p1 < ../kernpatch/linux-2.4.17-s390-1-lcs.diff 
	$ patch -p1 < ../kernpatch/linux-2.4.17-s390-2.diff 
	$ patch -p1 < ../kernpatch/linux-2.4.17-s390-kerntypes.diff 
	$ patch -p1 < /usr/src/dprobes-3.x.x/patches/dprobes-vx.x.x-core-x.patch
	$ patch -p1 < /usr/src/dprobes-3.x.x/patches/dprobes-vx.x.x-s390-x.patch
	$ make *config

The 2.4.19 s390 and s390x dprobes patches have no prerequisites. You, however, may
need additional patches for network drivers, etc.

The 2.4.7 s390 patches are specifically tailored for the SuSE SLES-7 kernel for 
dprobes / LTT integration. The LTT patch must be applied first and is available at:
http://opersys.com/ftp/pub/LTT/ExtraPatches/patch-ltt-0.9.5-linux-2.4.7-SuSE.gz
A separate paper with detailed installation instructions for dprobes and LTT on
S/390 will be available shortly on the dprobes web site.

The ppc port was tested on 2.4.19 without any additional patches. More
information on the ppc linux port can be found at http://penguinppc.org

The ppc64 port was developed and tested on 2.4.20 using the patch from
ftp://ftp.linuxppc64.org/pub/people/engebret/linuxppc64-2.4.20.patch.gz
This patch is required for the kernel to build. There is a newer patch
available, but we haven't had a chance to work with it yet. Probably
won't cause any problems, but hey, that's what the mailing list is for ;-)
More information on the ppc64 linux port can be found at:
http://penguinppc64.org

If the kernel is not build in /usr/src/linux, it should be linked to the actual
directory where the kernel is built. 

All the kernel build configuration options related to Dynamic Probes are under
the "Kernel Hacking" section. Select "Kernel Hook Support" (CONFIG_HOOK) first 
as it is required for DProbes. Then select "Dynamic Probes" (CONFIG_DPROBES)
to build it into the kernel ('y') or as a module('m'). Select 
"Global Debug Registers" (CONFIG_DR_ALLOC) if you require support for
watchpoint probes. After making appropriate configuration selections, build the
kernel as usual. 

If DProbes is compiled as a module, you need to also run "make modules"
and "make modules_install" and "insmod dp" to use dprobes. Please note that
the dprobes module is named dp.o.

Once the kernel is booted and dp.o is inserted (if DProbes is compiled as a 
module), find out the major number of /dev/dprobes character device from
/proc/devices and create a character device node /dev/dprobes with the
following command:
	$ mknod /dev/dprobes c <major> 0
Or, you could write a shell script for doing it as below:

rm -f /dev/dprobes
MAJOR=`cat /proc/devices | awk '$2=="dprobes" {print $1}'`
mknod /dev/dprobes c $MAJOR 0

You can use IBM's DProbes with Opersys' Linux Trace Toolkit(LTT) to
provide universal (dynamic) tracing capability for Linux. See the dprobes(8)
for details of tracing using DProbes. To use Linux Trace Toolkit as a target
for the log generated by DProbes, patches for LTT (version 0.9.5pre3 or later)
and DProbes need to applied to the kernel sources and compiled in by selecting 
the appropriate options in make *config. Note that while both the patches are 
designed to apply cleanly on unmodified kernel sources, one can apply LTT patch 
first and then the DProbes patch. The may be a few failed hunks, depending on 
the kernel version, which need to be applied manually. Refer to 
http://www.opersys.com/LTT/ for details about LTT and the required patches. 

To use exit n instruction in the probe program file(See dprobes.lang(8) to
know about exit n instruction) for interfacing with other debugging facilities,
the kernel should have support for those facilities. SGI kdb and SGI crash
dump (version 4.0 or above) are two such facilities which are available in 
the form of kernel patches. 

Note that the dprobes patch will not apply successfully if the SGI kdb
or the SGI crash dump patches are already applied. In such cases, some
hunks might have to be applied manually. Refer to http://oss.sgi.com for
KDB and crashdump patches.

You will also need to build the "dprobes" utility that provides the command
line user interface for this facility.
	$ cd /usr/src/dprobes-3.x.x/cmd
	$ make
	$ make install

If your include files are not present in /usr/src/linux/include, please
change INCLUDE_DIR in the dprobes-x.x.x/cmd/Makefile appropriately.

The make process uses the HOSTTYPE environment variable to decide which arch
to build for. If HOSTTYPE is not set to s390, s390x, ppc, or ppc64 then i386 
will be assumed. Note that ppc64 is not how HOSTTYPE is usually set on a ppc64
system. Your best bet is to read the next paragraph anyway, because building 
the dprobes command for ppc64 takes some effort at this time.

For a ppc64 environment you need to build the dprobes command as a 64bit 
application. Currently shipping ppc64 distros do not contain a 64bit userspace
toolchain and no 64bit glibc. There are instructions on http://penguinppc64.org
on how to build a 64bit tool chain. If you are installing the complete tool chain
on your ppc64 then you can build the command pretty much as normal, you may have
to modify the Makefile slightly for library paths. If you don't intend on having
the full 64bit runtime on that machine, you will want to build the tool chain 
statically (for the configure program this is usually the --disable-shared
option). A statically linked dprobes command will be about 5meg, but will work
fine. Problems can be addressed to the dprobes mailing list.


Please refer to the dprobes(8) man page for more information on the usage of
the command and to the dprobes.lang(8) man page for the format of the probe
program file and the syntax of the instructions available for use in the
probe handlers.

3. How to use
--------------
A typical usage of the Dynamic Probes facility involves the following:

- Identify the module which needs to be debugged. The module could be an
  executable program, shared library, kernel or kernel module.

- Identify the probe locations. The exact way in which the probe locations
  have to be specified is explained in the dprobes.lang(8) man page.

- Specify the actions to be taken (like examining relavent data structures,
  detecting error conditions and collecting specific data into a log buffer
  etc.) when a given probe is fired. This is the probe-handler part of each
  probe. All the probes that you want to place on a given module will have
  to be specified in a probe program file. The format of the probe program
  file and all the instructions available for a probe handler are detailed
  in the dprobes.lang(8) man page.

- Insert the probes into the module using the dprobes command. See dprobes(8)
  man page for more details of the command. Note that probes can be inserted
  into a module when it is already running or even when it is not yet loaded
  into memory (except, of course, in the case of the kernel).

- Run the program being debugged or wait for the probed code locations to be
  activated. When the probes are hit, the probe-handlers will be run and
  logs will be collected, as per the instructions in the probe-handlers.

- Examine the log collected. All the log records are prefixed with "DProbes" for
  easy identification. The major and minor codes can be used to uniquely
  identify each probe point in the system.

 - Starting with v3.6, we have the DProbes C compiler, dpcc, which provides 
   a high-level language interface to dprobes, that makes writing probe
   programs much easier. Please checkout dpcc/ directory for more details.

4. Security Considerations
--------------------------
Since this is a very low-level facility that operates in kernel mode and
provides almost unrestricted access to hardware registers and memory, it
needs to be used with caution. Only the root user has the authority to use
this facility.

Special care should be taken to make sure that the probe location specified
lies on an instruction boundary. If a probe is placed in the middle of an
instruction then the behaviour of the system may be unpredictable - it could
lead to data corruption or a system crash.

Dynamic Probes facility should be used only through the dprobes utility
provided. The dprobes system call should not be called directly by other
applications as the utility provided some extra safety checks that are
not done in the kernel. So, calling the dprobes syscall directly with
incorrect parameters could easily crash the system.

5. Known Bugs
-------------
On s390 and s390x you may run into problems single stepping with gdb across a
probe point. If you set a break point around the probe point, and use run or
continue there won't be any problems. 


6. Restrictions in the current release
--------------------------------------
Here are some of the features that are not yet available:

- s390, s390x, ppc and ppc64 ports don't support watchpoints

- dpcc only supports i386 at this time

7. License
----------
Copyright (C) 2000 IBM Corporation

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston,
MA 02111-1307, USA.

8. Version
----------
The current version of Dynamic Probes is 3.6.4.

9. Reporting Bugs
-----------------
If you encounter a bug, please report it to DProbes mailing list
	dprobes@www-124.ibm.com

10. Contributing
---------------
IBM Dynamic Probes is licensed under the GNU General Public License. If
you would like contribute to the development of this product, please
contact one of authors.

11. Authors
-----------
Richard J. Moore	richardj_moore@uk.ibm.com (Lead Architect)
Bharata B. Rao          rbharata@in.ibm.com
Michael Grundy		grundym@us.ibm.com (s390, s390x, ppc, ppx64 ports)
Subodh Soni             ssubodh@in.ibm.com
Suparna Bhattacharya    bsuparna@in.ibm.com
Thomas Zanussi		zanussi@us.ibm.com (dpcc)
Vamsi Krishna S.        vamsi_krishna@in.ibm.com (Maintainer)
