# dtrace -l
ID PROVIDER MODULE FUNCTION    NAME
1  dtrace                      BEGIN
2 dtrace                       END
3 dtrace                       ERROR
4 lockstat genunix mutex_enter adaptive-acquire
5 lockstat genunix mutex_enter adaptive-block
6 lockstat genunix mutex_enter adaptive-spin
7 lockstat genunix mutex_exit  adaptive-release
... many lines of output omitted ...
#

# dtrace -l | wc -l
30122


Thread local variables use "self->varname"
For performance reasons, use thread specific predicates whenever possible.

syscall::read:entry
{
   self->read = 1;
}

Use aggregation function. At the end of prog, aggregation is displayed:

syscall::write:entry
{
      @counts["write system calls"] = count();
}

# dtrace -s writes.d
dtrace: script ?./writes.d? matched 1 probe
^C
write system calls 179

tracemem()
void tracemem(address, size_t nbytes)

stack()
void stack(int nframes)
void stack(void)
The stack() action records a kernel stack trace to the directed buffer

ustack()
void ustack(int nframes, int strsize)
void ustack(int nframes)
void ustack(void)
The ustack() action records a user stack trace to the directed buffer. 
If nframes is not provided, the number of stack frames
recorded is the number specified by the ustackframes option. 
If strsize is specified and non-zero, ustack() will
allocate the specified amount of string space, and use it to perform 
address-to-symbol translation directly from the kernel.


# dtrace -n syscall::write:entry?/pid == $target/{ustack(50, 0);
exit(0)}? -c "java -version"

# dtrace -n syscall::write:entry?/pid == $target/{ustack(50, 500); exit(0)}?
-c "java -version"


Use of aggregation using ustack :

syscall::brk:entry
/execname == $$1/
{
   @[ustack(40)] = count();
}

To run this example for the Netscape web browser:

# dtrace -s brk.d .netscape.bin
dtrace: description ?syscall::brk:entry? matched 1 probe
^C
libc.so.1?_brk_unlocked+0xc
88143f6
88146cd
.netscape.bin?unlocked_malloc+0x3e
.netscape.bin?unlocked_calloc+0x22
.netscape.bin?calloc+0x26
.netscape.bin?_IMGCB_NewPixmap+0x149
.netscape.bin?il_size+0x2f7
.netscape.bin?il_jpeg_write+0xde
8440c19


jstack()
void jstack(int nframes, int strsize)
void jstack(int nframes)
void jstack(void)
jstack() is an alias for ustack() that uses the jstackframes option for the
number of stack frames the value specified by , and for the string space size 
the value specified by the jstackstrsize option.

#pragma D option destructive
#pragma D option quiet
proc:::signal-send
/args[2] == SIGINT/
{
printf("SIGINT sent to %s by ", args[1]->pr_fname);
system("getent passwd %d | cut -d: -f5", uid);
}

printf %S : unprintable characters are printed as hexadecimal.


If there are too many cross calls between applications :

Create a file named xcall.d and enter the following text into the file:
#!/usr/sbin/dtrace -s
sysinfo:::xcalls
{
@[execname] = count();
}


# ./xcall.d
The initial output of the script will indicate how many probes the specified 4-tuple
(sysinfo:::xcalls) matched.
dtrace: description 'sysinfo:::xcalls' matched 2 probes
DTrace will begin to execute and gather information on the cross calls being issued on behalf of the
processes running on the system. Wait as long as desired and break the DTrace execution with an
interrupt (typically by pressing Ctrl-D). DTrace will then report its findings:
bash 2
cron 9
uname 24
sched 423
dtrace 7415
java 50345


Example 	           Explanation
pid2439:libc:malloc:entry  entry into the malloc function in libc 
pid1234:a.out:main:return  return from main for process id 1234
pid1234:a.out::entry       entry into any function in 1234 
pid1234:::entry            entry into any function in any library



How to get the last 100 mallocs stacktraces by a specific process ?

dtrace:::BEGIN
{
i = 0;
}

dtrace:::END
{
trace("Counter at end is");
trace(i);
}

syscall::brk:entry
/execname == "thava.out"/
{
   @mystack[ i++ % 10 ] =  ustack(40);
}


Checkout /usr/jdk/latest/sample/dtrace/hotspot
and /usr/demo/dtrace scripts.

Running dtrace to monitor java program:

e.g.
cd /usr/jdk/latest/sample/dtrace/hotspot
./gc_time_stat.d -p java_pid

Running dtrace with chime GUI, See:
http://www.int.com/presentations/dtrace_chime/

To examine java process:
Use the JVM option: -XX:+ExtendedDTraceProbes
? Or use jinfo
jinfo -flag +ExtendedDTraceProbes <java_pid>

Note: use of dtrace function trunc(<aggregation>,<num_rows>) is preferred.