Notes for ST XXX: Kernel core dump analysis.

Use dumpadm command to show/change panic dump config.
This changes /etc/dumpadm.conf file.

By default the swap is used as dump device. Any other harddisk partition
or normal file system file could be used as dump device too.
By default only kernel pages are dumped. you can configure to dump all memory 
pages, too.

e.g. dumpadm -c all -d /dev/dsk/c0t0d0s2 -s /export/cores

The dumpadm.conf file is sourced during rc bootup and savecore is
executed if save core is enabled during reboot.

savecore: reads dumpadm output from the disk and generates kernel
pages (unix.0) and memory pages (vmcore .0) in separate files.

unix.0  is same as /dev/ksyms for running kernel. This is in ELF format
with symbol table and no data and no text.


Lab 01 :
Use swap -d to delete your swaps. It complains that the dumpadm
dumping is disabled. Then use dumpadm to specify the swap disk partition
to be dump device. Create mkfile file system file and add it for swap.
Create a save core file directory for reading the dump from the dump device
and create save core files unix.0 and vmcore.0.

Use savecore -L command to create core files of running live system.
Now crash the system by doing adb -k and rootdir/W 0 (i.e. make rootdir
global variable a null pointer).

Now reboot and see the savecore files being created during boot from
rc script. Look at save core files and see unix.1 and vmcore.1 got created.

Look at /var/adm/messages for mesgs before panic.

You can also sync and >g0 to execute illegal instruction and do a panic.

To capture deadman code (i.e. to find hard hang) set snooping=1 in
/etc/system. This detects the hang and reboots the system automatically.

Compressing vmcore/unix files gives around 3 times space savings.

* The script iscda (available from sunsolve) captures many standard system
  information from unix.1, vmcore.1 save core files (or from running system)

iscda unix.1 vmcore.1 > iscda.out

It is a good script which prints out many system information like utsname,
standard msgbuf, cpu structures etc. 


Lab 03: Inspect execution of live processes.
truss a simple program. sotruss is the utility to trace only shared
library procedure calls!!
See also: whocalls which prints who has called a specific function
truss -ulibc <program> traces both system call and library calls.
See man truss.

adb syntax:

addr,count [/?=] format
somesymbol,3/X
somesymbol/3X  --> above two are same.
Note that count could be expression. Conditionally it could be even 0.
The boolean expr #(X-Y) is true if (X != Y) ???
somesymbol=J --> print the symbol address as 8 bytes hex.
0t555=X --> display in hex
555=D   --> display in dec
$r -- display registers
<sp $< stack --- print stack
$>outputfile -- redirect output
$>  -- restore output to stdout
<g7=J  -- display g7 register in 64 bit value.
$<<file  -- nested macro. $<file is not nested macro.

*panicstr/s
prints string at panicstr. Note there is a two deref here since "/" 
has already applied one level of deref.
utsname+(2*0t257)/4t"release:"16ts
First 0t257 is a const. In the format 4t is 4th tab, then print as string.

addr/DDf --> if you have struct with int, int, float this will work !!

sparc assembly code:
sethi %hi(0x10800), %o0 --> set high 22 bits. Instns are left to right.
analyzing a small program, by stepping through by :s 

use: proc2u to find out uarea fields like open files entries, etc.

There are 32 general purpose registers : r0 to r31
From high to low: Input, Local, Output, Global registers.
Input Registers: r31 to r24 Or %i7 to %i0. 
       %i6 is frame pointer. 
Why do you need sp as well as fp ???

Local  Registers: r23 to r16: %l7 to %l0 
Output Registers: r15 to r8 : %o7 to %o0  
        %o6 : Stack pointer
        %o7 : return pc

Global Registers: %g0 to %g7
%g0 is always zero
%g7 is current kernel thread.

As soon as enter function:
kcd_demo:       save    %sp, -0xd0, %sp
This single instn (not macro) saves sp and decrements sp ???
Debugging kcd crash:

$c (stack trace)
$<msgbuf
Watch out for die() in $c, and second parameter is the address at
which all registers were stored.
die(31,2a1004fd690,10416020,0,14,2a1004fd690) + b0
2a1004fd690 $< regs
Note down pc:
1027e0dc/i
Find out stack pointer, ...
0x2a1004fcf31 $<stack
The display is 16 words. The 15th word is the fp which is the stack
pointer for the below function. Use that in sequence:
2a1004fd011 $< stack

You can search kernel memory area for a data content:
scb/M 0xconstant  : M is 64 bit. L is for 32 bit.
once you find . is moved to that addr. If you want to search more:
.+8/M 0xconstant

proc-->as-->seg-->segvn_data-->vnode-->inode

watch one process.
ps -ef |grep ksh

adb -k
0t413 $< setproc
<proc  (or . <proc)

In the proc find as.
30000038970 $< as

Find segs:

30000ef94c8 $< seg

From there, find "data", which is vnode type. Since the ops is
segvn_ops, the data is segvn_data

30000ebced8 $< segvn_data

amp is 0. There is a  vp which is vnode pointer:

30000b52780 $< vnode
The op says ufs_vnodeops ! The data here is really inode.

30000b526f0 $< inode
we got a number here: 201014

goofy:/# ncheck -i 201014
/dev/dsk/c0t0d0s0:
201014  /usr/bin/ksh
201014  /usr/bin/rksh


Given inumber, to find the file, you can check:
ncheck -i <inumber>
(Or use find -inum )

Instructor: marc@hybridmedia.com
rolando.dizon@ebay.sun.com

Take stack pointer, and use following command. It displays
all the calls down in the stack:
0t...$<pid2proc
procaddr$<procthreads
<stackp_addr>$<stackregs