Solaris core dump analysis with SUNWscat

I've previously tackled how Solaris core dumps can be investigated with mdb. There's another utility (comparable with Q4 on HP_UX), called SUNWscat. Scat is a tool to analyze kernel dumps (pun probably intended). Just download the SUNWscat package, install it on your server, and wait for a kernel crash to happen. When this happened, you'll find a unix.0 and vmcore.0 in the coredump directory (default /var/crash). When you fire up SUNWscat, you'll be presented with the following screen :

# scat 0
  Solaris[TM] CAT 4.1 (build 526) for Solaris 10 64-bit SPARC(sun4u)

  Copyright © 2003 Sun Microsystems, Inc. All rights reserved.
  Patents Pending. Use is subject to license terms.
  Sun Microsystems proprietary - DO NOT RE-DISTRIBUTE!

opening vmcore.0 ...dumphdr...symtab...core...done
loading core data: modules...panic...memory...time...misc...done
loading stabs...read_type_db: Wrong number of lines in database, or database
doesn't end in a newline
unable to load any stabs file
patches... - NOT AVAILABLE (No such file or directory) done

core file:      /var/crash/vmcore.0
user:           Super-User (root:0)
release:        5.10 (64-bit)
version:        Generic_112233-11
machine:        sun4u
node name:      boson
domain:         arda.org
hw_provider:    Sun_Microsystems
system type:    SUNW,Sun-Fire-V210
hostid:         837844c7
time of crash:  Tue Apr 22 11:49:52 EDT 2008
age of system:  22 hours 5 minutes 4.48 seconds
panic cpu:      0 (ncpus: 8)
panic string:   free: freeing free block, dev:0x200000016e, block:32032, ino:6057255, 
                fs:/homes

running sanity checks.../etc/system...ndd...sysent...misc...done
SolarisCAT(vmcore.0)>

The first thing you probably want to do, is investigating the crash reason :

SolarisCAT(vmcore.0)> analyze
PANIC: free: freeing free block, dev:0x%lx, block:%ld, ino:%lu, fs:%s
[...]
==== printing for generic panic information ====
cpu 0 had the panic

==== panic thread: 0x2a1003f7d40 ==== cpu: 0 ====
==== panic kernel thread: 0x2a1003f7d40  pid: 0  on cpu: 0 ====
cmd: sched

t_stk: 0x2a1003f7b50  sp: 0x1437751  t_stkbase: 0x2a1003f4000
t_pri: 60(SYS)  pctcpu: 0.000000  t_lwp: 0x0
t_procp: 0x1438518(proc_sched)  p_as: 0x1438400(kas)
last cpuid: 0
idle: 50 ticks (0.50 seconds)
start: Mon Apr 21 13:45:07 2008
age: 79485 seconds (22 hours 4 minutes 45 seconds)
stime: 2132 (22 hours 4 minutes 43.16 seconds earlier)
tstate: TS_ONPROC - thread is being run on a processor
tflg:   T_TALLOCSTK - thread structure allocated from stk
        T_DONTBLOCK - for lockfs
        T_PANIC - thread initiated a system panic
tpflg:  none set
tsched: TS_LOAD - thread is in memory
        TS_DONT_SWAP - thread/LWP should not be swapped
        TS_SIGNALLED - thread was awakened by cv_signal()
pflag:  SSYS - system resident process
        SLOAD - in core
        SLOCK - process cannot be swapped

pc: 0x104a720   unix:panicsys+0x44:     call    unix:setjmp
startpc: 0x11a53f8      ufs:ufs_thread_delete+0x0:      save    %sp, -0xd0, %sp

unix:panicsys+0x44 (0x14a3158, 0x2a1003f74a0, 0x1438120, 0x1, 0x0, 0x0)
[...]
unix:thread_start+0x4 (0x3000026e828, 0x0, 0x0, 0x0, 0x0, 0x0)
-- end of kernel thread's stack --

SolarisCAT(vmcore.0)>

The proc command, for example, can tell you about the processes that were running at the time your system crashed. These processes are listed by default in reverse PID order.

SolarisCAT(vmcore.0)> proc

    addr       pid    ppid   uid      size      rss     swresv   time  command
------------- ------ ------ ------ ---------- -------- -------- ------ ---------
0x30003c8e040    283      1      0    3776512  1646592  1302528  90118 /usr/sbin/ssmon
0x30003c96a50    279      1      0    9306112  2514944  1769472     19 /usr/sbin/ssserver
0x30003bee030    256      1      0   27656192  2596864  1138688     57 /usr/sbin/nscd
0x30003c8ea58    243      1      0    2506752  1703936   466944      7 /usr/sbin/cron
0x30003c96038    240      1      0   18874368  2170880  2711552      7 /usr/sbin/syslogd
0x30000f60010    225      1      0    7217152  2400256  1146880    170 /usr/lib/autofs/automountd
0x300020c4a40    217      1      0    2260992  1572864   598016      3 /usr/lib/nfs/lockd
0x300020c5458    213      1      1    4677632  1974272   876544      2 /usr/lib/nfs/statd
0x300020c4028    201      1      0    2629632  2048000   835584     12 /usr/sbin/inetd -s
[...]

Advanced use of scat requires an in-depth understanding of the Solaris kernel. However, you can get a lot of useful information by using just the basic commands.

Bart Coddens Wed, 10/27/2010 - 08:33

That's why I included this wonderfull tool in your default jumpstart profile :)

Eduardo Tue, 05/31/2011 - 22:15

I believe this tool is wonderful, but how can I get this tool ?
It´s really dificult to find this tool.

kristof Thu, 06/02/2011 - 16:05

In reply to by Eduardo

The transition from Sun to Oracle made the previous sunsolve website kind of a mess. I would suggest to retrieve the tool from the OpenSolaris webservers.
Oracle itself removed the tool apparently...

Maverick

I upgraded my machines to Maverick - nothing much too say, except that the upgrade went pretty smooth. The upgrade on my netbook was the most interesting : the Ubuntu Netbook Remix interface (UNR) was about to be replaced with a brand new interface, called Unity. Now, I still don't like the fact that Ubuntu forces the user interface with every last upgrade to the standard one, but with Unity, all now makes more sense :

  • The placing of the close-minimize-maximize buttons is excellent for minimal mouse gestures
  • The dashboard alike interface of UNR gets replaced with a vertical Windows7/OSX alike dock. With the default maximization of windows, this works great.

However, in Unity, there are still some annoying bugs :

  • I sometimes need to click twice on a startup button before the application starts, as if the dock is too less sensitive.
  • The window title sometimes gets overwritten with other stuff
  • The Ubuntu logo should display a menu if clicked, instead of minimizing all windows. If you want to fire up a non-docked application, you need to go to the Applications button, which displays all installed applications with their icon (which is kind of a mess)

Apart from that, very neat interface.

Rotating wallpaper

I've been working a while on a Windows 7 machine, and one of the nice desktop blings there is the rotating wallpaper item. There are quite some sets available, and you can download many more (eg like the famous Bing backgrounds). Linux has this functionality already since some time, but the sets of available wallpapers there are quite limited.

I've played around with Drapes, but the rotation time is quite small (I want maximum one wallpaper rotation per day). So why not write a script which downloads automatically some wallpaper from sites like APOD, Bing or National Geographic ?

Here are some scripts I found on the internet which do this for you. Place them in your crontab with a frequency of your liking.

Bing

#!/bin/bash
# modify this, if you want to save to other place
#  - $HOME: your home dir
#  - %F: full date (%Y-%m-%d)
#  - other date format: man date :P
OUTPUT=`date "+$HOME/Pictures/bing_bg.%F.jpg"`

# get the bing.com page, and separate the background image (DONT touch this)
IMG=`wget -q http://www.bing.com -O- | sed -r "s/[\]//g;s/.*g_img=\{url:'([^']+)'.*/\1/gp;d"`

# get the background image to the output (DONT touch this)
wget -q "http://www.bing.com$IMG" -O "$OUTPUT"
# error handle
if [ $? -gt 0 ]; then
        echo "there is a problem with downloading.."
        exit 1
fi
# Setting background-image.
# NB: Use an absolute URL
gconftool-2 -t string -s /desktop/gnome/background/picture_filename "~/Pictures/bing_bg.%F.jpg"
gconftool-2 -t string -s /desktop/gnome/background/picture_options "zoom"

National Geographic

#!/bin/sh
#Based on code of APOD, you can really find everywhere on the web.

#Downloading html of the Picture-Of-The-Day
wget -N http://photography.nationalgeographic.com/photography/photo-of-the-day/ -O /tmp/ngpod.html
#Getting the URL of the image
img_location=`egrep -o "http://images.nationalgeographic.com/[^]*\.jpg" /tmp/ngpod.html|head -n 2|tail -n 1`

rm /tmp/ngpod.html

#Download image
TODAY=$(date +'%Y%m%d')
wget $img_location -O ~/Pictures/$TODAY.jpg

# Setting background-image.
# NB: Use an absolute URL
gconftool-2 -t string -s /desktop/gnome/background/picture_filename "~/Pictures/$TODAY.jpg"
gconftool-2 -t string -s /desktop/gnome/background/picture_options "zoom"

B-klas, #2

Hoera ! Onze jongste hond Jitta, een goeie 7 maand oud ondertussen, heeft met vlag & wimpel de overgangsproef doorstaan voor de B-klas. De saaie oefeningen zijn voorbij, vanaf nu begint de training zonder leiband...

The quest for a new netbook

Netbooks are laptops done right. I had no idea how true this was before I actually bought a netbook myself. While only being slightly larger than a DVD-cover, my eeePC-900 was so portable, I've taken it with me around the world, both for work and holiday trips. My eeePC has been indeed so successful, it has completely wiped out my need for my laptop, which has been mostly gathering dust since the netbook purchase.

However, a (first generation) netbook still has some serious disadvantages :

  • disk is mostly a combination of SSD and SD, and very limited in space.
  • the keyboard is way too small to type comfortably
  • the battery is very limited in life expectancy.

My new netbook had to overcome those three limitations. Not a big deal, since most current netbooks deal with this already. In addition, I wanted a minimum of 2GB RAM and a CPU with virtualization possibilities. As I was very happy with the eeePC line, I almost opted for a eeePC-1201HA, which sports the Z520 CPU, which had Intel-V support. Unfortunately, the netbook got slaughtered in every review because of its slow performance.

My final choice was the Samsung N220 Premium Plus, a N450 based laptop with 2GB RAM and a 350GB hard disk. As the laptop is red, I feared a bit for too much of hardware bling, but the color is nicely darkish red, so it doesn't scream out in a meeting room. So far, I've been really happy with the netbook itself. The following are only (very) minor annoyances, but indeed are things that could have been better :

  • If some netbook manufacturer brands a netbook with a Premium label, I expect the least they can do, is to include a pouch with it. As a netbook gets carried along everywhere, you want a cover for it to avoid excessive scratching, unless it is not made in plastic. No pouch with my edition, though I've heard that latest Samsung netbooks come again with this addition.
  • The Power button comes in some sort of a slide button, which is on the front of the netbook. A true button would have been better, and would have been better if been protected by the lid.
  • If the lid is closed, it is very hard to see if the netbook has been suspended, as the LEDs are hidden by the lid itself.

What is exellent is that netbook is completely silent : the hard disk is perfectly mute, and the fan makes only a slight noise under high stress. The netbook comes with Windows 7 Home Premium Edition, which starts its installation when you power on for the first time. The good thing is that it lets you choose the partitioning, so the hard disk is split by default into 3 partitions :

  • first partition carries the Windows7 OS
  • the second partition is a 14GB restore partition
  • the third is an empty D-drive, which can be used for data & installed programs.

In a next post, I'll describe what tweaks were necessary to install & use a 64bits Ubuntu on the 3rd partition.