Kristof.Willen.be - Life in the trenches of the data center

The Ben Nevis distillery, now overtaken by the Japanese Nikka distillers, is named after & located at the foot of the highest mountain in Scotland (1334m). For one or other strange reason, Ben Nevis is called the Banana Whisky. My interest in Ben Nevis was sparked by my colluegue Peter B., who refers it as one of the best whiskies ever made, and very difficult to find. However, I had no problem locating it in my favorite dram shop.

The color : dark amber
The nose : orange with chocolate flavors. Some maltiness, then spices are flowing in.
The taste : *Very* malty, quite spiced, bit of pepper. Lots of dark chocolate, the very bitter taste of orange zest. Bit of spice & smoke. Smooth but very firm. Not complex at all, warm.

Where are the bananas ? Probably a referral to the littering of banana peels on the peak of Ben Nevis ?
This whisky has a very strong taste, especially empowered by the malty taste in combination with the bitterness of chocolate and orange. I'm sorry, Peter, too bitter for my cup of tea.

Temperatures are soaring outside, not really the time for a dram. Or is it ? Some people drink whisky at any season, others prefer a warm fireplace while the rain is pouring outside. Count me in the last group. However, duty calls, so here's a new tasting note. Springbank is the oldest independent distillery in Scotland, and created in Campbeltown. From a traveller's perspective, visiting the Campbeltown area is a small nightmare. One has to make a long trek to the Southern tip of the Kintyre peninsula on the Western coast of Scotland to reach Springbank - or the nearby Glen Scotia distillery. Except for the distilleries, there's not much happening in Campbeltown. And even for the people that want to 'get away from it all' for a few days, Springbank is a fairly poor destination. It's one of the few distilleries that isn't surrounded by the lush Scottish countryside - located in the middle of a busy town.

Springbank CV is an unusual whisky, for several reasons : first, it’s not quite clear what ‘C.V.’ used to mean. Some say ‘Chairman’s Vat – or Vatting’, others ‘Curriculum Vitae’... It contains a blend of different single malts between the age of 8 and 30, therefore presenting a 'taste visiting card' of Springbank. Then, there’s been several versions, notably an earlier ‘white cap’ version that’s the one we’ll have right now, and then a more recent ‘gold cap’ version. Second, it has a quite complex tasting :

Color : *very* pale gold, almost white wine.
Smell: complex. Spirits and grain. Spicy. Adding drops of water emerges a burst of pepper. Is that fruit there hiding in the back ?
Taste : oily, malt and more spices. So much pepper, it makes my mouth tingle ! Tears in me eyes. Again some fruit (lemon ? Pear ?) hiding in the pepper cloud. Lots of other stuff too, like liquorice, peat and some bitterness but almost killed immediately by spice and pepper.

Aaa-choum ! Did I mention the pepper ? This could have been a balanced complex whisky, but unfortunately too spicy for me.

Deze kievit komt regelmatig in onze tuin op bezoek. Van ver lijkt hij op een zwart-witte duif met een uitgesproken kuifje. Bij het inzoomen echter blijkt dit vogeltje een mooi metalig grijs-groen vederkleed te hebben. Je treft hem aan in vochtige weilanden met kort gras, waar hij ook graag in nestelt. De kievit komt in het najaar in grote groepen samen om naar het warme zuiden te trekken, tijdens dewelke hij zwaar bedreigt wordt, gezien hij graag geschoten wordt, voornamelijk in Denemarken en Ierland. De naam is vernoemd naar zijn 'kievit'-achtige roep.

With flights STS-133 (launching at October 29th 2010) and STS-134 (Febrauary 29th 2011), the space Shuttle program will come to a halt. The program started in 1981, and was meant to provide a cost-saving alternative for the 'throw-away' Saturn rockets which were used in the 60ties and 70ties. The space shuttles Discovery and Endeavour will be the last shuttles going into space, delivering new scientific equipment to the International Space Station.

If you want to 'participate' in the last two shuttle missions, you can : through the Face in Space website, you can upload your picture, which will be taken on the shuttles, thereby be launched into orbit and "become a part of history".

We've tackled previously how to look at kernel dumps on HP-UX, let's have a look now how to perform them same on OpenSolaris. The kernel debugger is actually 'quite' user-friendly, and gives you mostly enough information how to handle a crash. If your Solaris is too stable to generate crashes, then use the

savecore -L 

command to generate one on the fly. This will generate a dump in /var/adm/crash. Let's have a look at it with mdb :

# mdb -k unix.0 vmcore.0 
Loading modules: [ unix krtld genunix specfs dtrace cpu.AuthenticAMD.15 uppc pcplusmp ufs ip sctp usba lofs zfs random ipc md fcip fctl fcp crypto logindmux ptm nfs ]
>

The ::status command will display high level information regarding this debugging session. This is mostly a one-liner, which reveals the reason of the crash.

> ::status
debugging crash dump vmcore.0 (64-bit) from hostname
operating system: 5.11 snv_43 (i86pc)
panic message: BAD TRAP: type=e (#pf Page fault) rp=fffffe80000ad3d0 addr=0 occurred in module "unix" due to a NULL pointer dereference
dump content: kernel pages only

The ::stack command will prove you with a stack trace, this is the same thing trace you would have seen in syslog or the console.

> ::stack
atomic_add_32()
nfs_async_inactive+0x55(fffffe820d128b80, 0, ffffffffeff0ebcb)
nfs3_inactive+0x38b(fffffe820d128b80, 0)
fop_inactive+0x93(fffffe820d128b80, 0)
vn_rele+0x66(fffffe820d128b80)
snf_smap_desbfree+0x78(fffffe8185e2ff60)
dblk_lastfree_desb+0x25(fffffe817a30f8c0, ffffffffac1d7cc0)
dblk_decref+0x6b(fffffe817a30f8c0, ffffffffac1d7cc0)
freeb+0x89(fffffe817a30f8c0)
tcp_rput_data+0x215f(ffffffffb4af7140, fffffe812085d780, ffffffff993c3c00)
squeue_enter_chain+0x129(ffffffff993c3c00, fffffe812085d780, fffffe812085d780, 1, 1)
ip_input+0x810(ffffffffa23eec68, ffffffffaeab8040, fffffe812085d780, e)

The ::msgbuf command will output the message buffer at the time of crash; the message buffer is most commonly used by sysadmins through the "dmesg" command.

> ::msgbuf
MESSAGE                                                               
....
WARNING: IP: Hardware address '00:14:4f:xxxxxxx' trying to be our address xxxx
WARNING: IP: Hardware address '00:14:4f:xxxx' trying to be our address xxxx

panic[cpu0]/thread=fffffe80000adc80: 
BAD TRAP: type=e (#pf Page fault) rp=fffffe80000ad3d0 addr=0 occurred in module "unix" due to a NULL pointer dereference

sched: 
#pf Page fault
Bad kernel fault at addr=0x0

One of the coolest commands is the cpuinfo -v command, which will show more information about the running processes at the time of the crash, including some nicely ascii-art style formatting :

>  ::cpuinfo -v
 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  1 ffffffff983b3800  1f    1    0  59  yes    no t-0    fffffe80daac2f20 smtpd
                       |    |
            RUNNING <--+    +-->  PRI THREAD           PROC
              READY                99 fffffe8000bacc80 sched
           QUIESCED
             EXISTS
             ENABLE

Other interesting commands are the ::ps (info about running processes), and ::panicinfo, which will reveal thread information, which you can further investigate with the ::walkthread option.

In a following article, I'll write about the Solaris Core Analyzer, which is a Q4 comparabe tool on Solaris to walk through kernel dumps.