Kristof.Willen.be - Life in the trenches of the data center

Hoera ! Gisteren de overgangsproef op de hondenschool meegedaan, en onze hond is geslaagd voor de C-klas. Vanaf nu wordt het grootste deel van de oefeningen zonder leiband gedaan, en wordt van de hond verwacht dat ze de basis gehoorzaamheid volledig onder de knie heeft. Als dat maar goed afloopt ;)

I had a few spare hours yesterday, so I decided to upgrade to (the beta version of) Ubuntu 10.4, called Lucid Lynx. The upgrade went quite smooth, but I can't say I was completely happy with the end result. As it's getting usual, a typical Ubuntu upgrade is becoming two steps backwards & 3 forward. There were quite some things broken after the upgrade :

• no framebuffer with the new kernel. Fixed by upgrading to Grub2
• Gnome 1 called and wants its GDM theme back. No, seriously, GDM looks so horrible that I'm wondering there's something wrong with my config
• Some users got XFCE as a default window manager
• The upgrade to Thunderbird 3 removed some of the SMTP server settings for all users.
• The window control button placement is stupid (I know about the many discussions and the Canonical reasoning behind it, but still). Here's a guide how to place these buttons to the right place again.
• There is no support any more for the NVidia drivers. I knew I would run into this, I had to downgrade the drivers already in Karmic, but this means bye-bye to Compiz.

Not all is teeth-gnashing :

• Boot times are faster
• Overall desktop feeling is faster and snappier
• Fonts are displayed way better

So, mixed emotions with the upgrade. Hopefully there are some GDM changes before the release hits the road.

It's now about 6 weeks that I have my keyboard, the Yamaha PSR-e413. The keyboard has some cool functions, like accompaniment features (called styles), which are actually MIDI files containing some instrumentation banks, so it feels like you're part of a playing band. If you do have a Yamaha PSR, you can find some additional styles for your keyboard here.

I have been following keyboard lessons since a week of 4, and I'm making quite some progress. 'Mull of Kintyre' does however stays difficult to play, even after several training sessions.

I have been searching to some ways to extend the possibilities of the keyboard, so I could have some more synthesizer features. Therefore, I've been testing some demos of software synthesizers and audio equalizers. Cubase and Cakewalk are big names in the music producing industry, but the first had troubles identifying my keyboard, while the latter experienced quite some audio lag and stutter. Both were quite too extended for my demand, however. It was only until I looked at Reason 4, till I really found what I was looking for : a true audio synth giving me the possibility of using my gear as a master keyboard, controlling the tracks with a sound of my liking. Reason has unlimited audio modification possibilities, truely an incredible program.

I spent some time last week in Bratislava to sort out some issues with the consolidated backup environment for our Slovakian colleagues. We had some successful meetings and I could finally place a face behind some names. Meeting people face-to-face is so much better than electronic conversations.

Bratislava is a nice little city, with some Austrian accents. Unfortunately we did not have a lot of time to do much sightseeing, but I did find an Irish bar in the old town center. As it was St-Patrick day, the place was however too crowded to even consider staying there for a long time. Luckily for me, the other pub had a fine array of single malts too :)

An issue I lately encountered was that a collegue complained about several processes which kept hanging on a Solaris 10 machine. After investigation, processes like format, powermt and even a for diagnostics invoked dtrace kept hanging, and could not even be killed :

# pkill -9 format
# ps -ef |grep -c format
2

In such cases, a good old truss session mostly explains what's going on; but in this case, truss came back with a quite peculiar message :

# truss -p 26632
truss: unanticipated system error: 26632
#
# pstack 26632
pstack: cannot examine 26632: unanticipated system error
#
# pfiles 26632
pfiles: unanticipated system error: 26632

In those cases, the only option you have is to rely on the kernel debugger to determine the cause :

# mdb -k
Loading modules: [ unix genunix specfs dtrace ufs sd pcisch md ip hook neti sctp arp usba fcp fctl ssd nca lofs zfs cpc fcip random crypto logindmux ptm nfs ipc ]
> ::pgrep format
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R   1241      1    942    686      0 0x4a004900 000006001414c060 format
> 000006001414c060::thread
            ADDR    STATE  FLG PFLG SFLG   PRI  EPRI PIL             INTR
000006001414c060 inval/2000 1424 de50    0     0     0   0              n/a
> 000006001414c060::walk thread | ::findstack
stack pointer for thread 300012b7700: 2a10055cb01
[ 000002a10055cb01 cv_wait+0x38() ]
  000002a10055cbb1 PowerSleep+0x14()
  000002a10055cc71 PowerGetSema+0xe8()
  000002a10055cd31 power_open+0x364()
  000002a10055cea1 spec_open+0x4f8()
  000002a10055cf61 fop_open+0x78()
  000002a10055d011 vn_openat+0x500()
  000002a10055d1d1 copen+0x260()
  000002a10055d2e1 syscall_trap32+0xcc()

In this case, it was the PowerPath MPIO which was blocked on a semaphore. Further investigation revealed that the drivers for PowerPath were removed from the /etc/system file. Restoring the correct version of that file and a reboot solved the problem.