You are here



Multi-terabyte filesystems on Solaris


At our shop, we still use UFS as the Solaris' filesystem standard. I was puzzled when a colleague came to ask if he could use ZFS on his Solaris 10 server, because he needed a multi-terabyte filesystem, and UFS supported this, but in an awkward manner. Now, personally, I don't favor filesystems this big, because the backup alone needs special care, but sometimes some applications force you to do weird things. But I had a hard time believing that Solaris 10 couldn't support this. Seems my colleague is right :

In Solaris 10, yo need to add the -T option to newfs when trying to format a +1TB filesystem :

newfs -T /dev/rdsk/c0t1d0s1

This also will imply a hardcoded inode density, whereas each inode is 1MB big (nbpi reset to default 1048576). Which means your UFS filesystem will also only support 1 million files on your TB filesystem, which is *low*. The reason for this parameter is because a higher inode density on +TB filesystems could imply fsck times up to days. A corrupt filesystem is now a reason to invoke full disaster recovery.

AFAIK, the only possibility to change your inode settings on UFS filesystems this big, is to take the mkfs source code from OpenSolaris, change the nbpi parameter and compile it yourself.
Or change to ZFS or Veritas filesystems.

Solaris IP multipathing

Sun has a nice intro online how you can implement IP multipathing on Solaris, in case you have multiple network cards in your server, and want to loadbalance over it. Don't forget to contact your netwerk engineer !

Solaris' format coredumps on EFI-labelled disks


I tracked a nasty problem on a Solaris 10 host, which refused to start up format after adding extra disks :

# format
Segmentation fault.  Core dumped.  

Tracing the format command revealed that it barfed while reading a specific disk. Using format with specific disks worked flawlessly, but with that one disk, format kept segfaulting :

# format c2t1d42
Segmentation fault. Core dumped.

Digging further into the dump, it revealed the true reason why Solaris' format was crashing :

# pstack format.s0001620.6924.core
core 'format.s0001620.6924.core' of 6924:       format -d c2t1d42
 ff2542ec _malloc_unlocked (3c78, 5f188, 5f188, 5f188, 0, 0) + 22c
 ff2540a4 malloc   (3c78, 1, 94224, 0, ff2e8284, ff2f09b0) + 4c
 ff240e68 calloc   (1, 1, 3c78, 0, 0, 0) + 58
 ff350aac efi_alloc_and_read (5, ffbfe64c, 3c00, 1356c, 0, ff364000) + 24
 0001f6c4 read_efi_label (5, ffbfe6b8, 0, 0, d, 543f0) + 8
 0002d8c0 ???????? (ff315a12, ffbfee68, ffbfe8ec, 5, 54360, 0)

Apparently, format was choking while trying to read an efi label. An octal dump of the disk also revealed the label :

# od -c /dev/rdsk/c2t1d42s2|more
0000000  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
3140000   I   A   6   4   _   E   F   I  \0  \0  \0  \b  \0  \0  \0  \0
3140020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

Fact is that Solaris cannot cope with (probably corrupt) EFI labels; there are some patches lying around for x86, but this is a sparc machine, with no patches apparently :
"It's a nasty series of bugs that apparently have been lingering around since 1993, and were never completely fixed. Nowadays they are obscure bugs, and if you look into, for most of them there is no fix, on SunSolve there is no patch, etcetera, etcetera."

Metainit : no such file or directory


Solaris Disksuite is quite straightforward, but the error messages it spews are sometimes beyond comprehension. I was about to create a concat of 3 LUNs when I encountered this error :

# metainit d200 3 1 c0t1d97s0 1 c0t1d98s0 1 c0t1d99s0
metainit: eriador: d200: No such file or directory

It seems that the limit on metadevices is in the configuration file /kernel/drv/md.conf I had to change the "nmd" parameter from 128 to 200 to give me enough metadevices for my configuration, that is, if I wanted to keep the d200 naming (Changing kernel driver files requires a reboot, too).

The simplest solution here was to choose a lower number for the concat metadevice :

# metainit d70 3 1 c0t1d97s0 1 c0t1d98s0 1 c0t1d99s0
d70: Concat/Stripe is setup

LD_LIBRARY_PATH for setuid binaries


A colleague and I got a bit surprised when on a stripped Solaris, a program refused to work cause it couldn't find a shared library. But even after setting the LD_LIBRARY_PATH environment variable, the program behaved in the same way. In cases like that, one should try to modify the global ld path, which is /etc/ on Linux, but that file is not present on Solaris. Even greater frustration arose when it turned out that setud binaries cannot be trussed, so no list of syscalls to find out where the program is looking for its libraries.

Turns out that the use of LD_LIBRARY_PATH is restricted for setuid and setgid programs, as part of the Trusted Solaris environment. However, the list of shared library directories can be extended to the list of trusted directories in /var/ld/ld.config by use of the crle command.


Subscribe to RSS - Sun