Storage

Storage and backup

Howto restore deleted files from ext3 ?

Topics

It happens to everyone sooner or later: a split second after you hit Enter you realize your mistake, but it's too late; you just deleted a valuable file or directory for which no backup exists. Or maybe you have a backup, but it's a month old... and in shock you see the past month flash before your eyes as you realize in pain what you'll have to do all over again...

Fortunately, you remember that files are never really deleted, at most overwritten by new content. So, you remount the disk read-only as fast as possible. But now? On February 7th, 2008, Carlo Wood accidentaly deleted his whole home directory: over 3 GB of data, deleted with rm -rf. The only backup he had was from June 2007. Not being able to undelete was unacceptable. So, he ignored what everyone tried to tell him and started to learn how an ext3 file system really works, and what exactly happens when files are deleted... This is not for the faint of heart, though...

Taking backups may not allways be enough : sometimes, you better take a snapshot image of your boot partition (including /usr and /var), and your MBR. dd might be your friend, but be carefull about the block size. Freshmeat has an interesting article how to use partimage to create clones of partitions.

Snapshot-Backups with FlyBack

Topics

FlyBack is a tool similar to Apple's TimeMachine. It is intended to create snapshot-backups of selected directories or even your full hard drive. From the FlyBack project page: "FlyBack is a snapshot-based backup tool based on rsync. It creates successive backup directories mirroring the files you wish to backup, but hard-links unchanged files to the previous backup. This prevents wasting disk space while providing you with full access to all your files without any sort of recovery program. If your machine crashes, just move your external drive to your new machine and copy the latest backup using whatever file browser you normally use."

Git as a backup tool

Topics

Backup and version control tools are in fact the same : their job is to keep different versions of files. Using git as a backup solution is an interesting point of view for this hip version control tool. If you're interested in start using Git altogether, check out this tutorial.
Update : EigenClass is a git backup implementation in OCaml.
Update2 : an important remark is that Git does not preserve file ownership. This might be a no-go for restore operations, but for personal file backup this might well do the job you're looking for.

TSM as a home backup solution

Topics

I previously reported on how to install TSM on Ubuntu. Today I'm going to explain how to use TSM as a backup solution for everyday use. This might sound weird, as most people think professional backup solutions only offer VTL or tape backups; however, we'll create a FILE devclass as a virtual tape, which will allow us to use disk (eg an external USB disk) to store our backup data.

Here's the setup : a typical Windows/Linux client base, combined with one TSM server running on Linux. We're assuming that the backups will be put on an external USB disk, mounted at /mnt/DATA on the server, that TSM has been installed (and is up and running), and that the client package has been installed. In the install article about TSM, we placed our DB and log volumes in the TSM install directory. If you're really paranoid, you can place them on the external USB disk, but then you must take care that the disk is connected every time TSM starts up.

First, start up TSM by typing ./dsmserv from the TSM Server install directory (/opt/tivoli/tsm/server/bin); time to register an admin, and grant him system authorities :

reg admin myname mypasswd passe=9999
granth auth myname classes=system

Time to create the device class, we'll call it VTAPE, and we'll assign a capacity of 256 MB to it, which means our virtual tapes can hold 256 MB on them :

# mkdir -m 755 -p /mnt/DATA/Backups/TSM/Volumes
define devc VTAPE devtype=file maxcapacity=256M directory=/mnt/DATA/Backups/TSM/Volumes
define stgpool TAPEPOOL VTAPE pooltype=primary maxscratch=1000

The last command defines a tape pool containing maximum 1000 tapes, using our virtual tape device class.
Don't forget to change the default Policy Set's destination to the TAPEPOOL storage pool :

upd copygroup STANDARD destination=TAPEPOOL
validate policyset STANDARD STANDARD
activate policyset STANDARD STANDARD

We'll create a maintenance script with the TSM webadmin interface; an example of a maintenance script is something like :

backup db devc=vtape type=full wait=yes
expire inventory skipdirs=NO wait=yes
prepare
delete volhist todate=today type=stgreuse
delete volhist todate=today type=stgnew
delete volhist todate=today type=stgdelete

Time to configure the client(s). Let's back up our Ubuntu workstation by defining a client UBUNTU :

reg node UBUNTU passwd passexp=9999 USERID=NONE domain=STANDARD type=client

Create the following dsm.sys file in the /opt/tivoli/tsm/client/ba/bin directory which will backup the /home and /etc directories :

SErvername  mytsmsrv
   COMMmethod         TCPip
   TCPPort            1500
   TCPServeraddress   UBUNTU
  
   PASSWORDACCESS GENERATE
   SCHEDMODE POLLING
   NODENAME UBUNTU
   SCHEDLOGNAME /opt/tivoli/tsm/client/ba/bin/dsmsched.log
   errorlogretention 3
   resourceutilization 10
   SCHEDLOGRETENTION 3

   DOMAIN / /home 
   EXCLUDE /.../*
   INCLUDE /etc/.../*
   INCLUDE /home/.../*

Now time to test the client connection; run the dsmc binary, the first time it will ask you for a user id (just press enter, it's offering our client node name). The password is the 'passwd' password string we entered in the 'reg node' command :

# ./dsmc
IBM Tivoli Storage Manager
Command Line Backup/Archive Client Interface - Version 5, Release 2, Level 3.0 
(c) Copyright by IBM Corporation and other(s) 1990, 2004. All Rights Reserved.

Node Name: UBUNTU
Session established with server SERVER1: Linux/i386
  Server Version 5, Release 2, Level 2.0
  Server date/time: 07/30/2007 19:10:37  Last access: 07/30/2007 18:42:27

Please enter your user id : 

Please enter password for user id "UBUNTU": 

Session established with server SERVER1: Linux/i386
  Server Version 5, Release 2, Level 2.0
  Server date/time: 07/30/2007 19:10:56  Last access: 07/30/2007 19:10:37

That's it ! You can start a backup session using :

# ./dsmc i

Happy backupping !

Installing Tivoli Storage Manager Server on Ubuntu Feisty

Topics

I recently installed Tivoli Storage Manager, also called TSM, on my Ubuntu Feisty PC. The installation needs some clarification :

a. first of all, IBM offers a Linux version of TSM, but in rpm format. You need to convert these to deb's with the alien command. Don't bother to convert the preinst and postinst scripts. After that, install tivsm-server, tivsm-license, and if you like the webadmin interface, tivsm-webadmin and tivsm-webhelp.

b. If you try to startup dsmserv, it will die with the following error :

ANR9999D dsmserv.c(642) Unable to open language en_US for message formatting.
ANR9999D dsmserv.c(650) Unable to open language AMENG for message formatting.

If you strace the dsmserv, you'll notice that it is looking after a dsmen_us.txt; simply symlink this file to the present dsmen_US.txt.

c. Next, you'll need to create and format the DB and log volumes; TSM wants these apparently present while using the format command, so create the files by :

# cd /opt/tivoli/tsm/server/bin
# mkdir volumes
# dd if=/dev/zero of=./volumes/dbvol.dsm bs=1024 count=16000
# dd if=/dev/zero of=./volumes/logvol.dsm bs=1024 count=16000
# ./dsmserv format 1 ./volumes/logvol.dsm 1 ./volumes/dbvol.dsm
Don't worry about the sizes, we'll replace them later with better suited sized volumes. Time to start the TSM server :
# ./dsmserv
That's it, your backup server should be up & running. Create an admin user, and test it by using the webadmin on port 1580 of the server. Set up the correct DB and log volumes, and start creating devclasses and such. Happy backupping.

LTUG 2007, Denver

Topics

I just returned from a week in Denver, Colorado where I joined the LTUG 2007 conference. LTUG is the Large Tape User Group, a consortium of StorageTek customers. The 2007 conference sessions displayed roadmaps of future software and hardware and was this year first congress which featured Open System sessions. Take a guess how the road to the StorageTek headquarters is called ? Here are some more pictures.

TSM Web admin GUI in TSM5.3

Topics

Since TSM (IBM Backup Solution) version 5.3, the old web Admin GUI has been replaced by a servlet called the ISC console. Now, generally everyone dislikes ISC, cause it's slow, and uses quite some resources. One can, however, restore the old admin GUI in 5.3 :

IBM has made the old gui available for download. The site contains an installation procedure as well. After installation, you'll notice that the timeout parameter is set quite low. This is not the IdleTimeOut parameter found out in the options set. Modify the timeout value by issueing the command :

set webauthtimeout 60

which would define a web admin timout value of 60 minutes.

Gentlemen, start your backups !

Topics

It is now two weeks my workstation has died; the long wait for a technician (the box is still under guarantee) is finally over next Monday. In the meantime, I dug out my old workstation, a P3-900 MHz, which I didn't touch for more than one year. The box runs Debian unstable, for which I dist-upgraded most parts without much hassle : X, Gnome, ssh, Firefox, Thunderbird, and basically any client program which connects to the internet. The only problem is memory : the machine only has 384 MB RAM and a swap space of 256MB, which is insanely low with current desktops like Gnome or KDE. Browsing with IceWeasel (Debianese for Firefox) feels like driving a 20 ton truck. One week countdown...

This incident made it clear that I must give some priority to backups : previously I made a seldom backup on CD-ROM or DVD, but that is work-intensive, and not much fun to do. So - with the current storage prices plummeting - I decided to shop for an external hard disk and I chose for a Packard Bell hard disk of 320GB. A quick rsync one-liner backed up most of my current files, but when I have some more time, I need to look around for some other backup methods. Maybe snapshotting (rsync or dervish), or keeping my homedir in CVS seem nice and fun projects to do.