LTSPFS - The LTSP FileSystem

LTSPFS is now part of the official Ltsp-4.2 distribution.

Introduction:

ltspfs is a remote filesystem consisting of two parts: 1) A network server daemon that runs on the LTSP terminal. 2) A FUSE module that runs in user-space on the server, that connects with the daemon on the client.

The goals of ltspfs are:

1. Provide a lightweight file access mechanism that will be feasable on lower end hardware.
2. Provide a stateless file access method that will feature "atomic" reads and writes to minimize impact from client network disruptions.
3. Provide a network filesystem that handles client reboots and disconnections in a manner that doesn't leave inaccesible, unmountable filesystems on the LTSP server.
4. Provide a network filesystem that can easily handle the oddities of dealing with removable media, and integrate well with udev (LTSP's preferred device handling support).

Previous methods tried:

1. floppyd and mtools.
   • Simple to install and maintain, simple authentication method that doesn't rely on userids.
   • However, it's only really meant for floppies.
   • With floppyd, you don't really open a file and edit it. instead, the entire file gets copied up to the server, and when it is written, the entire file gets pushed back down to the client.
2. NFS exporting of local media:
   • Conceptually simple, NFS protocols already exist in Linux kernels.
   • However, NFS doesn't handle media changes well, and using automounter is out, as underneath the covers it uses NFS, and you can't re-export an NFS mounted filesystem.
3. Samba:
   • Runs as a user process on the terminal, handles media changes better, smb mounts supported under most versions of Linux kernel.
   • Sometimes, cdroms and floppies appear as 0-byte files
   • Samba doesn't properly deal with very small filesystems, such as 1.4mb floppies. The filesystem size appears as zero, causing Nautilus to refuse to write to it.
   • However, it's very large, extremely complicated, and has problems on less powerful terminals, or terminals with smaller amounts of memory.

How LTSPFS addresses these problems:

1. LTSPFS is lightweight: the size of the binary running on the thin client is under 20 kilobytes. Each file operation will normally consume under 10 kilobytes of stack space. Read and write operations are usually done with a 4 kilobyte block, which is malloc()'d and free()'d on the heap.
2. LTSPFS is removable filesystem friendly. It's $CWD stays at the root. When you "mount" an LTSPFS filesystem, it does path translation to make the filesystem semantics work correctly. For instance: Lets say you ltspfs mounted the /tmp/drives/floppy directory on the LTSP terminal to /home/sbalneav/drives, and wanted to read the file "foo.txt". Normal unix filesystem semantics makes this look like a read of "/foo.txt" on the remote end. The normal way to handle this sort of thing is to chroot() to "/tmp/drives/floppy" when you get the mount request, and then proceed with the filesystem call, which will now work, as "/foo.txt" is valid in the chrooted environment. However, if /tmp/drives/floppy on the LTSP terminal goes away (either because the floppy was ejected, or the usb floppy was unplugged from the terminal), the chroot()ed process is now in a bit of trouble, as it has lost it's root. The way to handle this is to stay in "/" (which should always exist), and perform the right set of path translations to turn filesystem operations on "/foo.txt" into filesystem operations on "/tmp/drives/floppy/foo.txt".
3. LTSPFS operations are "atomic". LTSPFS doesn't leave files open at all. By this, we mean that reading or writing a file in LTSPFS will perform the sequence open() -> lseek() -> read()/write() -> close(). This does have a performance impact, however, since LTSPFS is meant to be used with "slower" media such as floppies, cdroms, and memory sticks, it's not as big a concern. The advantage is that you've got a much better chance of network problems to a terminal not destroying files on removable media than you do with other methods. At some point in the future, stateful read()/write() operations may be added as an option, if it's deemed a desireable feature.
4. The LTSPFS protocol on the LTSP server is implemented with FUSE, or Filesystems in USErspace. You can find FUSE at http://fuse.sourceforge.net. FUSE has now been included in the mainline Linux kernel starting at 2.6.14, and it's an installable package under Debian Sarge and Ubuntu Hoary and is installed by default in Ubuntu Breezy Badger (Thanks to Fabio!).
5. LTSPFS uses the same, simple xauth based authentication scheme that mtools uses. Basically, an LTSPFS mount is authenticated using the MIT-MAGIC-COOKIE that your XWindows session generates. This is a simple authentication scheme, but one that works well in an LTSP environment, as it's difficult to do user based authentication, as the terminal normally has no concept of the user logged into it.

Current Availability:

LTSPFS is in production, and used in Ltsp-4.2. It is not part of the LTSP 4.1.1 distribution. The following instructions are for building LTSPFS from CVS, for people interested in playing with the source.

Requirements to use LTSPFS:

• You must have the FUSE module installed on your server.
• You must have ltspfsd installed on the terminal.
• You must have ltspfs installed on the LTSP server.
• Your DHCP server must make sure to set the LTSP client hostname.
• /tmp on the LTSP terminal must be writeable. (For authentication).
• You must modify your session scripts to mount/unmount the LTSPFS filesystem.

Installing FUSE on your LTSP server:

This is going to vary depending on your server distribution.

Installing FUSE on Debian Sarge:

Sarge uses an older version of FUSE, however, LTSPFS will work just fine with this version. The only issue is that a dead mount won't time out by itself, but rather will timeout and unmount when a filesystem operation is actually tried.

• I'm going to assume you're running 2.6.8-2 on your server. I've done all my testing with this kernel.

aptitude install kernel-image-2.6.8-2-386 kernel-headers-2.6.8-2-386

• Install the FUSE bits:

aptitude install fuse-source fuse-utils libfuse-dev libfuse2

• Install module-assistant, if you haven't already.

aptitude install module-assistant

• As root, build the fuse module

m-a a-i fuse

• Load the FUSE module into the kernel.

modprobe fuse

If this works, you'll probably want to add "fuse" to your /etc/modules file so that it'll be there whenever you reboot the server.

Installing FUSE on Ubuntu:

• The fuse kernel module is already included in the default Breezy kernel (Thanks Fabio!), so nothing needs to be done in terms of getting the kernel module installed. To install the fuse module, you can execute:

sudo modprobe fuse

• To make this a permanent on your machine, add "fuse" to your /etc/modules file.
• Install the fuse packages you'll need:

sudo aptitude install fuse-utils libfuse-dev libfuse2

• Add users you want to be able to use localdevices to the "fuse" group on the server.
• The ltspfsd has begun to be packaged by ogra, in #ltsp. Keep watching this space, as there's likely to be a lot of activity here shortly.

Installing FUSE on Fedora Core:

• If you are using the latest Fedora Core 4 kernel, then you have 2.6.14 (or later), and 2.6.14 and later come with FUSE.
• To install the needed fuse libraries, run this command:

yum install fuse fuse-libs fuse-devel

• To avoid the error fuse: failed to exec fusermount: Permission denied when running ltspfs, add each user to the group fuse on the terminal server. For more information about this requirement (and a method for disabling it), see /usr/share/doc/fuse-2.4.2/README.fedora.

Installing FUSE on Gentoo:

emerge fuse

Installing FUSE from source:

• Check out the instructions available at http://fuse.sourceforge.net. It's usually quite simple. On Debian, if you have the kernel-headers-2.6.8-2-386 package installed, you should simply be able to unpack the tarball, do a ./configure, make, and make install. Note that it will install the fuse libraries to /usr/local/lib, so you'll need to add that to your /etc/ld.so.conf file if you haven't got it there already.

Installing ltspfsd on your terminal:

• ltspfsd, the bit that runs on the thin client, should already be installed as part of LTSP 4.2.
• If not, grab a copy and compile it. Note that you will need the libx11-dev libraries.

cd /usr/local/src
cvs -d :pserver:anonymous@cvs.ltsp.org:/usr/local/cvsroot checkout ltspfsd
cd ltspfsd
./configure
make
cp ltspfsd /opt/ltsp/i386/bin

Installing ltspfs on your LTSP server:

• Grab a copy of ltspfs. This is the bit that runs on the server:

cd /usr/local/src
cvs -d :pserver:anonymous@cvs.ltsp.org:/usr/local/cvsroot checkout ltspfs
cd ltspfs
./configure
make
make install

Adjusting your dhcpd.conf:

If you're using assigned addressing (i.e. you have host {} entries), make sure you're specifying:

use-host-decl-names         on;

in the dhcpd.conf section where you're specifying the hosts.

For dynamic host assignment, you'll want:

get-lease-hostnames         true;

IMPORTANT: The two settings are exclusive, so if you've got use-host-decl-names turned on globally, you're going to have to turn it off in the dynamic section of the dhcpd.conf file.

Starting ltspfsd on the terminal:

LTSPFS is now included as part of Ltsp-4.2. The rc.localdev script in 4.2 starts the ltspfsd server. Make sure you've got the line:

   LOCAL_STORAGE  =  Y

in your lts.conf file.

Quick checklist:

• Boot the terminal.
• Log in as a user on the terminal.
• Press F2, and pop into the shell.
  • Check that /tmp is world writable.
  • Check that ltspfsd is running as root
• Pop back to X on F1
• Open a terminal window
• type "mkdir foo"
• If you're on terminal ws008.ltsp, type:
  • ltspfs ws008.ltsp:/tmp/drives $HOME/foo
• pop in a memory stick, and do an ls -al.

Good luck!

System integration

To integrate LtspFS such that mounting is automatic during login and such that users see disk icons in GNOME, see the page LocalMedia.

Troubleshooting

Error message "fusermount: failed to mountpoint for reading: Stale NFS file handle"

• Error message: fusermount: failed to mountpoint for reading: Stale NFS file handle
  • Reason: The mount point is on a NFS file system with root squash enabled.
  • Solution: Change the mount point or disable root squashing.

Technical Specification:

Command line options:

The ltspfsd server has the following command line options:

• -a for no authentication. The server will simply accept the XAUTH packet sent down, and always return OK.
• -r for read only. Specifying this causes the ltspfsd to behave as a readonly server, and won't allow any modification to the media.
• -d for debug. Server won't fork into the background, and will print messages to stderr.

The ltspfs command has the following format:

ltspfs hostname:/path/to/remote/dir /path/to/local/dir

So, as an example:

ltspfs ws185.ltsp:/tmp/drives $HOME/drives

Network port:

The ltspfs filesystem communicates on port 9220, which is unpriviledged.

LTSP Design Document

The Spec:

The following is a thumbnail view of the functional spec. The on-the-wire protocol will be XDR based, except for the actual reading and writing of the datablocks, which will be passed "as is" in binary.

Packets have the following format:

• A command packet, consisting of:
  • A 4 byte xdr_int of the size of the packet, in bytes. THIS SIZE INCLUDES THE PACKET COUNT BYTES!!!
  • A 4 byte xdr_int of the type of packet this is.
  • An optional number of xdr_* args, depending on the type of the command packet.
• An optional data packet, for the read/write commands, consisting of the data to be read/written. The data packet, if included, appears immediately after the command packet.

Currently, ltspfsd is running as root, as it is now doing the mounting and unmounting of media on the local terminal.

Packet Types:

The packet type is the xdr_int immediately following the length packet. The values are:

LTSPFS_GETATTR	0
LTSPFS_READLINK	1
LTSPFS_READDIR	2
LTSPFS_MKNOD	3
LTSPFS_MKDIR	4
LTSPFS_SYMLINK	5
LTSPFS_UNLINK	6
LTSPFS_RMDIR	7
LTSPFS_RENAME	8
LTSPFS_LINK	9
LTSPFS_CHMOD	10
LTSPFS_CHOWN	11
LTSPFS_TRUNCATE	12
LTSPFS_UTIME	13
LTSPFS_OPEN	14
LTSPFS_READ	15
LTSPFS_WRITE	16
LTSPFS_STATFS	17
LTSPFS_RELEASE	18	Not currently used
LTSPFS_RSYNC	19	Not currently used
LTSPFS_SETXATTR	20	Not currently used
LTSPFS_GETXATTR	21	Not currently used
LTSPFS_LISTXATTR	22	Not currently used
LTSPFS_REMOVEXATTR	23	Not currently used
LTSPFS_XAUTH	24
LTSPFS_MOUNT	25
LTSPFS_PING	26
LTSPFS_QUIT	27

Initial Commands:

These commands must be run before any subsequent commands. The commands must appear in the order of XAUTH, then MOUNT.

XAUTH

• First command sent to the daemon.
• Send the data packet generated by: xauth extract - $DISPLAY
• ltspfsd tries to do an XOpenDisplay using the cookie. If successful, you're authenticated.
• Whether successful or not, delete the cookie passed.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:XAUTH][xdr_int:size]==\n[size bytes of X cookie]
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

MOUNT

• Second command sent to the daemon.
• Mount command causes the daemon to set up the needed path translation.
• daemon protocol:
  • query: [xdr_int:MOUNT][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

Automounting

• ltspfsd handles mounting and unmounting the local usb stick, cdrom, whatever. This removes the need to have a separate kernel automounter, or supermount, or other methods. It also provides a much closer coupling between ltspfs data operations, and the media itself. You can this documented at LtspFSAutomount.

The FUSE API

Stateless functions required by fuse:

ltspfs_getattr:

• static int ltspfs_getattr(const char *path, struct stat *stbuf)
• returns 0 on success, -errno on error.
• Gets file attributes.
• Populate the stat buf based on attributes of file
• daemon protocol:
  • query: [xdr_int:length][xdr_int:GETATTR][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0][xdr_u_longlong_t:st_dev][xdr_u_longlong_t:st_ino][xdr_u_int:st_mode] [xdr_u_int:st_nlink][xdr_u_int:st_uid][xdr_u_int:st_gid][xdr_u_longlong_t:st_rdev] [xdr_longlong_t:st_size][xdr_long:st_blksize][xdr_longlong_t:st_blocks][xdr_long:st_atime] [xdr_long:st_mtime][xdr_long:st_ctime]
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_readlink:

• static int ltspfs_readlink(const char *path, char *buf, size_t size)
• returns 0 on success, -errno on error.
• use readlink function to read the destinantion of the link, and place the result in *buf, with size (size - 1)
• daemon protocol:
  • query: [xdr_int:length][xdr_int:READLINK][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0][xdr_string:resultpath] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_readdir:

• static int ltspfs_readdir(const char *path, void *buf, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi)
• returns directory listing. Needs some inode information from the dirent structure. Needed fields: d_ino, d_type, d_name.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:READDIR][xdr_string:path]
  • response: [xdr_int:length][xdr_int:2][xdr_u_longlong_t:d_ino][xdr_u_char:d_type][xdr_string:d_name]
  • Continued responses until the last response: [xdr_int:length][xdr_int:0] on last line if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_mknod:

• static int ltspfs_mknod(const char *path, mode_t mode, dev_t rdev)
• create a device node using the mknod call.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:MKNOD][xdr_u_int:mode][xdr_u_longlong_t:rdev][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_mkdir:

• static int ltspfs_mkdir(const char *path, mode_t mode)
• Create a directory.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:MKDIR][xdr_u_int:mode][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_symlink:

• static int ltspfs_symlink(const char *from, const char *to)
• create a symlink using symlink function call.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:SYMLINK][xdr_string:from][xdr_string:to]
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_unlink:

• static int ltspfs_unlink(const char *path)
• Remove a file.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:UNLINK][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_rmdir:

• static int ltspfs_rmdir(const char *path)
• Remove a directory.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:RMDIR|pathlen|path
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_rename:

• static int ltspfs_rename(const char *from, const char *to)
• rename a file.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:RENAME][xdr_string:from][xdr_string:to]
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_link:

• static int ltspfs_link(const char *from, const char *to)
• create a hard link
• daemon protocol:
  • query: [xdr_int:length][xdr_int:LINK][xdr_string:from][xdr_string:to]
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_chmod:

• static int ltspfs_chmod(const char *path, mode_t mode)
• chmod a file.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:CHMOD][xdr_u_int:mode][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_chown:

• static int ltspfs_chown(const char *path, uid_t uid, gid_t gid)
• chown a file
• daemon protocol:
  • query: [xdr_int:length][xdr_int:CHMOD][xdr_u_int:uid][xdr_u_int:gid][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_truncate:

• static int ltspfs_truncate(const char *path, off_t size)
• truncate a file to a given size using the truncate system call.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:TRUNCATE][xdr_longlong_t:offset][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_utime:

• static int ltspfs_utime(const char *path, struct utimbuf *buf)
• Change access times on a file. Need access time and mod time.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:UTIME][xdr_long:actime][xdr_long:modtime][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_statfs:

• static int ltspfs_statfs(const char *path, struct statfs *stbuf)
• Populates the statfs buffer. See statfs function call for details.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:STATFS][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0][xdr_int:f_type][xdr_int:f_bsize][xdr_u_longlong_t:f_blocks] [xdr_u_longlong_t:f_bfree][xdr_u_longlong_t:f_bavail][xdr_u_longlong_t:f_files] [xdr_u_longlong_t:f_ffree][xdr_u_longlong_t:f_fsid][xdr_int:f_namelen] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

Stateful function calls:

ltspfs_open:

• static int ltspfs_open(const char *path, struct fuse_file_info *fi)
• Stateless. Just opens the file to see if it can be opened. Then closes it immediately. No state is needed to be maintained int the protocol.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:OPEN|flags|pathlen|path
  • response: [xdr_int:length][xdr_int:0] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_read:

• static int ltspfs_read(const char *path, char *buf, size_t size, off_t offset, struct fuse_file_info *fi)
• Stateless read from a file. Opens file, reads size bytes at position offset, into buffer buf, from file at path. Then closes the file.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:READ][xdr_u_int:size][xdr_longlong_t:offset][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0][xdr_int:size] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_write:

• static int ltspfs_write(const char *path, char *buf, size_t size, off_t offset, struct fuse_file_info *fi)
• Stateless write to a file. Opens file, writes size bytes at position offset, from buffer buf, to file at path. Then closes the file.
• daemon protocol:
  • query: [xdr_int:length][xdr_int:WRITE][xdr_u_int:size][xdr_longlong_t:offset][xdr_string:path]
  • response: [xdr_int:length][xdr_int:0][xdr_int:size] if ok.
  • response: [xdr_int:length][xdr_int:1][xdr_int:errno] if error. errno = EACCES.

ltspfs_release:

• static int ltspfs_release(const char *path, struct fuse_file_info *fi)
• Since our reads and writes are stateless, this call does nothing.

ltspfs_rsync:

• static int ltspfs_rsync(const char *path, int isdatasync, struct fuse_file_info *fi)
• Since our reads and writes are stateless, this call does nothing.

-- ScottBalneaves - 19 May 2006