[AUUG-Talk]: XFS Experiences?

[Nathan Robertson]

Hi David,

On 04/01/2005, at 2:26 AM, David J N Begley wrote:

> Collective brains trust of AUUG - does anyone have any experience with 
> SGI's
> XFS (file system) on either Linux or IRIX boxen?  Whilst I am more 
> interested
> in negative (or even just mildly annoying) experiences, if anyone feels
> sufficiently motivated to sing XFS' praises I wouldn't mind hearing 
> those too.

Ok. We've been running XFS in production on an internal IMAP mail 
server for about 2-3 years now (since SGI shipped the XFS patches, well 
before it was in the mainline kernel). At the time the decision was 
made because:

1. XFS performance was better than any other filesystem in almost every 
test.
2. JFS (IBM) corrupts itself quite regularly (I retested this about 12 
months ago and was suprised to see it hadn't improved).
3. Support for ACLs (important when we ran it on a Samba server, and at 
the time no other filesystem supported ACLs - now others do)
4. Stability. We've never had a problem.

(BTW - we've had other servers running XFS, but the IMAP server seems 
to map closest to your scenario).

> I am in the process of preparing an I/O performance simulation in 
> order to
> pitch ext3fs, ReiserFS and XFS against one another for "real world" 
> (cough,
> splutter, choke) Squid access patterns;  on the thorny issue of 
> "reliability",
> however, I am left (at present) only examining documented behaviours 
> and
> design priorities for each file system.

Are you talking about reiserfs v3 or v4? v4 I would believe might come 
somewhere close to XFS (its only been out for a handful of months), but 
v3 I'd be very suprised. v4 is somewhat of a political football in the 
Linux community - I won't rehash it here, but it'll be a long road into 
mainstream kernel acceptance for v4, and may have a fair few features 
stripped before Linus will accept it. On the other hand, XFS is in both 
2.4 and 2.6 kernels.

Which distribution are you running? It can be slightly painful to 
install using XFS because the Red Hat (either Fedora or RHEL) installer 
doesn't have XFS on its GUI partitioner, you need third party CDs for 
Debian, SuSE Linux Enterprise Server supports it, but a bug in their 
kernel they shipped with SLES9 means that has a kernel "oops" as soon 
as you try to mount an XFS partition.

> Before anyone asks, RAID mirroring (or similar) approaches are not 
> applicable
> (only RAID0/striping).  Whilst the hardware will technically be on a 
> UPS,
> I have to assume that the Squid server will not have any knowledge of 
> any
> impending loss of power (internal politics - long boring story).

Ok, in other news, if you are running a softraid setup (I realise you 
aren't, but I want to get this on the record for others that read this, 
particularly in mailing list archives), you need to have your /boot as 
ext2 or ext3. Then your root mount and every other mount can be 
whatever you like (in our case XFS).

If power outages are a possibility, then I'd recommend you use a 
filesystem that has been around for a while, and fairly proven (ie. not 
reiser4). XFS did the job for us, as we had to go through this a few 
years ago before we started buying UPSs. But that was a few years ago, 
and I'm unsure how hard the server was being hit at the time.

> Of course, search the Web and everyone's got a "nightmare" story 
> regarding any
> of these three file systems (though fewer for ext3fs, admittedly);  
> however,
> the "ordered journalling" option of ext3fs and newer ReiserFS 
> implementations
> reduces the window (between data and metadata updates) for failures to 
> cause
> havoc.  Alas, XFS' behaviour is more akin to ext3fs' "writeback 
> journalling"
> option.

I think you are talking reiser4, which as I say, is likely to be 
outside the mainstream kernel for a while. Read the stories on reiser4 
inclusion from a few months back on http://www.kerneltrap.org/ (easier 
than reading LKML). If you're talking reiser3, then I'd be particularly 
interested in the results of your benchmarking.

> So... has anyone used XFS (IRIX or Linux since this appears to be an 
> aspect
> of XFS' design that is not specific to Linux) on a "busy" (I/O) system 
> that
> has accidentally lost power?  If so, what are your experiences with 
> data loss
> (either individual files, parts of files or entire file systems)?

Hmm. Well, we've never lost data, but I haven't run an XFS system 
without a UPS for a couple of years now. But from a couple of years 
ago, we didn't lose data. Couldn't tell you what I/O load the machine 
was under at the instant the power went off two years ago either. ;-)

What I can tell you is that our XFS IMAP server is used by 30+ people 
all day. IMAP is a fairly chatty protocol that is annoyingly expensive 
on I/O (for a reasonably simple job - browsing email), and might be 
comparable on I/O to your squid server. A fast reliable filesystem 
(XFS) combined with U160 SCSI disks (softraid1) and an IMAP server 
supporting Maildirs (courier-imapd) was the answer, and is humming 
along quite well. FWIW, I've had both Debian and Red Hat based servers 
running XFS, but said IMAP server is Debian.

Hope this helps.

Regards,
Nathan.