Re: [Gluster-devel] GlusterFS 3.3.0 and NFS client failures

[Ian Latter]

Interestingly, I never considered this as an "NFS" problem.

  1) Eg; http://www.novell.com/support/kb/doc.php?id=7008148
      (note that none of my machines are SLES).

  2) One NFS client in my environment has never hung on the 
      Gluster/NFS share, and it only does play-back (read).


This has opened up some possibilities.  Thanks for the idea bouncing.



----- Original Message -----
>From: "Ian Latter" <address@hidden>
>To: "Anand Avati" <address@hidden>
>Subject:  Re: [Gluster-devel] GlusterFS 3.3.0 and NFS client failures
>Date: Sun, 24 Mar 2013 11:35:52 +1000
>
> 
> Thanks for the extra queries;
> 
> > 1) when the system was "hung", was the client still flushing data to the
> > nfs server? 
> 
> I have no more detail outside of what has been supplied.  I.e. the other two 
> applications I've seen fail have been starting a guest from a Gluster NFS 
> share from VMWare Player (2nd most common, has happened four or five 
> times) and "cp" from bash (3rd most common, has happened twice) - each 
> on different hosts (NFS clients).
> 
> > Any network activity? 
> 
> On the myth backend server - No NFS network activity occurs after the
> event, but I can still SSH into it (and everything is fine in that session, 
> you
> just can't perform any NFS iops or you'll hang that session on the blocked
> io).
> 
> On the Gluster NFS server (Saturn) there's no issues at all - other 
> devices on the network continue to access NFS shares from that box.
> And no action is required on the Gluster NFS server for the failed host 
> to reconnect and operate normally again.
> 
> > The backtrace below shows that the system was just waiting for a long
> > time waiting for a write to complete.
> 
> Note that this state never recovers.  Sometimes its days before I find that 
> the backend server has hung (as in, it has been in a hung state for days - 
> like when it failed the day after I left for an overseas trip).  It doesn't 
> reset, 
> all future IOPS on that share hang.
> 
> So one diagnostic option is for me to write up a script that just creates and 
> destroys a shed-load of data on a Gluster NFS share with an strace 
> running over it .. but your comment suggests that if a "cp" hangs on write 
> with output like the below then you'll be unimpressed.  What about a 
> network trace instead?
> 
> 
> > 2) anything in the gluster nfs logs?
> 
> There are no other logs on Saturn, other than syslog and the console 
> (dmesg), and I'm not seeing any Gluster or NFS entries there.  Is there 
> a way to get Gluster to log to syslog rather than on-disk files?  I only 
> have a couple of Megabytes of disk space available.
> 
> 
> > 3) is it possible DHCP assigned a different IP while renewing lease?
> 
> No.  As a backend service it is issued a static DHCP assignment based 
> on its MAC address, and in the final case (13 days uptime) I removed 
> DHCP requesting completely by hard coding the IPv4 address in 
> Ubuntu's /etc/network* scripts on the Myth Backend host.  I.e. it fails 
> without a DHCP process involved.
> 
> 
> The feeling that I get is that this is a rare issue - in which case I'm 
> looking
> for something domestic - like a hardware fault or something bespoke in 
> my configuration.  The problem reminds of the error we found in one of 
> the Gluster modules that would kill replication for some files > 2GB but 
> not all, so one of my other thoughts was that it may be something related 
> to this (Saturn) being a 32bit platform.  However I have another site 
> running Saturn that doesn't have this problem, so I'm reluctant to believe
> that its the kernel-to-gluster stack either.  But seeing the failures in three
> NFS clients makes it look like its Gluster/Saturn side issue.
> 
> Let me try a packet capture and we'll see if there's anything odd in there.
> 
> The next failure is due in a week or so.  If I get a chance I'll also write up
> a test script to see if I can force a failure (it might also give a view on 
> the 
> volume of data required to trigger an event, and if it works it will give me
> the ability to separate testing from my production kit).
> 
> 
> 
> Thanks,
> 
> 
> 
> 
> ----- Original Message -----
> >From: "Anand Avati" <address@hidden>
> >To: "Ian Latter" <address@hidden>
> >Subject:  Re: [Gluster-devel] GlusterFS 3.3.0 and NFS client failures
> >Date: Sat, 23 Mar 2013 16:40:38 -0700
> >
> > Do you have any more details, like -
> > 
> > 1) when the system was "hung", was the client still flushing data to the
> > nfs server? Any network activity? The backtrace below shows that the system
> > was just waiting for a long time waiting for a write to complete.
> > 
> > 2) anything in the gluster nfs logs?
> > 
> > 3) is it possible DHCP assigned a different IP while renewing lease?
> > 
> > Avati
> > 
> > On Fri, Mar 22, 2013 at 3:04 AM, Ian Latter <address@hidden>wrote:
> > 
> > > Hello,
> > >
> > >
> > >   This is a problem that I've been chipping at on and off for a while and
> > > its finally cost me one recording too many - I just want to get it cured -
> > > any
> > > help would be greatly appreciated.
> > >
> > >   I'm using the kernel NFS client on a number of Linux machines (four, I
> > > believe), to map back to two Gluster 3.3.0 shares.
> > >
> > >   I have seen Linux Mint and Ubuntu machines of various generations and
> > > configurations (one is 64bit) hang intermittently on either one of the two
> > > Gluster shares on "access" (I can't say if its writing or not - the below
> > > log
> > > is for a write).  But by far the most common failure example is my MythTV
> > > Backend server.  It has 5 tuners pulling down up to a gigabyte per hour
> > > each directly to an NFS share from Gluster 3.3.30 with two local 3TB
> > > drives in a "distribute" volume.  It also re-parses each recording for Ad
> > > filtering, so the share gets a good thrashing.  The myth backend box would
> > > fail (hang the system) once each 2-4 days.
> > >
> > > The backend server was also updating its NIC via DHCP.  I have been using
> > > an MTU of 1460 and each DHCP event would thus result in this note in 
> > > syslog;
> > >  [  12.248640] r8169: WARNING! Changing of MTU on this NIC may lead to
> > > frame reception errors!
> > >
> > > I change the DHCP MTU to 1500 and didn't see an improvement.  So, the
> > > last change I made was a hard coded address and default MTU (of 1500).
> > > The most recent trial saw a 13 day run time which is well outside the 
> > > norm,
> > > but it still borked (one test only - may have been lucky).
> > >
> > > >> syslog burp;
> > > [1204800.908075] INFO: task mythbackend:21353 blocked for more than 120
> > > seconds.
> > > [1204800.908084] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > > disables this message.
> > > [1204800.908091] mythbackend   D f6af9d28     0 21353      1 0x00000000
> > > [1204800.908107]  f6af9d38 00000086 00000002 f6af9d28 f6a4e580 c05d89e0
> > > c08c3700 c08c3700
> > > [1204800.908123]  bd3a4320 0004479c c08c3700 c08c3700 bd3a0e4e 0004479c
> > > 00000000 c08c3700
> > > [1204800.908138]  c08c3700 f6a4e580 00000001 f6a4e580 c2488700 f6af9d80
> > > f6af9d48 c05c6e51
> > > [1204800.908152] Call Trace:
> > > [1204800.908170]  [<c05c6e51>] io_schedule+0x61/0xa0
> > > [1204800.908180]  [<c01d9c4d>] sync_page+0x3d/0x50
> > > [1204800.908190]  [<c05c761d>] __wait_on_bit+0x4d/0x70
> > > [1204800.908197]  [<c01d9c10>] ? sync_page+0x0/0x50
> > > [1204800.908211]  [<c01d9e71>] wait_on_page_bit+0x91/0xa0
> > > [1204800.908221]  [<c0165e60>] ? wake_bit_function+0x0/0x50
> > > [1204800.908229]  [<c01da1f4>] filemap_fdatawait_range+0xd4/0x150
> > > [1204800.908239]  [<c01da3c7>] filemap_write_and_wait_range+0x77/0x80
> > > [1204800.908248]  [<c023aad4>] vfs_fsync_range+0x54/0x80
> > > [1204800.908257]  [<c023ab5e>] generic_write_sync+0x5e/0x80
> > > [1204800.908265]  [<c01dbda1>] generic_file_aio_write+0xa1/0xc0
> > > [1204800.908292]  [<fb0bc94f>] nfs_file_write+0x9f/0x200 [nfs]
> > > [1204800.908303]  [<c0218454>] do_sync_write+0xa4/0xe0
> > > [1204800.908314]  [<c032e626>] ? apparmor_file_permission+0x16/0x20
> > > [1204800.908324]  [<c0302a74>] ? security_file_permission+0x14/0x20
> > > [1204800.908333]  [<c02185d2>] ? rw_verify_area+0x62/0xd0
> > > [1204800.908342]  [<c02186e2>] vfs_write+0xa2/0x190
> > > [1204800.908350]  [<c02183b0>] ? do_sync_write+0x0/0xe0
> > > [1204800.908359]  [<c0218fa2>] sys_write+0x42/0x70
> > > [1204800.908367]  [<c05c90a4>] syscall_call+0x7/0xb
> > >
> > > This might suggest a hardware fault on the Myth Backend host (like the
> > > NIC) but I don't believe that to be the case because I've seen the same
> > > issue on other clients.  I suspect that they are much more rare because
> > > the data volume on those clients pales in comparison to the Myth Backend
> > > process (virtual guests, etc - light work - months between failures,
> > > doesn't
> > > feel time related).
> > >
> > > The only cure is a hard reset (of the host with the NFS client) as any FS
> > > operation on that share hangs - including df, ls, sync and umount - so the
> > > system fails to shutdown.
> > >
> > > The kernel on the Myth Backend host isn't new ..
> > >
> > > >> uname -a;
> > > Linux jupiter 2.6.35-22-generic #33-Ubuntu SMP Sun Sep 19 20:34:50 UTC
> > > 2010 i686 GNU/Linux
> > >
> > > Is there a known good/bad version for the kernel/NFS client?  Am I under
> > > that bar?
> > >
> > >
> > > The GlusterFS NFS server an embedded platform (Saturn) that has been
> > > running for 74 days;
> > >
> > > >> uptime output;
> > > 08:39:07 up 74 days, 22:16,  load average: 0.87, 0.94, 0.94
> > >
> > > It is a much more modern platform;
> > >
> > > >> uname -a;
> > > Linux (none) 3.2.14 #1 SMP Tue Apr 10 12:46:47 EST 2012 i686 GNU/Linux
> > >
> > > It has had one error in all of that time;
> > > >> dmesg output;
> > > Pid: 4845, comm: glusterfsd Not tainted 3.2.14 #1
> > > Call Trace:
> > >  [<c10512d0>] __rcu_pending+0x64/0x294
> > >  [<c1051640>] rcu_check_callbacks+0x87/0x98
> > >  [<c1034521>] update_process_times+0x2d/0x58
> > >  [<c1047bdf>] tick_periodic+0x63/0x65
> > >  [<c1047c2d>] tick_handle_periodic+0x17/0x5e
> > >  [<c1015ae9>] smp_apic_timer_interrupt+0x67/0x7a
> > >  [<c1b2a691>] apic_timer_interrupt+0x31/0x40
> > >
> > > .. this occurred months ago.
> > >
> > > Unfortunately due to its embedded nature, there are no logs coming from
> > > this platform, only a looped buffer for syslog (and gluster doesn't seem 
> > > to
> > > syslog).  In previous discussions here (months ago) you'll see where I was
> > > working to disable/remove logging from GlusterFS so that I could keep it
> > > alive in an embedded environment - this is the current run configuration.
> > >
> > > The Myth Backend host only mounts one of the two NFS shares, but I've seen
> > > the fault on the hosts that only mount the other - so I'm reluctant to
> > > believe that its a hardware failure at the Drive level on the Saturn /
> > > Gluster
> > > server.
> > >
> > > The /etc/fstab entry for this share, on the Myth Backend host, is;
> > >
> > >   saturn:/recordings /var/lib/mythtv/saturn_recordings nfs
> > > nfsvers=3,rw,rsize=8192,wsize=8192,hard,intr,sync,dirsync,noac,noatime,nodev,nosuid
> > > 0  0
> > >
> > > When I softened this to async with soft failures (a config taken straight
> > > from the Gluster site/FAQ) it crashed out in a much shorter time-frame
> > > (less than a day, one test only - may have been unlucky);
> > >
> > >   saturn:/recordings /var/lib/mythtv/saturn_recordings nfs
> > > defaults,_netdev,nfsvers=3,proto=tcp 0  0
> > >
> > >
> > > Other than the high use Myth Backend host I've failed to accurately nail
> > > down the trigger for this issue - which is making diagnostics painful (I
> > > like
> > > my TV too much to do more than reboot the failed box - and heaven forbid
> > > the dad that fails to record Pepper Pig!).
> > >
> > >
> > > Any thoughts?  Beyond enabling logs on the Saturn side ...
> > >
> > > Is it possible this is a bug that was reaped in later versions of Gluster?
> > >
> > > Appreciate being set straight ..
> > >
> > >
> > >
> > >
> > >
> > > Cheers,
> > >
> > >
> > >
> > >
> > > --
> > > Ian Latter
> > > Late night coder ..
> > > http://midnightcode.org/
> > >
> > > _______________________________________________
> > > Gluster-devel mailing list
> > > address@hidden
> > > https://lists.nongnu.org/mailman/listinfo/gluster-devel
> > >
> > 
> 
> 
> --
> Ian Latter
> Late night coder ..
> http://midnightcode.org/
> 
> _______________________________________________
> Gluster-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/gluster-devel
> 


--
Ian Latter
Late night coder ..
http://midnightcode.org/