Re: [Gluster-devel] nodes don't use swap

[Jordi Moles]

hi,

did you guys manage to reproduce the issues i'm facing?

do you need any extra information?

thanks.

En/na Jordi Moles ha escrit:
> Hi,
>
> i've just read through the whole log files from the last few days, 
> when i started these massive tests.
>
> just to let you know in case i haven't done so before.... i'm running 
> two types of tests.
>
> 1. sending to the mail system several emails per second, postfixs just 
> keep delivering mails every second. A part from that, a script checks 
> several mailboxes via imap and pop3 every minute.
> 2. running some disk tests, like bonnie.
>
> anyway... a part from that issue with only 1 cpu used and also running 
> out of ram memory... i was wondering if anything would be logged. And 
> actually it is.
> The thing is that nodes haven't reported any error or warning at all 
> in days, nor have dovecots.
>
> But postfixs keep logging this from time to time. It doesn't happen 
> all the time, but when it does, it keeps logging this for several 
> minutes, every second.
>
> *************
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785301: 
> /dummymassiu/new/1205166062.V12I514f6M451321.postfix01gluster01.cdmon.com,S=1322 
> => 333046 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785307: 
> /dummymassiu/new/1205169576.V12I528b6M828341.postfix01gluster01.cdmon.com,S=2113 
> => 338102 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785308: 
> /dummymassiu/new/1205165435.V12I5115eM368089.postfix01gluster01.cdmon.com,S=1322 
> => 332126 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785310: 
> /dummymassiu/new/1205163678.V12I50696M779037.postfix01gluster01.cdmon.com,S=2110 
> => 329366 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785311: 
> /dummymassiu/new/1205166503.V12I517e8M525608.postfix02gluster01.cdmon.com,S=1321 
> => 333800 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785312: 
> /dummymassiu/new/1205168060.V12I521beM376767.postfix01gluster01.cdmon.com,S=1323 
> => 336318 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785313: 
> /dummymassiu/new/1205172223.V12I52adaM551011.postfix02gluster01.cdmon.com,S=1321 
> => 338650 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785316: 
> /dummymassiu/new/1205163863.V12I507c2M877288.postfix01gluster01.cdmon.com,S=2113 
> => 329666 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785320: 
> /dummymassiu/new/1205163951.V12I50862M599096.postfix01gluster01.cdmon.com,S=1323 
> => 329826 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785322: 
> /dummymassiu/new/1205169551.V12I52898M543504.postfix03gluster01.cdmon.com,S=2110 
> => 338072 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785324: 
> /dummymassiu/new/1205167646.V12I51f74M976792.postfix03gluster01.cdmon.com,S=2110 
> => 335732 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785326: 
> /dummymassiu/new/1205167981.V12I52149M809051.postfix04gluster01.cdmon.com,S=1323 
> => 336201 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785330: 
> /dummymassiu/new/1205166685.V12I51916M157626.postfix01gluster01.cdmon.com,S=2112 
> => 334102 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785332: 
> /dummymassiu/new/1205164950.V12I50e5eM93799.postfix01gluster01.cdmon.com,S=2112 
> => 331358 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785333: 
> /dummymassiu/new/1205164539.V12I50c5aM252840.postfix01gluster01.cdmon.com,S=2112 
> => 330842 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785336: 
> /dummymassiu/new/1205166068.V12I51505M733823.postfix04gluster01.cdmon.com,S=1323 
> => 333061 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785339: 
> /dummymassiu/new/1205164143.V12I5098dM237821.postfix04gluster01.cdmon.com,S=2113 
> => 330125 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785340: 
> /dummymassiu/new/1205169554.V12I5289bM892195.postfix02gluster01.cdmon.com,S=2110 
> => 338075 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785341: 
> /dummymassiu/new/1205169041.V12I52639M355241.postfix04gluster01.cdmon.com,S=1322 
> => 337465 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785344: 
> /dummymassiu/new/1205168969.V12I525f9M212765.postfix01gluster01.cdmon.com,S=2112 
> => 337401 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785345: 
> /dummymassiu/new/1205165036.V12I50f0bM828482.postfix03gluster01.cdmon.com,S=1321 
> => 331531 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785347: 
> /dummymassiu/new/1205167618.V12I51f40M129843.postfix03gluster01.cdmon.com,S=2110 
> => 335680 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785348: 
> /dummymassiu/new/1205165356.V12I510f0M733837.postfix02gluster01.cdmon.com,S=2111 
> => 332016 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785349: 
> /dummymassiu/new/1205166260.V12I51656M140344.postfix01gluster01.cdmon.com,S=1323 
> => 333398 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785350: 
> /dummymassiu/new/1205166740.V12I5197bM561623.postfix01gluster01.cdmon.com,S=1322 
> => 334203 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785353: 
> /dummymassiu/new/1205165612.V12I51257M218424.postfix02gluster01.cdmon.com,S=1321 
> => 332375 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785359: 
> /dummymassiu/new/1205166335.V12I516d7M906967.postfix03gluster01.cdmon.com,S=2110 
> => 333527 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785360: 
> /dummymassiu/new/1205166706.V12I5193bM376041.postfix02gluster01.cdmon.com,S=2111 
> => 334139 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785362: 
> /dummymassiu/new/1205167774.V12I52045M925044.postfix04gluster01.cdmon.com,S=1322 
> => 335941 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785363: 
> /dummymassiu/new/1205163717.V12I506d3M585524.postfix01gluster01.cdmon.com,S=2111 
> => 329427 Rehashing because st_nlink less than dentry maps
> 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] 
> glusterfs-fuse: 2785366: 
> /dummymassiu/new/1205166212.V12I5160dM479271.postfix04gluster01.cdmon.com,S=1322 
> => 333325 Rehashing because st_nlink less than dentry maps
> *************
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> En/na Jordi Moles ha escrit:
> > Hi,
> >
> > yes, sorry, that is a typo, i mean:
> >
> > dom0 is NOT running any server or client daemon.
> >
> > En/na Basavanagowda Kanur ha escrit:
> > > Jordi,
> > >
> > > I have one more question, inline with your the message below.
> > >
> > > On Wed, Mar 12, 2008 at 10:25 PM, Jordi Moles <address@hidden> wrote:
> > >
> > >  
> > > > Hi,
> > > >
> > > > I'm using Xen 3.2.0 on debian lenny machines, xen installed from 
> > > > apt-get
> > > > repositories.
> > > >
> > > > All machines, nodes, postfixs and dovecots are virtuals.
> > > >
> > > > nodes have 2GB of ram and 4GB of swap, dovecots and postfixs 512M 
> > > > of ram
> > > > and also 4GB of swap.
> > > >
> > > > dom0 is running any server or client gluster daemon.
> > > >     
> > >
> > >
> > > Did you mean 'dom0 is not running any server or client glusterfs 
> > > daemon'?
> > >
> > >
> > >  
> > > > I've attached a picture from cacti, which i use to monitor all 
> > > > parts of
> > > > gluster. There are two points at which  the node runs out of ram 
> > > > memory.
> > > > The first one is when i perform a postmark test, and the second one 
> > > > is a
> > > > bonnie test. The other readings which are close to 1GB refers to
> > > > "normal" use of my mail system, receiving mails, people checking their
> > > > mailboxes and so on.
> > > >
> > > > no errors or warnings are reported in the gluster log files.
> > > >
> > > > here's the xen config files from any kind of machine:
> > > >
> > > > NODES
> > > >
> > > > *************
> > > >
> > > > #
> > > > # Configuration file for the Xen instance node01gluster01, created
> > > > # by xen-tools 3.9 on Thu Feb 28 16:39:27 2008.
> > > > #
> > > >
> > > > #
> > > > #  Kernel + memory size
> > > > #
> > > > kernel      = '/boot/vmlinuz-2.6.18-6-xen-amd64'
> > > > ramdisk     = '/boot/initrd.img-2.6.18-6-xen-amd64'
> > > > memory      = '2048'
> > > > vcpus       = '2'
> > > >
> > > > #
> > > > #  Disk device(s).
> > > > #
> > > > root        = '/dev/sda2 ro'
> > > > disk        = [
> > > >                  
> > > > 'file:/home/xen/domains/node01gluster01/swap.img,sda1,w',
> > > >                  
> > > > 'file:/home/xen/domains/node01gluster01/disk.img,sda2,w',
> > > >              ]
> > > >
> > > >
> > > > #
> > > > #  Hostname
> > > > #
> > > > name        = 'node01gluster01'
> > > >
> > > > #
> > > > #  Networking
> > > > #
> > > > vif         = [
> > > > 'bridge=xenbr0,ip=192.168.1.204,mac=00:16:3E:B3:2F:18','bridge=xenbr1,ip= 
> > > >
> > > > 192.168.2.204,mac=00:16:3E:B3:2F:19'
> > > > ]
> > > >
> > > > #
> > > > #  Behaviour
> > > > #
> > > > on_poweroff = 'destroy'
> > > > on_reboot   = 'restart'
> > > > on_crash    = 'restart'
> > > >
> > > >
> > > > *************
> > > >
> > > > DOVECOTS
> > > >
> > > > *************
> > > >
> > > >
> > > > # Configuration file for the Xen instance 
> > > > dovecot02gluster01.cdmon.com,
> > > > created
> > > > # by xen-tools 3.9 on Thu Feb 28 16:51:32 2008.
> > > > #
> > > >
> > > > #
> > > > #  Kernel + memory size
> > > > #
> > > > kernel      = '/boot/vmlinuz-2.6.18-6-xen-amd64'
> > > > ramdisk     = '/boot/initrd.img-2.6.18-6-xen-amd64'
> > > > memory      = '512'
> > > >
> > > > #
> > > > #  Disk device(s).
> > > > #
> > > > root        = '/dev/sda2 ro'
> > > > disk        = [
> > > >
> > > > 'file:/home/xen/domains/dovecot02gluster01.cdmon.com/swap.img,sda1,w',
> > > >
> > > > 'file:/home/xen/domains/dovecot02gluster01.cdmon.com/disk.img,sda2,w',
> > > >              ]
> > > >
> > > >
> > > > #
> > > > #  Hostname
> > > > #
> > > > name        = 'dovecot02gluster01.cdmon.com'
> > > >
> > > > #
> > > > #  Networking
> > > > #
> > > > vif         = [
> > > > 'bridge=xenbr0,ip=212.x.x.x,mac=00:16:3E:B2:E7:69','bridge=xenbr1,ip=
> > > > 192.168.1.36,mac=00:16:3E:B2:E7:70'
> > > > ]
> > > >
> > > > #
> > > > #  Behaviour
> > > > #
> > > > on_poweroff = 'destroy'
> > > > on_reboot   = 'restart'
> > > > on_crash    = 'restart'
> > > >
> > > >
> > > > **************
> > > >
> > > > POSTFIXS
> > > >
> > > > **************
> > > >
> > > > #
> > > > # Configuration file for the Xen instance 
> > > > postfix02gluster01.cdmon.com,
> > > > created
> > > > # by xen-tools 3.9 on Thu Feb 28 16:40:20 2008.
> > > > #
> > > >
> > > > #
> > > > #  Kernel + memory size
> > > > #
> > > > kernel      = '/boot/vmlinuz-2.6.18-6-xen-amd64'
> > > > ramdisk     = '/boot/initrd.img-2.6.18-6-xen-amd64'
> > > > memory      = '512'
> > > >
> > > > #
> > > > #  Disk device(s).
> > > > #
> > > > root        = '/dev/sda2 ro'
> > > > disk        = [
> > > >
> > > > 'file:/home/xen/domains/postfix02gluster01.cdmon.com/swap.img,sda1,w',
> > > >
> > > > 'file:/home/xen/domains/postfix02gluster01.cdmon.com/disk.img,sda2,w',
> > > >              ]
> > > >
> > > >
> > > > #
> > > > #  Hostname
> > > > #
> > > > name        = 'postfix02gluster01.cdmon.com'
> > > >
> > > > #
> > > > #  Networking
> > > > #
> > > > vif         = [
> > > > 'bridge=xenbr0,ip=212.x.x.x,mac=00:16:3E:F9:47:0D','bridge=xenbr1,ip=
> > > > 192.168.1.212,mac=00:16:3E:F9:47:0E'
> > > > ]
> > > >
> > > > #
> > > > #  Behaviour
> > > > #
> > > > on_poweroff = 'destroy'
> > > > on_reboot   = 'restart'
> > > > on_crash    = 'restart'
> > > >
> > > >
> > > >
> > > > **************
> > > >
> > > > En/na Basavanagowda Kanur ha escrit:
> > > >   
> > > > > Jordi,
> > > > >   We will be trying to reproduce your problem at our setup. To go
> > > > >       
> > > > further,
> > > >   
> > > > > we have few questions for you:
> > > > >
> > > > > 1. Where are glusterfs servers running? On virtual machines or real
> > > > > machines?
> > > > > 2. Where are glusterfs clients running? On virtual machines or real
> > > > > machines?
> > > > > 3. Is the Xen dom-0 also using glusterfs client? If yes, what all
> > > > >       
> > > > processes
> > > >   
> > > > > are accessing the glusterfs mount.
> > > > > 4. If Xen dom-0 hosts the block device images for Xen guest OSes on
> > > > > glusterfs mount, then what is the access method for guest OSes? 
> > > > > loopback
> > > > > (disk=['file:'])  or blktp (disk=['tap:aio:']) or nfs rexport?
> > > > >
> > > > > Detailed description of your Xen setup in addition to answers to 
> > > > > above
> > > > > questions will help us debug the problem.
> > > > >
> > > > > --
> > > > > Gowda
> > > > >
> > > > >
> > > > >       
> > > >     
> > >
> > >
> > >   
> >
> >
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/gluster-devel