[Gluster-devel] atomic operations fails_

[Fredrik Widlund]

Hi,

 

I’ll try to make myself clearer than in the earlier thread, since I need some help here. I’m not sure if I am missing something.

 

I am not able to use atomic operations using glusterfs 3.0.2, on a Arch Linux 2.6.32.7-1 x86_64 server.

 

I’ve stripped everything down to the most simple configuration

Server: a posix store with locks

Client: a tcp client

 

I mount the glusterfs on /gfs, and use a “writer” and a “reader” as below:

[writer]                address@hidden> while :; do date > /gfs/.file; mv /gfs/.file /gfs/file; done

[reader]               address@hidden> while :; do cat /gfs/file > /dev/null; done

 

Leaving this running, I get a flood of

cat: /gfs/file: No such file or directory

 

This should not happen on a filesystem that supports atomic operations?!

 

What this means is that I am unable to replace a file as an atomic operation, creating a race-condition where the file is sometimes not present. Furthermore this actually “desyncs” the server in a more complex setup, requiring a restart of the server part to make “desynced” files available again.

 

This reproduces using XFS as well as EXT2 as the underlying fs.

 

[glusterfs.vol]

volume media0

  type protocol/client

  option transport-type tcp

  option remote-host 127.0.0.1

  option remote-subvolume media0

end-volume

 

mounted with “glusterfs -f glusterfs.vol /gfs”

 

[glusterfsd.vol]

(se below debug log)

 

-x-

 

[2010-02-09 18:18:23] D [glusterfsd.c:424:_get_specfp] glusterfs: loading volume file glusterfsd.vol

[2010-02-09 18:18:23] D [xlator.c:739:xlator_set_type] xlator: dlsym(notify) on /usr/lib/glusterfs/3.0.2/xlator/features/locks.so: undefined symbol: notify -- neglecting

================================================================================

Version      : glusterfs 3.0.2 built on Feb  8 2010 15:23:34

git: v3.0.2

Starting Time: 2010-02-09 18:18:23

Command line : glusterfs -f glusterfsd.vol --debug

PID          : 19486

System name  : Linux

Nodename     : n100.p1.x0.local

Kernel Release : 2.6.32-ARCH

Hardware Identifier: x86_64

 

Given volfile:

+------------------------------------------------------------------------------+

  1: volume media0-store

  2:   type storage/posix

  3:   option directory /mnt

  4: end-volume

  5:

  6: volume media0

  7:   type features/locks

  8:   subvolumes media0-store

  9: end-volume

 10:

 11: volume server

 12:   type protocol/server

 13:   option transport-type tcp

 14:   option auth.addr.media0.allow *

 15:   subvolumes media0

 16: end-volume

 

+------------------------------------------------------------------------------+

[2010-02-09 18:18:23] D [glusterfsd.c:1370:main] glusterfs: running in pid 19486

[2010-02-09 18:18:23] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so

[2010-02-09 18:18:23] N [glusterfsd.c:1396:main] glusterfs: Successfully started

[2010-02-09 18:18:43] D [addr.c:190:gf_auth] media0: allowed = "*", received addr = "127.0.0.1"

[2010-02-09 18:18:43] N [server-protocol.c:5812:mop_setvolume] server: accepted client from 127.0.0.1:1023

[2010-02-09 18:18:43] D [addr.c:190:gf_auth] media0: allowed = "*", received addr = "127.0.0.1"

[2010-02-09 18:18:43] N [server-protocol.c:5812:mop_setvolume] server: accepted client from 127.0.0.1:1022

[2010-02-09 18:18:58] D [server-resolve.c:238:resolve_path_deep] media0: RESOLVE RENAME() seeking deep resolution of /file

[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f79b

[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f7b3

[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f7cb

[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f79b

[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f7b3

[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f7cb

[2010-02-09 18:19:15] D [inode.c:916:inode_path] media0/inode: no dentry for non-root inode 502725

[2010-02-09 18:19:34] D [inode.c:916:inode_path] media0/inode: no dentry for non-root inode 502734

[2010-02-09 18:19:56] D [inode.c:916:inode_path] media0/inode: no dentry for non-root inode 502734

[…] repeated many times. Frequency of the “no dentry” entries depends on how aggressive the reader is.