Re: [Gluster-devel] new glusterfs logging framework

[Balamurugan Arumugam]

Hi all,


----- Original Message -----
> From: "Balamurugan Arumugam" <address@hidden>
> To: "Niels de Vos" <address@hidden>
> Cc: "Balamurugan Arumugam" <address@hidden>, address@hidden, address@hidden
> Sent: Friday, August 9, 2013 6:53:11 PM
> Subject: Re: [Gluster-devel] new glusterfs logging framework
> 
> 
> 
> ----- Original Message -----
> > From: "Niels de Vos" <address@hidden>
> > To: "Balamurugan Arumugam" <address@hidden>
> > Cc: "Balamurugan Arumugam" <address@hidden>, address@hidden,
> > address@hidden
> > Sent: Friday, August 9, 2013 3:19:42 PM
> > Subject: Re: [Gluster-devel] new glusterfs logging framework
> > 
> > On Fri, Aug 09, 2013 at 04:47:45AM -0400, Balamurugan Arumugam wrote:
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Niels de Vos" <address@hidden>
> > > > To: "Balamurugan Arumugam" <address@hidden>
> > > > Cc: "Balamurugan Arumugam" <address@hidden>,
> > > > address@hidden,
> > > > address@hidden
> > > > Sent: Thursday, August 8, 2013 2:46:38 PM
> > > > Subject: Re: [Gluster-devel] new glusterfs logging framework
> > > > 
> > > > On Thu, Aug 08, 2013 at 04:44:01AM -0400, Balamurugan Arumugam wrote:
> > > > > 
> > > > > 
> > > > > ----- Original Message -----
> > > > > > From: "Niels de Vos" <address@hidden>
> > > > > > To: "Balamurugan Arumugam" <address@hidden>
> > > > > > Cc: address@hidden, address@hidden
> > > > > > Sent: Tuesday, August 6, 2013 8:48:58 PM
> > > > > > Subject: Re: [Gluster-devel] new glusterfs logging framework
> > > > > > 
> > > > > > On Tue, Jul 30, 2013 at 05:29:43AM -0400, Balamurugan Arumugam
> > > > > > wrote:
> > > > > > > 
> > > > > > > Hi All,
> > > > > > > 
> > > > > > > Recently new logging framework was introduced [1][2][3] in
> > > > > > > glusterfs
> > > > > > > master branch.  You could read more about this on
> > > > > > > doc/logging.txt.
> > > > > > > In
> > > > > > > brief, current log target is moved to syslog and user has an
> > > > > > > option
> > > > > > > to
> > > > > > > this new logging at compile time (passing '--disable-syslog' to
> > > > > > > ./configure or '--without syslog' to rpmbuild) and run time
> > > > > > > (having
> > > > > > > a file /var/log/glusterd/logger.conf and restarting gluster
> > > > > > > services).
> > > > > > > 
> > > > > > > As rsyslog is used as syslog server in Fedora and CentOS/RHEL and
> > > > > > > default configuration of rsyslog does not have any rule specific
> > > > > > > to
> > > > > > > gluster logs, you see all logs are in /var/log/messages in JSON
> > > > > > > format.
> > > > > > 
> > > > > > As the below does not function on RHEL-6 with its own rsyslog (no
> > > > > > rsyslog-mmjsonparse available), I am creating an empty
> > > > > > /var/lib/glusterd/logger.conf which (after a restart) makes
> > > > > > glusterd
> > > > > > and
> > > > > > friends to log to their standard files again.
> > > > > > 
> > > > > > Do you have some progress on the rsyslog-mmjsonparse for rhel5 and
> > > > > > rhel6?
> > > > > > 
> > > > > 
> > > > > mmcount is a rsyslog plugin (developed by me) available rsyslog
> > > > > upstream which tracks each gluster log using error codes.  The error
> > > > > code is the entity enabled by gluster logging framework.  This module
> > > > > is back ported to rhel6 which does tracking and json parsing.  I am
> > > > > in
> > > > > the process of submitting this to rhel 6 downstream.  I haven't tried
> > > > > rhel5 yet and I will do that too.
> > > > 
> > > > Very cool!
> > > > 
> > > > > > Also, it would really be nice to include that
> > > > > > /etc/rsyslog.d/gluster.conf in the rpms and add a dependency on the
> > > > > > rsyslog-mmjsonparse package. Or, provide a
> > > > > > /var/lib/glusterd/logger.conf
> > > > > > for the distributions that do not have rsyslog-mmjsonparse.
> > > > > > 
> > > > > 
> > > > > Yes.  I had similar thought on this to do so.  I will send the fix
> > > > > soon.
> > > > 
> > > > Great, thanks. Also, consider to move the file to /etc/glusterd where
> > > > all configuration files that sysadmins may edit are kept.
> > > > 
> > > 
> > > the logger.conf file will be placed into /etc/glusterfs/ directory.
> > > The change is asked by jeo julian.  Let me know what you think.
> > 
> > Thanks, much better.
> > 
> 
> BTW, rhel 5 doesn't have rsyslog.  I thought of disabling this frame work for
> rhel 5.  Also I submitted below patches which fixes issues in this framework
> 
> http://review.gluster.org/5541
> http://review.gluster.org/5542
> http://review.gluster.org/5543
> 

These fixes set log format like below

/DAEMON/INFO [2013-08-12T16:09:33.575933+05:30] [9999] [devel error] 
[rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout 
to 600
/DAEMON/INFO [2013-08-12T16:09:33.576314+05:30] [9999] [devel error] 
[socket.c:3515:socket_init] 0-management: SSL support is NOT enabled
/DAEMON/INFO [2013-08-12T16:09:33.576541+05:30] [9999] [devel error] 
[socket.c:3530:socket_init] 0-management: using system polling thread
/DAEMON/INFO [2013-08-12T16:09:33.596181+05:30] [9999] [devel error] 
[rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout 
to 600
/DAEMON/INFO [2013-08-12T16:09:33.596639+05:30] [9999] [devel error] 
[socket.c:3515:socket_init] 0-management: SSL support is NOT enabled
/DAEMON/INFO [2013-08-12T16:09:33.596884+05:30] [9999] [devel error] 
[socket.c:3530:socket_init] 0-management: using system polling thread
/DAEMON/INFO [2013-08-12T16:09:33.624693+05:30] [9999] [devel error] 
[socket.c:2240:socket_event_handler] 0-transport: disconnecting now
/DAEMON/INFO [2013-08-12T16:09:33.625027+05:30] [9999] [devel error] 
[glusterd-handler.c:3653:__glusterd_brick_rpc_notify] 0-management: 
Disconnected from 192.168.122.200:/var/tmp/music-b1
/DAEMON/INFO [2013-08-12T16:09:33.628410+05:30] [9999] [devel error] 
[socket.c:2240:socket_event_handler] 0-transport: disconnecting now
/DAEMON/INFO [2013-08-12T16:09:33.807009+05:30] [9999] [devel error] 
[glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /var/tmp/music-b1 
on port 49152


I would like to get feedback on this.


Regards,
Bala


> 
> Help on reviewing these patches would be really helpful
> 
> Thanks,
> Bala
> 
> 
> 
> > Niels
> > 
>