Re: [Freeipmi-users] ipmiconsole sending unsollicited break

[Hank Bruning]

Hi,
I'm not 100% sure but think Al is right about the IPM Controller
sending/receiving SOL causing the problem. We(JBlade)  provide a Java
implemention of IPMI (Hemi) with a Java ANSII terminal emulator and see the
same thing.
It happens when a IPM Controller's mux changes between two serial devices
within a single SOL session or when you change from hardware flow control
to software flow control.
This is very difficult to test because it's not reproducible.
Hank Bruning

On Tue, Apr 2, 2013 at 1:27 PM, Albert Chu <address@hidden> wrote:

> On Tue, 2013-04-02 at 10:52 +0200, Fabien Wernli wrote:
> > Hi,
> >
> > On Fri, Mar 29, 2013 at 10:49:48AM -0700, Albert Chu wrote:
> > > A) It seems you are using a console manager of some sort.  We use
> Conman
> > > here and have never seen this issue.  Is it possible the console
> manager
> > > is doing something to send the breaks/sysrqs?
> >
> > We're using conserver. I seriously doubt the latter is to blame, as we've
> > never seen this using ipmitool.
> >
> > > Note that I am aware of two console managers (Conman & Conserver) that
> > > have "native" IPMI console support via FreeIPMI's libipmiconsole
> > > library.  So if you use one of those, you wouldn't have to run
> > > 'ipmitool' or 'ipmiconsole' in a separate process.
> >
> > Conserver doesn't support IPMI. Good to know Conman does, though.
>
> I patch for libipmiconsole was submitted to Conserver a long time ago,
> but I guess it was not added.
>
> > > B) The default escape character in ipmiconsole is '&', which may not
> > > match with whatever the console manager wants to use or would expect?
>  I
> > > think ipmitool's is '~'.
> >
> > Conserver uses a very long escape sequence, which is IMHO a good idea as
> it
> > reduces the chances to accidentally send a break sequence. As for our
> > previous setup, we were using '[' for the ipmitool sequence, but
> conserver
> > doesn't actually know/care about it. This being said, I don't think this
> is
> > relevant, as the SysRq were sent without anyone attached to the consoles,
> > so the break must have been sent internally, I guess.
> >
> > > C) You note there was a lot of garbage when you switched from ipmitool
> > > to ipmiconsole.  It is possible (depending on many factors) that
> > > ipmiconsole was receiving packets from the previous ipmitool's console
> > > session.  The garbage is corrupted b/c ipmiconsole doesn't have the
> > > right information to decrypt the data.  This can have unexpected side
> > > affects.
> >
> > This is what I was thinking, as ipmiconsole tries to stop the existing
> > connection. Maybe it wrongly assumes the latter is consumed by another
> > ipmiconsole process?  Moreover, I think this is a bug: freeipmi really
> > shouldn't send data without any user input, should it?
>
> It could certainly be a bug in freeipmi or conserver getting confused
> when there is garbage (i.e. possible buffer overflow), but now I think
> it's more likely it's a bug in the ipmi firmware.
>
> I believe the previous console session (via ipmitool) was not cleaned up
> properly, and that the second console session (via ipmiconsole) was
> receiving left over data from the previous ipmitool session.
>
> If the firmware is confused enough to believe that the new session
> (ipmiconsole) is the old session (ipmitool), then there's a good chance
> the buffers with input/output data in the ipmi firmware cannot be
> trusted.  With enough servers, a few probably got the magic sysrqs to
> reboot themselves.
>
> > > Do you continue to see this garbage on subsequent reboots?  If not, it
> > > may have been a one-time problem.
> >
> > No, after reboot all was fine, although it lasted quite long as one of
> the
> > servers was found in the boot configuration screen of its RAID adapter.
> We
> > were lucky the garbage didn't reconfigure the ARRAY using random
> characters
> > (or the complete works of William Shakespeare for that matter).
>
> Good, hopefully it's just a one time incident.
>
> > > Those are just some ideas.
> >
> > Thanks for looking into this. Maybe there is some evidence in the routine
> > handling the disconnection in the code.
> >
> > Oh, and the two servers this happened on were IBM (3550 and 3650).
> >
> > Cheers
> >
> >
> > _______________________________________________
> > Freeipmi-users mailing list
> > address@hidden
> > https://lists.gnu.org/mailman/listinfo/freeipmi-users
> --
> Albert Chu
> address@hidden
> Computer Scientist
> High Performance Systems Division
> Lawrence Livermore National Laboratory
>
>
>
> _______________________________________________
> Freeipmi-users mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/freeipmi-users
>