Re: [Freeipmi-users] ipmiconsole sending unsollicited break

[Albert Chu]

On Tue, 2013-04-02 at 10:52 +0200, Fabien Wernli wrote:
> Hi,
> 
> On Fri, Mar 29, 2013 at 10:49:48AM -0700, Albert Chu wrote:
> > A) It seems you are using a console manager of some sort.  We use Conman
> > here and have never seen this issue.  Is it possible the console manager
> > is doing something to send the breaks/sysrqs?
> 
> We're using conserver. I seriously doubt the latter is to blame, as we've
> never seen this using ipmitool.
> 
> > Note that I am aware of two console managers (Conman & Conserver) that
> > have "native" IPMI console support via FreeIPMI's libipmiconsole
> > library.  So if you use one of those, you wouldn't have to run
> > 'ipmitool' or 'ipmiconsole' in a separate process.
> 
> Conserver doesn't support IPMI. Good to know Conman does, though.

I patch for libipmiconsole was submitted to Conserver a long time ago,
but I guess it was not added.

> > B) The default escape character in ipmiconsole is '&', which may not
> > match with whatever the console manager wants to use or would expect?  I
> > think ipmitool's is '~'.
> 
> Conserver uses a very long escape sequence, which is IMHO a good idea as it
> reduces the chances to accidentally send a break sequence. As for our
> previous setup, we were using '[' for the ipmitool sequence, but conserver
> doesn't actually know/care about it. This being said, I don't think this is
> relevant, as the SysRq were sent without anyone attached to the consoles,
> so the break must have been sent internally, I guess.
> 
> > C) You note there was a lot of garbage when you switched from ipmitool
> > to ipmiconsole.  It is possible (depending on many factors) that
> > ipmiconsole was receiving packets from the previous ipmitool's console
> > session.  The garbage is corrupted b/c ipmiconsole doesn't have the
> > right information to decrypt the data.  This can have unexpected side
> > affects.
> 
> This is what I was thinking, as ipmiconsole tries to stop the existing
> connection. Maybe it wrongly assumes the latter is consumed by another
> ipmiconsole process?  Moreover, I think this is a bug: freeipmi really
> shouldn't send data without any user input, should it?

It could certainly be a bug in freeipmi or conserver getting confused
when there is garbage (i.e. possible buffer overflow), but now I think
it's more likely it's a bug in the ipmi firmware.

I believe the previous console session (via ipmitool) was not cleaned up
properly, and that the second console session (via ipmiconsole) was
receiving left over data from the previous ipmitool session.

If the firmware is confused enough to believe that the new session
(ipmiconsole) is the old session (ipmitool), then there's a good chance
the buffers with input/output data in the ipmi firmware cannot be
trusted.  With enough servers, a few probably got the magic sysrqs to
reboot themselves.

> > Do you continue to see this garbage on subsequent reboots?  If not, it
> > may have been a one-time problem.
> 
> No, after reboot all was fine, although it lasted quite long as one of the
> servers was found in the boot configuration screen of its RAID adapter. We
> were lucky the garbage didn't reconfigure the ARRAY using random characters
> (or the complete works of William Shakespeare for that matter).

Good, hopefully it's just a one time incident.

> > Those are just some ideas.
> 
> Thanks for looking into this. Maybe there is some evidence in the routine
> handling the disconnection in the code.
> 
> Oh, and the two servers this happened on were IBM (3550 and 3650).
> 
> Cheers
> 
> 
> _______________________________________________
> Freeipmi-users mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/freeipmi-users
-- 
Albert Chu
address@hidden
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory