Re: [Freeipmi-users] ipmiconsole sending unsollicited break

[Albert Chu]

Hi Fabian,

While there is always the possibility of a bug in FreeIPMI, I think it's
unlikely this is caused by ipmiconsole itself.  Some possibilities:

A) It seems you are using a console manager of some sort.  We use Conman
here and have never seen this issue.  Is it possible the console manager
is doing something to send the breaks/sysrqs?

Note that I am aware of two console managers (Conman & Conserver) that
have "native" IPMI console support via FreeIPMI's libipmiconsole
library.  So if you use one of those, you wouldn't have to run
'ipmitool' or 'ipmiconsole' in a separate process.

B) The default escape character in ipmiconsole is '&', which may not
match with whatever the console manager wants to use or would expect?  I
think ipmitool's is '~'.

C) You note there was a lot of garbage when you switched from ipmitool
to ipmiconsole.  It is possible (depending on many factors) that
ipmiconsole was receiving packets from the previous ipmitool's console
session.  The garbage is corrupted b/c ipmiconsole doesn't have the
right information to decrypt the data.  This can have unexpected side
affects.

Do you continue to see this garbage on subsequent reboots?  If not, it
may have been a one-time problem.

Those are just some ideas.

Al

On Fri, 2013-03-29 at 11:04 +0100, Fabien Wernli wrote: 
> Hi,
> 
> We just encountered a serious problem when switching from ipmitool to
> ipmiconsole: on at least two out of several hundred hosts, multiple
> 'SysRq' breaks were sent just after the sol session establishment, resulting
> in one server reset, and one power off.
> Here's some relevant snippets from one server:
> 
> --
> server24:/var/log/messages
> --------------------------
> Mar 28 14:30:41 server24 kernel: SysRq : HELP : loglevel0-8 reBoot Crashdump 
> tErm Full kIll thaw-filesystems(J) saK showMem Nice powerOff showPc unRaw 
> Sync showTasks Unmount shoWcpus
> Mar 28 14:36:03 server24 kernel: SysRq : HELP : loglevel0-8 reBoot 
> CrashdumptErm Full kIll thaw-filesystems(J) saK showMem Nice powerOff showPc 
> unRaw Sync showTasks Unmount shoWcpus
> Mar 28 14:36:04 server24 kernel: SysRq : Changing Loglevel
> Mar 28 14:36:04 server24 kernel: SysRq : HELP : loglevel0-8 reBoot Crashdump 
> tErm Full kIll thaw-filesystems(J) saK showMem Nice powerOff showPc unRaw 
> Sync showTasks Unmount shoWcpus
> Mar 28 14:36:06 server24 kernel: SysRq : Power Off
> Mar 28 14:36:06 server24 kernel: SysRq : HELP : loglevel0-8 reBoot Crashdump 
> tErm Full kIll thaw-filesystems(J) saK showMem Nice powerOff showPc unRaw 
> Sync showTasks Unmount shoWcpus
> 
> server24:[on the ipmiconsole]
> -----------------------------
> [-- Console down -- Thu Mar 28 14:30:25 2013]
> [-- Console up -- Thu Mar 28 14:30:28 2013]
> 2013-03-28T14:30:29+01:00  [SOL established]
> 2013-03-28T14:30:29+01:00  ¥ 0O Qw¥ATIMÏ
> [ ... lots of garbage ... ]
> 2013-03-28T14:30:36+01:00  Ìoçiî iîcorrect
> 2013-03-28T14:30:36+01:00
> 2013-03-28T14:30:36+01:00  Scientific Linux SL release 5.1 (Boron)
> 2013-03-28T14:30:36+01:00  Kernel 2.6.18-164.9.1.el5 on an x86_64
> 2013-03-28T14:30:36+01:00
> 2013-03-28T14:30:37+01:00  server24 login: ø¼ø>§2øDsaÃaÇ¢søJ0q2
> [ ... lots of garbage until 14:36:06 (time of the SysRq : Power Off)... ]
> 2013-03-28T14:48:17+01:00  [SOL established]
> 2013-03-28T15:17:25+01:00  CP: C3 RN50 200m/200e DDR1 BIOS
> --
> 
> The first two lines on the console show the switch from 'ipmitool
> sol' to 'ipmiconsole'.
> The last line is text from the POST/BIOS startup and evidence of the boot
> process (someone switched the server back on using a finger).
> Note there is no '[generate break]' message on the console, nor any evidence
> of SysRq, although the kernel logged it.
> 
> Any insight would be greatly appreciated as to the how and why. Currently
> we're considering disabling the dangerous SysRq keys in the kernels, at the
> cost of useful functionality. The switch to freeipmi was motivated by a
> memory leak in ipmitool sol.
> 
> Cheers
> 
> 
> _______________________________________________
> Freeipmi-users mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/freeipmi-users
-- 
Albert Chu
address@hidden
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory