RE: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect to various Intel servers

[Cress, Andrew R]

Al,

For the CTS_MASK, I'm not where I can pull this up, but I thought that this 
option was something that could be detected.  IIRC, I think all BMCs will work 
with this FALSE, but only some support it with TRUE.

For the status loop, I'd first assume a software bug, before pointing to the 
time-tested and fully validated firmware.  The point of the get activation 
payload status is to prevent multiple activate payload commands from being 
issued from one client.

Andy



-----Original Message-----
From: Al Chu [mailto:address@hidden
Sent: Monday, December 08, 2008 2:34 PM
To: Cress, Andrew R
Cc: address@hidden
Subject: RE: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect to 
various Intel servers

Hey Andrew,

For fi4.log I see a return code of CCh = "Invalid data field in Request"
for the Activate Payload command.  Comparing FreeIPMI's code to your
code in ipmiutil, I see:

      if (mfg == VENDOR_INTEL)
           ibuf[2] |= SOL_BMC_ASSERTS_CTS_MASK_TRUE;
      else ibuf[2] |= SOL_BMC_ASSERTS_CTS_MASK_FALSE;

So I would assume this is yet another intel specific workaround?  I
can't (code auditting wise) see another difference in the packets.

For fi3.log

I definitely got a cycle in my state machine.  I should (at minimum)
monitor and exit after some number of failed attempts for this state-
machine cycle.

It seems that the "get payload activation status" request always returns
that SOL is not active.  But whenever the "activate payload" command is
executed, it responds w/ 80h = "payload already active".  The current
state machine moves back to "get payload activation status" to
(hopefully) verify that the session is now active, so that it can then
move on to "deactivate payload".

Since one of these commands is continuously returning bad data, the
cycle results.   So I assume there is a bug in the Intel firmware
somewhere in here.  Perhaps the fix to the "activate payload" command up
above (from fi4.log) would be enough to make the activate payload
command work?

I'll put a fix in for these and e-mail you a beta link.

Thanks for all the help.

Al

On Mon, 2008-12-08 at 10:35 -0800, Al Chu wrote:
> Hey Andrew,
>
> Cool.  These logs show that we're fully authenticating, so the "hard"
> Intel IPMI 2.0 workarounds are over.  So there is something in the
> libipmiconsole state machine that it is stuck on.  Likely some corner
> case b/c I do things differently than ipmitool/ipmiutil.
>
> I'll take a look into it.
>
> Al
>
> On Mon, 2008-12-08 at 10:27 -0800, Cress, Andrew R wrote:
> > Al,
> >
> > Oops.  I forgot the -W intel20 option.
> > Here are logs with that included, and debug.
> > The problem seems to be with the Trailer now.
> > It flags a "BMC Error" message to one system and repeats the cycle forever 
> > on another system.
> >
> > Both of these can activate SOL sessions via ipmitool and ipmiutil.
> > It may be just one more tweak is needed.
> >
> > Andy
> >
> > -----Original Message-----
> > From: Al Chu [mailto:address@hidden
> > Sent: Monday, December 08, 2008 1:13 PM
> > To: Cress, Andrew R
> > Cc: address@hidden
> > Subject: RE: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect 
> > to various Intel servers
> >
> > Hey Andrew,
> >
> > Hmmm, the BMC Busy error seems justified.
> >
> > telco64: ------------------
> > telco64: [              93h] = message_tag[ 8b]
> > telco64: [               1h] = rmcpplus_status_code[ 8b]
> >
> > 1h = "Insufficient resources to create a session"
> >
> > So I think this error is legit given the response.  Any idea why the
> > board would respond with this type of error code?  Could it be to many
> > failed connection attempts?  I remember an Intel IPMI 1.5 board I played
> > with would not be very responsive if there were many failed connections
> > in a row.
> >
> > telco64: =====================================================
> > telco64: IPMI 2.0 Open Session Request
> > telco64: =====================================================
> > <snip>
> > telco64: IPMI Command Data:
> > telco64: ------------------
> > telco64: [              C5h] = message_tag[ 8b]
> > telco64: [               0h] = requested_maximum_privilege_level[ 4b]
> >
> > telco64: =====================================================
> > telco64: IPMI 2.0 Open Session Response
> > telco64: =====================================================
> > <snip>
> > telco64: IPMI Command Data:
> > telco64: ------------------
> > telco64: [              C5h] = message_tag[ 8b]
> > telco64: [               0h] = rmcpplus_status_code[ 8b]
> > telco64: [               0h] = maximum_privilege_level[ 4b]
> >
> > (ipmiconsole_checks.c,
> > ipmiconsole_check_open_session_response_privilege, 610):
> > hostname=telco64; protocol_state=0x2: open session response privilege
> > check failed; p = 3^M
> > [error received]: privilege level cannot be obtained for this user
> >
> > I'll have to look into this.  0h = IPMI_PRIVILEGE_LEVEL_HIGHEST_LEVEL,
> > which should not be done for the Intel boards when the intel workaround
> > is specified.
> >
> >       /* IPMI Workaround
> >        *
> >        * Intel IPMI 2.0 implementations don't support the highest level
> > privilege.
> >        */
> >       if (c->config.workaround_flags &
> > IPMICONSOLE_WORKAROUND_INTEL_2_0_SESSION)
> >         privilege_level = c->config.privilege_level;
> >       else
> >         privilege_level = IPMI_PRIVILEGE_LEVEL_HIGHEST_LEVEL;
> >
> > Al
> >
> > On Mon, 2008-12-08 at 06:38 -0800, Cress, Andrew R wrote:
> > > Al,
> > >
> > > Attached are the debug logs of the two errors I get.
> > > Same errors for any Intel server, the "BMC Busy" error in fi.log is the 
> > > most common.
> > >
> > > Andy
> > >
> > >
> > > -----Original Message-----
> > > From: Al Chu [mailto:address@hidden
> > > Sent: Friday, December 05, 2008 5:41 PM
> > > To: Cress, Andrew R
> > > Subject: RE: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect 
> > > to various Intel servers
> > >
> > > Hey Andrew,
> > >
> > > If you have time, do you think you could sanity check this tar.gz's
> > > ipmiconsole and some of the other stuff (ipmipower and ipmi-sensors
> > > would hit all the ipmi 2.0 "implementations") against an Intel machine.
> > > I'd appreciate it.
> > >
> > > http://   ftp.zresearch.com/pub/freeipmi/qa-
> > > release/freeipmi-0.7.4.beta0.tar.gz
> > >
> > > Al
> > >
> > > On Fri, 2008-12-05 at 11:31 -0800, Cress, Andrew R wrote:
> > > > Yep, I meant to save the draft and go finish that part but hit send 
> > > > instead.  :-)
> > > >
> > > > Andy
> > > >
> > > > -----Original Message-----
> > > > From: Al Chu [mailto:address@hidden
> > > > Sent: Friday, December 05, 2008 2:29 PM
> > > > To: Cress, Andrew R
> > > > Cc: address@hidden
> > > > Subject: RE: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot 
> > > > connect to various Intel servers
> > > >
> > > > Hey Andrew,
> > > >
> > > > On Fri, 2008-12-05 at 10:54 -0800, Cress, Andrew R wrote:
> > > > > Al,
> > > > >
> > > > >       /* IPMI Workaround (achu)
> > > > >        *
> > > > >        * Discovered on SE7520AF2 with Intel Server Management Module
> > > > >        * (Professional Edition)
> > > > >        *
> > > > >        * The username must be padded despite explicitly not being
> > > > >        * allowed.  "No Null characters (00h) are allowed in the name".
> > > > >        * Table 13-11 in the IPMI 2.0 spec.
> > > > >        */
> > > > > Hmmm.  This is because when the set username command is issued, the
> > > > > command data is padded with nulls to 16 bytes, as shown in secion
> > > > > 22.28, and that is common across most of the spec.  But Table 13-11 is
> > > > > an exception to the rest of IPMI username handling, in that it is not
> > > > > a fixed-length 16-byte entity.  This is implemented the same in most
> > > > > Intel BMCs that are currently available.  In ipmiutil, this RAKP1
> > > > > stuff is handled the same way that ipmitool does, by ...
> > > >
> > > > Forgot to write the end to this part? :-)
> > > >
> > > > Looking through the ipmiutil code, it doesn't seem like any padding is
> > > > done.  I wrote the Intel 2.0 workarounds a long time ago and probably
> > > > added things to "just make them work", not seeing if there was a cleaner
> > > > way to deal with it.
> > > >
> > > > Al
> > > >
> > > > >
> > > > >       /* IPMI Workaround (achu)
> > > > >        *
> > > > >        * Discovered on SE7520AF2 with Intel Server Management Module
> > > > >        * (Professional Edition)
> > > > >        *
> > > > >        * When the authentication algorithm is HMAC-MD5-128 and the
> > > > >        * password is greater than 16 bytes, the Intel BMC truncates 
> > > > > the
> > > > >        * password to 16 bytes when generating keys, hashes, etc.  So 
> > > > > we
> > > > >        * have to do the same when generating keys, hashes, etc.
> > > > >        */
> > > > > I doubt that the IMM firmware supported 20-byte passwords (the first 
> > > > > Intel BMC with IPMI 2.0), but some other Intel BMCs do, I believe.  
> > > > > Ipmiutil currently does not handle 20-byte passwords, and that could 
> > > > > be done conditionally on certain systems, but if we are talking about 
> > > > > managing passwords for hundreds of servers where some support 16 
> > > > > bytes and some support 20 bytes, all of the passwords would be 
> > > > > restricted to 16-bytes anyway, and the IPMI 2.0 spec requires 
> > > > > support/compatibility for 16-byte passwords for this reason.  We 
> > > > > don't have any customers who have a use-case where 20-byte passwords 
> > > > > would be used, so I haven't implemented the logic for it.
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: address@hidden [mailto:address@hidden On Behalf Of Al Chu
> > > > > Sent: Friday, December 05, 2008 1:27 PM
> > > > > To: Cress, Andrew R
> > > > > Cc: address@hidden
> > > > > Subject: Re: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot 
> > > > > connect to various Intel servers
> > > > >
> > > > > Hey Andrew,
> > > > >
> > > > > On Fri, 2008-12-05 at 09:14 -0800, Cress, Andrew R wrote:
> > > > > > Retry, first send failed.
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Cress, Andrew R [mailto:address@hidden
> > > > > > Sent: Friday, December 05, 2008 8:43 AM
> > > > > > To: Albert Chu; Bryan Henderson; Andy Cress; address@hidden
> > > > > > Subject: RE: [bug #24300] ipmiconsole cannot connect to various 
> > > > > > Intel servers
> > > > > >
> > > > > > Al,
> > > > > >
> > > > > > This was the first time I had tried ipmiconsole, so I don't know if 
> > > > > > it worked before or what changed.
> > > > > >
> > > > > > For an example of what is different with Intel boards, you can view 
> > > > > > the source to ipmitool or ipmiutil under the 'lanplus' protocol.  
> > > > > > It boils down to some different assumptions about defaults or 
> > > > > > special conditions.
> > > > > > In ipmitool, the syntax requires specifying "-o intelplus", but 
> > > > > > ipmiutil detects the manufacturer/product id first and doesn't need 
> > > > > > those options.
> > > > > >
> > > > > > From ipmiutil:
> > > > > > lib/lanplus/lanplus.c:is_sol_partial_ack() has an intelplus special 
> > > > > > case, which probably should apply to all other boards too (?)
> > > > >
> > > > > Yeah, I'm not quite sure why this would be an intel corner case.  Does
> > > > > intel return 0 for the accepted_character_count even if it accepted 
> > > > > all
> > > > > the data?  So a return of 0 is just assumed to be equivalent to "all
> > > > > data accepted"??  I don't currently handle this case in ipmiconsole.
> > > > > The assumption is if you didn't accept any data, then you resend data
> > > > > just as if there was a partial acceptance.
> > > > >
> > > > > > lib/lanplus/lanplus.c:ipmi_lanplus_open_session() has an intelplus 
> > > > > > condition for privilege defaults
> > > > >
> > > > > I handle this one:
> > > > >
> > > > >       /* IPMI Workaround
> > > > >        *
> > > > >        * Intel IPMI 2.0 implementations don't support the highest 
> > > > > level privilege.
> > > > >        */
> > > > >       if (c->config.workaround_flags & 
> > > > > IPMICONSOLE_WORKAROUND_INTEL_2_0_SESSION)
> > > > >         privilege_level = c->config.privilege_level;
> > > > >       else
> > > > >         privilege_level = IPMI_PRIVILEGE_LEVEL_HIGHEST_LEVEL;
> > > > >
> > > > > > lib/lanplus/lanplus_crypt.c:lanplus_rakp4_hmac_matches() has two 
> > > > > > intelplus cases
> > > > >
> > > > > handle this one:
> > > > >
> > > > >   /* IPMI Workaround
> > > > >    *
> > > > >    * Intel IPMI 2.0 implementations respond with the integrity check
> > > > >    * value based on the integrity algorithm rather than the
> > > > >    * authentication algorithm.
> > > > >    */
> > > > >   if (c->config.workaround_flags & 
> > > > > IPMICONSOLE_WORKAROUND_INTEL_2_0_SESSION)
> > > > >     {
> > > > >       if (c->config.integrity_algorithm == 
> > > > > IPMI_INTEGRITY_ALGORITHM_NONE)
> > > > >         authentication_algorithm = 
> > > > > IPMI_AUTHENTICATION_ALGORITHM_RAKP_NONE;
> > > > >       else if (c->config.integrity_algorithm == 
> > > > > IPMI_INTEGRITY_ALGORITHM_HMAC_SHA1_96)
> > > > >         authentication_algorithm = 
> > > > > IPMI_AUTHENTICATION_ALGORITHM_RAKP_HMAC_SHA1;
> > > > >       else if (c->config.integrity_algorithm == 
> > > > > IPMI_INTEGRITY_ALGORITHM_HMAC_MD5_128)
> > > > >         authentication_algorithm = 
> > > > > IPMI_AUTHENTICATION_ALGORITHM_RAKP_HMAC_MD5;
> > > > >       else if (c->config.integrity_algorithm == 
> > > > > IPMI_INTEGRITY_ALGORITHM_MD5_128)
> > > > >         /* achu: I have not been able to reverse engineer this.  So 
> > > > > accept it */
> > > > >         return 1;
> > > > >     }
> > > > >   else
> > > > >     authentication_algorithm = c->config.authentication_algorithm;
> > > > >
> > > > > > lib/lanplus/lanplus_crypt.c:lanplus_generate_rakp3_authcode() has 
> > > > > > an intelplus case for privilege defaults
> > > > >
> > > > > Ahh, this might be it.  In ipmipower and libfreeipmi's ipmi 2.0 code I
> > > > > handle this properly.  But in ipmiconsole I seem to have accidentally
> > > > > put this code in the RAKP1 section.  That might be the reason that I'm
> > > > > checking the return value from the RAKP2 incorrectly.
> > > > >
> > > > > I'll do some more auditing to see if I can find why the other fellow's
> > > > > example isn't working on the INTELs.  I noticed he is using a NULL
> > > > > username/password.  You mind given my next tar.gz a test run to see 
> > > > > if I
> > > > > caught everything?
> > > > >
> > > > > BTW, it seems as though ipmiutil does not implement all of the Intel
> > > > > workarounds I found.  There were a number of corner cases for
> > > > > username/password lengths.  Here's what I have in my comments.
> > > > >
> > > > >       /* IPMI Workaround (achu)
> > > > >        *
> > > > >        * Discovered on SE7520AF2 with Intel Server Management Module
> > > > >        * (Professional Edition)
> > > > >        *
> > > > >        * The username must be padded despite explicitly not being
> > > > >        * allowed.  "No Null characters (00h) are allowed in the name".
> > > > >        * Table 13-11 in the IPMI 2.0 spec.
> > > > >        */
> > > > >
> > > > >       /* IPMI Workaround (achu)
> > > > >        *
> > > > >        * Discovered on SE7520AF2 with Intel Server Management Module
> > > > >        * (Professional Edition)
> > > > >        *
> > > > >        * When the authentication algorithm is HMAC-MD5-128 and the
> > > > >        * password is greater than 16 bytes, the Intel BMC truncates 
> > > > > the
> > > > >        * password to 16 bytes when generating keys, hashes, etc.  So 
> > > > > we
> > > > >        * have to do the same when generating keys, hashes, etc.
> > > > >        */
> > > > >
> > > > > Does ipmiutil handle these too?  Or is it possible Intel fixed some
> > > > > issues but not others in newer firmware?
> > > > >
> > > > > Al
> > > > >
> > > > > > That's all that is different.
> > > > > >
> > > > > > Andy
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Albert Chu [mailto:address@hidden
> > > > > > Sent: Thursday, December 04, 2008 5:47 PM
> > > > > > To: Albert Chu; Bryan Henderson; Andy Cress; address@hidden
> > > > > > Subject: [bug #24300] ipmiconsole cannot connect to various Intel 
> > > > > > servers
> > > > > >
> > > > > >
> > > > > > Follow-up Comment #2, bug #24300 (project freeipmi):
> > > > > >
> > > > > > Sorry I didn't see these posts earlier.  Hopefully I've fixed the 
> > > > > > config on
> > > > > > Savannah so that bugs actually send out e-mails to the mailing list.
> > > > > >
> > > > > > I implemented the Intel workarounds a long time ago, but no longer 
> > > > > > have an
> > > > > > Intel motherboard.  So I've been forward porting the patches since 
> > > > > > then and
> > > > > > praying they still work and I didn't mess anything up along the 
> > > > > > way.  I guess
> > > > > > something is messed up or there is something new to workaround.
> > > > > >
> > > > > > Hopefully I can find an Intel mobo to try and fix this on.  I'm 
> > > > > > going through
> > > > > > the code right now visually and can't see a workaround issue.
> > > > > >
> > > > > > Al
> > > > > >
> > > > > > P.S.  Bryan, I can see how the wording of the manpage was 
> > > > > > misinterpreted to
> > > > > > make you think "I note the manual mentions this can happen with
> > > > > > --workaround=intel20, but it doesn't mention anything to do about 
> > > > > > it. ".  I'm
> > > > > > going to fix up the manpage to instead say:
> > > > > >
> > > > > > There are a number of Intel IPMI 2.0 bugs.  These problems may cause
> > > > > > "username invalid", "password invalid", or "k_g invalid" errors to 
> > > > > > occur.
> > > > > > They can be worked around by specifying the "intel20" workaround.
> > > > > >
> > > > > >
> > > > > >     _______________________________________________________
> > > > > >
> > > > > > Reply to this item at:
> > > > > >
> > > > > >   <http://      savannah.gnu.org/bugs/?24300>
> > > > > >
> > > > > > _______________________________________________
> > > > > >   Message sent via/by Savannah
> > > > > >   http://      savannah.gnu.org/
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Freeipmi-devel mailing list
> > > > > > address@hidden
> > > > > > http://      lists.gnu.org/mailman/listinfo/freeipmi-devel
> > > > > >
> > --
> > Albert Chu
> > address@hidden
> > Computer Scientist
> > High Performance Systems Division
> > Lawrence Livermore National Laboratory
> >
--
Albert Chu
address@hidden
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory