Re: [Freeipmi-devel] Re: [Freeipmi-users] querying sensors of Intel SE7210TP1-E board fails with ipmi_sdr_cache_create: internal IPMI error

[Al Chu]

Hey Christian,

Well, I poked on your mobo a bit today, but was not able to find the
smoking gun bug.  There is some subtlety of programming through
the /dev/i2c-0 driver that I'm clearly missing and can't see within all
the Linux documentation.  The SSIF driver in FreeIPMI was written a long
time ago, so it's always possible that there's been a subtle change in
the /dev/i2c driver implementation in newer Linuxes, or perhaps its a
subtlety with your motherboard b/c it is quite old (2003/2004 from what
I can see).

I was able to determine that the ENXIO errors are always coming from the
reads of responses, which is strange, strongly suggesting the writes are
succeeding.

When I get some time, I'll try and implement a userspace version of the
SSIF driver like they do in ipmiutil.  However, it's probably going to
be a long time till I can implement it.

Sorry, wish I could give you some better news for the short term.

Al

On Mon, 2011-02-14 at 13:37 -0800, Albert Chu wrote:
> Hey Christian,
> 
> Well, it seems that ipmiutil is doing direct io port calls to do their
> SSIF, while we're using /dev/i2c.  If there's a subtlety in the
> implementation difference, I'm not going to be able to figure it out in
> short order :-(
> 
> Access to a SSIF mobo would definitely help.  I'll e-mail you privately
> about this.
> 
> I've e-mailed the original driver authors to see if they have any
> immediate insight.
> 
> Al
> 
> On Mon, 2011-02-14 at 10:56 -0800, Christian Ruppert wrote:
> > # ipmiutil sel -v -x
> > ipmiutil ver 2.67
> > isel: version 2.67
> > ipmi_open: driver type = unknown
> > ipmi_open_mv: cannot open /dev/ipmi/0
> > ipmi_open_mv: cannot open /dev/ipmi0
> > ipmi_open_mv: cannot open /dev/ipmidev0
> > ipmi_open_mv: cannot open /dev/ipmidev/0
> > imbapi ipmi_open_ia: open(/dev/imb) failed: No such file or directory
> > smbios: Driver=8(SMBus), sa=84, Base=0x0400, Spacing=1
> > BMC SSIF/SMBus Interface at i2c=84 base=0x0400
> > ipmidir Cmd=01 NetFn=06 Lun=00 Sa=20 Data(0):
> > ipmidir Resp: status=0 cc=00, Data(11): 20 81 02 40 51 9f 22 03 00 11 43
> > open_direct: status=0, SMBus drv, ipmi=1
> > ipmi_open rc = 0 type = smb
> > Driver type smb, open rc = 0
> > ipmidir Cmd=01 NetFn=06 Lun=00 Sa=20 Data(0):
> > ipmidir Resp: status=0 cc=00, Data(11): 20 81 02 40 51 9f 22 03 00 11 43
> > -- BMC version 2.40, IPMI version 1.5
> > ipmidir Cmd=40 NetFn=0a Lun=00 Sa=20 Data(0):
> > ipmidir Resp: status=0 cc=00, Data(14): 51 00 00 c0 05 00 00 00 00 d3 17
> > 48 4d 02
> > GetSelInfo status = 0
> > SEL Ver 51 Support 02, Size = 92 records (Used=0, Free=92)
> > isel completed successfully
> > 
> > # bmc-info -D kcs --disable-auto-probe --driver-address=0x84
> > --register-spacing=1
> > ipmi-kcs-driver.c: 748: ipmi_kcs_write: error 'BMC busy' (7)
> > ipmi-kcs-driver-api.c: 251: _kcs_cmd_write: error 'BMC busy' (7)
> > ipmi-device-global-cmds-api.c: 87: ipmi_cmd_get_device_id: error
> > 'internal system error' (31)
> > ipmi_cmd_get_device_id: internal system error
> > 
> > On 02/14/2011 07:34 PM, Albert Chu wrote:
> > > Hey Christian,
> > > 
> > > Could you run
> > > 
> > > ipmiutil sel -v -x
> > > 
> > > the -x is to get some debug info, the "sel -v" is to just get a shorter
> > > output (don't need all the sensor debug info, just how ipmiutil is
> > > getting to the data :-).
> > > 
> > > One other random thought.  I think it's possible that your motherboard
> > > supports the KCS driver, it's just not "advertised".  Perhaps you can
> > > give
> > > 
> > > bmc-info -D kcs --disable-auto-probe --driver-address=0x84
> > > --register-spacing=1 a shot?
> > > 
> > > Al
> > > 
> > > On Sat, 2011-02-12 at 05:32 -0800, Christian Ruppert wrote:
> > >> Hi Albert,
> > >>
> > >> I don't get any data with the other commands at all.
> > >>
> > >> # ipmi-chassis --get-chassis-status
> > >> ipmi-ssif-driver.c: 269: _ipmi_i2c_smbus_access: errno '' (6)
> > >> ipmi-ssif-driver-api.c: 266: _ssif_cmd_read: error 'internal system
> > >> error' (12)
> > >> ipmi-chassis-cmds-api.c: 146: ipmi_cmd_get_chassis_status: error
> > >> 'internal system error' (31)
> > >> ipmi_cmd_get_chassis_status: internal system error
> > >>
> > >> # bmc-device --get-ssif-interface-capabilities
> > >> ipmi-ssif-driver.c: 269: _ipmi_i2c_smbus_access: errno '' (6)
> > >> ipmi-ssif-driver-api.c: 266: _ssif_cmd_read: error 'internal system
> > >> error' (12)
> > >> ipmi-messaging-support-cmds-api.c: 693:
> > >> ipmi_cmd_get_system_interface_capabilities_ssif: error 'internal system
> > >> error' (31)
> > >> ipmi_cmd_get_system_interface_capabilities_ssif: internal system error
> > >>
> > >> and so on...
> > >>
> > >> Somebody told me about ipmiutil, it seems to work.
> > >>
> > >> # ipmiutil sensor -s
> > >> ipmiutil ver 2.67
> > >> isensor: version 2.67
> > >> -- BMC version 2.40, IPMI version 1.5
> > >> Full sensor   [000a]| snum 0b | Baseboard 1.5V   | OK   | 1.52 Volts
> > >> GetSensorReading error cb Requested sensor, data, or record not present
> > >> Full sensor   [000b]| snum 0c | Baseboard 3.3V   | OK   | 0.00 Volts
> > >> GetSensorReading error cb Requested sensor, data, or record not present
> > >> Full sensor   [000c]| snum 0d | Baseboard 5.0V   | OK   | 0.00 Volts
> > >> GetSensorReading error cb Requested sensor, data, or record not present
> > >> Full sensor   [000d]| snum 0e | Baseboard 12V    | OK   | 0.00 Volts
> > >> Full sensor   [000e]| snum 10 | Processor Vccp   | OK   | 1.35 Volts
> > >> Full sensor   [000f]| snum 11 | Baseboard Temp   | OK   | 34.00 degrees C
> > >> ...
> > >>
> > >> I would be glad if we could fix that in FreeIPMI too :)
> > >> I could give you access to the box with that SSIF device, if you want :)
> > >>
> > >> On 02/12/2011 02:44 AM, Albert Chu wrote:
> > >>> Hey Christian,
> > >>>
> > >>>> No luck with 0x84 but 0x42 looks better, I hope it helps.
> > >>>
> > >>> That's good.  I guess the random luck of the order I probe devices isn't
> > >>> good for your motherboard.  I'd like to flip the probing order around
> > >>> for you, but I'm sort of afraid it'll mess up another persons system.
> > >>> Let me see if I can come up w/ something more creative to deal with
> > >>> this.
> > >>>
> > >>> But atleast in the short term you have something that would work for
> > >>> you.  If you configure /etc/freeipmi/freeipmi.conf, you can set the
> > >>> below as defaults so that you don't have to type them everytime.
> > >>>
> > >>>> ipmi-ssif-driver.c: 269: _ipmi_i2c_smbus_access: errno '' (6)
> > >>>
> > >>> This is really odd, errno 6 = ENXIO which means "no such device or
> > >>> address", but clearly it just worked w/ the previous packet.  I wonder
> > >>> if the ENXIO refers to something deeper inside IPMI, not something
> > >>> related to the actual device.
> > >>>
> > >>> Could you try other random IPMI commands, ipmi-chassis, bmc-device,
> > >>> ipmi-sensors, ipmi-sel, and see what they output.  Perhaps I need to
> > >>> handle ENXIO a special way, maybe it means something.
> > >>>
> > >>> Al
> > >>>
> > >>> On Fri, 2011-02-11 at 16:05 -0800, Christian Ruppert wrote:
> > >>>> No luck with 0x84 but 0x43 looks better, I hope it helps.
> > >>>>
> > >>>> # bmc-info -D ssif --disable-auto-probe --driver-address=0x84
> > >>>> --driver-device=/dev/i2c-0 --register-spacing=1
> > >>>> ipmi-ssif-driver.c: 685: ipmi_ssif_ctx_io_init: errno '' (22)
> > >>>> ipmi-api.c: 1012: ipmi_ctx_open_inband: error 'internal error' (13)
> > >>>> ipmi-api.c: 2029: ipmi_ctx_close: error 'device not open' (16)
> > >>>> ipmi_ctx_open_inband: internal error
> > >>>>
> > >>>>
> > >>>> bmc-info -D ssif --disable-auto-probe --driver-address=0x42
> > >>>> --driver-device=/dev/i2c-0 --register-spacing=1
> > >>>>
> > >>>> Device ID             : 32
> > >>>> Device Revision       : 1
> > >>>> Device SDRs           : supported
> > >>>> Firmware Revision     : 2.40
> > >>>> Device Available      : yes (normal operation)
> > >>>> IPMI Version          : 1.5
> > >>>> Sensor Device         : supported
> > >>>> SDR Repository Device : supported
> > >>>> SEL Device            : supported
> > >>>> FRU Inventory Device  : supported
> > >>>> IPMB Event Receiver   : supported
> > >>>> IPMB Event Generator  : unsupported
> > >>>> Bridge                : unsupported
> > >>>> Chassis Device        : supported
> > >>>> Manufacturer ID       : National Semiconductor (802)
> > >>>> Product ID            : 17169
> > >>>>
> > >>>> ipmi-ssif-driver.c: 269: _ipmi_i2c_smbus_access: errno '' (6)
> > >>>> ipmi-ssif-driver-api.c: 266: _ssif_cmd_read: error 'internal system
> > >>>> error' (12)
> > >>>> ipmi-device-global-cmds-api.c: 422: ipmi_cmd_get_device_guid: error
> > >>>> 'internal system error' (31)
> > >>>> ipmi_cmd_get_device_guid: internal system error
> > >>>>
> > >>>>
> > >>>> On 02/09/2011 10:56 PM, Albert Chu wrote:
> > >>>>> Hey Christian,
> > >>>>>
> > >>>>> Thanks for the traces.  I noticed something peculiar in the 
> > >>>>> ipmi-locate
> > >>>>> output.
> > >>>>>
> > >>>>> Probing SSIF device using DMIDECODE... done
> > >>>>> IPMI Version: 1.5
> > >>>>> IPMI locate driver: DMIDECODE
> > >>>>> IPMI interface: SSIF
> > >>>>> BMC driver device: /dev/i2c-0
> > >>>>> BMC SMBUS slave address: 0x42
> > >>>>> Register spacing: 1
> > >>>>>
> > >>>>> Probing SSIF device using SMBIOS... done
> > >>>>> IPMI Version: 1.5
> > >>>>> IPMI locate driver: SMBIOS
> > >>>>> IPMI interface: SSIF
> > >>>>> BMC driver device: /dev/i2c-0
> > >>>>> BMC SMBUS slave address: 0x84
> > >>>>> Register spacing: 1
> > >>>>>
> > >>>>> The probing finds 2 different slave addresses.  I'm not sure why.
> > >>>>> Perhaps you could try setting the driver values manually to see if it
> > >>>>> changes things??
> > >>>>>
> > >>>>> bmc-info -D ssif --disable-auto-probe --driver-address=0x84
> > >>>>> --driver-device=/dev/i2c-0 --register-spacing=1
> > >>>>>
> > >>>>> and also try 0x42 for the driver-address.
> > >>>>>
> > >>>>> Assuming the FreeIPMI ssif driver doesn't have bugs (I don't have a
> > >>>>> machine that uses SSIF, so I've never tried it, I can only assume the
> > >>>>> original writers did it without bugs), it's possible the OpenIPMI 
> > >>>>> kernel
> > >>>>> driver probes for addresses in an alternate order to the way FreeIPMI
> > >>>>> does, and by happen chance gets the right values.  That might explain
> > >>>>> things.
> > >>>>>
> > >>>>> Al
> > >>>>>
> > >>>>> On Wed, 2011-02-09 at 10:14 -0800, Christian Ruppert wrote:
> > >>>>>> Hey guys,
> > >>>>>>
> > >>>>>> take a look at the attachments for "impi-locate" and "bmc-info 
> > >>>>>> --debug"
> > >>>>>> with debug/trace enabled.
> > >>>>>>
> > >>>>>> The kernel driver is only available through OpenIPMI and the patches 
> > >>>>>> are
> > >>>>>> only available up to kernel 2.6.35. I didn't get to the OpenIPMI 
> > >>>>>> kernel
> > >>>>>> drivers yet to port them to .36 and above. So I'd like to use 
> > >>>>>> FreeIPMI
> > >>>>>> instead if it works without any kernel drivers at all.
> > >>>>>>
> > >>>>
> > >>
> > >>
> > 
> > 
-- 
Albert Chu
address@hidden
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory