Skip to content

Instantly share code, notes, and snippets.

@ganawaj
Last active November 19, 2019 03:09
Show Gist options
  • Select an option

  • Save ganawaj/dcfc4c6bea942d952dc2bdd11497d655 to your computer and use it in GitHub Desktop.

Select an option

Save ganawaj/dcfc4c6bea942d952dc2bdd11497d655 to your computer and use it in GitHub Desktop.
sudo add-apt-repository universe
sudo apt update
sudo apt install vim libncurses5 p7zip-full

After rebooting and using the live Linux image, take note of the current SAS address used by the card. We need to rely on MegaCli for this, which is available also for Linux:

Download firmware We are going to be using this card with FreeNAS 9.3.1 which requires v20 IT firmware to be installed. To download the correct file, head over to Broadcom’s product pages and select your model of HBA card, in my case the 9211 which is a 6Gb/s model. EDIT: Here’s a direct link to make life a bit easier. https://docs.broadcom.com/docs/12350520 We need 2118it.bin

wget https://docs.broadcom.com/docs-and-downloads/raid-controllers/raid-controllers-common-files/8-07-14_MegaCLI.zip
unzip 8-07-14_MegaCLI.zip
7z x Linux/MegaCli-8.07.14-1.noarch.rpm
7z x MegaCli-8.07.14-1.noarch.cpio
chmod 755 opt/MegaRAID/MegaCli/MegaCli64
opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -a0 | grep SAS\ Address

The SAS address is a 16-byte hexadecimal number that will be restored afterwards. Technically, unless there are multiple HBAs in a single system, all with the same SAS address, it may be left blank (which makes it default to a known value).

Next, the following software should be fetched and built:

wget https://github.com/exactassembly/meta-xa-stm/raw/master/recipes-support/lsiutil/files/lsiutil-1.72.tar.gz
tar xzf lsiutil-1.72.tar.gz
make -C lsiutil -f Makefile_Linux
git clone https://github.com/marcan/lsirec
make -C lsirec

Before running the programs just built, it is required to unload the kernel module currently driving the HBA:

rmmod megaraid_sas

If drives are attached to the server via the HBA, this will fail. Hugepages should now be enabled. These will be used to load the IT mode firmware from the host to the card:

echo 16 > /proc/sys/vm/nr_hugepages

The card now has to be put in reset mode. For this step, it is required to know its full PCI address. You may use lspci to identify the card:

lspci -nnv | grep H310 -B1
01:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID SAS 2008 [Falcon] [1000:0073] (rev 03)
 Subsystem: Dell PERC H310 Mini Monolithics [1028:1f51]
 
lsirec/lsirec 0000:01:00.0 unbind
lsirec/lsirec 0000:01:00.0 halt

At this point, the SBR should be extracted from the card. The SBR is a 256-byte blob stored onto a serial EEPROM which contains several pieces of information, most notably how the card should identify itself.

lsirec/lsirec 0000:01:00.0 readsbr H310MM.sbr

When flashing to IT mode using the regular utilities, the SBR is erased (typically blanking with the 'megarec.exe -writesbr 0 sbrempty.bin' or the 'sas2flsh. exe -o -e 7' command). This is the step that bricks the card, as the server uses the PCI Subsystem Vendor and Product ID loaded from the SBR to check if the card is allowed or not in the dedicated storage slot. Additionally, the SBR includes checksums, edits should update them as needed or, as we'll do, a tool can be used:

python3 lsirec/sbrtool.py parse H310MM.sbr H310MM.cfg 
vim H310MM.cfg 

The H310MM.cfg file should have at least two entries modified, PCIPID should be set to 0x0072 and Interface should be set to 0x00 (these two values signal that the card is running in IT/IR mode rather than in megaRAID mode).

The modified file should be turned back into a binary blob and written to the card's SBR:

python3 lsirec/sbrtool.py build H310MM.cfg mod_H310MM.sbr

I've chosen to use the freshly dumped H310MM.cfg as the base for the new SBR, as I've preferred to be minimal in the changes performed, given I do not know the meaning of the other fields. It is however possible (and I have tested the result is working) to use the lsirec/sample_sbr /sbr_dell_h200e_itir.cfg as the base, modifying the SubsysVID and SubsysPID fields to 0x1028 and 0x1f51 respectively. As long as the card presents the correct SubsysVID and SubsysPID, it will not brick upon reboot. To get the appropriate driver (for megaRAID or IT/IR mode) bound to the card, the PCIPID and Interface fields have to be set to the values described above.

lsirec/lsirec 0000:01:00.0 writesbr mod_H310MM.sbr
Device in MPT mode
Using I2C address 0x54
Using EEPROM type 1
Writing SBR...
SBR written from mod_H310MM.sbr

You can check that the card's PCI ID has been updated via the following command:

lspci -vns 0000:01:00.0 -A intel-conf1 | head -n2

The kernel however hasn't noticed this and will be updated later. Now, you'll want to get the card out from reset mode and instruct it to load the IT mode firmware loaded from the host (rather than the one stored internally)

lsirec/lsirec 0000:01:00.0 hostboot 2118it.bin

At this point we can let the kernel know the card has changed, as it is able to operate in IT mode properly as if it had been flashed:

lsirec/lsirec 0000:01:00.0 rescan

As the kernel rescans the card, it will also bind the mpt3sas driver to it. This is fine as the next steps, involving actually flashing the firmware to the card to make IT mode persistent, make use of the /dev/mptctl device created by the driver.

lsiutil/lsiutil -e
Unk00 = 0x0022f661
Unk04 = 0xb34f2000
Unk08 = 0x91d700f8
PCIVID = 0x1000
PCIPID = 0x0072
Unk10 = 0x0000
HwConfig = 0x0104
SubsysVID = 0x1028
SubsysPID = 0x1f51
Unk18 = 0x00000000
Unk1c = 0x00000000
Unk20 = 0x00000000
Unk24 = 0x00000000
Unk28 = 0x00000000
Unk2c = 0x00000000
Unk30 = 0x40000000
Unk34 = 0x0300c650
Unk38 = 0x00000000
Unk3c = 0x00000000
Interface = 0x00
Unk41 = 0x2c
Unk42 = 0x0000
Unk44 = 0x00000000
Unk48 = 0x0000
Unk4a = 0x00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment