Translate

Image of XSLT 2.0 and XPath 2.0 Programmer's Reference (Programmer to Programmer)
Image of Beginning Google Maps API 3
Image of Operating System Concepts
Image of Linux Kernel Development (3rd Edition)

Utilities for Parsing Intel Microcode

A recent issue on a Linux platform with an Intel CPU prompted me to check to see if there was a microcode patch available from Intel to fix the issue. This blog post provides the source code for some of the Python utilities I wrote to assist me in determining if a microcode update was available for my particular issue or not.

Intel distributes microcode updates in the form of a text file consisting of groups of big endian 32-bit integers represented as hexadecimals. As an example, here is a portion of one such file:

/*  Fri Nov 4 16:09:13 CST 2016  */
/*  m02f0a15.inc  */
0x00000001,	0x00000015,	0x08212002,	0x00000f0a,
0x63e49a0c,	0x00000001,	0x00000002,	0x00000000,
0x00000000,	0x00000000,	0x00000000,	0x00000000,
0xffffffe9,	0x80e0e61a,	0x7cb2c607,	0xbe1e2fc9,
0xacaf1526,	0x31514e28,	0x9252d426,	0xb999ebf8,
0x8469bb8a,	0x95e72fd2,	0x5f7c5472,	0x254adcf0,
0xf706b236,	0x21ef3efb,	0x82d05526,	0x8b10bd77,
0x268ab8bd,	0x929739a5,	0x0f180e02,	0x3ad1cc5a,
0x23a10814,	0xbc4257f8,	0x026ded8d,	0x84bf45be,
0xe13e69f3,	0xbb0edf16,	0x85218fd2,	0x898af4e1,
0xcd635f26,	0x846ab0c6,	0x85a5d5cd,	0x077b4a16,
0x71514de0,	0xbff893a6,	0x47536ab0,	0x1fc4b546,
0x906f4b85,	0xc1f997d5,	0x0ba4b594,	0xa823eb50,


The first 16 integers (48 bytes) represent a standard microcode header which is fully documented in Section 9.11 (Microcode Update Facilities) of the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A: System Programming Guide, Part 1.

Here is Table 9-6 from that section:


This table also details an extended header (> 48 bytes, starting with the field named Update Data) which does not form part of this blog post.

Here is a simple Python script which converts the downloaded microcode.dat ASCII file and converts it into a little endian binary file suitable for use on a Linux system which supports loading microcode updates.

#!/usr/bin/python3
#
#   Author: Finnbarr P. Murphy
#  License: BSD
#     Date: November 2016
#    Usage: scriptname microcode.dat microcode.bin
#


import sys
import binascii

def convert2int(sval) :
    sval = sval.replace('\t','').replace('0x','').replace(' ','')
    return int.from_bytes(binascii.unhexlify(sval), 'little')


if len(sys.argv) != 3 :
    print("ERROR: An input and output filename is required")
    exit(1)

try :
    infile = open(sys.argv[1], "r")
except IOError :
    print("ERROR: Failed to open input file")
    exit(1)

try :
    outfile = open(sys.argv[2], "wb")
except IOError :
    print("ERROR: Failed to open output file")
    infile.close()
    exit(2)
 
for line in infile : 
    if line[0] == '/' :
        continue
    hvals = line.split(',')
    for hval in hvals :
        if hval == '\r\n' or hval == '\n' or hval == '\r' :
            continue
        outfile.write(bytes.fromhex('%0.8X' % convert2int(hval)))

infile.close()
outfile.close()

exit(0)


Note that this script does not break the individual microcode updates into individual files as some tools such as iucode_tool do. This functionality could easily be added if desired.

The following script takes the downloaded file and outputs a line of information for each microcode header found in the file. Note that the script only parses the standard microcode header and not any extended headers. Functionality to parse any extended headers found in the file could easily be added if desired but I did not include such code as I had no need for this information.

#!/usr/bin/python3
#
#   Author: Finnbarr P. Murphy
#  License: BSD
#     Date: November 2016
#    Usage: scriptname < microcode.dat 
#


import sys
import binascii
import re


def CpuFamily(sig) :
    fam = (sig >> 8) & 0xF 
    if fam == 0xF :
       fam += (sig >> 20) & 0xFF
    return fam


def CpuModel(sig) :
    fam = CpuFamily(sig)
    model = (sig >> 4) & 0xF
    if fam >= 0x6 :
        model += ((sig >> 16) & 0xF) << 4
    return model
 

def CpuStepping(sig) :
    return sig & 0xF


def convert2int(sval) :
    sval = sval.replace('\t','').replace('0x','').replace(' ','')
    # return int.from_bytes(binascii.unhexlify(sval), 'little')
    return int(sval, 16)

 
ints = 0
outline = ""
total = 0
matched = 0
datestr = "[A-Z][a-z]{2}\ [A-Z][a-z]{2}\ \d{1,2}\ \d{2}:\d{2}:\d{2}\ [A-Z]{3}\ \d{4}"


for line in sys.stdin:
    if line[0] == '/' :
        if line[1] != '*' :
            continue
        line = line.replace('\n','').replace('\r','').replace('/*','').replace('*/','')
        if not matched :
            match = re.search(datestr, line)
            if match is not None :
                matched = 1
                print("Release Date: {0}".format(line.strip()))
                print("\nDate          Sig F/M/S   Flags       Datasize      Name\n")
                continue
        if outline :
            print("{0}    {1}".format(outline, name))
            total += 1
            outline = ""
        name = line.upper().replace('.INC','')
        ints = 0
        continue
    hvals = line.split(',')
    for hval in hvals:
        if hval == '\r\n' or hval == '\r' or hval == '\n' :
            continue
        if ints == 2 :      # date
            outline += "{0:s}-{1:s}-{2:s}".format(hval[7:], hval[3:5], hval[5:7])
        elif ints == 3 :    # signature
            sig = convert2int(hval)
            fms = "     {0:d}/{1:d}/{2:d}".format(CpuFamily(sig), CpuModel(sig), CpuStepping(sig))
            outline += fms.ljust(12) 
        elif ints == 6 :    # processor flags
            procflags = convert2int(hval)
            outline += "    {0:08b}".format(procflags)
        elif ints == 7 :    # data size
            datasize = convert2int(hval)
            if datasize == 0 :
               datasize = 2000
            outline += "    0x{0:06X}".format(datasize)
        ints += 1

print("\nTotal: {0}".format(total))

exit(0)


Note that the above script does not elegantly handle any errors it encounters when invoked. I leave it up to you the reader to modify the script to make it more robust.

Here is the output generated by the above script when run against the latest Intel Linux microcode (November 4th, 2016) download:

Release Date: Fri Nov 4 16:09:13 CST 2016

Date          Sig F/M/S   Flags       Datasize      Name

2002-08-21     15/0/10    00000010    0x0007D0      M02F0A15  
2004-08-26     15/2/5     00010000    0x0007D0      M10F252C  
1999-06-28     6/5/3      00000001    0x0007D0      MU165310  
2001-02-20     6/11/1     00100000    0x0007D0      MU16B11D  
2003-06-04     15/2/7     00001000    0x0007D0      M08F2739  
1999-05-05     6/6/10     00001000    0x0007D0      MU166A0D  
2002-07-16     15/0/10    00000100    0x0007D0      M04F0A14  
2000-03-06     6/10/1     00000100    0x0007D0      MU26A101  
1999-05-25     6/5/0      00000010    0x0007D0      MU165041  
2005-04-21     15/4/9     10111101    0x0007D0      MBDF4903  
1999-09-21     6/8/1      00001000    0x0007D0      MU16810F  
1998-06-10     6/3/2      00000000    0x0007D0      MU163202  
2000-05-05     6/8/6      00000001    0x0007D0      MU168607  
2004-11-09     6/9/5      00100000    0x0007D0      M2069507  
1999-05-05     6/6/13     00100000    0x0007D0      MU166D07  
1999-05-25     6/5/0      00001000    0x0007D0      MU165045  
2004-08-11     15/2/9     00001000    0x0007D0      M08F292F  
2004-10-17     6/13/6     00100000    0x0007D0      M206D618  
2002-01-10     6/11/4     00010000    0x0007D0      MU16B401  
2005-04-21     15/3/3     00001101    0x0007D0      M0DF330C  
2000-01-10     6/10/0     00000100    0x0007D0      MU26A003  
1999-05-20     6/5/3      00000100    0x0007D0      MU26530B  
2000-05-04     6/8/6      00000010    0x0007D0      MU16860A  
2006-07-14     15/6/8     00100010    0x0007D0      M22F6809  
1999-05-05     6/6/5      00010000    0x0007D0      MU166503  
2004-05-11     15/3/2     00001101    0x0007D0      M0DF320A  
2004-11-09     6/9/5      00010000    0x0007D0      M1069507  
2001-02-15     6/11/1     00010000    0x0007D0      MU16B11C  
2000-05-04     6/8/6      00000100    0x0007D0      MU268602  
2004-08-11     15/2/5     00000010    0x0007D0      M02F252A  
2003-05-02     15/1/2     00000100    0x0007D0      M04F122E  
1999-09-21     6/8/1      00000100    0x0007D0      MU268110  
1999-09-21     6/8/1      00100000    0x0007D0      MU16810E  
2004-08-11     15/2/9     00000100    0x0007D0      M04F292E  
1999-05-25     6/5/0      00000001    0x0007D0      MU165040  
2000-12-07     6/8/10     10000000    0x0007D0      MU168A05  
2004-08-11     15/2/9     00000010    0x0007D0      M02F292D  
1999-05-05     6/6/10     00000010    0x0007D0      MU166A0C  
2005-12-14     15/4/10    01011100    0x0007D0      M5CF4A04  
2002-01-11     6/11/4     00100000    0x0007D0      MU16B402  
1999-05-18     6/5/3      00000010    0x0007D0      MU16530C  
1999-10-15     6/8/3      00001000    0x0007D0      MU168308  
1999-03-12     6/6/13     00001000    0x0007D0      MU166D06  
2003-06-05     15/2/4     00000010    0x0007D0      M02F241F  
2003-06-04     15/2/7     00000100    0x0007D0      M04F2737  
1999-09-10     6/7/3      00000100    0x0007D0      MU26732E  
1999-03-12     6/6/13     00000010    0x0007D0      MU166D05  
2000-11-02     6/8/10     00010000    0x0007D0      MU168A01  
1999-09-21     6/8/1      00000001    0x0007D0      MU16810D  
1999-05-12     6/5/2      00000100    0x0007D0      MU26522B  
2000-12-07     6/8/10     00100000    0x0007D0      MU168A04  
2004-11-09     6/9/5      10000000    0x0007D0      M8069547  
2000-05-04     6/8/6      10000000    0x0007D0      MU16860C  
2003-06-04     15/2/7     00000010    0x0007D0      M02F2738  
2005-04-21     15/4/3     10011101    0x0007D0      M9DF4305  
1999-05-12     6/5/2      00000001    0x0007D0      MU16522A  
2004-08-05     15/2/6     00000010    0x0007D0      M02F2610  
2003-06-10     15/2/4     00010000    0x0007D0      M10F2421  
2000-11-15     15/0/7     00000010    0x0007D0      2F0708  
1999-05-17     6/5/2      00000010    0x0007D0      MU16522C  
1999-05-18     6/5/3      00001000    0x0007D0      MU16530D  
2002-07-16     15/0/10    00000001    0x0007D0      M01F0A13  
1999-05-05     6/6/10     00100000    0x0007D0      MU166A0B  
1999-10-15     6/8/3      00100000    0x0007D0      MU168307  
1998-08-11     6/7/1      00000100    0x0007D0      MU267114  
1999-09-21     6/8/1      00010000    0x0007D0      MU168111  
2005-06-10     15/4/10    01011101    0x0007D0      M5DF4A02  
2002-07-16     15/0/7     00000001    0x0007D0      M01F0712  
2003-06-05     15/2/4     00000100    0x0007D0      M04F241E  
1999-09-22     6/7/2      00000100    0x0007D0      MU267238  
1999-05-05     6/6/0      00000001    0x0007D0      MU16600A  
1999-05-25     6/5/1      00000001    0x0007D0      MU165140  
2000-05-05     6/8/6      00010000    0x0007D0      MU168608  
2004-08-11     15/2/5     00000001    0x0007D0      M01F2529  
2004-08-11     15/2/5     00000100    0x0007D0      M04F252B  
2010-10-03     6/15/11    00100000    0x000FD0      M206FBBA  
2005-04-22     15/4/1     10111101    0x0013D0      MBDF4117  
2010-10-03     6/15/11    00001000    0x000FD0      M086FBBB  
2010-09-30     6/15/6     00000001    0x000FD0      M16F6D0  
2009-08-25     6/28/10    00000100    0x0013D0      M04106CA107  
2009-08-25     6/28/10    00000001    0x0013D0      M01106CA107  
2010-09-29     6/23/7     00010000    0x000FD0      M101067770A  
2010-10-03     6/15/11    10000000    0x000FD0      M806FBBA  
2005-04-21     15/4/4     10011101    0x000BD0      M9DF4406  
2005-04-21     15/4/7     10011101    0x000BD0      M9DF4703  
2005-06-30     15/4/8     01011111    0x000BD0      M5FF4807  
2005-12-15     15/6/4     00000001    0x000BD0      M01F6402  
2010-10-04     6/22/1     10000000    0x000FD0      M801066144  
2010-09-29     6/23/6     00000001    0x000FD0      M011067660F  
2010-10-01     6/15/6     00000100    0x000FD0      M46F6D2  
2010-09-29     6/23/6     10000000    0x000FD0      M801067660F  
2005-12-15     15/6/2     00000100    0x000BD0      M04F620F  
2010-10-02     6/15/13    00100000    0x000FD0      M206FDA4  
2010-10-03     6/15/11    01000000    0x000FD0      M406FBBC  
2010-10-02     6/15/13    00000001    0x000FD0      M16FDA4  
2009-04-10     6/28/2     00000001    0x0013D0      M01106C2217  
2010-10-03     6/15/11    00010000    0x000FD0      M106FBBA  
2005-12-23     15/6/4     00110100    0x000BD0      M34F6404  
2010-09-29     6/23/6     00010000    0x000FD0      M101067660F  
2009-04-10     6/28/2     00000100    0x0013D0      M04106C2218  
2009-08-25     6/28/10    00010000    0x0013D0      M10106CA107  
2009-04-10     6/28/2     00001000    0x0013D0      M08106C2219  
2010-10-02     6/15/13    10000000    0x000FD0      M806FDA4  
2005-04-21     15/4/1     00000010    0x0013D0      M02F4116  
2005-11-15     6/14/8     00100000    0x000FD0      M206E839  
2010-10-03     6/15/11    00000001    0x000FD0      M016FBBA  
2006-05-01     6/14/12    00100000    0x000FD0      M206EC54  
2010-10-03     6/15/11    00000100    0x000FD0      M046FBBC  
2009-08-25     6/28/10    00001000    0x0013D0      M08106CA107  
2009-10-23     6/38/1     00000001    0x0013D0      M0120661104  
2010-10-04     6/22/1     00000010    0x000FD0      M021066142  
2010-10-02     6/15/2     00000001    0x000FD0      M16F25D  
2006-04-26     15/6/5     00000001    0x0007D0      M01F6508  
2010-10-01     6/15/6     00100000    0x000FD0      M206F6D1  
2006-05-08     15/4/8     00000001    0x000BD0      M01F480C  
2005-04-21     15/3/4     00011101    0x001BD0      M1DF3417  
2010-10-02     6/15/7     00010000    0x000FD0      M106F76A  
2006-09-12     6/14/12    10000000    0x000FD0      M806EC59  
2010-10-02     6/15/2     00100000    0x000FD0      M206F25C  
2010-10-02     6/15/7     01000000    0x000FD0      M406F76B  
2008-01-15     15/4/8     00000010    0x000BD0      M02F480E  
2011-07-18     6/38/1     00000010    0x0013D0      M0220661105_CV  
2010-10-02     6/15/10    10000000    0x000FD0      M806FA95  
2010-10-04     6/22/1     00000001    0x000FD0      M011066143  
2013-08-20     6/30/5     00010011    0x001BD0      M13106E5_00000007  
2014-05-29     6/62/4     11101101    0x0033D0      MED306E4_00000428  
2010-09-28     6/23/10    10100000    0x001FD0      MA01067AA0B  
2013-06-26     6/37/2     00010010    0x001FD0      M1220652_0000000E  
2013-06-21     6/26/5     00000011    0x0027D0      M03106A5_00000019  
2010-09-28     6/23/10    00010001    0x001FD0      M111067AA0B  
2013-06-17     6/45/7     01101101    0x0043D0      M6D206D7_00000710  
2013-06-28     6/37/5     10010010    0x000BD0      M9220655_00000004  
2013-06-21     6/26/4     00000011    0x0037D0      M03106A4_00000012  
2013-06-19     6/62/6     11101101    0x002BD0      MED306E6_00000600  
2012-05-22     6/45/6     01101101    0x003FD0      M6D206D6_00000619  
2013-06-12     6/42/7     00010010    0x0027D0      M12206A7_00000029  
2014-05-29     6/62/7     11101101    0x003BD0      MED306E7_0000070D  
2015-02-26     6/58/9     00010010    0x002FD0      M12306A9_0000001C  
2016-10-07     6/79/1     11101111    0x0063D0      MEF406F1_0B00001F  
2016-06-22     6/78/3     11000000    0x017BD0      MC0406E3_0000009D_0000009E  
2016-04-29     6/61/4     11000000    0x0043D0      MC0306D4_00000024  
2016-04-01     6/69/1     01110010    0x004FD0      M7240651_0000001F  
2016-04-01     6/70/1     00110010    0x005FD0      M3240661_00000016  
2016-04-29     6/71/1     00100010    0x002BD0      M2240671_00000016  
2015-12-12     6/86/2     00010000    0x006FD0      M1050662_0000000F  
2016-06-07     6/63/4     10000000    0x003BD0      M80306F4_0000000D  
2016-06-22     6/94/3     00110110    0x017BD0      M36506E3_0000009D_0000009E  
2016-03-16     6/60/3     00110010    0x0057D0      M32306C3_00000020  
2016-10-07     6/63/2     01101111    0x007FD0      M6F306F2_00000039  
2013-06-18     6/47/2     00000101    0x0033D0      M05206F2_00000037  
2010-09-28     6/23/10    01000100    0x001FD0      M441067AA0B  
2010-09-29     6/23/6     01000000    0x000FD0      M401067660F  
2010-09-30     6/29/1     00001000    0x000FD0      M08106D129  
2010-09-29     6/23/6     00000100    0x000FD0      M041067660F  
2016-06-02     6/86/4     00010000    0x0053D0      M1050664_0F00000A  

Total: 155


The first column is the individual microcode release date, the second column is the processor family, model and stepping which were extracted from the Processor Signature field, the third column represents processor type bits extracted from the Processor Flag field, the next column is the size of the microcode code in bytes, and the final column is the identification string used by Intel.

Well, that is all for now. The source code I have provided above should be enough to enable you to develop your own Python tool(s) to slice and dice the Intel microcode file in whatever manner you wish. Both scripts ware developed using Python 3.5 on Windows 10 using the Subsystem for Linux; thus there is no guarantee that the code will work without modification on Python 2.7.

Enjoy the holiday season and Merry Christmas to all my readers.

Leave a Reply