Setting custom SMBIOS data in Xen DomUs

December 4th, 2015

The length of this post is directly related to the number of hours I’ve wasted working this all out so you don’t have to. You have been warned…

SMBIOS data contains all sorts of useful information about the system, such as manufacturer name, product, serial number and memory configuration. When using Xen PVHVM virtualisation there are default value that are set to tell the domU some information about the host. In Linux the tool to read this data is dmidecode. For example,

root@domu:/# dmidecode
# dmidecode 2.12
SMBIOS 2.4 present.
12 structures occupying 342 bytes.
Table at 0xFC001000.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
	Vendor: Xen
	Version: 4.4.1
	Release Date: 06/11/2015
	Address: 0xE8000
	Runtime Size: 96 kB
	ROM Size: 64 kB
	Characteristics:
		PCI is supported
		EDD is supported
		Targeted content distribution is supported
	BIOS Revision: 4.4

Handle 0x0100, DMI type 1, 27 bytes
System Information
	Manufacturer: Xen
	Product Name: HVM domU
	Version: 4.4.1
	Serial Number: 7b84005b-456e-40f1-b896-804a4a91bd6e
	UUID: 7B84005B-456E-40F1-B896-804A4A91BD6E
	Wake-up Type: Power Switch
	SKU Number: Not Specified
	Family: Not Specified
...

It would be nice to be able to update some of this information to pass data from the host to the domU. In this case hopefully to provide information to iPXE for netbooting the host, but could be for other reasons, such as finding out which physical host the domU is running on (though lldpd works really well for that).

But finding out how to set this data has proved hard. There seems to be no good documentation about it on the Internet anywhere. Having spent far too long working out how to get this to actually work, the following explanation might help.

SMBIOS is presented as a set of tables of data. Table type 0 contains BIOS information; type 1, system information; type 2, motherboard data, and many more. These tables are specified in a document, the System Management BIOS (SMBIOS) Reference Specification.

Don’t use QEMU options

When Xen is running in PVHVM mode it emulates the BIOS and hardware using QEMU. The first search result was to find that QEMU has a command-line option, “-smbios”, which can set SMBIOS data. Using this option it is possible to set a file to load, or specific values for type 0 or type 1 data. For example

-smbios type=1,manufacturer=Myself

So the problem then becomes how to pass these arguments into Xen for the domU. A search eventually found the “device_model_args” option for the domU configuration, so the following would go in the config file:

device_model_args = [ '-smbios', 'type=1,manufacturer=Myself' ]

This passes the options to QEMU as intended, as can be seen by running ps | grep qemu on the host, but unfortunately doesn’t have any effect in the guest. It turns out that SMBIOS data to Xen domU guests is provided not by QEMU, but by the domU loader, hvmloader.

Method 1: smbios_firmware

So there are currently at least two ways to set the SMBIOS data with hvmloader. The first is to use the smbios_firmware option in the xl.cfg file. That is documented in the man page, but is rather confusing as it doesn’t explain in detail what that file should exactly contain.

The file contains one or more SMBIOS tables, each preceded by a 32bit length field. The length is necessary because the code that reads this file has to skip over the different tables that may be present and there is no length in the actual data, so without a full parser this would be hard.

But what data to use for the file? It is possible to read the specification and put together some data that way, but the easiest is to take the data from a running system and modify it. Thankfully, dmidecode can output the raw data that can be edited. Let’s look at the type 1 (system information) table:

root@domu:/# dmidecode -t1 -u
# dmidecode 2.12
SMBIOS 2.4 present.

Handle 0x0100, DMI type 1, 27 bytes
        Header and Data:
                01 1B 00 01 01 02 03 04 09 94 42 3A 7B 0E 4D 0D
                80 EE BE 99 D8 10 75 68 06 00 00
        Strings:
                58 58 58 00
                "Xen"
                48 58 58 20 64 6F 6D 55 00
                "HVM domU"
                34 2E 34 2E 31 00
                "4.4.1"
                30 39 39 34 34 32 33 61 2D 37 62 30 65 2D 34 64
                30 64 2D 38 30 65 65 2D 62 65 39 39 64 38 31 30
                37 35 36 38 00
                "0994423a-7b0e-4d0d-80ee-be99d8107568"

The format (see the spec) is a short 4-octet header followed by the data, followed by strings. Each string is numbered from 01, and referred to by number in the data section. In the above the header is ’01 1B 00 01′ (table type 1, length 1B, handle 00 01), then for example the first three values are strings, ’01 02 03′, which refer to the NULL-terminated strings following the data section, namely in this case, “Xen”, “HVM domU” and “4.4.1”.

So we can easily edit the binary values of the strings here. Let’s update the manufacturer from “Xen” to “James”:

# dmidecode 2.12
SMBIOS 2.4 present.

Handle 0x0100, DMI type 1, 27 bytes
        Header and Data:
                01 1B 00 01 01 02 03 04 09 94 42 3A 7B 0E 4D 0D
                80 EE BE 99 D8 10 75 68 06 00 00
        Strings:
                4A 61 6D 65 73 00
                "Xen (this string doesn't matter)"
                48 58 58 20 64 6F 6D 55 00
                "HVM domU"
                34 2E 34 2E 31 00
                "4.4.1"
                30 39 39 34 34 32 33 61 2D 37 62 30 65 2D 34 64
                30 64 2D 38 30 65 65 2D 62 65 39 39 64 38 31 30
                37 35 36 38 00
                "0994423a-7b0e-4d0d-80ee-be99d8107568"

Then we need to convert this to a binary file for the smbios_firmware option. Assuming we sent the output of dmidecode to “smi.txt” and have edited that, the following will do this quite nicely:

$ awk '/^\t\t[^"]/' smi.txt | xxd -r -ps > smbios.bin

However, using this with Xen won’t work quite yet. Remember the length required at the start? It also turns out that the SMBIOS spec calls for a second terminating NULL after the strings that dmidecode doesn’t show us, so let’s add these. Start with the terminating NULL:

# dmidecode 2.12
SMBIOS 2.4 present.

Handle 0x0100, DMI type 1, 27 bytes
        Header and Data:
                01 1B 00 01 01 02 03 04 09 94 42 3A 7B 0E 4D 0D
                80 EE BE 99 D8 10 75 68 06 00 00
        Strings:
                4A 61 6D 65 73 00
                "Xen"
                48 58 58 20 64 6F 6D 55 00
                "HVM domU"
                34 2E 34 2E 31 00
                "4.4.1"
                30 39 39 34 34 32 33 61 2D 37 62 30 65 2D 34 64
                30 64 2D 38 30 65 65 2D 62 65 39 39 64 38 31 30
                37 35 36 38 00
                "0994423a-7b0e-4d0d-80ee-be99d8107568"

                00

(remember to indent the new line with two TABs, not spaces, or add the 00 on the end of the last string). We can then convert this to a binary file again with the above command:

$ awk '/^\t\t[^"]/' smi.txt | xxd -r -ps > smbios.bin
$ ls -l smbios.bin
-rw-r--r-- 1 user user 86 Dec  4 22:24 smbios.bin
$

In this case we ended up with a 86 byte file. This is the length of the actual table. This is 56 in hex, and we’re using a little-endian x86 system, so update the text file once again to add the length to the start (again, two tabs, because awk regex):

# dmidecode 2.12
SMBIOS 2.4 present.
                56 00 00 00

Handle 0x0100, DMI type 1, 27 bytes
        Header and Data:
                01 1B 00 01 01 02 03 04 09 94 42 3A 7B 0E 4D 0D
...

Finally, boot up the domU and run dmidecode, and the manufacturer should now be “James”! Multiple tables can be included in the binary file – each should follow the above format (length then SMBIOS tables data), so the easiest is probably to create each table file separately as above, then concatenate them for the smbios_firmware file.

Method 2: xenstore data

The second way of setting this data is slightly easier, but I have yet to find out a way to set the data in the domU configuration file so the domain has to be created paused, the data set, and then resumed.

Proceed as follows. Start the domain up as usual, but paused:

# xl create -p domu.cfg

Now get the domain id for the new domU:

# xl list domu
Name             ID   Mem VCPUs      State   Time(s)
domu             20  1024     2     -b----     311.3
#

In this case we find that our new domU has domain ID ’20’.

We can now set SMBIOS options in the paused domain:

# xenstore-write "/local/domain/20/bios-strings/system-manufacturer" "James"

and unpause the domain:

# xl unpause domu

When the system has booted up, dmidecode should now show the new system manufacturer as “James” rather than “Xen”. Note that once the domU has booted then changing these bios-strings options will have no effect. They have to be set before hvmloader creates the SMBIOS tables at boot time.

A complete lists of possible options that can be set is found in the source header files. It would just be nice to be able to set these in the domu.cfg file rather than having to set the in the paused domain. But at least this method can easily be scripted, so it’s not too much of a problem.

So, there we go. Two ways to set SMBIOS data for Xen domUs.

Tested with Xen 4.4 on Debian 8.0 (Jessie). With thanks to Terry Burton for letting me know that this might be possible, but not indicating how long it would take…

First experiences of the ZyXEL VMG1312-B10A

October 6th, 2015

A technical moan, I’m afraid.

Since upgrading from 3/0.4Mbps ADSL to 40/10Mbps FTTC Internet today, I’ve had to use a new router to replace my very aged ZyXEL Prestige 660R which has, aside from no recent software updates, been pretty reliable.

So I’ve had a new ZyXEL VMG1312-B10A running on the line for about 24 hours, and first impressions are not good. Not that it’s not working as a VDSL modem, just that it’s not really working as much else.

There are enough things wrong that it’s worth listing them somewhere so I can remind myself how painful this thing is.

I should comment that I’ve managed to get it working in bridge mode enough that the line is usable, and I’m getting better speeds through it than with the previous ADSL line, which is one good thing!

Software version is 1.00(AAJZ.5)C0. There’s apparently a newer version, but the release notes don’t indicate that any of the below have been fixed, and given its current state I’m not sure I want to risk flashing it.

Wireless

An unexpected feature as I only needed a router, but turns out it won’t do WPA-Enterprise, so totally useless. Even 5 year old Netgear boxes at the same price can do that.

The User’s Guide is really helpful in giving information on what RADIUS, 802.1X (though they use a lowercase x) and EAP are, even though the product doesn’t do them.

Static routes

RIP works, which is nice. It even plants routes in the routing table. The router can ping out to IP addresses on those routes, and the destination responds.

But the firewall on the router drops any packets that aren’t on the immediate subnet. “Disabling” the firewall actually leaves it running and the broken rules in place.

Workaround is that, as a bonus, the iptables command works at the CLI (undocumented), so the following will remove the broken rules:


> iptables -F LAN_ONLY_FORWARD
> iptables -F LAN_ONLY_INPUT

sadly, they come back at boot time. So this thing is only barely usable as a “router”.

HTTP server

The web interface is awful, obviously designed to try and be as simple as possible, but including completely unnecessary javascript which slows it down to a crawl. What’s more, it’s now decided to totally break so I can’t use it at all (TCP packets come back from it at a snails pace and the browser eventually times out). Then it got stuck at high CPU. Thankfully again, top is available to show the horrible mess:


Mem: 57188K used, 1696K free, 0K shrd, 0K buff, 18736K cached
CPU: 0.0% usr 50.0% sys 0.0% nic 50.0% idle 0.0% io 0.0% irq 0.0% sirq
Load average: 3.94 3.72 3.76 3/73 7641
PID PPID USER STAT VSZ %MEM CPU %CPU COMMAND
7624 257 supervis R 10604 17.9 1 45.7 httpd -m 0 -s
7640 7638 supervis R 2148 3.6 1 4.1 top
260 257 supervis S 8272 14.0 0 0.0 ssk
7603 7582 supervis S 8060 13.6 1 0.0 sshd -m 0
...

but, unthankfully, kill isn’t.

Looking at the syslog data isn’t very reassuring either:


> syslog dump brcm
...
2015-10-06T22:56:10 (none) daemon.warn System: BadVA : 00000000
2015-10-06T22:56:10 (none) daemon.warn System: PrId : 0002a080 (Broadcom4350)
2015-10-06T22:56:10 (none) daemon.warn System: httpd/3318: potentially unexpected fatal signal 11.
2015-10-06T22:56:10 (none) daemon.warn System: Cpu 1
2015-10-06T22:56:10 (none) daemon.warn System: $ 0 : 00000000 00000001 004fbc1f 2ab50014
2015-10-06T22:56:10 (none) daemon.warn System: $ 4 : 004fbc1e 00000000 00000000 00000000
2015-10-06T22:56:10 (none) daemon.warn System: $ 8 : 07ab9b5b 00000001 00000297 00000000
...

(repeated many times) So it looks like httpd is stuck in a segfault/restart loop eating up CPU resources and power.

The web server was usable before switching into bridge mode, so I’m suspecting that has something to do with it, but I can’t yet work out how to disable bridge mode from the CLI…

…which leads me neatly on to:

CLI

help helpfully lists all the available commands. Unhelpfully, several of them don’t work


> help
...
interfaceGroup
udpechod
tr69c
webstyle
radvdconf
...
> tr69c
sshd:error:95.474:processInput:544:unrecognized command tr69c
>

I have no idea what the tr69c command is there to do, and it looks like I’m not going to find out either. TR-69 looks like the protocol used by the ISP to configure the router, so I probably want this off anyway, but no chance now that the web server is broken.

What’s more, there are extra commands not listed in the help, such as iptables, top and ps. But being the really useful thing it is, sh and/or bash are apparently missing.

This thing runs Linux. Just give us a shell or write a decent CLI for it; don’t leave us with a few working commands, a few Linux commands to save writing replacements, and the rest just broken.

Shell!

Ah! A way to get shell access of sorts. The echo command works, and takes backtick expansion:


> echo hi
hi
> echo `whoami`
supervisor
> echo `ls /`
bin data dev etc home lib linuxrc log mnt opt proc sbin sys tmp usr var vmlinux.lz webs
>

cat also works, which makes things a little easier:


> cat /proc/version
Linux version 2.6.30 (root@CjLai2Ubuntu) (gcc version 4.4.2 (Buildroot 2010.02-git) ) #1 SMP PREEMPT Wed Mar 25 15:37:10 CST 2015
>

You can even pipe into commands:


> ps | ls /
bin etc linuxrc opt sys var
data home log proc tmp vmlinux.lz
dev lib mnt sbin usr webs
>

So we can now get shell access with:


> cat | sh
whoami
supervisor

Root shell, not bad. Why not make it easier?

[update: login as user “supervisor” rather than “admin”, and you can then type sh at the prompt to get a shell!]

Segfault

Trying out a few commands, you don’t expect to hit this in production code.


> pppoectl
Aiee, segfault! You should probably report this as a bug to the developer
Connection to a.b.c.d closed.
$

Auto-reconnect

So I happily got my new DSL line working and headed off into work. An hour later, the router dropped the line. When I get home I look at its log and it just reports that the Internet link went down. Then it sat there for the rest of the day without trying to reconnect. This might be a first time fluke, but I need a router that will keep trying to connect as fast as its little CPU can cope, not that gives up at the first try.

User guide

Extensive, but gives no information about the CLI at all that I can see. Maybe they were too embarrassed to mention it…

Summary

So far my only advice is to turn off everything and put it into bridge mode, run pppoe on an internal server to terminate the connection and forget about the router.

Come on ZyXEL, you can do better than this! I know, because I’ve been using a P660R for the last 10 years.

Finding the winbind privileged pipe

March 19th, 2015

When connecting FreeRADIUS or ntlm_auth to winbind, you may need to know where winbind’s privileged pipe is so that the file permissions are set correctly. This is sometimes obvious, but may not be (e.g. if you’ve compiled Samba yourself and installed it in a non-standard location).

The following hack (which works on winbind 3.6 to 4.2 at least) will query winbind and ask it where the privileged pipe is.

    $ perl -e 'print "\x30\x08\0\0\x2f".("\0"x2091)'|socat - UNIX-CONNECT:/tmp/.winbindd/pipe|strings|grep '/'
    /var/run/samba/winbindd_privileged
    $

Then you just need to know where the unprivileged pipe is, which is probably not /tmp/.winbindd/pipe if you installed in a non-standard location. :-)

Note this hack is only semi-serious…

FreeRADIUS Active Directory authentication performance issues

March 11th, 2015

This is really just a post for me to keep track of sites that have got or had performance issues with FreeRADIUS when authenticating to Active Directory via ntlm_auth and Samba.

The issue crops up on the FreeRADIUS mailing list with an increasing frequency, so trying to collate posts to keep track of what people have tried as a solution.

Note the vast majority of sites are universities. I believe this is because there is a unique problem that does not occur in business: once an hour, many students and staff all walk around the site at the same time, authenticating to different wireless access points. This causes massive increase in concurrent authentications.

September 2014 – Georgia Tech

October 2013 – Imperial College, Loughborough University, Bristol University

October 2013 – University of the West of England, Georgia Tech

October 2013

September 2014 – Inverse, Inc.

October 2014 – University of York

October 2014

December 2014

January 2015 – McMaster University

Some of my posts on the issue:

January 2015

February 2015

March 2015

Looking forward to a break

December 22nd, 2014

I’ve been thinking about the things I’ve been up to over the last month or so and really starting to wonder why I’m not called “Jack”. If I did them as professions I would have been all of the below. Not surprisingly, I’m pretty sure I would also qualify for the title ‘Master of None’.

  • Accountant
  • Artist – clay modeller
  • Computer programmer
  • Furniture maker / woodworker
  • Knitter (er, yes…)
  • Motorcycle mechanic
  • Network engineer
  • Pianist
  • Photographer
  • Plumber
  • Project manager
  • Systems administrator
  • Teacher
  • Upholsterer
  • Videographer
  • Wireless architect

It shouldn’t come as a surprise that I’m looking forward to the Christmas holiday! Just one more day at work to go.