guest - flak

OpenBSD on a Sun T5120

I’ve been looking for a sparc64 system for a while and noticed the Sun Enterprise T5120 models have become very affordable. They’re interesting machines and great for testing due to the built in virtualization support.

T5120

There are two console ports on the back, serial and net. If you’re lucky, the network port will come in a usable configuration; otherwise you’ll need a “Cisco” RJ45 serial cable. I was lucky.

Plug in network and power, wait a bit, watch dhcpd logs for a new request to come in. There it is. ssh root@10.10.10.10 and use the default passwrod, changeme. This lands at the ilom prompt, which is a little weird, but not too ridiculous. Let’s get the rest of this thing fired up. I recommend donning noise canceling headphones at this point. If the CS24 is a hairdryer, the T5120 is a vacuum cleaner.

start /SYS start /SP/console

This should eventually (wait for it, wait for it) land at the classic ok prompt. I installed OpenBSD by booting a CD. It is theoretically possible to netboot, but if you haven’t done so with a Sun before, I’d steer clear. I started down that path, but gave up somewhere after rarpd and /etc/ethers but before rpc.bootparamd and /etc/bootparams. pxeboot4life.

T2

The T2 CPU is hyperhyperthreaded. OpenBSD doesn’t yet scale all that well to 64 CPUs, so you are well advised to chop this thing up into manageable pieces. Each core consists of 8 threads, but has two integer ALUs, so dividing into units of 8 seems reasonable.

Perhaps my understanding is flawed, but in practice this doesn’t work as well as theorized. I haven’t been able to map out exactly how ALUs are divided among ldoms, but it’s not as straightforward as I’d like to believe. Running CPU benchmarks in parallel in various ldoms led to some inexplicable results. Play around with it.

ldomize

The man page for ldomctl is unfortunately rather barren; this is based on Mark’s undeadly article.

First, don’t forget to enable ldomd in rc.conf. Start things off by copying the factory config.

mkdir default cd default ldomctl dump cd .. cp -r default myconfig cd myconfig

Create some disks. Instead of using a second disk for the installer, I overwrite the front of the main disk instead.

dd if=/dev/zero of=vdisk1 bs=1m count=8000 dd if=miniroot55.fs of=vdisk1 bs=64k conv=notrunc

Now we create an ldom.conf. I created an explicit primary domain. This would normally be created automatically using whatever resources are left, but if you’re unlucky, ilom won’t like generated config and you’ll end up booting the factory-default again. I was unlucky.

domain primary { vcpu 8 memory 8G } domain puffy { vcpu 8 memory 4G vdisk "/home/tedu/vms/v1/vdisk1" vnet } domain huffy { vcpu 8 memory 4G vdisk "/home/tedu/vms/v1/vdisk2" vnet }

Install and reboot.

ldomctl init-system ldom.conf cd .. ldcomctl download myconfig halt (back at ok prompt) #. (back at ilom prompt) reset /sys

To later switch configs in ilom:

set /HOST/bootmode config=configname

You can finish installing OpenBSD in the ldoms by running cu -l ttyV0 (ttyV1, etc.). In order to get networking working, we probably want to bridge the ldoms together.

ifconfig vnet0 -inet6 up ifconfig vnet1 -inet6 up ifconfig bridge0 up add vnet0 add vnet1

At this point it’s possible to bridge the interfaces to a physical em device, but I did something a little different. My primary ldom has IP address 10.10.10.76. I’m going to left shift that and create a 10.10.76.0 network inside the T5120. All the little ldoms are plugged into our bridge (switch), but we need to create another virtual interface to plug the primary domain in.

ifconfig vether0 up 10.10.76.1 ifconfig bridge0 add vether0

Then I started dhcpd listening on vether0, and configured puffy and huffy to use dhcp.

All in all, the ldom support is pretty solid, thanks to Mark Kettenis. I did all this without ever touching Solaris, or even installing any pkgs.

dmesg

From before carving into ldoms.

console is /virtual-devices@100/console@1 Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. Copyright (c) 1995-2014 OpenBSD. All rights reserved. http://www.OpenBSD.org OpenBSD 5.5 (GENERIC.MP) #173: Tue Mar 4 14:47:47 MST 2014 deraadt@sparc64.openbsd.org:/usr/src/sys/arch/sparc64/compile/GENERIC.MP real mem = 34225520640 (32640MB) avail mem = 33670987776 (32111MB) mainbus0 at root: SPARC Enterprise T5120 cpu0 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu1 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu2 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu3 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu4 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu5 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu6 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu7 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu8 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu9 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu10 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu11 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu12 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu13 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu14 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu15 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu16 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu17 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu18 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu19 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu20 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu21 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu22 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu23 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu24 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu25 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu26 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu27 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu28 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu29 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu30 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu31 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu32 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu33 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu34 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu35 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu36 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu37 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu38 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu39 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu40 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu41 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu42 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu43 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu44 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu45 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu46 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu47 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu48 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu49 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu50 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu51 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu52 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu53 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu54 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu55 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu56 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu57 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu58 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu59 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu60 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu61 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu62 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz cpu63 at mainbus0: SUNW,UltraSPARC-T2 (rev 0.0) @ 1165.379 MHz vbus0 at mainbus0 "flashprom" at vbus0 not configured "tpm" at vbus0 not configured cbus0 at vbus0 vldc0 at cbus0 vldcp0 at vldc0 chan 0x14: ivec 0x228, 0x229 channel "spds" "sunvts" at vldc0 chan 0x6 not configured "sunmc" at vldc0 chan 0x7 not configured "explorer" at vldc0 chan 0x8 not configured "led" at vldc0 chan 0x9 not configured "flashupdate" at vldc0 chan 0xa not configured "ipmi" at vldc0 chan 0xc not configured "system-management" at vldc0 chan 0xd not configured vldc1 at cbus0 "spfma" at vldc1 chan 0x5 not configured vldc2 at cbus0 vldcp1 at vldc2 chan 0x0: ivec 0x200, 0x201 channel "hvctl" "ldom-primary" at vldc2 chan 0x1 not configured "fmactl" at vldc2 chan 0x3 not configured vldc3 at cbus0 "ldmfma" at vldc3 chan 0x4 not configured "n2cp" at vbus0 not configured "ncp" at vbus0 not configured vrng0 at vbus0 vcons0 at vbus0: ivec 0x111, console vrtc0 at vbus0 "niu" at mainbus0 not configured vpci0 at mainbus0: bus 2 to 18, dvma map 80000000-ffffffff pci0 at vpci0 ppb0 at pci0 dev 0 function 0 "PLX PEX 8533" rev 0xaa pci1 at ppb0 bus 3 ppb1 at pci1 dev 1 function 0 "PLX PEX 8533" rev 0xaa pci2 at ppb1 bus 4 ppb2 at pci2 dev 0 function 0 "PLX PEX 8517" rev 0xad pci3 at ppb2 bus 5 ppb3 at pci3 dev 1 function 0 "PLX PEX 8517" rev 0xad pci4 at ppb3 bus 6 ppb4 at pci4 dev 0 function 0 vendor "PLX", unknown product 0x8111 rev 0x21 pci5 at ppb4 bus 7 ohci0 at pci5 dev 0 function 0 "NEC USB" rev 0x43: ivec 0x16, version 1.0 ohci1 at pci5 dev 0 function 1 "NEC USB" rev 0x43: ivec 0x17, version 1.0 ehci0 at pci5 dev 0 function 2 "NEC USB" rev 0x04: ivec 0x14 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "NEC EHCI root hub" rev 2.00/1.00 addr 1 usb1 at ohci0: USB revision 1.0 uhub1 at usb1 "NEC OHCI root hub" rev 1.00/1.00 addr 1 usb2 at ohci1: USB revision 1.0 uhub2 at usb2 "NEC OHCI root hub" rev 1.00/1.00 addr 1 ppb5 at pci3 dev 2 function 0 "PLX PEX 8517" rev 0xad pci6 at ppb5 bus 8 em0 at pci6 dev 0 function 0 "Intel 82571EB" rev 0x06: ivec 0x17, address 00:14:4f:97:cd:a4 em1 at pci6 dev 0 function 1 "Intel 82571EB" rev 0x06: ivec 0x14, address 00:14:4f:97:cd:a5 ppb6 at pci3 dev 3 function 0 "PLX PEX 8517" rev 0xad pci7 at ppb6 bus 9 em2 at pci7 dev 0 function 0 "Intel 82571EB" rev 0x06: ivec 0x14, address 00:14:4f:97:cd:a6 em3 at pci7 dev 0 function 1 "Intel 82571EB" rev 0x06: ivec 0x15, address 00:14:4f:97:cd:a7 ppb7 at pci1 dev 2 function 0 "PLX PEX 8533" rev 0xaa pci8 at ppb7 bus 10 mpi0 at pci8 dev 0 function 0 "Symbios Logic SAS1068E" rev 0x02: msi scsibus0 at mpi0: 112 targets sd0 at scsibus0 targ 0 lun 0: <SEAGATE, ST914602SSUN146G, 0400> SCSI3 0/direct fixed naa.5000c5000615a07f sd0: 140009MB, 512 bytes/sector, 286739329 sectors sd1 at scsibus0 targ 1 lun 0: <ATA, OCZ-AGILITY2, 1.25> SCSI3 0/direct fixed naa.5e83a97f7a590cc2 sd1: 85857MB, 512 bytes/sector, 175836528 sectors ppb8 at pci1 dev 8 function 0 "PLX PEX 8533" rev 0xaa pci9 at ppb8 bus 11 ppb9 at pci9 dev 0 function 0 "PLX PEX 8533" rev 0xaa pci10 at ppb9 bus 12 ppb10 at pci10 dev 1 function 0 "PLX PEX 8533" rev 0xaa: msi pci11 at ppb10 bus 13 ppb11 at pci10 dev 2 function 0 "PLX PEX 8533" rev 0xaa: msi pci12 at ppb11 bus 14 ppb12 at pci10 dev 8 function 0 "PLX PEX 8533" rev 0xaa: msi pci13 at ppb12 bus 15 ppb13 at pci10 dev 9 function 0 "PLX PEX 8533" rev 0xaa: msi pci14 at ppb13 bus 16 ppb14 at pci10 dev 10 function 0 "PLX PEX 8533" rev 0xaa: msi pci15 at ppb14 bus 17 ppb15 at pci1 dev 9 function 0 "PLX PEX 8533" rev 0xaa: msi pci16 at ppb15 bus 18 "pci-performance-counters" at mainbus0 not configured ebus0 at mainbus0 com0 at ebus0 addr ca0000-ca0007 ivec 0x13: ns16550a, 16 byte fifo umass0 at uhub0 port 2 configuration 1 interface 0 "Texas Instruments Incorporated 6250 FW: 03.03.02.22.562AD423" rev 2.00/2.00 addr 2 umass0: using SCSI over Bulk-Only scsibus1 at umass0: 2 targets, initiator 0 cd0 at scsibus1 targ 1 lun 0: <TSSTcorp, CD/DVDW TS-T632A, SR03> SCSI0 5/cdrom removable serial.0451625fACCC3275122F uhub3 at uhub0 port 4 "Cypress Semiconductor USB2 Hub" rev 2.00/0.0b addr 3 vscsi0 at root scsibus2 at vscsi0: 256 targets softraid0 at root scsibus3 at softraid0: 256 targets bootpath: /pci@0,0/pci@0,0/pci@2,0/scsi@0,0/disk@0,0 root on sd0a (ccfe4bf26855e98c.a) swap on sd0b dump on sd0b
Posted 2014-03-18 17:00:20 by tedu Updated: 2015-01-08 18:05:37
Tagged: computers openbsd