Board Server Hardware

The board server hardware is hosted by a Xilinx FX12 FPGA, which contains a PowerPC 405 CPU. The FX12 mini module that we are using (left) is sold by Avnet: here is the product page. It is a tiny circuit board comprising a Xilinx Virtex-4 FPGA, SDRAM, Flash and an Ethernet header. The board is capable of running a full Linux 2.6 system on the PowerPC CPU within the FPGA, which is documented elsewhere.

The bit file for this FPGA is currently built using Xilinx ISE/EDK 10.1 service pack 3. Based on past experience, it is quite likely that future versions of ISE/EDK will not be able to generate the hardware correctly due to incompatibilities introduced by upgrades. For this reason, you should maintain a "frozen" ISE/EDK 10.1 installation that will never be upgraded.

The hardware consists mostly of IP cores provided by Xilinx and is generated entirely by EDK. It was originally based on an Avnet reference design for FX12. There are two custom IP cores. Here is the memory map:


Device typeInstance nameBase address High addressNote
RAMDDR_SDRAM_32Mx160x000000000x03ffffff
GPIOBaseBoard_LEDs_3Bit0x814000000x8140ffffNote
Int Cntlrxps_intc_00x818000000x8180ffff
MACTriMode_MAC_GMII0x81c000000x81c0ffffNote
UARTLiteRS2320x840000000x8400ffffNote
Customtp0 (UART 0)0x8f0000000x8f00ffffNote
Customtp1 (UART 1)0x8f0100000x8f01ffff
Customtp2 (UART 2)0x8f0200000x8f02ffff
Customtp3 (UART 3)0x8f0300000x8f03ffff
GPIOJTAG_Host_In0x8f1000000x8f10ffffNote
GPIOJTAG_Host_Out0x8f1100000x8f11ffff
GPIORelay_Out0x8f1200000x8f12ffff
Customconsoleswitch_00x8f2000000x8f20ffffNote
FlashFLASH_2Mx160xff8000000xffbfffffNote
Block RAMxps_bram_if_cntlr_10xfffff0000xffffffffNote

Using EDK

To rebuild the FX12 bit file within EDK, obtain the file fx12-linux-hardware.zip from the download page and unpack it in a suitable location. Open the system.xmp project file using EDK. Then follow this procedure:

  • Select: Project -> Clean All Generated Files.
  • Select: Software -> Software Platform Settings. Ensure that "OS" is standalone. Do not select any of the libraries (xilmfs, xilflash, etc.). Click "OS and Libraries" and ensure that stdout and stdin are both set to RS232. Click Ok.

    This step seems counter intuitive, because the intention is to run Linux on the platform. However, the first program that runs will be the boot loader, which is "standalone" and built into FPGA block RAM.

  • Select: Hardware -> Clean Netlist.

    This step has the side effect of generating the Makefile.

  • Launch an EDK Shell (using the Start Menu group for EDK on Windows, or by sourcing the EDK settings.sh file on Linux). Move to the project directory and run:
            make -f system.make init_bram
    
    This step is also counter intuitive, because the same operation is supposed to be provided by Update Bitstream in the Device Configuration menu. However, ISE/EDK 10.1 has some bugs that make some menu options unreliable. These bugs tend to mask real errors with GUI exceptions. Therefore, it is best to build the project at the command line.
  • When the process completes, use project dir/implementation/download.bit as the bit file.

If you add new devices or change the memory map, you have to make a new version of the Linux kernel. For this, you need to generate a new "fdt.dts" file in EDK. The procedure is as follows:

  • Ensure that you have installed "gen-mhs-devtree" in EDK. This is included in fx12-linux-software.zip from the download page.
  • Select: Software -> Software Platform Settings. Change "OS" to fdt. Click Ok.
  • On the "OS and Libraries" page, change "bootargs" to read "console=ttyUL0 root=/dev/ram0 init=/sbin/init rw".
  • Select: Software -> Generate Libraries and BSPs.
  • When the process completes, search for fdt.dts in the project directory, and copy it to the Linux kernel directory linux-2.6-xlnx/arch/powerpc/boot/dts. You will need this in order to build the Linux kernel.
  • Consider restoring the standalone OS settings (see above) to avoid the risk of building a bit file without a working boot loader.

Baseboard LEDs

The three baseboard LEDs are used to report the status of the system. The codes vary depending on the baseboard being used:

LED State Meaning
Avnet FX12 Prototyping Baseboard In-House Virtex 4 Motherboard
DS1DS2DS3 J261J263J264
off off off on on on Undefined
off off on on off on Boot loader running (waiting 10 seconds before starting kernel)
OR Embedded board server software starting
off on off on on off Boot loader expecting Intel Hex records
OR Embedded board server software has exited abnormally
off on on on off off Linux kernel is booting
on off off off on on User space has been reached - init scripts are running
on off on off off on User space was reached, but either the network update failed, or the Flash memory was not initialised with configuration data
on on off off on off Embedded software is doing something
on on on off off off Embedded software is waiting for a command

MAC

The FX12 FPGA includes two Ethernet MACs which can operate at 10MB/s, 100MB/s and 1000MB/s speeds. (Note: the software driver has only been tested at 100MB/s.) One of these MACs is connected to the RJ45 socket on the FX12 board, and the custom Linux kernel has been built with the appropriate driver. The driver uses "local link FIFO" buffers to store received data and data waiting to be sent. Xilinx doesn't support this configuration any more; it has been deprecated in favour of DMA operation, but a memory corruption bug may exist in the current LLTEMAC DMA code, so FIFO mode is used instead.

Version 1.01.b of the xps_ll_temac IP core is used. Earlier versions have an packet loss bug that is triggered by unrecognisable packets, such as the 802.1d messages generated by some Ethernet switches. The bug discards the packet following the 802.1d packet, causing problems with virtually all network connections.

The IP and MAC address of the Ethernet connection can be set by editing /jffs2/network.

UARTLite

This component is the system console; it is connected to the serial port on the FX12 prototyping board (if available). The serial settings are fixed: 115200 baud, with 8 bits, no parity and 1 stop bit.

When the FX12 bit file is loaded, the boot loader uses this component to display a boot message and receive Intel Hex records if booting from Flash is prevented. As the Linux kernel boots, it writes debugging information to this component. Once Linux has started, a user can log in using a terminal connected to the serial port. The Linux device name for this component is /dev/ttyUL0 (major 207, minor 187).

UARTLite has to be used during initial setup of an FX12 board because an Intel Hex record download is the only way to boot the Linux kernel when the Flash is empty. Once Linux has booted, the Flash can be loaded with the kernel and root file system.

The physical port that is connected to UARTLite on bootup can be exchanged with the physical port of UART 1. This allows both of the physical RS232 ports on the Virtex 4 motherboard to be used as virtual lab UART ports. To activate this functionality, run fx12tool switch within a bootup script, e.g. /jffs2/custom.

Teaports

Four UARTs are used for communication between the FX12 board and virtual lab FPGAs. These UARTs support standard baud rates below 500000 baud, and some other non-standard baud rates. The Linux device names of these components are /dev/ttyS0 through /dev/ttyS3: the major and minor numbers match the Linux standard. The supported baud rates are:


Baud rateBaud rate
300115200
600230400
1200460800
1800500000
2400568181
4800892857*
96001041666*
192001562500*
384002083333*
576003125000*'
6250000*^

* = this baud rate may require a special serial cable.
' = not recommended if more than two FPGA boards are in use simultaneously.
^ = not recommended if more than one FPGA board is in use.

In early versions of the FX12 system, these UARTs were 16550A-compatible devices (instances of the xps_uart16550 component from Xilinx). However, the 16550A design has two significant disadvantages: (1) the FIFO size is a puny 16 bytes, and (2) every byte read from the UART requires two entire words to cross the data bus, creating a total overhead of at least 64 clock cycles per byte. This makes the 16550A unsuitable for operation above 115200 baud; and even at that relatively low speed, data was occasionally lost when more than one UART was active.

To support higher speeds, a custom UART named Teaport was implemented. It has its own page. The design uses a block RAM as a FIFO, giving 1kb send and receive buffers. It also permits data and status bits to be fetched for three bytes simultaneously, speeding up the interrupt service routine by a factor of six. Its disadvantage is a poorer feature set: it only supports "8 bit, no parity, 1 stop bit", cannot use RTS or CTS, and does not send breaks.

JTAG GPIOs

These components are used for programming FPGAs via JTAG. They are standard Xilinx GPIOs; the JTAG-specific features are all implemented in kernel and user software. JTAG_Host_In is a single bit GPIO that is used to sample TDO. JTAG_Host_Out is a three bit GPIO that is used to generate TCK, TMS and TDI. The pin assignments are set in jtag.h in the board server source code:

  • TCK - Least significant bit
  • TMS
  • TDI - Most significant bit

All GPIOs are controlled using Linux character devices. The major and minor device ids assigned by the kernel are:

  • major 10 minor 185 - Baseboard LEDs
  • major 10 minor 186 - Baseboard DIP switches
  • major 10 minor 187 - JTAG_Host_Out
  • major 10 minor 188 - JTAG_Host_In
  • major 10 minor 189 - Relay_Out

JTAG programming can be carried out by using ioctl commands to manipulate these GPIOs. But this is slow, since every operation requires a system call. Therefore, a custom JTAG programming driver exists within the custom Linux kernel.

Console switch

This component allows the console port (UARTLite) to be exchanged with the ports for UART 1. It provides a single register which is 1 on reset, meaning that exchange is disabled. The fx12tool program can change this register to activate the exchange, allowing two FPGA boards to be connected to a single FX12 on a Virtex 4 motherboard. fx12tool can also make the change semi-permanent so that it survives reboots (but not FPGA reprogramming).

Flash

This component must not be confused with the platform Flash which holds the FPGA bit stream. This Flash memory is part of the system memory map and holds two things:

  • A 64kbyte read/write filesystem for configuration data, using the JFFS2 filesystem driver. Configuration data includes the MAC address, the IP address and the board server configuration. The root user can change these files once Linux has booted. The Linux device name is /dev/mtdblock0. A new filesystem can be created by using the following sequence of commands on the embedded system:
        mkdir /tmp/newfs
        cd /tmp/newfs
        touch dummyfile
        mkfs.jffs2 -v -o /tmp/newfs.bin
        flashcp -v /tmp/newfs.bin /dev/mtd0
    
    Having initialised the filesystem in this way, you should mount it (mount /jffs2) then install and edit the system-specific configuration files.
  • A 4032kbyte data area for the Linux kernel and compressed root filesystem. The Linux device name is /dev/mtd2. A zImage.bin file (produced by the build-kernel script) should be loaded into this space by a command like:
        flashcp -v zImage.bin /dev/mtd2
    
    Danger! Don't turn off the power during the flashcp process as this will damage the Flash memory.

    You can use the wget utility to download zImage.bin to an FX12 board from any web server. thttpd is useful here.

Initially, the Flash memory is empty and unformatted. The best way to prepare a new system is to run /install, which makes the the /jffs2 filesystem, downloads zImage.bin, and flashcps it into the Flash memory. The only additional step is editing of the files in /jffs2 to set up the network and configure the VL2 software.

Having loaded a kernel into Flash, you won't need to send the kernel via the serial line again, unless you flashcp a buggy kernel that won't boot.

Boot loader

The boot loader is a short C program. It is not standard Xilinx software; it was written in-house for this project. This is because Xilinx boot loaders don't include functionality to allow programs to be loaded. Xilinx expects you to load programs by putting the PowerPC or Microblaze CPU into debug mode, then use debugging commands to write to memory. This isn't suitable in this environment because it would force you to connect each FX12 board to a PC running EDK for the initial setup. Therefore, like the Linux/Microblaze company Petalogix, we have developed our own boot loader that allows new programs to be loaded. (Petalogix code has not been reused due to licencing restrictions.)

The boot loader uses the UARTLite serial port to communicate with the user. It begins by printing a message. Then it looks for a bootable Linux kernel at memory address 0xff810000. It expects kernels to begin with the machine code:

      48 00 00 05     bl      ff810004 
      7c 13 ea a6     mfccr0  r0

If it sees this code, then it waits for 10 seconds. During this waiting time, you can press Enter to abort the boot process. When the boot process is aborted, the boot loader waits for Intel Hex records, which are executed according to the Intel Hex record specification. If a bootable kernel is not found, then the boot loader begins waiting for Intel Hex records immediately.

The build-kernel script produces both a binary kernel image (zImage.bin) and an Intel Hex version (zImage.hex). Once the boot loader is waiting for Intel Hex records, you should send the latter file. If you are using a Linux system as your terminal, then you can simply:

    cat zImage.hex >/dev/ttyS0

to do this. Otherwise, look for a "send ASCII file" feature in your terminal. When the file is being sent, a rotating - symbol will be displayed. When transmission is complete, the Linux kernel will boot.

Programming the FX12 Platform Flash

The FX12 board includes an XCF08P platform Flash IC that can store up to 8MB of bitfile data, which is loaded onto the FX12 FPGA on power up. Each FX12 bitfile uses about 4.7MB. A compression feature is available; this allows more than one bitfile to be stored on the IC, although it is not clear why this is useful in this particular application. The main benefit of compression is faster updates.

You'll need to program the XCF08P in two situations:

  • You're updating the hardware on a virtual lab board, perhaps to add a feature or fix a bug. In this case, you will have used EDK to generate a new "download.bit" file.
  • You're adding a new virtual lab board. In this case, you'll be reusing an existing "download.bit" file.

You won't need to program the XCF08P if you're just testing a new bitfile. You can program the FPGA directly in that case.

To program the XCF08P, plug the FX12 board into the Avnet/Memec prototyping board, then connect the prototyping board to a PC via a JTAG connector. Then set up the Xilinx IMPACT software on the PC. Version 8.1 is usable with service packs; later versions presumably also work. (Because IMPACT is required for many aspects of virtual lab maintenance, it is worth keeping a PC specifically for IMPACT.)

In IMPACT, select "Create a new project", then "Prepare a PROM file". On the next page, you want to "target a Xilinx PROM". Give the PROM file a suitable name and location. Select "Serial mode" if you get the option. On the next page, select "xcf08p" and click Add. Then click Finish. Now you add the "download.bit" file (produced by EDK) to the PROM. This is the "device file". IMPACT should tell you that the startup clock has been changed to "CCLK" - this is normal. You only have one device file to add, so don't attempt to add any more if IMPACT asks.

You should now see that "xcf08p" is about 56% full. Click "Generate File" (on the left). Once "PROM File Generation Succeeded", you have created three PROM programming files ending in .mcs, .cfi, and .sig. Click "Boundary Scan" (on the left) and then select "File -> Initialize Chain". Assign your new .mcs file to the XCF08P device, and click Bypass for the FPGA. Now click Program and ensure that "Erase before programming" is selected before pressing Ok. When programming is complete, power cycle the board. If you did it right, the bootloader message will be written to the prototyping board's serial port (at 115200 bps, 8N1).

At this point, you might want to download the Linux kernel and root filesystem to the separate software Flash memory.

Wiring for the Prototypes

Avnet FX12 Prototyping Baseboard In-House Virtex 4 Motherboard
The figure below shows the wiring arrangements for the FX12 board in the prototype VL2 system. This setup exposes four UARTs and the JTAG interface. This wiring arrangement does not necessarily match that of any finalised version of VL2. The figure below shows the wiring arrangements for the FX12 board on the Virtex 4 motherboard made in-house. This setup exposes one UART, the JTAG interface, and two relay outputs. The JTAG header has the pins shown below.