Tutorial: Controlling SpiNNaker machines¶
SpiNNaker machines consist of a network of SpiNNaker chips and, in larger systems, a set of Board Management Processors (BMPs) which control and monitor systems’ power and temperature. SpiNNaker (and BMPs) are controlled using SCP packets (a protocol built on top of SDP) sent over the network to a machine. Rig includes a set of high-level wrappers around the low-level SCP commands which are tailored towards SpiNNaker application developers.
Note
Rig does not aim to provide a complete Python implementation of the full (low-level) SCP command set. Users who encounter missing functionality as a result of this are encouraged to submit a patch or open an issue as the developers are open to (reasonable) suggestions!
In addition to these high-level interfaces, Rig includes a lower-level interface for sending and receiving application-defined SDP and SCP packets to running applications via a socket.
The two high-level machine control interfaces are:
MachineController
- Interact with and control SpiNNaker chips, e.g. boot, load applications, read/write memory.
BMPController
- Interact with and control BMPs, e.g. control power-supplies, monitor system temperature, read/write FPGA registers. Only applicable to machines based on SpiNN-5 boards.
The low-level SDP and SCP interfaces are:
A tutorial for each of these interfaces is presented below.
MachineController
¶
To get started, let’s instantiate a MachineController
. This is as
simple as giving the hostname or IP address of the machine:
>>> from rig.machine_control import MachineController
>>> mc = MachineController("spinnaker_hostname")
Note
If you’re using a multi-board machine, give the hostname of the (0, 0) chip. Support for connecting to multiple Ethernet ports of a SpiNNaker machine is not currently available but should be automatic.
Booting¶
You can boot()
the system like so:
>>> mc.boot()
True
If the machine could not be booted for any reason a
rig.machine_control.machine_controller.SpiNNakerBootError
will be
raised. If no exception is raised, the machine is booted and ready to use. The
return value of boot()
indicates whether the
machine was actually booted (True
), or if it was already booted and thus
nothing was done (False
), most applications may consider the boot to be a
success either way.
If you’re using a SpiNN-2 or SpiNN-3 board booted without arguments, only LED 0
will be usable. To enable the other LEDs, instead boot the machine using one of
the pre-defined boot option dictionaries in rig.machine_control.boot
,
for example:
>>> from rig.machine_control.boot import spin3_boot_options
>>> mc.boot(**spin3_boot_options)
True
Probing for Available Resources¶
The get_system_info()
method returns a
SystemInfo
object
describing which chips, links and cores are alive and also the SDRAM
available:
>>> system_info = mc.get_system_info()
This object can also be used to guide Rig’s place and route utilities (see
rig.place_and_route.place_and_route_wrapper
,
rig.place_and_route.utils.build_machine
and
rig.place_and_route.utils.build_core_constraints
).
Loading Applications¶
The load_application()
method will,
unsurprisingly, load an application onto an arbitrary set of SpiNNaker cores.
For example, the following code loads the specified APLX file to cores 1, 2 and
3 of chip (0, 0) and cores 10 and 11 of chip (0, 1):
>>> targets = {(0, 0): set([1, 2, 3]),
... (0, 1): set([10, 11])}
>>> mc.load_application("/path/to/app.aplx", targets)
Alternatively, this method accepts dictionaries mapping applications to
targets, such as those produced by
rig.place_and_route.place_and_route_wrapper
.
load_application()
verifies that all applications
have been successfully loaded (re-attempting a small number of times if
necessary). If not all applications could be loaded, a
SpiNNakerLoadingError
exception is raised.
Many applications require the sync0 signal to be sent to start the
application’s event handler after loading. We can wait for all cores to reach
the sync0 barrier using
wait_for_cores_to_reach_state
and then send the
sync0 signal using send_signal
:
>>> # In the example above we loaded 5 cores so we expect 5 cores to reach
>>> # sync0.
>>> mc.wait_for_cores_to_reach_state("sync0", 5)
5
>>> mc.send_signal("sync0")
Similarly, after application execution, the application can be killed with:
>>> mc.send_signal("stop")
Since the stop signal also cleans up allocated resources in a SpiNNaker machine
(e.g. stray processes, routing entries and allocated SDRAM), it is desirable
for this signal to reliably get sent even if something crashes in the host
application. To facilitate this, you can use the
application()
context manager:
>>> with mc.application():
... # Main application code goes here, e.g. loading applications,
... # routing tables and SDRAM.
>>> # When the above block exits (even if due to an exception), the stop
>>> # signal will be sent to the application.
Note
Many application-oriented methods accept an app_id argument which is given
a sensible default value. If the MachineController.application()
context manager is given an app ID as its argument, this app ID will become
the default app_id within the with block. See the section on context
managers below for more details.
Loading Routing Tables¶
Routing table entries can be loaded using
load_routing_tables()
like so:
>>> routing_tables = {
... (0, 0): [RoutingTableEntry(...), ...],
... (0, 1): [RoutingTableEntry(...), ...],
... ...
... }
>>> mc.load_routing_tables(routing_tables)
This command allocates and then loads the requested routing table entries onto
each of the supplied chips. The supplied data structure matches that produced by
rig.place_and_route.place_and_route_wrapper()
.
Allocating/Writing/Reading SDRAM¶
Many SpiNNaker applications require the writing and reading of large blocks of
SDRAM data. The recommended way of doing this is to allocate blocks of SDRAM
using sdram_alloc()
with an identifying ‘tag’. The
The SpiNNaker application can later use this tag number to look up the address
of the allocated block of SDRAM. Not only does this avoid the need to
explicitly communicate SDRAM locations to the application it also allows SARK
to safely allocate memory in the SDRAM.
read()
and write()
methods are provided which can read and write arbitrarily large blocks of data
to and from memory in SpiNNaker:
>>> # Allocate 1024 bytes of SDRAM with tag '3' on chip (0, 0)
>>> block_addr = mc.sdram_alloc(1024, 3, 0, 0)
>>> mc.write(block_addr, b"Hello, world!")
>>> mc.read(block_addr, 13)
b"Hello, world!"
Rig also provides a file-like I/O wrapper
(MemoryIO
) which may prove
easier to integrate into applications and also ensures reads and writes are
constrained to the allocated region.
>>> # Allocate 1024 bytes of SDRAM with tag '3' on chip (0, 0)
>>> block = mc.sdram_alloc_as_filelike(1024, 3, 0, 0)
>>> block.write(b"Hello, world!")
>>> block.seek(0)
>>> block.read(13)
b"Hello, world!"
File-like views of memory can also be sliced to allow a single allocation to be safely divided between different parts of the application:
>>> hello = block[0:5]
>>> hello.read()
b"Hello"
The sdram_alloc_for_vertices()
utility
function is provided to allocate multiple SDRAM blocks simultaneously. This
will be especially useful if you’re using Rig’s place and route
tools, since the utility accepts the place-and-route tools’
output format. For example:
>>> placements, allocations, application_map, routing_tables = \
... rig.place_and_route.wrapper(...)
>>> from rig.machine_control.utils import sdram_alloc_for_vertices
>>> vertex_memory = sdram_alloc_for_vertices(mc, placements, allocations)
>>> # The returned dictionary maps from vertex to file-like wrappers
>>> vertex_memory[vertex].write(b"Hello, world!")
Context Managers¶
Many methods of MachineController
require arguments such as x,
y, p or app_id which can quickly lead to repetitive and messy code. To
reduce the repetition Python’s with
statement can be used:
>>> # Within the block, all commands will affect chip (1, 2)
>>> with mc(x = 1, y = 2):
... block_addr = mc.sdram_alloc(1024, 3)
... mc.write(block_addr, b"Hello, world!")
BMPController
¶
A limited set of utilities are provided for interacting with SpiNNaker BMPs
which are contained in the BMPController
class. In systems with
either a single SpiNN-5 board or a single frame of SpiNN-5 boards which are
connected via a backplane, the class can be constructed like so:
>>> from rig.machine_control import BMPController
>>> bc = BMPController("bmp_hostname")
For larger systems which contain many frames of SpiNNaker boards, at least one IP address or hostname must be specified for each:
>>> bc = BMPController({
... # At least one hostname per rack is required
... (0, 0): "cabinet0_frame0_hostname",
... (0, 1): "cabinet0_frame1_hostname",
... ...
... (1, 0): "cabinet1_frame0_hostname",
... (1, 1): "cabinet1_frame1_hostname",
... ...
... # Individual boards can be given their own unique hostname if
... # required which overrides those above
... (1, 1, 0): "cabinet1_frame1_board0_hostname",
... })
Boards are referred to by their (cabinet, frame, board) coordinates:
2 1 0
Cabinet --+-------------+----------------+
| | |
+-------------+ +-------------+ +-------------+ Frame
| | | | | | |
| +---------+ | | +---------+ | | +---------+ | |
| | : : : : | | | | : : : : | | | | : : : : |--------+ 0
| | : : : : | | | | : : : : | | | | : : : : | | |
| +---------+ | | +---------+ | | +---------+ | |
| | : : : : | | | | : : : : | | | | : : : : |--------+ 1
| | : : : : | | | | : : : : | | | | : : : : | | |
| +---------+ | | +---------+ | | +---------+ | |
| | : : : : | | | | : : : : | | | | : : : : |--------+ 2
| | : : : : | | | | : : : : | | | | : : : : | | |
| +---------+ | | +---------+ | | +---------+ | |
| | : : : : | | | | : : : : | | | | : : : : |--------+ 3
| | : : : : | | | | : : : : | | | | : : : : | |
| +---------+ | | +|-|-|-|-|+ | | +---------+ |
| | | | | | | | | | |
+-------------+ +--|-|-|-|-|--+ +-------------+
| | | | |
Board -----+-+-+-+-+
4 3 2 1 0
Power Control¶
Boards can be powered on using set_power()
:
>>> # Power off board (0, 0, 0)
>>> bc.set_power(False)
>>> # Power on board (1, 2, 3)
>>> bc.set_power(True, 1, 2, 3)
>>> # Power on all 24 boards in frame (1, 2)
>>> bc.set_power(True, 1, 2, range(24))
Note
Though multiple boards in a single frame can be powered on simultaneously, boards in different frames must be powered on separately.
Note
By default the set_power()
method adds a delay
after the power on command has completed to allow time for the SpiNNaker
cores to complete their self tests. If powering on many frames of boards,
the post_power_on_delay argument can be used to reduce or eliminate this
delay.
Reading Board Temperatures¶
Various information about a board’s temperature and power supplies can be read
using read_adc()
(ADC = Analogue-to-Digital Converter)
which returns a bmp_controller.ADCInfo
named tuple containing many
useful values:
>>> adc_info = bc.read_adc() # Get info for board (0, 0, 0)
>>> adc_info.temp_top # Celsius
23.125
>>> adc_info.fan_0 # RPM (or None if not attached)
2401
Context Managers¶
As with MachineController
, BMPController
supports the
with
syntax for specifying common arguments to a series of commands:
>>> with bc(cabinet=1, frame=2, board=3):
... if bc.read_adc().temp_top > 75.0:
... bc.set_led(7, True) # Turn on LED 7 on the board
Sending/receiving SDP and SCP packets to/from applications¶
A number of low-level facilities are provided for users who wish to send and receive SCP and SDP packets directly. The most common use for these APIs is to send and receive SDP packets to and from a running SpiNNaker application to allow realtime monitoring and communication with the underlying application via an IP Tag. A minimal example of each is presented below.
Example: Sending SDP packets to a running application¶
In your SpiNNaker application you should register a callback handler for the
arrival of SDP packets. For example, using the spin1_api
:
spin1_callback_on(SDP_PACKET_RX, on_sdp_from_host, 0);
To send SDP packets to this application, you must open a UDP socket with which
to send SDP packets to your SpiNNaker system. Note that (slightly confusingly)
SpiNNaker listens for incoming SDP packets on the SCP port
.
>>> import socket
>>> from rig.machine_control.consts import SCP_PORT
>>> out_sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>>> out_sock.connect((hostname, SCP_PORT))
With the port opened, you can use the
rig.machine_control.packets.SDPPacket
and
rig.machine_control.packets.SCPPacket
classes to pack your data
into properly formatted SDP or SCP packets. Since sark
and spin1_api
(unfortunately) make packing/unpacking SDP packets rather clumsy it is common
to use SCP packets.
Note
SCP packets are just SDP packets with some additional fields placed in the SDP data payload. When a port number other than 0 is used SCP packets are passed to the application like any other SDP packet
As an example, to send an SCP packet core 1 on chip (0, 0) with a cmd_rc
of
123
:
>>> from rig.machine_control.packets import SCPPacket
>>> data = b"Hello world!\0"
>>> packet = SCPPacket(
... dest_port=1,
... dest_x=0, dest_y=0, dest_cpu=1,
... cmd_rc=123
... data=data
... )
>>> out_sock.send(packet.bytestring)
On the receiving core the on_sdp_from_host
callback might then look like
this:
void on_sdp_from_host(uint mailbox, uint port)
{
sdp_msg_t *msg = (sdp_msg_t *)mailbox;
if (msg->cmd_rc == 123)
{
io_printf(IO_BUF,
"Got SCP packet from host with data: %s\n",
msg->data);
}
spin1_msg_free(msg);
}
Note
SpiNNaker can only receive packets up to a certain size. This size can be
determined using MachineController
’s
scp_data_length()
property
This property defines the maximum length of the data-field in an SCP packet
sent to the machine.
Example: Receiving SDP packets from a running application¶
To receive SDP packets from an application there must first be an open socket ready to receive the packets. For example:
>>> import socket
>>> PORT = 50007
>>> in_sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>>> in_sock.bind(("", PORT))
Next, you must set up an ‘IP tag’ on every Ethernet-connected SpiNNaker chip through which SDP packets may be sent back to the host which informs SpiNNaker of the IP address these packets should be sent to.
A list of the Ethernet-connected chips in a typical SpiNNaker machine can be
produced using
rig.machine_control.MachineController.get_system_info
and an IP tag
configured on each using
rig.machine_control.MachineController.iptag_set
like so:
>>> from rig.machine_control import MachineController
>>> # Get the IP and port of the socket we opened
>>> addr, port = in_sock.getsockname()
>>> # Set-up IP Tag 1 on each ethernet-connected chip to forward all SDP
>>> # packets to this socket.
>>> mc = MachineController("spinnaker-machine-hostname")
>>> si = mc.get_system_info()
>>> for (x, y), chip_ip in si.ethernet_connected_chips():
... mc.iptag_set(1, addr, port, x, y)
You can now listen for incoming packets and unpack them using
rig.machine_control.packets.SDPPacket.from_bytestring()
and
rig.machine_control.packets.SCPPacket.from_bytestring()
. For example,
to unpack SCP packets received from the machine:
>>> from rig.machine_control.packets import SCPPacket
>>> while True:
... data = self.in_sock.recv(512)
... if not data:
... break
... packet = SCPPacket.from_bytestring(data)
... print("Got SCP packet from core {packet.src_cpu} "
... "of chip ({packet.src_x}, {packet.src_y}) "
... "with cmd_rc {packet.cmd_rc} and data "
... "{packet.data}.".format(packet=packet))
Note
We use a 512 byte UDP receive buffer since at present the largest SDP
packet supported by the machine at the time of writing is 256 bytes + 24
bytes SCP header. Using power-of-two sized receive buffers is recommended
on most operating systems for performance reasons. The
MachineController
’s
scp_data_length()
property
can be used to get the actual value.
SCP packets might be sent from a SpiNNaker application using code such as:
sdp_msg_t msg;
void send_scp_packet(const char *data)
{
// Send to the nearest Ethernet-connected chip.
msg.tag = 1;
msg.dest_port = PORT_ETH;
msg.dest_addr = sv->eth_addr;
// Indicate the packet's origin as this chip/core. Note that the core is
// indicated in the bottom 5 bits of the srce_port field.
msg.flags = 0x07;
msg.srce_port = spin1_get_core_id();
msg.srce_addr = spin1_get_chip_id();
// Copy the supplied data into the data field of the packet and update
// the length accordingly.
int len = strlen(data) + 1; // Include the null-terminating byte
spin1_memcpy(msg.data, (void *)data, len);
msg.length = sizeof (sdp_hdr_t) + sizeof (cmd_hdr_t) + len;
// and send it with a 100ms timeout
spin1_send_sdp_msg(&msg, 100);
}