Performance

The performance of IPbus is dependent on many variables, such as:

  • Network topology/latency

  • Quality of networking components (NIC, switches, etc) and related drivers

  • Mode of operation (ipbusudp-2.0, chtcp-2.0, etc)

  • Interrupt coalescence settings of your NIC (see the Performance tweaks with ethtool section below)

  • CPU type/speed and throttle state

The performance via the ControlHub has been measured using very similar network layout and hardware to that planned for final deployment in the CMS experiment. These performance measurements are described in the TWEPP-14 IPbus presentation, but some notable results are summarised below (for chtcp-2.0 communication - i.e. via ControlHub - on 1Gbit/s network infrastructure):

  • For one software client controlling one device, the single-word read/write latency is approximately 250 micro-seconds

    • Although this single-word latency is larger than VME/PCIe-based control, for multiple transactions or large block transfers this is compensated by concatenating multiple (up to order of 100) transactions into each packet, and by having multiple packets in flight around the system at any given time

    • Hence, optimal performance will be obtained if network dispatches are only performed when necessary.

  • The 1-client-to-1-device block read/write throughput for payloads larger than 1 Mbyte is above 0.5 Gbit/s

  • The total block read/write throughput is above 0.75 Gbit/s for three or more boards in a single MicroTCA crate.

Given the many variables affecting IPbus performance, your experience will almost certainly vary from the values given above. However if you experience substantially worse (more that a factor 2) performance, we might be able to assist - please submit a support ticket.

Performance tweaks with ethtool

IPbus performance gains (either lower latency for small transactions and/or higher throughput for larger writes/reads) can sometimes be attained by changing the interrupt coalescence settings of the Ethernet NIC that handles the IPbus packets. These settings can be changed using ethtool, although beware that these settings may revert after a system reboot.

Assuming in this example that the Ethernet adapter performing the UDP transactions to the hardware is eth1, you can list the coalescence parameters using this command:

sudo /sbin/ethtool -c eth1

Changing the rx-usecs parameter to 0 has been observed to substantially improve (~double) the throughput for small block reads/writes on some NICs. This can be done like so:

sudo /sbin/ethtool -C eth1 rx-usecs 0

However, your mileage with this performance tweak may vary…