.. _performance:

Performance
-----------

The performance of IPbus is dependent on many variables, such as:

* Network topology/latency
* Quality of networking components (NIC, switches, etc) and related drivers
* Mode of operation (ipbusudp-2.0, chtcp-2.0, etc)
* Interrupt coalescence settings of your NIC (see the :ref:`performance-tweaks` section below)
* CPU type/speed and throttle state

The performance via the ControlHub has been measured using very similar network layout and hardware to that planned for final deployment in the CMS experiment. These performance measurements are described in the `TWEPP-14 IPbus presentation <https://indico.cern.ch/event/299180/session/5/contribution/46>`_, but some notable results are summarised below (for ``chtcp-2.0`` communication - i.e. via ControlHub - on 1Gbit/s network infrastructure):

* For one software client controlling one device, the single-word read/write latency is approximately 250 micro-seconds

  * Although this single-word latency is larger than VME/PCIe-based control, for multiple transactions or large block transfers this is compensated by concatenating multiple (up to order of 100) transactions into each packet, and by having multiple packets in flight around the system at any given time
  * Hence, optimal performance will be obtained if network dispatches are only performed when necessary.

* The 1-client-to-1-device block read/write throughput for payloads larger than 1 Mbyte is above 0.5 Gbit/s
* The total block read/write throughput is above 0.75 Gbit/s for three or more boards in a single MicroTCA crate.

Given the many variables affecting IPbus performance, your experience will almost certainly vary from the values given above. However if you experience substantially worse (more that a factor 2) performance, we might be able to assist - please submit a `support ticket <https://github.com/ipbus/ipbus-software/issues/new>`_. 


.. _performance-tweaks:

Performance tweaks with ethtool
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

IPbus performance gains (either lower latency for small transactions and/or higher throughput for larger writes/reads) can sometimes be attained by changing the interrupt coalescence settings of the Ethernet NIC that handles the IPbus packets.  These settings can be changed using ``ethtool``, although beware that these settings may revert after a system reboot.

Assuming in this example that the Ethernet adapter performing the UDP transactions to the hardware is ``eth1``, you can list the coalescence parameters using this command:

.. code-block:: sh

  sudo /sbin/ethtool -c eth1

Changing the ``rx-usecs`` parameter to 0 has been observed to substantially improve (~double) the throughput **for small block reads/writes** on some NICs. This can be done like so:

.. code-block:: sh

  sudo /sbin/ethtool -C eth1 rx-usecs 0

However, your mileage with this performance tweak may vary...