Control Hub¶

The ControlHub is a software application that forms a single point of access for IPbus control of each device; specifically, it arbitrates simultaneous access from multiple control applications to one or more devices, and it implements the IPbus reliability mechanism for the UDP packets travelling between the ControlHub and the IPbus target devices. Since the Controlub is a software application, the uHAL-ControlHub communication uses TCP, which has sophisticated congestion mitigation and flow-control algorithms.

Note

This article assumes that you have already installed the Control Hub software and its dependencies via yum or the appropriate IPbus RPMs, according to the installation instructions found here

Usage¶

The Control Hub is entirely configuration free and can be considered akin to a network switch that you turn on and forget about. Once switched on, the Control Hub listens for TCP connections on port 10203, so you should ensure that the firewall is configured to allow connections on this port if you wish to connect to the Control Hub from an external machine (i.e. not just over localhost).

The scripts that are used to start, stop, and query the status/configuration of the ControlHub are installed under /opt/cactus/bin by the ControlHub RPM. However, since version 2.3.1, the ControlHub is deployed as a Linux service when installed by RPM, and so the standard Unix service management commands should be used to start/stop the ControlHub. In Red Hat 7 (and hence also CentOS 7), the default service management interface changed from SysV init scripts to systemd, and as a result the service control command changed from /sbin/service to /sbin/systemctl. The ControlHub should be controlled, and its status queried, using the following commands:

Function	RedHat < 7 [e.g. SL(C) 6]	RedHat >= 7 [e.g. CERN CentOS7]
Start the ControlHub	`service controlhub start`	`sudo systemctl start controlhub`
Stop the ControlHub	`service controlhub stop`	`sudo systemctl stop controlhub`
Restart the ControlHub (if running)	`service controlhub restart`	`systemctl restart controlhub`
Check if ControlHub is currently running	`service controlhub status`	`systemctl status controlhub`
Check configuration of ControlHub (if running)	`service controlhub info`	`/opt/cactus/bin/controlhub_info`
Print ControlHub statistics report	`service controlhub stats`	`/opt/cactus/bin/controlhub_stats`

Each of the above commands will return exit code 0 if completed successfully, or a non-zero exit code (with explanatory error message printed) if a problem occurs.

The expected output of these commands in various typical scenarios is shown at the bottom of this page.

Statistics report¶

The first port of call when you have problems should be to check the statistics report returned via the /opt/cactus/bin/controlhub_stats command. This will allow you to tell if your client application successfully managed to connect, and how many packets have been sent, and if there were timeouts, etc; this information is displayed for each board that is currently being controlled, as well as for each client (uHAL HwInterface instance) connected to the ControlHub. The statistics report should look something like this (for a ControlHub with one connected uHAL client, controlling one board):

$ /opt/cactus/bin/controlhub_stats
Control Hub Stats Report
------------------------

CLIENT  All-time connections: 2
          Active connections: 1 (peak: 1)
  @ 127.0.0.1:42295
           Requests received: 51484 (of which 0 were malformed)
              Responses sent: 51430
  Old clients
           Requests received: 101 (of which 0 were malformed)
              Responses sent: 101

Board @ 127.0.0.1:60001, since 14:01:54.454, 08-03-2017 UTC
              Packets Out: 51538
               Packets In: 51533
                 Timeouts: 0 in normal operation (of which 0 were recovered)
                             (0 request packets got lost, 0 reply packet)
           Other timeouts: 0 (status), 0 (in resends)

Old UDP
              Packets Out: 0
               Packets In: 0
                 Timeouts: 0 in normal operation (of which 0 were recovered)
                             (0 request packets got lost, 0 reply packet)
           Other timeouts: 0 (status), 0 (in resends)

Tip: If you want to see the stats at regular intervals - say every 3 seconds - then use the watch command: watch -n 3 controlhub_stats

Log file¶

The ControlHub logs are sent to the local syslog by default, and the ControlHub RPM installs a syslog configuration file (/etc/rsyslod.d/controlhub.conf) that writes these log messages to /var/log/controlhub/controlhub.log. The ControlHub RPM also installs a logrotate configuration file (/etc/logrotate.d/controlhub.conf) causing the log file to be rotated weekly, keeping 4 old copies of the log file (with rotation date appended to filename)

If the Control Hub isn’t starting correctly, or appears to be behaving oddly, you should check the log file found at /var/log/controlhub.log. This log-file is minimal, and will only contain critical process startup reports and any process crash-reports that might occur. One possible cause of Control Hub startup failure is that port 10203 might already be taken.

Configuration settings¶

When the ControlHub is running, you can check its version and configuration settings through the controlhub_info command, which produces the following output:

$ /opt/cactus/bin/controlhub_info
The ControlHub is running. General info ...
    started at 2019-05-21 11:30:57 UTC
    uptime:    0 days, 0 hours, 0 minutes, 5 seconds
    version:   2.6.0
    config file: /etc/controlhub/default/sys.config

Device client parameters ...
    max packets in flight to each board = 16
    timeout for replies from board = 20ms
    device client lifetime when inactive = 15000ms

TCP parameters ...
    Listening on port 10203

The above output shows the default value of the ControlHub’s settings. These settings are specified by a configuration file that contains a list of Erlang terms and is typically named sys.config.

In RPM-based or system-wide installations, the ControlHub will use /etc/controlhub/sys.config as its configuration file if that file exists, otherwise it will use /etc/controlhub/default/sys.config. The ControlHub RPM installs the default configuration file /etc/controlhub/default/sys.config, so this file should not be edited by users; instead, in order to specify non-default values for settings, users should create a new file, /etc/controlhub/sys.config.

The full list of configuration parameters that can be specified in this file are:

Configuration setting	Name in configuration file	Default value	Unit
TCP port on which ControlHub listens for connections from clients (i.e. uHAL)	`tcp_listen_port`	10203
Maximum number of UDP packets in flight to each IPbus target	`max_udp_in_flight`	16
Timeout for communication with IPbus targets (typically FPGA-based devices)	`device_response_timeout`	20	ms
Time after which a device client process will shut down if not communicating with IPbus target	`device_client_shutdown_after`	15000	ms

Note

The ControlHub must be restarted after the configuration file has been edited, in order to pick up the changes.

The easiest way to change the ControlHub settings is to copy /etc/controlhub/default/sys.config to /etc/controlhub/sys.config, then edit the ‘controlhub’ section as desired. For example, the default timeout that the ControlHub uses when communicating with IPbus targets (boards) is 20 milliseconds; in order to increase this to 500 milliseconds (N.B. not normally necessary), you should edit /etc/controlhub/sys.config so that it contains the following:

[
  %% Take application-level settings (e.g. for logging) from 'core.config' file
  "/opt/cactus/lib/controlhub/core.config",

  %% Override default values of ControlHub settings
  {controlhub,
    [{device_response_timeout, 500}]
  }
].

Or, in order to both increase the default timeout to 500ms, and change the TCP port to 10204, you would update /etc/controlhub/sys.config to:

[
  %% Take application-level settings (e.g. for logging) from 'core.config' file
  "/opt/cactus/lib/controlhub/core.config",

  %% Override default values of ControlHub settings
  {controlhub,
    [{device_response_timeout, 500},
     {tcp_listen_port, 10204}
     ]
  }
].

After saving the new contents of /etc/controlhub/sys.config and then restarting the ControlHub, you should check that the new settings have been picked up by running the controlhub_info command again; the output of that command also states which configuration file is being used.

Expected output of ControlHub commands¶

Successfully starting the ControlHub

SLC6:

$ service controlhub start
Starting ControlHub ..... ok

CC7: no output

Checking the ControlHub’s status when it is running

SLC6:

$ service controlhub status
The ControlHub is up (version 2.4.1, pid=719)

CC7:

$ systemctl status controlhub
● controlhub.service - IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications)
   Loaded: loaded (/usr/lib/systemd/system/controlhub.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2017-03-08 11:26:42 CET; 37s ago
     Docs: http://ipbus.web.cern.ch/ipbus/doc/index.html
  Process: 3631 ExecStop=/opt/cactus/bin/controlhub_stop (code=exited, status=0/SUCCESS)
  Process: 4105 ExecStartPost=/bin/sh -c /opt/cactus/lib/controlhub/bin/controlhub getpid > /var/run/controlhub.pid (code=exited, status=0/SUCCESS)
  Process: 3793 ExecStart=/opt/cactus/bin/controlhub_start (code=exited, status=0/SUCCESS)
 Main PID: 3906 (beam.smp)
   Memory: 63.6M
   CGroup: /system.slice/controlhub.service
           └─3906 /opt/cactus/lib/controlhub/erts-5.10.2/bin/beam.smp -zdbbl 2097151 -sbt db -scl false -sbwt very_long -swt low -- -root /opt/cactus/lib/controlhub -progname controlhub -- -home $CACTUSROOT -- -boot /opt/cactus/...

Mar 08 11:26:31 tsw-cc7.cern.ch systemd[1]: Starting IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications)...
Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.585 [notice] <0.1084.0> - Starting the ControlHub application (version 2.4.1).
Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.585 [notice] <0.1085.0> - ControlHub root supervisor starting now.
Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.585 [notice] <0.1086.0> - Initialising the stats server.
Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.585 [notice] <0.1087.0> - Initialising the device client registry.
Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.586 [notice] <0.1088.0> - Initialising the TCP listener (port 10203).
Mar 08 11:26:40 tsw-cc7.cern.ch controlhub_start[3793]: Starting ControlHub ........ ok
Mar 08 11:26:42 tsw-cc7.cern.ch systemd[1]: Started IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications).

Successfully stopping the ControlHub

SLC6:

$ service controlhub stop
Stopping ControlHub ....... ok

CC7: no output

Stopping the ControlHub, or checking its status/configuration, when the ControlHub is not running

CC7, systemctl stop controlhub: no output, exit code zero

CC7, systemctl status controlhub:

$ systemctl status controlhub
● controlhub.service - IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications)
   Loaded: loaded (/usr/lib/systemd/system/controlhub.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Wed 2017-03-08 11:30:54 CET; 43s ago
     Docs: http://ipbus.web.cern.ch/ipbus/doc/index.html
  Process: 4170 ExecStop=/opt/cactus/bin/controlhub_stop (code=exited, status=0/SUCCESS)
  Process: 4105 ExecStartPost=/bin/sh -c /opt/cactus/lib/controlhub/bin/controlhub getpid > /var/run/controlhub.pid (code=exited, status=0/SUCCESS)
  Process: 3793 ExecStart=/opt/cactus/bin/controlhub_start (code=exited, status=0/SUCCESS)
 Main PID: 3906 (code=exited, status=0/SUCCESS)

Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.585 [notice] <0.1087.0> - Initialising the device client registry.
Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.586 [notice] <0.1088.0> - Initialising the TCP listener (port 10203).
Mar 08 11:26:40 tsw-cc7.cern.ch controlhub_start[3793]: Starting ControlHub ........ ok
Mar 08 11:26:42 tsw-cc7.cern.ch systemd[1]: Started IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications).
Mar 08 11:30:48 tsw-cc7.cern.ch systemd[1]: Stopping IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications)...
Mar 08 11:30:49 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:30:49.638 [notice] <0.1088.0> - TCP listener shutting down.
Mar 08 11:30:49 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:30:49.638 [notice] <0.1087.0> - Device client registry shutting down.
Mar 08 11:30:49 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:30:49.638 [notice] <0.1084.0> - Stopping the ControlHub application.
Mar 08 11:30:54 tsw-cc7.cern.ch controlhub_stop[4170]: Stopping ControlHub ....... ok
Mar 08 11:30:54 tsw-cc7.cern.ch systemd[1]: Stopped IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications).

Otherwise:

$ service controlhub status
The ControlHub is down.

N.B: The controlhub commands should be run from a directory that the executing user has read access to; otherwise the following error message may appear:

[user@pc ~]$ sudo /opt/cactus/bin/controlhub_start
{"init terminating in do_boot",{badarg,[{erl_prim_loader,check_file_result,3,[]},{init,get_boot,1,[]},{init,get_boot,2,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ()