.. _controlhub: Control Hub =========== The ControlHub is a software application that forms a single point of access for IPbus control of each device; specifically, it arbitrates simultaneous access from multiple control applications to one or more devices, and it implements the IPbus reliability mechanism for the UDP packets travelling between the ControlHub and the IPbus target devices. Since the Controlub is a software application, the uHAL-ControlHub communication uses TCP, which has sophisticated congestion mitigation and flow-control algorithms. .. note:: This article assumes that you have already installed the Control Hub software and its dependencies via yum or the appropriate IPbus RPMs, according to the installation instructions found :ref:`here ` Usage ----- The Control Hub is entirely configuration free and can be considered akin to a network switch that you turn on and forget about. Once switched on, the Control Hub listens for TCP connections on port 10203, so you should ensure that the firewall is configured to allow connections on this port if you wish to connect to the Control Hub from an external machine (i.e. not just over localhost). The scripts that are used to start, stop, and query the status/configuration of the ControlHub are installed under ``/opt/cactus/bin`` by the ControlHub RPM. However, since version 2.3.1, the ControlHub is deployed as a Linux service when installed by RPM, and so the **standard Unix service management commands should be used to start/stop the ControlHub**. In Red Hat 7 (and hence also CentOS 7), the default service management interface changed from SysV init scripts to systemd, and as a result the service control command changed from ``/sbin/service`` to ``/sbin/systemctl``. The ControlHub should be controlled, and its status queried, using the following commands: +------------------------------------------------+---------------------------------+-------------------------------------------+ | Function | RedHat < 7 | RedHat >= 7 | | | [e.g. SL(C) 6] | [e.g. CERN CentOS7] | +================================================+=================================+===========================================+ | Start the ControlHub | ``service controlhub start`` | ``sudo systemctl start controlhub`` | +------------------------------------------------+---------------------------------+-------------------------------------------+ | Stop the ControlHub | ``service controlhub stop`` | ``sudo systemctl stop controlhub`` | +------------------------------------------------+---------------------------------+-------------------------------------------+ | Restart the ControlHub (if running) | ``service controlhub restart`` | ``systemctl restart controlhub`` | +------------------------------------------------+---------------------------------+-------------------------------------------+ | Check if ControlHub is currently running | ``service controlhub status`` | ``systemctl status controlhub`` | +------------------------------------------------+---------------------------------+-------------------------------------------+ | Check configuration of ControlHub (if running) | ``service controlhub info`` | ``/opt/cactus/bin/controlhub_info`` | +------------------------------------------------+---------------------------------+-------------------------------------------+ | Print ControlHub statistics report | ``service controlhub stats`` | ``/opt/cactus/bin/controlhub_stats`` | +------------------------------------------------+---------------------------------+-------------------------------------------+ Each of the above commands will return exit code 0 if completed successfully, or a non-zero exit code (with explanatory error message printed) if a problem occurs. The expected output of these commands in various typical scenarios is shown at the :ref:`bottom of this page `. Statistics report ----------------- The first port of call when you have problems should be to check the statistics report returned via the ``/opt/cactus/bin/controlhub_stats`` command. This will allow you to tell if your client application successfully managed to connect, and how many packets have been sent, and if there were timeouts, etc; this information is displayed for each board that is currently being controlled, as well as for each client (uHAL ``HwInterface`` instance) connected to the ControlHub. The statistics report should look something like this (for a ControlHub with one connected uHAL client, controlling one board): .. code-block:: shell-session $ /opt/cactus/bin/controlhub_stats Control Hub Stats Report ------------------------ CLIENT All-time connections: 2 Active connections: 1 (peak: 1) @ 127.0.0.1:42295 Requests received: 51484 (of which 0 were malformed) Responses sent: 51430 Old clients Requests received: 101 (of which 0 were malformed) Responses sent: 101 Board @ 127.0.0.1:60001, since 14:01:54.454, 08-03-2017 UTC Packets Out: 51538 Packets In: 51533 Timeouts: 0 in normal operation (of which 0 were recovered) (0 request packets got lost, 0 reply packet) Other timeouts: 0 (status), 0 (in resends) Old UDP Packets Out: 0 Packets In: 0 Timeouts: 0 in normal operation (of which 0 were recovered) (0 request packets got lost, 0 reply packet) Other timeouts: 0 (status), 0 (in resends) Tip: If you want to see the stats at regular intervals - say every 3 seconds - then use the watch command: ``watch -n 3 controlhub_stats`` Log file -------- The ControlHub logs are sent to the local syslog by default, and the ControlHub RPM installs a syslog configuration file (``/etc/rsyslod.d/controlhub.conf``) that writes these log messages to ``/var/log/controlhub/controlhub.log``. The ControlHub RPM also installs a logrotate configuration file (``/etc/logrotate.d/controlhub.conf``) causing the log file to be rotated weekly, keeping 4 old copies of the log file (with rotation date appended to filename) If the Control Hub isn't starting correctly, or appears to be behaving oddly, you should check the log file found at ``/var/log/controlhub.log``. This log-file is minimal, and will only contain critical process startup reports and any process crash-reports that might occur. One possible cause of Control Hub startup failure is that port 10203 might already be taken. Configuration settings ---------------------- When the ControlHub is running, you can check its version and configuration settings through the ``controlhub_info`` command, which produces the following output: .. code-block:: shell-session $ /opt/cactus/bin/controlhub_info The ControlHub is running. General info ... started at 2019-05-21 11:30:57 UTC uptime: 0 days, 0 hours, 0 minutes, 5 seconds version: 2.6.0 config file: /etc/controlhub/default/sys.config Device client parameters ... max packets in flight to each board = 16 timeout for replies from board = 20ms device client lifetime when inactive = 15000ms TCP parameters ... Listening on port 10203 The above output shows the default value of the ControlHub's settings. These settings are specified by a configuration file that contains a list of Erlang terms and is typically named ``sys.config``. In RPM-based or system-wide installations, the ControlHub will use ``/etc/controlhub/sys.config`` as its configuration file if that file exists, otherwise it will use ``/etc/controlhub/default/sys.config``. The ControlHub RPM installs the default configuration file ``/etc/controlhub/default/sys.config``, so this file should not be edited by users; instead, in order to specify non-default values for settings, users should create a new file, ``/etc/controlhub/sys.config``. The full list of configuration parameters that can be specified in this file are: +--------------------------------------------------------------------------------+----------------------------------+---------------------+----------+ | Configuration setting | Name in configuration file | Default value | Unit | +================================================================================+==================================+=====================+==========+ | | TCP port on which ControlHub listens | ``tcp_listen_port`` | 10203 | | | | for connections from clients (i.e. uHAL) | | | | +--------------------------------------------------------------------------------+----------------------------------+---------------------+----------+ | | Maximum number of UDP packets in flight | ``max_udp_in_flight`` | 16 | | | | to each IPbus target | | | | +--------------------------------------------------------------------------------+----------------------------------+---------------------+----------+ | | Timeout for communication with IPbus | ``device_response_timeout`` | 20 | ms | | | targets (typically FPGA-based devices) | | | | +--------------------------------------------------------------------------------+----------------------------------+---------------------+----------+ | | Time after which a device client process | ``device_client_shutdown_after`` | 15000 | ms | | | will shut down if not communicating | | | | | | with IPbus target | | | | +--------------------------------------------------------------------------------+----------------------------------+---------------------+----------+ .. note:: The ControlHub must be restarted after the configuration file has been edited, in order to pick up the changes. The easiest way to change the ControlHub settings is to copy ``/etc/controlhub/default/sys.config`` to ``/etc/controlhub/sys.config``, then edit the 'controlhub' section as desired. For example, the default timeout that the ControlHub uses when communicating with IPbus targets (boards) is 20 milliseconds; in order to increase this to 500 milliseconds (**N.B. not** normally necessary), you should edit ``/etc/controlhub/sys.config`` so that it contains the following: .. code-block:: erlang [ %% Take application-level settings (e.g. for logging) from 'core.config' file "/opt/cactus/lib/controlhub/core.config", %% Override default values of ControlHub settings {controlhub, [{device_response_timeout, 500}] } ]. Or, in order to both increase the default timeout to 500ms, and change the TCP port to 10204, you would update ``/etc/controlhub/sys.config`` to: .. code-block:: erlang [ %% Take application-level settings (e.g. for logging) from 'core.config' file "/opt/cactus/lib/controlhub/core.config", %% Override default values of ControlHub settings {controlhub, [{device_response_timeout, 500}, {tcp_listen_port, 10204} ] } ]. After saving the new contents of ``/etc/controlhub/sys.config`` and then restarting the ControlHub, you should check that the new settings have been picked up by running the ``controlhub_info`` command again; the output of that command also states which configuration file is being used. ----- .. _controlhub-commands-expected-output: Expected output of ControlHub commands -------------------------------------- .. rubric:: Successfully starting the ControlHub SLC6: .. code-block:: shell-session $ service controlhub start Starting ControlHub ..... ok CC7: no output .. rubric:: Checking the ControlHub's status when it is running SLC6: .. code-block:: shell-session $ service controlhub status The ControlHub is up (version 2.4.1, pid=719) CC7: .. code-block:: shell-session $ systemctl status controlhub ● controlhub.service - IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications) Loaded: loaded (/usr/lib/systemd/system/controlhub.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2017-03-08 11:26:42 CET; 37s ago Docs: http://ipbus.web.cern.ch/ipbus/doc/index.html Process: 3631 ExecStop=/opt/cactus/bin/controlhub_stop (code=exited, status=0/SUCCESS) Process: 4105 ExecStartPost=/bin/sh -c /opt/cactus/lib/controlhub/bin/controlhub getpid > /var/run/controlhub.pid (code=exited, status=0/SUCCESS) Process: 3793 ExecStart=/opt/cactus/bin/controlhub_start (code=exited, status=0/SUCCESS) Main PID: 3906 (beam.smp) Memory: 63.6M CGroup: /system.slice/controlhub.service └─3906 /opt/cactus/lib/controlhub/erts-5.10.2/bin/beam.smp -zdbbl 2097151 -sbt db -scl false -sbwt very_long -swt low -- -root /opt/cactus/lib/controlhub -progname controlhub -- -home $CACTUSROOT -- -boot /opt/cactus/... Mar 08 11:26:31 tsw-cc7.cern.ch systemd[1]: Starting IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications)... Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.585 [notice] <0.1084.0> - Starting the ControlHub application (version 2.4.1). Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.585 [notice] <0.1085.0> - ControlHub root supervisor starting now. Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.585 [notice] <0.1086.0> - Initialising the stats server. Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.585 [notice] <0.1087.0> - Initialising the device client registry. Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.586 [notice] <0.1088.0> - Initialising the TCP listener (port 10203). Mar 08 11:26:40 tsw-cc7.cern.ch controlhub_start[3793]: Starting ControlHub ........ ok Mar 08 11:26:42 tsw-cc7.cern.ch systemd[1]: Started IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications). .. rubric:: Successfully stopping the ControlHub SLC6: .. code-block:: shell-session $ service controlhub stop Stopping ControlHub ....... ok CC7: no output .. rubric:: Stopping the ControlHub, or checking its status/configuration, when the ControlHub is not running CC7, ``systemctl stop controlhub``: no output, exit code zero CC7, ``systemctl status controlhub``: .. code-block:: shell-session $ systemctl status controlhub ● controlhub.service - IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications) Loaded: loaded (/usr/lib/systemd/system/controlhub.service; enabled; vendor preset: disabled) Active: inactive (dead) since Wed 2017-03-08 11:30:54 CET; 43s ago Docs: http://ipbus.web.cern.ch/ipbus/doc/index.html Process: 4170 ExecStop=/opt/cactus/bin/controlhub_stop (code=exited, status=0/SUCCESS) Process: 4105 ExecStartPost=/bin/sh -c /opt/cactus/lib/controlhub/bin/controlhub getpid > /var/run/controlhub.pid (code=exited, status=0/SUCCESS) Process: 3793 ExecStart=/opt/cactus/bin/controlhub_start (code=exited, status=0/SUCCESS) Main PID: 3906 (code=exited, status=0/SUCCESS) Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.585 [notice] <0.1087.0> - Initialising the device client registry. Mar 08 11:26:39 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:26:39.586 [notice] <0.1088.0> - Initialising the TCP listener (port 10203). Mar 08 11:26:40 tsw-cc7.cern.ch controlhub_start[3793]: Starting ControlHub ........ ok Mar 08 11:26:42 tsw-cc7.cern.ch systemd[1]: Started IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications). Mar 08 11:30:48 tsw-cc7.cern.ch systemd[1]: Stopping IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications)... Mar 08 11:30:49 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:30:49.638 [notice] <0.1088.0> - TCP listener shutting down. Mar 08 11:30:49 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:30:49.638 [notice] <0.1087.0> - Device client registry shutting down. Mar 08 11:30:49 tsw-cc7.cern.ch controlhub[3906]: 2017-03-08 11:30:49.638 [notice] <0.1084.0> - Stopping the ControlHub application. Mar 08 11:30:54 tsw-cc7.cern.ch controlhub_stop[4170]: Stopping ControlHub ....... ok Mar 08 11:30:54 tsw-cc7.cern.ch systemd[1]: Stopped IPbus ControlHub server daemon (mediates simultanous IPbus-based hardware access from multiple control applications). Otherwise: .. code-block:: shell-session $ service controlhub status The ControlHub is down. **N.B:** The controlhub commands should be run from a directory that the executing user has read access to; otherwise the following error message may appear: .. code-block:: shell-session [user@pc ~]$ sudo /opt/cactus/bin/controlhub_start {"init terminating in do_boot",{badarg,[{erl_prim_loader,check_file_result,3,[]},{init,get_boot,1,[]},{init,get_boot,2,[]},{init,do_boot,3,[]}]}} init terminating in do_boot ()