appfwk: The DUNE DAQ Application Framework

Overview

In a single sentence:

appfwk consists of a generic DAQ application (daq_application) which can be configured to run different types of DAQ modules performing various DAQ-related tasks, where DAQ modules are user-developed implementations of appfwk's DAQModule interface class.

In a bit more detail:

appfwk provides the scaffolding on which all DUNE DAQ software processes can be developed. The running DAQ typically consists of multiple distinct processes assigned various tasks: filtering data, requesting it, saving it to storage, etc. There are many different types of process, some of which may not even have been conceived of yet, and it would be cumbersome to recompile multiple different types of process across many packages every time one wanted to change the behavior of the DAQ. To solve this problem, the approach that's been taken is to have a standard DUNE DAQ software process daq_application which can be configured at runtime by Run Control in order to perform some particular function in the DAQ.

daq_application is designed as a flexible container of "DAQ modules" (units of code designed to perform specific tasks) and "connections" (designed to move data between DAQ modules and DAQ applications). These specific tasks can vary widely; they include producing fake data for testing purposes, putting data into long term storage, and so forth. DAQ modules will typically execute user-defined functions when receiving standard transitions from Run Control: "conf", "start", etc. appfwk provides the DAQModule base class which users should derive their DAQ module class from in their own packages.

Running daq_application

In general, in a full blown DAQ system users won't be running daq_application from the command line. However, it can be instructive to know what options daq_application takes. Details can be found here.

Writing DAQ modules

Basics of the DAQModule interface

Be aware that much of the boilerplate code described below can be automatically generated using the create_dunedaq_package script

When implenting a DAQ module, you'll want to #include the DAQModule.hpp header, and derive your DAQ module from the DAQModule base class. The most important parts of DAQModule.hpp to an implementor of a DAQ module are the following:

• DEFINE_DUNE_DAQ_MODULE: This is a macro which should be "called" at the bottom of your DAQ module's source file with an "argument" of the form dunedaq::<your_package_name>::<your DAQ module name>. E.g., DEFINE_DUNE_DAQ_MODULE(dunedaq::dfmodules::DataWriter) at the bottom of the dfmodules package's DataWriter module's source file

• register_command: takes as arguments the name of a command and a function which should execute when the command is received. The function is user defined, and takes an instance of DAQModule::data_t as argument. DAQModule::data_t is aliased to the nlohmann::json type and can thus be thought of as a blob of JSON-structured data. While in principle any arbitary name could be associated with any function of arbitrary behavior to create a command, in practice implementors of DAQ modules define commands associated with the DAQ's state machine: "conf", "start", "stop", "scrap". Not all DAQ modules necessarily need to perform an action for each of those transitions; e.g., a module may only be designed to do something during configuration, and not change as the DAQ enters the running state ("start") or exits it ("stop"). It also supports an optional third argument which lists the states that the application must be in for the command to be valid.

• init: this pure virtual function's implementation is meant to create objects which are persistent for the lifetime of the DAQ module. It takes as an argument the type DAQModule::data_t. Typically it will use parameters from this JSON argument to define the persistent objects. For persistent objects of types which don't have an efficient copy assigment operator, a common technique is to declare as member data a unique_ptr to the type of interest and then, in init, to allocate the desired object on the heap using values from the JSON and point the unique_ptr member to it. Connection objects are commonly allocated in init; they'll be described in more detail later in this document.

An conceptual example of what this looks like is the following simplified version of a DAQ module implementation.

// This file would be called plugins/MyDaqModule.hpp
// Functions would typically be defined in plugins/MyDaqModule.cpp

class MyDaqModule : public dunedaq::appfwk::DAQModule {
  public:
     MyDaqModule(const std::string& name) : // A DAQ module instance is meant to have a unique name
        dunedaq::appfwk::DAQModule(name)
          {
          register_command("conf",  &MyDAQModule::do_conf);
          register_command("start", &MyDAQModule::do_start);
          register_command("stop",  &MyDAQModule::do_stop);
          register_command("scrap", &MyDAQModule::do_scrap);
     }

     void init(const data_t& init_data) override;

  private:

     void do_conf(const data_t& conf_data);
     void do_start(const data_t& start_data);
     void do_stop(const data_t& stop_data);
     void do_scrap(const data_t& scrap_data);
};

Overview of additional function-by-function descriptions

A set of programming idioms have been developed over the first year of DAQ module development which, while not strictly necessary for implementing DAQ modules, have proven to be quite useful. They'll be described within the context of the functions above, as well as some new functions described later. While writers of DAQ modules should follow the coding guidelines presented in the DUNE DAQ C++ Style Guide, the techniques which will be presented below concern a higher-level view of the code and unlike the guidelines are more recommendations than requirements. Of course, you can also see the techniques in action by looking at the source code of actual DAQ modules.

The DAQ module's constructor

While of course all member data will be initialized here in a technical (as opposed to logical) sense, in general the only things about a DAQ module instance which are meaningfully defined in its constructor are:

1. Its command set, already discussed

2. Its unique name, via the argument to its constructor

A word needs to be said about the concept of a "unique name" here. Looking in DAQModule.hpp, you'll see that the DAQModule base class itself inherits from an appfwk class called NamedObject. The instances of any class which inherits from NamedObject will require unique names; while this of course would include DAQ modules as they inherit from DAQModule, other types of class can require unique names as well.

The init function

Already touched upon above, this function takes a data_t instance (i.e., JSON) to tell it what objects to make persistent over the DAQ module's lifetime. A very common example of this is the construction of the iomanager connections which will pipe data into and out of an instance of the DAQ module. A description of this common use case will illustrate a couple of very important aspects of DAQ module programming.

When a DAQ module writer wants to communicate with other DAQ modules, they use the iomanager. The iomanager Sender and Receiver objects needed by a DAQ Module get built in the call to init based on the JSON configuration init receives . A definition of init, then, can look like the following:

void MyDaqModule::init(const data_t& init_data) {
    auto ci = appfwk::connection_index(init_data, {"name_of_required_input"});
    m_required_input_ptr = dunedaq::get_iom_receiver(ci["name_of_required_input"]));
}

In the code above, the call to connection_index, defined in DAQModuleHelper.cpp, returns a map which connects the names of connections with the ConnectionRef objects consumed by IOManager. It will throw an exception if any provided names don't appear - so in this case, if name_of_required_input isn't found in init_data, an exception will be thrown. If the name is found, then m_required_input_ptr, which here is an std::shared_ptr_ to a iomanager::Receiver of MyType_ts, gets pointed to the appropriate Receiver. When the DAQ enters the running state, we could have MyDaqModule receive MyType_t instances from m_required_input_ptr for processing.

The do_conf function

As one might expect, there are many values which a DAQ module may rely on to perform its calculations when in the running state that ideally should be settable during the conf transition. The typical technique is to have some member data which in the DAQ module constructor intentionally gets initialized either to zero or to implausible values (e.g. m_calibration_scale_factor(-1), m_num_total_warnings(0)) and then to set them properly during the config transition. You'll see in the code below that the type of the data instance data which gets extracted from the JSON is mydaqmodule::Conf, and then data is used to set the member(s).

void MyDaqModule::do_conf(const data_t& conf_data)
{
  auto data = conf_data.get<mydaqmodule::Conf>();

  m_calibration_scale_factor = data.calibration_scale_factor;
  // ...and then set the other members which take per-configuration values...
}

This of course raises the question: what is mydaqmodule::Conf? It's a struct, but rather than being manually written the code for it is generated by the DUNE DAQ build system itself, using a jsonnet file as input. It's in the jsonnet file that the logical contents of the struct are defined; an example of this type of file can be found here. This approach allows automatic compile-time checks on the variable (here mydaqmodule::Conf) retrieved by the module, reducing the workload on the implementor of do_conf. Note also that in fact many functions in a DAQ module, including init, can use JSON as input to control their actions, not just do_conf. Further details on the generation of code from jsonnet files are beyond the scope of appfwk documentation and are instead covered in this section of the daq-cmake documentation.

The do_start function

Most DAQ modules are designed to loop over some sort of repeated action when the DAQ enters the running state, and it's in the do_start function that this repeated action begins. A very common technique for the do_start function is, "Set an atomic boolean stating that we're now in the running state, and then start one or more threads which perform actions in loops which they break out of if they see that the atomic boolean indicates we're no longer in the running state".

While it's of course possible to accomplish this using the existing concurrency facilities provided by the C++ Standard Library, the utilities package provides a class, WorkerThread, which makes this easier. WorkerThread is covered in detail here; when in use the do_start function can be as simple as follows:

void MyDaqModule::do_start(const data_t& /*args*/) {
    m_thread.start_working_thread();  // m_thread is an `utilities::WorkerThread` member of MyDaqModule
}

Note that start_working_thread takes an optional argument which gives the WorkerThread instance a name, potentially allowing shifters to keep track of various threads for debugging purposes.

The do_stop function

Quite simple, basically the reverse of do_start:

void MyDaqModule::do_stop(const data_t& /*args*/) {
    m_thread.stop_working_thread();  // m_thread is an `utilities::WorkerThread` member of MyDaqModule
}

Note that if your do_start function also allocates any resources (hardware, memory, etc.) it should be deallocated here. Also, the queues which send data to your DAQ module should be drained. The idea is that you want your DAQ module to be able to accept a "start" transition after receiving a "stop" transition without anything from the previous run interfering.

The do_scrap function

This is the reverse of do_config. Often this function isn't even needed since the values which get set in do_conf are completely overwritten on subsequent calls to do_conf. However, as the point of this function is to bring the DAQ module back to a state where it can be configured again, it's important that any hardware or memory resources which were acquired in do_conf are released here in do_scrap.

The get_info function

Not yet mentioned, you can see in DAQModule.hpp that there's a virtual function called get_info which defaults to a no-op:

virtual void get_info(opmonlib::InfoCollector& /*ci*/, int /*level*/) { return; };

It's meant to be implemented by DAQ module writers to supply metrics about the DAQ module; an example of this can be found here.

The full code

Given the code features described above, MyDaqModule would look something like the following, ignoring things irrelevant to the pedagogy presented here, like proper error handling, log statements, #includes, etc. Pretend the name of the package MyDaqModule in is "mypackage":

• MyDaqModule.hpp:

  class MyDaqModule : public dunedaq::appfwk::DAQModule {
    public:
  
       alias MyType_t = double; // Pretend this module processes an incoming stream of doubles 
  
       MyDaqModule(const std::string& name) : // A DAQ module instance is meant to have a unique name
          dunedaq::appfwk::DAQModule(name),
          m_thread(std::bind(&MyDaqModule::do_work, this, std::placeholders::_1)),
          m_calibration_scale_factor(-1)
          {
            register_command("conf",  &MyDAQModule::do_conf);
            register_command("start", &MyDAQModule::do_start);
            register_command("stop",  &MyDAQModule::do_stop);
            register_command("scrap", &MyDAQModule::do_scrap);
       }
  
       void init(const data_t& init_data) override;
  
    private:
  
       void do_conf(const data_t& conf_data);
       void do_start(const data_t& start_data);
       void do_stop(const data_t& stop_data);
       void do_scrap(const data_t& scrap_data);
  
       void do_work(std::atomic<bool>&);
       dunedaq::utilities::WorkerThread m_thread; 
       double m_calibration_scale_factor;
       std::shared_ptr<dunedaq::iomanager::Receiver<MyType_t>> m_required_input_ptr; 
  };
  

• MyDaqModule.cpp:

  void MyDaqModule::init(const data_t& init_data) {
      auto ci = appfwk::connection_index(init_data, {"name_of_required_input"});
      m_required_input_ptr = dunedaq::get_iom_receiver(ci["name_of_required_input"]));
  }
  
  void MyDaqModule::do_conf(const data_t& conf_data)
  {
    auto data = conf_data.get<mydaqmodule::Conf>();
  
    m_calibration_scale_factor = data.calibration_scale_factor;
    // ...and then set the other members which take per-configuration values...
  }
  
  void MyDaqModule::do_start(const data_t& /*args*/) {
      m_thread.start_working_thread();  // m_thread is an `utilities::WorkerThread` member of MyDaqModule
  }
  
  void MyDaqModule::do_stop(const data_t& /*args*/) {
      m_thread.stop_working_thread();  
  }
  
  void MyDaqModule::do_scrap(const data_t& /*args*/) {
      m_calibration_scale_factor = -1;
  }
  
  void MyDaqModule::do_work(std::atomic<bool>& running_flag)
  {
     while (running_flag.load()) {
        // Here we'd receive data from m_required_input_ptr and presumably use m_calibration_scale_factor when processing the data
     }
  }   
  
  DEFINE_DUNE_DAQ_MODULE(dunedaq::mypackage::MyDaqModule)
  

Final thoughts on writing DAQ modules

Now that you've been given an overview of appfwk and how to write DAQ modules, you're encouraged to look at the various repos to see how other DUNE DAQ developers have written DAQ modules. The listrev package is maintained as an example of simple DAQ modules, and another package with plenty of real DAQ modules to study is dfmodules, modules used for dataflow purposes. Keep in mind that not all DAQ modules will adhere to the model described above, and you can judge for yourself what techniques you feel will make it easiest to write and maintain a DAQ module.

API Diagram

---

Last git commit to the markdown source of this page:

Author: John Freeman

Date: Wed Sep 6 10:18:49 2023 -0500

If you see a problem with the documentation on this page, please file an Issue at https://github.com/DUNE-DAQ/appfwk/issues