Integration tests¶
Felix Firmware Integration Testing status¶
Initial setup¶
Firmware:¶
Download a copy of the FELIX firmware here.
Note, to get the 10 link build with the TPG firmware download the file starting with flx712-dune-ipbus-pod-10hf
To program the FELIX card, you can use Vivado if you have not setup a dunedaq envirnomnent (see below), otherwise, you can use the low level tool fflashprog
to flash the correct partition on the card. Each method uses a different file type, both of which are provided in the download.
Vivado:¶
To program with Vivado, you need to ensure you have X-11 forwarding or a VNC if you are connecting to the machine remotely, a copy of the .bit file of the firmware and can run as sudo
or root
on the machine you are working on. Run vivado:
vivado
and a GUI should appear, under Tasks select the option "Open Hardware Manager", then "tools" -> "open new target". This will open a window where you can connect to hardware locally or via a remote server. Assuming this is the machine with the FELIX card installed connect to local server and in the next window look for a hardware device (bottom table) with the device name including xkcu115
. Click next then finish. You should see the device appear on the hardware tab on the left. Now click "tools" -> "program device" -> "xkcu115" and then specify the bitstream (.bit) file and click program. I the programming worked then you should see no errors in the tcl console (some warnings may appear but can be ingored), and now you can close vivado and reboot the machine with the FELIX card:
sudo reboot
fflashprog
:¶
To program with fflashprog
ensure you have a copy of the .mcs file of the firmware, already have a working dunedaq environment or a copy of the low level tools and can run as sudo
or root
on the machine you are working on. Then run the commands:
fflashprog -c 0 -f 3 <.mcs file to program> prog
-c
). Then to check if the firmware is programmed with the correct firmware:
fflashprog -c 0 -f 3 <.mcs file to program>
Note, the -f command indicates which partition to program, if programming on partition 3 doesn't work, run flx-info
to get the correct partition number or program the card using vivado.
Drivers:¶
To install drivers for the FELIX card follow the instructions to setup local drivers:
https://github.com/DUNE-DAQ/flxlibs/wiki/Local-driver
Ensure the FELIX is programmed, the first check is to enable and get the status of the drivers:
sudo <location of local drivers>/drivers_flx_local start
sudo <location of local drivers>/drivers_flx_local status
if you see the following output after starting the drivers:
Starting cmem driver
major number for cmem_rcc is 238
Starting io_rcc driver
major number for io_rcc is 237
2 flx PCIe endpoints found
Starting flx driver
major number for flx is 236
creating node /dev/flx0
creating node /dev/flx1
then it has successfully worked, if you see errors, then reboot the machine and try again.
Note you can ignore error messeges about regmap verision mismatches
DUNEDAQ software:¶
First ensure you have setup a dunedaq enivronmet, following these instructions: https://dune-daq-sw.readthedocs.io/en/latest/packages/daq-buildtools/#running-a-release-from-cvmfs
Now to get access to the integration test scripts and create test configurations, clone the flxlibs repository:
cd sourcecode
git clone https://github.com/DUNE-DAQ/flxlibs.git
dbt-workarea-env
dbt-build.py
cd ..
Clone dtp-pattens in your working directory, a repository containing fake data patterns to test with the FELIX:
git clone ssh://git@gitlab.cern.ch:7999/dune-daq/readout/dtp-patterns.git
<!--### DTP software (only for dunedaq versions less then v2.10.0):
To test hitfinding with the appropriate firmware, dtp-controls
and wupper-toybox
are needed to control the TPG (Note wupper-toybox
to be replaced with DUNE-DAQ/uhallibs
).
In your sourcecode
directory, clone wupper-toybox
:
git clone ssh://git@gitlab.cern.ch:7999/dune-daq/readout/wupper-toybox.git
and make a clean build of the dunedaq software:
dbt-build.sh -c
Next, clone dtp-controls
in the working directory:
git clone ssh://git@gitlab.cern.ch:7999/dune-daq/readout/dtp-controls.git -b aearle/felixSupport
to begin using dtp-controls
:
source dtp-controls/env.sh
Each time you open a new session, alongside setting up the dunedaq environment you must also setup dtp-controls
, so the full set of startup commands are:
source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt dunedaq-v2.8.2
dbt-workarea-env
source dtp-controls/env.sh
```-->
## Running tests
### test dataflow with low level tools:
Integration test scripts are located in `flxlibs/scripts/integration-tests`
#### basic card communication
run the bash script:
```bash
./test-communication.sh
test dataflow of ADC links using the internal emulator¶
run the bash script:
./test-adc-dataflow-emu.sh
test dataflow of ADC links with optical links¶
to test receiving data over optical links you need to first run the following command:
./test-adc-dataflow-fe.sh
test dataflow of TP links with the wibulator¶
First run on a SLR 0 and 1 separately:
./test-tpg-dataflow-wib-slr0.sh
./test-tpg-dataflow-wib-slr1.sh
fdaq
, then configure the firmware TPG with hfButler.py
and then spy on the link processors. if successful you should see the link processor state change over time for ~60s. After 60s the page will stop updating and you can safely kill the process (ctrl-C). Note you will also get an output file which logs the output of fdaq
, as well as a binary file with the dumped TP's processed by the card.
Now run with both SLR's simultaneously:
./test-tpg-dataflow-wib.sh
fdaq
for each SLR for 60s and as before a log file plus binary file of dumped tp's are produced.
test dataflow of TP links with the internal FELIX emulator¶
Run the bash script
./test-tpg-dataflow-emu.sh
fdaq
then configure the hitfinding tpg in gbt mode i.e. process data from elinks. The outputs of fdaq are stored in log files. if successful you should see the data rate increase twice, once when the adc data is received and then again when the tpg is configured.
test dataflow of TP links with optical links¶
First run the bash script
./test-tpg-dataflow-fe.sh
./set_gbt.sh
fdaq
run and if you want to monitor the link processors of the tpg then run set_gbt.sh
with these arguements instead:
./set_gbt.sh watch-0 # watch processor on SLR 0
./set_gbt.sh watch-1 # watch processor on SLR 1
test readout with DUNE-DAQ¶
First ensure you are able to ssh into the machine you are working in, as nanorc will run the various DAQ modules as localhosts.
A custom command to raw record data is provided: flxlibs/scripts/integration-tests/record-cmd.json
, which should be copied in the work area.
Test ADC recording with the internal FELIX emulator¶
to generate a 10 ADC link configuration run the following command:
python -m flxlibs.app_confgen -n 10 -t 0 -m "0-4:0-4" -e -E app_flx_10
-e
enables generation of fake timestamps when using fake data patterns and -E
enables the internal emulator.
Then run the following:
./test-communication.sh
nanorc app_flx_10 test-emu
boot conf start <run number>
info_333.json
:
num_blocks_processed
num_chunks_processed
num_subchunks_processed
rate_blocks_processed
rate_chunks_processed
num_blocks_processed_with_error
num_chunks_processed_with_error
num_short_chunks_processed_with_error
num_subchunk_crc_errors
num_subchunk_errors
num_subchunk_trunc_errors
num_subchunks_processed_with_error
To record the raw ADC to file you need to run the expert command which runs the json file record-cmd.json
.
expert_command app_flx_10/app_flx_10/readout_app record-cmd.json
To stop the test run the following:
stop_run
scrap
quit
if the application terminates correctly nanorc will print an exit code 255.
Test ADC recording with the optical links¶
run the following command:
python -m flxlibs.app_confgen -n 10 -t 0 -m "0-4:0-4" -e app_flx_10
./test-communication.sh
femu -d 0 -n
femu -d 1 -n
nanorc app_flx_10 test-fe
boot conf start <run number>
To record the raw ADC to file:
expert_command app_flx_10/app_flx_10/readout_app record-cmd.json
To stop the test run the following:
stop_run
scrap
quit
Test TP dataflow with the wibulator¶
run the following command:
python -m flxlibs.app_confgen -n 0 -t 2 -m "5:5" -e app_flx_2
./test-communication.sh
femu -d 0 -n
femu -d 1 -n
nanorc app_flx_2 test-wib
boot conf start <run number>
./set_wibulator.sh
watch-0
or watch-1
as an argument.
If successful you should see dataflow on the two link handlers.
To record the raw ADC to file:
expert_command app_flx_2/app_flx_2/readout_app record-cmd.json
To stop the test run the following:
stop_run
scrap
quit
Test TP dataflow with the internal emulator¶
run the following command:
python -m flxlibs.app_confgen -n 10 -t 2 -m "0-5:0-5" -e -E app_flx_12
./test-communication.sh
nanorc app_flx_12 test-emu
boot conf start <run number>
./set_gbt.sh
To record the raw ADC to file:
expert_command app_flx_12/app_flx_12/readout_app record-cmd.json
To stop the test run the following:
stop_run
scrap
quit
Test TP dataflow with the optical links¶
run the following command:
python -m flxlibs.app_confgen -n 10 -t 2 -m "0-5:0-5" -e app_flx_12
./test-communication.sh
femu -d 0 -n
femu -d 1 -n
nanorc app_flx_12 test-fe
boot conf start <run number>
Now enable the hitfinding TPG in GBT mode:
./set_gbt.sh
To record the raw ADC to file:
expert_command app_flx_12/app_flx_12/readout_app record-cmd.json
To stop the test run the following:
stop_run
scrap
quit
Test TP dataflow with nanorc
¶
Create the following nanorc configuration:
fddaqconf_gen -f -e --host-ru localhost --region-id 0 --host-df localhost -o . -s 1 --number-of-data-producers 10 --enable-firmware-tpg --enable-raw-recording --enable-tpset-writing --trigger-activity-config 'dict(prescale=500)' --trigger-candidate-config 'dict(prescale=20)' --tpg-channel-map ProtoDUNESP1ChannelMap flx-fw-json
record-cmd.json
with the following contents:
{
"data": {
"modules": [
{
"data": {
"duration": 1
},
"match": ""
}
]
},
"id": "record"
}
In a new terminal (with the dunedaq environment setup), configure the card:
./test-communication.sh
In another terminal run nanorc:
nanorc flx-fw-json/ boot test init conf start resume wait 60 stop scrap terminate
Which will boot the localhosts, initilise and configure the card, start dataflow, wait 60s stop and terminate the program. During the wait period (a small loading bar will appear), configure the firmware tpg to process data from the elinks:
./set_gbt.sh
curl -d "@record.json" -H "Content-Type: application/json" -H "X-Answer-Port: 9876" -X POST <localhost-url>:3336/command
if you are unable to run the above commands in time then extend the wait period, and if you want to record for more than 1s you can modify record-cmd.json but be warned, the data rate is very high. if successful, you will see output files for each link with a rather with file sizes of ~1GB.
Last git commit to the markdown source of this page:
Author: roland-sipos
Date: Wed Sep 27 09:53:05 2023 +0200
If you see a problem with the documentation on this page, please file an Issue at https://github.com/DUNE-DAQ/flxlibs/issues