🤖 SCRIPTS

🤖 SCRIPTS#

Obtain file paths#

Before starting with the analysis you may need to get the paths of the HDF5 files you want to use. They could be already availbale in the folder: /eos/experiment/neutplatform/protodune/experiments/ProtoDUNE-VD/ruciopaths/ (for NP02) or you can get the rucio_path using the tools listed below:

  • setup_rucio_a9.sh: script to setup the rucio environment variables in your terminal. After that you can produce all the paths you need without re-authenticating.

  • fetch_rucio_replicas.py: script that wraps the needed tools to save the paths of the HDF5 files you want to decode.

This can be summarized in the following steps:

Get path for several runs (without re-authenticating)#

For using these scripts you need valid FNAL credentials. Have a look at the RUCIO DOCS for more details of this process.

source setup_rucio_a9.sh
python fetch_rucio_replicas.py --runs <run_number>, <run_number2>,...

.. warning:: The following five paragraphs (00_HDF5toROOT, C++, Python, 01_Process, 02_RawAna) are valid only for NP04 data

00_HDF5toROOT#

We have developed two decoders (C++ and Python) which output is a root file with the same structure. After running them check /eos/experiment/neutplatform/protodune/experiments/ProtoDUNE-II/PDS_Commissioning/waffles/2_daq_root/run_YOUR_RUN_NUMBER/ for the output root files.

To use the decoders can follow the next instructions:

C++#

  1. Compile the scripts (just the first time)

cd scripts/cpp_utils # Go to the C++ scripts folder
./compile_decoder.sh # Run the script to compile the C++ scripts (will clone the HDF5 library and compile it together with the decoder)

  1. Run the decoder

cd ..               # Go back to the scripts folder
sh 00_HDF5toROOT.sh # Run the bash script to manage the C++ macros

# You can also give the run number as an argument (sh 00_HDF5toROOT.sh 27632)

During the execution of the bash script you will be asked for the run number to be processed:

[INFO] Welcome to the script decoding the HDF5 files using the CPP TOOLS [make sure you have compiled the decoder]!. To execute the script just run: sh 00_HDF5toROOT.sh. Optionally:

* [1st argument] the run number (separated by commas: 02ABC,02XYZ),

* [2nd,3rd] first and number of hdf5 files to process (default 0, -1 --> ALL) and

* [4th] path to save the root files (default /eos/experiment/neutplatform/protodune/experiments/ProtoDUNE-II/PDS_Commissioning/waffles/2_daq_root/).

You can run the script without any flag and the required arguments will be asked

Example: sh 00_HDF5toROOT.sh 27644,27645 3 5 --> will process 5 files starting from the 4rd (index 3) file of the run 27644 and 27645 and save the output in the default path

Enjoy! :)

The user should input the run number (27632 or a list as 26755,26756) and choose the running mode (1=decoder, 2=duplications check):

  • Please provide a run(s) number(s) to be analysed, separated by commas :)

  • Do you want to run the DUMP an output root file in eos (1) or just a DUPLICATION check (2)? (1/2)

The outputs will be saved in the default eos path /eos/experiment/neutplatform/protodune/experiments/ProtoDUNE-II/PDS_Commissioning/waffles/2_daq_root/run_XXXX or in the path provided by the user.

Tip

CPP UTILS

Once you have compiled the cpp macros (HDF5toROOT_decoder and HDF5LIBS_duplications) you can use them from the command line from wherever you are. The bash script 00_HDF5toROOT.sh is just a wrapper to manage the input arguments and the output paths. Therefore if you have an output .hdf5 file not included in rucio you can decode it by running: HDF5toROOT_decoder /path/to/your/file.hdf5

Python#

  1. Run the decoder

python 00_HDF5toROOT.py # Run the Python script to decode the HDF5 files

# You can also give the run number as an argument (python 00_HDF5toROOT.py --runs 27632)

In order to use this decoder you need to have ROOT installed in your virtual environment. If you don’t have it, you can install it by running:

UNDER TESTING

# INSIDE THE VIRTUAL ENVIRONMENT (you have source env.sh before)

git clone --branch latest-stable --depth=1 https://github.com/root-project/root.git root_src

mkdir root_build root_install && cd root_build

cmake -DCMAKE_INSTALL_PREFIX=../root_install -Ddataframe=OFF ../root_src

cmake --build . -- install -j1

If some error appears during the installation related with Roofit, just disable it by adding the command -Droofit=OFF on the cmake.

After that, every time you log in, you need to source ROOT, or you can edit env.sh and add this command:

source ../root_install/bin/thisroot.sh

01_Process#

This script is used to convert the raw .hdf5 files into waffles classes and save it locally in .pkl format. In order to run this script make sure you have a waffles/data folder to store the output files. In summary what this script does is:

import waffles.input_output.raw_hdf5_reader as reader

rucio_files = f"/eos/experiment/neutplatform/protodune/experiments/ProtoDUNE-II/PDS_Commissioning/waffles/1_rucio_paths/028602.txt"
allfilepath = reader.get_filepaths_from_rucio(rucio_filepath)
waveformset = reader.WaveformSet_from_hdf5_files( filepaths[:int(file_lim)], read_full_streaming_data = False)
with open(f"data/{run}_full_wfset_raw.pkl", "wb") as f:
            pickle.dump(wfset, f)

Some decorators are included to require the user inputs, such as the run number and the number of files to process. This allows the analysers to have a common starting point and avoid the need to change the code every time they want to process a new run.

02_RawAna#

Following the same structure as the previous script, this one reads the WaveformSet objects stored in the .pkl files from 01_Process and performs some basic analysis. The goal of this script is to provide the user quick visualization of the BasicAnalysis output channel by channel. Make sure you have a waffles/config/run_number.json folder and file with the input parameters for the analysis. A template for these configuration files could be:

{
    "standard": {                                       #analysis_label
                    "base_lim": [0, 70, 900, 1000],     #limits for the baseline computation
                    "int_ll": 50,                       #lower limit for the integral computation
                    "int_ul": 130,                      #upper limit for the integral computation
                    "amp_ll": 0,                        #lower limit for the amplitude computation
                    "amp_ul": 1000                      #upper limit for the amplitude computation
                },
    "avanced": { ... }
}

After filtering the full WaveformSet object, a plot will be displayed using plot_ChannelWsGrid from the waffles.plotting module. This plot will show 10 Waveform objects together with the baseline and analysis limits. Additionally, the analyser can plot the waveforms objects using the plot_WaveformAdcs function from the same module with a loop as follows:

    for wf in filter_wfset.waveforms:
        plot_WaveformAdcs(  wf, figure = figure,
                            plot_analysis_markers = True,
                            show_baseline_limits = True, 
                            show_baseline = True,
                            show_general_integration_limits = True,
                            show_general_amplitude_limits = True,
                            show_spotted_peaks = True,
                            show_peaks_integration_limits = False)

07_save_structured_from_config.py#

This script is used to convert the raw .hdf5 files into waffles classes and save it in a compressed .hdf5 format.

Contents