Modules

In NuRadioReco, the process of reconstruction event properties from data is done by modules. In principle, each step of an analysis should be done by a dedicated module. For example, one module applies a filter to the channel spectrum and a different module reconstructs the electric field from that channel afterwards. In the case of simulation studies, modules can also be used to simulate aspects of the detector response, e.g. the efieldToVoltageConverter, which calculates the voltages in the antennas from the electric field at each antenna.

In principle, modules can be arranged in any order, including loops and if/or branching, though some some modules may require others to be run beforehand. For example, using a module to apply a filter to an electric field only makes sense if the field has been reconstructed by another module beforehand. It is even possible to combine several modules into a single module, for example to make a standard reconstruction model, that performs a reconstruction with the default settings deemed best for a given experiment.

Basic Module Structure

Each module consists of four components:

  • A constructor to create the module. In a reconstruction, this is usually called to create the modules before looping over all events in the event file.

  • The begin method. This is used to specify settings that will not change on an event-by-event basis.

  • The run method. This is executed for each event and executes the task that the module was built for.

  • The end Method. This is run after the last event was processed for cleanup or to print information on what the module has done.

These methods should all be called in that order, though the begin and end function can be skipped for some modules.

Logging

In version 2.3.0, the way in which we handle logging changed. We switched to a custom logging class, which avoids any potential clashes with other packages. This custom class is set as the logging class when using NuRadioMC/NuRadioReco. Furthermore, we now create the NuRadioMC and NuRadioReco loggers automatically when importing the packages. This means that no particular setup is required from the user, as long as the module loggers follow the general naming scheme “NuRadioMC.MODULE” or “NuRadioReco.MODULE”.

Tip

If you are writing a module for a specific experiment, it is recommended to follow the naming scheme “NuRadioReco.EXPERIMENT.MODULE” to avoid any potential clashes with other experiments. It also allows to adjust the logging level for all modules of a specific experiment at once, by manually the settings of the logger “NuRadioReco.EXPERIMENT”.

Note

Please note that here we only show examples from NuRadioReco. For NuRadioMC applications one should simply replace NuRadioReco with NuRadioMC in the following.

Logging in NuRadioReco is handled using a custom class called NuRadioLogger, which inherits from logging.Logger . We use the fact that loggers can inherit handlers from logger created in parent modules. To achieve this behaviour, every module file MODULE.py should initialise a logger using

import logging

logger = logging.getLogger('NuRadioReco.MODULE')

The logger can then be used across the module and will report using the name given in the getLogger() function. Notice that we do not set the logging level explicitly in the module. This ensures that the logging level is inherited from the parent logger, which allows users to easily control the logging level of all modules at once. If you want to provide the functionality to change the logging level of the module, for example in the begin() function, please make sure the default is logging.NOTSET.

logger = logging.getLogger('NuRadioReco.MODULE')

class MODULE:
    def __init__(self):
        self.logger = logger

    def begin(self, logging_level=logging.NOTSET):
        self.logger.setLevel(logging_level)

The parent logger, with the name “NuRadioReco”, is automatically initialised when importing NuRadioReco. It is this parent logger which processes all the logging events from the NuRadioReco modules, and it determines the overall logging level of the script that is inherited by all modules. By default the level is set to STATUS (see below), but this can easily be changed by using the set_general_log_level() function from the NuRadioReco logging module, located in NuRadioReco/utilities/logging.py . This allows to turn DEBUG on for all, for example. It is still possible to change the logging level for individual modules as an overwrite. This can be useful when debugging a particular module, as in the following example. Here we first set the general logging level to ERROR and then use the begin() function to set the logging level of the module to DEBUG. This will show DEBUG messages for this module, while only showing ERROR messages for all other modules.

from NuRadioReco.utilities.logging import set_general_log_level
import logging

set_general_log_level(logging.ERROR)  # set all modules logging level to ERROR

module = MODULE()
module.begin(logging_level=logging.DEBUG)  # show DEBUG messages for this module

Next to the standard logging levels that Python provides, NuRadioReco implements an additional level called STATUS. Its value is defined in NuRadioReco/utilities/logging.py as LOGGING_STATUS. As of February 2024, this value is 25, which is between WARNING and INFO (meaning that it will only be printed if the logging level is set to STATUS, INFO or DEBUG). In order to log a message with the STATUS level, you can use the my_logger.status() method.

Another interesting feature of the Python logging module, is the option to add multiple handlers which each output to a different location. Each of these can have a different logging level and can be formatted differently. To apply the same formatting as the NuRadioReco logger, you can use the get_fancy_formatter() function from the NuRadioReco/utilities/logging.py module. Regarding the logging level, please note that the logger’s level will take priority (this diagram shows the control flow).

For example, if you wanted to save all the logging statements to a file for later reference, you can add a FileHandler to the general logger in your script using

from NuRadioReco.utilities.logging import get_fancy_formatter, set_general_log_level
import logging

set_general_log_level(logging.DEBUG)  # set all modules logging level to DEBUG

# create the handler, set the level and apply same formatting as NuRadioReco logger
f_handler = logging.FileHandler('debug.log')
f_handler.setLevel(logging.DEBUG)
f_handler.setFormatter(get_fancy_formatter())

# retrieve parent logger and add file handler to it
logger = logging.getLogger("NuRadioReco")
logger.addHandler(f_handler)

In the code block above we create a handler which will direct log statements to a file called debug.log, configure that handler to take all logging output with level DEBUG or above and format them using the same formatting as the NuRadioReco logger. We then add that handler to the NuRadioReco logger. Notice that we also had to set the general logging level to DEBUG, to ensure that DEBUG messages will actually be logged. Another way could be to set the logging level of some specific modules to DEBUG. It also possible to set a different formatter to this handler, or add more handlers to the parent logger.

Hint

If you wish to log records with a level lower than the general logging level to a file, you can edit the StreamHandler of the NuRadioReco logger which is created by default (you should be able to retrieve this one via parent_logger.handlers[0]). The idea would be to set the logger’s level as low as possible, but up the level of the StreamHandler to only output messages of a certain level or higher. The level of the FileHandler would then be set to the lowest level you want to log to the file. Note that this might mess up the standard logging behaviour, so only do this when you know what you are doing.