NuRadioReco.modules.io.LOFAR.rawTBBio module

This module implements an interface for reading LOFAR TBB data.

This module is strongly based on pyCRtools module tbb.py by Pim Schellart, Tobias Winchen, and others. However, it has been completely re-written for use with LOFAR-LIM

Author: Brian Hare

Definitions: LOFAR is split into a number of different stations. There are three main types:

1. Core Stations (CS)

2. Remote Stations (RS)

3. international stations

Each station contains 96 low band antennas (LBA) and 48 high band antennas (HBA). Each antenna is dual polarized.

Each station is referred to by its name (e.g. “CS001”), which is a string, or its ID (e.g. 1), which is an integer. In general, these are different! The mapping, however, is unique and is given in utilities.py.

There are a few complications with reading the data.

1. The data from each station is often spread over multiple files There is a class below that can combine multiple files (even from different stations)

2. It is entirely possible that one file could contain multiple stations This feature is not used, so I assume that it isn’t a problem (for now)

3. Each Station has unknown clock offsets. Technically the core stations are all on one clock, but there are some unknown cable delays. This is a difficult problem, not handled here

4. Each Antenna doesn’t necessarily start reading data at precisely the same time. The code below picks the latest start time so this problem can be ignored by the end user

5. The software that inserts metadata (namely antenna positions and calibrations) sometimes “forgets” to do its job The code below will automatically read the metadata from other files when necessary

6. LOFAR is constantly changing So..keeping code up-to-date and still backwards compatible will be an interesting challenge

7. LOFAR only has 96 RCUs (receiver control units) per station (at the moment). Each RCU is essentially one digitizer. Each antenna needs two RCS to record both polarizations. The result is only 1/3 of the antennas can be read out each time.

   LOFAR keeps track of things with two ways. First, the data is all referred to by its RCUid. 0 is the 0th RCU, ect… However, which antenna corresponds to which RCU depends on the antennaSet. For LOFAR-LIM the antenna set will generally be “LBA_OUTER”. This could change, and sometimes the antenna set is spelled wrong in the data files. (This should be handled here though)

   In the code below each RCU is referred to by ANTENNA_NAME or antennaID. These are the same thing (I think). They are however, a misnomer, as they actually refer to the RCU, not antenna. The specific antenna depends on the antenna set. For the same antenna set, however, the ANTENNA_NAME will always refer to the same antenna.

   Each ANTENNA_NAME is a string of 9 digits. First three is the station ID (not name!), next three is the group (no idea, don’t ask), final 3 is the RCU id

   For LBA_INNER data set, even RCU ids refer to X-polarized dipoles and odd RCU ids refer to Y-polarized dipoles. This is flipped for LBA_OUTER antenna set. X-polarization is NE-SW, and Y-polarization is NW-SE. antenna_response.py, which handles the antenna function, assumes the data is LBA_OUTER.

NuRadioReco.modules.io.LOFAR.rawTBBio.read_antenna_pol_flips(fname)

NuRadioReco.modules.io.LOFAR.rawTBBio.read_bad_antennas(fname)

NuRadioReco.modules.io.LOFAR.rawTBBio.read_antenna_delays(fname)

NuRadioReco.modules.io.LOFAR.rawTBBio.read_station_delays(fname)

NuRadioReco.modules.io.LOFAR.rawTBBio.decode_if_needed(input_decode)

class NuRadioReco.modules.io.LOFAR.rawTBBio.TBBData_Dal1(filename, force_metadata_ant_pos=False, forcemetadata_delays=True, metadata_dir=None)

Bases: object

A class for reading one station from one file.

However, since one station is often spread between different files, use filePaths_by_stationName combined with MultiFile_Dal1 below.

Methods

get_ITRF_antenna_positions([copy])	returns the ITRF positions of the antennas.
get_LOFAR_centered_positions([out])	returns the positions (as a 2D numpy array) of the antennas with respect to CS002.
get_all_sample_numbers()	return numpy array that contains the sample numbers of each antenna.
get_antenna_names()	return name of antenna as a list of strings.
get_antenna_set()	return the antenna set as a string.
get_data(start_index, num_points[, ...])	return the raw data for a specific antenna, as an 1D int16 numpy array, of length num_points.
get_filter_selection()	return a string that represents the frequency filter used.
get_full_data_lengths()	get the number of samples stored for each antenna.
get_nominal_data_lengths()	return the number of data samples that are usable for each antenna, accounting for different starting sample numbers.
get_nominal_sample_number()	return the sample number of the 0th data sample returned by get_data.
get_sample_frequency()	gets samples per second.
get_station_ID()	returns the ID of the station, as an integer.
get_station_name()	returns the name of the station, as a string
get_timestamp()	return the POSIX timestamp of the first data point
get_timing_callibration_delays([...])	return the timing calibration of the antennas, as a 1D np array.
get_timing_callibration_phases()	only a test function for the moment, do not use
needs_metadata()	return true if this file does not have metadata

close_file

close_file()

needs_metadata()

return true if this file does not have metadata

get_station_name()

returns the name of the station, as a string

get_station_ID()

returns the ID of the station, as an integer.

This is not the same as StationName. Mapping is given in utilities

get_antenna_names()

return name of antenna as a list of strings. This is really the RCU id, and the physical antenna depends on the antennaSet

get_antenna_set()

return the antenna set as a string. Typically “LBA_OUTER”

get_sample_frequency()

gets samples per second. Typically 200 MHz.

get_filter_selection()

return a string that represents the frequency filter used. Typically “LBA_10_90

get_timestamp()

return the POSIX timestamp of the first data point

get_full_data_lengths()

get the number of samples stored for each antenna. Note that due to the fact that the antennas do not start recording at the exact same instant (in general), this full data length is not all usable returns array of ints

get_all_sample_numbers()

return numpy array that contains the sample numbers of each antenna. Divide this by the sample frequency to get time since the timestamp of the first data point. Note that since these are, in general, different, they do NOT refer to sample 0 of “get_data” in general

get_nominal_sample_number()

return the sample number of the 0th data sample returned by get_data. Divide by sample_frequency to get time from timestamp of the 0th data sample

get_nominal_data_lengths()

return the number of data samples that are usable for each antenna, accounting for different starting sample numbers. returns array of ints

get_ITRF_antenna_positions(copy=False)

returns the ITRF positions of the antennas. Returns a 2D numpy array. If copy is False, then this just returns the internal array of values

get_LOFAR_centered_positions(out=None)

returns the positions (as a 2D numpy array) of the antennas with respect to CS002. if out is a numpy array, it is used to store the antenna positions, otherwise a new array is allocated

get_timing_callibration_phases()

only a test function for the moment, do not use

get_timing_callibration_delays(force_file_delays=False)

return the timing calibration of the antennas, as a 1D np array. If not included in the metadata, will look for a data file in the same directory as this file. Otherwise returns None

get_data(start_index, num_points, antenna_index=None, antenna_ID=None)

return the raw data for a specific antenna, as an 1D int16 numpy array, of length num_points.

First point returned is start_index past get_nominal_sample_number(). Specify the antenna by giving the antenna_ID (which is a string, same as output from get_antenna_names(), or as an integer antenna_index. An antenna_index of 0 is the first antenna in get_antenna_names().

class NuRadioReco.modules.io.LOFAR.rawTBBio.MultiFile_Dal1(filename_list, metadata_dir, force_metadata_ant_pos=False, polarization_flips=None, bad_antennas=None, additional_ant_delays=None, station_delay=0.0, only_complete_pairs=True, pol_flips_are_bad=False)

Bases: object

A class for reading the data from one station from multiple files

Methods

close_file()	Properly close all the TBBData_Dal1 files.
get_ITRF_antenna_positions([out])	returns the ITRF positions of the antennas.
get_LOFAR_centered_positions([out])	returns the positions (as a 2D numpy array) of the antennas with respect to CS002.
get_all_sample_numbers()	return numpy array that contains the sample numbers of each antenna.
get_antenna_names()	return name of antenna as a list of strings.
get_antenna_set()	return the antenna set as a string.
get_data(start_index, num_points[, ...])	return the raw data for a specific antenna, as an 1D int16 numpy array, of length num_points.
get_filter_selection()	return a string that represents the frequency filter used.
get_full_data_lengths()	get the number of samples stored for each antenna.
get_geometric_delays(source_location[, out, ...])	Calculate travel time from a XYZ location to each antenna.
get_nominal_data_lengths()	return the number of data samples that are usable for each antenna, accounting for different starting sample numbers.
get_nominal_sample_number()	return the sample number of the 0th data sample returned by get_data.
get_sample_frequency()	gets samples per second.
get_station_ID()	returns the ID of the station, as an integer.
get_station_name()	returns the name of the station, as a string
get_time_from_second([out])	return the time (in units of seconds) since the second of each antenna (which should be get_timestamp).
get_timestamp()	return the POSIX timestamp of the first data point
get_timestamp_as_datetime()	return the POSIX timestamp of the first data point as a python datetime localized to UTC
get_timing_callibration_delays([out, ...])	return the timing calibration of the antennas, as a 1D np array.
get_timing_callibration_phases()	only a test function for the moment, do not use
get_total_delays([out])	Return the total delay for each antenna, accounting for all antenna delays, polarization delay, station clock offsets, and trigger time offsets (nominal sample number).
has_antenna(antenna_name)	if only_complete_pairs is False, then we could have antenna names without the data.
needs_metadata()
set_polarization_flips(even_antenna_names)	given a set of names(IDs) of even antennas, flip the data between the even and odd antennas
set_station_delay(station_delay)	set the station delay, should be a number

find_and_set_polarization_delay
set_odd_polarization_delay

Parameters:

filename_list: list

List of filenames for this station for this event

force_metadata_ant_posbool, default=False

If True, then load antenna positions from a metadata file and not the raw data file

polarization_flipslist

List of even antennas where it is known that even and odd antenna names are flipped in file. This is assumed to apply both to data and timing calibration

bad_antennaslist

Antennas that should not be used. Each item in the list is a tuple, first item of tuple is name of even antenna, second item is a 0 or 1 indicating if even or odd antenna is bad. assumed to be BEFORE antenna flips are accounted for

additional_ant_delaysdict

Each key is name of even antenna, each value is a tuple with additional even and odd antenna delays. This should rarely be needed. assumed to be found BEFORE antenna flips are accounted for

station_delayfloat

A single number that represents the clock offset of this station, as a delay

only_complete_pairsbool

If True, discards antenna if the other in pair is not present or is bad. If False, keeps all good antennas with a ‘none’ value if other antenna in pair is missing

pol_flips_are_badbool

If True, antennas that are in pol-flips are included in bad_antennas

Notes

This module always defaults to using antenna timing calibration from metadata.

Also, polarization_flips, bad_antennas, additional_ant_delays, and station_delay can now be strings that are file names. If this is the case, they will be read automatically

Methods

close_file()	Properly close all the TBBData_Dal1 files.
get_ITRF_antenna_positions([out])	returns the ITRF positions of the antennas.
get_LOFAR_centered_positions([out])	returns the positions (as a 2D numpy array) of the antennas with respect to CS002.
get_all_sample_numbers()	return numpy array that contains the sample numbers of each antenna.
get_antenna_names()	return name of antenna as a list of strings.
get_antenna_set()	return the antenna set as a string.
get_data(start_index, num_points[, ...])	return the raw data for a specific antenna, as an 1D int16 numpy array, of length num_points.
get_filter_selection()	return a string that represents the frequency filter used.
get_full_data_lengths()	get the number of samples stored for each antenna.
get_geometric_delays(source_location[, out, ...])	Calculate travel time from a XYZ location to each antenna.
get_nominal_data_lengths()	return the number of data samples that are usable for each antenna, accounting for different starting sample numbers.
get_nominal_sample_number()	return the sample number of the 0th data sample returned by get_data.
get_sample_frequency()	gets samples per second.
get_station_ID()	returns the ID of the station, as an integer.
get_station_name()	returns the name of the station, as a string
get_time_from_second([out])	return the time (in units of seconds) since the second of each antenna (which should be get_timestamp).
get_timestamp()	return the POSIX timestamp of the first data point
get_timestamp_as_datetime()	return the POSIX timestamp of the first data point as a python datetime localized to UTC
get_timing_callibration_delays([out, ...])	return the timing calibration of the antennas, as a 1D np array.
get_timing_callibration_phases()	only a test function for the moment, do not use
get_total_delays([out])	Return the total delay for each antenna, accounting for all antenna delays, polarization delay, station clock offsets, and trigger time offsets (nominal sample number).
has_antenna(antenna_name)	if only_complete_pairs is False, then we could have antenna names without the data.
needs_metadata()
set_polarization_flips(even_antenna_names)	given a set of names(IDs) of even antennas, flip the data between the even and odd antennas
set_station_delay(station_delay)	set the station delay, should be a number

find_and_set_polarization_delay
set_odd_polarization_delay

set_polarization_flips(even_antenna_names)

given a set of names(IDs) of even antennas, flip the data between the even and odd antennas

set_odd_polarization_delay(new_delay)

set_station_delay(station_delay)

set the station delay, should be a number

find_and_set_polarization_delay(verbose=False, tolerance=1e-09)

close_file()

Properly close all the TBBData_Dal1 files.

needs_metadata()

get_station_name()

returns the name of the station, as a string

get_station_ID()

returns the ID of the station, as an integer. This is not the same as StationName. Mapping is given in utilities

get_antenna_names()

return name of antenna as a list of strings. This is really the RCU id, and the physical antenna depends on the antennaSet

has_antenna(antenna_name)

if only_complete_pairs is False, then we could have antenna names without the data. Return True if we actually have the antenna, False otherwise. Account for polarization flips.

get_antenna_set()

return the antenna set as a string. Typically “LBA_OUTER”

get_sample_frequency()

gets samples per second. Typically 200 MHz.

get_filter_selection()

return a string that represents the frequency filter used. Typically “LBA_10_90

get_timestamp()

return the POSIX timestamp of the first data point

get_timestamp_as_datetime()

return the POSIX timestamp of the first data point as a python datetime localized to UTC

get_full_data_lengths()

get the number of samples stored for each antenna. Note that due to the fact that the antennas do not start recording at the exact same instant (in general), this full data length is not all usable returns array of ints

get_all_sample_numbers()

return numpy array that contains the sample numbers of each antenna. Divide this by the sample frequency to get time since the timestamp of the first data point. Note that since these are, in general, different, they do NOT refer to sample 0 of “get_data”

get_nominal_sample_number()

return the sample number of the 0th data sample returned by get_data. Divide by sample_frequency to get time from timestamp of the 0th data sample

get_nominal_data_lengths()

return the number of data samples that are usable for each antenna, accounting for different starting sample numbers. returns array of ints

get_ITRF_antenna_positions(out=None)

returns the ITRF positions of the antennas. Returns a 2D numpy array. if out is a numpy array, it is used to store the antenna positions, otherwise a new array is allocated. Does not account for polarization flips, but shouldn’t need too.

get_LOFAR_centered_positions(out=None)

returns the positions (as a 2D numpy array) of the antennas with respect to CS002. if out is a numpy array, it is used to store the antenna positions, otherwise a new array is allocated. Does not account for polarization flips, but shouldn’t need too.

get_timing_callibration_phases()

only a test function for the moment, do not use

get_timing_callibration_delays(out=None, force_file_delays=False)

return the timing calibration of the antennas, as a 1D np array.

If not included in the metadata, will look for a data file in the same directory as this file. Otherwise returns None. If out is a numpy array, it is used to store the antenna delays, otherwise a new array is allocated. This takes polarization flips, and additional_ant_delays into account (assuming that both were found BEFORE the pol flip was found). Also can account for a timing difference between even and odd antennas, if it is set.

get_total_delays(out=None)

Return the total delay for each antenna, accounting for all antenna delays, polarization delay, station clock offsets, and trigger time offsets (nominal sample number). This function should be preferred over ‘get_timing_callibration_delays’, but the offsets can have a large average. It is recommended to pick one antenna (on your reference station) and use it as a reference antenna so that it has zero timing delay. Note: this creates two definitions of T=0. I will call ‘uncorrected time’ is when the result of this function is used as-is, and a reference antenna is not chosen. (IE, the reference station can have a large total_delay offset), ‘corrected time’ will be otherwise.

get_time_from_second(out=None)

return the time (in units of seconds) since the second of each antenna (which should be get_timestamp). accounting for delays. This is literally just the opposite of get_total_delays

get_geometric_delays(source_location, out=None, antenna_locations=None)

Calculate travel time from a XYZ location to each antenna.

out can be an array of length equal to number of antennas. antenna_locations is the table of antenna locations, given by get_LOFAR_centered_positions(). If None, it is calculated. Note that antenna_locations CAN be modified in this function. If antenna_locations is less then all antennas, then the returned array will be correspondingly shorter. The output of this function plus??? get_total_delays plus emission time of the source is the time the source is seen on each antenna.

get_data(start_index, num_points, antenna_index=None, antenna_ID=None)

return the raw data for a specific antenna, as an 1D int16 numpy array, of length num_points.

First point returned is start_index past get_nominal_sample_number(). Specify the antenna by giving the antenna_ID (which is a string, same as output from get_antenna_names()) or as an integer antenna_index. An antenna_index of 0 is the first antenna in get_antenna_names().