Filtering the data

Before splitting the data, you should filter and synchronize them.

The data from the files can be erroneous, so you can apply some filters on them. You can create new filters or use the ones provided in hibou.filters. To create a new filter, you just nedd to make a function that takes in arguments the inputs data, the targets and as many keyword arguments as you want:

def rain_filter(inputs: pd.DataFrame, targets: pd.DataFrame, max_value: float = 0.2):
    """Filters out all time steps whose rainfall is greater than the given value (0.2 by default)."""
    return (inputs[config.Inputs.RAIN] < max_value)  # returns True when the data should be kept

Then we can create a Filters instance and add the filters you want to apply on the data in it:

my_filters = filters.Filters()
my_filters.add(filters.h_thresholding)  # add a default filter
my_filters.add(filters.le_thresholding, min_value=-20)  # override min value
my_filters.add(rain_filter)  # using the default value of 0.2
# my_filters.add(rain_filter, max_value=0) : overrive max value

We can get the data from data_reader and filter them:

def open_data():
    weather_data = data_reader.open_weather_data()
    flux_data = data_reader.open_flux_data()

Warning

Be careful, as all missing values will lead to a missing row in the data, do not leave unused variables into your pd.DataFrame!

When all the unused variables have been deleted from the DataFrame we can synchronized the two and filter them:

# deleting unused variables
weather_data = weather_data[list({
    Inputs.DATETIME,
    Inputs.RAIN,
    *INPUTS_VAR
})]
flux_data = flux_data[list({
    Targets.DATETIME,
    *TARGETS_VAR
})]

# synchronizing data
inputs, targets, timestamps = preprocessing.sync(
    weather_data,
    flux_data,
    filters=my_filters,
    inputs_var=INPUTS_VAR,
    targets_var=TARGETS_VAR
)

utils.write_input(inputs, targets, timestamps)  # as sync can take several minutes, we can save the results into files

Then, the inputs and targets data are ready for split.