Reading data
============
The opening of the files that contains the data are on your own. But there's some care to take.

In this guide we assume that the data from the average weather stations and the data from the fluxes stations are into two separate files. The whole programm use the ``pandas.DataFrame`` object. So you should use ``pandas`` to open the files.

Let's start with the ``data_reader.py`` script and the header::
    
    from datetime import datetime
    
    import pandas as pd
    import numpy as np
    
    from hibou import utils, config

* ``datetime`` for time parsing

* ``pandas`` for opening and storing data

* ``numpy`` for vectorize operations here

* ``hibou`` provides some tools to compute sinusoidal time parameters and MetPy calculations

Once the header is done, we can start by defining our variables to train the artificial neural network (ANN). This step allows ``hibou`` to access the defined variables during the preprocessing of the data. These variables names are stored into the ``config`` module and more precisely into ``config.Inputs`` and ``config.Targets`` classes::

    # redefining variables names for inputs
    config.Inputs.TEMPERATURE = "Ts_Avg"
    config.Inputs.PRESSURE = "Pa_Avg"
    config.Inputs.WIND_SPEED = "wind_speed"
    config.Inputs.RAIN = "P_mm_Avg"  # we can even define new variables
    config.Inputs.MIXING_RATIO = "mr_Avg"
    config.Inputs.RELATIVE_HUMIDITY = "RelHum_Avg"
    
    # redefining variables names for targets
    config.Targets.H = "sensible_flux"
    config.Targets.LE = "latent_flux"

Then, just before opening files, we create a small function that cast a string into a ``datetime`` object::

    def parse_date(timestamp: str):
        """Cast a string formatted like YYYY-MM-DD HH:MM:SS into a datetime object."""
        return datetime.strptime(timestamp, "%Y-%m-%d %H:%M:%S")
        
Once this step is done, we can create two functions (one per file)::
    
    def open_weather_data():
        data = pd.read_csv("data/weather_data.csv")
        
        # parse dates
        data[config.Inputs.DATETIME] = np.vectorize(parse_date)(data["date"])
        
        # converting pressure from kPa to hPa
        data[config.Inputs.PRESSURE] = data[config.Inputs.PRESSURE] * 10
        
        # reconstruct wind components
        data[config.Inputs.U] = - data[config.Inputs.WIND_SPEED] * np.sin(data["wind_dir"] * np.pi / 180)
        data[config.Inputs.V] = - data[config.Inputs.WIND_SPEED] * np.cos(data["wind_dir"] * np.pi / 180)
        
        # sinusoidal time parameters
        (
            data[config.Inputs.ANNUAL_SIN],  # even if the columns doesn't exists, they will be created automatically
            data[config.Inputs.ANNUAL_COS],
            data[config.Inputs.DAILY_SIN],
            data[config.Inputs.DAILY_COS]
        ) = utils.get_sinusoidal_time_parameters(data[config.Inputs.DATETIME].values)

    return data

The second function is very similar to the first one::

    def open_flux_data():
        data = pd.read_csv("data/flux_data.csv")
        
        # parse dates
        data[config.Targets.DATETIME] = np.vectorize(parse_date)(data["date"])
        
        return data