Process Tall Tower Data

This R Markdown processes raw data from the Tall Tower database. The processing consists of reading the .csv input file with raw data and computing the average value between measurements from sensors (humidity,pressure, wind direction and speed) installed at different booms of the mast at the same height. It requires data in .csv format derived this repository: Tower2csv

Let’s start by simply reading the data.

df <- read.csv(file="2009.csv", header=TRUE)

Now, we continue by visualizing the data frame. If browsing the data frame by clicking in the black right arrow, it’s possible to see that there is more than one sensor for each parameter. For instance, there are three wind direction sensors at 116 m (S1, S2 and S3), namely, wdiragl116S1, wdiragl116S2 and wdiragl116S3.

head(df)

Next, a function to process the data named “processData” is defined. It averages data from different sensors at the same height. This function requires two inputs: (i) a data frame, (ii) and a list of heights. It is worth noting that the average can be directly computed because the Tall Tower database has been pre-processed and the mast shadow effect was previously fixed.

processData <- function(df,allHeights){
## FUNCTION TO PROCESS DATA ####################################################
# Defining the sensors' names
  allSensors = list("huragl","psagl","taagl","wdiragl","windagl")
# New data frame to receive processed data
  dfProcessed <- df["time"]

  for (sensor in allSensors){
    for (height in allHeights){
    # Sensor full name
      sensorFullName = paste(sensor, height, sep = "")
    # All columns from "sensor" at the same height 
      columns = grep(sensorFullName, colnames(df), value = TRUE)
      
    # Checking if the columns was found
      if (length(columns) == 0) { # If none columns was found
        print(paste(sensorFullName," column was not found"))
        next # Continue with the next iteration step
      }
      print(paste("Reading", toString(columns)))
      data_tmp = df[columns]  # Data from "sensor"
      
    # Computing the mean value between different sensors at the same height
      dfProcessed[sensorFullName] <- rowMeans(data_tmp, na.rm = T)
    }
  }
  return(dfProcessed)
}

In the next code cell/chunk we call the function that displays which sensor was found.

allHeights = list(21,70,116) # list of heights
dfProcessed <- processData(df,allHeights)

[1] "Reading huragl21S1"
[1] "Reading huragl70S1"
[1] "Reading huragl116S1"
[1] "Reading psagl21S1"
[1] "psagl70  column was not found"
[1] "psagl116  column was not found"
[1] "Reading taagl21S1"
[1] "Reading taagl70S1"
[1] "Reading taagl116S1"
[1] "Reading wdiragl21S1, wdiragl21S2, wdiragl21S3, wdiragl21S4"
[1] "Reading wdiragl70S1, wdiragl70S2, wdiragl70S3, wdiragl70S4"
[1] "Reading wdiragl116S1, wdiragl116S2, wdiragl116S3, wdiragl116S4"
[1] "Reading windagl21S1, windagl21S2, windagl21S3"
[1] "Reading windagl70S1, windagl70S2, windagl70S3"
[1] "Reading windagl116S1, windagl116S2, windagl116S3"

In the following, we visualize the processed data.

head(dfProcessed)

Finally, the processed data frame can be saved or exported as .csv file by using the following command lines

## Exporting dataframe to a .csv file ##########################################
saveFileName = paste(getwd(),'/',"2009_avg",".csv", sep = "")
write.csv(dfProcessed, saveFileName, row.names=FALSE, quote=FALSE, 
          fileEncoding = "UTF-8")