By Ladislav Šigut, Postdoc, Department of Matter and Energy Fluxes, Global Change Research Institute CAS, Czechia
In this blog, we introduce several publicly available software/packages that the flux scientists/engineers have developed (or are developing) for data acquisition/processing/QAQC/manipulation. The list is intended for information sharing and may not be comprehensive. We welcome any feedback or comment to help improve the list.
We thank all the original software/package developers for sharing and maintaining such great services. For details of each software/package, please refer to the developing team through the external links. Please contact the blog author Ladislav Šigut or editor Housen Chu if you’d like to add/revise any to this list.
[First publish: October, 2017]
[Last update: November, 2017] — add ‘FluxnetLSM’ R package
[Last update: January, 2018] — add ‘bigleaf’ R package
[Last update: April, 2018] — add ‘Fluxpart‘ Python library
[Last update: October, 2018] — add ‘openeddy‘, ‘phenocamr‘, ‘daymetr‘, ‘MODISTools‘ R packages, ‘ONEFlux‘, ‘daymetpy‘ Python libraries
[Last update: April, 2019] — add ‘esmwfr‘ R package
[Raw Data Processing/QAQC]
The following software and packages are generally designed for data acquisition and/or processing raw (high frequency) flux data. While most of them share the common feature of flux calculation, each of them differs in terms of the extended features provided (e.g., QAQC, footprint analysis)
- Allows acquisition, visualization and processing of eddy covariance data
- Software itself can be obtained through direct contact with the authors (Kolle, O. and Rebmann, C.)
- OS support for Windows
- Application for processing eddy covariance data. It computes fluxes of water vapor (evapotranspiration), carbon dioxide, methane, other trace gases, and energy with the eddy covariance method. Includes three footprint models and other features.
- OS support for Windows, Mac and Linux. (While executables are only for Win and Mac, both the engine and the GUI can be compiled from source code (openly available) and used under any Linux distribution)
- Application for processing eddy covariance data. It computes fluxes of water vapor (evapotranspiration), carbon dioxide, methane, other trace gases, and energy with the eddy covariance method. Includes extended quality assurance tools and other features.
- OS support for Windows and Linux.
- Software for post-processing eddy covariance data. The software is a MATLAB based program with a graphical user interface. EddyUH can process data obtained from various different sonic anemometer and gas analyser combinations and from various different measurement sites. The software can process CO2, H2O, temperature, O3, CH4, N2O and particle high frequency data.
- OS support for Windows, Mac and Linux.
- EdiRe is a fast, flexible,and user friendly software tool for micrometeorologists with a focus on eddy covariance and microclimatological measurement analysis. EdiRe is adaptable to most eddy covariance raw data formats and capable of incorporating microclimate data.The graphical user interface simplifies development of processing routines and allows rapid redesign of routines to enhance question/answer cycle of data analysis.
- OS support for Windows.
- the first EC flux processor building on the open-source, reproducible and community-extensible DevOps software development model. Amongst others it produces NEON’s first EC data, and provides an environment for advanced end-to-end analytics and modelling for EC flux and rich contextual data (e.g., Metzger et al., 2013; Vaughan et al., 2016; Xu et al, 2017).
- R software based – OS support for Windows, Mac and Linux.
- Encompasses eddy-covariance calculation scheme which is a fork of the well-known EC-PACK and a quality control and assessment (QC/QA) scheme developed in course of the German TERENO project.
The following software and packages are generally designed for taking in post-processing data (i.e., calculated half-hourly flux and met data), generating value-added data products (e.g., gap-filled, partitioned gross primary production/ecosystem respiration, footprint) and facilitating further QAQC.
- R package (accessible also through online interface) that allows application of ustar filtering, gap-filling and flux partitioning of eddy covariance data.
MDI Meteo gap-filling tool: http://www.bgc-jena.mpg.de/~MDIwork/meteo/index.php
- This meteo gap filling tool takes advantage of the re-analyzed meteorological fields provided by ECMWF. The local site conditions are related to the gridded spatial data from their ERA-Interim product. The mapping from grid to point is performed by multiple linear regression with the nearest nine pixels.
- A user-friendly open source package GaFiR was developed to fill data gaps in turbulent flux observations of evapotranspiration and carbon dioxide.
- R package allowing despiking, ustar filtering, plotting, footprint modelling and general post-processing of a standard EddyPro full output file.
- Developed in Python, but the package can be used via a simple GUI and the editing of text files.
- PyFluxPro includes the functions of QAQC, corrections (e.g., calibration drifts), gap filling of meteorological and flux data, u* threshold detection (Barr’s method), GPP/RECO partitioning (daytime/nighttime approaches).
- Met data gap-filling can be done using data from a paired flux tower, from automated weather stations (AWS) operated as part of the global observing network, from the ERA-Interim re-analysis data set and from high spatial and temporal numerical weather prediction (NWP) models where available. Flux data gap-filling (H, LE and FC) is done using a robust neural network (Hsu et al., 2002).
- Its predecessor (OzFluxQC) is described in a recent paper in the OzFlux Special Issue of Biogeosciences (https://www.biogeosciences.net/14/2903/2017/).
- Python 3 module for partitioning water vapor and carbon dioxide fluxes.
- Implements the Scanlon and Sahu (2008) procedure for partitioning eddy covariance measurements of water vapor and carbon dioxide fluxes into stomatal (transpiration, photosynthesis) and nonstomatal (evaporation, respiration) components.
- Includes capabilities for applying basic QA/QC to high-frequency eddy covariance data, for correcting high frequency data for external fluctuations associated with air temperature and vapor density, and for estimating leaf-level water use efficiency.
- The module was recently described in Agricultural and Forest Meteorology (https://doi.org/10.1016/j.agrformet.2018.02.019).
- R package for eddy covariance data handling, quality checking (similar to Mauder et al., 2013), processing, summarising and plotting. It aims to standardise the automated quality checking and make data processing reproducible.
- The vignette for the package is in development but you can already try it out on an example dataset using the workflow files (1. QC, 2. Storage, 3. GF & FP, 4. Summary) (https://github.com/lsigut/EC_workflow)
- Although most of the functions have general application, the processing is tuned to EddyPro output files. Together with REddyProc, openeddy forms complete processing chain that produces annual sums of fluxes with further variables, statistics and visualization.
- Developed in Python version 2.7 (it should work with Python version 3.5 or later, but was not fully tested with these versions).
- ONEFlux (Open Network-Enabled Flux processing pipeline) consolidates multiple computations to process half-hourly (or hourly) flux inputs in an automatic fashion, including friction velocity threshold estimation methods and filtering, gap-filling of micrometeorological and flux variables, partitioning of CO2 fluxes into ecosystem respiration and gross primary production, uncertainty estimates, and more.
- The current version of the code is compatible with the code base used to create the FLUXNET2015 dataset. The steps implemented are detailed in the data processing description page.
- R package for post-processing FLUXNET2015 and La Thuile data for use in land surface modelling
- Package transforms the data into community standard NetCDF files that are directly readable by LSMs
- It also provides multiple methods for gap-filling and quality controlling the data
- R package primarily developed for the analysis of air pollution measurement data but which is also of more general use in the atmospheric sciences (e.g. creation of wind rose).
- The package consists of many tools for importing and manipulating data, and undertaking a wide range of analyses to enhance understanding of air pollution data.
Kljun footprint model: reference – https://www.geosci-model-dev.net/8/3695/2015/
- Codes: http://footprint.kljun.net/ (Matlab, R, Python)
- Online data processing: http://geography.swansea.ac.uk/nkljun/ffp/www/
[Auxiliary Data Processing]
The following software and packages are generally designed to process/visualize/analyze ancillary data that are often collected at the flux tower site or are relevant in analyzing flux data.
- R package for the derivation of physical and physiological bulk ecosystem properties from eddy covariance data
- functions for calculating surface conductance, aerodynamic conductance, decoupling, surface temperature and humidity, potential evapotranspiration, intrinsic water-use efficiency, bulk intercellular CO2 concentration, etc.
- The package focuses on visualising large amounts of hourly environmental data
- See examples e.g. here:
- Calculation of Cumulative Basal Area Inflection (CuBI) height. CuBI height is the representative canopy height of a forest stand determined as the height of the inflection point of a sigmoid-shaped relationship between tree height and cumulative basal area.
- Available as R and Matlab software.
- R package for phenology from digital images of vegetation. It contains the most up-to-date processing techniques along with novelties including the combination of different fit/phenophase extraction methods
- R package facilitates the retrieval and post-processing of PhenoCam time series.
- The post-processing of PhenoCam data includes outlier removal and the generation of data products such as phenological transition dates.
- If requested complementary Daymet climate data will be downloaded and merged with the PhenoCam data for modelling purposes.
- For a detailed overview of the assumptions made during post-processing, please refer to publications by Hufkens et al. (2018) and Richardson et al. (2018).
- An open-source, interactive tool for processing sap flux data from thermal dissipation probes. Baseliner enables users to QA/QC data and process data using a combination of automated steps, visualization, and manual editing.
- Matlab software.
Landflux tools: http://landflux.org/Tools.php
- Provides various Excel sheet based tools mostly with ecophysiological relevance.
- Contains sheets for A/Ci Curve Fitting, Pressure/volume curvefitting, Photosynthetic Light Response curve fitting, Keeling Plot Curve Fitting, Leaf Energy Balance, Two-Pool Decomposition Model 4.0, Uncertainty in Partitioning Using Two-Source Mixing Models, Green Fraction from Digital Images, Leaf Transpiration Calculator
- R package to the Daymet web services (http://daymet.ornl.gov/).
- Allows for easy downloads of Daymet climate data directly to your R workspace or your computer. Routines for both single pixel data downloads and gridded (netCDF) data are provided.
- R package allowing for easy downloads of ECMWF (European Centre for Medium-Range Weather Forecasts) data (http://apps.ecmwf.int/datasets/).
- Python library for easy, integrated access to gridded daily Daymet weather data.
- R package for programmatic interface to the ‘MODIS Land Products Subsets’ web services (https://modis.ornl.gov/data/modis_webservice.html).
- Allows for easy downloads of ‘MODIS’ time series directly to your R workspace or your computer.
About Ladislav Šigut — A postdoc at the Department of Matter and Energy Fluxes in the Global Change Research Institute CAS (Brno, Czechia). His main research interests are plant ecophysiology, CO2 fluxes on the leaf and ecosystem scale and their response to micrometeorological parameters, plant acclimation to different environmental parameters and application of eddy covariance method in the mountainous ecosystems. Currently he is developing a R package for reproducible quality checking of eddy covariance data and a flux partitioning tool that combines advantages of both nighttime and daytime approaches.