Date: Aug 31, 2020 Version: 3.0
BestiaPop: A python package to automate the extraction and processing of climate data for crop modelling¶
Climate data is an essential input for crop models to predict crop growth and development using site-specific (point) or gridded climate data. However, Crop Modelling software expects input data to be encapsulated in custom file formats (
WTH, etc.) which don’t conform to a common standard and require various customizations, depending on the prediction engine that generates crop models. Moreover, source data providers like SILO or NASA POWER are usually neutral in the type of data output files they provide as part of their API services which leads to a gap between source raw data and processed data required by crop modelling suites to develop their models. We developed BestiaPop (a spanish word that translates to pop beast), a Python package which allows model users to automatically download SILO’s (Scientific Information for Land Owners) or NASAPOWER gridded climate data and convert this data to files that can be ingested by Crop Modelling software like APSIM or DSSAT.
The package offers the possibility to select a range of grids (5 km × 5 km resolution) and years producing various types of output files: CSV, MET (for APSIM), WTH (for DSSAT) and soon JSON (which will become part of BestiaPop’s API in the future).
Curently, the code downloads data from two different climate databases:
Crop Modelling Software¶
There are two major crop modelling suites in use by the scientific community in Australia: APSIM and DSSAT.
[yoni to expand]
What is APSIM?¶
The Agricultural Production Systems sIMulator (APSIM) is internationally recognised as a highly advanced platform for modelling and simulation of agricultural systems. It contains a suite of modules that enable the simulation of systems for a diverse range of crop, animal, soil, climate and management interactions. APSIM is undergoing continual development, with new capability added to regular releases of official versions. Its development and maintenance is underpinned by rigorous science and software engineering standards. The APSIM Initiative has been established to promote the development and use of the science modules and infrastructure software of APSIM.
What is a MET file?¶
The APSIM MET module provided daily meteorological information to all modules within an APSIM simulation. The APSIM Met Module requires parameters to specify the climate of the site for each APSIM time step. This information is included in a MET file.
APSIM MET files consist of a section name, which is always weather.met.weather, several constants consisting of name = value, followed by a headings line, a units line and then the data. Spacing in the file is not relevant. Comments can be inserted using the ! character.
At a minimum three constants must be included in the file: latitude, tav and amp. The last two of these refer to the annual average ambient temperature and annual amplitude in mean monthly temperature. Full details about tav and amp can be found here: tav_amp.
The MET file must also have a year and day column (or date formatted as yyyy/mm/dd), solar radiation (MJ/m2), maximum temperature (°C), minimum temperature (°C) and rainfall (mm). The column headings to use for these are year and day (or date), radn, maxt, mint, rain. Other constants or columns can be added to the file. These then become available to APSIM as variables that can be reported or used in manager script.
While point data is usually available in MET format at the SILO webpage, gridded data is provided in NetCDF file format which is difficult to store and convert to an input file readable by APSIM or other crop models. BestiaPop takes care of generating the required input files for APSIM.
Can I use this script to generate climate files for other process-based crop models?¶
[yoni to expand/fix]
So far, the code is producing CSV or MET files to be directly used by APSIM, however, it also could be applied to produce input climate data for other crop models such as DSSAT and STICS. Decision Support System for Agrotechnology Transfer (DSSAT) is a software application program that comprises dynamic crop growth simulation models for over 40 crops. DSSAT is supported by a range of utilities and apps for weather, soil, genetic, crop management, and observational experimental data, and includes example data sets for all crop models. The STICS (Simulateur mulTIdisciplinaire pour les Cultures Standard, or multidisciplinary simulator for standard crops) model is a dynamic, generic and robust model aiming to simulate the soil-crop-atmosphere system.
[yoni to expand] –> we need a section similar to what we wrote for APSIM but for DSSAT I guess…
SILO is a database of Australian climate data from 1889 to the present. It provides daily meteorological datasets for a range of climate variables in ready-to-use formats suitable for biophysical modelling, research and climate applications.
NetCDF and API data variations from SILO¶
There are particular years where some data may be different when you read the data from a
NetCDF SILO file, as opposed to reading the same data from the
SILO API. Below is a detailed description of the SILO team as to why this might happen.
When you request point data at grid cell locations, the data is extracted from the relevant grid cell in the NetCDF files. This data is then passed through a simple filter that checks if each datum is within an acceptable range:
- daily rainfall: 0 – 1300 mm
- maximum temperature: -9 - 54 ᵒC
- minimum temperature: -20 - 40 ᵒC
- class A pan evaporation: 0 - 35 mm
- solar radiation: 0 - 35 MJ/m2
- vapour pressure: 0 - 43.2 hPa
- maximum temperature > minimum temperature
If a given datum (or pair of data values for the Tmax > Tmin check) fails the check, it/they may be erroneous so SILO provides the long term daily mean(s) instead. This is represented by a number 75 in the value for a particular datum.
If you request data at station locations the same checks are done; the main difference is observed data are provided where possible, and gridded data are provided if observed data are not available on a given day(s).
Differences between the API and NetCDF values only occur when a datum fails one of the aforementioned range checks, for example, when the interpolated maximum temperature is lower than the interpolated minimum temperature. Such situations typically arise due to errors in the observed data (leading to errors in the gridded surface), or in regions where there are very few recording stations. We expect there to be more errors in the gridded surfaces for the early years, as there were relatively few stations recording data (other than rainfall) before 1957. Plots showing the number of stations recording each variable as a function of time are provided in our 2001 paper (see the Publications section on SILO).
- Clone this repo
- Install required packages.
Using Anaconda (preferred)¶
conda create -y --name bestiapop python==3.7
conda install --name bestiapop -y -c conda-forge --file requirements.txt
conda activate bestiapop
This option might take a very long time due to the multiple dependencies that Anaconda might have to solve on your default base environment. Preferably, install using the next method which creates a custom environment.
- Open your anaconda prompt for the base environment (default)
conda install -y -c conda-forge --file require
- Change directory to the repo folder
pip install -r requirements.txt
The following papers implemented this code and can be used as references
- Ojeda JJ, Eyshi Rezaei E, Remeny TA, Webb MA, Webber HA, Kamali B, Harris RMB, Brown JN, Kidd DB, Mohammed CL, Siebert S, Ewert F, Meinke H (2019) Effects of soil- and climate data aggregation on simulated potato yield and irrigation water demand. Science of the Total Environment. 710, 135589. doi:10.1016/j.scitotenv.2019.135589
- Ojeda JJ, Perez D, Eyshi Rezaei E (2020) The BestiaPop - A Python package to automatically generate gridded climate data for crop models. APSIM Symposium, Brisbane, Australia.