A spatio temporal spectral framework for plant stress phenotyping

An overview of the framework described in this paper. This work presents a methodology for building and benchmarking machine learning models that can infer plant stress using remotely sensed, multi-modal data. Our framework consists of a spatio-temporal spectral dataset, image pre-processing and classification algorithms along with reference plant trait measurements and stress type and severity level labels to serve as ground truth. Our generic, plant agnostic pipeline starts with raw input imagery from RGB, stereo infrared (IR) and multispectral cameras, followed by pre-processing steps of vegetation segmentation, 3D reconstruction and reflectance normalization to transform this raw data into plant trait indicators such as canopy cover, average height and normalized narrow-band reflectances over time. We then train machine learning models which can use these indicators to predict severity levels for Water, Nitrogen and Weed stress simultaneously. We show the effectiveness of our framework by using the trained models to accurately predict stress severity levels on novel test data. We release the collected dataset [1] and accompanying pre-processing and classification software [2] under an open source license for the broader plant research communit