From image processing to computer vision: plant imaging grows up

In recent years, plant phenotyping has evolved into a technology-driven discipline, and can be seen in increasingly common use in much plant and crop science research. An understanding of the genotype has driven research in the past, and an inability to accurately and quickly characterise phenotypes has caused a gap to develop between what is possible with the genotype and phenotype. This has occurred not only in plant science, but in all walks of biology (Houle et al. 2010). Now, new technology – both hardware and software – are causing this gap to be closed, and a decrease in equipment cost is allowing systems to become more widespread. Detailed, high-throughput phenomic measurements of plants are allowing us to understand more completely the functions of genes and the interplay with the environment (Furbank and Tester 2011; Dhondt et al. 2013). However, the process of automatically acquiring images of plants and their subsequent analysis is complex, and requires an understanding of the software processes at work, and their limitations (Pridmore et al. 2012). The combination of rapid image capture and high throughput, objective analysis, though, has been shown to lead to new genetic insight (Atkinson et al. 2015). These potential rewards then, and the necessity for specialist knowledge to apply and develop the underlying systems, provide the motivation for the timely application and development of image analysis to plant biology.

Taking automated measurements from images of plants is not an easy task. When experienced biologists take measurements for experiments, as well as having expert plant biology knowledge, they also have an understanding of the world which allows them to interact with the plants in the first place. This fundamental ability to understand and interact with the real world is a complex challenge for computers. Teaching a computer to understand when there is a plant in an image, let alone programming it to take measurements of the plant is an immensely challenging task when all you have is the raw numerical data representing a digital picture. With the right knowledge, techniques developed for use in other domains can be adapted and applied to plant images. Where these techniques do not exist, the image analyst must develop new techniques using existing image processing building blocks to begin, or entirely from scratch. The top-most computer vision challenges are great enough to rival anything artificial intelligence has to offer – the challenges in just developing a software tool should not be overlooked. The image analysis components of these tools in the most part fall into the areas of image processing and/or computer vision, and both can help with the analysis of data in plant biology, as we shall see in this issue.

Image processing is properly understood as the use of computational tools to help people to interpret images. The methods of image processing range from straightforward image enhancement and noise reduction techniques, to image-based interactive measurement systems. Within computer-assisted biology there are many examples of this kind of work (Li et al. 2014). An image is in a sense a measurement of raw data, and the tools which help biological scientists to interpret and obtain meaningful measurements from images are the tools of image processing.

More complicated approaches involve computer vision on top of processing; the aim here is to build algorithms and systems that can ‘see’ – algorithms that can perform measurements and detections from an image, or even better from a series of images, without human intervention. Computer vision is a research field in its own right: an interdisciplinary one, linked to fields as diverse as animated graphics for the movie industry, robotics and self-driving cars, and medical imaging to name just a few. For plant biologists, computer vision is the field that can offer a solution to the challenge of high throughput phenotyping: if you have ten thousand images, clicking on each to define growth points is a prohibitively expensive task. Building a computer vision system that can do this with reasonable accuracy can save time, save money, and generate more objective results. Of course, we are now left with the problem of defining reasonable accuracy versus people: if someone is measuring plants manually, are they sure they are measuring the right thing? And how late a night did they have the day before…?

In this special issue, we report work from both areas, both image processing and computer vision, the aims of which are to aid the plant biology community with new approaches and techniques designed to advance the field of plant image analysis. Several of these techniques were presented at the International Workshop on Image Analysis Methods for the Plant Sciences (University of Nottingham, 2–3 September 2013), an annual meeting of researchers working in the field of plant image analysis.

The papers themselves span a diverse range of topics, representing the broad application areas of such techniques. Boyle et al. (2015) present work using texture patterns to characterise oat panicles to determine the time of flowering. The combination of existing image processing and data clustering approaches, adapted to work with plant image data, provides an encouraging correlation with manually observed data, which can be obtained automatically on a high throughput phenotyping installation.

Kempthorne et al. (2015) address the problem of accurately reconstructing the shapes of leaf surfaces for use in modelling agrichemical spraying of whole plants, addressing the requirement of high-accuracy rather than high-throughput for phenotyping. Challenges produced by leaves bending and twisting, which complicates the application of traditional algorithms, were also investigated. Such high-level computer vision model-fitting approaches are applicable to a wide variety of datasets featuring long, thin leaves, from laser-scan derived data, to point clouds derived from stereo imaging methods, for example. High-detail modelling approaches such as this are necessary for accurate and useful simulations of plant growth and interactions with the environment.

Strange et al.’s (2015) work derives from a completely different imaging modality which is increasing in popularity – X-ray-derived CT (Computer Tomography) data. Here, an image processing pipeline (a collection of techniques used in sequence) is used to characterise the morphology of wheat grains imaged in situ using CT technology.

Mairhofer et al. (2015) also work in the CT domain, but investigating root architectures rather than grain morphology. Their system attempts to reconstruct something that is simply not recoverable from hand-measurement: the architecture of roots in soil. The technique involves segmenting roots (identifying which voxels represent root, and those that are soil) from CT, and then tracking these throughout the root column. One difficult aspect of this work is the evaluation; and here the authors are thorough, comparing to artificial data, plant roots washed from soil, and to other root-extraction software.

Nelson et al. (2015) compare an object-based (computer-vision style) method to a more traditional pixel-based (image-processing style) measure, when quantifying the colocalisation of labels in fluorescent microscopy images of NET1A with plasmodesmata. Through identifying biological phenomena as objects, computationally speaking, they are able to better model change, and to extract distance information.

Finally, Horgan et al. (2015) use coarse histograms in RGB-space to find a measurement that correlates with leaf area in pepper plants. By placing a blue screen behind the plants, they are easily able to identify plant material in their images, and they go on to show that the differences in colour measurements for different pepper genotypes can be followed through into QTL analysis. This paper falls firmly in the category of computer vision for high-throughput phenotyping.

We hope these papers can provide a valuable insight into how image processing and analysis can be used in a wide variety of ways, using a combination of different imaging technologies, to image a wide variety of plants and scales, both in a high-throughput and high-accuracy manner. The approaches themselves can be mathematically founded, high-level approaches, which can be applied to a dataset from the top down; that is, how the data was captured is less important than the type of data it results in (for example, the 3D leaf modelling work of Kempthorne et al. (2015)). Or they could be a combination of modified existing, bottom-up processing components which produce a robust and reliable measure, and can be implemented in a high-throughput manner in a phenotyping centre.

We need an ongoing dialogue between image analysis scientists and plant scientists which will lead to computer vision techniques becoming more broadly adopted. When we have plants and genotypes that can be captured at a rate of thousands of images per day, automated analysis of those images is essential. With good computer vision systems, quantitative data can be extracted from these images, and data analysis rather than data acquisition becomes the true bottle neck. Such techniques can automate current laborious manual protocols, or indeed lead to entirely new biological insights into data and plants which would just not be possible without new hardware imaging approaches and accompanying software analysis.