Image of the Nisqually River

Gramener and Microsoft AI for Earth help Nisqually River Foundation automate identification of fish species

Download case study

The Nisqually River Foundation is tasked with the successful implementation of a watershed stewardship plan. As a part of this plan, they assist the Nisqually Indian Tribe in Washington State to measure and monitor the fish species present in the Nisqually River. To do this, the Nisqually Indian Tribe installed a video camera and infrared sensors in a fish ladder at a dam on the river. The camera is triggered to capture 30 seconds of video when any fish swims past the infrared sensors.

A cumbersome manual process

Throughout the year, more than 3,000 videos are generated by the counter camera. As part of their original process, a trained biologist needed to view each video to manually identify and record the species of each fish. This manual process of fish species identification in captured videos is resource intensive, from a time, human resources, and cost perspective. “The staff of the Nisqually Indian Tribe’s Salmon Recovery program and the Nisqually River Foundation are small and their time is already fully committed to projects,” said Ganes Kesari, Co-Founder of Gramener Technologies. “This work is slow and repetitive and is much better suited to automation than manual analysis.”

“Gramener has been pioneering the use of AI technologies to solve tough business challenges faced by our enterprise clients. It was great partnering with Microsoft AI for Earth team to address the business problem faced by Nisqually River Foundation.”

— Naveen Gattu, COO, Gramener

An AI-driven solution

Gramener, in conjunction with the Microsoft AI for Earth program, worked with the Nisqually River Foundation to attempt to automate the detection and identification of fish species from the video clips. The Nisqually salmon detection application was built as a web app to automate the process of video feed input, detection, and classification. The automated AI solution leverages the latest deep learning algorithms implemented using the Microsoft Azure and Cognitive Services platform stack.

According to Kesari, the process begins with the video feeds being processed to extract the relevant frames. Deep learning models are then trained to detect the fish by drawing bounding boxes and accurately identifying the species within the video frames.

Given the nature of the problem and the format of the video files, processing power was a key requirement for the training and validation phases. “A GPU (Graphics Processing Unit) machine was the natural choice to run the object detection models, hence we selected a GPU virtual machine (VM) in the Azure portal,” Kesari said.

“For a reliable cloud solution with machine learning capabilities, Microsoft Azure Data Science VMs was chosen. For the purpose of extracting frames from the videos and tag them, Microsoft Visual Object Tagging Tool (VOTT) came in handy. The final object detection algorithm chosen was a YOLO V3 video detection algorithm.” YOLO stands for “you only look once” and is a state-of-the-art, real-time object detection system.

The first challenge was to process the videos and tag the fish, Kesari said. The heavy manual work involved in this was automated by leveraging the Microsoft VOTT tool. The tagged frames were then used to train a model using Microsoft Cognitive Toolkit (CNTK). This model was then tested against more frames extracted from the videos. While this solution was good, it lacked speed and real-time video detection capabilities. As an enhancement to the solution, Gramener moved to video object detection using YOLO V3, which provides a faster solution with real-time capabilities.

“This project by Gramener is a good showcase of an AI-driven solution that addresses challenges faced by the Nisqually River Foundation.”

— Lucas Joppa, Chief Environmental Scientist, Microsoft AI for Earth

“Throughout the project, the Microsoft AI for Earth team was a key enabler and influencer for project success. Timely access to Microsoft technical teams for support and resolution of queries on CNTK and Azure data science virtual machines (DSVMs) was invaluable,” Kesari said.

Naveen Gattu, Chief Operating Officer (COO) at Gramener, said the partnership with Microsoft AI for Earth was beneficial. "Gramener has been pioneering the use of AI technologies to solve tough business challenges faced by our enterprise clients. It was great partnering with Microsoft AI for Earth team to address the business problem faced by Nisqually River Foundation.”

Lucas Joppa, Chief Environmental Scientist at Microsoft AI for Earth, agreed. “This project by Gramener is a good showcase of an AI-driven solution that addresses challenges faced by the Nisqually River Foundation,” he said.

Saving valuable hours

The web-based AI solution is expected to save the Nisqually Indian Tribe and the Nisqually River Foundation valuable hours of expert biologist time and the infrastructure costs for manually viewing videos.

In addition, Kesari said that an enhanced version of the solution has just been rolled out to the Nisqually Indian Tribe. “This updated version has been predicted to deliver a nearly 5 times increase in speed of species determination. A task which would previously take more than 100 hours in a year could be reduced to about 20 hours of analysis time,” he said.