Undersea Visionaries: How the Mantis Shrimp is Changing the Way We See the World
Researchers create a unique approach to polarization imaging
Humankind has often looked to nature for inspiration. Many species have developed extraordinary ways of surviving, sometimes over millions of years. Studying them can help us find solutions a lot faster and without as much experimentation. Scientists have been looking at the mantis shrimp for inspiration and have discovered that it could change the way we see everything.
It’s hard to believe this little 6-inch crustacean is making serious waves in the scientific, medical, and even comedic community. Although quite funny looking, with a bulbous head and a long awkward body, this shrimp is considered to be one of the most vicious predators under the sea. Its front appendages, affectionately known as ‘smashers’, have inspired the designs of both body armour and football helmets, due to their ability to withstand considerable, repeated blows without trauma. But an even more noteworthy ability of the mantis shrimp is it’s extraordinary eyesight – it may have the best in all of the animal kingdom.
Eye can’t even imagine
To properly appreciate how remarkable a mantis shrimp’s eyes are it helps to compare them to our own. The human eye has three colour-receptive cones (green, blue and red) which enable us to not only see those colours, but all their derivatives. e.g. red + blue = purple, and so on. The mantis shrimp however, has sixteen colour-receptive cones! We literally can’t even imagine the spectrum of colour they see. In addition to their ability to see a multitude of colours, the mantis shrimp can also see different kinds of light. Their eyes, which are mounted on stalks and can work independently of each other, have three pupils each. These pupils give them trinocular vision that not only allows for amazing depth perception but enables them to see both UV and polarized light, which humans cannot see.
Scientist believe they have developed this remarkable eyesight due to the way they hunt – they are considered ambush, or sit-and-wait predators – which means they stay in a concealed position and scan an area until their prey arrives. Because the area they are scanning is complex, i.e. the varied nature of the coral reefs, the constant motion of the sea, and the depth at which they live – it’s pretty clear why the mantis-shrimp needs an exceptional set of eyes to see their prey.
Bioinspiration and CMOS image sensor design
Scientists continue to find biological systems that propose solutions to the challenges of digital imaging. For example, the demand for higher resolutions and frame rates have led to bottlenecks in other parts of the imaging system, either at data transmission across cables or in the image processing stage. With very high-bandwidth visual sensors (their eyeballs), the human visual cortex actually compresses the information sent from the eyes before it’s passed on to higher brain areas. Researchers have even looked at how the brain might change that ‘bandwidth’ at specific times, perhaps in response to threats, giving people a sense of time slowing. It turns out that is not the case – we don’t perceive more when scared, we actually remember more, which affects our perception of time. But the idea is sound, and can be found in the digital imaging world in features like TurboDrive.
Another example is found in the mantis shrimp: Its sophisticated eyes are paired to a simple brain. It overcomes this potential processing bottleneck of massive input from its 12 photoreceptors doing very minimal processing in the brain, and instead directly analyzes the presence of very specific colours. It’s the equivalent of a Look-Up Table to simplify output instead of combining the output of all the different color photoreceptors on an underpowered brain. Smart cameras also increase the amount of processing done on-camera to reduce the data transmission and processing demands.
But there are other aspects of the mantis shrimp’s eyes that can be applied to vision systems – particularly how to incorporate light polarization detection directly on the the sensor:
“The compound eye of the mantis shrimp contains a group of individual photocells called an ommatidium. Each ommatidium has a cornea that focuses external light. The focused light is filtered through a pigment cell for color sensitivity and passes through a series of photosensitive retinular cells (Formula-cells). In the mantis shrimp, these cells contain an array of microvilli that can act as polarization filters. The photosensitive Formula-cells will signal the brain via the optic nerve, and the brain extracts visual information based on input from the array of ommatidia.”
This research, by a group of scientists from Queensland Brain Institute, Washington University, the Washington University School of Medicine, the University of Maryland Baltimore County and the University of Bristol are working to develop the technology.
Advances in sensor design and production have allowed the researchers to take this approach to create relatively small, low power, integrated CMOS image sensors that can detect both visible and polarized light at high frame rates. A series of nanowires are laid as arrays over the CMOS imager at 0, 45, 90, and 135 degrees, acting as linear polarization filters. In this “bioinspired” solution, vision system, the photosensitive elements are monolithically integrated with aluminum nanowires, which act as linear polarization filters. The resulting camera package can fit onto an endoscope, allowing non-invasive tumor imaging that also doesn’t require the compromises of an MRI or CT scan.
The immediate application for this mantis shrimp-inspired camera is to detect cancer, which is not easily visible to the human eye or with conventional imaging solutions. Cancerous tissue looks almost identical to healthy tissue except for its disorganized structures that scatters light differently. “The camera that we’ve developed in close collaboration with the US and UK scientists shoots video and could provide immediate feedback on detecting cancer and monitoring the activity of exposed nerve cells. It converts the invisible messages into colours that our visual system is comfortable with,” explains Professor Justin Marshall at the Queensland Brain Institute. Mounted on an endoscope, the experimental camera was inserted into lab mice, looking at cancerous tissue.
One member of the team, Dr. Viktor Gruev of Washington University (St. Louis), told Photonics about how the end application was not the driving force behind the technological innovation. The possibilities were realized after the fact:
“My idea was that if we want to utilize nanotechnology, we have to find a way to merge it with CMOS. Along the way, we want to create extraordinary devices. This effort was purely driven by technology and not by application. Once we solved some of the engineering and material science questions, my colleagues kept asking me what [I was] going to use this technology for. This made me think even harder and outside my comfort zone.”
Looking Forward, Deeply
The possibilities of polarization imaging extend far beyond medical applications. Polarized light can also reveal imperfections or differences in numerous other materials and situations. This new imaging modality, with differing technological solutions, can be integrated into many different applications.
Satellite imaging, security and defense applications would benefit from the fact that just like human tissue, fog, clouds, dust, and water all scatter light in ways that prevent clear imaging with conventional means. Polarization can open the door to more reliable optical imaging in these kinds of scattering media. In fact, researchers have already looked into how polarization imaging can complement thermal imaging to make it perform even better.
Polarization imaging will also allow more more reliable imaging of shiny and translucent surfaces, such as those used in web inspection applications.
The inspection of specular metallic objects is difficult, as careful (i.e. comprehensive and expensive) lighting and multiple cameras are required to get an useful image in industrial conditions. With polarization imaging, it would be possible to extract the shape information of specular metallic surfaces from a single view.
Polarization imaging can also be used to measure physical properties that are not easily detectable. Surface, geometry, birefringence, composition, roughness, film thickness, 3D etc. can be measured based on phase differences in the materials. Quality assurance can also be accomplished, with material qualities like stress, defects, and contamination also showing up.
To date, polarization imaging cameras for machine vision have been based on area scan cameras. But, with so many opportunities for high-performance material and web inspection, line scan seems to be a logical next step. Teledyne Dalsa has already integrated this technology into the world’s first line scan polarization camera. The ability to use this imaging modality in a continuous manner, with easier lighting requirements, will allow innovative system integrators to create new uses for the technology.
With this added ability, these vision systems can now capture different polarization states per pixel and detect birefringence, stress, composition, thickness and surface morphology, making polarization ideal for manufacturers of materials and objects that require intense attention to detail, like glass, optical films, semiconductors, electronics, biomedicine, sensing and security. The possibilities are endless.
Bob’s Imaging Fundamentals #1: Look-Up Tables 
BRAIN Research: They Wish It Was Only Rocket Science 