Devices that can detect, track and decode movements in their surroundings can have countless valuable applications in fields ranging from robotics to health care, the entertainment industry, sports, and more. Wearable sensors can be particularly effective in detecting and recording the movements of human users, as they can be strategically placed and may pick up subtler movements with greater precision.
Researchers at Ajou University, Korea University and the Korea Institute of Science and Technology (KU-KIST) recently developed a new system that can recognize the movements of a human user’s fingers in real-time. Their proposed device, presented in Nature Electronics, is based on a wearable sensor and an array of so-called artificial synapses (i.e., hardware components that replicate the function of synapses in the brain).
“The evolution of human-machine interfaces is moving toward a more seamless integration with our daily lives,” Prof. Sungjun Park (Ajou University), one of the researchers who carried out the study, told Tech Xplore. “Ideally, we’d interact with technology without even being consciously aware of it. One pivotal way of achieving this is through enabling machines to recognize natural human movements, especially intricate actions like finger motions. Such recognition is not just a novelty but holds profound implications for areas like robotics, health care, and communication.”
Despite recent advancements in the development of wearable motion recognition systems, many of the solutions proposed so far still do not achieve optimal results. Most recently developed systems for recognizing human movements rely on two or more stationary devices, such as 3D depth cameras, infrared cameras, and inertial measurement units.
“The types of setups inadvertently hamper the very natural movements they seek to recognize,” Prof. Gunuk Wang (KU-KIST) explained. “Furthermore, their underlying algorithms are resource-intensive, both in terms of time and energy, mainly because they need to differentiate fingers from other objects and then track their movements over successive time frames. Our recent paper in Nature Electronics sought to overcome these barriers.”
Building on their previous research efforts, Prof. Park, Wang and their colleagues set out to develop an “unrestricted” motion recognition system that could reliably detect and recognize the movements of a user’s individual fingers. To do this, they integrated widely used optical sensors with artificial synapses onto a very thin substrate, which had a thickness of approximately 2 micrometers (i.e., about 1/20th of a human hair’s diameter).
“The genesis of this idea came from two pivotal works,” Prof. Park said. “The first is my previous work on highly sensitive and ultra-flexible organic photovoltaics for self-powered ECG sensors, while the second is Gunuk Wang’s development of synaptic devices with robust, stable operation over extended periods.”
The researchers’ finger motion recognition system has two primary components, namely an optical sensor (organic proximity sensors) and a so-called synaptic device. By carefully placing these two components onto a highly thin substrate, the team was able to produce a device that is easy to wear on top of the skin, while also adapting to the creases that naturally form on the skin while humans are moving.
This unique design allows users to move as freely as they want in their surroundings, while the system records and recognizes their movements. The system achieves finger motion recognition by completing three different steps: motion capturing, signal conversion and learning/recognition.
“While wearing the sensor, users can draw patterns in the air (for our demonstrations we used numbers 0 through 9),” Prof. Park explained. “This is captured by the optical sensor placed on the index finger’s top and side. To provide lighting conditions, we positioned two different lights in vertical alignment. For illustration, if you drew the number ‘3,’ your finger would trace a unique route along both horizontal (x-axis) and vertical (y-axis) planes. As the optical sensor gets closer or farther from the light, it generates varying voltage patterns.”
As a second step, the team’s system translates the light captured it by the motion sensor into an electrical signal in real-time. The resulting unique voltage patterns, derived from the movements of fingers, are then transformed into digital images representing the magnitude of the voltage. Finally, these digital images are fed to the synaptic device, which proceeds to predict a user’s finger motions by running a machine learning algorithm.
“This component is key for learning and recognizing patterns,” Prof. Wang said. “It adjusts and updates synaptic weights (w) based on input, with its ability to change and maintain conductance playing a crucial role. In essence, a more pronounced input voltage leads to a more significant change in conductance, which is held for a specific duration. Over time, the device is trained to recognize patterns for numbers 0 through 9, making it capable of identifying new, random patterns based on the synaptic weights it has learned.”
In contrast with many other motion recognition systems introduced in the past, the researchers’ system is more reliable, durable and accurate in its predictions. Remarkably, their device can tolerate strains of up to 60% and can retain its functioning after it is repeatedly bent more than 1,200 times.
The system also works in poor lighting conditions, including in environments with scarce illumination or at times of the day when there is little light. In initial evaluations, it achieved a remarkable accuracy rate of up to 95%, thus predicting the finger movements of human users far better than countless other solutions introduced in the past.
“We’ve made significant strides in the intersection of neuromorphic computing and wearables,” Prof. Park said. “By synergizing a sensor and synaptic device onto an extraordinarily pliable substrate, we’ve crafted a system that not only melds seamlessly with the user but also showcases outstanding computational prowess. This is characterized by reduced power demands, heightened accuracy, and commendable resilience—both mechanically and electrically.”
In the future, the new finger detection system introduced by this team of researchers could be implemented and tested in various settings that can benefit from the reliable monitoring of human movements. Its underlying design could also inspire further efforts in the field, leading to the development of new state-of-the-art motion tracking and recognition technologies.
“Beyond just finger motion, this technology can be adapted for recognizing other bodily movements,” Prof. Park said. “Furthermore, its implications are far-reaching, spanning diverse sectors, including robotics, where it could facilitate precise human-machine interactions; health care, where it could revolutionize patient monitoring and rehabilitation; communications, where it could enable more intuitive interfaces; wearable technology, where it could aid development of more user-friendly and adaptable devices; and the realms of augmented and virtual reality, where it could help to create more immersive and naturalistic experiences.”
In their current and upcoming studies, Prof. Park and his colleagues plan to continue working on their device to further advance its capabilities and performance. In addition, they would like to use their design to create new technologies that can be used to interpret other body movements and physiological signals.
“Our recent work centered on the creation of a skin-conformable system specifically tailored for real-time finger motion recognition, achieved by merging an optical sensor and an artificial synapse on an exceptionally pliable substrate,” Prof. Park added.
“The potential of this platform doesn’t end here. In fact, we’re exploring the integration of diverse sensor types, such as strain or thermal sensors, to interpret an even broader range of data. The versatility of the optical sensor also piques our interest; its potential to gauge other vital metrics, like blood oxygen levels or heartbeats, is intriguing. Essentially, this underlines our system’s expansive adaptability across numerous applications.”