Researchers from Florida Atlantic University’s College of Engineering and Computer Science have developed a cutting-edge, real-time American Sign Language (ASL) interpretation system that could revolutionize how deaf and hard-of-hearing individuals communicate in a digital world.
Designed to address the limitations of traditional ASL recognition tools—such as poor performance in real-time settings, misclassifications due to visually similar gestures, and a lack of adaptability to diverse users—the innovative system leverages the power of YOLOv11 object detection and MediaPipe hand tracking. This dual-tech approach allows for accurate, contact-free translation of ASL alphabet gestures into text using only a standard webcam.
Bader Alsharif, the study’s first author and a Ph.D. candidate
“What makes this system especially notable is that the entire recognition pipeline – from capturing the gesture to classifying it – operates seamlessly in real time, regardless of varying lighting conditions or backgrounds,” said Bader Alsharif, the study’s first author and a Ph.D. candidate in FAU’s Department of Electrical Engineering and Computer Science.
The research, recently published in the journal Sensors, achieved an impressive 98.2% accuracy (mean Average Precision, mAP@0.5) with minimal latency—making it ideal for fast-paced, interactive applications like live video processing and real-time communication tools.
ASL is composed of complex hand gestures that can be difficult to interpret using existing technologies, especially when differentiating between similar signs such as “A” and “T” or “M” and “N.” To address these challenges, the FAU team created a dataset of 130,000 images capturing ASL gestures under varying lighting conditions, backgrounds, and hand orientations. Each image is annotated with 21 key hand landmarks to help machine learning models distinguish between subtle gesture differences with high precision.
“This project is a great example of how cutting-edge AI can be applied to serve humanity,” said Imad Mahgoub, Ph.D., co-author and Tecore Professor in the Department. “By fusing deep learning with hand landmark detection, our team created a system that not only achieves high accuracy but also remains accessible and practical for everyday use.”
The deaf and hard-of-hearing community includes approximately 11 million individuals in the U.S. alone, with 37.5 million American adults experiencing some degree of hearing difficulty. By translating ASL gestures into readable text in real time, this new system promises to break down communication barriers across education, employment, healthcare, and social interaction.
Mohammad Ilyas, Ph.D., co-author and professor in FAU’s Department of Electrical Engineering and Computer Science
“The significance of this research lies in its potential to transform communication for the deaf community,” said Mohammad Ilyas, Ph.D., co-author and professor in FAU’s Department of Electrical Engineering and Computer Science. “Our study contributes to the advancement of assistive technologies that promote accessibility and inclusivity.”
Future development will aim to expand the system’s capabilities from recognizing individual letters to interpreting complete ASL sentences, making communication even more fluid and natural.
“This research highlights the transformative power of AI-driven assistive technologies,” said Stella Batalama, Ph.D., dean of FAU’s College of Engineering and Computer Science. “By enabling real-time ASL recognition, this system fosters a more inclusive society and empowers individuals with hearing impairments to more fully engage with the world around them.”
Study co-authors include Easa Alalwany, Ph.D., a recent graduate of FAU and assistant professor at Taibah University in Saudi Arabia, and Ali Ibrahim, Ph.D., a fellow graduate of the College.
To learn more about the College of Engineering & Computer Science at Florida Atlantic University, visit www.fau.edu/engineering