HawkEye Vision Assistance
View project ↗
Smart-glasses software that describes lecture whiteboards for blind and visually impaired students using voice, OCR, and AI.
The idea
When my Hack the North team received access to AdHawk MindLink smart glasses, I thought of a blind classmate and the difficulty of following work written on lecture boards.
HawkEye uses the glasses' outward-facing camera to read a live scene and dictate text and equations through earbuds.
The solution
Voice commands are transcribed and classified into OCR requests, general questions, or no action. Visual requests activate the camera and Google Cloud Vision; general questions bypass it.
- Python with PyAudio and wake-word detection
- Google Speech-to-Text and Whisper transcription
- Google Cloud Vision OCR on sampled 1080p frames
- OpenAI function calling to route OCR, answers, and null actions
Result
The project won Best Use of Syro and Best Use of Google Cloud. It also made the potential for faster, more accessible classroom tools feel tangible despite the constraints of an early and lightly documented hardware SDK.
