Temi Healthcare Guidance Robot
Built an interactive healthcare guidance robot using Whisper, retrieval-augmented generation, and a mobile service-robot interface.
Elderly patients in a hospital need to get from the lobby to the right department without already knowing the building. A mobile service robot can guide them, but only if speech-to-text, retrieval, navigation, and motion-guidance control all hold up around vague speech, crowds, and uneven floors.
Department recommendation is not a chatbot demo: it has to map noisy voice to one-to-four clinical departments using hospital documents as the knowledge base, then hand a goal to the robot.
- Built the Android application on the Temi platform (Kotlin, adb) with navigation and IoT connectivity over TCP/HTTP, plus obstacle avoidance, uneven-floor detection, and motion-guidance control.
- Implemented the voice-input department recommender: OpenAI Whisper speech-to-text, retrieval-augmented generation over hospital documents (LangChain / ChromaDB), and an LLM. Accuracy was 93% on a 1,000-scenario evaluation set, then 99.6% after a reselect model for edge cases.
- Co-first author on two papers: NSSSE 2025 Best Paper and ICSSE 2024 Best Session Presentation.
Perception
Whisper for speech-to-text, including the vague and unclear patterns common with elderly speakers. On the mobility side, onboard sensing supports obstacle avoidance and uneven-floor detection.
Planning
Map-based navigation on the Temi platform to consulting rooms, examination rooms, and the dispensary, with electronic fences for restricted areas and speed adapted to walking pace.
ML / Data
RAG over hospital documents in ChromaDB, with an LLM that recommends one to four departments. A reselect-department model handles cases the first pass misses — that is the jump from 93% to 99.6% on the custom 1,000-scenario set.
Infrastructure
Temi SDK on Android, a Python RAG server, HTTP/TCP to the robot and a Windows toast-notification path so staff see arrivals. The interface is a service robot, not a web form.
Retrieval quality is the product. Whisper errors and vague symptoms are expected; the system has to retrieve the right department anyway, which is why the reselect model moved the number after the first RAG pass.
A hospital robot is a distributed system: Android on the Temi, a RAG process, and a staff-side notifier. Most integration bugs were timeouts and IP assumptions, not model quality.