MedHelp: enhancing medication compliance for demented elderly people with wearable visual intelligence
Scientific Phone Apps and Mobile Devices volume 2, Article number: 3 (2016)
Dementia results in much stress in senior citizens and and immensely affects their quality of life. It also incurs huge financial and emotional burdens to their family members. Personal information assistance may alleviate such a problem by enhancing the sufferers’ ability to perceive the environment, understand their personal status, and make judgments and decisions. Leveraging on visual intelligence technologies, we develop a wearable system that provides personal assistance to enhance the independence and well-being of elderly people with dementia. An application called MedHelp was developed on Google Glass paired with a mobile phone to improve medication compliance. MedHelp has three functions: it sends out reminders to a user who is on regular medication; it helps a user to recognize the correct medicine container and provides dosing instructions; and it tracks the user’s medicine-taking activities so as to plan the time of next medications.
Dementia includes a spectrum of symptoms of cognitive decline including memory loss and difficulties with thinking, problem-solving or language. It results in much stress in senior citizens and and immensely affects their quality of life. It also incurs huge financial and emotional burdens to their family members (Luengo-Fernandez et al. 2010). Considering the growing number of people with dementia, and the declining old-age support ratio (http://population.sg/whitepaper/resource-files/population-white-paper.pdf), it could become a challenging social and economical problem in near future. Meanwhile, with limited public resources and facilities, and to the preference of most senior citizens, a significant portion (e.g. 70 % in UK (Prince et al. 2014)) of the elder population will seek care in their own homes. Enabling technologies for assisted independent living are of great value to tackle the problem.
Technological assistance has been developed to help demented people in various ways, such as prospective memory (Oriani et al. 2003), health monitoring and personal assistance (Sun et al. 2014), medicine management (Khan et al. 2010), object recall and search (Huang et al. 2011), etc. This app focuses on enhancing medication compliance in the home environment. Medication compliance has huge impact on the elder individuals’ health. In fact, inadequate compliance results in increased morbidity and mortality from various illnesses, and causes significant increase in healthcare costs (Cramer et al. 2008). However, it has been a tough challenge to ensure medication compliance. First, people with memory loss constantly forget about the medications especially when they have many medicines to take. Second, many elderly people have reduced vision acuity so that they have difficulties in identifying the correct medicines. Third, they may forget whether they have taken certain medicines or not, leading to omissions or overdoses.
In view of the difficulties faced by demented people, we develop wearable solutions that provide personal assistance in the home environment. In particular, we present MedHelp as a medication management tool. The application is built on image recognition techniques. A unique feature of the app is its accessibility because it is worn by a user rather than being stationed in a fixed place. We are aware that a comprehensive solution framework for home care based solely on wearable devices is technically and socially challenging. Therefore, this application is intended to explore and demonstrate the scenarios where Information and Communication Technology (ICT) is appropriate and desirable.
The application serves as an external memory of the user and provides instant help when needed. To do so, basic medication information is pre-registered in the system, typically with the help of caregivers. For example, MedHelp requires the caregiver to specify which medicines should be taken by the user (i.e. the elder person), at what time, and how to take them (i.e. the dosage instruction where necessary). One or a few images of each medicine is captured and stored in the database. These images will be used for medicine verification through image matching. For the purpose of evaluation, we have trained the system to recognize four types of medicines. Interested readers may find the images of the medicine for testing in the MedHelp user guide (http://perception.i2r.a-star.edu.sg/medhelp/Medhelp_instructions-v2.pdf).
The system consists of a wearable camera (i.e. a Google Glass) and a smart phone running Android OS. It should be noted that the system can be implemented on other types of devices and platforms. The two devices are connected via Bluetooth. The application adopts a client-server-cloud structure. At the front-end, the client (i.e. Google Glass) acquires images, displays results, and issues voice instructions. The server runs on the mobile phone that relays the images to the cloud. The image is processed in the cloud to determine if it matches a registered medicine. The result is sent back and is further displayed to the user. Figure 1 shows a photo of a user demonstrating the glass and smart phone running MedHelp. The mobile phone must have Internet access to connect to the cloud server.
The core technology of the application is image matching. The Glass camera captures image sequences continuously at a resolution of 640*480 pixels. To reduce the data flow between the devices, images are cropped out from the original images, so that only the central region (320*240 pixels) is used for processing.
MedHelp adopts the image recognition algorithm in (Kusuma et al. 2012). The image recognition algorithm will recognize the images captured from the camera of Google Glass by matching them to pre-defined images of medicine containers in the database. An image with distinctive features will be identified by the image recognition engine running on the cloud and the pre-associated content such as text, video or audio will be retrieved and displayed on the glass or played to the users. The apk files can be downloaded for the mobile phone http://perception.i2r.a-star.edu.sg/medhelp/medhelpserver-debug.apkhttp://perception.i2r.a-star.edu.sg/medhelp/medhelpserver-debug.apk and Google Glass http://perception.i2r.a-star.edu.sg/medhelp/medhelpglass-debug.apk. Instructions on the installation and usage of the app are available in the user guide.
We design the interaction protocol considering the reduced cognitive abilities of demented users. To do so, we simplify the user interface so that it requires minimal input from the user, and it provides concise and intuitive feedback. Nevertheless, it is required that the user wears the Google Glass and keeps the phone in its vicinity so as to enable Bluetooth connection.
The interaction protocol of MedHelp is as follows.
At a pre-set time, a voice prompt is issued from the Glass telling the user “It’s time to take your medicine - XXX”, where XXX is the name of medicine. Meanwhile, an image of the corresponding medicine is displayed on the Glass screen together with its name. The camera of Google Glass starts to capture live image feeds and sends them to the phone (Fig. 2 a). It should be noted that the current app does not activate the clock-based reminder. Instead a user manually press a button on the phone app to trigger the medicine reminder.
The user holds the medicine container in the hand so that it is ‘visible’ to the Glass camera. The phone receives the images and sends them to the cloud server, where they are compared against pre-registered ones in the database. If a correct match is found, it will send a confirmation followed by dosage instructions. For example, it will say “You have found XXX. Please take two capsules with warm water.” (Fig. 2 b & c). If the system detects a wrong medicine or if no match is found after 5 attempts, it will send a warning, “This is not XXX. Please try looking for the correct medicine again!”. This is repeated until the user finds the correct medicine (Fig. 2 d & e).
After the correct medicine is detected, the system will prompt the user to confirm that the medicine is taken. The user needs to tap on the touchpad of Google Glass. Once the system detects the tap gesture, it sends a confirmation message, e.g. “You have taken your medicine XXX. It is saved in your medicine log. The next time to take your medication is 6pm.” (Fig. 2 f). Meanwhile, the medicine-taking activity is recorded in the dataset. This will allow the user, and importantly, the caregiver to check if the medicines have been taken as expected.
The procedure is shown in a video that can be downloaded at http://perception.i2r.a-star.edu.sg/medhelp/MedHelp_UI2015.12.mp4.
With regards to the confirmation of medicine-taken, an alternative and more intuitive way is to automatically recognize user activity. After a correct medicine is detected, the system enters a mode of activity detection to recognize the medicine-taking activity. The system can read sensor data (e.g. gyroscope and accelerometer) from the Google Glass to detect the head-up activity, which typically happens when a user swallows the medicine. The app has implemented this simplified strategy. However, since medication is a critical application scenario, we deactivated the activity recognition function in the current app. We plan to improve this with more comprehensive activity recognition methods.
First, the usability of the system needs to be improved to better address the special needs of elder users, particularly those with cognitive decline. Second, in real applications, one needs to consider multivariate practical constraints, including hardware, logistics, and variations caused by different usage patterns. Finally, the robustness of the system has not been tested with regard to potential safety issues.
Despite these limitations, we believe that these applications are useful technological components that can be integrated into a more comprehensive solution framework. Putting together they have great potential to improve the quality of life of elderly people who suffer from cognitive decline.
We present the design of an application in assisted living for aging. The application leverages on visual intelligence to help the elderly people better perceive the environment and makes sensible decisions on specific tasks. We investigate the possibility of wearable solutions in such context to achieve seamless ICT support in the home environment. In particular, MedHelp provides comprehensive support to medication compliance in the elderly on regular medication. Visual intelligence plays a unique role in the application owning to the richness of visual information and a shared point-of-view between the wearable camera and the user. Thus, it has good potential to support home care.
Written informed consent was obtained from Shue Ching Chia featured on Fig. 1 for publication of this image.
Cramer, JA, Roy A, Burrell A, Fairchild CJ, Fuldeore MJ, Ollendorf DA, Wong PK. Medication compliance and persistence: Terminology and definitions. Value Health. 2008; 11(1):44–7. http://population.sg/whitepaper/resource-files/population-white-214paper.pdf. Accessed 05 March 2016.
Huang, PS, Huang CP, Hsieh CH, Hwang BJ, Chiou CY, Hsiao KF. A Vision-based System to Help Senior Citizens for Memory Recall of Object-placing. In: International Conference on Instrumentation, Measurement, Computer, Communication and Control. IEEE: 2011.
Khan, DU, Siek KA, Meyers J, Haverhals LM, Cali S, Ross SE. Designing a Personal Health Application for Older Adults to Manage Medications. In: IHI’10. SpringerLink: 2010. p. 849–58.
Kusuma, GP, Szabo A, Li Y, Lee JA. Appearance-Based Object Recognition Using Weighted Longest Increasing Subsequence. In: ICPR. IEEE: 2012. p. 3668–671.
Luengo-Fernandez, R, Leal J, Gray A. Dementia 2010: The economic burden of dementia and associated research funding in the United Kingdom. Cambridge: Alzheimer’s Research Trust; 2010.
Oriani, M, Moniz-Cook E, Binetti G, Zanieri G, Frisoni GB, Geroldi C, Vreese PD, Zanetti O. An electronic memory aid to support prospective memory in patients in the early stages of alzheimer’s disease: a pilot study. Aging Ment Health. 2003; 7(1):22–7.
Prince, M, Knapp M, Guerchet M, McCrone P, Prina M, Comas-Herrera A, Wittenberg R, Adelaja B, Hu B, King D, Rehill A, Salimkumar D. Dementia uk update, second edition. UK, London: Alzheimer’s Society; 2014. Technical report.
Sun, M, Burke LE, Mao ZH, Chen Y, Chen HC, Bai Y, Li Y, Li C, Jia W. eButton: A Wearable Computer for Health Monitoring and Personal Assistance. In: DAC’14. New York: ACM: 2014.
The work is funded by the Singapore A*STAR JCO VIP - REVIVE (Project No. 1335h00098). http://www.i2r.a-star.edu.sg/revive/.
The authors declare that they have no competing interests.
QX proposed the design concept, system structure and user interface. SCC developed Android program and writes the codes. JHL proposed the application case. BM developed part of the backend API. YL developed the main image matching algorithm. LL contributed to system design and implementation. All authors read and approved the final manuscript.
About this article
Cite this article
Xu, Q., Chia, S.C., Lim, JH. et al. MedHelp: enhancing medication compliance for demented elderly people with wearable visual intelligence. Sci Phone Appl Mob Devices 2, 3 (2016). https://doi.org/10.1186/s41070-016-0006-5