Esoma Exercise System - Cardiac Rehab Using the Kinect

Esoma Exercise System - Cardiac Rehab Using the Kinect

The management of heart failure is the classic example the success of telemedicine. Hospital systems and clinics throughout the world have adopted solutions to measure daily weights and vital signs from patients. They have assembled teams to monitor the data and to communicate with the patients when trends suggest decompensation. Emergency rooms visits are prevented and healthcare costs are reduced.

The use of telemedicine for heart failure is a great step in the right direction, but there is definitely room for improvement. The problem with most systems is that their philosophy revolves around doctors or nurses siphoning data from patients and returning directives. This approach does not promote patient self-efficacy and is costly because of the heavy reliance on professionals to interpret data and make decisions. It helps to keep patients stable, but it does not motivate them to improve their condition.

The Esoma Exercise System aims to shift cardiac rehabilitation to a patient-centered approach. Patients will play games powered by Microsoft Kinect and computer vision algorithms that recognize the execution of rehab exercises. Patients will wear physiological sensors while playing the games that monitor heart rate, blood oxygenation, etc. Rich feedback about their progress and improvement in physiological state will be embedded in the game to motivate self-efficacy. There will still be clinicians on the other side of the server, but they will act more as coaches that help patients make decisions for themselves based on the data. The system is being built as a CollaboRhythm plug-in so that it can benefit from the powerful tools for data visualization, communication, and social support.

During Health and Wellness Innovation 2011, Ryan Orendorff from Tufts built a prototype of Esoma that tracks typical cardiac rehabilitation exercises for patients with relatively poor reserve. He is developing algorithms that allow the system to be taught new exercise through example. The system will then be able to track those trained exercises for any patient. The next steps will be to incorporate the algorithms into a game, to incorporate the physiological sensors, and to integrate with CollaboRhythm.


Patients with heart failure and those recovering from invasive cardiac surgery should perform cardiac rehabilitation exercises to optimize recovery. Although home physiotherapy has been proven to be effective in these cases, adherence is low. The exercises are boring and patients are afraid to push themselves due to their cardiac conditions. These patients also need to be monitored for safety, but current systems only provide intermittent, low-fidelity feedback to clinicians.

The Esoma Exercise System helps to make cardiac rehabilitation fun and engaging by transforming the exercises into a game. The system uses the Microsoft Kinect hardware and open-source computer vision algorithms to track and quantify patient exercise performance. It will also use physiologic sensors to track heart rate and blood oxygenation. Patients strive to achieve higher scores as they receive immediate feedback that highlights how their physiological state is improving.

This feedback aims to increase adherence by providing a sense of  accomplishment and a feeling of safety. As patients perform exercises, the data is fed directly to the supervising clinician through the patient’s personal health record. This allows improved assessment of the patient’s progress and more informed clinical decisions.

The Esoma Exercise System has potential to increase adherence to home cardiac rehabilitation and improve the efficiency and effectiveness of clinicians overseeing this care. Currently, the system is capable of tracking simple cardiac rehab exercises. The next steps include adapting the feedback into a 3D game, incorporating the physiological sensors, and devising strategies to maximize the impact of feedback.