This article was originally published on iThome on September 12, 2022. [https://www.ithome.com.tw/people/152960] Reprinted with permission from iThome.

LongGood Meditech created an AR interactive game, relying on AI to identify the patient’s rehabilitation movements, which has become an excellent helper for tele-rehabilitation, and also makes it easier for therapists to track the progress of rehabilitation. They have also developed gait analysis, named GaitBEST, to help physicians measure rehabilitation outcomes more accurately. The next step is to become an Android platform for traditional rehabilitation software and hardware devices.

“It used to take at least an hour for family members to go to the hospital to see a doctor!" recalled Ray Chen, the founder and CEO of LongGood Meditech, who was from a remote township in Su’ao. This distance is not only a pain point for people in remote villages to seek medical treatment, but also a major obstacle for patients who need to return to the hospital regularly for rehabilitation. They often only return to the hospital once or twice before giving up. This dilemma of medical resources inequivalently distributed, Ray has always kept in mind, and later became his motivation to invest in software-assisted rehabilitation. He not only needs to use software to become the right and left hand of a rehabilitation therapist, but also saves the public from running around and can do rehabilitation at home.

So, he creates a lively and interesting AR interactive game coding line by line, allowing people to dance and move through the level. For example, some people turned into butterflies and fluttered their wings vigorously, while others shuttled through traditional markets, picking fruit and vegetables from vegetable baskets. These scenes can be seen in more than 100 medical institutions, community centers, people’s homes, and even more than 130 aboriginal cultural and health stations in Taitung.

These games are not only fun, but also sophisticated software for rehabilitation. LongGood inhouse-build computer vision model can instantly identify people’s every move. It can not only calculate the joint flexibility from the arm swing angle, but also deduce the strength from various movements, and then judge the correctness of the posture. Ray digged into this field, and has been doing it for 10 years, and he still enjoys it.

Utilizing power of data to solve traditional rehabilitation medical problems

Because, he believes, software-assisted rehabilitation can solve many problems.

For example, the labor-intensive problems faced by traditional rehabilitation. He explained that the general hospital rehabilitation situation is that after the specialists evaluate(see) the patient, they will issue prescriptions and list the items that need rehabilitation, and then the therapists in the hospital rehabilitation center will conduct rehabilitation one-on-one training and evaluation. Assuming that people see a doctor once a month, during that period of time, patient need to go to the hospital regularly to find a therapist for constantly rehabilitation.

But the current situation in Taiwan is that a rehabilitation therapist has to care for rehabilitation patients over their capacity, and each patient has to spend a lot of time training and evaluation, extra time in transportation time should be included. In addition, some patients may perform the same rehabilitative movements as others, tantamount to repeating the work for the therapist.

When Ray and his team designed the system, they take those clinical unmet need into consideration. First of all, the interactive game rehabilitation system he created has a built-in prescription interface, which doctors can use to drag and drop rehabilitation exercises for different joints, so that doctors can combine training content suitable for patients and optimize the traditional process. Furthermore, this rehabilitation game is equipped with an AI model and a depth sensor, which can accurately identify and calculate the patient’s joint movement angle, and directly generate a report for the therapist’s reference to share the therapist’s evaluation work.

Moreover, they also designed that this software is not only for individuals, but also supports multiple people (up to 6 people) to rehabilitate at the same time, recognize movements at the same time, and generate reports. In this way, the phenomenon of labor intensive work of the therapist has been changed, and the therapist can also grasp the patient’s state more quickly and further adjust the treatment plan.

Even, this set of rehabilitation software is further extended to distance rehabilitation. Ray pointed out that the traditional rehabilitation method requires patients to go to the hospital rehabilitation center for rehabilitation, but people in remote areas, even patients who live in cities but have no elevators in their apartments, can easily give up going to the hospital for rehabilitation due to inconvenience and distance. . Coupled with the raging epidemic in the past two years, some people are afraid to step into the hospital, and the demand for remote rehabilitation is rising.

LongGood’s rehabilitation system uses iot and cloud-based architecture to break down distance barriers. Ray said that at present, as long as the doctor prescribes the rehabilitation prescription, as long as the patient has a TV screen at home and a iot-box equipped with a depth sensor, they can follow the interactive game to do rehabilitation at home, and the system can also identify people in real time. Action, output report. The therapist can check the report, grasp the progress of the patient’s rehabilitation, and determine whether the disease is worsening without being on the scene. When the patient returns to the clinic, the physician can also view the report, instead of listening to the oral description of the public, and use the quantitative data to adjust the treatment strategy.

This leading telerehabilitation in Asia, home rehabilitation is the promotion goal of LongGood this year. They need to develop a virtual rehabilitation center that supports distance from medical centers, so that people can rehabilitate at home, and also expand the concept of distance to more community centers to shorten the time it takes for people in remote areas and patients with disabilities to go to the hospital for rehabilitation. distance.

Not only that, the more important significance of tele-rehabilitation is to allow patients to train in a timely manner during the golden rehabilitation period to improve their original strength after recovery. For example, a stroke within 3 months is the golden rehabilitation period. If the patient is not hospitalized, he will receive hospital rehabilitation according to the traditional process. However, many people have difficulty in arriving at the hospital due to factors such as inconvenience of movement and distance. From the rehabilitation software, come to in-situ training.

We will modularize the core functions to integrate with rehabilitation equipment manufacturers and become the Android platform in the rehabilitation hardware industry. ── Ray Chen, founder of LongGood Meditech

Inhouse-built AI models and analysis reports, developed more than 200 AR game scenarios

In order to improve the traditional process of these rehabilitation software, it must first arouse the interest of patients and increase their willingness for rehabilitation, and then generate rehabilitation reports to therapists through the ability to recognize and analyze patients’ movements in real time. How does LongGood make it?

First of all, in order to attract patients to rehabilitate, LongGood uses interactive games to package rehabilitation activities. For example, they designed a cotton picking game for stroke patients with better recovery, neurosurgery patients and sub-health elders in the community. The game was conceived from the early flourishing cotton industry in Taiwan. It has nostalgic features and is mainly used to test the patient’s sense of balance.

In the game screen, there are 3 conveyor belts on the left and right to send the dirty cotton to the center. This game measures how the patient’s body shifts its center of mass and maintains balance under a fixed support floor area. Among them, there are 3 conveyor belts from top to bottom, which can not only train the extension ability of the arms, but also use the muscle strength of the upper and lower limbs. The design of the left and right conveyor belts is suitable for the activities of patients with hemiplegia such as stroke, and encourages lateral limb movement.

Not only does he pick cotton, but LongGood also designs related themed rehabilitation games according to different festivals to attract patients and the elderly to move their muscles and joints. “Our training content has expanded from the original 5 to more than 200 now!" Ray said.

After the patient is involved in rehabilitation, the next step is to make the software have the ability to “see". The core of LongGood’s api is based on an open-source AI model, which is modified and retrained to observe patient movements. For example, the model is equipped with a depth sensor, which can identify the angle of the arm swinging up and down to determine the flexibility of the shoulder joint, or identify the flexion angle of the patient’s elbow joint to determine whether the patient’s elbow movement is in line with rehabilitation training. Target.

After the model sees the action, it has to digest the information it sees and convert it into a patient’s rehabilitation report. Taking the shoulder report as an example, the patient has completed the shoulder rehabilitation training, and the system will generate a series of parameter values ​​according to the patient’s movements, such as the absolute angle of the shoulder joint, flexion angle, horizontal abduction angle, rotation of the left and right arms swinging up and down. The angle, as well as the absolute angle of the elbow joint and the flexion angle of the knee joint, etc., can be accurate to one decimal place. Next to the patient angle column, the system will also calculate the standard deviation angle, which is convenient for the therapist to interpret.

In addition to individual joint angles, the system also produces an overview of rehabilitation activities, which maps the patient’s activity history with the two coordinates of time and angle into a graph, from which the therapist can understand the amplitude, and endurance of the patient’s movements. In addition, the system can further generate a cross-sectional view of the shoulder joint, which the therapist can use to determine the degree of limited joint mobility of the patient, as well as abnormal movement strategies, and even view the raw data as a reference for correction.

The current content of LongGood is diverse. In order to manage various content, they have also created 3 kinds of back-end systems, including patient presentation layer, therapist presentation layer and internal presentation layer. The presentation layer refers to management. For example, the therapist presentation layer is the backend used by therapists to manage the patient’s rehabilitation data, while the patient presentation layer is the management backend for the game content and analysis content by LongGood

LongGood develops an interactive rehabilitation game system named HappyGoGo that supports simultaneous activities of multiple people (up to 6 people), and is equipped with a computer vision model and a depth sensor to instantly identify the joint movement angles of multiple elders for rehabilitation therapists to judge.。

Incorporate gait analysis, named GaitBEST as a rehabilitation assessment tool

Rehabilitation has two essential elements: training and evaluation. Training is used to improve the patient’s physical activity, and evaluation is an important tool to measure the effectiveness of training. LongGood has developed hundreds of interactive game scenarios for rehabilitation training, and in 2019, they were further included in the evaluation as part of the product blueprint.

At that time, they asked the clinicians of the first-line hospital, went to Kaohsiung Chang Gung Neurology Department, and locked the gait analysis as a rehabilitation evaluation tool. They first targeted patients with Parkinson’s disease, and launched a four-year study to create a proprietary system to determine the risk of falling and the risk of falling based on data such as the patient’s pace, stride length, and joint angles (such as hip, knee). the severity of the disease.

They found that reductions in stride length and pace were warning signs of a fall. These two parameters have become important indicators to monitor whether patients are deteriorating. Hospitals can analyze the gait of patients before and after rehabilitation and before and after medication, and formulate treatment strategies.

Later, LongGood also worked with other hospitals to optimize the gait analysis system, and further included the risk assessment of stroke, sarcopenia and other symptoms. In 2021, the system was also certified as a medical material by the US Food and Drug Administration FDA Class II.

The biggest challenge is to meet the needs and create a universal version

However, like many medical software based company, LongGood has encountered many challenges in system development, the biggest of which is generalization. Because Taiwan’s medical care is a highly customized industry, most hospitals have different needs, but the strategy of LongGood is “customized development one by one." Ray pointed out that his principle is to clarify needs and create a public version. The system, including training content, analysis templates and background management system, is used by medical institutions.

However, with the development of time, the LongGood market is getting bigger and bigger, and more and more medical institutions are using it. They have to face the need to adjust the public version system architecture at any time. Because this public version system has to take into account the common needs of hundreds of hospitals, and also meet the customized requirements of hospitals. For example, a hospital puts forward its own needs and hopes that rehabilitation patients will come to the hospital, insert their own health insurance cards and complete rehabilitation training. , LongGood has to adjust the public version system architecture according to this individual requirement and on the premise of maintaining versatility. “That’s what we’re working towards," he said.

To modularize the product, the future will be Android in the field of rehabilitation industry

Another goal that LongGood is working on is product modularization. For a long time, the core functions of LongGood’s software have been integrated, tying the database, interactive training content, analysis content, evaluation content and cloud computing programs together.

But now, Ray believes that rehabilitation software should be integrated with hardware equipment in order to achieve greater benefits. To this end, LongGood must first disassemble and modularize the original core functions of the software for system partners to access and connect data in order to integrate services, and modularization is also beneficial to LongGood’s subsequent pricing operations. At present, the team is thinking about how to divide the training content to be more efficient and systematic on Android devices, such as TVs and set-top boxes, to provide on-demand functions, which is one of the challenges to be overcome this year.

Even Ray focused on the integration of software with hardware equipment such as lower limb training robots and exoskeleton robots, “which can achieve twice the rehabilitation effect!" He said that there are many excellent rehabilitation hardware manufacturers in Taiwan, and the world’s major countries are willing to pay for these. If hardware is added with software services, it will be able to exert higher value and provide better services. He revealed that LongGood has cooperated with Taiwan’s No.1 lower limb training robot manufacturers, and will cooperate with more manufacturers in the next step.

This is the goal that the LongGood aims at, to provide software interfaces and functions for rehabilitation hardware equipment, just like Google provides the Android operating system to many smartphone manufacturers, “We want to become the Android of the rehabilitation industry!"

Ray Chen Founder and CEO of LongGood MediTech

Education: Graduated from the Institute of Bioengineering, National Taiwan University

Experience: After graduating from the Institute of bioengineering of National Taiwan University, he joined the Optoelectronic Medical Center of National Taiwan University Hospital as an imaging management engineer in 2006. Since 2009, he has worked as a software engineer and product manager at Fisher & Paykel, a medical equipment manufacturer in New Zealand. In 2011, the Keel King Preparatory Office was set up, and it was officially established in 2013, specializing in the creation of rehabilitation training software and evaluation tools. In 2017, he went to UCBerkeley as a visiting scholar with funding from the National Academy of Sciences of Taiwan.