When consumption meets AI | From 'Metaverse' to 'heart universe,' human organs such as the heart and brain are being rebuilt

The digital twin technology is changing the medical paradigm. Nowadays, doctors can reconstruct various organs from the heart to the brain of patients through three-dimensional images. In the near future, it is also expected to build a virtual twin of the entire human body, making surgery and diagnosis more precise and predictable.

With the development of large-scale artificial intelligence technology, stronger computing power will make the construction of future 3D models faster and more convenient. After integrating AI imaging information, doctors will also be able to make a more comprehensive assessment of patients' health.

From virtual heart to virtual brain

Six months ago, software engineer Steven Levine underwent a 12-hour brain tumor removal surgery. After being diagnosed with a benign tumor the size of a golf ball in his brain, doctors modeled his brain, pinpointed the exact location of the tumor, and performed an intervention surgery guided by 3D imaging. Six weeks later, Levine fully recovered from the surgery.

"Although the tumor will not immediately endanger my life, it has eroded part of my skull, nasal sinuses, and compressed the optic nerve," Leben told Caijing reporters. "The tumor also affects pituitary function, secreting excessive growth hormone, causing my feet and hands to slowly enlarge."

Thomas Beaumont, a neurosurgeon at the University of California, San Diego, used digital twin technology to reconstruct Levin's brain, including tumors. During the operation, using images on the screen in the operating room, Beaumont inserted a camera into one of Levin's nostrils and sent surgical tools into the other nostril to remove the tumor block by piece and repair the damaged tissue, performing the operation in a non-invasive manner.

As the bone structure is different for each patient. The carotid arteries also bend differently. All of this must be visualized in 3D to ensure the success of the surgery. With 3D images, doctors don't have to imagine the anatomy of the patient's brain in their heads, but can see them directly, which puts the doctor in a more certain state during the operation and makes the operation more precise.

Levin's surgery was very successful. Half a year later, he was back to work as usual. In fact, the digital twinning technology used by Dr. Beaumont was developed by Levin himself, making him the beneficiary of his own developed technology. His experience has made him even more convinced that this technology will be able to help more patients in the future and alleviate the burden on doctors.

Currently, Livn is advancing digital twin technology from the laboratory to the clinic, including digital twins of various human organs such as the heart, brain, and liver. Livn has been working on this for over ten years.

In 2014, Lai Wen, in his company Dassault Systemes, first launched the Living Heart project, which was the first digital human heart modeling tool approved by the FDA in the United States. It can help interventional cardiologists better understand the structure of patients' hearts before or during surgery, thus finding the best surgical plan.

Levin told the First Financial Journalist that the inspiration for his virtual heart project came from his daughter Jesse. Jesse was born with a rare and serious heart defect. Doctors implanted a pacemaker in her heart when she was born. Jesse is now 35 years old and has had 5 pacemakers replaced in her heart.

Doctors faced a lot of uncertainty when treating Jesse's heart disease, mostly relying on experience and guesswork, which left a deep impression on Levin. Since then, Levin has been hoping to develop a digital system that can help doctors better study the development of congenital heart disease in children like Jesse.

"Just as no one would now manufacture a real plane or car and then drive it for a test drive," Leven told the First Financial reporter, "Before these products are actually assembled, software programs and computer systems allow designers to first create and test these parts in a virtual environment, so why can't heart surgery and coronary intervention devices be tested and planned in advance?"

Levin, an engineer by training, has always thought about this with a physical mindset, and he believes that since the heart is a "pump", it should be subject to the laws of physics. This idea led to the creation of his Virtual Heart project.

"If the constructed model is correct, the final product is correct," he said. And this success comes from the open-source nature of the system. Levine said that the open-source collaboration of the project has been going on for ten years. Hundreds of doctors, engineers, industry standard setters, and government officials from around the world have participated in the project, contributing their professional skills with the goal of building the first fully functional 3D heart in a virtual environment.

Lai Wen told First Financial Journal that at first, many people doubted this project. After all, every heart surgery is related to life, and currently, heart surgery still relies on the skills and experience of doctors. However, gradually, the three-dimensional model is created from common CT scans and magnetic resonance imaging (MRI) images, the software is gradually improved, and the project is gradually verified, performing well in animal and final human tests.

The project driving the virtual heart is an evolving ecosystem, where we collaborate with top cardiovascular researchers, medical device developers, regulatory agencies, and cardiac experts, aiming to develop clinically validated, highly precise, commercializable personalized digital human heart models and novel digital therapies, according to Levin.

Children with congenital heart disease are expected to benefit first

After years of effort, Levin's team has successfully built a fully functional virtual heart and made it available to pediatric cardiac surgeons at Boston Children's Hospital. In Levin's view, as pediatric cardiologists lack predictive tools for the development of congenital heart disease in children, pediatric cardiology is expected to be the first department to commercialize digital twin technology.

Now, twice a week, dozens of experts gather in the cardiac department of Boston Children's Hospital to plan the most complex heart surgeries. They analyze three-dimensional digital images of the heart projected on the screen, where every damaged vessel or malformed ventricle could pose a threat to the child's life and health.

These three-dimensional digital images can be rotated or broken down into chunks on a computer screen, allowing surgeons to precisely plan upcoming surgeries. With the assistance of biomedical engineers, doctors can observe the flow of blood and oxygen, the electrical signals of the heart, and how the pressure of the valves affects cardiac function. Doctors can even use digital images to predict the impact of the valves they plan to use on heart repairs.

Levin said that someday, sensors or wearable devices will be added to these digital technologies to create a pathway for human hearts to transmit data to patients' virtual hearts. This feedback loop will create a 'digital twin' of the human heart, providing doctors with new methods to ensure that their surgical plans are optimal.

David Hoganson, a pediatric cardiac surgeon at Boston Children's Hospital and director of the 3D computer visualization project, leads the project. As of now, his team has completed about 2000 surgeries using digital heart models.

At the Children's Heart Center of Xinhua Hospital affiliated to Shanghai Jiaotong University, Chen Sun, the director of the Cardiology Department, has to deal with a large number of complex congenital heart disease patients every day. These children sometimes need treatment just after birth due to poor ventricular development, and some develop Kawasaki disease or cardiomyopathy at the age of 2 or 3. Many mechanisms of these diseases are still not clear.

Chen Sun told First Financial Journalists that his team is also conducting industry-enterprise scientific research cooperation in digital twinning, based on the fusion of multimodal images of echocardiography and enhanced CT of the heart and great blood vessels, developing relevant algorithms, constructing 3D and 4D models of pediatric congenital heart disease, to better understand the development of pediatric congenital heart disease.

Chen Sun frankly stated that the main reason why foreign digital technology is leading in clinical applications is based on two advantages: first, they have a mature payment system, so products can achieve commercialization faster; second, the overseas ecosystem is relatively developed, forming a system of interdisciplinary research in basic medicine, engineering, and clinical fields.

Dr. Peng Yongxuan, chief physician of the Children's Heart Center at Xinhua Hospital, told First Financial Reporter, "Virtual heart has become a research hotspot in the domestic cardiovascular field. Digital twin technology is an inevitable trend to further develop the medical level, and will bring revolutionary changes to the medical field."

He stated that in the field of pediatric heart disease in China, virtual heart technology is in the early stage of research and optimization through AI algorithms, multi-modal image fusion, and other amplifications, continuously improving the accuracy of digital twin models. In the future, further exploration will be conducted on its application in actual clinical settings.

"As a new type of technology and auxiliary tool, the powerful functions of digital twin hearts still need a systematic method to be used, for example, some long-used two-dimensional parameters (diameter, area) will be replaced by three-dimensional parameters (area, volume). Preoperative planning, surgical simulation, and prediction of disease progression, etc., all need to develop corresponding methodologies according to different diseases," Peng Yong declared.

In recent years, "digital twins" have matured in the medical field, developing models for the lungs, liver, brain, joints, eyes, blood vessels, and other parts of the body. It is also expected to create a virtual twin of the entire human body in the near future. Currently, this new technology has been used to test new medical devices and predict the effects of new drug molecules on organs and cells. In the future, it may reduce or even replace animal experiments.

These models will establish a unified basis for cardiovascular computational medicine, promote the development of education and training, medical device design, clinical trials, and clinical diagnosis, and provide a more effective pathway for cutting-edge medical device innovation. Currently, the FDA has extended its collaborative research agreement with Dassault Systèmes' virtual heart for 10 years to evaluate the implantation, placement, and performance of cardiovascular devices including pacemakers.

With the development of generative AI, Dassault's virtual heart project is introducing large language models. Leven told the First Financial Reporter that his team is now testing a new generation of virtual heart models, which can be configured for individual patients or patient groups, and the AI-driven customization and automation capabilities will help simplify and accelerate the development of medical devices.

"One advantage of AI models is that in the future, we may not need a large amount of patient data. We only need a small amount of data to allow the model to continuously learn and generate a large amount of data. This is an important impact of AI development on digital twin technology," Levine said.

In addition, with the help of AI, the past manual modeling of each individualized heart will also undergo changes. "Manual modeling often takes a long time, but now it can be shortened from days to minutes through one-click automated modeling, which will be revolutionary for the entire process," he said.

The era of AI medicine is coming

In China, clinicians are also exploring the application prospects of virtual twin technology. Recently, in the operating room of Renji Hospital affiliated to Shanghai Jiao Tong University School of Medicine, a man-machine collaborative "precision war" was launched. Zhou Hongyu's team from the Functional Neurology Group of Renji Hospital imported the patient's CT and MRI image data into a surgical robot system called Sino, which can automatically reconstruct the intracranial 3D model and accurately outline the three-dimensional outline of the abscess through the AI algorithm system.

With the help of three-dimensional data imaging, under the shadowless light, Dr. Guo Liemei, deputy chief physician of the Neurosurgery Department of Renji Hospital, skillfully manipulates the mechanical arm of the surgical robot, and according to the surgical path planned by the robot, slowly inserts the drainage tube into the center of the abscess through the preset channel with a repeat positioning accuracy of 0.1 millimeters.

Zhou Hongyu explained to the First Financial reporter that in the past, this type of surgery relied on the doctor's 'feel' and experience. Now, with the development of neural imaging technology, image data from CT and MRI can be post-processed to construct deep brain structures that could only be observed through craniotomy in the past. Based on the three-dimensional system, surgical paths can be planned to avoid blood vessels and important functional areas. Precision positioning and surgical navigation can be achieved through robots, breaking through the physiological limits of the human eye and hand.

"The system displays real-time puncture depth, angle offset, and can cleverly avoid blood vessels and functional areas, which is a perspective that traditional surgery cannot achieve," he said.

Professor Feng Junfeng, director of the Neurosurgery Department at Renji Hospital, told First Financial Reporter: 'Neurosurgery has entered the era of 'millimeters'. Machine learning continues to accumulate surgical data, and in the future, it may autonomously optimize puncture paths, becoming the 'AI assistant' of doctors.'

A similar scenario is expected to be played out in the cardiac catheterization laboratory of the hospital in the future. Academician Ge Junbo, director of the Department of Cardiology of Zhongshan Hospital Affiliated to Fudan University, described a "Metaverse Catheterization Laboratory" scenario to the CBN reporter: the catheterization laboratory is composed of an artificial intelligence-assisted decision-making system, a voice-assisted control system, a robot-assisted and haptic feedback system, a mixed reality holographic digital system and a high-speed Internet.

"In this catheter room, before the patient undergoes surgical treatment, all the information has been mirrored to a set of software systems worn by the doctor using mixed virtual devices like Vision Pro, which can simulate the surgery and anatomy, and the process of the surgery has been rehearsed in advance." Ge Junbo told the First Financial Journalist.

He believes that the manifestation and dimensions of the metaverse are equally applicable to the diagnosis and treatment of heart disease. From the metaverse to the heart universe, future digital twins can accurately construct a digital organ (digital human) with the help of AI technology, outlining possible scenarios that may occur in real situations (virtual), enabling doctors and patients to understand the consequences of a disease factor, and so on.

"This will help understand the correlation between various complex vascular diseases, such as how atherosclerotic diseases affect the entire vascular network of the human body," Ge Junbo said. "This is of great significance for the diagnosis and management of pan-vascular diseases. Traditional knowledge is no longer able to predict all the consequences of the disease. In the future, the integration of patient characteristics, clinical manifestations, biomarker omics, and imaging omics will become a trend."

Ge Junbo said that the combination of digital twinning technology and artificial intelligence large models can accurately predict systemic vascular diseases in the future, and can play a greater role in the field of valve replacement surgery, helping doctors make predictions and decisions on when to intervene in diseases.

Just last week, the Cardiology Department of Zhongshan Hospital released the first large-scale heart model observation system in China (CardioMind). By integrating multimodal diagnostic data with top doctors' experience, the model achieves full-process intelligence from medical history collection to assisted diagnosis. More importantly, the system breaks through single-text data analysis and achieves integrated reasoning of multimodal data such as electrocardiograms, ultrasound images, and laboratory tests.

However, the standardization of this data still faces challenges. "Digital twins describe organs based on various medical 3D images and physiological signals. CT/MRI 3D images are the foundation of digital twins. Due to differences in organ structures among individuals, these real-time images can be used to adjust AI models for personalized applications," said a medical imaging expert to a reporter from the First Financial Daily. "Currently, there is no unified standard for organ digital twins, but they can be simplified according to specific needs, such as for surgical guidance scenes."

Some clinicians also pointed out to the first financial reporter that due to the large subjective variability of the physiological data of the human body, this will increase the difficulty of standardization. "Human body data is extremely complex, with hundreds of millions of variables interacting with each other, and it is very difficult to accurately simulate it." Professor Pan Wenzhi, chief physician of the Department of Cardiology of Zhongshan Hospital, told the first financial reporter, "In order to prove whether a therapy is feasible, even if it is a simple hypothesis, it is usually necessary to include thousands of patients for several years of clinical trials, which costs hundreds of millions of yuan." Because the patient cannot be repeated trial and error. ”

He also said that the application of AI large models has the greatest advantage for objective data, such as image data, pathology, and blood data. By entering these data locally, specific auxiliary functions can be provided to doctors.

Levin told First Financial News that the virtual heart project is still awaiting evaluation by regulatory authorities, and large-scale entry into clinical trials is still pending. When talking about challenges, he said: "The lack of standards is the biggest challenge. Not only is every company developing its own products without unified standards, but also the standards in each country are different. Therefore, the industry needs to quickly establish standards for digital twin technology, provide references for regulatory approval, and push the technology into clinical trials as soon as possible."

(Source: First Financial)

Source: Oriental Fortune Network

Author: First Financial