Published on in Vol 10, No 11 (2022): November

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/35138, first published .
Realizing the Potential of Computer-Assisted Surgery by Embedding Digital Twin Technology

Realizing the Potential of Computer-Assisted Surgery by Embedding Digital Twin Technology

Realizing the Potential of Computer-Assisted Surgery by Embedding Digital Twin Technology

Authors of this article:

Jiaxin Qin1 Author Orcid Image ;   Jian Wu1 Author Orcid Image

Viewpoint

Institute of Biomedical Engineering, Shenzhen International Graduate School, Tsinghua University, Shenzhen, China

Corresponding Author:

Jian Wu, PhD

Institute of Biomedical Engineering

Shenzhen International Graduate School

Tsinghua University

Building L, 3th Fl.

Taoyuan Street, University Town

Shenzhen, 518055

China

Phone: 86 13751113096

Email: wuj@sz.tsinghua.edu.cn


The value of virtual world and digital phenotyping has been demonstrated in several fields, and their applications in the field of surgery are worthy of attention and exploration. This viewpoint describes the necessity and approach to understanding the deeper potential of computer-assisted surgery through interaction and symbiosis between virtual and real spaces. We propose to embed digital twin technology into all aspects of computer-assisted surgery rather than just the surgical object and further apply it to the whole process from patient treatment to recovery. A more personalized, precise, and predictable surgery is our vision.

JMIR Med Inform 2022;10(11):e35138

doi:10.2196/35138

Keywords



As is well known, computer-assisted surgery (CAS) has been widely used in the field of surgery. It enables the precise positioning and visualization of the patient’s deconstructive structures and surgical instruments by integrating medical imaging, spatial positioning technology, and various computer technologies. Therefore, with CAS, the surgeon can plan the surgical procedure in a precise manner before the operation, such as the access point, implant selection, and placement [1]. However, intraoperative planning cannot be implemented accurately during surgery due to intraoperative disturbances such as respiratory movements and tissue deformation. Moreover, CAS is still lacking in postoperative prediction, prevention of surgical complications, and postoperative evaluation.

Digital twin (DT) is a concept or technology that refers to create a multiphysical, multiscale, and high-fidelity virtual representation of physical entity in the virtual space. We call this virtual representation the DT of its corresponding physical entity. The virtual representation can be dynamically updated when the physical entities change, enabling real-time mapping from physical space to virtual space. DT originates in the industry [2] and is now used in many fields such as precision health [3], manufacturing, construction, product design, and weather prediction [4]. Considering the features of CAS and DT, we propose to bring DT to CAS and explain why and how DT can be used to enhance the application of CAS in all its phases, especially the potential value in remote surgery when combining CAS with DT.


Figure 1 illustrates our proposed DT-based CAS solution. We collect data related to surgical objects, surgical instruments, and medical devices in the preoperative period, and use these data to build their multiphysical, multiscale, and high-fidelity DT models in virtual space by mathematical simulation and modeling [5]. DT models can dynamically simulate a wide range of properties of the patient’s tissues and organs, such as geometric, physical, physiological, and behavioral properties. Therefore, preoperative planning, surgical simulation, and postoperative prediction based on these models will be more accurate and comprehensive.

Information, as well as the changes of information from surgical instruments and the patient (such as anatomical structure, position, posture, etc), are acquired during surgery using real-time imaging technologies and a variety of sensors. Due to the high requirements for real-time performance, it is much more difficult to obtain whole information intraoperatively compared to preoperatively. For example, the clarity of the patient’s anatomical structure obtained by ultrasound imaging is much less than that of preoperative CT (computed tomography) and MRI (magnetic resonance imaging). However, we can quickly capture local feature information from ultrasound images through advanced methods such as deep learning [6]. Combining this local information with the preoperative DT models, which already have physical, physiological, and behavioral characteristics, it is possible to obtain the overall dynamic changes of the patient’s anatomical structure. In this way, intraoperative changes can be evaluated in real time, and all the information in the virtual space can be visualized when using extended reality (virtual reality, augmented reality, and mixed reality), and it also provides the surgeon with real-time, accurate surgical navigation.

Sensors can collect a variety of data that are needed to update the DT during the procedure. Commonly used sensors are physiological, mechanical, and position sensors. Among them, position sensors play a key role in positioning and tracking in surgical navigation and can help achieve a baseline position mapping relationship between the real surgical space and the DT surgical space. Optical sensors have higher accuracy and real-time performance compared with other position sensors. They are able to collect position and posture data of the optical marker in real time in the form of quaternions or transformation matrix. We usually fix the optical marker on the object of interest. For example, we can fix the optical marker on the ultrasound probe before the procedure and obtain the conversion between the optical marker coordinate system and the ultrasound image coordinate system by ultrasound probe calibration [7]. Intraoperatively, we can not only collect real-time position and posture information of the ultrasound probe in real time but also obtain the spatial position of any point displayed on the ultrasound image.

An application programming interface is a software intermediary that can create data links between different devices. We can use application programming interfaces of medical devices to obtain operational data in real-time and update the DTs. Through these DTs, we can monitor, control, and manage medical devices in a uniform way.

While the surgery is being performed in real space, a digital record of the surgery is updated simultaneously in virtual space, which we call the process twin of the whole surgery process. This record can be used for postoperative evaluation of the patient, including the evaluation of the actual surgery, the surgical procedure, the choice of strategy, and so on. It can also act as an important reference for postoperative follow-up.

Figure 1. Digital twin–based computer-assisted surgery (CAS) solution. With the support of digital twin surgery, we can perform monitoring, optimization, recording, and prediction for the real surgery. API: application programming interface; AR: augmented reality; CT: computed tomography; MR: mixed reality; MRI: magnetic resonance imaging; VR: virtual reality.
View this figure

It is the establishment of an accurate match between the real world and the virtual space that enables effective remote surgery on real patients using DTs. The surgeon can obtain the real-time situation of the surgery at the remote end by the dynamically updated DTs and can use a haptic device to remotely control the robotic arm to perform the surgery [8] through high-speed, low-latency communication technologies such as 5G. Since the DTs contain rich information about the surgical object, the surgeon can obtain important feedback information [9] at the remote end through the haptic device without any other sensors. For each surgical device, it can also be monitored and controlled remotely through their DTs in virtual space.


Data-driven approach is one of the core approaches to implement DT. The process of dynamically updating the DTs intraoperatively requires a large amount of data computation and interchange. By using cloud computing technology [10], hardware costs can be effectively reduced. Medical Cyber Physical Systems are the networked health care integration of medical devices [11]. They provide a superior way to capture, store, and securely access large amounts of medical data. In the future, its development may provide important data support for the application of DT [12]. Additional attention needs to be paid to the fact that the collection of private health data on human individuals may raise complex ethical issues [13]. Thus, data security should be carefully considered for the storage and retrieval of operation data, and data process encryption should be embodied in DT.


Bringing DT to CAS is to monitor, optimize, record, and predict the surgical process in the real space by creating a virtual twin surgical space (including the DTs of the surgical object, surgical instruments, and medical equipment). In this way, it can leverage and integrate data from the entire surgical phase and is applied to the patient from treatment to recovery. This also determines the higher level of complexity of the DT systems. Different types of data from multiple devices need to be integrated within the same system and ensure the system’s stable operation. The real-time dynamic response of the DTs requires high data transmission speed and network speed, especially in the remote surgery. In addition, the simulation of complex physiological signals of biological tissues is still a challenge that needs to be faced if a more detailed patient model is desired. In the future, the virtual twin space can be used as a carrier to establish an integrated, digital surgical process management system and to form a new clinical implementation system. Under such a system, surgical treatment will be more personalized, precise, and predictable.

Acknowledgments

The authors would like to thank the support of the Shenzhen Basic Research Project (JCY20200109142805928, JCYJ20160428182053361) and the National Key Research and Development Program of China (2019YFC0119500).

Conflicts of Interest

None declared.

  1. Joskowicz L. Computer-aided surgery meets predictive, preventive, and personalized medicine. EPMA J 2017 Mar 07;8(1):1-4 [FREE Full text] [CrossRef] [Medline]
  2. Tao F, Zhang H, Liu A, Nee AYC. Digital twin in industry: State-of-the-art. IEEE Trans. Ind. Inf 2019 Apr;15(4):2405-2415. [CrossRef]
  3. Fagherazzi G. Deep digital phenotyping and digital twins for precision health: Time to dig deeper. J Med Internet Res 2020 Mar 03;22(3):e16770 [FREE Full text] [CrossRef] [Medline]
  4. Fuller A, Fan Z, Day C, Barlow C. Digital twin: Enabling technologies, challenges and open research. IEEE Access 2020;8:108952-108971. [CrossRef]
  5. Corral-Acero J, Margara F, Marciniak M, Rodero C, Loncaric F, Feng Y, et al. The 'Digital Twin' to enable the vision of precision cardiology. Eur Heart J 2020 Dec 21;41(48):4556-4564 [FREE Full text] [CrossRef] [Medline]
  6. Shen C, He J, Huang Y. Discriminative correlation filter network for robust landmark tracking in ultrasound guided intervention. 2019 Presented at: International Conference on Medical Image Computing and Computer-Assisted Intervention; October 13-17, 2019; Shenzhen, China p. 646-654   URL: https://doi.org/10.1007/978-3-030-32254-0_72 [CrossRef]
  7. Wen T, Wang C, Zhang Y, Zhou S. A novel ultrasound probe spatial calibration method using a combined phantom and stylus. Ultrasound Med Biol 2020 Aug;46(8):2079-2089 [FREE Full text] [CrossRef] [Medline]
  8. Huang T, Li R, Li Y, Zhang X, Liao H. Augmented reality-based autostereoscopic surgical visualization system for telesurgery. Int J Comput Assist Radiol Surg 2021 Nov 07;16(11):1985-1997. [CrossRef] [Medline]
  9. Tholey G, Desai J, Castellanos A. Force feedback plays a significant role in minimally invasive surgery: results and analysis. Ann Surg 2005 Jan;241(1):102-109. [CrossRef] [Medline]
  10. Dang LM, Piran M, Han D, Min K, Moon H. A survey on Internet of Things and cloud computing for healthcare. Electronics 2019 Jul 09;8(7):768 [FREE Full text] [CrossRef]
  11. Dey N, Ashour AS, Shi F, Fong SJ, Tavares JMRS. Medical cyber-physical systems: A survey. J Med Syst 2018 Mar 10;42(4):74. [CrossRef] [Medline]
  12. Kocabas O, Soyata T, Aktas MK. Emerging security mechanisms for medical cyber physical systems. IEEE/ACM Trans. Comput. Biol. and Bioinf 2016 May 1;13(3):401-416. [CrossRef]
  13. Bruynseels K, Santoni de Sio F, van den Hoven J. Digital twins in health care: Ethical implications of an emerging engineering paradigm. Front Genet 2018;9:31 [FREE Full text] [CrossRef] [Medline]


CAS: computer-assisted surgery
CT: computed tomography
DT: digital twin
MRI: magnetic resonance imaging


Edited by C Lovis; submitted 22.11.21; peer-reviewed by S Mallick; comments to author 16.10.22; revised version received 23.10.22; accepted 25.10.22; published 08.11.22

Copyright

©Jiaxin Qin, Jian Wu. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 08.11.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.