High-Definition 3D Stereoscopic Microscope Display System for Biomedical Applications

Abstract

Biomedical research has been performed by using advanced information techniques, and micro-high-quality stereo images have been used by researchers and/or doctors for various aims in biomedical research and surgery. To visualize the stereo images, many related devices have been developed. However, the devices are difficult to learn for junior doctors and demanding to supervise for experienced surgeons. In this paper, we describe the development of a high-definition (HD) three-dimensional (3D) stereoscopic imaging display system for operating a microscope or experimenting on animals. The system consists of a stereoscopic camera part, image processing device for stereoscopic video recording, and stereoscopic display. In order to reduce eyestrain and viewer fatigue, we use a preexisting stereo microscope structure and polarized-light stereoscopic display method that does not reduce the quality of the stereo images. The developed system can overcome the discomfort of the eye piece and eyestrain caused by use over a long period of time.

1. Introduction

Humans receive a great deal of information through the five senses and receive about 70% by the visual sense. Video techniques based on the visual sense are extending to television broadcasting, machine, and image processing applications as well as to other applications. Video techniques have developed rapidly with the realization of HDTV broadcasting. Recently, we have tried to reproduce high-resolution three-dimensional stereoscopic images in a way that allows a direct watching of microscopic imaging. Because of the perception of depth, the 3D stereoscopic imaging has various applications [1–3].

The industry has been trying to create a 3D display for more than 10 years by using a wide range of pure optical schemes. Glasses and helmets, microlens screens, and a parallax barrier, which are all well known, did not satisfy customers and had insignificant commercial success [4]. There are a number of reasons for this: poor quality of 3D images, physiological and psychological problems including the "fatigue" effect, high price of the end unit, and dramatic changes needed for adoption to entertainment, and monitor manufacturing industries. Recently, there has been significant development in the field of stereoscopic imaging devices because of technology improvements, and all the research on stereoscopic devices has focused on the new 3D display technology. Hence, we saw a need to develop an image acquisition, recording, and playback device and search for a stable display method in the operating microscope field.

Two earlier studies [4–7] show that in medical education depth perception can be enhanced by displaying stereoscopic video content to medical students on personal stereoscopic displays. Although multimedia teaching cannot replace hands-on experience or direct contact with patients, in many cases physical presence in a theatre is no longer necessary to understand the idea of surgical concepts that are designed to remove pathology and preserve intact structures plus restore function. From these earlier results we concluded that a stereoscopic image displayed in high resolution on one large screen could facilitate orientation for the junior surgeon so that the right surgical plane is found and maintained while image resolution is still comparable to the original optical image.

There have been few reports on the evaluation of stereoscopy in surgery. Von Pichler and coworkers noted in their study on a laparoscopic model that stereoscopy made no difference to the operation time of an experienced surgeon, although it facilitated orientation and reduced operation time for inexperienced surgeons compared to monoscopic vision [8]. Although the system can require all viewers to wear polarized filter glasses, the stereoscopic viewing impression is robust against position changes and the number of viewers.

Stereoscopy devices have been used in the operation theatre before, either as a standalone solution or in combination with three-dimensional volume rendering from computer-assisted surgery [9]. However, these devices have rarely served as the main source of visual information for the surgeon, as their resolution is limited. In our study, stereoscopic video in high-definition resolution has been used to replace the direct microscopic view to perform surgical applications.

Our work focused on the development of a full HD stereoscopic imaging device with acquisition, display, recording, and playback technology for the operating microscope, experiments on animals, HD stereoscopic content development in biomedical applications, and so on.

The stereoscopic image acquisition device consists of a 3D camera using a stereo microscope with two small HD cameras. The recording system consists of a 3D Multiplexer/DeMultiplexer device for making one channel image from two channels images of the left and right channel, and vice versa [10]. Last, the assessment of a wide number of display types has led to the choice of well-matched conventional television displays with images separated by polarized light filters for our medical applications. The goal of our approach is to develop a system which allows a full HD stereoscopic imaging system for the surgeon or for experiments on animals through stereo microscope optics.

This paper is organized as follows. Section 2 illustrates our proposed full HD 3D stereoscopic microscope imaging system which can be applied to biomedical applications. Section 3 describes the evaluation results obtained from real animal surgeon test by using the developed system, and finally, the results of this paper are summarized and future research directions will be presented.

2. Full HD 3D Stereoscopic Microscope Imaging System

2.1. Overview of the HD Stereoscopic Microscope Imaging System

Many problems of using a microscope relate to the fact that the surgeon's eyes must remain fixed on the microscope eyepieces. In this case, eye flicker affects eye fatigue very much. If an observer's eyes flicker, it is difficult to focus on the object. It also cannot be observed by the other surgeons at once. In the case of a 2D image display using a microscope, a wrong diagnosis or mistake in a surgical operation can occur because of the lack of depth information and difference between the 2D image and the human visual system. Using an SD (640 480 pixel) quality image can be difficult to precisely recognize.

Figure 1 shows microthread image samples to compare the limit of resolution with full HD and SD images. The microthread is the use of microsurgery; in the case of the SD image, a small object cannot display exactly. When high ratio zooming is used for an accurate image, the field of view (working area) becomes narrow. For a more precise image display, we must use a high-resolution video system like a standard full HD video system, which can facilitate the search for components. In Figure 1, the distance of the scale mark is 0.001 mm, and the diameter of the microthread is 0.002 mm.

Figure 1

Microthread image samples for comparing the limit of the resolution with HD and SD images. (a) Sample of a full HD image of a microthread: Object Lens 0.5, Zoom magnification 2, CCD Lens 0.35, HD CCD Size 1/3 inch 3CCD, 24 inch TFT-LCD monitor. (b) Sample of a SD image of a microthread: Object Lens 0.5, Zoom magnification 2, CCD Lens 0.35, SD CCD Size 1/2 inch CCD, 24 inch TFT-LCD monitor.

The basic components of the HD stereoscopic imaging system are illustrated in Figure 2. The system is divided into three major functional modules: image acquisition, image processing, and display. The image acquisition part acquires a stereoscopic (left and right) image at the structure of the pre-existing stereo microscope and the image processing. The recording part includes a 3D multiplexer (Mux) and demultiplexer (DeMux) which has the function of compressing or decompressing two channel video signals into one channel or vice versa. Last, the display part displays the stereoscopic image. In our implementation, two 1920 1080 p HD cameras (ICONX Inc., USA) are used to operate the microscope. The cameras are attached at the left and right photo adapters that are spilt by the beam splitter, which functions as a divide between the two optical paths. To display a direct signal from the HD video acquisition part of microscope system, two channels of HD-SDI signal are transferred to the left and right TFT-LCD monitor without HD video recording part. To record the HD stereoscopic video, 3D Mux and DeMux are connected to HD video acquisition part.

Figure 2

Diagram of the stereoscopic microscope imaging system.

2.2. Stereoscopic Image Acquisition of the Stereo Microscope

The stereo microscope uses two separate optical paths with two objectives and two eyepieces to provide slightly different viewing angles to the left and right eyes. In this way it produces a 3D visualization of the sample being examined. There are two major types of magnification systems in the stereo microscopes [11–13].

One is the Geenough type, which is fixed magnification in which primary magnification is achieved by a paired set of objective lenses with a set degree of magnification. The other is a center mounted object (CMO) type that has zooming or pan pancreatic magnification, which are capable of a continuously variable degree of magnification across a set range. In order to acquire the stereoscopic image we have to use the CMO type, which is easy to attach to the CCD camera or order device. Figure 3 shows a schematic diagram of the stereoscopic image acquisition system using the COM type of the stereo microscope. By adding the beam splitters in the two optical paths of the stereo microscope we can acquire stereoscopic video images through the CCD cameras and the eyepieces are always the same. The system is a manufactured basic structure of the parallel stereo camera, which is two small HD CCD heads attached on a special camera base device.

Figure 3

Schematic diagram for stereoscopic image acquisition of the stereo microscope.

The stereo microscope can be analyzed like stereovision because of the two slightly different optical paths about one object. Figure 4 shows the schematic diagram for the stereovision analysis part at the optical structure of the stereo microscope. The CCD is assumed to be the coordinate system; and represent the points which are the object position . The distance is estimated by the stereovision method as follows:

Figure 4

Principle of stereovision method.

(1)

where is the focal distance of the cameras, and s is the distance between the left camera and the right camera. Like the simple equation (1), it is easy to calculate the depth or distance.

2.3. Stereoscopic Display System

In our study of the stereoscopic display system for a microscope in medical applications, the most important conditions are precision to make accurate observations and stability for long time use and minimized watching fatigue. Table 1 shows an explanation of the typical stereoscopic display methods. There are other stereoscopic display devices which have advantages as well as disadvantages. The combination of cathode ray tube monitors (or LCD) with shutter glasses is a widely proven method [9, 14]. Its main drawback is the need for synchronization between the display and the glasses, either by a cable or by an infrared link. Autostereoscopic displays avoid the necessity to wear glasses. However, the range of movement is extremely limited as there are only a few defined positions in front of the screen to obtain a proper stereoscopic image.

Table 1 Stereoscopic display methods.

This study develops the use of the polarized-light technique for medical applications. The polarized light stereoscopic display technique, which uses undulation of the light wave, separates a combined stereoscopic image into right and left images using orthogonal oriented filter sheets. Although such displays are bulkier than many alternatives, they do give a display which does not reduce the information content of the picture and only requires the viewer to wear polarized spectacles. Figure 5 shows a schematic of the polarized light stereoscopic monitor. The display using TFT-LCD monitors uses a half beam splitter to accurately combine the two pictures. The monitor faces are arranged at to one another horizontally. The TFT-LCD monitors use the polarizing filter; so our developed stereoscopic monitor is not needed. Through the polarizing glasses, we can watch the right image directly and the left image is a reflected image by a semireflective mirror.

Figure 5

Schematic diagram for the polarized light stereoscopic display.

2.4. Stereoscopic Image Processing Device for Recording and Playback

The HD stereoscopic image recording uses the general rule of the 3D format because stereoscopic images are limited by memory capacity, processing time, synchronization problem between left and right video signals, and processing time for the en/decoding. The devices for the general rule of the 3D format are generally called 3D Mux (multiplexer) and 3D DeMux (demultiplexer). The 3D Mux has the function of compressing two channel video signals into one channel and the 3D DeMux has the reverse function. There are two types of 3D Mux/DeMux of the frame sequential and spatial compression (side by side) methods. Figure 6 shows 3D Mux/DeMux of the two types of 3D formatter. In this work, we developed a 3D HD formatter which is available in two types through the program.

Figure 6

Schematic diagram of 3D Mux/DeMux for two types of the 3D format.

2.5. System Development

In this section, we describe the system development of the HD stereoscopic display system of the stereo microscope. The system consists of a stereo microscope (Olympus SZX model), 3D Mux/DeMux, and stereoscopic monitor. The stereoscopic image can be acquired by each of the two Full HD CCD camera ports of the Olympus SZX stereo microscope.

The 3D Mux/DeMux device plays an important role in the image recording and playback with a recording device like a tape recorder or semiconductor memory. The 3D Multiplexer input receives from the synchronized HD component signals of two HD CCD cameras. A full HD1080i analog image converts to a digital image through the ADC7403 (Analog Device Company), which can control the video mode or clock speed by I2C. The function of Multiplexing/De Multiplexing is realized by the process of the writing and reading in the frame buffer about the input image.

The stereoscopic monitor is a polarization-mode stereoscopic monitor using two high resolution TFT-LCDs that show the stereoscopic image through two HD input images. Figure 7 and Table 2 show the developed stereo microscope stereoscopic display system and its specifications.

Table 2 Specifications of the full HD microscope stereoscopic image display system.

Figure 7

Embodied Stereoscopic microscope display system: (a) HDTV stereoscopic microscope, (b) 24-inch polarized light stereoscopic monitor, and (c) 3D Mux/DeMux.

3. System Evaluation

In general microscope magnification environment, it is difficult to display marks of sutures (11/0) appeared after surgical operations through SD images more clearly, but these marks can be confirmed in full HD vivid images. Since full HD video standards are established for highest resolution, we make design to full HD microscope stereoscopic display system which consists of the microscope stereoscopic image acquisition, recording/playback device, and stereoscopic display monitor. And also, the embodiment system proposed in this paper has been evaluated as the best device in the field of stereoscopic display system.

Now we have performed various tests on animals with the full HD stereoscopic display system of the stereo microscope. One of these animal experiments was a heart transplant of a rat in the Department of Thoracic and Cardiovascular Surgery at the Medical College of Chungbuk National University. Because the SD image has low quality or resolution, we have evaluate just comparing directly watching microscope eyepieces (depend on optical system) with our proposed system. Figure 8 shows scenes of the animal test using the proposed full HD stereoscopic displays system of stereo microscope.

Figure 8

Scenes of the surgical test and video sample images: (a), (b) scenes of the animal test; (c), (d) left and right image samples in the HD stereoscopic video.

The heart transplant of a rat is often time consuming (about 2 to 3 hours), and like a microsurgery, visual information is mostly dependent on the stereo microscope. Because of the fixed eyepieces, after the experiment is finished, the surgeon feels a great deal of eye fatigue. To adjust to the changing environment of the experiment, the tester exercised simple or repetitive work for 30 minutes using the proposed display system. The evaluation has been performed by comparing optical microscope depending systems with the full HD stereoscopic display system of the stereo microscope. Even though the evaluation can be done at various viewpoints, the evaluation viewpoints used in this paper are focused on accuracy of the 3D image by depth information, eyestrain for the long time operation, operation efficiency, reusability for captured information, and system configuration settings. The results of the system evaluation are shown in Table 3.

Table 3 The evaluation results of the proposed system compared with existing optical microscope system.

In the case of an optical microscope system, image quality is very good, but it is limited at the actions because of the fixed eyepieces, which is a major cause of eyestrain and a serious problem of the optical microscope system. Our proposed system on the other hand needs time to adapt for the display system but more effort can improve accuracy.

According to the evaluation test, an ideal microscope display system will have to improve or develop like as follows:

(i)microscope image acquisition device—wider of field of view (FOV), color quality like human vision, and low-impact illumination by changing of environment around;

(ii)stereoscopic video recording/playback device—minimum changing of resolution or image quality when 3D multiplexing or demultiplexing;

(iii)stereoscopic display device—problem of the display monitor size and wearing the special glasses.

4. Discussion and Conclusion

The long hours of intensive work looking through an operating microscope have always been accepted as the price paid for working in this surgical subspecialty. Looking through microscope eyepieces, while performing a complex microsurgical reconstruction for hours sitting or standing relatively motionless often in uncomfortable positions, is not uncommon. It is not unreasonable to see how years of performing surgery under these strenuous physically demanding conditions could lead microsurgeons to prematurely retire from active practice, and in doing so, deprive the field of what could be their most fruitful years. The use of video technology in many surgical specialties is now considered routine. As an alternative to the operating microscope, advances in video technology can now permit the surgeon to view a microsurgical field on a video monitor in three dimensions without the necessity of physically looking through the microscope eyepieces. The conventional microscope magnifies the operative field and brings it mechanically and stereoscopically into the surgeon's view.

In conclusion, in the present study, we evaluated the use of the full HD stereoscopic display system of a stereo microscope and found that this technique provides several advantages over conventional microworking techniques. However, our evaluation of the technology showed less than optimal loss of illumination, inadequate par focal capability, and loss of depth and width. This study's preliminary data has guided the refinement of the stereoscopic display system for the stereo microscope.

Development of a full HD 3D stereoscopic microscope display system capable of providing a clear and accurate sense of depth perception has been a critical requirement in the rapidly evolving field of minimally invasive surgery and inspection. Our stereoscopic display system for the stereo microscope will be ready for widespread implementation and will positively affect the way microscopes work. The employment of surgical microscopy in the field of surgery has significantly contributed not only to the advancement of surgical technique but also to the operative outcome as well.