Deep-learning cell imaging through Anderson localizing optical fiber Download: 852次
1 Introduction
In biomedical science and clinical applications, visualizations of real-time cell activity, morphology, and overall tissue architecture are crucial for fundamental research and medical diagnosis.1,2 This usually requires real-time in vivo imaging to be performed in a minimally invasive way with the ability to deeply penetrate into organs or tissues. Due to their miniature size and flexible imaging transfer capability, fiber-optic imaging systems (FOISs) have been widely applied to this domain.2
Recent burgeoning deep-learning technology and the latest discoveries of the novel properties of glass-air Anderson localizing optical fibers (GALOFs) open an avenue for overcoming these challenges and fundamentally promoting the overall performance of FOISs. Deep-learning technology is a fast-developing research field that has gained great success in imaging applications and demonstrated better performance than conventional model-based methods.23
The use of DCNNs for simple binary image recovery and classification after transport through optical fibers has been reported recently.32
Nevertheless, the design of the existing GALOF imaging system is based on a previous limited understanding of the localized modes in GALOFs. Hence, it is faced with several challenges limiting its practical application. First, the system only demonstrated success in imaging of low-resolution sparse objects, such as the binary MNIST handwritten numbers. There is a chasm between sparse binary object reconstruction and the reconstruction of biological objects, which are typically different types of cells or tissue with complicated morphologic features. Second, the demonstrated transfer-learning capability of the previous DCNN-GALOF system was limited to binary sparse testing objects that shared image features quite similar to those of the objects in the training data.32 For many practical applications, it would be highly desirable if the system would be able to perform transfer learning using objects that are significantly different from the training data. Third, the previous DCNN-GALOF system performs high-quality imaging under coherent laser illumination. The ability to perform imaging under incoherent broadband illumination in the fiber imaging system would be another important step toward practical applications. For example, white-light transmission cellular micrographs are already very familiar to histopathologists, and they prefer similar white light illumination for endoscopic images.8 Furthermore, the coherence of lasers results in speckle patterns, which often reduce the image quality. Last but not least, the high intensity of lasers light might be damaging to biological objects such as living cells and the cost of lasers is relatively high. In contrast, incoherent broadband illumination generally avoids speckle problems and the lower intensity of incoherent light sources helps to protect cells against photobleaching and phototoxicity during the imaging process. At the same time, the cost of incoherent light sources, such as LEDs, is much lower compared with laser systems. The latest research progress on mode properties of GALOFs offers a new possibility to overcome all of these barriers and enhance the system performance of FOISs. Recently, we prove that the wavefront quality of localized modes in GALOFs is close to that of an ideal fundamental Gaussian mode.42 Meanwhile, the mode density is orders of magnitude higher than that of MMF and MCF.42 Other related research further shows that localization lengths, comparable to point spread function, of localized modes are independent of wavelengths.44 Based on these latest discoveries, the GALOF has the potential to support a high-quality imaging process using a broadband incoherent light source. In addition, DCNN itself does not raise any requirements for illumination. Therefore, it should be possible to achieve high-quality imaging of biological objects under incoherent broadband illumination using the combination of GALOFs and DCNNs.
In this work, we develop an incoherent broadband light illuminated DCNN-GALOF imaging system with the capability to image various cell structures. Within this system, a DCNN model with a design tailored to the cell imaging task is applied, and a low-cost LED works as the light source. We call the new system Cell-DCNN-GALOF. We demonstrate that it is able to transfer high-quality, artifact-free images of different types of cells in real time. We further prove that the imaging depth of this system can reach up to several millimeters without any distal optics. In addition, we show that the image reconstruction process is remarkably robust with regard to external perturbations, such as temperature variation and fiber bending. Last but not least, the transfer-learning capability of the new system is confirmed using cells of different morphologies and classes for testing. The work presented here introduces a new platform for various practical applications, such as biomedical research and clinical diagnosis. The system performance of the Cell-DCNN-GALOF is superior to state-of-the-art systems. It is also a new cornerstone for imaging research based on waveguide devices using transverse Anderson localization.
2 Methods
The experimental setup and details of DCNN are shown in
In
3 Results
3.1 Imaging of Multiple Cell Types
To demonstrate the imaging reconstruction capability, two different types of cells, human red blood cells and cancerous human stomach cells, serve as objects. By scanning across different areas of the cell sample, we collect 15,000 reference and raw images as the training set, 1000 image pairs as the validation set, and another 1000 image pairs as the test set for each type of cell. During the first data acquisition process, the GALOF is kept straight and at room temperature of about 20°C. The imaging depth is 0 mm, meaning that the image plane is located directly at the fiber input facet. The training data are loaded into the DCNN [see
Fig. 2. Cell imaging of different types of cells: (a)–(c) test data for human red blood cells and (d)–(f) test data for cancerous human stomach cells. All data are collected with straight GALOF, at room temperature with 0-mm imaging depth. The length of the scale bar in (a1) is . (a1)–(f1) The reference images. (a2)–(f2) The corresponding raw images. (a3)–(f3) The images recovered from the raw images. [Video S1 , avi, 10 MB (URL: https://doi.org/10.1117/1.AP.1.6.066001.1 )].
3.2 Cell Imaging at Various Depths
Distal optics located at the fiber input end hinders conventional FOIS from miniaturizing the size of the imaging unit. Here we are investigating the ability of our Cell-DCNN-GALOF system to image objects located at various distances from the fiber input facet without distal optics. As illustrated in
Fig. 3. Multiple depth cell imaging: (a)–(f) Test data for human red blood cells. All data are collected with straight GALOF at room temperature. All three images in each column are from the same depth. The length of the scale bar in (a1) is . (a1)–(f1) The reference images; (a2)–(f2) the corresponding raw images. The distance between the image of the object and the fiber input facet is defined as the depth. Initially, the image of the object is located at the GALOF’s input facet with 0-mm depth. Then the imaging depth is increased in steps of 1 mm by moving the fiber input end using a translation stage. As illustrated in (g), (a2)–(f2) are obtained by varying the imaging depth from 0 to 5 mm with steps of 1 mm. (a3)–(f3) The images recovered from the corresponding raw images. (h) The averaged test MAE for each depth with the standard deviation as the error bar. More sample results, including reference, raw, and recovered images, are shown in Fig. S2 in the Supplementary Material .
3.3 Cell Imaging with Temperature Variation and Fiber Bending
In practical applications, the optical fiber of the FOIS often needs to be inserted deeply into the cavity of living organs. This requires the imaging system to tolerate thermal variation and fiber bending. For MMF-based FOIS, the increase of temperature or bending of the fiber when inserting the fiber into organs or tissues induces strong variations of the mode coupling. These variations decrease the performance of MMF-based imaging systems due to induced changes of the TM.9 This problem can be overcome using GALOF since most of the modes embedded in GALOF show single-mode characteristics, which increase the system tolerance and can make it immune even to rather strong perturbations. We first investigate the effect of temperature variation on our Cell-DCNN-GALOF system by changing the temperature of a 10-mm-long GALOF segment with a heater. During the data collection, we keep the GALOF straight and at 0-mm imaging depth. We collect 15,000 image pairs at 20°C as the training data. For test data, we record three sets of test data where the GALOF segment is heated to 20°C, 35°C, and 50°C, respectively. Each set of test data consists of 1000 image pairs. The DCNN model is only trained utilizing the training data collected at 20°C. Subsequently, the trained model is applied to perform test image reconstruction of data acquired at all three temperatures. In
Fig. 4. Cell imaging at different temperatures. (a1)–(c1) Test raw images of human red blood cells collected at 20°C, 35°C, and 50°C, respectively. The scale bar length in (a1) is . (a2)–(c2) Images recovered from (a1)–(c1); (a3)–(c3) the corresponding reference images. All data are collected with straight GALOF at 0-mm imaging depth. (d) The averaged test MAE for each temperature with the standard deviation as the error bar. More test sample results, including reference, raw, and recovered images, are provided in Fig. S3 in the Supplementary Material .
Next, we test the effect of fiber bending on the performance of our Cell-DCNN-GALOF system. We keep the temperature of the fiber at room temperature and the imaging depth at 0 mm. We collect 15,000 image pairs with straight GALOF as the training data and record five sets of separate test data corresponding to five different bending states. Each test set consists of 1000 image pairs. Experimentally, the bending is induced by moving the fiber end by a specified offset distance as illustrated in
Fig. 5. Cell imaging under bending. (a)–(e) Data in each column correspond to examples with the bending offset distance listed above. The definition of offset distance is illustrated in Fig. 1(b) . The bending angle range corresponding to offset distances between 0 and 2 cm is about 3 deg. For more details, see Sec. 2 . (a1)–(e1) Raw images collected at different bending offset distances. The scale bar length in (a1) is . (a2)–(e2) Images reconstructed from (a1)–(e1); (a3)–(e3) the corresponding reference images. (f) Averaged test MAE for five different bending states with the standard deviation as the error bar. More sample results of human red blood cells, including reference, raw, and recovered images, are provided in Fig. S4 in the Supplementary Material .
3.4 Cell Imaging Transfer Learning
We have shown that our DCNN is able to perform high-fidelity image restoration when training and testing are performed with the same types of cells. In practical applications, the Cell-DCNN-GALOF system would be a more efficient and higher functionalized tool if it was able to transfer its learning capability to reconstruct different types of cells that never appeared in the set of training data. To enable transfer-learning reconstruction with high fidelity, a training dataset with high diversity would certainly be beneficial. As a proof-of-concept experiment, we apply a training set with just three different types of images. Sample images are shown in
Fig. 6. Cell imaging transfer learning. (a)–(c) Sample cell images in the set of training data. The scale bar length in (a) is . There are three different types of cells in the set of training data: (a) an image of human red blood cells, (b) an image of frog blood cells, and (c) an image of polymer microspheres. (d)–(f) Test process using data from images of bird blood cells. (d1)–(d4) Raw images of bird blood cells transported through straight GALOF taken at 0 mm imaging depth and at room temperature. (e1)–(e4) Images reconstructed from (d1)–(d4); (f1)–(f4) the corresponding reference images of bird blood cells. (g) Training and validation accuracy improvement curves using MAE as the metric over 80 epochs. (h) Averaged test MAE of the bird blood cell images with the standard deviation as the error bar.
As the test data, we record 1000 image pairs from a totally different type of cell, namely bird blood cells. The raw images of the bird blood cells obtained after passing through straight GALOF are shown in
4 Discussion and Conclusion
The system performance of an FOIS is mainly determined by the imaging processing method and the physical properties of the optical fiber. Recently developed FOISs using MMFs and MCFs heavily rely on the TM method, which requires phase-shifting interferometry and adaptive optic devices, such as spatial light modulators (SLMs) or digital micromirror devices (DMDs).5,9,14,19 Although TM-based systems have demonstrated remarkable performance, several inherent shortcomings are associated with this approach.17,20,22 First, TM-based systems require phase-shifting interferometry for calibration, which results in complicated configurations, vulnerability to environmental noise, and incompatibility with incoherent broadband illuminations. Second, the slow refresh rate of the wavefront-shaping device (SLM or DMD) limits the calibration and imaging speed (
The Cell-DCNN-GALOF system overcomes many of the abovementioned drawbacks of current FOISs through the combination of DCNN and GALOF. Regarding the image processing technique, the DCNN does not impose restrictions on coherence, bandwidth, or polarization of the light. Training and test data are intensity images recorded by conventional CCD cameras without any wavefront shaping device or interferometric measurement process. On the other hand, the unique properties of the GALOF make high-quality imaging possible. Schirmacher et al.44 confirmed that the localization lengths of modes in transverse Anderson localizing fiber remain almost constant over a very broad spectral bandwidth of
Due to the combination of DCNN and GALOF, the imaging system performance makes a big leap forward. High-speed (
Although the Cell-DCNN-GALOF system has demonstrated superior system performances, optimizations on both GALOF structure and DCNN design can potentially enhance its performance further. The current setup uses a transmission mode, whereas reflection mode operation is more suitable for practical in vivo imaging. Reflection mode operation could be implemented using illumination coupled into the GALOF proximal end and guided either by parts of the disordered structure or by separate channels integrated into the GALOF’s nondisordered cladding.42,47 Even if the lensless imaging depth of the Cell-DCNN-GALOF system is able to satisfy lots of practical applications, extending the depth further is always desired. The imaging quality of the current system gradually reduces with increasing depth. This might be attributed to the fact that, under incoherent illumination, high-frequency features of the intensity objects are gradually lost with increasing depth, leading to a corresponding rise of the MAE. In future work, this problem might be relieved by optimizing the structural parameters of the GALOF. Perhaps the robustness of the system performance, for instance, the bending tolerance, can also be improved in next-generation Cell-DCNN-GALOF systems. Since this environmental stability is strongly related to the presence of transversally extended modes, the path to more robust performance might be further optimization of the random structure inside the GALOF with a maximized scattering in the transverse plane and, therefore, further enhanced Anderson localization. Maximizing the scattering should be achieved by adapting new materials with different refractive indices and optimizing the air–hole filling fraction and the air–hole size distribution. Enhanced Anderson localization creates transverse localization sites with smaller radii and fewer variations.31,42,48 Finally, more universal transfer-learning imaging can be achieved using training data with larger diversity. However, generating highly diverse biological training data for an FOIS remains a formidable practical challenge, and the computational power available to process large amounts of training data is often an additional bottleneck. We believe that these challenges can be addressed in next-generation FIOSs for biological objects with the help of further optimized DCNN architectures.
In conclusion, it is the combination of unique GALOF properties and tailored DCNN design that enables the remarkable capabilities of the presented Cell-DCNN-GALOF imaging system. Both components, GALOF design and DCNN architecture, still have room for improvements, and future research will consider both components and their interplay. We are very optimistic that the presented architecture can be the basis for future high-fidelity imaging systems that are minimally invasive and demonstrate robust performance in dynamic environments.
[1] F. Koenig, J. Knittel, H. Stepp. Diagnosing cancer
[3] B. A. Flusberg, et al.. Fiber-optic fluorescence imaging. Nat. Methods, 2005, 2(12): 941-950.
[13] J. M. Stone, et al.. Low index contrast imaging fibers. Opt. Lett., 2017, 42(8): 1484-1487.
[17]
[23] Y. Rivenson, et al.. Deep learning microscopy. Optica, 2017, 4(11): 1437-1443.
[34] N. Borhani, et al.. Learning to see through multimode fibers. Optica, 2018, 5(8): 960-966.
[38] G. Ruocco, et al.. Disorder-induced single-mode transmission. Nat. Commun., 2017, 8: 14571.
Article Outline
Jian Zhao, Yangyang Sun, Hongbo Zhu, Zheyuan Zhu, Jose E. Antonio-Lopez, Rodrigo Amezcua Correa, Shuo Pang, Axel Schulzgen. Deep-learning cell imaging through Anderson localizing optical fiber[J]. Advanced Photonics, 2019, 1(6): 066001.