A 25 micron-thin microscope for imaging upconverting nanoparticles with NIR-I and NIR-II illumination

Abstract

Rationale: Intraoperative visualization in small surgical cavities and hard-to-access areas are essential requirements for modern, minimally invasive surgeries and demand significant miniaturization. However, current optical imagers require multiple hard-to-miniaturize components including lenses, filters and optical fibers. These components restrict both the form-factor and maneuverability of these imagers, and imagers largely remain stand-alone devices with centimeter-scale dimensions.

Methods: We have engineered INSITE (Immunotargeted Nanoparticle Single-Chip Imaging Technology), which integrates the unique optical properties of lanthanide-based alloyed upconverting nanoparticles (aUCNPs) with the time-resolved imaging of a 25-micron thin CMOS-based (complementary metal oxide semiconductor) imager. We have synthesized core/shell aUCNPs of different compositions and imaged their visible emission with INSITE under either NIR-I and NIR-II photoexcitation. We characterized aUCNP imaging with INSITE across both varying aUCNP composition and 980 nm and 1550 nm excitation wavelengths. To demonstrate clinical experimental validity, we also conducted an intratumoral injection into LNCaP prostate tumors in a male nude mouse that was subsequently excised and imaged with INSITE.

Results: Under the low illumination fluences compatible with live animal imaging, we measure aUCNP radiative lifetimes of 600 μs - 1.3 ms, which provides strong signal for time-resolved INSITE imaging. Core/shell NaEr0.6Yb0.4F4 aUCNPs show the highest INSITE signal when illuminated at either 980 nm or 1550 nm, with signal from NIR-I excitation about an order of magnitude brighter than from NIR-II excitation. The 55 μm spatial resolution achievable with this approach is demonstrated through imaging of aUCNPs in PDMS (polydimethylsiloxane) micro-wells, showing resolution of micrometer-scale targets with single-pixel precision. INSITE imaging of intratumoral NaEr0.8Yb0.2F4 aUCNPs shows a signal-to-background ratio of 9, limited only by photodiode dark current and electronic noise.

Conclusion: This work demonstrates INSITE imaging of aUCNPs in tumors, achieving an imaging platform that is thinned to just a 25 μm-thin, planar form-factor, with both NIR-I and NIR-II excitation. Based on a highly paralleled array structure INSITE is scalable, enabling direct coupling with a wide array of surgical and robotic tools for seamless integration with tissue actuation, resection or ablation.

Introduction

Highly sensitive imaging in hard-to-access areas remains a persistent challenge for optical surgical navigation. This is increasingly prevalent in the modern era of minimally invasive cancer surgeries where visualizing microscopic disease over the entirety of a small, complex tumor bed is hindered by the form-factor of intraoperative optical imagers, and consequently microscopic residual disease (MRD) is common [1]. MRD significantly increases cancer recurrence [2, 3], necessitating substantial additional treatment, resulting in additional toxicity and cost. Despite these efforts, clinical outcomes of additional therapy often remain inferior to complete resection upfront [4]. MRD remains common despite the advent of new targeted molecular imaging agents [5] and intraoperative optical imagers [6-8].

Intraoperative imagers have made significant progress for precision guided surgeries [7, 8]. Leveraging the growing number of specific optical probes to label tumor cells in vivo, stand-alone widefield fluorescence microscopes are increasingly common for surgical guidance [6], and advances in engineering have decreased sizes to centimeter scales [9-16]. While intraoperative imagers have had significant impacts on surgical outcomes, current designs are limited in both miniaturization and adaptability. In particular, current fluorescent probes that target proteins or cells necessitate the use of rigid, bulky optical lenses to distinguish probe emission from excitation light [17]. This need for rigid optical components has been a barrier to miniaturization. While advances in microfabrication have achieved small lenses [18-20], fundamental physical limits and fabrication challenges hinder further miniaturization, and ultra-small lenses face challenges in producing high resolution images. Furthermore, the requirement for precise positioning of the tissue sample in the focal plane and imaging over a broad area hinder utilization of these micro-scale lenses intraoperatively. Optical lenses are not easily planarized - precluding an ultra-thin form-factor necessary for broad device integration. Fiber-optic based imagers address this problem by guiding light out of the tumor bed to a large standalone imager, but fundamental tradeoffs between imaging area and flexibility hinder achieving both high maneuverability and rapid imaging of the entire tumor cavity.

Chip-based imagers eliminate the need for optics in favor of contact imaging [21, 22] placing the imager in direct tissue contact to capture light before it diverges, microfabricated patterns to focus light [23], or computational approaches [24-26]. While these imagers are significantly more compact without the use of lenses, the high-performance optical filters required to image fluorophores with small Stokes shifts demands precision optical alignment [27, 28], substantially hindering further miniaturization for intraoperative imaging.

Chip-based tissue imagers built around optical probes that enable time-resolved imaging [29, 30] would obviate the need for both filters and lenses, and allow for exceptionally small and flexible microscopes. To date this approach has been limited for tissue imaging by both the nanosecond radiative decay times of organic optical labels, and the similarity of decay times with tissue autofluorescence, masking small signals with background. Imager sensitivity is fundamentally limited by scattering [31] and cellular autofluorescence [32, 33] , which is strongest with UV and visible excitation and cannot be removed with filtering [34, 35]. Optical labels with longer lifetimes and photoexcitation at longer wavelengths in the NIR-I (700 - 1000 nm) or NIR-II (typically, 1000 - 1700 nm) [36] might circumvent these problems.

As long-lifetime optical probes, lanthanide-based upconverting nanoparticles (UCNPs) are able to sum the energies of multiple NIR photons and emit at higher energies in the NIR or visible spectrum. UCNP luminescence efficiencies are up to 10 orders of magnitude higher than those of the best 2-photon fluorophores [37, 38] and show no overlap with cellular autofluorescence, no measurable photobleaching, even under prolonged single-particle excitation [39-42]. UCNPs make use of energy transfer upconversion between neighboring lanthanide ions, in which sensitizer ions sequentially transfer absorbed energy to luminescent emitter ions. Numerous studies have shown that the sensitizer/emitter pair of Yb³⁺/Er³⁺ doped into host matrix NaYF₄ nanocrystals is most efficient and emits both green and red light with either continuous wave NIR-I (980 nm) or NIR-II (1550 nm) lasers. Recent work has shown that alloyed upconverting nanoparticles (aUCNPs), in which host matrix metals are replaced entirely with lanthanides, are significantly brighter than their doped counterparts, particularly at low laser fluences compatible with living systems [37]. A key advantage of UCNPs is that they can be excited with low fluences of either NIR-I or NIR-II light, minimizing interactions with both cells and semiconductors, while emitting at the visible wavelengths appropriate for most common imaging detectors. Human maximum permitted exposures [43] are ~200 W/cm² for 5-ms pulses of either 980 nm or 1550 nm light (Table S1), suggesting NIR-excitable probes can be imaged during surgery with minimal phototoxicity.

In this study, we have integrated both custom integrated circuit imager design with nanoparticle engineering, and introduced INSITE, a 25 micron-thin microscopic imaging platform for imaging aUCNPs. We achieve this level of miniaturization by eliminating conventional fluorophores and their requisite optics in favor of a scalable, planar chip-based microscope, custom designed and fabricated in a 0.18-micron complementary metal oxide semiconductor (CMOS) process, in synergistic combination with long-lifetime NIR I and II-excited upconverting nanoparticles. To determine the optimal UCNPs composition to use with INSITE, we first investigate three different types of lanthanide-based aUCNPs and we subsequently present experimental results obtained using INSITE from an intratumorally-injected mouse prostate tumor specimen.

Materials and Methods

Imager design and fabrication

An ultra-thin imager array was designed as an application-specific integrated circuit (ASIC) imaging array consisting of 2,880 pixels [44]. The resulting INSITE chips were custom fabricated in a 0.18 μm CMOS process, and thinned down to 25 μm. Each pixel contains a silicon photodiode (44 μm [Image Omitted; See PDF] 44 μm), followed by a 4-transistor front-end amplifier which enables a current integration over a custom-made MOM (metal-oxide-metal) capacitor. A subsequent sample-hold block controls pixel timing.

Eliminating lenses and angle-selective gratings

To obtain spatial resolution with minimum form-factor, we used on-chip angle-selective gratings using designs similar to those previously reported [22, 44, 45]. Angle selective gratings are an array of microfabricated collimators patterned directly over each photodiode, effectively coupling each photodiode to the tissue directly opposite it, blocking obliquely incident background light.

Nanocrystal synthesis

Growth of aUCNP cores: aUCNPs were grown as previously described [37, 46] For, 8-nm NaEr_0.6Yb_0.4F₄ cores: YbCl₃ hexahydrate (64 mg, 0.16 mmol) and ErCl₃ (66 mg, 0.24 mm) were stirred in oleic acid (OA, 3.25 g, 10.4 mmol) and 4 mL of ODE and heated under vacuum for 1 h at 110 °C. The reaction was cooled to room temperature under N₂, followed by the addition of sodium oleate (381 mg, 1.25 mmol), NH₄F (74 mg, 2.0 mmol) and 3 mL of ODE. The reaction was then held under vacuum for 20 min, followed by 3 cycles of refilling with N₂ and purging. The reaction temperature was then increased to 315 °C. After 45 min, nanocrystal growth was halted by removing the heating mantle and cooling the flask to 75 °C with a stream of air, followed by the addition of 20 mL of EtOH and 20 mL of acetone. The UCNPs were precipitated twice with EtOH and redispersed in 10 mL of hexane with 0.2% (v/v) OA.

Growth of core/shell aUCNPs: Epitaxial 4-nm NaY_0.8Gd_0.2F₄ shells were overgrown using adaption of a layer-by-layer method [46]. A 0.10 M solution of 80:20 Y/Gd oleate (Y/Gd-OA) was prepared by heating YCl₃ (78 mg, 0.40 mmol) and GdCl₃ (26 mg, 0.10 mmol) to 110 °C in OA (2 mL) and ODE (3 mL) and stirred for 15 min under vacuum. The flask was filled with N₂ and heated to 160 °C for 30 min, followed by another 15 min at 110 °C under vacuum. In a separate flask, a 0.40 M NaTFA-OA precursor solution was prepared by dissolving sodium trifluoroacetate (163 mg, 1.20 mmol) in oleic acid (3 mL) and applying vacuum at room temperature for 20 min.

Purified aUCNP cores (27 μmol) in hexane were added to 4 mL of OA and 6 mL of ODE. The mixture was held under vacuum for 30 min at 70 °C to remove the hexane. The shell growth was performed under N₂ at 280 °C, with alternating injections of Y/Gd-OA and NaTFA-OA precursor performed at 15 min intervals. After the last injection, the reaction was maintained at 280 ° C for an additional 30 min to allow for complete shell growth, followed by rapid cooling. Nanoparticles were purified and stored as described for the aUCNP cores.

UCNP polymer encapsulation: Hydrophobic core/shell aUCNPs were dispersed in hexane with 0.2% (v/v) oleic acid to 5 μM. For aqueous dispersions [37, 46], 6 mg of poly(maleic anhydride-alt-1-octadecene) copolymer (MW 20-25k, Aldrich) was dissolved to 17 μM in 0.5 mL of acetone and 15 mL of CHCl₃. aUCNPs (0.5 nmol) in 100 μL of hexane were added with stirring, and the solvents were removed under a gentle stream of N₂ overnight. The aUCNP/polymer residue was resuspended in a solution of MeO-PEG₈-NH₂ (ThermoFischer, 10 μmol) in 10 mL of 100 mM NaHCO₃ buffer, pH 8.2, with 1% (v/v) DMSO. This suspension was sonicated for 60 min, heated in an 80 °C water bath for 60 min, slowly cooled to room temperature, and then sonicated for 30 min. Excess polymer was removed by spin dialysis (Amicon, 100 kDa MWCO), washing with 4 × 15 mL of 100 mM HEPES, pH 7.4. Retained aUCNPs were concentrated to 680 μL and filtered through a 0.2-μm filter into a sterile glass vial.

Nanocrystal characterization

X-ray diffraction: 1 mL of a stock solution of the nanoparticles in hexane was precipitated with addition of 2 mL of EtOH. The nanoparticle slurry was spotted onto a glass coverslip or silicon wafer multiple times until an opaque white film formed, and the sample was allowed to air dry completely. XRD patterns were obtained on a Bruker AXS D8 Discover GADDS X-ray diffractometer system with Co Kα radiation (λ= 1.78897 Å) from 2θ of 15 to 65 °.

Electron microscopy: UCNPs were precipitated, resuspended in hexane to 10 nM, and 7 μL was dropped onto ultra-thin carbon film/holey carbon grid, 400 mesh copper (Ted Pella). Images of the nanoparticles were obtained using a Zeiss Gemini Ultra-55 analytical scanning electron microscope. Dark-field images were collected in transmission (STEM) mode with 30 kV beam energy. HR-TEM images were acquired on a JEOL 2100-F 200 kV field-emission analytical transmission electron microscope.

Dynamic Light Scattering: Aqueous nanocrystal size was determined by dynamic light scattering measurements on a Malvern Zetasizer. Samples were prepared from aqueous stocks by dilution with ddH₂O to ~50 nM. Hydrodynamic diameters were determined by instrument software based on volume fittings.

Optical characterization: aUCNPs emission and lifetimes were characterized as functions of illumination intensity, illumination pulse duration, each with either 980 or 1550 nm excitation. Vials of hydrophobic aUNCPs dispersions in hexane (400 μL of the 0.68 μM) were placed above the imager array and excited with time-gated collimated lasers. The beam was positioned 2 mm above the surface of the imager.

INSITE samples were excited with a 980-nm wavelength-stabilized, single-mode, fiber-coupled laser diode (Qphotonics QFBGLD-980-500) followed by an adjustable collimator (Thorlabs ZC618FC-B) set to a beam diameter of 1.27 mm; or a 1550-nm single-mode, fiber-coupled laser diode (Qphotonics QFLD-1550-150S) collimated by an aspheric collimator (Thorlabs CFS2-1550-APC) with a beam diameter of approximately 0.3 mm. Both lasers were driven by a temperature-controlled mount driver (Arroyo Instruments 6310 ComboSource). Unless otherwise indicated, illumination parameters and laser settings used for INSITE for 980 nm and 1550 nm are given in Table S1.

Radiative lifetimes ([Image Omitted; See PDF]) were modeled as a single exponential decay and were calculated by extracting decay profiles with a fixed moving integrating window (T_int). Assuming the dark current intensity (i_d) is constant over time, we derive the integrated pixel value I_A(t) from the current density i(t):

[Image Omitted; See PDF]

[Image Omitted; See PDF]

where τ and I_D are the emission decay lifetime and dark current level in the pixel, respectively [47]. Dark current level was subtracted from waveforms.

Effects of excitation (T_exc) pulse duration on emission signal intensity: To extract the excitation duration dependency, the aUCNPs were excited for increasing durations of time (T_exc) and the emission intensity was measured. This duration represents how long the nanoparticles are illuminated with the excitation light source before the start of the time-resolved imaging sequence.

Fabrication of PDMS micro-wells

We fabricated 500-micron cubic wells for aUCNP coating purposes by micro-patterning wells into a PDMS substrate. We subsequently coated the internal surface of the PDMS micro-well with 10 μL of 0.68 μM 16-nm NaEr_0.6Yb_0.4F₄ core/shell aUCNPs in hexane, giving a final surface concentration of 5 pM/mm² of adsorbed UCNP. This aUCNP-coated structure was imaged with INSITE using a 1550 nm laser at 60 W/cm².

Imaging aUCNPs in tissue with INSITE

Animal experiments were conducted according to protocols approved by the UCSF Animal Care and Use Committee. Tumor-bearing mice were generated by subcutaneous implantation of prostate cancer (LNCaP) cells bilaterally over the flank of a nude male mouse (5-week-old, Tarconic Farms). Tumors were allowed to grow to 1 cm in diameter. Mice were anesthetized and single tumors were injected with injected with 25 μL of 250 nM polymer-encapsulated 26-nm (16-nm cores with 5-nm shells) NaEr_0.8Yb_0.2F₄ core/shell aUCNP dispersions on the left side tumor of the mouse (ventral left), and the right side tumor was used as a reference and negative control. Mice were imaged with an IVIS Spectrum (In Vivo Imaging System, PerkinElmer) equipped with a 980-nm continuous wave laser (Qphotonics) and 780-nm short pass filter (Chroma), to reject 980-nm interference on the IVIS camera. Emission was collected from 650-670 nm in the ⁴F_9/2 Er³⁺ band using 2.5-s integration times.

Tumors with aUCNPs were excised and imaged by INSITE with a 1550-nm pulsed laser (Qphotonics) scanned across the sample in 300-μm increments. Excised tumors were also imaged under a widefield microscope (Leica DMIRB) customized to image aUCNPs using a 980 nm laser (Qphotonics) at 1 W/cm².

Results and Discussion

Optimization of UNCPs for INSITE Imaging

To determine the optimal composition of aUCNPs for use with INSITE (Figure 1B), we synthesized a series of core/shell aUCNPs with varying Yb³⁺ and Er³⁺ content to measure emission using either 980 nm or 1550 nm photoexcitation. aUCNP cores (8 nm) were synthesized with 20/80, 40/60, or 80/20 Yb³⁺/Er³⁺ ratios and overgrown with inert 4-nm shells [37]. These hydrophobic nanocrystals were characterized by transmission electron microscopy (TEM) to measure size, dispersity, and crystallinity (Figures S1-2). With these nanocrystals in the experimental setup (Figure 2), we measured the initial emission intensity, [Image Omitted; See PDF], luminescence lifetime, [Image Omitted; See PDF], (Figure 3) and total integrated emission count, [Image Omitted; See PDF], at a given power density (8 W/cm² at either 980 and 1550 nm) (Figure 4). Measured decays for varying aUCNP compositions range from 600 μs to 1.3 ms at either 980 nm or 1550 nm excitation. These values are longer than measured [37] or calculated, but consistent with UCNP power-dependence given the lower excitation powers used here.

While most Yb³⁺/Er³⁺ upconversion is nominally a 2-photon process with following 980-nm excitation of the Yb^{3+ 2}F_5/2 manifold, 1550 nm excitation of the Er^{3+ 4}I_13/2 manifold leads to upconversion via a nominal 3-photon process (Figure 1a). For all aUCNP compositions, NIR-I excitation produces a stronger signal than with NIR-II, consistent with significantly higher efficiency of 2-photon versus 3-photon upconversion processes [48]. At both excitation wavelengths, NaEr_0.6Yb_0.4F₄ aUCNPs are the brightest composition (Figure 4) as measured by INSITE, although single nanoparticle studies at low 980-nm fluences have shown little difference in absolute brightness between aUCNP compositions [37]. Differences in brightness here may be due to subtle variations in radiative lifetime, since the INSITE 100-μs time gating (T_delay) will cause greater losses in aUCNPs with faster upconversion. Kinetic modeling has shown that Yb³⁺/Er³⁺ lifetimes are complex and deeply power-dependent [42, 48] but alloyed compositions do show kinetic differences in both green (⁴S_3/2, ²H_11/2) and red (⁴F_9/2) Er³⁺ emission [48].

Figure 1 INSITE imaging of upconverting nanoparticles. (A) Multiphoton energy absorption, transfer, and emission in Yb3+/Er3+-based aUCNPs following either NIR-I (980 nm) or NIR-II (1550 nm) excitation. (B) Cartoon of INSITE directly integrated onto surfaces, such surgical glove. (C) Diagram of the time-resolved image acquisition scheme. Texc, pulse excitation time; Tint, emission signal integration time; Tdelay, time gating delay. (D) Angle-selective gratings used to achieve lensless image acquisition and block obliquely incident background light.

Figure 2 INSITE configuration for imaging of aUCNP dispersions. Laser beam width is 300 [Mu] m.

Figure 3 Emission decays of aUCNPs (Tint = 1 ms, Texc = 5 ms) as measured by pixel output at (A) 8 W/cm2 of 980 nm excitation, or (B) 60 W/cm2 of 1550 nm excitation. Tint is integation time; Texc is duration of exciation light pulse.

Figure 4 Integrated visible emission as a function of aUCNP composition at either 980 and 1550 nm excitation, both with 8 W/cm2 excitation power density. Hexane blank is without aUCNPs.

Long decay lifetimes enable time-resolved imaging in modern CMOS technologies, and alleviate the need for optical filters entirely. While time resolved imaging has been demonstrated with organic and protein fluorophores, their nanosecond radiative lifetimes [17] make large, dense array based imaging impossible, as arrayed CMOS sensors cannot readily detect on timescales shorter than tens of microseconds [49, 50]. Although single photon avalanche diodes that require specialized fabrication processes have been demonstrated to operate at these timescales [51], we require a massively parallel array-based approach for larger spatial coverage, high fill factor, and adequate spatial resolution necessary for efficient chip-based imaging. Consequently, a chip-based imager using time resolved imaging requires optical probes with microsecond lifetimes, such as upconverting nanoparticles [37, 47, 48]. The design of UCNPs with a core-shell prevents rapid quenching, as tested here, significantly elongating the decay lifetime. The 0.6-1.3 ms decay lifetimes open the door to time-resolved imaging in an array-based CMOS imager.

The chip-based time-resolved imaging method takes advantage of these uniquely long emission lifetimes the aUCNPs studied here, and alleviates the need for high performance frequency-selective (color) filters by separating the emission and excitation signals in the time domain rather than in frequency domain, a strategy that can be implemented in modern high-speed integrated circuit design. In a chip-based imager, we implement this by briefly pulsing the excitation light (T_exc = 5 ms duration) while the imaging pixels are not integrating. After the excitation light is turned off, the pixels are turned on, and integrate the emission signal from the aUCNPs for 1 ms (Figure 1C). Since there is no background excitation light at this time, the need for an optical filter is eliminated. The illumination and signal acquisition scheme (Figure 1C), where the excitation light source is pulsed for T_exc, and subsequently turned off, after which the emission signal from the aUCNPs is acquired and integrated by the imager. Any interference and unwanted signal caused by the excitation light can be rejected by delaying (T_delay) the integration window start point.

aUCNP brightness as a function of NIR-I and NIR-II excitation intensity and pulse duration

In order to minimize overall imaging time for rapid imaging which consists of illumination, integration and chip readout, and duration of optical exposure for tissue, we investigated the optimal time for the illumination pulse. We find increases in emission intensity for all three aUCNP compositions with increasing T_exc up to a threshold of 5 ms, after which emission intensity plateaus, and longer integration pulses cause a measurable increases in detector noise (Figure S3). Pulsed illumination is safer than continuous wave (CW) illumination, and human maximum permitted exposures are ~3 orders of magnitude higher for pulsed versus CW at both NIR-I and NIR-II wavelengths [43]. These 5-ms pulses then permit significantly higher excitation fluences, with resulting increased sensitivity and signal-to-background ratios.

To quantify the relationship between excitation light intensity and emission intensity, all 3 aUCNP compositions were excited at increasing illumination intensities at both 980 nm and 1550 nm, and emission intensity was measured by INSITE (Figure S4). UCNP excitation is nominally a 2- and 3-photon process, but the relationship between illumination power density and emission intensity is complex and varies with both excitation and detection schemes [48]. While varying illumination intensity over orders of magnitude below saturation elicits non-linear responses [37, 39], the linear response we observed here may be due to the small range of illumination intensities, time-gating, and pulse sequence.

Optical probes excited in the NIR-I and NIR-II spectrum are ideal for both in vivo imaging and CMOS-based tissue imaging chips. NIR-I and NIR-II illumination enables deeper tissue penetration and lower scatter than illumination in the visible spectrum, common to most optical labels. Furthermore, aUCNP excitation in the NIR-I and NIR-II regions is particularly attractive for CMOS-based imagers, as it minimally interacts with silicon. At these wavelengths, the thin imager itself is effectively transparent and minimal background and noise is generated in the imager by the illumination light.

Despite the stronger aUCNP emission signal with NIR-I excitation, NIR-II light is more weakly absorbed by silicon and may be advantageous for chip-based microscopes. The silicon bandgap energy corresponds to ~1100 nm [52] and at wavelengths above that, photons pass through with relatively little interaction. At shorter wavelengths, particularly in the visible, photon interaction with the chip can introduce substantial background signal. Excitation with 980 nm light (just below the silicon bandgap) produces a small background signal with INSITE (Figure 4 and Figure S4, hexane), while for 1550 nm illumination (substantially above the silicon bandgap), such background is absent. Because background substantially affects signal to noise ratios, using higher excitation powers at 1550 nm to compensate for lower aUCNP emissions may ultimately lead to better image quality.

Upconversion, along with NIR illumination and long decay times, enables high contrast and sensitivity imaging by eliminating autofluorescence background. Upconversion places the emission photos in the high responsivity (sensitivity) range of silicon-based detectors, enabling imaging of these nanoparticles [37]. In comparison, light from a conventional fluorophore excited in the NIR I or NIR II spectrum, will emit a lower energy (longer wavelength) photon, which would pass through silicon undetected.

INSITE imager design

In order to determine whether an ASIC can image optical labels without the use of conventional focusing lenses and optical filters, we designed and fabricated an imaging array capable of time resolved imaging (Figure 1). Absent optics, the imaging chip can be easily thinned to just 25 microns and placed directly on tissue, increasing sensitivity through proximity - capturing light from optical labels before it diverges. The key to this platform is the transformation of molecular imaging from the color (frequency) domain to the time domain enabled by aUCNPs [37]. We accomplish this using synergistic design of modern integrated-circuits and upconverting nanoparticles. A key advantage of using CMOS-based imaging platform is the ability to integrate in-pixel electronics enabling signal processing directly on-chip, eliminating the need for optical lenses. This imager addresses both elimination of lenses and filters simultaneously, making it possible to obtain optical images with a much smaller form-factor.

INSITE spatial resolution

To determine the imaging quality achievable with a 25-micron thin microscope, we used a 1550 nm excitation source to excite the aUCNP-coated microstructure and acquired the image using time-resolved imaging. The custom-fabricated PDMS micro-well and the CMOS imager demonstrate that INSITE is able to resolve the spatial features of the micro-well (Figure 5A and 5B) with nearly single-pixel sharpness (55 μm), translating into a spatial resolution performance sufficient for detecting microscopic residual disease. A cross-section of the acquired signals (Figure 5C), shows three different regions of the image. Aside from the micro-well and the background, the intermediate zone represents the PDMS surrounding the micro-well. Due to the porosity of the PDMS, small amounts of aUCNPs diffuse into the surrounding area, generating a small signal in this area. For a concentration of 10¹² aUCNPs per mm², the measured average signal-to-background ratio is 6.5. The background is dominated by the dark pixel current, which can ultimately be subtracted out.

Figure 5 (A) Photograph of CMOS contact imager and PDMS micro-well holding aUCNPs, fabricated for this study. (B) Time-resolved image of the aUCNP-coated surface of the micro-well. Normalized emission intensity shown as in grayscale legend. Section (I) is background, (II) is outer PDMS with aUNCPs diffused into PDMS, and (III) is microwell with aUCNP sample. (C) Cross-section emission profile for the three regions in (B).

INSITE achieves this spatial resolution without the use of conventional lenses through both proximity to the tissue sample and direct integration of on-chip microfabricated collimators, and is limited only by the pixel size. INSITE uses angle-selective gratings (ASG) to improve spatial resolution with chip-based imaging [22, 45]. ASG are arrays of microcollimators fabricated directly on each pixel using only the inherent metal interconnect layers common to all CMOS process - obviating the need for any postprocessing and not adding any thickness to the imager itself. The versatility of CMOS fabrication technology has led to the on-chip integration of a variety of optical components such as wavelength-selective optical filters [21, 53] that could be tuned to be compatible with quantum dot applications [54], or stacked diffraction gratings for lensless 3D imaging [55] to reject angled incoming light and decrease blur in the image. Other lensless imaging platforms have also been reported in [24, 26, 56] that leverage computational techniques. As demonstrated here, the elimination of optical filters and focusing optics enables placement of the custom designed INSITE imaging chip directly against the sample itself, capturing light before it diverges, achieving both spatial resolution and increased sensitivity without optics.

INSITE tumor imaging

To determine the applicability of INSITE to tissue imaging, we injected a prostate tumor with aqueous 250 nM polymer-encapsulated 26-nm NaEr_0.8Yb_0.2F₄ aUNCPs. Hydrophobic, as-synthesized nanocrystals were transferred to water by encapsulation within amphiphilic polymers terminated with short PEG chains [57, 58]. Previous work with these and similar amphiphilic polymers [39, 46] has shown that nanoparticle brightness is fully preserved by retention of the oleic acid surfactant layer. Polymer encapsulation increases nanoparticle diameters by 3-5 nm (Figure S2 and [59]) and present multiple surface carboxylates that, along with the PEG chains, may minimize non-specific endocytosis [57, 58]. aUCNP-injected mice were imaged a custom-modified IVIS imager using NIR-I illumination, showing colocalization of the tumor and aUCNPs (Figure 6A). Images of a tumor on the contralateral side of the mouse without aUCNP injection shows no measurable visible emission (background dark current only). To ensure aUCNPs were being imaged, the spectrum of the acquired emission signal was measured (Figure 6B), displaying the characteristic emission spectrum of the nanoparticles, with the two major visible emission bands of the aUCNPs at 545 nm and 655 nm. Using the aUCNP emission band at 650-670 nm, the signal-to-background ratio is 15.

To determine if the aUCNPs injected into the tumor can be visualized with INSITE alone, we next excised the injected tumor and imaged with the INSITE chip imager. Figure 7 shows the photograph of the excised tumor sample on the 25-micron imaging chip. For reference, we image the excised tumor on a microscope (Figure 8A), and the image acquired with INSITE reveals a distinct area of aUCNPs in the excised tissue (Figure 8B) with a signal to background ratio of 9, closely matching the performance of the IVIS imager, which incorporates high performance optical filters and a cooled CCD camera. Results from these experiments demonstrate that the ultra-thin time-resolved CMOS imager, custom designed to image engineered aUCNPs, is capable of imaging within tissue, with no background autofluorescence and with little to no additional interference from the excitation light. More extensive work in toxicology and biodistribution with immunotargeted aUCNPs will be needed to assess the suitability for these nanoparticles for tumor targeting in mice or human tissue.

Figure 6 Live mouse images of intratumorally-injected NaEr0.8Yb0.2F4 aUCNPs with 8 W/cm2 980 nm excitation. (A) Images of aUCNPs-injected into mouse prostate tumor (left) and non-injected side (right). Emission intensity as in colored legend. (B) Measured emission spectrum of the injected and non-injected sides, showing tumor-specific Er3+ emission bands at 545 and 655 nm.

Figure 7 Photograph of the tumor injected with aUCNPs (left side of mouse, from Figure 6) is excised and placed directly on INSITE for imaging. The area within the excised tissue where the aUCNPs are located (aUCNP spot) is circled. The remainder of the tissue does not contain aUCNPs. The path of the illumination laser is drawn in red. *This is for illustrative purposes only, and is not an image of the actual laser beam, which is not visible.

Figure 8 (A) Microscope image of excised tumor injected with 25 [Mu] L of a 250 nM aqueous dispersion of aUCNPs, excited with 1 W/cm2 980 nm, showing distinct localization of aUCNPs. (B) Images of NIR-II laser scanning of tumor, from top to bottom in increments of 300 [Mu] m (numbered 1-8). At each position, an image of the tumor sample is acquired with INSITE using a 5 millisecond-pulsed 60 W/cm2 1550-nm laser for illumination.

We have designed a 25-micron thin microscope, INSITE, and optimized it for imaging aUCNPs with NIR-I or NIR-II excitation. UCNPs have been successfully used for in vivo animal imaging [60-64], but to date the imagers remain too bulky and illumination power requirements are too high to be used in minimally invasive surgeries. Using complementary design of imager and optical label, INSITE synergistically integrates the advantages of both modern integrated circuit design with nanoparticle engineering. Leveraging the engineered long luminescent lifetimes and upconversion of aUCNPs we demonstrate the optimal aUCNP composition for chip-based time-resolved imaging. We custom design our chip-based imager around the excitation and emission wavelengths of aUCNPs and their decay time constants. This eliminates optical filters and the requisite focusing optics without sacrificing background rejection. Pulsed illumination allows a 200x-1000x increase in instantaneous optical illumination power, while still respecting ANSI safety limits. Absent optics, the INSITE platform is thinned to just 25 microns at which point it becomes flexible, allowing integration on virtually any planar or curved surgical instrument without disruption of form-factor. On-chip integration of both optical sensing and signal processing enables conversion of an optical signal to an electronic one (and amplification) at the point of imaging - enabling only thin flexible wires to transmit power and data from the chip, achieving a level of flexibly and maneuverability not achievable with current approaches. The scalable design, inherent to array-based integrated circuits allows further customization to any device or tumor bed size, while the parallel operation of the imaging array ensures imaging time remains constant, regardless of size. This maintains the rapid imaging needed for seamless intraoperative evaluation of large surface areas. Future form-factors for in vivo imaging may fully integrate illumination with optical detection, such as by surrounding the imaging chip with laser diodes, or leveraging the transparency of silicon above 1100 nm to illuminate tissue via laser diodes affixed to the chip back-side. Conjugation of aUCNPs to molecules targeted to tumor cells can further enhance tumor specificity of these imagers, and INSITE has the potential to significantly reduce the rate of positive margins in cancer surgeries, significantly impacting patient outcomes across the cancer spectrum.

Conclusion

This work provides a new strategy to bring NIR-I and NIR-II excitation-based imaging, with high sensitivity, into intraoperative imaging by introducing a time-resolved contact CMOS imaging array that no longer requires optics to resolve the image and can easily achieve a surgically-compatible form-factor. The integration of probe design, in terms of long-lifetime aUCNPs, with microscope engineering represents an initial foray into co-design of nanoparticles and imagers, and can be further optimized through iterative rounds of optimization for each element. The scalability inherent to CMOS technology will allow fabrication of larger arrayed image sensor to achieve greater spatial coverage, while preserving spatial resolution and imaging speed, than current imaging platforms.

Abbreviation

INSITE: immunotargeted nanoparticle single-chip imaging technology; UV: ultraviolet; UCNP: upconverting nanoparticle; aUCNP: alloyed upconverting nanoparticle; PDMS: polydimethylsiloxane; IR: infrared; NIR: near infrared; CMOS: complementary metal-oxide-semiconductor; LNCaP: lymph node carcinoma of the prostate; CW: continuous wave; OA: oleic acid; MOM: metal-oxide-metal; ANSI: American national standards institute; MRD: microscopic residual disease; ASIC: application-specific integrated circuit; ODE: 1-octadecene; ASG: angle-selective gratings.

Acknowledgements

MA and HN were supported by NIH R21EB027238 and the UCSF Academic Senate Committee on Research. Work at the Molecular Foundry was supported by the Director, Office of Science, Office of Basic Energy Sciences, Division of Materials Sciences and Engineering, of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. TSMC provided chip manufacturing through the University Shuttles Program and UC Berkeley.

Competing Interests

The authors have declared that no competing interest exists.

References

1. Orosco RK, Tapia VJ, Califano JA, Clary B, Cohen EEW, Kane C. et al. Positive surgical margins in the 10 most common solid cancers. Sci Rep. 2018;8:5686

2. Karakiewicz PI, Eastham JA, Graefen M, Cagiannos I, Stricker PD, Klein E. et al. Prognostic impact of positive surgical margins in surgically treated prostate cancer: multi-institutional assessment of 5831 patients. Urology. 2005;66:1245-50

3. Moran MS, Schnitt SJ, Giuliano AE, Harris JR, Khan SA, Horton J. et al. Society of Surgical Oncology-American Society for Radiation Oncology consensus guideline on margins for breast-conserving surgery with whole-breast irradiation in stages I and II invasive breast cancer. J Clin Oncol. 2014;32:1507-15

4. Tendulkar RD, Agrawal S, Gao T, Efstathiou JA, Pisansky TM, Michalski JM. et al. Contemporary Update of a Multi-Institutional Predictive Nomogram for Salvage Radiotherapy After Radical Prostatectomy. J Clin Oncol. 2016;34:3648-54

5. Zhang RR, Schroeder AB, Grudzinski JJ, Rosenthal EL, Warram JM, Pinchuk AN. et al. Beyond the margins: real-time detection of cancer using targeted fluorophores. Nat Rev Clin Oncol. 2017;14:347-64

6. Troyan SL, Kianzad V, Gibbs-Strauss SL, Gioux S, Matsui A, Oketokoun R. et al. The FLARE intraoperative near-infrared fluorescence imaging system: a first-in-human clinical trial in breast cancer sentinel lymph node mapping. Ann Surg Oncol. 2009;16:2943-52

7. Schols RM, Connell NJ, Stassen LPS. Near-infrared fluorescence imaging for real-time intraoperative anatomical guidance in minimally invasive surgery: a systematic review of the literature. World J Surg. 2015;39:1069-79

8. de Boer E, Harlaar NJ, Taruttis A, Nagengast WB, Rosenthal EL, Ntziachristos V. et al. Optical innovations in surgery. Br J Surg. 2015;102:e56-72

9. Gambin Y, Legrand O, Quake SR. Microfabricated rubber microscope using soft solid immersion lenses. Applied Physics Letters. 2006;88:174102

10. Flusberg BA, Nimmerjahn A, Cocker ED, Mukamel EA, Barretto RPJ, Ko TH. et al. High-speed, miniaturized fluorescence microscopy in freely moving mice. Nat Methods. 2008;5:935-8

11. Ghosh KK, Burns LD, Cocker ED, Nimmerjahn A, Ziv Y, Gamal AE. et al. Miniaturized integration of a fluorescence microscope. Nat Methods. 2011;8:871-8

12. Breslauer DN, Maamari RN, Switz NA, Lam WA, Fletcher DA. Mobile phone based clinical microscopy for global health applications. PLoS ONE. 2009;4:e6320

13. Wei Q, Qi H, Luo W, Tseng D, Ki SJ, Wan Z. et al. Fluorescent imaging of single nanoparticles and viruses on a smart phone. ACS Nano. 2013;7:9147-55

14. Senarathna J, Yu H, Deng C, Zou AL, Issa JB, Hadjiabadi DH. et al. A miniature multi-contrast microscope for functional imaging in freely behaving animals. Nat Commun. 2019;10:99

15. Cai DJ, Aharoni D, Shuman T, Shobe J, Biane J, Song W. et al. A shared neural ensemble links distinct contextual memories encoded close in time. Nature. 2016;534:115-8

16. Aharoni D, Khakh BS, Silva AJ, Golshani P. All the light that we can see: a new era in miniaturized microscopy. Nat Methods. 2019;16:11-3

17. Berezin MY, Achilefu S. Fluorescence lifetime measurements and biological imaging. Chem Rev. 2010;110:2641-84

18. Fletcher DA, Crozier KB, Quate CF, Kino GS, Goodson KE, Simanovskii D. et al. Near-field infrared imaging with a microfabricated solid immersion lens. Applied Physics Letters. 2000;77:2109-11

19. Brady DJ, Hagen N. Multiscale lens design. Optics Express. 2009;17:10659-74

20. Wu D, Wang J-N, Niu L-G, Zhang XL, Wu SZ, Chen Q-D. et al. Bioinspired Fabrication of High-Quality 3D Artificial Compound Eyes by Voxel-Modulation Femtosecond Laser Writing for Distortion-Free Wide-Field-of-View Imaging. Advanced Optical Materials. 2014;2:751-8

21. Hong L, Li H, Yang H, Sengupta K. Nano-plasmonics and electronics co-integration in CMOS enabling a pill-sized multiplexed fluorescence microarray system. Biomed Opt Express. 2018;9:5735-58

22. Papageorgiou EP, Zhang H, Giverts S, Park C, Boser BE, Anwar M. Real-time cancer detection with an integrated lensless fluorescence contact imager. Biomed Opt Express. 2018;9:3607-23

23. Khorasaninejad M, Chen WT, Devlin RC, Oh J, Zhu AY, Capasso F. Metalenses at visible wavelengths: Diffraction-limited focusing and subwavelength resolution imaging. Science. 2016;352:1190-4

24. Antipa N, Kuo G, Heckel R, Mildenhall B, Bostan E, Ng R. et al. DiffuserCam: lensless single-exposure 3D imaging. Optica. 2018;5:1

25. Adams JK, Boominathan V, Avants BW, Vercosa DG, Ye F, Baraniuk RG. et al. Single-frame 3D fluorescence microscopy with ultraminiature lensless FlatScope. Sci Adv. 2017;3:e1701548

26. Greenbaum A, Luo W, Su T-W, Göröcs Z, Xue L, Isikman SO. et al. Imaging without lenses: achievements and remaining challenges of wide-field on-chip microscopy. Nat Methods. 2012;9:889-95

27. Dandin M, Abshire P, Smela E. Optical filtering technologies for integrated fluorescence sensors. Lab Chip. 2007;7:955-77

28. Lichtman JW, Conchello J-A. Fluorescence microscopy. Nat Methods. 2005;2:910-9

29. Huang T-cD, Sorgenfrei S, Gong P, Levicky R, Shepard KL. A 0.18-m CMOS array sensor for integrated time-resolved fluorescence detection. Solid-State Circuits, IEEE Journal of. 2009;44:1644-54

30. Cubeddu R, Comelli D, D'Andrea C, Taroni P, Valentini G. Time-resolved fluorescence imaging in biology and medicine. J Phys D Appl Phys. 2002;35:R61-R76

31. Jacques SL. Optical properties of biological tissues: a review. Phys Med Biol. 2013;58:R37-61

32. Bornhop DJ, Contag CH, Licha K, Murphy CJ. Advance in contrast agents, reporters, and detection. J Biomed Opt. 2001;6:106-10

33. Yuanlong Y, Yanming Y, Fuming L, Yufen L, Paozhong M. Characteristic autofluorescence for cancer diagnosis and its origin. Lasers Surg Med. 1987;7:528-32

34. Croce AC, Bottiroli G. Autofluorescence spectroscopy and imaging: a tool for biomedical research and diagnosis. Eur J Histochem. 2014;58:2461

35. Frangioni JV. In vivo near-infrared fluorescence imaging. Curr Opin Chem Biol. 2003;7:626-34

36. Zhao J, Zhong D, Zhou S. NIR-I-to-NIR-II fluorescent nanomaterials for biomedical imaging and cancer therapy. J Mater Chem B. 2018;6:349-65

37. Tian B, Fernandez-Bravo A, Najafiaghdam H, Torquato NA, Altoe MVP, Teitelboim A. et al. Low irradiance multiphoton imaging with alloyed lanthanide nanocrystals. Nat Commun. 2018;9:3082

38. Garfield DJ, Borys NJ, Hamed SM, Torquato NA, Tajon CA, Tian B. et al. Enrichment of molecular antenna triplets amplifies upconverting nanoparticle emission. Nat Photonics. 2018;12:402-7

39. Wu S, Han G, Milliron DJ, Aloni S, Altoe V, Talapin DV. et al. Non-blinking and photostable upconverted luminescence from single lanthanide-doped nanocrystals. Proc Natl Acad Sci USA. 2009;106:10917-21

40. Nam SH, Bae YM, Park YI, Kim JH, Kim HM, Choi JS. et al. Long-Term Real-Time Tracking of Lanthanide Ion Doped Upconverting Nanoparticles in Living Cells. Angewandte Chemie-International Edition. 2011;50:6093-7

41. Ostrowski AD, Chan EM, Gargas DJ, Katz EM, Han G, Schuck PJ. et al. Controlled synthesis and single-particle imaging of bright, sub-10 nm lanthanide-doped upconverting nanocrystals. ACS Nano. 2012;6:2686-92

42. Gargas DJ, Chan EM, Ostrowski AD, Aloni S, Altoe MVP, Barnard ES. et al. Engineering bright sub-10-nm upconverting nanocrystals for single-molecule imaging. Nat Nanotechnol. 2014;9:300-5

43. American National Standards I. American national standard for the safe use of lasers, ANSI Z136.1-2014. New York: National Standards Institute. 2014

44. Papageorgiou E, Boser B, Anwar M. Chip-Scale Fluorescence Imager for In Vivo Microscopic Cancer Detection. Symposia V, editor. 2017 Symposium on VLSI Technology and Circuits: IEEE. 2017

45. Papageorgiou EP, Boser BE, Anwar M. An angle-selective CMOS imager with on-chip micro-collimators for blur reduction in near-field cell imaging. 2016 IEEE 29th International Conference on Micro Electro Mechanical Systems (MEMS): IEEE. 2016:2016 337-40

46. Tajon CA, Yang H, Tian B, Tian Y, Ercius P, Schuck PJ. et al. Photostable and efficient upconverting nanocrystal-based chemical sensors. Opt Mater (Amst). 2018;84:345-53

47. Najafiaghdam H, Papageorgiou EP, Torquato NA, Tajon CA, Zhang H, Park C. et al. A Molecular Imaging" Skin" A Time-resolving Intraoperative Imager for Microscopic Residual Cancer Detection Using Enhanced Upconverting Nanoparticles. 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC): IEEE. 2018:2018 1-4

48. Teitelboim A, Tian B, Garfield DJ, Fernandez-Bravo A, Gotlin AC, Schuck PJ. et al. Energy Transfer Networks within Upconverting Nanoparticles Are Complex Systems with Collective, Robust, and History-Dependent Dynamics. J Phys Chem C. 2019;123:2678-89

49. Dutton NAW, Gyongy I, Parmesan L, Gnecchi S, Calder N, Rae BR. et al. A SPAD-Based QVGA Image Sensor for Single-Photon Counting and Quanta Imaging. IEEE Trans Electron Devices. 2016;63:189-96

50. Dutton NA, Parmesan L, Holmes AJ, Grant LA, Henderson RK. 320× 240 oversampled digital single photon counting image sensor. 2014 Symposium on VLSI Circuits Digest of Technical Papers. 2014:1

51. Mora AD, Tosi A, Zappa F, Cova S, Contini D, Pifferi A. et al. Fast-gated single-photon avalanche diode for wide dynamic range near infrared spectroscopy. IEEE Journal of Selected Topics in Quantum Electronics. 2009;16:1023-30

52. Green MA, Zhao J, Wang A, Reece PJ, Gal M. Efficient silicon light-emitting diodes. Nature. 2001;412:805-8

53. Hong L, Li H, Yang H, Sengupta K. Fully Integrated Fluorescence Biosensors On-Chip Employing Multi-Functional Nanoplasmonic Optical Structures in CMOS. IEEE J Solid-State Circuits. 2017;52:1-19

54. Anikeeva PO, Halpert JE, Bawendi MG, Bulović V. Quantum dot light-emitting devices with electroluminescence tunable over the entire visible spectrum. Nano Lett. 2009;9:2532-6

55. Wang A, Gill P, Molnar A. Angle sensitive pixels in CMOS for lensless 3D imaging. 2009 IEEE Custom Integrated Circuits Conference (CICC): IEEE. 2009:2009 371-4

56. Mudanyali O, Tseng D, Oh C, Isikman SO, Sencan I, Bishara W. et al. Compact, light-weight and cost-effective microscope based on lensless incoherent holography for telemedicine applications. Lab Chip. 2010;10:1417-28

57. Dante S, Petrelli A, Petrini EM, Marotta R, Maccione A, Alabastri A. et al. Selective Targeting of Neurons with Inorganic Nanoparticles: Revealing the Crucial Role of Nanoparticle Surface Charge. ACS Nano. 2017;11:6630-40

58. Mann VR, Powers AS, Tilley DC, Sack JT, Cohen BE. Azide-Alkyne Click Conjugation on Quantum Dots by Selective Copper Coordination. ACS Nano. 2018;12:4469-77

59. Wichner SM, Mann VR, Powers AS, Segal MA, Mir M, Bandaria JN. et al. Covalent Protein Labeling and Improved Single-Molecule Optical Properties of Aqueous CdSe/CdS Quantum Dots. ACS Nano. 2017;11:6773-81

60. Min Y, Li J, Liu F, Padmanabhan P, Yeow EKL, Xing B. Recent Advance of Biological Molecular Imaging Based on Lanthanide-Doped Upconversion-Luminescent Nanomaterials. Nanomaterials (Basel). 2014;4:129-54

61. Cheng L, Yang K, Zhang S, Shao M, Lee S, Liu Z. Highly-sensitive multiplexed in vivo imaging using pegylated upconversion nanoparticles. Nano Res. 2010;3:722-32

62. Park YI, Lee KT, Suh YD, Hyeon T. Upconverting nanoparticles: a versatile platform for wide-field two-photon microscopy and multi-modal in vivo imaging. Chem Soc Rev. 2015;44:1302-17

63. Cheng L, Wang C, Liu Z. Upconversion nanoparticles and their composite nanostructures for biomedical imaging and cancer therapy. Nanoscale. 2013;5:23-37

64. Zhou J, Liu Z, Li F. Upconversion nanophosphors for small-animal imaging. Chem Soc Rev. 2012;41:1323-49