Two-photon fluorescence microscopy signal formation in highly scattering media: theoretical and numerical simulation

Using the radiative transfer theory and Monte Carlo simulations, we analyse the effect of scattering in a medium and of the size of the detector pinhole on the formation of the fluorescent signal in standard two-photon fluorescence microscopy (TPFM) systems. The theoretical analysis is based on a small-angle diffusion approximation of the radiative transfer equation, adapted to calculate the propagation of focused infrared radiation in media similar to the biological tissues in their optical properties. The accuracy of the model is evaluated by comparing the calculated excitation intensity in a highly scattering medium with the results of Monte Carlo simulations. To simulate a tightly focused Gaussian beam by the Monte Carlo method, the so called "ray-optics" approach that correctly takes into account the finite size and shape of the beam waist is applied. It is shown that in the combined confocal and two-photon scanning microscopy systems not equipped with an external "nondescanned" detector, the scattering significantly affects both the nonlinear excitation efficiency in the medium and the fluorescence collection efficiency of the system. In such systems, the rate of the useful TPFM signal in-depth decay is 1.5 — 2 times higher than in systems equipped with a "nondescanned" detector.