Comparative Study On Optical Coherence Tomography (OCT) Retinal Disease Segmentation

Abstract

Optical Coherence Tomography (OCT) is a non-invasive imaging technique that generates high-resolution cross-sectional images of the retina, and has become a critical modality for the diagnosis of retinal diseases. Diabetic Macular Edema (DME) represents one of the most clinically prevalent such conditions, causing significant vision loss and requiring early and precise diagnosis, particularly in resource-constrained settings such as Nepal, where no AI-assisted OCT analysis tool currently exists. This study presents a systematic comparative evaluation of three deep learning-based segmentation architectures — U-Net, U-Net++, and Y-Net — for automated identification of DME and Intraretinal Fluid (IRF) regions in OCT scans. A dataset of retinal OCT images was manually annotated using the Computer Vision Annotation Tool (CVAT) and augmented with geometric transformations to improve model generalization. Models were trained using a hybrid Focal Tversky and Focal Cross-Entropy loss and evaluated on a held-out validation set using standard segmentation metrics: Dice Similarity Coefficient (DSC), Intersection over Union (IoU), precision, and recall. Among the evaluated models, Y-Net achieved the best overall performance, attaining an F1 score of 0.8651, IoU of 0.8142, precision of 0.9210, and recall of 0.8424, outperforming both U-Net++ (F1: 0.8432, IoU: 0.7925) and the baseline U-Net (F1: 0.8246, IoU: 0.7599). These results demonstrate the feasibility of deep learning-based OCT segmentation as a diagnostic support tool in resource-constrained clinical environments.