Diabetic retinopathy (DR) occurs in almost all patients with diabetes and remains as one of the major causes of vision loss worldwide. Nevertheless, the molecular mechanisms underlying the pathogenesis of DR remain elusive. The present study aimed to investigate the role and association of Notch signaling and NADPH oxidase 4 (Nox4)-mediated oxidative stress in high glucose (HG)-treated retinal cells. Human retinal endothelial cells were cultured for various durations in RPMI-1640 medium containing 30 mM glucose (HG) or 30 mM mannitol (MN) as an osmotic control; apoptotic cell death and reactive oxygen species (ROS) levels were assessed, respectively. Alterations in the expression profiles of Nox and Notch proteins were evaluated using reverse transcription-quantitative polymerase chain reaction and western blot analysis. Knockdown of Nox4 and recombination signal-binding protein J (RBPj) was generated by transfection with specific small interfering (siRNA). Persistent activation of Notch signaling was induced via the overexpression of Notch intracellular domain (NICD). In the present study, time-dependent increases in ROS production and cell death were detected in HG-treated cells. Depletion of ROS by diphenyleneiodonium decreased HG-induced cell death, and suppressed increases in caspase 3 activity and B-cell lymphoma 2-associated X protein levels. In HG-treated cells, Nox4 expression was upregulated at the mRNA and protein levels, and inhibition of Nox4 by GKT137831 or knockdown of expression by siRNA Nox4 significantly reduced ROS levels and cell death. In the presence of HG, Notch1 expression levels were elevated, and increased NICD abundance was detected in whole cell lysates and nuclear fractions. Additionally, HG-induced cell death was decreased by treatment with -secretase inhibitor (GSI), but increased via the overexpression of NICD. The application of GSI or knockdown of RBPj by siRNA RBPj prevented increases in Nox4 expression within HG-treated cells. The findings of the present study demonstrated that Nox4-mediated ROS serves an important role in HG-induced retinal cell damage, in which the activation of Notch signaling may be responsible for Nox4 upregulation. Therefore, inhibition of Notch signaling or Nox4 expression may be considered as potential therapeutic targets in patients with DR.