The increasing intensity of train use in large cities has resulted in some damage to railway tracks. One of the defects that appears is on the surface of the railway track, and this maintenance will require workers around the clock to inspect and maintain the railway track surface periodically. This will impact the need for labor if done manually for periodic inspections, and of course, will increase the need for operational and other costs. To streamline time in field inspections, we analyzed and compared deep learning-based surface defect detection systems using several variations of the You Only Look Once (YOLO) Algorithm: YOLOv6, YOLOv7, YOLOv8, and YOLO NAS. In addition, our experiments focused on the training results using the RSSD dataset, which consists of high-speed rail (type 1) and heavy rail (type 2). In the research results, YOLOv8 is a model variation that balances precision and recall and has a high enough computation to detect damage on the type 1 also for type 2 datasets with [email protected]:0.95 value, reaching 0.69 and 0.474. Several studies are needed to obtain high-quality data so that it can be implemented in real time.