Marine debris, a hazardous threat to the marine environment, has emerged as a critical global environmental issue. Manual removal is the method most commonly applied to remove marine debris from the marine environment. One solution to this problem is to use deep learning-based visual marine debris detectors to detect marine debris automatically. However, in terms of marine debris detection, there are several challenges, one of which is because the complex marine environment makes visual detectors able to detect marine debris accurately and in real-time to avoid damaging the marine environment. In this research, three variants of the RTMDet (Real-Time Models for Object Detection) model were trained and evaluated using the TrashCan-Instance dataset. One of them is the RTMDet-l model with improvisation to replace the loss bbox in the head model with DIoU (Distance-Intersection over Union) and improvisation to add a sampling strategy with a Class-aware Sampling technique to handle the imbalanced data problem that has obtained mAP50 (Mean Average Precision at threshold 50%) accuracy of 71.3%. This has made the model's object detection accuracy on the TrashCan-Instance dataset the best while maintaining detection speed. These results prove that the model proposed in this study can be a vital consideration for further development in detecting marine debris. This contribution aims to address the global challenges related to marine debris and stimulate the development of more effective and efficient object detection models in complex marine environments.