With the potential of Cavendish banana in the global food industry, Indonesia as one of the top producer of banana still lacks behind compared to other countries. This can be attributed to the traditional method which can be time-consuming, labor-intensive, and subjective due to human nature. This research investigates the potential of using Convolutional Neural Networks (CNN), specifically ResNet50 strengthened with Convolutional Block Attention Module (CBAM) architecture, as an alternative to the traditional methods. We proposed performance evaluation and optimization of ResNet50 architecture with the addition of attention module to further heightened the performance based on the dataset of Cavendish bananas. The chosen attention module, CBAM, is favored due to its performance boost, easily implementable aspect, and small, lightweight architecture. This attention-enhanced ResNet50 model pushed the already excellent base model by helping it focus on important features within the input image. Accordingly, the resulting model is evaluated using metrics such as: accuracy, precision, recall, and F1-Score. The dataset of banana images contains various conditions of lighting and backgrounds to further better train the model. The results showed that the proposed model was able to correctly classify with an accuracy of 94.42\% proving its capability of ripeness classification task. These results implicated that a attention-fitted CNN model was able to perform better than the baseline model and we recommend a further investigation into other potential parameters within the ResNet50 model and CBAM block, along with implementation of CBAM block with other CNN models in order to explore the potential of attention-enhanced models.