Detecting cloves quality in Indonesia still uses manual labor. Therefore, errors often occur in sorting cloves. The quality of cloves depends on good weather. Unpredictable weather will prolong the clove drying process and make the cloves damaged and moldy. In this study, we use the CNN architecture with several combinations, namely the number of convolutions, the number of dense layers, and the size of the layers. The CNN architecture is trained using several variations of the color space and several variations of image segmentation to classify the clove quality. This study used three color spaces (RGB, HSV, and YCbCr) and two segmentation methods (Otsu segmented and HSV color segmentation). The best accuracy is obtained by using the HSV color space, original dataset, and E-5C-64LS-4D architecture that is 96% accuracy, precision, recall, and F1-Score. To get a better model, we use the proper image segmentation method and composing the suitable color space to improve the CNN architectural performance.