In the field of natural language processing (NLP), keywords are crucial for enhancing information retrieval (IR) and content summarization, as well as for optimizing search engines and organizing documents. As the volume of generated information increases, identifying keywords manually from large documents becomes more challenging and no longer feasible. Therefore, automatic keyword extraction is necessary as a cost-effective method to handle large documents and to provide scalable solutions for various applications in NLP and information management. In the academic domain, automatic keyword extraction simplifies the process of finding and categorizing scientific publications, enabling paper repositories to optimize their IR and document organizing systems. However, many methods of keyword extraction use either global semantic features based on pre-trained embedding models or local statistical features separately, which yields low results. Since a good keyword must be identified by both external knowledge and local statistical features, this paper proposes a method to improve the performance of keyword extraction from scientific publications by combining local statistical features with embedding models. The proposed method outperforms the baseline methods with an F-score of 0.70 on the SemEval2017 dataset using the SciBERT model and SVM classifier. This research confirms that both local statistical information and contextualized semantic information are important to identify keywords.