Big Data Analytics of Knowledge and Skill Sets for Web Development Using Latent Dirichlet Allocation and Clustering Analysis

Abstract

Web development is a data-centric field and fundamental component of data science. The advent of big data analytics has significantly transformed the processes, knowledge domains, and competencies associated with Web development. Accordingly, educational programs must adjust to contemporary advancements by initially determining the abilities required for big data web developers to satisfy industry demands and adhere to current trends. This study aims to identify the knowledge areas and abilities essential for big data analytics and to create a taxonomy by correlating these competences with currently popular tools in web development. A mixed method consisting of semi-automatic and clustering methods is proposed for the semantic analysis of the text content of online job advertisements associated with the development of big data web applications. This methodology uses Latent Dirichlet Allocation (LDA), a probabilistic topic modeling tool, to uncover hidden semantic structures within a precisely specified textual corpus and average linkage hierarchical clustering as a clustering analysis technique for web developers. The results of this study are a web development competency map which is expected to help evaluate and improve the knowledge, qualifications and skills of IT professionals being hired. It helps to identify the roles and competencies of professionals in the company’s personnel recruitment process; and meet industry skill requirements through web development education programs. The competency map consists of knowledge domains, skills and essential tools for web development such as basic knowledge, frameworks, design and user experience, database design, web development, cloud computing and other soft skills. Furthermore, the proposed model can be extended to several types of jobs in the IT sector.