1. Introduction
As a result of the leadership of today’s rapidly advancing information and communication technologies, big data has become an indispensable strategic information resource and a potential power more than ever before. The ever-increasing participation and utilization of online communication and interaction networks have led to the exponential growth of various types of data produced and shared by individuals, institutions, devices, and systems [1]. Big data consists of massive and frequently unstructured datasets made available by a variety of data sources, such as online platforms, social networks, and organizations [2]. These large-scale data sources have the potential to facilitate increased productivity, quicker and more effective decision-making, more accurate forecasts, and greater adaptability [3]. However, processing this large volume of diverse types of unstructured data with efficient processes and methodologies and obtaining meaningful inferences requires extensive knowledge, skills, and abilities [4]. Big data competencies include a variety of data-driven operational skills, such as all types of data collection, data logging, data cleansing, data analytics, data visualization, data security and privacy, scaling, deployment, empowerment, and querying [3,5,6]. Among the responsibilities of big data specialists, interpreting data and identifying their strategic and potent potential through the use of data mining and analytic models is the most essential responsibility. Big data expertise requires familiarity with the required software and hardware to acquire data and manage multiple data sources [1,7,8].
To derive all possible benefits from sources of big data, both government and commercial organizations must possess the necessary big data knowledge and skills [9]. Big data competencies are crucial skills that will play an important role in today’s data-driven decision-making across all industries [3]. As a result, the need for professionals with the knowledge and skills to effectively manage and interpret big data has become a pressing issue that must be addressed. There are new career opportunities for big data professionals every day, particularly in the fields of big data processing and big data analytics [10,11]. As the use of services and applications centered on big data has become more pervasive, the demand for qualified labor in the field of big data has steadily increased, and big data has become one of the most important sources of employment in the current labor market [4,5,8]. In this context, big data competency requirements provide IT professionals with diverse career opportunities [6]. For these workers, big data jobs have become increasingly tangible over the past decade. Throughout the life cycle of big data-oriented applications, big data professionals may assume various responsibilities and roles, depending on the scope and nature of their work [4,12]. By analyzing and defining the roles (titles and areas of expertise) and responsibilities (competencies and skills) of big data professionals, it is possible to gain a better understanding of all available big data professions [4,5]. There is a clear research gap in identifying the roles and related skills required for the most prominent big data jobs and defining the academic supply required to meet this demand.
Given the current background, we aim to close the aforementioned knowledge gap by empirically identifying and defining the job roles and skills necessary for businesses to effectively exploit the potential of big data. In particular, we intend to (1) identify and categorize competency requirements for big data professionals from a practitioner’s perspective and (2) establish correlations between these requirements and professional roles. To achieve this, we utilized online job postings as a data source and conducted a topic modeling-based semantic content analysis on 2280 big data job postings. This analysis resulted in the identification of 32 topics (skills) for big data professions, which define seven expertise roles and their respective competencies. For each of these roles, a conceptual job description has been developed, and the primary roles and responsibilities of big data professionals have been identified. Consequently, the findings of this study can provide insightful contributions to big data communities with a wide spectrum of profiles, and to industries in different arenas that are embracing the potential and benefits of big data.
2. Background and Related Work
Since the term “big data” first appeared in the scientific literature, it has been a dynamic industry closely related to human resources and a significant source of employment [13]. The requirement for expertise in big data, which has emerged as a natural consequence of the widespread use of big data, has been a fundamental issue frequently emphasized in the literature, particularly over the past decade [14]. Recent research indicates that, unlike traditional data proficiency, big data-driven services and applications require more advanced and specialized professional knowledge and skills.
Numerous synchronized processes based on storage, processing, analytics, and visualization comprise big data workflows [2]. Services and applications based on real-time data processing and analysis utilizing dynamic data flows have spawned new specializations for big data. Notable big data applications include online analytical processing (OLAP), streaming analytics, business intelligence, and business data analytics [15]. Consequently, meeting the growing demand for skilled labor in the big data industry has emerged as a problem that must be addressed in the near future [12]. This issue has been identified in the literature as a big data skills gap and has been highlighted in numerous scientific studies and industry reports [8,12,14]. On the other hand, a limited number of specific studies have been conducted based on the analysis of job postings to reveal the required knowledge and skills in the big data industry [5]. The focus of these studies was on various aspects of big data competency requirements, including big data analytics, data science, and other big data specializations [3,6,8,16].
Among them, the study by Debortoli et al. (2014) is one of the most prominent. They compiled a dataset with 1357 business intelligence and 450 big data job postings [3]. They analyzed business intelligence and big data job postings using text mining procedures and the latent semantic analysis (LSA) technique. De Mauroa et al. (2018) analyzed 2786 job postings related to big data utilizing the latent Dirichlet allocation (LDA) method in a separate study [4]. They identified four roles (business analysts, data scientists, developers, and system administrators) and nine skills that correspond to these roles. Using text mining techniques, Gardiner et al. (2018) analyzed 1216 big data job postings [8]. In a similar study, Radovilsky et al. (2018) surveyed 1050 online job postings using semantic content analysis to identify the knowledge domains and skill sets that distinguish data scientist and data analyst positions [16].
Gurcan (2019) analyzed 2175 big data job postings using latent Dirichlet allocation (LDA), a generative method for topic modeling, and identified 60 topics for big data competencies [17]. Using the LDA-based topic modeling technique, Gurcan and Cagiltay (2019) analyzed 2638 job postings and revealed the big data skills required for software engineering [5]. Using text mining techniques, Persaud (2020) analyzed 3009 job postings for big data analytics. This study found that cognitive and functional skills, as well as social and meta-skills, are necessary for big data analytics [10]. Using the LSA method, Halwani et al. (2022) conducted a study on big data and data science skills by analyzing 1200 job postings. As a result, they determined the connections between academic and sectoral perspectives [6].
These studies, which investigate the various contexts of big data careers, have revealed the knowledge domains and skill sets required for big data professionals. The studies aimed to close the talent gap in the big data industry, benefiting both industry and academia. In this regard, our study on the competency requirements for big data extends the methodology and findings of the previous research.
3. Materials and Methods
Using quantitative procedures based on topic modeling, we will investigate the major dynamics, themes, and tendencies of the big data labor market. This study’s phases include data collection, preprocessing, and analysis. An empirical dataset of big data job postings was subjected to a semantic content analysis based on probabilistic topic modeling using a semi-automated methodology. The overall research design and methodological workflow are illustrated in Figure 1, which provides a visual guide to the sequential steps of the analysis. The phases of data collection, preprocessing, analysis, and interpretation are each addressed in detail in the following subsections to outline the methodological workflow of the study.
3.1. Data Collection and Preprocessing
The World Wide Web (www), which contains a vast amount of information in a variety of formats and structures, is an ideal source of information for such scientific studies. To create the most relevant dataset, we sought out a reputable data source (online employment website) from which we could receive unrestricted job postings. Indeed.com [18], a well-known employment website with extensive search, filter, and retrieval capabilities, was selected as the data source for this study. Indeed.com is the most popular job site in the world, with over 250 million monthly visitors [17,18,19]. All job postings containing the phrase “big data” in the job title were considered big data jobs in this context. Using our search query, you can view examples of the job postings that comprise our empirical dataset on the Indeed website. Using an API developed by Indeed.com, 2280 big data job postings published between July 2024 and December 2024 were retrieved [18]. The job postings were saved in a database, and an experimental dataset was compiled. Each job listing in the dataset is divided into two sections: job title and job description.
After creating the dataset, sequential data preprocessing steps were applied to the dataset consisting of texts from job postings [20]. Data preprocessing is essential, especially for the success of web-based unstructured text-driven data analysis [21]. In this analysis, the experimental dataset was subjected to a series of sequential preprocessing steps. Initially, text was tokenized, or divided into words, to obtain meaningful attributes [22,23]. All text was converted to all lowercase letters. Then, unnecessary punctuation, web links, HTML tags, and characters were removed. Then, stop words in English were removed from the texts to save space. The process of lemmatization was applied to the texts in order to reduce the terms from their derived form to their simple form [24,25,26]. The remaining words were used to create a document–term matrix (DTM) in order to conduct the intended numerical analysis [27,28,29].
3.2. Data Analysis and Interpretation
In the first phase of the empirical analysis, we presented a taxonomy for big data specializations by classifying job titles based on their essential expertise roles. The job title of each big data position is a significant indicator of the expertise role of the candidate who will work in that position, i.e., in which role they will work [4]. In this regard, based on the words in the job titles, each position has been assigned a role associated with the big data area of expertise. For instance, job titles such as big data engineer, big data systems engineer, and big data infrastructure engineer, which contain derivatives of the word “engineering”, fall within the engineer role’s competency area. Similarly, the words in all other big data job titles were identified and assigned to the corresponding competency roles. Thus, seven primary expertise roles for big data professions have been identified.
After identifying the expertise roles, latent Dirichlet allocation (LDA) topic modeling was performed on the textual corpus of big data job postings [27,29,30]. LDA is a probabilistic and generative method for demonstrating the semantic structure of a textual corpus [27]. Text documents contain hidden semantic patterns referred to as “topics”. Multiple topics may appear in varying proportions in a text document. LDA-based topic modeling is based on determining these topics and their proportions. LDA employs an unsupervised learning model to discover hidden topics in an unstructured text corpus without training or explanation. Consequently, the LDA algorithm is utilized for the topic modeling analysis of large datasets [29,31,32,33].
To fit and apply the LDA model to the big data job postings corpus, we used Gensim, a Python (3.12) library specialized for topic modeling [22,29]. The LDA model was then applied with K values ranging from 10 to 60 in order to determine the optimal number of topics [29,34]. For each K value change within this range, a model’s coherence score was calculated. When we examined the calculated coherence scores for each K in this range, we discovered that the coherence score decreased as the number of topics increased. The maximum coherence score, which expresses the best semantic consistency of the discovered topics, was obtained for K = 32 topics [22,34]. For each of the 32 topics, the top 10 descriptive keywords with the highest frequency were determined [19,29,30]. Using the open card sorting method, each topic’s topic name was determined and assigned after descriptive keywords were considered [35]. Additionally, the distribution rates of the discovered topics per document, the distribution of words per topic, and the proportions of the topics in the entire corpus were calculated [19,31]. In all subsequent analyses, these 32 topics discovered by LDA were used to represent big data skills.
4. Results
4.1. Identification of Expertise Roles
This section presents the results of an analysis of job postings involving big data. Expertise roles and job titles were extracted from the job postings’ title texts. Taking into account the distribution of job titles across the roles, seven specialization roles were determined. Table 1 lists these positions, their top five job titles, and their respective percentages.
According to Table 1, the “developer” role has been the most prevalent specialty role, accounting for 34.69%. The rate for “engineer” is then 33.99%. It follows the role of “architect” with a proportional decrease of 10.48%. The remaining positions are “analyst” (9.15%), “manager” (5.18%), “administrator” (3.39%), and “consultant” (3.21%). According to these results, “developer” and “engineer” roles account for nearly 70% of the big data positions. The remaining five roles account for roughly 30%.
4.2. Identification of Skill Sets
This phase of our analysis reveals the fundamental competencies and information domains associated with big data. Thirty-two skills have been identified as a result of LDA-based topic modeling analysis. Table 2 provides the percentage distribution of the identified competencies along with the keywords that define each skill-based topic label.
The results depicted in Table 2 reveal a wide range of technical, analytic, and soft skills within the domain of big data. These 32 discovered skills also emphasize the interdisciplinary nature of big data competencies. As shown in the table, “big data processing” (6.82%) is the most popular skill related to big data, followed by “big data tools” (5.93%). “Communication skills” (5.02%) is positioned third. This demonstrates that soft skills are as important as technical skills for jobs involving big data. The lowest-ranked competencies in the distribution are “Azure Cloud” (1.77%), “team working” (1.76%), and “decision-making” (1.63%), indicating a relatively lower emphasis on these topics.
4.3. Taxonomy of the Skill Sets by Competency Areas
In this phase of the analysis, the knowledge domains and skill sets were categorized and presented in a more systematic way. The knowledge domains and skills denoted by the 32 topics were allocated into six competency areas, and a systematic competency map was developed for big data skills. The percentages of the skills according to the competency areas are presented in Figure 2. As shown in Figure 2, the first competency area, “big data knowledge” (36.30%), contains ten knowledge and skill items consisting of “big data processing”, “big data tools”, “big data architecture”, “information security”, “scalable systems”, “database”, “distributed systems”, “data warehousing”, “Hadoop ecosystem”, and “data streaming”. The second, “developer skills” (20.25%), has six items, namely “remote development”, “programming languages”, “Agile development”, “business applications”, “application development”, and “testing”. The third, “big data analytics” (14.88%), contains four items, specifically “data analytics”, “analytical skills”, “business analytics”, “machine learning”, “data visualization”, and “decision-making”. The fourth, “cloud services” (10.51%), has four items: “Amazon EMR”, “Google Cloud”, “AWS Data Services”, and “Azure Cloud”. The fifth, “soft skills” (10.16%), consists of three items: “communication skills”, “project management”, and “team working”. The sixth, “technical background” (7.90%), has three items comprising “technical knowledge”, “bachelor degree”, and “troubleshooting”.
This taxonomy systematically reveals the most in-demand competency areas within the big data ecosystem. The fact that “big data knowledge” ranks first, with a high proportion of 36.30%, highlights the central role of technically grounded expertise in the industry. The substantial shares of “developer skills” and “big data analytics” further emphasize the importance of practical software development and analytical capabilities. In contrast, the lower percentages associated with “soft skills” and “technical background” suggest that while technical competencies are prioritized, complementary skills are also necessary. In summary, this distribution reflects the industry’s strong demand for professionals who possess both domain-specific technical knowledge and applied problem-solving abilities.
4.4. Mapping Skill Sets to Expertise Roles
At this stage of analysis, skill sets and roles are matched. From this perspective, a systematic taxonomy of roles and skills is provided for the field of big data. The proportional distributions, correlations, and skills in the field of big data are presented in Table 3. The table’s skills are arranged according to the percentages in the final column. This table displays each role’s dominant and passive skills in green and red, respectively. The seven roles are ordered from largest to smallest in columns.
Even though “big data processing” has the highest rate in the first row of Table 3, it is dominant in the “engineer” role and relatively passive in the “administrator” role. “Big data tools” and “communication skills” are also essential “developer” skills. With these findings, it is clear that the “developer” role in the big data field requires both technical and soft skills.
In order to provide a clearer understanding of the results of the analysis, we have summarized the first ten skills of each role in Table 4. As seen in the table, 24 of the 32 skills are included in the first 10 skills of each role’s list.
As shown in Table 4, “communication skills” is one of the top ten skills for all big data positions. Second, the term “big data tools” is present in six of the seven roles. Additional dominant skills include “analytical skills”, “big data architecture”, and “big data processing”. Similar to Table 3, Table 4 applies a coloring scheme (from green to red) from dominant to passive skills. For instance, the top three competencies for the “developer” position are “big data tools”, “programming languages”, and “remote development”. Their order numbers are displayed in green. In contrast, the last three highlighted skills are “Amazon EMR” (8), “scalable systems” (9), and “business applications” (10). The order numbers of these last three are shown in red. To identify the most dominant skills associated with other roles, refer to Table 4 by focusing on the corresponding column for each role. Each row in this table indicates the relative importance of each skill. For instance, in Row 2 of Table 4, “big data tools” ranks first for “developer”, second for “engineer”, and fourth for “architect”. However, this skill is not among the top ten for “analyst”.
5. Discussion
Using text mining and probabilistic topic modeling techniques, this study analyzed the semantic content of job postings for big data jobs. This analysis uncovered seven expertise roles, 32 skills, and six competency areas required for big data professions. A high percentage (70%) of big data professionals have “developer” or “engineer” as their primary role, which is a significant inference. This finding was also emphasized in previous studies. De Mauro et al. (2018) identified four roles as “Business Analysts”, “Data Scientists”, “Developers”, and “System Managers” in their study [4]. Our findings overlap with three of these four roles. We determined the results for “engineer”, “architect”, “manager”, and “consultant” in different ways. We found that the “architect” position is approximately three times more in demand than the “administrator” position. However, De Mauro et al. (2018) added “Architect” to “System Managers” [4].
These results are significant indicators of the diversification and evolution of big data roles and skills over time. According to their study, the “Architect” role is part of the engineering family [4]. However, according to our findings, the percentage of the “architect” role is extremely high. It is also seen that they should have different skills from “engineers” and “developers”. On the other hand, the “consultant” role among the specialist roles, unlike other studies, is another remarkable finding. Due to the rapidly changing and transforming work dynamics, it is seen that the role of “consultant” is becoming increasingly important for big data professions. De Mauro et al. (2018) described program managers and data consultants as roles within the family of business analysts [4]. In the literature, “consultant” has not been highlighted as the primary role [4,10,14]. One of our further findings is that “project management” proficiency is the most sought-after skill for “manager” and “consultant” positions. In this sense, experts who assume the role of “consultant” are expected to have senior management skills.
Our second discovery is that big data skills contain six core proficiency areas that include “big data knowledge” (36.30%), “developer skills” (20.25%), “big data analytics” (14.88%), “cloud services” (10.51%), “soft skills” (10.16%), and “technical background” (7.90%). This finding revealed that the required skills for big data jobs include a wide range of competency areas. The wide spectrum of skills required for big data jobs is an important finding frequently mentioned in previous studies [4,5,8,10]. Compared to earlier studies such as [4,5], our findings confirm a broader spectrum of competencies, especially highlighting the simultaneous importance of technical and interpersonal skills in the current big data labor market. The emergence of soft skills—most notably “communication skills” and “team working”—among the top competencies across multiple roles reflects a shift towards more human-centric expectations in technically demanding roles, consistent with the evolving nature of data-driven team work and organizational adaptability [3,6,10].
Our findings clearly demonstrate that the vast majority of big data professions require employees at all levels to possess a collective set of technical, developer, analytical, and soft skills in order to drive decision-making processes that combine technical and analytical understandings with intuition and vision [10,16,17,36]. This alignment suggests that the big data profession is increasingly interdisciplinary, requiring a fusion of analytical thinking, programming expertise, and interpersonal competencies, rather than purely technical proficiency. Our results reinforce the argument that in environments characterized by complex data infrastructures and agile development teams, soft skills play a decisive role in enabling collaboration, problem-solving, and cross-functional communication. It has been revealed by our study that valuable insights and predictions for businesses are provided by combining machine learning and business analytics approaches with big data skills [15,37,38,39].
Our study also showed that big data is a multidisciplinary field comprised of various disciplines, as stated in previous studies [4,8,17]. The aforementioned studies argued that a variety of skills are required to analyze, interpret, and maintain big data in order to make effective business decisions, emphasizing that the discipline of big data is complex and multifaceted [4,8,17]. According to our topic modeling analysis, the top five skills emerged as “big data processing”, “big data tools”, “communication skills”, “remote development”, and “big data architecture”. The topic of “communication skills” is another significant finding of our analysis. There are many studies supporting this finding [4,6,8,17,40]. As shown in Table 4, “communication skills” is the only skill required for all roles and is among the top ten skills. With a rate of 5.02%, it also ranks third among big data skills. This result demonstrates the significance of communication-focused abilities. Despite the increasing demand for new technological skills, many studies have clearly emphasized the necessity of communication and social skills for big data professions [4,5,6,8,10,17,20,26,41].
“Team working” is another soft skill that should be emphasized. Employers expect a coordinated and collaborative team work style from employees with various qualifications. This is because big data is a work environment that requires different skill sets from experts in different roles (see Table 3). Another remarkable finding from our analysis is the topic “bachelor degree”. As seen in Table 4, “bachelor degree” is only included in the tenth skill of the “analyst” role and remains passive compared to other skills. So, this skill was not in the top ten skills of the other six roles. As a result, it is seen that professional experience and competencies are more dominant than undergraduate degrees in the competencies required for big data professions.
Among big data knowledge and skills, the number and high rates of competencies based on cloud services such as “Amazon EMR”, “Google Cloud”, “AWS Data Services”, and “Azure Cloud” have revealed the close relationship between big data and cloud services (see Figure 2). Likewise, the fact that many tools and technologies used in big data processes are cloud-based confirms this finding. Our findings regarding cloud services show that, as stated in other studies [5,8,19], big data specialists prefer cloud-based services and platforms over traditional platforms. This is because cloud computing platforms and services provide big data specialists with more innovative paradigms and solutions for storing, analyzing, managing, and processing big data [42].
Although the big data knowledge and skill sets that emerged as a result of our analysis are similar to other studies, some specific skills have changed proportionally over time. The change and diversification in industrial skill demands have caused some competencies to fall into the background and some new skills to emerge [5,43,44,45]. This is due to the dynamic structure of the field, which is constantly updated and renews itself. Information technologies and big data-oriented applications and services are constantly evolving and changing [33,44]. In addition to identifying key competencies, this study emphasizes the practical and ethical implications of digital transformation. Educational programs should integrate both technical and interpersonal skills, while human resources departments must adapt to continuous upskilling demands. The growing need for inclusive, lifelong learning highlights the importance of providing training that is not only relevant but also socially equitable [26,46]. Based on our findings, we predict that in the near future, the roles of expertise and related competencies in the big data field will become more diverse, and different sub-disciplines will emerge as the umbrella of the big data field. The big data phenomenon is becoming more popular day by day, and therefore it is likely that big data undergraduate and graduate programs in various scopes will emerge from the big data discipline.
6. Conclusions
In this study, we aim to bridge the skills gap between industry and academia by empirically defining the roles and skills required for big data expertise from the practitioner’s perspective, thereby enabling industries to effectively exploit the potential gains of big data. To this end, a machine learning-based semantic content analysis was conducted on 5444 online job postings for big data positions using text mining and topic modeling procedures. As a result of this analysis, seven expertise roles, six proficiency areas, and 32 competencies (knowledge, skills, and abilities) required for big data professions were revealed. Based on these findings, we developed a practitioner-oriented conceptual taxonomy of the knowledge, skills, and abilities required for big data professions. This taxonomy validates the multifaceted nature of big data competencies as well as their complexity and diversity.
Based on our findings, it can be said that the big data competencies revealed in this research can be an informative guide for data science education and lifelong learning programs in order to satisfactorily meet the qualified human resources needs of the industry. It is anticipated that the findings of this study will contribute to the following issues: (1) to companies in human resource management and employment of a qualified workforce in the field of big data; (2) to big data experts in the evaluation and development of their competencies; (3) to academic institutions and organizations in the development of up-to-date big data curricula to meet emerging industrial workforce demands; and (4) to students and candidates in guiding their careers in the field of big data.
As with many studies, this research has a number of potential limitations. Considering the fact that the big data discipline and the employment market are evolving rapidly, it should be noted that our analysis covers only a certain time period. It should be taken into account that such analyses, in the near future, may produce different findings over time as the qualified human resources requirements of industries are constantly changing. As a data source, only the job postings on Indeed.com, the world’s leading job search site, were included in this study. One limitation of this study is the reliance on a single data source (Indeed.com), which may introduce potential sampling bias. Future research can benefit from incorporating multiple job platforms such as LinkedIn, Glassdoor, and Monster to ensure broader representativeness and cross-platform validation. In this regard, our investigation can be expanded by including different employment sites as data sources. The methodology of the study is based on the LDA topic modeling approach, which is extensively used in text mining. By using different alternative approaches for text mining and topic modeling, future studies can expand our methodology and findings and provide descriptive implications that reveal the time-based evolution of big data proficiencies.
Literature review and research background, B.G., F.G., G.G.M.D. and M.D.; conceptualization, B.G., F.G., G.G.M.D. and M.D.; methodology, B.G., F.G., G.G.M.D. and M.D.; software, B.G., F.G., G.G.M.D. and M.D.; validation, B.G., F.G., G.G.M.D. and M.D.; investigation, B.G., F.G., G.G.M.D. and M.D.; resources, B.G., F.G., G.G.M.D. and M.D.; data curation, B.G., F.G., G.G.M.D. and M.D.; writing—original draft preparation, B.G., F.G., G.G.M.D. and M.D.; writing—review and editing, B.G., F.G., G.G.M.D. and M.D.; visualization, B.G., F.G., G.G.M.D. and M.D.; supervision, B.G., F.G., G.G.M.D. and M.D.; project administration, B.G., F.G., G.G.M.D. and M.D. All authors have read and agreed to the published version of the manuscript.
Not applicable.
Not applicable.
The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy.
The authors declare no conflicts of interest.
Footnotes
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Figure 1 An overview of the research methodology.
Figure 2 Taxonomy of big data skills by competency areas.
Expertise roles and related job titles for big data professions.
| Role | Related Titles | Rate (%) |
|---|---|---|
| Developer | Big Data Developer; Big Data Software Engineer; Big Data Spark Developer; Big Data Hadoop Developer; Java Big Data Developer | 34.69 |
| Engineer | Big Data Engineer; Senior Big Data Engineer; Lead Big Data Engineer; Principal Big Data Engineer; Big Data Platform Engineer | 33.99 |
| Architect | Big Data Architect; Big Data Solution Architect; Google Cloud Big Data Architect; Senior Big Data Architect | 10.48 |
| Analyst | Big Data Analyst; Big Data Analytics; Business Analyst; BI analyst; AWS Big Data Analytics | 9.05 |
| Manager | Big Data Lead; Big Data Technical Lead; Big Data Program Manager; Big Data Solution Manager; Big Data Product Manager | 5.18 |
| Administrator | Big Data DBA; Big Data Admin; Big Data Administrator; Big Data Hadoop Administrator; Big Data Platform Administrator | 3.39 |
| Consultant | Big Data Consultant; Big Data Hadoop Consultant; Big Data Solution Consultant; Big Data Technology Consultant; AWS Big Data Consultant | 3.21 |
The 32 topics (skills) discovered by LDA.
| Topic Name | Keywords | % |
|---|---|---|
| Big Data Processing | data pipeline process build engineer platform analytic etl lake develop | 6.82 |
| Big Data Tools | spark hadoop hive hbase java knowledge scala developer python sql | 5.93 |
| Communication Skills | skill strong ability communication write environment excellent problem good service | 5.02 |
| Remote Development | remote developer warehouse software reliably telecommuting connect good location part-time | 4.69 |
| Big Data Architecture | solution design architecture architect technical pipeline system structure enterprise implement | 4.64 |
| Programming Languages | software solution development programming tool practice java python language scala | 4.38 |
| Agile Development | software development design agile scrum product modeling team cross-functional customer | 4.03 |
| Information Security | information safety security privacy data financial banking risk prevent threat | 3.75 |
| Project Management | project management technical plan identify lead manage process manager program | 3.38 |
| Scalable Systems | scalability platform engineering engineer deliver structure scale service analytic healthcare | 3.28 |
| Amazon EMR | amazon system design software service emr aws engineer development distribute | 3.23 |
| Data Analytics | data analysis model business develop tool sql quality analytic knowledge | 3.16 |
| Business Applications | application development business support integration deploy delivery software agility design | 3.09 |
| Analytical Skills | analysis analytics critical prediction capability inferential analytical problem-solving report | 3.05 |
| Business Analytics | business analytics drive product customer leadership partner organization role strategy | 3.01 |
| Google Cloud | cloud platform build google service gcp infrastructure kubernete docker engine | 2.99 |
| Technical Knowledge | technical knowledge expert strong skill background look grow diversity professional | 2.98 |
| Database | database nosql sql stream management system distribute relational kafka mongodb | 2.93 |
| Bachelor Degree | science computer degree engineering relate field security system minimum qualification | 2.73 |
| AWS Data Services | aws cloud redshift sql emr python tool lambda glue engineer | 2.53 |
| Distributed Systems | distribute apache system kafka process parallel hdfs storm computing oracle | 2.39 |
| Data Warehousing | warehouse data warehousing process storage repository store tool model server | 2.33 |
| Hadoop Ecosystem | system hadoop performance issue cluster support database infrastructure security admin | 2.28 |
| Machine Learning | learn machine data science ml build model algorithm scientist learning | 2.25 |
| Troubleshooting | customer support service troubleshooting aws help technical engineer application amazon | 2.19 |
| Application Development | application develop process system software programming platform qualify language code | 2.11 |
| Data Streaming | data processing spark hadoop kafka real-time frameworks streaming apache storm | 1.94 |
| Testing | test testing code quality design unit automation etl qa agile | 1.94 |
| Data Visualization | data visual report visualization graph view time tableau chart infogram design | 1.78 |
| Azure Cloud | client azure solution consult delivery consultant databrick professional microsoft sql | 1.77 |
| Team Working | teamwork lead member join independently collaborate member contact solidarity | 1.76 |
| Decision-making | data business work decision-making core key judgment successful system | 1.63 |
Distribution of skills by expertise roles.
| Topics (Skills) | Developer | Engineer | Architect | Analyst | Manager | Administrator | Consultant | Rate |
|---|---|---|---|---|---|---|---|---|
| Big Data Processing | 1.90 | 3.26 | 0.90 | 0.29 | 0.26 | 0.10 | 0.11 | 6.82 |
| Big Data Tools | 2.93 | 1.88 | 0.50 | 0.09 | 0.22 | 0.16 | 0.15 | 5.93 |
| Communication Skills | 1.93 | 1.69 | 0.46 | 0.29 | 0.30 | 0.12 | 0.22 | 5.02 |
| Remote Development | 2.17 | 1.55 | 0.45 | 0.12 | 0.17 | 0.11 | 0.11 | 4.69 |
| Big Data Architecture | 1.01 | 1.46 | 1.42 | 0.17 | 0.23 | 0.17 | 0.18 | 4.64 |
| Programming Languages | 2.22 | 1.43 | 0.30 | 0.16 | 0.12 | 0.08 | 0.07 | 4.38 |
| Agile Development | 1.88 | 1.43 | 0.28 | 0.19 | 0.14 | 0.07 | 0.05 | 4.03 |
| Information Security | 2.01 | 0.88 | 0.28 | 0.16 | 0.11 | 0.12 | 0.19 | 3.75 |
| Project Management | 0.92 | 1.00 | 0.35 | 0.26 | 0.55 | 0.06 | 0.23 | 3.38 |
| Scalable Systems | 1.20 | 1.37 | 0.20 | 0.23 | 0.17 | 0.05 | 0.07 | 3.28 |
| Amazon EMR | 1.86 | 0.47 | 0.41 | 0.19 | 0.06 | 0.01 | 0.22 | 3.23 |
| Data Analytics | 0.74 | 1.56 | 0.27 | 0.39 | 0.10 | 0.03 | 0.07 | 3.16 |
| Business Applications | 1.17 | 1.00 | 0.29 | 0.17 | 0.24 | 0.07 | 0.15 | 3.09 |
| Analytical Skills | 0.61 | 0.27 | 0.35 | 1.15 | 0.26 | 0.23 | 0.17 | 3.05 |
| Business Analytics | 0.65 | 0.36 | 0.47 | 1.04 | 0.30 | 0.04 | 0.16 | 3.01 |
| Google Cloud | 0.81 | 1.28 | 0.53 | 0.10 | 0.08 | 0.05 | 0.14 | 2.99 |
| Technical Knowledge | 1.08 | 1.15 | 0.26 | 0.20 | 0.12 | 0.07 | 0.10 | 2.98 |
| Database | 1.17 | 1.22 | 0.27 | 0.08 | 0.09 | 0.05 | 0.05 | 2.93 |
| Bachelor Degree | 0.81 | 1.19 | 0.19 | 0.27 | 0.13 | 0.08 | 0.07 | 2.73 |
| AWS Data Services | 0.81 | 1.20 | 0.28 | 0.07 | 0.09 | 0.04 | 0.04 | 2.53 |
| Distributed Systems | 0.70 | 0.97 | 0.16 | 0.17 | 0.14 | 0.12 | 0.13 | 2.39 |
| Data Warehousing | 0.43 | 1.03 | 0.26 | 0.05 | 0.06 | 0.47 | 0.02 | 2.33 |
| Hadoop Ecosystem | 0.51 | 0.83 | 0.25 | 0.15 | 0.05 | 0.46 | 0.04 | 2.28 |
| Machine Learning | 0.71 | 0.50 | 0.08 | 0.76 | 0.05 | 0.02 | 0.13 | 2.25 |
| Troubleshooting | 0.34 | 0.96 | 0.14 | 0.30 | 0.07 | 0.34 | 0.05 | 2.19 |
| Application Development | 1.11 | 0.64 | 0.12 | 0.09 | 0.08 | 0.04 | 0.04 | 2.11 |
| Data Streaming | 0.34 | 0.96 | 0.34 | 0.10 | 0.07 | 0.08 | 0.05 | 1.94 |
| Testing | 0.94 | 0.69 | 0.07 | 0.08 | 0.09 | 0.03 | 0.04 | 1.94 |
| Data Visualization | 0.52 | 0.12 | 0.06 | 0.87 | 0.17 | 0.02 | 0.02 | 1.78 |
| Azure Cloud | 0.40 | 0.65 | 0.33 | 0.10 | 0.15 | 0.03 | 0.11 | 1.77 |
| Team Working | 0.56 | 0.85 | 0.09 | 0.11 | 0.07 | 0.05 | 0.03 | 1.76 |
| Decision-making | 0.24 | 0.12 | 0.14 | 0.65 | 0.42 | 0.03 | 0.02 | 1.63 |
Ranking of the top 10 skills for each of the expertise roles.
| Topics (Skills) | Developer | Engineer | Architect | Analyst | Manager | Administrator | Consultant | Total |
|---|---|---|---|---|---|---|---|---|
| Communication Skills | 5 | 3 | 6 | 9 | 3 | 7 | 3 | 7 |
| Big Data Tools | 1 | 2 | 4 | 9 | 6 | 9 | 6 | |
| Analytical Skills | 10 | 1 | 5 | 4 | 6 | 5 | ||
| Big Data Architecture | 6 | 1 | 8 | 5 | 5 | 5 | ||
| Big Data Processing | 6 | 1 | 2 | 8 | 6 | 5 | ||
| Remote Development | 3 | 5 | 7 | 10 | 4 | |||
| Business Analytics | 5 | 2 | 4 | 7 | 4 | |||
| Business Applications | 10 | 7 | 8 | 3 | ||||
| Google Cloud | 10 | 3 | 10 | 3 | ||||
| Amazon EMR | 8 | 8 | 2 | 3 | ||||
| Information Security | 4 | 8 | 4 | 3 | ||||
| Project Management | 9 | 1 | 1 | 3 | ||||
| Scalable Systems | 9 | 9 | 2 | |||||
| Agile Development | 7 | 8 | 2 | |||||
| Data Visualization | 3 | 10 | 2 | |||||
| Troubleshooting | 7 | 3 | 2 | |||||
| Data Analytics | 4 | 6 | 2 | |||||
| Programming Languages | 2 | 7 | 2 | |||||
| Decision-making | 5 | 2 | 2 | |||||
| Bachelor Degree | 10 | 1 | ||||||
| Distributed Systems | 9 | 1 | ||||||
| Machine Learning | 4 | 1 | ||||||
| Hadoop Ecosystem | 2 | 1 | ||||||
| Data Warehousing | 1 | 1 |
1. Chen, H.; Chiang, R.H.L.; Storey, V.C. Business Intelligence and Analytics: From Big Data to Big Impact. Mis Q.; 2012; 36, pp. 1165-1188. [DOI: https://dx.doi.org/10.2307/41703503]
2. Philip Chen, C.L.; Zhang, C.-Y. Data-Intensive Applications, Challenges, Techniques and Technologies: A Survey on Big Data. Inf. Sci.; 2014; 275, pp. 314-347. [DOI: https://dx.doi.org/10.1016/j.ins.2014.01.015]
3. Debortoli, S.; Müller, O.; Vom Brocke, J. Comparing Business Intelligence and Big Data Skills: A Text Mining Study Using Job Advertisements. Bus. Inf. Syst. Eng.; 2014; 6, pp. 289-300. [DOI: https://dx.doi.org/10.1007/s12599-014-0344-2]
4. De Mauro, A.; Greco, M.; Grimaldi, M.; Ritala, P. Human Resources for Big Data Professions: A Systematic Classification of Job Roles and Required Skill Sets. Inf. Process. Manag.; 2018; 54, pp. 807-817. [DOI: https://dx.doi.org/10.1016/j.ipm.2017.05.004]
5. Gurcan, F.; Cagiltay, N.E. Big Data Software Engineering: Analysis of Knowledge Domains and Skill Sets Using LDA-Based Topic Modeling. IEEE Access; 2019; 7, pp. 82541-82552. [DOI: https://dx.doi.org/10.1109/ACCESS.2019.2924075]
6. Halwani, M.A.; Amirkiaee, S.Y.; Evangelopoulos, N.; Prybutok, V. Job Qualifications Study for Data Science and Big Data Professions. Inf. Technol. People; 2022; 35, pp. 510-525. [DOI: https://dx.doi.org/10.1108/ITP-04-2020-0201]
7. Kantardzic, M. Data Mining: Concepts, Models, Methods, and Algorithms; 2nd ed. Institute of Electrical and Electronics Engineers: Piscataway, NJ, USA, 2011; ISBN 9780470890455 (ISBN)
8. Gardiner, A.; Aasheim, C.; Rutner, P.; Williams, S. Skill Requirements in Big Data: A Content Analysis of Job Advertisements. J. Comput. Inf. Syst.; 2018; 58, pp. 374-384. [DOI: https://dx.doi.org/10.1080/08874417.2017.1289354]
9. Deb, D.; Fuad, M. Integrating Big Data and Cloud Computing Topics into the Computing Curricula: A Modular Approach. J. Parallel Distrib. Comput.; 2021; 157, pp. 303-315. [DOI: https://dx.doi.org/10.1016/j.jpdc.2021.07.012]
10. Persaud, A. Key Competencies for Big Data Analytics Professions: A Multimethod Study. Inf. Technol. People; 2021; 34, pp. 178-203. [DOI: https://dx.doi.org/10.1108/ITP-06-2019-0290]
11. Verma, A.; Yurov, K.M.; Lane, P.L.; Yurova, Y.V. An Investigation of Skill Requirements for Business and Data Analytics Positions: A Content Analysis of Job Advertisements. J. Educ. Bus.; 2019; 94, pp. 243-250. [DOI: https://dx.doi.org/10.1080/08832323.2018.1520685]
12. Miller, S. Collaborative Approaches Needed to Close the Big Data Skills Gap. J. Organ. Des.; 2014; 3, 26. [DOI: https://dx.doi.org/10.7146/jod.9823]
13. Gandomi, A.; Haider, M. Beyond the Hype: Big Data Concepts, Methods, and Analytics. Int. J. Inf. Manag.; 2015; 35, pp. 137-144. [DOI: https://dx.doi.org/10.1016/j.ijinfomgt.2014.10.007]
14. Debao, D.; Yinxia, M.; Min, Z. Analysis of Big Data Job Requirements Based on K-Means Text Clustering in China. PLoS ONE; 2021; 16, e0255419. [DOI: https://dx.doi.org/10.1371/journal.pone.0255419]
15. Najafabadi, M.M.; Villanustre, F.; Khoshgoftaar, T.M.; Seliya, N.; Wald, R.; Muharemagic, E. Deep Learning Applications and Challenges in Big Data Analytics. J. Big Data; 2015; 2, 1. [DOI: https://dx.doi.org/10.1186/s40537-014-0007-7]
16. Radovilsky, Z.; Hegde, V.; Acharya, A.; Uma, U. Skills Requirements of Business Data Analytics and Data Science Jobs: A Comparative Analysis. J. Supply Chain Oper. Manag.; 2018; 16, pp. 82-101.
17. Gurcan, F. Extraction of Core Competencies for Big Data: Implications for Competency-Based Engineering Education. Int. J. Eng. Educ.; 2019; 35, pp. 1110-1115.
18. Indeed. Indeed Job Search. Available online: https://www.indeed.com/ (accessed on 16 January 2024).
19. Ozyurt, O.; Gurcan, F.; Dalveren, G.G.M.; Derawi, M. Career in Cloud Computing: Exploratory Analysis of In-Demand Competency Areas and Skill Sets. Appl. Sci.; 2022; 12, 9787. [DOI: https://dx.doi.org/10.3390/app12199787]
20. Montandon, J.E.; Politowski, C.; Silva, L.L.; Valente, M.T.; Petrillo, F.; Guéhéneuc, Y.G. What Skills Do IT Companies Look for in New Developers? A Study with Stack Overflow Jobs. Inf. Softw. Technol.; 2021; 129, 106429. [DOI: https://dx.doi.org/10.1016/j.infsof.2020.106429]
21. Gurcan, F. Major Research Topics in Big Data: A Literature Analysis from 2013 to 2017 Using Probabilistic Topic Models. Proceedings of the 2018 International Conference on Artificial Intelligence and Data Processing, IDAP 2018; Malatya, Turkey, 28–30 September 2018; pp. 1-4.
22. Řehůřek, R.; Sojka, P. Gensim—Statistical Semantics in Python; Masaryk University: Brno, Czech Republic, 2011; Volume 6611.
23. Ningrum, P.K.; Pansombut, T.; Ueranantasun, A. Text Mining of Online Job Advertisements to Identify Direct Discrimination during Job Hunting Process: A Case Study in Indonesia. PLoS ONE; 2020; 15, e0233746. [DOI: https://dx.doi.org/10.1371/journal.pone.0233746]
24. Uysal, A.K.; Gunal, S. The Impact of Preprocessing on Text Classification. Inf. Process. Manag.; 2014; 50, pp. 104-112. [DOI: https://dx.doi.org/10.1016/j.ipm.2013.08.006]
25. Murakami, R.; Chakraborty, B. Investigating the Efficient Use of Word Embedding with Neural-Topic Models for Interpretable Topics from Short Texts. Sensors; 2022; 22, 852. [DOI: https://dx.doi.org/10.3390/s22030852] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/35161598]
26. Calanca, F.; Sayfullina, L.; Minkus, L.; Wagner, C.; Malmi, E. Responsible Team Players Wanted: An Analysis of Soft Skill Requirements in Job Advertisements. EPJ Data Sci.; 2019; 8, 13. [DOI: https://dx.doi.org/10.1140/epjds/s13688-019-0190-z]
27. Blei, D.M. Probabilistic Topic Models. Commun. ACM; 2012; 55, pp. 77-84. [DOI: https://dx.doi.org/10.1145/2133806.2133826]
28. Subakti, A.; Murfi, H.; Hariadi, N. The Performance of BERT as Data Representation of Text Clustering. J. Big Data; 2022; 9, 15. [DOI: https://dx.doi.org/10.1186/s40537-022-00564-9]
29. Egger, R.; Yu, J. A Topic Modeling Comparison Between LDA, NMF, Top2Vec, and BERTopic to Demystify Twitter Posts. Front. Sociol.; 2022; 7, 886498. [DOI: https://dx.doi.org/10.3389/fsoc.2022.886498]
30. Blei, D.M.; Ng, A.Y.; Jordan, M.I. Latent Dirichlet Allocation. J. Mach. Learn. Res.; 2003; 3, pp. 993-1022. [DOI: https://dx.doi.org/10.1017/9781009218245.012]
31. Blei, D.M.; Lafferty, J.D. Correction: A Correlated Topic Model of Science. Ann. Appl. Stat.; 2007; 1, 634. [DOI: https://dx.doi.org/10.1214/07-AOAS136]
32. Gurcan, F.; Erdogdu, F.; Cagiltay, N.E.; Cagiltay, K. Student Engagement Research Trends of Past 10 Years: A Machine Learning-Based Analysis of 42,000 Research Articles. Educ. Inf. Technol.; 2023; 28, pp. 15067-15091. [DOI: https://dx.doi.org/10.1007/s10639-023-11803-8]
33. Alibasic, A.; Upadhyay, H.; Simsekler, M.C.E.; Kurfess, T.; Woon, W.L.; Omar, M.A. Evaluation of the Trends in Jobs and Skill-Sets Using Data Analytics: A Case Study. J. Big Data; 2022; 9, 32. [DOI: https://dx.doi.org/10.1186/s40537-022-00576-5]
34. Mimno, D.; Wallach, H.M.; Talley, E.; Leenders, M.; McCallum, A. Optimizing Semantic Coherence in Topic Models. Proceedings of the EMNLP 2011—Conference on Empirical Methods in Natural Language Processing; Edinburgh, UK, 27–29 July 2011.
35. Katsanos, C.; Avouris, N.; Stamelos, I.; Tselios, N.; Demetriadis, S.; Angelis, L. Cross-Study Reliability of the Open Card Sorting Method. Proceedings of the Conference on Human Factors in Computing Systems—Proceedings; Glasgow, Scotland, 4–9 May 2019.
36. Han, F.; Ren, J. Analyzing Big Data Professionals: Cultivating Holistic Skills Through University Education and Market Demands. IEEE Access; 2024; 12, pp. 23568-23577. [DOI: https://dx.doi.org/10.1109/ACCESS.2024.3363876]
37. Kılınç, M.; Aydın, C.; Tarhan, Ç. Do Machine Learning and Business Analytics Approaches Answer the Question of ‘Will Your Kickstarter Project Be Successful?. Istanbul Bus. Res.; 2021; 50, pp. 255-274. [DOI: https://dx.doi.org/10.26650/ibr.2021.50.0117]
38. Gurcan, F. What Are Developers Talking about Information Security? A Large-Scale Study Using Semantic Analysis of Q&A Posts. PeerJ Comput. Sci.; 2024; 10, e1954. [DOI: https://dx.doi.org/10.7717/peerj-cs.1954]
39. Bonesso, S.; Bruni, E.; Gerli, F. How Big Data Creates New Job Opportunities: Skill Profiles of Emerging Professional Roles. Behavioral Competencies of Digital Professionals; Springer: Berlin/Heidelberg, Germany, 2020; pp. 21-39.
40. Wowczko, I.A. Skills and Vacancy Analysis with Data Mining Techniques. Informatics; 2015; 2, pp. 31-49. [DOI: https://dx.doi.org/10.3390/informatics2040031]
41. Karakolis, E.; Kapsalis, P.; Skalidakis, S.; Kontzinos, C.; Kokkinakos, P.; Markaki, O.; Askounis, D. Bridging the Gap between Technological Education and Job Market Requirements through Data Analytics and Decision Support Services. Appl. Sci.; 2022; 12, 7139. [DOI: https://dx.doi.org/10.3390/app12147139]
42. Yang, C.; Huang, Q.; Li, Z.; Liu, K.; Hu, F. Big Data and Cloud Computing: Innovation Opportunities and Challenges. Int. J. Digit. Earth; 2017; 10, pp. 13-53. [DOI: https://dx.doi.org/10.1080/17538947.2016.1239771]
43. Boselli, R.; Cesarini, M.; Mercorio, F.; Mezzanzanica, M. Classifying Online Job Advertisements through Machine Learning. Futur. Gener. Comput. Syst.; 2018; 2, pp. 31-49. [DOI: https://dx.doi.org/10.1016/j.future.2018.03.035]
44. Gurcan, F. What Issues Are Data Scientists Talking about? Identification of Current Data Science Issues Using Semantic Content Analysis of Q&A Communities. PeerJ Comput. Sci.; 2023; 9, e1361. [DOI: https://dx.doi.org/10.7717/peerj-cs.1361]
45. Aljohani, N.R.; Aslam, M.A.; Khadidos, A.O.; Hassan, S.U. A Methodological Framework to Predict Future Market Needs for Sustainable Skills Management Using AI and Big Data Technologies. Appl. Sci.; 2022; 12, 6898. [DOI: https://dx.doi.org/10.3390/app12146898]
46. Moreno, A.M.; Sanchez-Segura, M.I.; Medina-Dominguez, F.; Carvajal, L. Balancing Software Engineering Education and Industrial Needs. J. Syst. Softw.; 2012; 85, pp. 1607-1620. [DOI: https://dx.doi.org/10.1016/j.jss.2012.01.060]
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Details
; Gudek Beyza 1
; Menekse Dalveren Gonca Gokce 2
; Derawi Mohammad 3 1 Department of Management Information Systems, Faculty of Economics and Administrative Sciences, Karadeniz Technical University, Trabzon 61080, Turkey
2 Department of Computer Engineering, Izmir Bakircay University, Izmir 35665, Turkey
3 Department of Electronic Systems, Faculty of Information Technology and Electrical Engineering, Norwegian University of Science and Technology, 7034 Gjøvik, Norway