Content area
In the academic year 2020, the Naval Postgraduate School Systems Engineering Department began requiring students in the Systems Engineering Management for Acquisition program to take a course focused on engineering research. The evaluation criteria include the level that the students have demonstrated mastery of the discipline, rigorous application of research and analysis methods, critical thinking, and publication worthiness of the document. A case analysis of students in the Systems Engineering Management for Acquisition (522) program at the Naval Postgraduate School (NPS) provides evidence of how a course in engineering research can improve students' capstone research efforts. [...]the success of engineering research is based on the meticulous nature of the scientific process embedded in general research.
Abstract: This paper highlights the impact of implementing a course about engineering research on the quality of capstone projects by master's students with limited science, technology, engineering, and math background. In the academic year 2020, the Naval Postgraduate School Systems Engineering Department began requiring students in the Systems Engineering Management for Acquisition program to take a course focused on engineering research. The graduation requirement for students in this program is a capstone team project and report. Prior to implementing an engineering course in the academic matrix, faculty assessments of the final reports identified shortcomings, specifically in the research approach, analysis, value proposition, and overall scholarship. The Department responded by developing a course in engineering research with an emphasis on applying engineering reasoning and critical thinking within the systems engineering design process. After the student's completion and submission of a final capstone report to the thesis processing office, faculty advisors provide a subjective assessment of the work. The evaluation criteria include the level that the students have demonstrated mastery of the discipline, rigorous application of research and analysis methods, critical thinking, and publication worthiness of the document. Using five years of data, the author conducted a comparative analysis of evaluations from project advisors before and after the course was required. The results of the analysis show significant improvements in the areas that were lacking in student work before the introduction of the engineering research course. Additionally, the author discusses specific elements of the course that indicate their contributions to improvements in the research approach and the overall quality of the work and final products. This case analysis asks the engineering community to consider elements of the engineering research course in future investigations or as a guide to develop a similar program of instruction.
Keywords: Engineering Research, Engineering Reasoning, Systems Engineering, System Design, Case Analysis
1. Background
It is acceptable to continue with convergent thinking and iterative processes that employ customary research methods as long as systems remain relatively simple. However, systems continue to grow in their dimensionality. The surge in technological advancements, greater depth of interactions between operators and the system of interest, as well as the exponential necessity for developing systems of systems to address increasingly complex issues demand novel approaches to augment conventional research and design practices. Engineering research, with an emphasis on combining engineering reasoning and systems engineering design, is a course that assists students to become better critical thinkers and solvers of complex engineering problems (Hernandez 2018). Before 2020, the Systems Engineering Department had no research methods course in any of its curriculum tracks.
A case analysis of students in the Systems Engineering Management for Acquisition (522) program at the Naval Postgraduate School (NPS) provides evidence of how a course in engineering research can improve students' capstone research efforts. Students in the 522 program have little engineering or mathematical background. Students in this program come from the U.S. Army Acquisition workforce that populates the program management offices for the development of Army systems. As such, approximately 22% of students have a degree in science, technology, engineering, and math (STEM). The rest of the 522 students have non-STEM degrees, with a majority having undergraduate degrees in business administration, political science, and criminal justice (Hernandez 2020).
Senior leaders in Army Acquisition have made it imperative for the workforce to have the acumen and technical vocabulary to discuss engineering matters with the contractors and manufacturers with which they must negotiate. The recent past has shown headlines of different cases of systems engineering failures that can be traced to poor practices that program offices should have identified (Schwartz 2024). Loss of life, equipment, time, and resources justify the move toward educating the work force in engineering fundamentals and their linkages to program management. Systems Engineering Management is the discipline of managing the activities, personnel, and resources of many types of engineering fields to create a new system. Overall, 522 students have limited experience in the processes of scientific inquiry. These conditions tend to negatively affect student projects. The final products lack elements of scholarship that faculty members and the engineering community demand. The Systems Engineering Department responded by developing and requiring 522 students to take the course, Methods in Engineering Research, which is referenced as SE3077 for the remainder of this paper.
1.1 Engineering Research
Engineering research follows the norms of scientific discovery. This paper does not claim that engineering research is more rigorous than scientific research. In fact, the success of engineering research is based on the meticulous nature of the scientific process embedded in general research. The distinction in engineering research is that new knowledge serves the primary purpose of developing functional design solutions to an engineering problem, whereas general research can be performed with discovery as an end in itself (Creswell and Creswell 2018). Another important difference between engineering and general research is time. Engineering research requires an engineering solution in a finite amount of time while general research may not necessarily have a definitive deadline.
Research within the engineering community is decidedly quantitative with good reason. Evidence based decisions are critical to system design solutions. Experimentation and numerical analysis are primary methods (Nagabhushan 2016). Generally, a research plan will combine several approaches that center on numeric techniques (Thiel 2014). However, as systems increase in complexity, the greater the need for different methods, including qualitative and multimethod approaches (Johnson and Hernandez 2016). Engineering accepts the need for diverse techniques with an end state of solving the engineering problem.
The engineering problem is based on three elements: design knowledge - K, design specifications - 5, and design variables - V (Summers 2005). Engineering research is a systematic exploration of these elements during the Systems Engineering design process. The language that engineers use is based on rational thinking (Paul, Niewoehner, and Elder 2013). Charles Peirce distinguished three classes of reasoning: deductive, inductive, and abductive (Nozawa, 2008). Engineering thinking during research is a deliberate effort to apply the appropriate logic type to examine an engineering element in any phase of the design. Whitcomb and Hernandez (2017) summarize each type of logic in terms of V, K, and S (Table 1).
The objective for applying а specific type of engineering logic is to obtain the missing design element in the relevant stage of the design process. For instance, deductive thinking focuses on the study of design variables and the design team's current knowledge to develop design specifications. Similarly, inductive and abductive reasoning surface in other phases of engineering design. The aim of tying engineering logic with systems engineering design is to assist students in training their thinking during an often complex, iterative process which is discussed in the following section and summarized in Table 2. A formal course to teach the connection between engineering research and engineering logic can be valuable to the engineer's design approach. This paper explains how such a course has improved the research efforts of 522 students.
1.2 Engineering Logic
Engineers do not have all the relevant information at the beginning of the design process. Research is necessary to fill the gaps in the known data (Paul, Niewoehner, and Elder 2013). Peircean Science of Inquiry eventually evolves into the scientific method and subsequently helps frame engineering research (Nozawa 2008). The following explains the different types of reasoning and their cumulative application.
1.2.1 Overview of Deductive, Inductive, and Abductive Thinking
Deductive reasoning applies when the design variables and design knowledge about the engineering problem are known. This is usually the case at the beginning of the design process. As a simple engineering example, consider the need for a beverage container. Design variables can include the amount of liquid that the container holds, weight of the container itself, or the frequency that it will be used. Knowledge about the problem consists of the density of liquids and the properties of potential material for the container. Other information about the engineering problem includes rules about the relationship between the strength of the material and its ability to hold the weight and density of a particular liquid. Given these design elements, deductive reasoning analytically determines the specifications of the container. The designer would make conclusions (specifications) about the maximum size of the container, the minimum amount of liquid it should hold, its durability, or the amount of fatigue (weakening of material) incurred for a given applied load. These can measure the suitability of the design.
Inductive reasoning is a process to derive design knowledge. Knowledge is a set of facts about the system or similar systems, and the system designer's level of experience and study about the system. During the design process, it may happen that the accumulated knowledge about the system is insufficient to continue with its design. This situation prompts the design team to obtain additional information through inductive thinking. At this point, the engineer would have a set of design variables and design specifications. However, the information about causal relationships between variables or even the concept of operations for the system may be vague. Experimentation is a natural technique in inductive thinking. Test runs keep certain design variables constant while changing the values of other variables and measuring the impact on the design specifications. Analysis develops a mathematical expression for the relationship. Continuing with the beverage container example, the engineer would hold the amount of fluid constant, vary the container material, and subsequently measure the fatigue that the container sustains during the experiment. The resulting relationship between material properties and fatigue informs the engineering problem.
When design knowledge and specifications are given, abductive thinking aims to discover new design variables, or to re-examine design variables that the engineer had previously considered as unimportant to the problem. Again, referencing the beverage container example suggests how abductive thinking could take place. Material properties of the container have been established as an important factor to the container's level of fatigue. However, it is not certain that only the material properties contribute to the fatigue that the container may exhibit. For instance, the conditions of the experiment may involve other factors such as the temperature of the liquid, the air temperature surrounding the container, or the speed at which the liquid enters. Examination of these new factors in an experiment may prove significant to the level of fatigue that occurs. In this example, not only does this approach identify new design variables (temperature, speed), but it also alters the context for using the container-system knowledge.
1.2.1 Retroduction, an Iterative Application of Logic - the Retroductive Design Process
Retroduction is not a class of logic. It is a process that applies the different types of engineering reasoning in conjunction with the iterative cycle of engineering design. Retroduction is deliberately cyclic. Table 2 is just one instantiation and depends on the step in the design process that the engineer is addressing. Within the table, the cycle is linearized to associate it with the engineering design process (Summers, 2005). The last column of Table 2 indicates potential research methods that can be applied.
2. Course Development - Combining Engineering Research and Engineering Logic
Engineering logic and engineering research are the foundations for developing SE3077 (Hernandez 2018). The course is intended to prepare engineers on how to investigate complex engineering problems. Incorporating engineering reasoning within the systems engineering design process provides context and motivation for engineering students to pursue different methods of inquiry for an engineering solution.
Central to the course is an actual topic and problem that each student or student team must resolve for a primary stakeholder. The instructor uses the engineering problem to show how course concepts and techniques manifest themselves, and to which engineering students can relate. The interaction between the student, stakeholder, and advisory team creates a realistic environment that adds a dimension for selecting a research design and communicating, as well as negotiating on the road to a problem solution.
2.1 Course Textbook and Study Material
Selection of course textbooks considered student unfamiliarity with formal research and lack of understanding the importance of research in engineering sciences. For SE3077, the course textbooks include Research Methods for Engineers (Thiel 2016), Research Design (Creswell and Creswell 2018), and The Thinker's Guide to Engineering Reasoning (Paul, Niewohner, and Elder 2013).
The course provides additional information that expands the students' horizons about research methods. The material is meant to educate the students on the versatility of systems engineering and other techniques and the ability to integrate them to investigate difficult problems. The instructor selects articles that are relevant to course objectives. Invited lecturers are expert practitioners of specific methods of inquiry who present recent examples of their work. Discussions from these interactions go beyond textbook readings.
2.2 Overview of Course Modules
The course is an eleven-week program of instruction that consists of three major sections and thirty-three contact hours. The course includes fieldwork where students engage their advisory team and stakeholders. Means of evaluation are scheduled in major milestones in the instruction and will be discussed in the following section.
Part | (weeks 1 - 5) is an introduction to basic research techniques, engineering research, and the design process. Material from Wasson (2016), Blanchard and Fabrycky (2011), and Dym and Little (2000) provide basic systems engineering techniques. Discussion begins by defining course objectives, learning outcomes of the lessons. It reviews the systems engineering design process. A study of traditional research methods provides the student with basic techniques (Creswell and Creswell 2018). An important aspect of instruction includes problem definition and the literature review. This section of the course ends with a midterm examination.
Part Il (weeks 6 - 8) involves the different classes of logic and retroductive thinking and their application in the system design process (Summers 2005). A short history on engineering reason and logic begins this section of the program (Paul, Niewoehner, and Elder 2013; Nozawa 2008). Study modules for abductive, inductive, and deductive thinking are the primary focus in this part of the course. A discussion of the retroductive process shows the iterative application of the major classes of engineering logic. The discussion links design and engineering reasoning in terms of engineering elements.
Part Ill (weeks 9 - 11) consists of lessons for developing a research plan and proposal, post-research activities to include publishing scholarly articles, as well as thesis and report writing. Final course modules contain advanced discussion of quantitative and qualitative research techniques that follow steps in the design process, corresponding tasks or objectives of each design step, and the class of engineering reasoning that governs how to address the tasks. The students combine lessons and all past work to develop a research proposal.
2.3 Evaluating Achievement of Course Objectives
Grades are based on the students' demonstrated knowledge and facility with topics and concepts in the course. There are three graded assignments and a midterm exam in the course. There is no final exam. Grades are awarded in accordance to thresholds from the "Evaluating Student Work in Engineering" section of The Thinker's Guide to Engineering Reasoning (Paul, Niewoehner, and Elder 2013). The following discussion provides details about course work that helps the instructor assess student achievement of learning outcomes and objectives.
Textbook exercises reinforce daily lessons and key concepts in the course. They are the foundation for addressing graded requirements, midterm examination, and discussing assigned articles. Students attempt homework exercises prior to class. Unclear topics good discussion points during the class session. Students are encouraged to revisit the homework after class to ensure that they have a clear understanding of the concepts.
Assignment 1. Problem statement and summary development is a graded assignment. Research begins with clearly understanding a given problem. A researcher must be able to define the problem for which a research effort is applied. Each student will use information from their capstone or individual thesis topic and any material that their primary stakeholder and\or advisor has provided. The Ishikawa or Fishbone Diagram (Ishikawa 1974), initial readings or documents about the topic area, and their own critical thinking, enable each student to independently develop a 2-page problem statement and summary.
Assignment 2. The literature review is a cornerstone for research and will span the majority of the class (Creswell and Creswell 2018). Based on the topic and engineering problem statement and summary, students collaborate with their advisory team and primary project stakeholders to learn more about appropriate documents, articles, and other references to study the problem. For the purposes of the course, students will use up to thirty different sources for the literature review. Part one of the literature review begins with five to ten sources.
Midterm Examination. At the end of Week 6, students take an online midterm examination to demonstrate their understanding of the topics discussed from Weeks 1 - 6. The exam is a timed, individual effort, consisting of a range and different forms of questions that total 100 points. It is an open book examination. Use of the internet is strictly prohibited except access to the online exam.
Assignment 3. The final graded assignment for the course is a draft capstone or thesis proposal. The proposal is an end-of-course deliverable. Each capstone team develops a written proposal that incorporates ideas from engineering reasoning and the engineering design process. Teams work with their respective advisory team and stakeholders for this assignment. Days for fieldwork are part of the course agenda.
3. Advisor Team Scoring of Capstone Projects
The primary objective of SE3077 is to impart the necessary knowledge that will help the students in their future endeavors and improve their research efforts (Hernandez 2018). The aggregate effort in which the effectiveness of SE3077 can be measured is the capstone project and final report. The capstone faculty advisory team assesses the final report in terms of the following criteria (Table 3):
The most relevant of these evaluation areas, and that which concerned faculty before the introduction of SE3077, were criteria 1, 2, 3, 4 and 8. Criteria 5, 6, and 7 are highly dependent on the problems that are being sponsored and therefore beyond the students' control. They are not part of this case analysis. It may be argued that the results of criteria 1 through 4 would determine whether the resultant report was publishable. However, for this study, it is important to discuss how specific elements of SE3077 map onto these criteria.
4. Analysis of Advisor Scoring of Capstone Project Reports
The focus of this study is on students in the 522 program in the Systems Engineering Department.
4.1 Data Analysis
The 522 program began in the summer quarter of academic year 2018 with one cohort of 30 students that graduated in academic year 2020. Inclusion of SE3077 in the 522 academic matrix began after graduation of the first cohort. Capstone teams consist of four to six students. The assessment data are from 35 capstone research efforts involving nearly 180 students. Each project report is evaluated by each member of the associated advisory team which can number from two or three faculty members. The element of analysis is the evaluation score from each advisory team member. The Likert scores are treated numerically (Norman 2010). The number of evaluations of reports that were produced by students who had not taken SE3077 was 15. The number of evaluations of reports from students who had SE3077 in their academic program was 71. The following summary data shows the average scores for criteria 1, 2, 3, 4, and 8 prior to and after inclusion of SE3077 in the 522 academic matrix (Table 4).
Visual comparison of the data indicates that scores in the assessment areas are higher after inclusion of SE3077. The author analyzed these results using a one-sided t-test with a significance level of 0.10 and assuming unequal variances (Montgomery and Runger 2014). However, because the sample sizes are drastically different, the author used Welch's t-test, a variation of the traditional t-test (Hogg and Tanis 1997).
For each t-test, the null hypothesis is that the difference between the average scores is zero. The alternative hypothesis is that after inclusion of SE3077 the average score is higher. The results show that the differences between each pair (before and after) of averages are statistically significant. This empirical analysis of the data provides evidence that SE3077 is a factor in improving capstone research efforts.
4.2 Interpretation of Results and Trace to SE3077
The analytic evidence to show SE3077's value for improving student research is accompanied with a discussion of relevant course elements with the assessment criteria. Table 5 briefly explains the linkages.
5. Conclusions and Recommendations
Implementing a course about engineering research has proven to be an effective approach to improve the quality of capstone projects by engineering master's students with little STEM background or experience with formal research. Faculty assessments of capstone reports revealed weaknesses in the student's ability to properly apply the scientific method and analysis in their work. As a remedy, the academic department created SE3077, a course based on engineering research and engineering reasoning.
Features of SE3077 were specifically developed to address identified shortfalls in student products. The course revisited the engineering design methodology that is fundamental to the systems engineering program. Entwining research methods in the design process provided context for students to recognize the importance of research techniques and focus on specific engineering methods. The result was greater competence in the systems engineering discipline. Introducing basic and advanced research methods was a primary objective in SE3077. However, linking these methods in the deliberate application of a specific type of logic to derive information about an engineering element in the problem was a key objective. This combination increased the students' use of research techniques and critical thinking skills. Each was rigorously evaluated in the course. The aggregate measure of effectiveness of the course was the publication worthiness of the capstone report.
Five years of evaluation data from advisory team members was collected to compare capstone products from students after inclusion of SE3077 in the program of instruction with products before SE3077 was introduced. Analysis of the data showed that the improvements of products after SE3077 was added were statistically significant. In all five of the most relevant criteria, assessment scores increased.
The discussion in this paper argues for immersing engineering students to engineering research and engineering reasoning to equip them with skills to solve complex problems. The major features of SE3077 are specifically designed to increase problem solving abilities, logical implementation of research techniques in alignment with the engineering design process, and scholarly writing. The author offers SE3077 to the engineering community as a foundation for future scientific inquiry or as a guide to develop a similar program of instruction in engineering departments or short courses in commercial organizations.
Ethical Declaration
| declare that this manuscript is original work and has not been published elsewhere.
Al Declaration
| have not used any Al tools or technologies to prepare this manuscript.
References
Bates, S. (2008) "Research Methods for Engineering Technology", [online], San Jose State University, http://www.unhas.ac.id/rhiza/arsip/kuliah/Metode-Penelitian/REFERENCES/01 Introductory Lecture.pdf.
Blanchard, B.S. and Fabrycky, W.J. (2011) Systems Engineering and Analysis 5th Edition, Pearson Prentice Hall, Upper Saddle River.
Buede, D.M. and Miller, W.D. (2016) The Engineering Design of Systems, Models and Methods 3 Edition, John Wiley & Sons, Inc., Hoboken.
Creswell, J.W. and Creswell J.D. (2018) Research Design 5" Edition, Sage Publications, Inc., Thousand Oaks, CA.
Dym, C.L. and Little, P. (2000) Engineering Design: A Project-Based Introduction, John Wiley & Sons Inc., New York.
Hernandez, A.S. (2020) "Summary of Degrees that Applicants and Enrolled Students Possess Prior to Entering the 522 Program," Presentation to Program Sponsor, U.S. Naval Postgraduate School, Monterey, CA.
Hernandez, A.S. (2018) "An Engineering Reasoning-Based Course on Research Methodologies for Systems Engineers," Proceedings of the 17 European Conference on Research Methodology for Business and Management Studies, Rome, Italy, July 2018.
Hogg, R.V. and Tanis, E.A. (1997) Probability and Statistical Inference, Pearson Hall, Inc., Upper Saddle River.
Ishikawa, K. (1974) Guide to Quality Control. Translated by the Asian Productivity Organization, JUSE Press Ltd., Tokyo.
Johnson, B. and Hernandez, A.S. (2016) "Exploring Engineered Complex Adaptive Systems of Systems", Complex Adaptive Systems, Publication 6, Procedia Computer Science, Vol. 95C, pages 58-65, November.
Montgomery, D.C. and Runger, G.C. (2014), Applied Statistics and Probability of Engineers 6th Edition, John Wiley & Sons Inc., Hoboken.
Nagabhushan, P. (2016) "Research Methods for Engineering Educators: A Primer", [online], Australian National University, https://iuceedotcom.files.wordpress.com/2016/06/research-methods-for-engineering-educators.pdf.
Norman, G. (2010) 'Likert Scales, Levels of Measurement and the 'Laws' of Statistics", Advance in Health Science Education, 15:625-632. DOI 10.1007/s10459-010-9222-y.
Nozawa, E.T. (2008) "A Conceptual Systems View of Peircean Logic, Art of Reasoning, and Critical Thinking", Proceedings of the 2008 International Symposium on Collaborative Technologies and Systems, Irvine, CA, May.
Paul, R., Niewohner, R., and Elder, L. (2013) The Thinker's Guide to Engineering Reasoning 2"" Edition, Foundation for Critical Thinking Press, Tomales.
Summers, J.D. (2005) "Reasoning in Engineering Design", Proceedings of the ASEM IDETC/CIE 2005, pp 1-6, New Orleans, LA, September.
Schwartz, B. (2024) "Written in Blood: Case Studies of Systems Engineering Failure" https://www.engineeringforhumans.com/systems-engineering/written-in-blood-case-studies-of-systemsengineering-failure/.
Thiel, D.V. (2014) Research Methods for Engineers, Cambridge University Press, Cambridge.
Wasson, C.S. (2016) Systems Engineering Analysis, Design, and Development 2nd Edition, John Wiley & Sons, Hoboken.
Whitcomb, C. and Hernandez, A. (2017) "Engineering Reasoning in Set-Based Design", Proceedings of the 2017 American Society of Naval Engineers, Design Sciences Series: Set-Based Design, Washington, DC, September.
Copyright Academic Conferences International Limited 2025