Oral Exams Improve Engineering Student Performance, Motivation

Article Content

The use of oral exams in engineering courses increased students’ motivation to learn, particularly in first-generation college students, according to preliminary findings from a study of undergraduate students at the University of California San Diego. The oral exams also improved students' understanding of the content, evidenced by outcomes on subsequent written tests. 

Oral exams are not commonly used in engineering courses at public universities in the United States, in part due to concerns surrounding cost, validity and scalability. The onset of the COVID-19 pandemic and abrupt transition to remote learning provided a unique opportunity for engineering and computer science faculty at UC San Diego to study the effectiveness of oral exams, and find ways to scale them up in large engineering courses. 

With funding from the National Science Foundation, a team of researchers across campus set out to study the use of oral exams to improve the conceptual mastery of engineering students in large engineering courses. 

In initial research papers from this three-year project, published at the ASEE conference in 2022, the researchers found that the inclusion of oral exams not only improved some students’ scores on traditional written exams, but also increased student motivation to succeed in the course. 

“We started way back in early 2020, as we were transitioning to remote learning. There were many challenges with remote learning and assessment– for example students didn't turn on their cameras during online lectures, they felt socially isolated, and there were also concerns about academic integrity for remote exams. All of these made oral exams an appealing option,” said Huihui Qi, an assistant teaching professor in the Department of Mechanical and Aerospace engineering at UC San Diego, and the principal investigator of the NSF grant. 

Qi and a team of faculty from the departments of Mechanical and Aerospace Engineering, Electrical and Computer Engineering, Computer Science and Engineering, and Psychology, as well as education researchers from the UC San Diego Teaching and Learning Commons, set out to quantify the results of using oral exams in large engineering and computer science courses. Though prior work had shown that this type of exam provided instructors with more insight into students’ understanding, how it would impact student success remained to be seen. 

In one of the early published studies from this NSF project, 560 engineering students in six classes took 15-minute oral exams and filled out a survey about their experience. In one of the classes, a comparison trial was performed with students randomly assigned to one of three groups. After the first written midterm, one group had an oral exam with the instructor, one group had an oral exam with the Teaching Assistant (TA), and the last group did not have an oral exam. In the second written midterm the group with the instructor-proxied oral exam increased their grade by 14%, the group with the TA increased by 3%, and the group with no oral exam had a negligible change in performance. 

In addition to this measured increase in student success, 70% of the students who responded to a survey either Strongly Agreed or Agreed that oral exams increased their motivation to learn. This figure was particularly strong for first-generation students, with 78% indicating that oral exams increased their motivation to learn, compared to 66% of non-first generation students. 

“We’ve gotten some good and interesting preliminary findings so far,” said Qi. “There are many benefits to oral exams and we are already starting to see it in our outcomes. From increased student written test scores, to students feeling more comfortable attending office hours and seeking help from the instructional team, we think oral exams show great promise in increasing student success and learning, even in large engineering courses.”

Conditional vs. procedural learning

One potential reason for the increase in students’ conceptual understanding is that oral exams, due to their dialogical nature, allow for follow-up questions about students’ problem solving and decision making processes. Qi said that oral exams help instructors gauge a student’s conditional knowledge– the ability to apply certain knowledge to a specific situation and explain why– instead of just the procedural knowledge– plug-and-chug equations– that is more easily tested in written exams. 

“Oral exams provide more capacity for students to elaborate their thought process and the problem solving strategy, not just rush right into the problem solving process and outcome,” Qi said. “At times, some students are too eager to rush to solve a problem without too much strategy planning. That's a key difference between an expert and beginner learner: an expert will put more thought into the process and planning than the beginning learner, and usually can do a deeper and wider knowledge search.”

While working toward conditional knowledge is key, having a firm grasp on the procedural knowledge is also crucial for engineering and computer science students, and is one reason why oral exams in this study were not replacing the written exams that can be better gauges of a student’s procedural knowledge, but rather were used in addition as a separate metric of learning. 

“I would say oral exams are much more rigorous even than a written exam, because for a written exam you can prepare almost perfectly using memorization and notes (granted you spend the time) to the point where you can pass the test on exam day with little effort,” one student in a course that implemented oral exams wrote in an anonymous survey. “I believe that it would be impossible to achieve something similar for an oral exam, as base memorization will only get you so far, as the instructor can ask you pretty much any question (within reason) about the subject matter; so it requires a deeper understanding than rote memorization.”

Scaling up oral exams

Though these initial results are promising, there is still work to be done to be able to implement oral assessments in engineering more broadly. The researchers are now working to understand how to best design and prepare students for oral exams to maximize their potential benefits in improving academic performance, increasing student self-efficacy and learning motivation. They’re working with education specialists in the Teaching and Learning Commons at UC San Diego to devise strategies to more effectively train graduate student TAs to administer these exams, since that will be a key aspect of scaling oral exams to more large classes. 

“It’s important that TAs not only have content knowledge when administering oral exams, but also knowledge about how to administer the exams from an equity-minded approach,” said Carolyn Sandoval, director of the Engaged Teaching Hub, associate director of the Teaching + Learning Commons at UC San Diego, and co-investigator on the NSF grant. “This includes training on how to mitigate the potential for and impact of implicit bias in the exam process. Providing training on the potential benefits and challenges of using oral exams to promote student learning is also essential to ensuring that TAs are intentional and supportive during their interactions with students.”

Future studies will also involve research into scoring rubrics to increase the consistency,validity and reliability of oral exam scores across different assessors. 

“We are very excited for this in-progress research,” write the researchers. “An early objective was to determine if there are any measurable benefits to oral exams, and this paper illustrated benefits of both increases in written exam grades following an oral exam, and a large percentage of students indicating that oral exams increased their motivation to learn. Ultimate outcomes of this study need to recognize that administering oral exams is labor intensive. However, by committing to this initiative of administering oral exams to a large number of students, we are beginning to quantify the benefits of oral exams, so that appropriate tradeoffs can be made.”

—------

NSF award for Improving the Conceptual Mastery of Engineering Students in High Enrollment Engineering Courses through Oral Exams: principal investigator is Huihui Qi, with co-PIs Nathan Delson, Curt Schurgers, and Marko Lubarda from the Jacobs School of Engineering, and Carolyn Sandoval from the Teaching and Learning Commons. The rest of our research project team members are: Saharnaz Baghdadchi, Joanna Boval, Xuan Gedney, Maziar Ghazinejad, Minju Kim, Leah Klement, Leo Liu, Mia Minnes, Alex Phan, Celeste Pilegard, Josephine Relaford-Doyle and Tony Wang. Learn more at the project website: https://oralexamucsd.wixsite.com/ucsdoralexam