|
ASSESSING TEACHING
METHODS FOR A COURSE IN NUMERICAL
METHODS Abstract Effectiveness of four instructional
delivery modalities – 1) Traditional lecture, 2) Web-enhanced lecture, 3)
Web-based self-study, and 4) Combined web-based self-study & classroom discussion,
was investigated for a single instructional unit (Nonlinear Equations) over
separate administrations of an undergraduate course in Numerical Methods. Two assessment instruments – 1) student
performance on a multiple-choice examination, and 2) a student satisfaction
survey were used to gather relevant data to compare the delivery modalities. Statistical analysis of the assessment
data indicates that the second modality where web-based modules for
instruction were used in conjunction with a face-to-face lecture delivery
mode resulted in higher levels of student performance and satisfaction. Background and Rationale Web-based modules have been
developed for a junior-level Numerical Methods course delivered in the The features of the web-based
modules are addressed indirectly since the complete details are readily available
in Ref1, 2. Stating in
brief, the unique features of the web-based modules are that they are both holistic and customized. Holistically, the web-based modules review essential course background information; present
numerical methods through several options - textbook notes, lecture videos,
PowerPoint presentations, simulations and assessments; show how course content
covered is applied in real life; tell stories to illustrate special topics
and pitfalls; and give historical perspectives to the material1,2. Faculty and students are able to choose a
customized view based on their preferred computational system - Maple3,
Mathcad4, Mathematica5, Matlab6, and choice
of engineering major - Chemical, Civil, Computer, Electrical, General,
Industrial, and Mechanical.
The focus of this research is
to compare four different modes of instructional delivery, namely 1) Traditional lecture, 2) Web-enhanced lecture, 3) Web-based self-study, and 4) Web-based self-study/discussion The present study is a
follow-up of findings reported in a previous paper7 where we
addressed only the first two modalities.
Since the previous study was completed, the course has been delivered
twice more, once with a web-based self-study and another with combined
web-based self-study followed by a classroom discussion. In
recent years, there has been a substantial amount of research exploring how
to enhance student learning across disciplines, including science,
mathematics, engineering, and technology (SMET) courses. Research in this area spans academic disciplines
and professional preparation, from medicine8 to education9
and computing to business10.
Furthermore, the research base is exploring how e-learning, as internet-based
education is often referred to, has different benefits based on
characteristics of the individual student.
The British Journal of International Technology devoted an entire
edition to this issue alone11 addressing, among other things, the
need to be cognizant that distance learning has a unique ability to provide
students with different learning modalities with varied resources and
strategies. Techniques and tools to be
used to enhance learning using the web include effective and adaptive
navigation as well as addressing multiple and diverse needs and interests of
the student12. The
text, How People Learn13
provides a foundation for many of the issues facing current educators who are
encountering an increasingly diverse and multi-faceted student population. This literature was foundational to the
exploration of various modalities of course delivery considered in this study. According to How People Learn, experts (in this case, faculty) “often forget what is easy and what is
difficult for students13, p. 32.” Relative to this issue, the modules and
instructional materials developed through this study offer both students and
faculty a comprehensive instructional package for simplifying and enhancing
the teaching of numerical methods across the engineering curriculum. Further,
research has demonstrated that it is beneficial to provide “instruction that enables students to see
models of how experts organize and solve problems” and that “the level of complexity of the models must
be tailored to the learners’ current levels of knowledge and skills13,
p. 37.” The design and format of
the web-based modules helps students see how experts apply fundamental
numerical methods to solve real world engineering problems both within and
across different engineering disciplines.
And
finally, citing again from this same synthesis of research findings, we know
that “A major goal of schooling is to
prepare students for flexible adaptation to new problems and settings13,
p. 65” and that “knowledge that
is taught in only a single context is less likely to support flexible
knowledge transfer than is knowledge that is taught in multiple contexts13,
p. 66.” Our effort was to
provide instruction opportunity to suit different learning styles14. By enabling
students to select both a preferred computational system as well as to select
one or more illustrative examples drawn from seven popular engineering majors
within each topic area, these interactive instructional modules maximize the
likelihood of lasting and flexible learning transfer of essential numerical
methods course content. Implementation
& Assessment Instruments
The previous study7 compared the first two
modalities 1) Traditional lecture, and 2) Web-enhanced
lecture for the two topics of Nonlinear Equations and Interpolation. In this paper, the focus is narrowed to the
topic of Nonlinear Equation, but the scope of data is broadened by looking at
four modes of delivering the content. The
four modalities were implemented in four separate semesters - Summer 2002,
Summer 2003, Summer 2004 and Spring 2005[1]
semesters, respectively. In
Summer 2002 semester, students in the Numerical Methods course were
instructed on Nonlinear Equations using the traditional, face-to-face lecture
method without the use of the web-based modules, hereafter referred to as the
Traditional Lecture mode of
delivery. We used a popular engineering
numerical methods textbook15 for reading assignments and problem
sets. In
Summer 2003 semester, students were instructed on the same topic of Nonlinear
Equations using both lecture and the web-based resources that were developed
for the course, hereafter referred to as the Web Enhanced Lecture. Before
discussing numerical methods for a mathematical procedure, we conducted an
in-class and informal diagnostic test on the background information via
several multiple-choice questions.
This allowed us to review specific material that most students
struggle with. We used PowerPoint
presentations to present the topics.
These presentations were continually supplemented with discussions based
on spontaneous instructor and student questions. Several times during the presentation,
students were also paired in class to work out an iteration or two for a numerical
problem. We also met during the weekly
computer laboratory session where each student had access to a computer. Simulations for various numerical methods
were conducted. In Summer
2004 semester, students received instruction through a distance format
without a classroom lecture component, hereafter called the Web-Based Self Study mode. Same resources were available to students
as they were in Summer 2003. In
addition, lecture videos that were video recorded in a studio were available
online. Since the students were
learning the material themselves, regular class periods and the weekly lab
session that were devoted to the topic of Nonlinear Equations as in Summer
2003 were cancelled. At the end of the
week, as part of their graded homework assignment, students were asked to
submit answers to 18 short questions (6 on each of the 3 subtopics of
Background, Bisection Method, and Newton-Raphson Method) that were based on
six levels of Bloom’s taxonomy. The
reading assignments and problem sets were the same as in Summer 2003. In
Spring 2005 semester, students used the same self-study methods as those in Summer
2004 but were required to meet in the weekly lab session to discuss the
lesson. This mode hereafter is called Web-based Self Study/Class Discussion. Although attending the weekly lab
session was mandatory, they were not required to ask questions. Before the weekly lab session, as part of
their graded homework assignment, students were asked to submit answers to 9
short questions (3 on each of the 3 subtopics) based on first three levels of Bloom’s taxonomy. After the weekly lab session, they were
asked to submit answers to 9 more short questions (3 on each of the 3
subtopics) based on last three
levels of Bloom’s taxonomy. The
reading assignments and problem sets were the same as in Summer 2003. To
measure the student performance, four[2]
questions were asked in the Nonlinear Equations portion of the final
examination. Two of the four questions
were selected at the lower levels of Bloom’s taxonomy, while the other two were
chosen from the upper levels of Bloom’s taxonomy. Student performance on these four questions
was examined as a function of the four course delivery modes. To measure
student satisfaction, a survey that gathered information on students’
perceptions of the presentation and
how it impacted their learning of the material was developed. This data was both quantitative and
qualitative in nature, thus permitting exploration of the reasons behind
student ratings. The instrument
consisted of eight Likert17 items (see Table 4) using a scale from
1 (truly inadequate) to 7 (truly outstanding). Instruments for the selected response
options was consistent across semesters, however, qualitative data varied
based on the mode of delivery. No
qualitative data was gathered in the initial (Summer 2002) year of course
delivery. For the other three years, questions
varied slightly, based on delivery mode.
While in Summer 2003, only one open-ended question was asked, “In
what way can the class presentations be improved for Nonlinear Equations“. In Summer 2004 and Spring 2005, four
questions were asked. In addition to
the question asked in 2003, the other three questions for 2005 were: “How did
you learn the material for Nonlinear Equations?” What did you like most about the web-based
and class presentation for Nonlinear Equations?”, and “What did you like
least about the web-based and class presentations for Nonlinear Equations”. The 2004 questions were similar but did not
address class presentations since they were not a part of the instruction in
2004. The answers were analyzed
thematically to identify trends as well as strengths and weaknesses of the
course as perceived by the students. To evaluate the effectiveness of
the various modes of delivery, the same sources of assessment data was used
across the four years as well as the survey data previously discussed. Student performance was examined relative
to their starting abilities, as reflected in their combined GPA across four
prerequisite courses, Calculus I, Calculus II, Calculus Assessment Results Students in each of the
four classes were typically in the later stages of their academic career and
were identified as coming from three different sources. They were identified as either transfer
students from the Community College (CC), First Time in College (FTIC), or
Other (OT[3]). Since the Mean GPA (MPGPA) of the four
prerequisite courses was of interest as a predictor, there was a concern
about the equity of class composition as a function of where the students
might have taken the four prerequisite courses. Chi-Square Goodness of Fit tests were
conducted to determine if each of the classes delivered under the four
different modalities contained similar students. The results of these analyses revealed that
there were no statistically
significant differences (using a Type I error rate of 0.5) across classes
based on gender or location of previous course work (c2= 1.07,
p=0.7849 for gender and c2= 18.96, p=0.4410 for location of previous
course work). Two assessment
instruments7 were used to explore the impact of course delivery
mode on student achievement and satisfaction.
1.
multiple choice question
final examination7 based on Bloom’s taxonomy, and 2.
student satisfaction
survey7. The summer semesters of
2002 and 2003 were 6 weeks long, the summer 2004 semester was 10 weeks long, and
the Spring 2005 semester was 16 weeks long.
As such, results must be considered with regards to this potentially
influential factor. A. Multiple-Choice Final Examination Based on Bloom’s
Taxonomy Four
multiple-choice questions on Nonlinear Equations were used to gauge how well
students performed in this area of the course delivered under the four
different modalities. Two questions
were asked at lower levels (Knowledge, Comprehension,
and Application) of Bloom’s taxonomy and two questions were asked at upper
levels (Analysis, Synthesis, and
Evaluation) of Bloom’s taxonomy. Each
correct answer was given a score of one while an incorrect answer was scored
as a zero, for two possible points for each of the lower and upper level sets
of questions. For each of the four
classes, Table 1 contains the sample size
and the means for the incoming GPA on the four prerequisite courses (Calculus
I, Calculus II, Calculus
To test
the potential for different modalities of delivery to impact student performance,
a two-factor Analysis of Variance (ANOVA) was conducted. Students were classified into one of three
groups: low, medium, and high
ability. Classification into a
category was based on incoming GPA on the four prerequisite courses. The low category was comprised of student
in the 25th percentile of the sample, the medium category was
comprised of students who scored in the middle half of the percentile scores,
and the high category was comprised of students in the 75th
percentile or higher. The distribution
of these students is presented in Table 2. Table 2 – Sample
Size of Students in Each Ability Level by Cohort
The results of the two-factor Analysis of Variance
(ANOVA) using the MPGPA and Course Delivery Modality was used to examine
student performance on the two sets of questions representing the lower and
upper levels of Bloom’s Taxonomy are presented in Table 3. The F
statistics used in the ANOVA analysis are used to draw conclusions about mean
differences in the population based upon the observed data. Each F
statistic is the ratio of a variance estimate based upon differences among
group means and an estimate based upon differences among scores within
groups. Large values of F are associated with group mean
differences that are greater than would be expected from only sampling error.
The p-value is the probability of obtaining an F statistic as large as the one observed or larger, if the null
hypothesis (that is, equal means in the population) is true. The smaller the p-value is, the less we believe that the null hypothesis is true.
When the p-value is smaller than a pre-specified criterion (called a), we officially declare the null hypothesis false and
conclude that the population means are not the same. Conversely, if the p-value is larger than a, we declare that we fail to reject the null hypothesis. The pre-specified value, a, is the probability of rejecting a null hypothesis when
it is true (a decision that is called a Type I error). These results of the two-factor, with interaction, design
of experiment are interpreted relative to a level of confidence of a=0.10 (or 90% confidence that the claim can be made) in
the results18, chapter 10. This
Type I error rate is consistent with the baseline study7 to
determine statistical significance of findings. Similar to findings in the previous study7,
and as might be expected, the MPGPA was a significant predictor of student
performance. The method of delivery
was not statistically significant as a main effect. However, an interaction between MPGPA and
mode of delivery was evident for the scores on the lower level taxonomy
questions. Further follow-up tests
identified that students in the 2002 and 2005 classes performed similarly,
but the 2003 cohort performed significantly higher than the students in the
2002 and 2005 classes. The results of
the contrast tests of significance between the groups revealed that, using an
alpha of 0.10, the students in 2004 performed similar to the remaining three
groups.
The results in Table 3 can be
summarized as follows: · Effect of pre-requisite
GPA (Factor A) – The
effect of the pre-requisite GPA (MPGPA) on the final examination score is
significant with a 90% confidence level (a = 0.10) for Nonlinear Equations upper
and lower level Bloom scores. Students
with prerequisite GPA higher than MPGPA perform better on these scores.
· Effect of course delivery mode (Factor
B) – The effect of course delivery mode on the final
examination score was not significant at the 90% confidence level (a = 0.10) for Nonlinear Equations upper and lower level
Bloom scores. Thus, students receiving
instruction under the different modalities did not vary significantly across
the different methods of instruction. ·
Effect of pre-requisite GPA and course delivery
mode interaction – The effect of the interaction between
GPA and delivery modality on the lower level Bloom questions was significant
(a = 0.10). This
indicates that different ability level students may perform better based on
mode of course delivery.
Based on
the findings reported above, as well as an examination of the mean scores
(see Table 1), there is support that the use of web-based modules positively
impact student performance. Although
not all statistical analyses had statistically significant findings, students
in the 2003 cohort consistently outscored their peers in the other
classes. Furthermore, the interaction
between mode of delivery and incoming ability level suggests that the use of
web-based modules provides students coming in with a lower ability (as
indicated by GPA on the four prerequisite courses) with an enhanced ability
to be successful on the material presented. B. Student Satisfaction Survey Student
satisfaction surveys were given on the presentations used to teach Nonlinear
Equations. The survey consisted of
eight selected response questions, and, depending on the class, included zero
to four open-ended questions. Quantitative Analysis A
seven-point Likert scale was used for the eight selected response items,
ranging from 1 (Truly Inadequate) to 7 (Truly Outstanding). In addition, an Analysis of Variance was
conducted on each of the items. The
results of these analyses are provided in Table 4. The results of all eight items are
statistically significant at the set Type I error rate of 0.10. In all cases, students in the 2003 cohort
had notably higher scores than in the other three classes of modality
delivery. Contrast statements support
the contention that this group of students rated these items higher than
their peers in the other classes at an alpha level of 0.05. Table 4 – Results of Presentation Items on Surveys on
Nonlinear Equations (number of samples, means, F-values, and p-values)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
