Chapter 3: Methodology
The Pilot Study
Attitudinal Data
On both the pre-test and the post-test, subjects were presented with the same eighteen statements about mathematics and asked to indicate for each one whether they strongly agreed, agreed, weren’t sure, disagreed, or strongly disagreed with the statement. Appendix C gives the statements and the numbers of day and night subjects agreeing and disagreeing with each statement on the pre- and post-tests, with “strongly agree” and “agree” are combined into one category, as are “strongly disagree” and “disagree.” Pre-test results are given twice, once for all subjects and once for only those subjects who went on to take the post-test.5 Appendix C also gives the results grouped by topic.
For most statements, there was little or no difference between sections or between pre-test and post-test results. On both pre-tests and post-tests, both sections generally agreed with the following statements:
1. There is always a rule to follow in solving a mathematics problem.
2. I expect to use graphs in my job.
4. Mathematics is a good field for creative people.
6. Using graphs makes information easier to understand.
7. Mathematics is useful in solving everyday problems.
9. I am looking forward to taking more mathematics.
10. I expect to use computers in my job.
11. I am better at working with graphs than most people are.
13. I usually understand what we are talking about in a mathematics class.
14. Using a computer can help you learn many different mathematical topics.
17. I expect to use mathematics in my job.
On both pre- and post-tests, both sections generally disagreed with the following statements:
3. Using computers makes mathematics more mechanical and boring.
5. Most people do not use mathematics in their jobs.
8. There have not been any new discoveries in mathematics for a long time.
12. Mathematics is harder for me than for most people.
15. Information presented in a graph is usually difficult to understand.
16. Learning mathematics involves mostly memorizing.
18. In mathematics, problems can be solved without using rules.
Scoring “strongly agree,” “agree,” “not sure,” “disagree,” and “strongly disagree” as 1, 2, 3, 4, and 5, respectively, I computed the change from pre-test to post-test for each subject’s response to each statement, and used t-tests of individual means of the changes (equivalent to paired t-tests on the pre-test and post-test results) with a per-contrast error rate of a = 0.10 to determine whether the changes were significant. Four statements showed significant changes from pre-test to post-test: “10. I expect to use computers in my job” (p < 0.10); “11. I am better at working with graphs than most people are” (p < 0.10); “16. Learning mathematics involves mostly memorizing” (p < 0.10); and “18. In mathematics, problems can be solved without using rules” (p < 0.10). For statement 10, the change was mostly between “agree” and “strongly agree”; disregarding strength of agreement leaves almost no change in this item. For statements 16 and 18, the change was concentrated in the night class, with the day class showing little or no change.
Next, I used independent two-sample t-tests with a per-contrast error rate of a = 0.10 to determine whether the change on each statement from pre-test to post-test was significantly different between the two sections. For four of the statements the two sections were significantly different: 1. There is always a rule to follow in solving a mathematics problem (p < 0.01); 7. Mathematics is useful in solving everyday problems (p < 0.05); 3. Using computers makes mathematics more mechanical and boring (p < 0.10); and 12. Mathematics is harder for me than for most people (p < 0.10).
Nature of Mathematics
These students describe mathematics as a good field for creative people, where new discoveries are still being made and learning is not primarily memorization. However, they also believe that there is always a rule to follow to solve every mathematics problem and that without the rules one cannot solve mathematics problems.
This result is similar to that reported by Sosniak, Ethington, and Varelas (1994). In their study of the responses eighth-grade mathematics teachers in the US gave to attitude items from the IEA Second International Mathematics Study (SIMS), including some of the same items used in the present study, they found that teachers generally expressed progressive views of mathematics as a discipline, but reported very traditional classroom practice. Supporting a progressive view of mathematics as a discipline, the teachers generally agreed with the statement “there are many different ways to solve most mathematics problems,” and disagreed with the statement “learning mathematics involves mostly memorizing.” (In regard to the latter statement, they agreed with the students in the present study.) However, supporting a traditional view of classroom practice, 87% of the teachers reported that they spent more time on lectures and teacher explanations than on group work.
The lack of a coherent orientation was particularly clear with regard to practices that the teachers rated as important for effective teaching. Sosniak, Ethington, and Varelas found that nearly three quarters of the teachers in their study rated both of the following, seemingly contradictory, practices as being of major or highest importance: “provide an opportunity for students to discover concepts for themselves” and “give students detailed step-by-step instructions on what they are to do.”
Sosniak, Ethington, and Varelas note that the teachers in their study tended to be more progressive in their views of issues that are abstract and distant from the classroom, such as the nature of mathematics, and more traditional with regard to matters that are close to classroom practice, such as the amount of time spent in lecture. They suggest that traditional practice is simply the collection of strategies that teachers have found to be effective within the demands and constraints of the typical school situation. Thus, as long as the schools remain substantially the same as they were when the traditional practices were developed, those practices will remain effective and the teachers will continue to use and value them, despite whatever views the teachers may hold on more abstract issues.
A similar conclusion could be drawn with regard to the students in the present study. They have found that following rules is an effective way of dealing with the problems they encounter in their homework, quizzes, and exams, just as the teachers responding to the SIMS questions had found that lecturing was an effective way to deal with the demands of school boards and the constraints of their classroom environments. Thus, when presented with questions that bear directly on their classroom experience, they respond in traditional ways, emphasizing following rules to solve problems. However, when presented with questions that are more distant from their classroom experience, they give more progressive answers, acknowledging continuing discovery in mathematics, and valuing creativity over memorization.
These conflicting statements appear to indicate two disparate images of mathematics. On one hand, there is mathematics as practiced by mathematicians, where creativity is an asset--perhaps even a requirement--and new discoveries are still being made. On the other, there is the mathematics of the classroom, where rules are taught and problems are solved by following examples given in the text. The constructivist unit used in this study is designed to enable the students to experience in the classroom some of the discovery and creativity associated with the mathematics of mathematicians, thus reducing their expectation that problems will be solved by application of rules. However, a few hours of instruction, especially instruction that is set apart from the classroom routine by being labeled as an experiment, is unlikely to make a large change in the image of classroom mathematics held by students who have experienced many years of conventional mathematics instruction.
The greatest difference between sections occurred on statement, “1. There is always a rule in solving a mathematics problem.” Of the night students who later took the post-test, eight agreed with this statement on the pre-test, one disagreed and one was unsure. When these students took the post-test, only five agreed with the statement, three disagreed, and two were unsure. According to a t-test, this change was significant at p < 0.10. A similar statement, “18. In mathematics, problems can be solved without using rules,” showed a similar change (p < 0.10) for the night class. Of the night students who later took the post-test, eight disagreed with this statement on the pre-test, one agreed and one was unsure. When these students took the post-test, only five disagreed with the statement, four agreed, and one was unsure. This is the direction of change that was expected. Since the unit asks students to answer questions and solve problems for which they have not been given rules, it is expected to challenge students’ belief that there will always be a rule.
The day students had a different reaction. Of the six day students who later took the post-test, three agreed and three disagreed on the pre-test that “1. There is always a rule in solving a mathematics problem.” All six of these students agreed with this statement on the post-test. This change was significant at p < 0.05. It is not clear why the unit might have had the effect of convincing students that problems in mathematics can always be solved by following rules. During the unit, the students discovered some principles, which might be thought of as rules, about the way graphs are used to present information, but no rules were presented by the instructor or the lesson packet. The day class showed no change on the other statement about rules, “18. In mathematics, problems can be solved without using rules.” Of the day students who took the post-test, five agreed and one disagreed with statement 18 on both the pre-test and the post-test. While this shows that the day students expect rules to be used in solving mathematics problems, it leaves some doubt about the extent of change in their views as a result of this unit.
In another counter-intuitive result, the night class significantly (p < 0.10) changed its view on memorization, moving in the direction of greater agreement with the statement “16. Learning mathematics involves mostly memorizing.” One the pre-test, nine of the night students who went on to take the post-test disagreed with the statement and only one agreed. On the post-test, three agreed, five disagreed, and two were unsure. This is odd both because there was no memorization involved in the unit and because the night students had moved away from agreement with the image of mathematics as involving application of rules. I’m not sure what to make of this, except that this area should get more attention in the main study. On both the pre-test and the post-test, five out of the six day students disagreed with statement 16.
Utility of Mathematics
Although the statement “7. Mathematics is useful in solving everyday problems” showed a significant difference (p < 0.05) between sections in the amount of change from pre-test to post-test, the change was all between “strongly agree” and “agree.” When strength of agreement is disregarded, there is no change from pre-test to post-test for either section on that statement. Students in both sections, on both pre- and post-tests, were nearly unanimous6 in agreeing that they expected to use mathematics and computers in their jobs, that most people use mathematics in their jobs, and that mathematics is useful in solving everyday problems. While they were divided on the question of whether they would use graphs in their jobs, there was little change from pre-test to post-test on this item.
Using Computers in Mathematics
Although the statement “3. Using computers makes mathematics more mechanical and boring” showed a significant difference (p < 0.10) between sections in the amount of change from pre-test to post-test, only two subjects in each section actually changed their responses. The significance comes from the fact that both changes in the day section were in the direction of greater disagreement with the statement, while both changes in the night section were in the direction of greater agreement. This is counter to what was expected, because the computers in the night section were used with motion sensors, which involved the students moving around the room, while the computers in the day section were used with the Java applet, which involved moving the mouse, presumably a more familiar and perhaps more boring format. Despite its statistical significance, I hesitate to ascribe importance to this result. With 18 items on the attitude survey, one would expect a couple of tests to appear significant at the a = 0.10 level even if the instruction had not made any difference. I suspect that is the cause of this counter-intuitive result. The main study should help confirm or refute this.
The subjects generally agreed that “using a computer can help you learn many different mathematical topics.” On the pre-test, the agreement on this item was slightly stronger in the day class than in the night class, but the difference was not significant. On the post-test, the night students showed slightly greater agreement with this item than they had shown on the pre-test, although this change also was not significant. It seems likely that the younger day students have had more experience with computer-aided instruction than the older night students have had, and so were more likely to see it as useful, and that the instruction made more of a difference to the attitudes of the night students than those of the day students because the experience was more novel for the night students. If the main study were to be conducted in a community college, this would be an area to examine further. Subjects in both sections on pre- and post-tests were nearly unanimous in agreeing that they would use computers in their jobs.
Subjects’ Perceptions of Their Own Abilities in Mathematics
In both sections, there was significant (p < 0.10) movement toward greater agreement with the statement “11. I am better at working with graphs than most people are.” This was the expected direction of change, since the students spent the week of the unit working with graphs constantly, and were always successful. The unit was structured in such a way that all students progressed through it in a reasonably successful manner, although perhaps at different speeds.
There was a significant (p < 0.10) difference between sections in the changes from the pre-test to the post-test responses to the statement “12. Mathematics is harder for me than for most people.” Both sections generally disagreed with the statement. The day class moved in the direction of greater disagreement; the only subject who had not disagreed on the pre-test moved from “not sure” to disagreeing on the post-test, while the others continued to disagree with the statement. This is consistent with the change on statement 11 toward greater confidence in their graphing abilities. In contrast, the night class moved toward greater agreement with statement 12, changing from eight disagreeing, one agreeing, and one not sure on the pre-test to seven disagreeing,two agreeing, and one not sure on the post-test. None of the subjects in either section appeared to find the unit difficult, so it is not clear why the unit would cause any subject to move in the direction of greater agreement with statement 12.
Main Study
The main study will be conducted at the University of Illinois at Urbana-Champaign (UIUC) in the fall of 1999. Subjects will be first-semester calculus students enrolled in the standard lecture/discussion course, Math 120. Since it is not feasible to use class time, students will be asked to come in during the evenings for the instruction offered as part of the study. Their primary incentive will be getting free help with a key concept in a difficult course. This is likely to mean that only students who expect to have trouble with calculus will volunteer, which will not be representative of all students. However, by taking volunteers and meeting them outside of class, I will be able to randomly (subject to the constraints of the subjects’ schedules) assign the subjects to groups, making for a true experiment rather than a quasi-experiment. If necessary, incentives may be provided to encourage volunteers.
Approximately half of the volunteers will be assigned to groups in which the students will use motion sensors to work through the instructional unit. The other half will be assigned to groups using the Java applet to complete the same unit. After administering a pre-test to all volunteers, I will meet with the students in groups of about twenty, introduce them to the equipment, and encourage them to work through the lesson packet over the course of two sessions of about two and a half hours each. After completing the instruction, the students will be asked to return for a post-test. Some incentive may be provided to encourage them to complete the post-test.
If possible, all students in the classes from with the volunteers are drawn will be tested at the beginning and end of the semester, to determine how the subjects compare to their peers and to measure what each group retains. Since assessing student’s understanding of the concept of derivative is a normal part of the exams in the course, I anticipate being able to get the instructors’ cooperation to place on the exams questions that are useful both for my purposes in this study and for the instructors’ purposes in the course. Of course, no data will be used from any student who has not given consent.
In preparation for the main study, the lesson packet will be revised and the achievement test will be substantially rewritten, as described above. The pre-test, including attitude survey, took the subjects only about 15 minutes to complete, so there is room to add more questions without making the test too long and burdensome. For new questions, I will interview some UIUC calculus instructors and ask what they consider to be appropriate ways of assessing students’ understanding of the concept of derivative. This input should enable me to create an assessment instrument more suitable for the population I will be studying. The attitude survey will also be expanded, with particular attention given to questions about the students’ image of mathematics. Those statements used in Sosniak, Ethington, and Varelas’s analysis that were not on the survey for the pilot study will be added for the main study, so that I can check on the extent to which these subjects agree with the teachers who participated in SIMS.
I will research the feasibility of including a few questions to determine which subjects are primarily kinesthetic learners. It would be interesting to discover whether the effects of the two types of instruction on kinesthetic learners are different from the effects on other types of learners. However, since my main interest is to determine what instruction is appropriate for students in general, in situations where it is not feasible to separate out different types of learners, the overall effect on the population at large is more relevant than the effect on various subpopulations.
Pilot study sessions were videotaped and audiotaped. Before the main study, I will analyze this data and settle on some method of recording the main study to acquire the types of data that appear most useful.
Potential Benefits of this Study
This study is intended to have very practical and immediate benefits for teachers. The most obvious of these is the creation of a convenient, economical, and effective alternative to motion sensor instruction. In addition, the particular instructional unit used to teach students about the derivative was developed for this study. If it proves to be effective, as observations from the pilot study suggest that it will, then it could be used, together with either motion sensors or the Java applet, to improve future calculus classes.
There are also more theoretical benefits to be expected from this study. If significant differences appear between the groups, this will confirm the speculations of the researchers who argue for the importance of the kinesthetic sense in graphing instruction. If, as I suspect, no significant differences are found, it will suggest that we should look elsewhere for the factors critical to the well-documented success of motion-sensor instruction. Either way, this study will shed some light on questions about the role of the kinesthetic sense in the learning of mathematics. Thus, this study is likely to contribute to our understanding of the learning process at a basic level.
1 Of course, since several means were being compared simultaneously, it would be appropriate to apply a stricter standard. One might choose to control the familywise error rate, rather than the per-contrast error rate, at a = 0.10. However, in this case even the looser standard of controlling the per-contrast error rate showed no significant differences, so we really don’t have to worry about familywise error rates here.
2 I didn’t get information about high school coursework from these students. I intend to include a question about that on the pre-test in the main study this fall.
3 I now realize that, just as the instructor needed to repeatedly remind the students to get them to go to the testing center and take the post-test, I needed to remind the instructor, and keep count of how many tests were completed and how many were left to be done. I hope to get a much higher proportion of the subjects in the main study to complete the post-test.
4 Simple subtraction was reasonable in this case because of the extreme similarity of the pre- and post tests.
5 One subject in the night class took the post-instruction achievement test, but not the post-instruction attitude survey, so I have post-test achievement results for eleven night students but attitude results for only ten.
6 One student who chose “not sure” on questions related to expectations for future jobs added a note indicating that these questions did not apply to her. She is a homemaker of approximately retirement age, and does not anticipate having a job in the future. This is not likely to be an issue with subjects in the main study.
Back to the Table of Contents for this paper.
Back to Lisa's Papers page.
Back to Lisa's Academic Activities page.
Back to Lisa's personal home page, Over the Rainbow.
This page last revised January 18, 2000