Tuesday, April 29, 2014

Does educational software make a difference?


Does educational software make a difference?  A summary of: Campuzano, L., Dynarski, M., Agodini, R., and Rall, K. (2009). Effectiveness of Reading and Mathematics Software Products: Findings From Two Student Cohorts—Executive Summary (NCEE 2009-4042). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
S. Krashen, May, 2014
Hat-tip: Susan Ohanian
Campuzano et. al. investigated whether software for reading and math had any effect of achievement, as measured by standardized tests. The reading software was used in grades 1 and 4, and the math software was used in grade 6 (pre-algebra) and for algebra (mostly grade 9).
The study lasted one year, and was replicated the second year.  The first year, 16 softward products were tested in 33 districts, 132 schools, and with 428 teachers in either classes that used the softeware or comparison classes that did not.. The second year included 10 products, 23 districts, 77 schools, 176 teachers, and 3,280 students. A variety of well-known standardized tests were used. For reading, the SAT-9 was used in grade 1 and the SAT-10 in grade 4 for reading and grade 6 for math.  The study used the Educational Testing Services’  End-of-Course Algebra Assessment for grade 9 (algebra I). Other tests were also used to confirm that the groups had similar levels of competence in reading and math beore the treatment began.
With the exception of scores on the ETS algebra test, scores were converted to normal curve equivalent (NCE) units. Algebra I scores for the ETS test are reported as percent correct.
The results
NO IMPACT OF USING SOFTWARE
There was no significant difference between test scores of students who used the software and those who did not at the end of the first year.
First year results
READING
software
comparison
grade 1
50.2
49.5
grade 4
42.1
41.7



MATH


grade 6
52.2
50.8
grade 9
37.3
38.1
Results presented in normal curve equivalents, except for grade 9, in percentiles.

NO CONSISTENT EFFECT OF EXPERIENCE WITH SOFTWARE
Some teachers (n = 115) taught both years with the software.  For reading, these teachers' scores were not significantly higher the second year. For math, the 6th grade scores got significantly worse, and the 9th grade scores got significantly better.
Difference between first and second year scores for teachers who used software for both years
READING
year 1
year 2
grade 1
0.86
-1.28
grade 4
2.65
4.67



MATH


grade 6
-0.44
-1.24 *
grade 9
-0.34
2.58 *
* statistically significant difference

LITTE DIFFERENCE AMONG THE PROGRAMS
A third analysis was the impact of ten of the software programs used in either the first or second year. Only one difference out of ten was statistically significant and in this case software group's test score was only two NCE's (about three percentiles) higher than the comparison group.
Impact of reading programs
grade 1

Destination Reading
1.91
Headsprout Early Reading
0.29
PLATO
0.5
Waterford
0.42
grade 4

Academy of Reading
-0.16
Leap Track
1.97 *
·      statistically significant difference
·       

Impact of math programs

grade 6

PLATO/Achieve now
-0.58
Larson Pre-Algebra
2.37
grade 9

Larson Algebra I
-0.1
Cognitive Tutor
-1.28

CONCLUSION
Out of 18 comparisons, 15 showed no difference, two showed a significant but small effect in favor of the software, and one showed a small negative effect.  We would expect one comparison out of 20 to be significant because of chance. 
The results are consistent with the conclusion that these educational software programs did not enhance learning. The results also indicate that experience with the programs did not improve their effectiveness.  Nine of the ten programs investigated had no positive effect. The only one that was significantly effective did only slightly better than the comparison group.









9 comments:

  1. It would be interesting to see descriptions and examples of the software and how it differs from the comparison group materials. Anecdotally, when I taught middle-school mathematics, the available software was just a different format for the same material, and it had the expected disadvantage of not relating to individual students affectively, something that was important to students and often made the difference between success and failure. Of course, that was several years ago, and one would expect there to be improvements. But I don't know how they could make up for the relationship between the student and the teacher.

    ReplyDelete
  2. "But I don't know how they could make up for the relationship between the student and the teacher"

    Exactly that's the thing that always drove me crazy when computers and software were being everywhere touted as being great for education because they are "interactive." What could be more interactive than an actual person.

    ReplyDelete
  3. Institute Manager is one of the best Institute Management Solutionsavailable for Educational Institutions. Contact us to get more information on our world class education software today! We provide complete Institute Management solutions for schools and colleges.

    ReplyDelete
  4. I needs to spend some time learning more or understanding more. Thanks for excellent information I was looking for this information.

    Thin Clients & Zero Client

    ReplyDelete
  5. software companies in delhi providing end to end software solutions. As a software development company based out of India,

    ReplyDelete
  6. The collage paper editing services are sometimes necessary for students and i wish them all the best.

    ReplyDelete
  7. Nice article i was reached for a good tips.thanks for sharing your blog.
    educational software companies

    ReplyDelete
  8. What a wonderful post you wrote! This will help many beginners like me! It's very clear, precise and descriptive really! Thank you very much, author. Keep writing for us like this :)

    DedicatedHosting4u.com

    ReplyDelete