Does educational software make a difference?

Does educational software make a difference?  A summary of: Campuzano, L., Dynarski, M., Agodini, R., and Rall, K. (2009). Effectiveness of Reading and Mathematics Software Products: Findings From Two Student Cohorts—Executive Summary (NCEE 2009-4042). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

S. Krashen, May, 2014
Hat-tip: Susan Ohanian

Campuzano et. al. investigated whether software for reading and math had any effect of achievement, as measured by standardized tests. The reading software was used in grades 1 and 4, and the math software was used in grade 6 (pre-algebra) and for algebra (mostly grade 9).

The study lasted one year, and was replicated the second year.  The first year, 16 softward products were tested in 33 districts, 132 schools, and with 428 teachers in either classes that used the softeware or comparison classes that did not.. The second year included 10 products, 23 districts, 77 schools, 176 teachers, and 3,280 students. A variety of well-known standardized tests were used. For reading, the SAT-9 was used in grade 1 and the SAT-10 in grade 4 for reading and grade 6 for math.  The study used the Educational Testing Services’  End-of-Course Algebra Assessment for grade 9 (algebra I). Other tests were also used to confirm that the groups had similar levels of competence in reading and math beore the treatment began.

With the exception of scores on the ETS algebra test, scores were converted to normal curve equivalent (NCE) units. Algebra I scores for the ETS test are reported as percent correct.

The results

NO IMPACT OF USING SOFTWARE

There was no significant difference between test scores of students who used the software and those who did not at the end of the first year.

First year results

READING	software	comparison
grade 1	50.2	49.5
grade 4	42.1	41.7
MATH
grade 6	52.2	50.8
grade 9	37.3	38.1

Results presented in normal curve equivalents, except for grade 9, in percentiles.

NO CONSISTENT EFFECT OF EXPERIENCE WITH SOFTWARE

Some teachers (n = 115) taught both years with the software.  For reading, these teachers' scores were not significantly higher the second year. For math, the 6^th grade scores got significantly worse, and the 9^th grade scores got significantly better.

Difference between first and second year scores for teachers who used software for both years

READING	year 1	year 2
grade 1	0.86	-1.28
grade 4	2.65	4.67
MATH
grade 6	-0.44	-1.24 *
grade 9	-0.34	2.58 *

* statistically significant difference

LITTE DIFFERENCE AMONG THE PROGRAMS

A third analysis was the impact of ten of the software programs used in either the first or second year. Only one difference out of ten was statistically significant and in this case software group's test score was only two NCE's (about three percentiles) higher than the comparison group.

Impact of reading programs

grade 1
Destination Reading	1.91
Headsprout Early Reading	0.29
PLATO	0.5
Waterford	0.42
grade 4
Academy of Reading	-0.16
Leap Track	1.97 *

·      statistically significant difference

·       

Impact of math programs
grade 6
PLATO/Achieve now	-0.58
Larson Pre-Algebra	2.37
grade 9
Larson Algebra I	-0.1
Cognitive Tutor	-1.28

CONCLUSION

Out of 18 comparisons, 15 showed no difference, two showed a significant but small effect in favor of the software, and one showed a small negative effect.  We would expect one comparison out of 20 to be significant because of chance. 

The results are consistent with the conclusion that these educational software programs did not enhance learning. The results also indicate that experience with the programs did not improve their effectiveness.  Nine of the ten programs investigated had no positive effect. The only one that was significantly effective did only slightly better than the comparison group.

Poll of the uninformed

Published in the San Francisco Chronicle, April 28, 2014

"Most in state approve of policy shifts in education," April 24:

Yes, 69 percent of Californians polled said they liked the Common Core standards after hearing a brief statement about them. The statement said only that they "are designed to ensure that students graduating from high school have the knowledge and skills they need to enter college programs or the workforce." I'm surprised anybody would object.

Not mentioned is the fact that the standards call for a uniform but untested curriculum and an astonishing amount of testing. Not mentioned is the fact that the Common Core does not address the largest factor influencing school performance: poverty. When researchers control for the effect of poverty, American students score at the top of the world on international tests.

Eighty percent of those interviewed said they knew either a little or nothing at all about the Common Core. The description they heard, devoid of meaningful detail, made them even less informed.

Stephen Krashen

Incentives for Reading Aloud to Children?

Published in the School Library Journal, 2014.

Introduction by the editors of the School Library Journal:
In School Library Journal‘s January 2014 print issue, Suffolk Cooperative Library System’s youth services coordinator Lisa G. Kropp wrote the article “Vivacious Vocabulary” about how in the interest of building students into critical thinkers and strong readers, libraries are embracing the “1,000 Books Before” program that encourages parents to read 1,000 books to their kids before kindergarten.
One SLJ reader, Stephen Krashen, language expert and professor emeritus at University of California, responds to Kropp’s article in an email:


Read alouds are very pleasant for both parents and children. The vast majority of children say they enjoy being read aloud to: Ninety-seven percent in the study Walker and Kuerbitz (1979), 95 percent in Mason and Blanton (1971), and many parents agree. Eighty-nine percent of mothers interviewed in Newson and Newson (cited in Wells, 1985) said their children liked to be read to.

However, giving certificates, getting one’s picture in the local newspaper, and other incentives could send the message that reading and hearing stories is not pleasant and that nobody would do it without a bribe. This could reduce the amount of reading aloud when the rewards are withdrawn (for research, see Kohn, 1999).

But, I agree with the article’s author Lisa Kropp in that we need to encourage reading aloud. The first step is to make sure that families have books. The second is to make sure that parents, and other caretakers, understand the benefits of read alouds.

Stephen Krashen
 

Kohn, A. (1999). Punished by Rewards: The Trouble with Gold Stars, Incentive Plans, A's, and Other Bribes. Boston: Houghton Mifflin.
Mason, G., and W. Blanton. (1971). Story content for beginning reading instruction. Elementary English, 48, 793-796.
Walker, G., and I. Kuerbitz. (1979). Reading to preschoolers as an aid to successful beginning reading. Reading Improvement 16: 149-154.
Wells, G. (1985). Language Development in the Pre-School Years. Cambridge: Cambridge University Press.

Original article: http://www.slj.com/2014/01/standards/early-learning/vivacious-vocabulary-turning-little-ones-into-strong-readers-first-steps/

Address Poverty to boost education

Published in USA Today, May 1, 2014

Address poverty to boost education: Your Say

Original title: Expert Opinion on the Common Core

USA TODAY's editorial does not mention the real opposition to the "Common Core" standards from professional educators and scholars ("'Common Core' demonized as Obamacore: Our view").
Common Core does not deal with the reason for unspectacular performance by American students. The United States has the second-highest level of child poverty among 35 economically advanced countries (now over 23%, compared with Finland's 5.3%). 
[IMPORTANT SENTENCE OMITTED BY USA TODAY EDITORS HERE: When researchers control for the effect of poverty, American scores on international tests are at the top of the world.]
Poverty means poor nutrition, inadequate health care and lack of access to books, among other things. All of these profoundly impact school performance. Instead of protecting children from the effects of poverty, the Common Core provides "tough" standards and more testing.

Stephen Krashen, professor emeritus; University of Southern California; Los Angeles
http://www.usatoday.com/story/opinion/2014/04/30/common-core-education-your-say/8540559/

Original article:
http://www.usatoday.com/story/opinion/2014/04/23/common-core-education-standards-test-scores-editorials-debates/8071745/


Sources (not included in published letter)
Levels of poverty: UNICEF Innocenti Research Centre 2012, ‘Measuring Child Poverty: New league tables of child poverty in the world’s rich countries’, Innocenti Report Card 10, UNICEF Innocenti Research Centre, Florence.

Control for poverty:
Payne, K. and Biddle, B. 1999. Poor school funding, child poverty, and mathematics achievement. Educational Researcher 28 (6): 4-13; Bracey, G. 2009. The Bracey Report on the Condition of Public Education. Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. http://epicpolicy.org/publication/Bracey-Report. Berliner, D. 2011. The Context for Interpreting PISA Results in the USA: Negativism, Chauvinism, Misunderstanding, and the Potential to Distort the Educational Systems of Nations. In Pereyra, M., Kottoff, H-G., & Cowan, R. (Eds.). PISA under examination: Changing knowledge, changing tests, and changing schools. Amsterdam: Sense Publishers. Tienken, C. 2010. Common core state standards: I wonder? Kappa Delta Phi Record 47 (1): 14-17. Carnoy, M and Rothstein, R. 2013, What Do International Tests Really Show Us about U.S. Student Performance. Washington DC: Economic Policy Institute. 2012. http://www.epi.org/).

“Poverty means poor nutrition, inadequate health care, and lack of access to books”:
Berliner, D. 2009. Poverty and Potential:  Out-of-School Factors and School Success.  Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. http://epicpolicy.org/publication/poverty-and-potential;   Krashen, S. 1997. Bridging inequity with books. Educational Leadership  55(4): 18-22.

More testing:
Krashen, S. 2012. How much testing? http://dianeravitch.net/2012/07/25/stephen-­‐ krashen-­‐how-­‐much-­‐testing/
 and: http://www.washingtonpost.com/blogs/answer-sheet/

Increasing testing does not mean greater achievement:
Nichols, S., Glass, G., and Berliner, D. 2006. High-stakes testing and student achievement: Does accountability increase student learning? Education Policy Archives 14(1).  http://epaa.asu.edu/epaa/v14n1/. OECD. Tienken, C., 2011. Common core standards: An example of data-less decision-making. Journal of Scholarship and Practice. American Association of School Administrators [AASA], 7(4): 3-18. http://www.aasa.org/jsp.aspx.

MOST BIASED POLL EVER

The AP reported that according to a recent poll, 7 out of 10 Californians support the common core.  Those who took the poll first read a description of the common core:

“The Common Core State Standards are a single set of K–12 English language arts and math standards that most states, including California, have voluntarily adopted. The state leaders who developed the standards say they are designed to ensure that students graduating from high school have the knowledge and skills they need to enter college programs or the workforce. In general, do you favor or oppose these standards?”  (Public Policy Institute of California)

This must be the most biased poll ever done. Only 19% said they were very familiar with the common core. This description, devoid of meaningful detail, made them even less informed.

http://www.edweek.org/ew/articles/2014/04/24/694460_ap.html?cmp=ENL-EU-NEWS2

For an alternative, please see: Multiple Choice Quiz on US Education (The Progressive, 1/26/13)  S. Ohanian and S. Krashen http://sdkrashen.com/articles.php?cat=4

The brave new technology

Sent to the Wall Street Journal, April 22, 2014

I do not share the enthusiasm about new digital technology in education ("Founding principles in the digital age," April 22).

There is no evidence that the brave new online tests that promise "real time testing" will be any better than those we have now. In fact, we have not even seen them in action: According to Fair Test (fairtest.org), current common core tests are of the old multiple choice variety.

Nor is there evidence that digital tools that present lectures online and reduce interaction with teachers ("flipped classrooms") will improve achievement. This technology is being pushed on classrooms without proper research.

Instead of careful experimentation, the plan is to use nearly the entire student population as experimental subjects.

Jumping in without proper preparation wastes our students' time and will cost taxpayers much more money in the long run.

Stephen Krashen
Professor Emeritus
University of Southern California

original article: http://online.wsj.com/news/articles/SB10001424052702304279904579515444132790598

Lack of evidence for flipped classrooms: http://www.ascd.org/publications/educational-leadership/mar13/vol70/num06/Evidence-on-Flipped-Classrooms-Is-Still-Coming-In.aspx
Fairtest: http://www.fairtest.org/common-core-assessments-factsheet

A weapon of mass distraction

Response to (“We Need to Talk About the Tests,” Op-Ed, April 10), published under the heading 

"A Need to Go Public With Test Data"  

Published in the New York Times, April 16, 2014

Yes, there’s a big problem with the state tests. But there’s a bigger problem: the whole idea of the Common Core standards. Accurately described by Susan Ohanian, a writer and former teacher, as “a radical untried curriculum overhaul” and “nonstop national testing,” the Common Core is an outrageous scheme with no justification and no empirical support.

The problems described by Elizabeth Phillips will eventually be solved, or at least reduced enough to stop complaints from coming, but the Common Core boondoggle will continue, with new and very expensive tests delivered online.

I suspect that these bad tests are a weapon of mass distraction, so that we forget what the real problem is.

STEPHEN KRASHEN
Los Angeles, April 11, 2014

The writer is professor emeritus at Rossier School of Education, University of Southern California.

This letter and other responses: http://www.nytimes.com/2014/04/17/opinion/a-need-to-go-public-with-test-data.html?partner=rssnyt&emc=rss&_r=0

Original version:

Yes, there's a big problem with the tests ("We need to talk about the tests," April 10). but there's a bigger problem: The whole idea of the common core. Accurately described by Susan Ohanian as  “a radical untried curriculum overhaul and ... nonstop national testing," the common core is an outrageous scheme with no justification and no empirical support.

The problems described by Elizabeth Phillips will eventually be solved, or at least reduced enough to stop complaints from coming, but the common core boondoggle will continue, with new and very expensive tests delivered online.

I suspect that the tests are horrible on purpose, to encourage resistance and debate over  details. When changes are made, it will give critics a sense of accomplishment, while they forget what the real problem is. The lousy tests are a weapon of mass distraction.

Stephen Krashen

What scientific research says about bilingual education

Published in the Boston Globe, April 10, 2014 as "MCAS scores are no way to as­sess lan­guage pro­gram"

Not mentioned in the discussion of dual language schools in Dever ("Teachers, parents protest end of bilingual program," April 7) is the overwhelming evidence that bilingual education works.

MCAS scores are unscientific. They are influenced by many factors that have nothing to do with the the efficacy of bilingual education, eg. students changing schools, changes in the test,  changing levels of poverty, changing definitions of English proficiency, and teaching factors.  Scientific studies are set up to probe the effect of bilingual education, all other factors held constant.

Scientific studies consistently show that language minority students in bilingual programs outperform similar students enrolled in English-only programs on tests of English reading. 

Stephen Krashen

Source: McField, G. and McField, D. 2014. "The consistent outcome of bilingual education programs: A meta-analysis of meta-analyses." In Grace McField (Ed.) 2014. The Miseducation of English Learners. Charlotte: Information Age Publishing. pp. 267-299.

Story at: http://www.bostonglobe.com/metro/2014/04/06/dever-school-parents-teachers-assail-state-plan-scrap-dual-language-program/qfQea68Wy0qeV9Chs7jCiJ/story.html

Extensive reading: good for old folks

Sent to the Japan Times, April 5, 2014

Prof. Matthew Claftin's successful effort to improve the English book collections in the Iwakura Public Library is an important step forward in English Education ("Read up on ways that can help us learn English," April 6).  There is overwhelming evidence that extensive self-selected reading is the major source of advanced vocabulary and grammar in both first and second languages and leads to better reading and writing.

Prof. Claftin is right about the positive impact of extensive reading older people. Prof. Beniko Mason of Shitennoji University in Osaka has done a series of studies with former EFL students who engaged in extensive English reading on their own, after completing Prof. Mason's EFL class. All made excellent progress on the TOEIC test, making better gains per hour of reading than students typically do in standard TOEIC test-preparation classes. One of her subjects was 75 years old, another was 66.
Stephen Krashen

Original article: http://www.japantimes.co.jp/community/2014/04/06/issues/read-up-on-ways-that-can-help-us-learn-english/#.U0GYP8ded9I

Sources:
Overwhelming evidence: Krashen, S. 2004. The Power of Reading. Portsmouth, NH: Heinemann, and Westport, CONN: Libraries Unlimited (second edition).
Beniko Mason case studies:
Mason, B. (2011). Impressive gains on the TOEIC after one year of comprehensible input, with no output or grammar study. The International Journal of Foreign Language Teaching, 7(1).  http://www.tprstories.com/images/ijflt/IJFLTNovember2011.pdf
Mason, B. (2013). Substantial gains in listening and reading ability in English as a second language from voluntary listening and reading in a 75 year old student. The International Journal of Foreign Language Teaching, 8(1), 25-27.
Mason, B. (in press). The case of Mr. Kashihara: Another case of substantial gains in reading and listening without output or grammar study, Shitennoji University (IBU) Journal.

Testing Talk: A Weapon of Mass Distraction

Stephen Krashen

We are invited to give our opinions about the content of common core tests on a website called "testing talk" (http://testingtalk.org). We are not invited to discuss whether we need the tests or for that matter whether we should have the common core. For those who haven't been paying attention, the common core will impose more testing than has ever been seen on this planet, far more than is helpful or necessary. There is no research shwoing that increasing testing improves achievement, and the amount of money involved is staggering, especially the cost of technology, as the tests will be delivered online.

Those who accept the invitation to discuss the content of the tests will have the impression they have a seat at the table. In reality, invitations to discuss the standards and tests are a means of control, diverting attention from the real issues:

"The smart way to keep people passive and obedient is to strictly limit the spectrum of acceptable opinion, but allow very lively debate within that spectrum … That gives people the sense that there's free thinking going on, while all the time the presuppositions of the system are being reinforced by the limits put on the range of the debate" (N. Chomsky, The Common Good, p. 42, 2002)

The problem in American education is not a matter of getting the right tests. The problem is poverty. Our students from middle-class families who attend well-funded schools score at the top of the world on international tests. The US has the highest level of child poverty among all industrialized countries. If all our children were protected from the effects of poverty our overall international test scores would be spectacular.

Poverty means little health care, poor nutrition and little access to books and has a devastating effect on school achievement. The best teaching is ineffective when children are hungry, ill, and have nothing to read. The impact of poverty could be profoundly reduced if we invested more on food programs, health care, and libraries, instead of on useless standards and tests.

We have been told not to worry about these things but instead to debate about the details of the tests.  We can be sure that PARCC et al will repair the minor problems (e.g. boring passages, not enough time), but presuppositions of the system will be reinforced, as Chomsky predicts.

Testing talk is a weapon of mass distraction.

PS: I posted this note on the testing talk website: http://testingtalk.org/response/testing-talk-a-weapon-of-mass-distraction?messages[0]=response-create