Flocabulary?

Exaggerated Claims of "Educational Hip-Hop": A Review of Flocabulary Research

Stephen Krashen

http://languagemagazine.com/?page_id=6649

Advertised as "educational hip-hop" (flocabulary.com), Flocabulary is a direct instruction program designed to help middle school students increase their academic, or "tier 2" vocabulary. It consists of units, each containing 15 target words likely to appear on state tests, with each word presented eight times in the form of "traditional exposures like reading passages and use in sentence," rap songs (see appendix for an example), charades, and writing, "using the word in context" (Rappaport, p. 21).  (For details, see http://www.flocabulary.com/how-it-works/).

I have found four empirical studies of Flocabulary. In no case was Flocabulary compared to reading, and in no case are the results impressive. Three of the four were supported by the company that owns Flocabulary and are available at the Flocabulary website.

Study 1: "Outcomes assessment report of the 2009 Campus Summer Flcabulary Word Up Program"  http://flocabulary.s3.amazonaws.com/pdfs/flat/asany-assessment-flocabulary-2009.pdf

Ninety-seven children, most (92%) between ages 10 and 13, participated in a ten week summer program that included "Flocabulary" for two hours per session, a total of 20 hours covering 60 "mature" vocabulary words.

Participants were given and pre- and post-tests on 30 of the 60 words and improved from 13.64  correct on the pre-test to 20.71 on the post-test. The participants did a lot better than comparison students, who studied "robotics" instead of vocabulary, but these gains are not impressive in terms of the number of words acquired. If Flocabulary students gained seven points (actually 7.07) on a test of 30 of the words, it is likely that they gained about 14 on the full set of 60. That's 14 words gained from 20 hours of work, less than one word per hour of instruction.

Study 2:  The Word Up Project: "A study of flocabulary's The Word Up project on increasing vocabulary achievement." November 2009. Educational Research Institute of America, Report 371. http://flocabulary.s3.amazonaws.com/pdfs/flat/efficacy-study-summary.pdf

This study examined the impact of Flocabulary in schools in six different states. The treatment consisted of seven Flocabulary units.  Each unit presents 15 target words, and the duration of the study was "for an entire academic year" (p. 23), which means that students were exposed to a total of 540 words (15 * 36).

For all students combined, students scored an average of 53.5% on the pretest covering 40 words and 78.1% correct on the posttest, which was identical to the pretest, a gain of about 25% (24.6). Extrapolating to 540 words, this means that students doing Flocabulary learned an average of about 135 words in one year (.25 * 540). 

Flocabulary claimed that this study showed "a dramatic increase in vocabulary proficiency" (http://www.flocabulary.com/results/).  This is clearly not the case.

Table 1 summarizes the results for each subgroup.

Table 1: Estimated gains for a full academic year using Flocabulary

group	n	Pre-test % correct	Post-test % correct	% gain	Gain in words	Estimated gain in words for one year
Overall	599	53.5	78.1	24.6	9.84	132.84
Alabama	89	51.5	76.8	25.4	10.16	137.16
California	70	42.3	70.7	28.4	11.16	153.36
Mass	87	65.4	81.6	16.2	6.48	87.48
New York	76	65.5	80	14.5	5.8	78.3
PA	26	74	91.4	17.4	6.96	93.96
Texas 6	41	38	74	36	14.4	194.4
Texas 7	111	55.2	75.5	20.3	8.12	109.62
Texas 8	98	43	78	35	14	189
HIGH SES	266	57.6	78.8	21.2	8.48	114.48
LOW SES	333	50.2	77.5	27.3	10.92	147.42

Study 3: A Study of Flocabulary's The Word Up Project Program's Influence on State Reading/Language Arts Test Scores: A Treatment and Control Group Design."  Educational Research Institute of America, Report 371.Report 372, October 2009. http://flocabulary.s3.amazonaws.com/pdfs/flat/state-test-study-summary.pdf

This study was a different analysis of the students involved in study 2. In five of the eight schools, the claim is made that more Flocabulary students scored in the highest (or second highest) category of state exams at the end of the year than comparison group students. In the other three, Flocabulary students did not do better, but we are informed that the comparison groups were not equivalent to the Flocabulary group. No data is provided. 

Of the five cases in which Flocubary students were superior, two look convincing.

In the Texas school studied, there were 39 students in each group. Of the Flocabulary group, 59% scored above 2100 on the TAKS reading test (23 students) and none of the comparison students did. Also, only 13% (5) students did poorly on the TAKS and 69% (27) of the comparisons did. This school also made the largest gain on the vocabulary test given at the end of the year (see above).

Table 2: Performance on end-of-year exams – Texas 6

	n	below 2000	2001-2099	2100 +
Flocabulary	39	13% (5)	28% (11)	59% (23)
comparison	39	69% (27)	31% (12)	0

Test: TAKS reading test

California's data is also impressive and it is interesting that California made the second best gain of all schools on the Flocabulary vocabulary test (see above).

Table 3: Performance on end-of-year exams - California

	N	below 250	250-299	300 +
Flocabulary	64	14% (9)	36% (23)	50% (32)
comparison	64	17% (11)	81% (52)	2% (1)

Test: STAR Vocabulary Assessment

In Alabama, there was no advantage for Flocabulary in the "exceeds standards" category, but more Flocabulary students met standards, thanks to the performance of 16 students.

Table 4: Performance on end-of-year exams - Alabama

	N	Doesn't meet, partially meets Stds	meets Stds	exeeds Stds
Flocabulary	82	0	66% (54)	34% (28)
comparison	98	25% (25)	39% (38)	36% (35)

Test: Reading portion of ARMT (Alabama Reading and Math Test)

The Flocabulary advantage in New York is due to the performance of students in the 670+ group of the NYSELA (English Language Arts) test. Of the Flocabulary group, 38% scored over 670 (34 students) and 20% of comparisons did (24 students).  The Flocabulary advantage in New York is thus due to the performance of 10 students. The New York Flocabulary students made the lowest gain on the vocabulary test of all the groups.

Table 5: Performance on end-of-year exams – New York

	n	below 650	650-669	670+
Flocabulary	89	17% (15)	45% (40)	38% (34)
comparison	119	18% (21)	62% (74)	20% (24)

Test: NYSELA tests (NY State English Language Arts)

In Pennsylvania, 23% of the Flocabulary group, 23% (6 students) exceeded the 1700 level on the end of year state assessment, and 11% (3 students) of the comparisons. Thus, the Flocabulary advantage at the highest level is based on only three students. At the intermediate level, it is based on the performance of only four students.

Table 6: Performance on end-of-year exams - Pennsylvania

	below 1500	1500-1600	above 1700
Flocabulary	23% (6)	54% (14)	23% (6)
comparison	54% (14)	37% (10)	11% (3)

Test: PSSA (Pennsylvania System of School Assessments: Reading)

Thus, of the eight groups, the Flocabulary impact was only obvious in two. In three others, differences were the result of performance by a small group of students, and in three other studies, there was no difference.

Previous year's state test scores were not provided. This would give us some idea of how much gain there was over the academic year. Also, it is difficult to believe that such modest gains in vocaublary in Texas and California, as shown in study 2, could result in profound differences on standardized tests of reading, unless Flocabulary was lucky enough to target just those words that appeared on the Texas and California state tests.

The data is suggestive, but is clearly insufficient to support Flocabulary's claim that "Flocabulary is proven to raise scores on state reading tests" (http://www.flocabulary.com/results/).

4. Martin, H. 2011. Affects (sic) of using Flocabulary: A Rap Based Vocabulary Program on Middle School ELLs. MA Thesis, University of Wisconsin-Whitewater.

Students in this study were middle school seventh graders, some ELL and some non-ELL, and the duration was eight weeks. Martin focused used only on 10 of the 15 words in each unit; five words were not targeted because teacher felt students already knew them or the word was infrequent. Each session lasted about 20 minutes.

Performance on quizzes given at the end of each unit ranged from 6.5 words correct to 9.59 words correct for non-ELLs and from 6.03 to 9.72 correct for ELLs. (table 4.1 Non-ELLs learned about 8.5 words per week (sd = 1.3) and ELLs learned about 8.3 words (sd = 1.5). If the program were done all year (36 weeks) and students remembered all the words they got right on quizzes, this would amount to a gain of slightly more than 300 words a year or about 5 words per hour.

SUMMARY AND CONCLUSIONS

In study 1, it was estimated that students learned about 14 words in 20 hours during the summer. In study 2, it was estimated that students learn about  135 words in one academic year.  In study 4, students learned about 5 words per hour, assuming no forgetting and extrapolating from eight weeks to a full year.

These results are far less than the rate of vocabulary acquisition from reading, calculated to be about .25 words per minute (Nagy, Herman and Anderson, 1985) or from listening to stories (Mason and Krashen, 2004).  It has been estimated young people in school acquire several thousand words per year, and little of this comes from instruction (Nagy and Herman, 1987).

Of course readers gain far more from reading than just vocabulary; they improve in writing, grammar, spelling, and in academic knowledge as well as knowledge of the world (Krashen, 2004).

It could be argued that Flocabulary is worthwhile because it focuses on more academic vocabulary, which students may not encounter in "light" reading. But light reading contains a surprising number of academic words (Krashen, 2013), Also, those who do light reading gradually expand their choices over the years (LaBrant, 1958), and there is good evidence that extensive self-selected reading provides the linguistic basis that will make more demanding texts comprehensible (Krashen, 2013). "Light," self-selected reading is the bridge between conversational and academic language.

References

Krashen, S. 2004. The Power of Reading. Portsmouth, NH: Heinemann, and Westport, CONN: Libraries Unlimited (second edition).

Krashen, S. 2012. Developing academic proficiency: Some hypotheses. International Journal of Foreign Langauge Teaching, (2): 8-15. (available at ijflt.com)

Krashen, S. 2013. Reading and vocabulary acquisition: Supporting evidence and some objections.  Iranian Journal of Language Teaching Research, 1(1): 27-43, 2013.

LaBrant, L. (1958). “An evaluation of free reading.” In C. Hunnicutt and W. Iverson (Eds.), Research in the Three R’s (pp. 154-161). New York: Harper and Brothers.

Mason, B. and Krashen, S. 2004. Is form-focused vocabulary instruction worthwhile? RELC Journal 35(2): 179-185.

Nagy, W. and P. Herman. 1987. Breadth and depth of vocabulary knowledge:

Implications for acquisition and instruction. The nature of vocabulary acquisition,

ed. M. McKeown and M. Curtiss. Hillsdale, NJ: Erbaum. pp. 19-35.

Nagy, W., Herman, P., and Anderson, R. 1985. Learning words from context.

Reading Research Quarterly 17: 233-255.

Rappaport, A. 2013. Closing the vocabulary gap. Language Magazine. 12(10): 20-23.

Appendix: A Flocabulary rap with ten target words

 (http://www.flocabulary.com/5-were-going-to-ride/)

Unit 5 (for third graders)  "We're Going to Ride"

"This song includes third grade vocabulary words that students are likely to encounter on state tests. It teaches the following words: active, coast, device, drift, gradual, marsh, nation, rate, schedule and swift. This song tells the true story of four guys who take a cross-country road trip . . . on skateboards!"

You know what I like? I like to skate,
I don’t mean on ice skates or Rollerblades.
I mean a skateboard, it’s a nice device,
It’s a cool invention; it’s nice and light.
As I got older, I thought it’d be amazing
To skate across the whole country, the nation.
So, me and my amigos packed up some Cheetos,
Got on our skateboards, figured we’d be heroes.
We lived in Oregon, so that’s where we would begin,
We had a schedule, so our time was planned.
We would kick, push and coast from coast to coast,
And we knew where we were supposed to go.
The going was slow, and at this rate,
At this speed, it would take us more than 20 days.
But that’s cool, we didn’t care, ’cause the feeling of
The wind in our hair was fresh like mountain air.

We’re going to ride… "Come on, take a ride."
Everybody take a ride with me. "Come on."
We're going to ride... "Take a ride, y'all."
Everybody take a ride with me.
We're going to ride... "Take a ride."
Everybody take a ride with me.
We're going to ride. "Take a ride."
Everybody take a ride with me. "Yeah! Come on!"

We saw a cactus in the desert where it’s extra dry,
The skating kept us active, ’cause it’s exercise.
But slowly, over time, gradually,
I’d get so weak that I could barely keep going.
At the top of the mountains, it was snowing,
Frozen flakes fell, the wind was blowing.
Imagine flying down a mountain on a skateboard,
I nearly hit a tree, I moved so swiftly and quickly.
Sometimes the wind would push my board,
Back and forth, and I would drift.
And one time, I even saw a wild horse
Run alongside my board, and it was hip.
We traveled east through wet swamps and marshes,
People in town would say, "Oh my lord it’s
Four guys skating so far, I can’t believe it,"
Well, this land is beautiful. Y’all should step outside and see it!

We’re going to ride… "Come on, take a ride."
Everybody take a ride with me. "Come on."
We're going to ride... "Take a ride, y'all."
Everybody take a ride with me.
We're going to ride... "Take a ride."
Everybody take a ride with me.
We're going to ride. "Take a ride."
Everybody take a ride with me. "Yeah! Come on!"