Word Generation-related Studies

Recent Research Findings

Word Generation is not only aligned with the established research base on effective instruction for supporting vocabulary, writing, and reading comprehension, but it has also demonstrated positive and significant results in several research studies.

Word Generation was most recently studied as part of the Catalyzing Comprehension through Discussion and Debate (CCDD) project, funded by the Institute of Education Sciences (IES) as part of the Reading for Understanding Initiative. The study, a large-scale randomized trial across four districts, was conducted during the 2012-2013 and 2013-2014 school years. Twenty-five schools were randomized within pairs that were matched on multiple variables (such as size, socio-demographic characteristics, etc.), and 7,773 students in grades 4-7 participated in the study. Fourth and fifth grade treatment classrooms were provided WordGen Elementary materials. Middle grades treatment classrooms were provided a selection of units from the original Word Generation program (now called WordGen Weekly) and grade-specific Social Studies Generation and Science Generation units (Jones, et al., under revision).

Numerous analyses were conducted using CCDD’s experimental evidence. Tables 1 and 2 include some overall results from the 2013-14 school year.
Data related to study
As these results show, elementary students made significant gains in taught vocabulary, perspective articulation and positioning skills, academic language skills, and deep reading comprehension, while students in the middle grades showed significant gains in taught vocabulary, perspective positioning skills, and deep comprehension. Larger impacts in the elementary grades can potentially be attributed to the greater “malleability” of younger students (Jones, et al., under revision). Another possible explanation is that implementation may have been easier in the elementary classrooms, where students generally remain with one teacher all day, than in the middle grades, where successful implementation is dependent on coordination among teachers in each of four content areas. Also, as interviews indicated, elementary teachers appreciated the new curricular resources, since they have little prescribed curriculum, whereas 6th and 7th grade teachers have prescribed curricula to cover and content standards to meet, making it more challenging to find sufficient instructional time for implementing WG (Jones, et al., under revision).

In addition, analysis of CCDD data shows specific benefits of Word Generation on deep reading comprehension, as measured by the Global Integrated Scenario-based Assessment (GISA). Academic language, perspective taking, and complex reasoning all explained small but significant amounts of variance in end-of-year GISA scores, suggesting that these domains should be given greater attention (LaRusso, Kim, Selman, Uccelli, Dawson, Jones, Donovan, & Snow, 2016).

More nuanced results arose from further analysis of the CCDD data. For example, the results reported above reflect the results obtained in the second year of the study. Impacts were stronger in year two than in year one, which makes sense given that, in the first year, teachers were introducing new practices, and student materials were still open to revisions in response to feedback (Jones, et al., under revision). Disaggregated results show that, in general, effects were larger in classrooms that implemented more of the curriculum.2 For example, students in the highest implementing elementary WG classrooms performed better in taught vocabulary (ES=.42, p<.001), perspective positioning (ES=.22, p<.01), and students in the highest implementing middle grades classrooms performed better on taught vocabulary (ES=.32, p<.001). Implementation challenges leading to variation in classroom implementation levels include things like lack of time in schedule, multiple new programs/curricula at once, and time lost to testing and test prep. These challenges are not unique to Word Generation; not only did control teachers report similar challenges, but they are common in implementation of Tier 1 programs more generally (LaRusso, Donovan, & Snow, 2016).

New Title

Additional analyses have yet to be published, but preliminary results indicate that Word Generation classrooms were rated as having significantly higher quality interactions than control classrooms for the following dimensions: Regard for Adolescent Perspectives, Content Understanding, Analysis and Inquiry, Quality and Feedback, Instructional Learning Formats, and Instructional Dialogue.3 Further, Word Generation teachers had these higher quality classroom interactions even when implementing other curricula, indicating that changes in improvements in teachers’ practices carried over even when Word Generation was not being implemented (LaRusso, Jones et al., SREE, 2016). Figure 1 illustrates these findings.

Figure 1: WG Impact on Classroom Interaction Quality:
Data related to WordGen
Preliminary results from the second year of the randomized control trial also suggest that the English-learners had significantly higher gains than their non-EL peers in social perspective articulation and academic language skills, suggesting that the Word Generation program can help students catch up to their peers on these dimensions (Kim et al., SREE, 2017).

Another set of preliminary analyses explores whether Word Generation impacts vary by students’ initial skill level. Results suggest that 4th and 5th grade students with higher initial levels of reading comprehension made greater gains on the GISA when they were in WG classrooms. However, these results contradict findings from the Word Generation efficacy trial (discussed below), which suggested that students with lower baseline vocabulary scores showed greater growth on standardized test scores than their peers (Lawrence, Francis, Paré-Blagoev, & Snow, 2016).
  • Earlier Findings

    The original Word Generation program was first studied as a quasi-experiment in partnership with Boston Public Schools from 2006-2009. At this time, only the original Word Generation units existed for the middle grades (6th-8th), and assessment of the program primarily focused on vocabulary acquisition. Results of this study indicated that students in schools implementing Word Generation outperformed students in the comparison schools on study-designed vocabulary tests; descriptive results imply that this difference is approximately equivalent to the difference between 8th and 6th graders on the pre-test (in other words, two years of incidental learning) (Snow & Lawrence, 2011; Snow, Lawrence, and White, 2009). Improvements on this study-designed test also predicted performance on the Massachusetts Comprehensive Assessment System (MCAS) ELA test (R2 = .486, p<.001) (Snow, Lawrence, and White, 2009). In addition, this study found that language minority students in Word Generation schools acquired academic vocabulary faster than their English-only classmates or than language minority students in control schools (Snow, Lawrence, and White, 2009). Furthermore, their advantage persisted in a follow-up assessment a year after instruction had ended (Lawrence, Capotosto, Branum-Martin, White, & Snow, 2012). On the other hand, another analysis (Lawrence, Rolland, Branum-Martin, Snow, 2014) found that students who were on the whole better readers improved the most from program participation.


    Following the initial Boston Public Schools study, the original Word Generation program was tested in a randomized trial across three large urban districts from 2009-2012. Generally referred to as the Word Generation efficacy trial, this study moved beyond vocabulary acquisition, with analyses also focused on classroom discussion quality. Results found that Word Generation classes had “dramatically” higher average discussion scores4 than control classes across all subject areas, particularly in math (Cohen’s d = 0.38-1.13) (Lawrence, Cross, Paré-Blagoev, Snow, 2015). The study measured little to no effect on vocabulary knowledge independent of the impact on discussion. The finding that improved discussion mediates treatment effects on student vocabulary learning is, to our knowledge, the first of its kind (Lawrence, Cross, Paré-Blagoev, Snow, 2015).5  Further analysis by Lawrence, Francis, Paré-Blagoev, and Snow (2016) found that, although there was no main treatment effect on the standardized tests, students in treatment schools who had low baseline vocabulary scores showed greater growth on standardized assessments than their peers.


    Data6 from the efficacy trial were also used in an analysis by Hwang, Lawrence, Mo, and Snow (2014) to explore differential effects of Word Generation for language minority students. Results reported a vocabulary learning advantage for newly reclassified English learners over English-only classmates; that advantage declined but remained significant for up to four years after reclassification, an effect that also persisted for reading comprehension (Hwang, Lawrence, Mo, and Snow, 2014). Analyses were also conducted on surveys targeting civic engagement and communicative self-efficacy (confidence to participate in discussions).7  Word Generation students reported both higher communicative self-efficacy on recently covered topics (Lin, Lawrence, Snow, & Taylor, 2016) and higher civic engagement than their peers (Lin, Lawrence, & Snow, 2015).

  • Notes

    1 Vocabulary was a measure of knowledge of word meaning from a sample of target words from the Word Generation curriculum. Perspective articulation and perspective positioning were measured using the SPTAM, which asks questions based on social scenarios commonly occurring in middle school. Academic language skills were measured using the Core Academic Language Skills Instrument (CALS-I), which focuses on linguistic features common in school content discourse but not necessarily common in casual conversation. Deep reading comprehension was measured using the GISA, an ETS scenario-based assessment that asks students to evaluate, integrate, and synthesize source materials for, as an example, a fictional class discussion.


    2 The level of implementation in Word Generation classrooms was calculated in two ways. Jones, et al. (under revision) did so by calculating levels of workbook exposure, defined as the total workbook activities with work done by any student in the class, and found exposure levels of 57% for fourth- and fifth-grade classes and 46% for sixth- and seventh-grade classes. LaRusso, Donovan, and Snow (2016), calculated classroom implementation levels using the average number of activities completed across all students in each class, finding average program engagement levels of 40.2% for fourth- and fifth-grade classes and 31.2% for sixth- and seventh-grade classes.


    3 These dimensions were evaluated using the Classroom Assessment Scoring System, Upper Elementary, which is an observational instrument used to assess classroom interactions (CLASS; Pianta, Hamre, & Mintz, 2012).


    4 Discussion quality was measured using a composite discussion quality rating from classroom observations.


    5 Lawrence, Crosson, Paré-Blagoev, and Snow (2015) only use data from year one of implementation, which only included two districts. Analyses in Lawrence, Francis, Paré-Blagoev, and Snow (2016) use data from year two of implementation, during which Word Generation was implemented in an additional (third) district.


    6 The authors only used data from one participating district in order to obtain a sample with enough English learners.


    7 The authors only administered surveys in one participating district to conduct these analyses (due to geographic proximity).

  • References

    Jones, S. M., LaRusso, M., Kim, J., Kim, H. Y., Selman, R., Uccelli, P., Barnes, S., Donovan, M. S., Snow, C. (under revision). Experimental effects of Word Generation on vocabulary, academic, language, perspective taking, and reading comprehension in high poverty schools.


    Kim, H. Y., LaRusso, M., Jones, S. Donovan, S., & Snow, C. (2017). Reducing the Academic Inequalities for English Language Learners: Variation in Experimental Effects of Word Generation in High Poverty Middle Schools. Presented at Society for Research on Educational Effectiveness, Washington, D.C.


    LaRusso, M., Donovan, S., & Snow, C. (2016). Implementation challenges for Tier One and Tier Two school-based programs for early adolescents. In B. Foorman (Ed.), Challenges to implementing effective reading intervention in schools. New Directions for Child and Adolescent Development, 154, 11–30.


    LaRusso, M., Jones, S. M., Kim, H. Y., Kim, J., Barnes, S., Donovan, S., & Snow, C. (2016). Impacts of a discussion-based academic language program on classroom interactions in 4th through 7th grades. Presented at the Society for Research on Educational Effectiveness, Washington, D.C.


    LaRusso, M., Kim, H. Y., Selman, R., Uccelli, P., Dawson, T., Jones, S., Donovan, M. S., & Snow, C. (2016). Contributions of academic language, perspective taking, and complex reasoning to deep reading comprehension. Journal of Research on Educational Effectiveness, 9(2), 201-222.


    Lawrence, J. F., Capotosto, L., Branum-Martin, L., White, C., & Snow, C. E. (2012). Language proficiency, home-language status, and English vocabulary development: A longitudinal follow-up of the Word Generation program. Bilingualism: Language and Cognition, 15(03), 437-451.


    Lawrence, J. F., Crosson, A. C., Paré-Blagoev, E. J., & Snow, C. E. (2015). Word Generation randomized trial: Discussion mediates the impact of program treatment on academic word learning. American Educational Research Journal, 52(4), 750-786.


    Lawrence, J. F., Francis, D., Paré-Blagoev, E. J., & Snow, C. E. (2016). The poor get richer: Heterogeneity in the efficacy of a school-level intervention for academic language. Journal of Research on Educational Effectiveness. http://dx.doi.org/10.1080/19345747.2016.1237596


    Lawrence, J. F., Rolland, R. G., Branum-Martin, L, & Snow, C. E. (2014). Generating vocabulary knowledge for at-risk middle school readers: Contrasing program effects and growth trajectories. Journal of Education for Students Placed at Risk (JESPAR) 19(2), 76-97.


    Lin, A. R., Lawrence, J. F., & Snow, C. E. (2015). Teaching urban youth about controversial issues: Pathways to becoming active and informed citizens. Citizenship, Social and Economics Education, 14(2), 103-119.


    Lin, A. R., Lawrence, J. F., Snow, C. E., & Taylor, K. S. (2016). Assessing adolescents’ communicative self-efficacy to discuss controversial issues: Findings from a randomized study of the Word Generation program. Theory & Research in Social Education, 44(3), 316-343.


    Snow, C. E., & Lawrence, J. F. (2011). Word Generation in Boston Public Schools: Natural history of a literacy intervention. The Senior Urban Education Research Fellowship Series. Volume III. Council of the Great City Schools.


    Snow, C. E., Lawrence, J. F., & White, C. (2009). Generating knowledge of academic language among urban middle school students. Journal of Research on Educational Effectiveness, 2(4), 325-344.


    Uccelli, P., Galloway, E. P., Barr, C. D., Meneses, A., & Dobbs, C. L. (2015). Beyond vocabulary: Exploring cross‐disciplinary academic‐language proficiency and its association with reading comprehension. Reading Research Quarterly, 50(3), 337-356.

Development of Word Generation was led by Catherine Snow (Harvard University) and Suzanne Donovan (SERP). Major SERP contributors to program development include: Claire White, Alyse Krantz, Halley Wheeless, Matt Ellinger, David Dudley, and Patrick Hurley. Boston Public Schools and other districts in Massachusetts and Maryland collaborated with SERP to develop Word Generation.


Support for Word Generation was provided by the Carnegie Corporation of New York, the William and Flora Hewlett Foundation, the Noyce Foundation, the Spencer Foundation, the Leon Lowenstein Foundation and the Institute of Education Sciences, U.S. Department of Education through grant numbers R305A090555 and R305F100026. The information provided does not represent views of the funders.