Simplification and Defaults Affect Adoption and Impact of Technology but Decision Makers Do Not Realize It
Researchers
* Peter Bergman | UT-Austin
Jessica Lasky-Fink | Harvard University
Todd Rogers | Harvard University
Scope
→ 7,000 Families
→ 12 Middle and High Schools in the District of Columbia Public Schools (DCPS)
→ Academic text messaging field experiment conducted for a school year using a Learning Management System (LMS)
Key Findings
School districts committed to empowering families with actionable, accessible information in support of student success should consider not only what digital communications tools they use, but how they are made available.
When parents and caregivers were told by schools to manually “opt-in” to automated text messages about their child’s academic progress, only 1% of families did. On the flip side, automatic, “opt-out” enrollment resulted in 95% of families adopting the new technology. Among the families adopting the technology, their students reaped the evidential academic benefits of the program.
Other Findings
Compared to students whose families were asked to opt in, those whose families were automatically enrolled scored higher GPAs, by an average of 0.06 points. Their course failures were also 9% lower (in other words, one in five students did not fail a class they would have otherwise).
Families who were automatically enrolled were more likely than others to say they would use the program in the future.
District decision-makers were unaware of how much automatic, opt-out enrollment boosts participation. Once made aware, they reported being willing to spend 140% more per student to transition to automatic, opt-out enrollment and recapture the intended benefits of the initial investment.
Methodology & Data Highlights
Field experiment involving 12 middle and high schools in the DCPS system (N = 7,000 student households)
Online survey of 130 decision-makers in 55 school districts
Summary
In several studies conducted in partnership with school districts, researchers have found that automated, inexpensive text messages to parents can boost student attendance and grades. [1] Parents tend to be overconfident about their child’s effort in school, the research finds, but they are able to motivate their children if given more frequent updates on grades, missed assignments, and classes.
Given the potential for these digital tools to boost student outcomes and the financial investment required to implement new technology, it’s important that districts consider barriers to their uptake. Different methods of enrollment might affect student outcomes. Which enrollment method(s) help overcome barriers to ensure all students and families have access?
This paper examines the implementation of an automated text message program across twelve middle and high schools in the District of Columbia Public Schools (DCPS). In a randomized field experiment that tested different methods for enrolling parents in the program, the researchers found that families were significantly more likely to participate if they were automatically enrolled, and that automatic enrollment was the best method for helping lower-performing students reap the benefits of the program.
[1] Learning Collider and its researchers use the term parents as inclusive of all caregivers and/or legal guardians in students’ households.
How the experiment was designed
The D.C. program sent weekly, automated text messages to parents over the course of one school year about low grades, missed classes, and missed assignments. Data from the district’s digital gradebook and student information system fed into the messages.
Parents in the twelve schools were randomly offered one of three procedures for enrolling in the text message program:
Parents could sign up online. Parents received a text message telling them they could enroll in the program on the school district website. (At the time of the study, this was the district’s current enrollment practice.)
Parents could opt-in via text message. Parents received a text message saying they can enroll by replying “start.”
Parents were automatically enrolled but could opt-out. Parents received a text message saying they were already enrolled and could opt-out by replying “stop.”
Automatic enrollment boosted grades and put schools in regular contact with families
Only 1% of families in the first group enrolled in the program and adopted the new technology. About 11% of parents in the second group enrolled. However, the third method, in which parents were automatically enrolled, resulted in an incredible 95% adoption rate.
Importantly, automatic enrollment not only increased the number of families that signed up but affected those who participated. Parents in the first two groups who opted-in to the program tended to have higher-achieving children and be more engaged in their child’s education to begin with, as measured by prior engagement with the student portal. In contrast, automatically enrolling parents helped schools engage families with lower-performing students–something districts often struggle with.
At the end of the school year, students in the automatic enrollment group had - on average - higher grades and fewer course failures than students in the first two groups:
0.06 point increase in GPA per student, the equivalent of each student boosting a C+ to a B- in one course
0.2 course reduction in course failures per student, the equivalent of 1 in 5 students not failing a course they might have otherwise failed
Figure 3: Percentage of parents in each condition who received alerts each week (vertical black lines indicate school holidays/breaks)
Ultimately, given the positive effects of the program on academic outcomes and the inability of the opt-in method to reach families with lower-performing students, how districts enrolled families mattered in terms of equity and efficiency.
Simple enrollment results in greater parent satisfaction
Simple enrollment options also made parents more likely to request the technology moving forward. In a school district survey, parents who were able to opt-in or opt-out via text message were more likely to say they’d like to use the technology the next school year than parents who had to sign up on the district’s website.
Given the potential for communication tools to boost academic achievement, districts concerned about equity in student resources should consider not only what tools they make available, but how they make them available.
All of these findings show how critical it is for school district decision-makers to understand that when it comes to the adoption and effectiveness of new technologies, roll-out matters – for financial investment and the intended efficacy.
Opt-out should be the default enrollment method; when provided with evidence, school districts adapt their enrollment systems
Fortunately, school decision-makers surveyed by the researchers showed a strong willingness to adapt their enrollment systems.
In a survey of 130 superintendents, principals, and family engagement coordinators in 55 school districts, the researchers show that these decision-makers vastly overestimate how likely families are to sign up for a new tool on the district’s website and vastly underestimate how many families would stay enrolled if automatically enrolled.
About 80 percent of respondents said they currently enroll parents in programs that are similar to the one studied in this paper via an opt-in method. Once they learned the real take-up rates, however, respondents were willing to pay 140% more per student to shift to an opt-out system of automatic enrollment.
*Peter Bergman is the founder and director of Learning Collider. This research is published in Organizational Behavior and Human Decision Processes, May 2020.