In the last few years, math curriculum provider ASSISTments ran into a major problem: The in-person, one-on-one coaching it provided teachers was proving too expensive and not scalable.
The issue emerged as demand for extra support in math reached an all-time high, amid widespread student struggles in the subject following the pandemic.
To keep up, the free curriculum-based platform explored swapping out the traditional face-to-face professional development in favor of a virtual professional learning community.
But the change raised questions about how the all-digital experience would impact the overall success of the product, and its ability to generate gains in students’ math scores.
During a panel at the ISTE+ASCD conference on Monday, researchers from WestEd walked through the rigorous study the research organization conducted on ASSISTments after it moved from an in-person to a virtual PD model.
The results, which were finalized in the last few months, found that the product moved the academic needle for students — albeit not dramatically — especially for low-performers.
When coupled with other studies WestEd has run, the organization concluded that “virtual professional learning communities work,” Research Director Linlin Li said in an interview with EdWeek Market Brief after the panel.
Switching to virtual professional development may be of particular interest to vendors looking for ways to keep costs low for their school district clients. Many districts are currently grappling with tight or uncertain budgets due to major forces in the market, including enrollment loss, shrinking state allocations, and uncertainty about the future of federal funding.
The idea is especially appealing in rural communities, Li said, where bringing people together face-to-face in the same room may require long drives or expensive travel. Teachers in those districts may also be the only math educators in their grade level or otherwise isolated from their peers in their subject areas.
The Results
After a year of use, WestEd found that students at all academic performance levels who used ASSISTments overall scored slightly better than those who didn’t, although not by a large enough margin to be statistically significant.
There was a statistically significant difference, however, between how low-performing students using ASSISTments performed versus those who didn’t. Which was also true when looking only at students in rural areas.
The ASSISTments program asks students a series of math questions, with the goal of providing information about where they are stuck and answering questions correctly. Teachers can access data on commonly submitted incorrect answers, which questions were skipped, and how often they used a hint.
There was a positive correlation between teachers who used this data to adjust their practice and how students scored, the study found. Meaning, the teachers who put the data into action had students perform better.
Panelist Ed Dieterle, founder of Education Research Partners, theorized that the results are an indication of the quality of the virtual professional development provided.
The tie between teachers’ use of data and student success suggests “professional development should emphasize, not just the tool we use,” he said, “but specifically how to interpret the data and how do we keep it usable within the classroom?”
In addition, the fact that so many teachers also participated in ASSISTments virtual community throughout the course of the study signals its popularity, Li said. WestEd defined the amount of training it hoped educators would utilize, and almost all of the participating teachers reached that bar.
“If it doesn’t help them … they won’t join,” Li said. “But they actually joined the community.”
Solving the Control Group Challenge
In order to conduct the study, WestEd had to navigate a common hurdle in education research, which is that it’s difficult to put together a control group. Schools generally balk at the idea that some students will get access to a resource while others don’t get the same opportunity because they were chosen as a control.
Post-pandemic, WestEd said their organization struggled to find teachers who would participate. As a solution, they opted for a virtual control group, which pulled students from different schools with similar demographics.
Join Us In Person at the EdWeek Market Brief Fall Summit
Education company officials and others trying to figure out what’s coming next in the K-12 market should join our in-person summit, Nov. 3-5 in Denver. You’ll hear from school district leaders on their biggest needs, and get access to original data, hands-on interactive workshops, and peer-to-peer networking.
However, it’s worth noting that this could be one reason the results are less stark.
Unlike a traditional research approach, in which the control group would’ve received no math supports, it’s likely given the ubiquity of ed tech that students in the control group had access to some form of support aimed at boosting their math scores. They just didn’t use ASSISTments.
K-12 companies looking to do research on their products should consider this workaround for creating a control group, Li said, rather than letting their research project hit a wall.
“Either in the treatment or in the control [group], you see the overall contrast [between the two] has changed,” Li said.