Why learning data is easy to collect and hard to interpret
Language learning has moved online in several ways. Many people use language learning apps on their phones, others join video calls with conversation partners abroad, and many more pick up a language informally through subtitles, music and online communities. All of this leaves digital traces, and traces are tempting to count: downloads, daily active users, lesson completions, streak lengths, minutes of practice.
These counts describe behaviour in the app. They say much less about what a person can actually do in the language afterwards. That gap is the central issue in judging any claim about digital language learning.
What the common numbers actually measure
Downloads and sign-ups measure interest, or at least curiosity. Many people try an app once and stop, so a large download figure typically says little about sustained use or learning.
Active users depend on the definition. A “monthly active user” might be anyone who opened the app once in a month. Companies and analysts define these terms differently, so figures from different sources are often not comparable. This is the same problem that affects engagement metrics on social platforms generally.
Streaks and completion rates show habit formation, which matters because regular practice is generally believed to help. But a streak can be kept alive by the quickest possible exercise, and a completed lesson shows that the lesson was finished, not that its content was retained.
Time spent is a better indicator of effort than downloads, but time is not equal to learning. Fifteen minutes spent matching familiar words differs from fifteen minutes spent producing new sentences.
How learning is evaluated more directly
To say something about fluency, researchers need some form of language assessment. Several approaches are common.
- Proficiency tests. Standardised tests of reading, listening, writing and speaking can be given before and after a period of study. These come closer to measuring skill, though a test only samples what it asks about.
- Framework levels. Many programmes describe ability using graded levels, such as the widely used European scale running from beginner to advanced. Mapping an app’s content onto these levels is not always straightforward.
- Task-based assessment. Participants may be asked to hold a conversation, write a short message or follow a recording. Rating these involves human judgement, which adds some subjectivity.
The cleanest way to ask whether a tool helps is a comparison. One group uses the app, another group does something else or nothing, and both are assessed in the same way. Without a sensible comparison group, improvement could be due to many things: classes, travel, television, or simply the passage of time.
Common weaknesses in learning studies
Studies of learning tools often run into the same difficulties, and these are worth knowing when you read a headline.
Self-selection. People who download a language app and keep using it are usually motivated already. If they improve, motivation may explain it as much as the app. This is a selection effects problem, and it is why a comparison of “users” and “non-users” is often misleading unless participants were assigned randomly or matched carefully.
Dropout. Many learners stop early. If a study only follows those who finish, it can overstate typical results. Honest reports describe how many started and how many remained.
Self-reported progress. Learners asked whether they have improved tend to answer generously or inconsistently. Reliance on self-reported behaviour and self-assessment is common because it is cheap, but it can diverge from tested ability.
Short follow-up. Gains measured soon after a course may fade, and the reverse can also occur. Studies that follow people for months or years are rarer and more expensive.
Narrow measures. A test of vocabulary recognition may show gains while saying nothing about speaking or understanding fast speech. A study of one skill should not be summarised as general fluency.
Funding and publication. Studies of commercial products are sometimes carried out or paid for by the companies involved. That does not make them wrong, but it is reasonable to look for independent replication and to read the methods section for how outcomes were chosen.
Online exchange and what it adds
Conversation exchange platforms and video calls with speakers elsewhere add something an app lesson usually cannot: unscripted interaction with a real person. Researchers interested in this often look at how much learners actually speak, how partners respond to errors, and whether learners keep going after the first awkward sessions.
Exchange also carries cultural information. Learners pick up idioms, humour and norms of politeness that textbooks tend to skip. That overlaps with ideas from travel research about cultural immersion, except that here the immersion happens through a screen, and it is difficult to quantify. The usual evidence comes from interviews, diaries and observation of recorded sessions rather than from large datasets.
Limits apply here too. Exchange outcomes depend heavily on the partner, the topic and the time invested. A well-matched pair may learn a lot, and a mismatched pair may stop after a few weeks. Averages hide this variation.
Reading claims about learners and apps
A few questions help when you see a statistic about digital language learning.
- What was counted: sign-ups, active use, completed lessons, or tested ability?
- Was there a comparison group, and how were people placed in it?
- How many people dropped out, and are they included in the results?
- Was proficiency measured by an independent test, or by the learner’s own report?
- How long after the learning period was the measurement taken?
None of this means apps are useless. Regular, enjoyable practice is valuable, and an app that keeps someone coming back may do real good, particularly alongside speaking practice. The point is simply that engagement is a step toward learning, not proof of it, and the usual statistics describe the step far better than the result.