We caught up with Dr. Roumen Vesselinov, PhD, at City University of New York who has been researching language learning app effectiveness since 2008. In his latest paper published earlier this year, results showed that just 19 hours of 1-on-1 language study on italki is the same as a full college semester of language learning - now that's pretty cool.

“Since 2008 we have completed nine language studies on efficacy and motivation for Rosetta Stone, Duolingo, Busuu, Babbel, Hello English, and italki. Our research team conducts the study, collects the data and performs the statistical analysis independently. So far the italki efficacy is the best as a point estimate.”

- Dr. Roumen Vesselinov, PhD at City University of New York

For those of you on a tight schedule, here’s your 20 second takeaway:

What is this?

The efficacy study measures the language progress (language used for consideration - Spanish) per one hour of study between different language learning apps.

Why does this matter?

Learners of foreign languages are bombarded with unsubstantiated advertisements claiming the impossible. This study is an objective and independent measure of the effectiveness of italki compared to other language learning apps.

How does this help you?

If you want to find the most effective method to learn a language (outside of getting on a plane and flying to the country itself), you’ve found it. Just 19 hours of 1-on-1 learning on italki is proven to be equivalent to one college semester of classes which is better than any other language learning app!

Read on below for the full, unedited interview with Dr. Roumen Vesselinov, PhD on the study:

Q: What is the background of this research and why was it conducted?

A: Since 2008 our research team has worked on evaluating the efficacy of language apps. Our particular focus is on measuring the progress per one hour of study.

Learners of foreign languages are bombarded with unsubstantiated advertisements stating that you can “learn foreign language in ten days”, etc. These claims do not elaborate what exactly you're going to learn for ten days and how much you'll have to study; is it ten minutes a day or ten hours a day? Our goal is to have an objective and independent measure of the efficacy of each language app.

Q: How are these language studies conducted?

A: All our studies are based on a well-established statistical impact study design. We measure the language level of the participants in the beginning of the study, then let them study for about two months, and measure their level at the end of the study.

Q: What kind of tests are used?

A: We use well-known language tests developed by independent testing companies.

The first instrument for evaluating the level of knowledge is the Web Based Computer Adaptive Placement Exam (WebCAPE). It is an established university placement test and it is offered in ESL, Spanish, French, German, Russian, and Chinese. It was created by Brigham Young University and is maintained by the Perpetual Technology Group.

The second test used in the studies is the Oral Proficiency Interview by Computer® (OPIc) created by Language Testing International (LTI). LTI is the exclusive licensee of the American Council on the Teaching of Foreign languages (ACTFL).

Q: How is the efficacy measured?

A: The efficacy measure includes both the amount of progress made by each study participant and the amount of their effort. This is a direct and objective measure of efficacy. Direct, because it includes directly the effect and the effort. Objective, because the effect is measured by an independent college placement test (WebCAPE) and the effort is measured by the actual study time for each participant.

The efficacy is computed by dividing the difference between the final and initial WebCAPE score by the study time in hours. The results show the number of points gained per one hour of study.

Q: How can we interpret the efficacy measure?

A: WebCAPE college placement test guidelines state that if a student’s score is below 270 points he or she will be placed in first semester course. We use this cut-off point of 270 and divide it by the average efficacy (points per one study hour). For example, a result of 20 would mean that on average one would need 20 hours of study during a two month period to cover the requirements of one semester of college study.

Q: Who were the study participants and how would they define their experience with the language app?

A: Study participants are randomly selected after advertising about each study. In some studies the participants were from around the world and in some studies they were only from the U.S. The requirements specified that participants were at least 18 years of age and not advanced learners of the language. The reason for the latter is that the main test is a college placement test which is better suited for beginners. At the end of each study we asked the participants to evaluate their experience with the language app. In general, users were very satisfied with their foreign language study experience.

Q: What factors affect the language learning?

A: In all studies the results were not affected by age, gender, education, native language, etc. The only important factor is the initial level of knowledge; true beginners show the biggest progress per one hour of study. More advanced learners have slower progress.

Q: How independent are these studies?

A: Since 2008 we have completed nine language studies on efficacy and motivation for Rosetta Stone, Duolingo, Busuu, Babbel, Hello English, and italki. Each respective company covers the expenses of the study, particularly the language tests. Our research team conducts the study, collects the data and performs the statistical analysis independently.

Q: What are the results of the efficacy studies so far?

A: All studies use the same methodology so in general the results can be compared, with one exception. In the 2009 Rosetta Stone study, the hours of study were self-reported while for the rest of the studies we collected objective computer time from the servers.

The hours necessary to cover the requirements for one college semester varied between 19 and 30 hours during two-month study period. So far the italki efficacy is the best as a point estimate.

So there you have it! We'll let the numbers speak for themselves. In case you missed it, here’s a link to The italki Efficacy Study for you to read.