Using AI for Automated Scoring to Transform Language Examinations Launch of “PROGOS”, an innovative English speaking assessment system
RareJob, Inc. (hereinafter “RareJob”), which operates an English-related business, has developed “PROGOS”, an innovative English speaking assessment system. At the same time, we will launch a service called “the RareJob Speaking Test powered by PROGOS” for individuals and corporations, an English speaking test that utilizes this system.
(Left) Image of the test.
(Right) Image of test feedback from “RareJob Speaking Test powered by PROGOS”
Background of Development
In recent years, global business development by Japanese companies has become an important part of corporate activities aimed at achieving sustainable growth.
In addition, the coming of “with Corona era” has greatly accelerated the introduction of ICT in business situations. With the spread of remote work and the promotion of job-based human resources, there is a growing tendency to place greater emphasis on output and results than in the past. In addition, it will become difficult to communicate with face-to-face, including traveling abroad, and it is expected that collaboration and management with foreign employees will be required remotely. Given these social conditions, it is necessary to use English not only for overseas business trips and overseas postings, but also for every business situation, and the tendency to place greater emphasis on “being able to do business in English and deliver results” than before is expected to increase in the future.
However, while many Japanese companies recognize that recruiting and developing human resources capable of promoting global business expansion is a management issue, the reality is that they are still searching for ways to put them into practice. The reason for this is that even if we conduct English training, a common indicator for quantitatively measuring the speaking ability of English have not been established, and we have no choice but depending on qualitative judgments, such as experience and sensibilities. A survey of client companies’ training trends*, which we conducted, indicates that “no clear goals are set even with English training.” In addition, some companies that have introduced the existing language exam did not know how to utilize the results of the exam for training.
In this way, in order to develop English skills to make results in the global business, a consistent learning cycle of assessment, recommendation and learning is effective.
Towards Solving Issues
In order to solve these issues, RareJob has established two policies.
・ Establish and disseminate indicators that can quantitatively measure English speaking ability
・ Provide practical tools for managing the above indicators
In parallel with these policies, we aim to properly measure our English speaking abilities and to visualize, externalize, and commonly recognize business skills.
The standard for PROGOS is CEFR for foreign language communication skills, an internationally recognized standard. CEFR has been adopted in the English tests of Cambridge Assessment English, British Council, and other tests, and is widely used worldwide as a reliable standard. We have also adopted it for services such as the Smart Method® Course.
In addition, in order to provide high-quality test opportunities to a large number of customers quickly, easily, and with a reasonable price, we have adopted “scoring automation” using AI such as speech recognition and natural language processing. We also completed the development of technologies from AI engine to data linkage APIs in-house as it enable us to respond swiftly to customer and market needs and improve our service experience using data. In addition, we provide feedback on how individual learners can improve their English speaking abilities together with the CEFR-based results.
With the PROGOS system, English learners can check English speaking abilities regularly, with the common standard, and learn efficiently, just like heath management by weight scale. We believe that business people need to manage skills by themselves. PROGOS is the tool that enable business people to improve English speaking abilities.
About PROGOS https://info.progos.ai/
Overview | English Speaking Assessment System |
Content | Business English Communication tests |
Test format | Follow the prompts on the screen and automated voice to answer |
Test time | Approx. 20 minutes |
Examination method | Individual exam (PCs with microphone and speakers) |
Valuation form | Overall assessment: CEFR compliance (CEFR-J) level, Assessment by indicator: Evaluation according to 6 speech qualities defined by CEFR |
Scoring methodology | Automatic scoring by AI, and manual scoring by scorers |
(Origin of Name)
PROGOS=Progress + Diagnostic
Features of PROGOS
① CEFR-based quantitative assessment
② Feedback and advice for learning
③ Automatic scoring and quick return of test results
Supervised by the leading scholars in CEFR research and testing theories for test design and scoring quality
Leading person in Japanese CEFR research
Professor of Institute Global Studies at Tokyo University of Foreign Studies
Mr. Yukio Tono
・ Representatives of CEFR-J project
・ Lecturer at NHK Radio “Basic English 3”
・ NHK Educational TV
“Start in 100 words! English conversation” lecturer
Leading person in English testing theories
Professor of Institute Global Studies at Tokyo University of Foreign Studies
Mr. Masashi Negishi
Participated in the development of a variety of large-scale language tests and national English proficiency studies. Leading CEFR-J R&D projects with Mr. Tono, developing CEFR-J language-testing in particular
Providing PROGOS products
As the first phase of our PROGOS testing service, we will release the “RareJob Speaking Test powered by PROGOS” for individuals and corporations at the same time. As one of Japan’s largest providers of online English tutorial services, we can provide more effective English learning opportunities by combining measurement of English speaking ability and detailed learning advice. For Learning, i.e. with the positioning of testing for learning, we provide feedback sheets that describe specific advice on what learners can learn from the outcomes of PROGOS.
The prices are as follows.
Automatic scoring βversion (for individuals): ¥500 (excluding tax)
Manual scoring version (for corporate clients): ¥2980 (excluding tax)
Going forward, we aim at achieving a total volume of 1 million per year in three years. In the future, we are also considering the provision of PROGOS as a stand-alone system to other companies. We will propose new value to the language testing market while solving social issues through our assets of educational knowledge and cutting-edge technology in the English-related business that we have cultivated.
Service website of “RareJob Speaking Test powered by PROGOS”
For individuals: https://www.rarejob.com/experiences/speakingtest_progos/
For corporations: https://www.rarejob.com/corporate/speakingtest_progos/
* Source: Trend Survey on Corporate Global Human Resource Development (released in February 2020)
Question “Issues and concerns regarding general English training (including our services) (multiple responses),” 48% responded “no goal set for training”
Press Contacts
PR Department, RareJob, Inc.
Email: press@rarejob.co.jp