Build Winning Teams with iMocha

The Making of AI-EnglishPro: Our AI-Based Business English Proficiency Test

Read More

Company News, Diversity & Inclusion, iMocha Engineering Product Updates Remote Hiring Skills Assessment

All Posts
15 December, 2020

The phrase Artificial Intelligence, sweetly abbreviated as AI when put in front of a product, raises the expectations from a product beyond limits.  

During the early pre-launch surveys with our clients on our AI-EnglishPro, we knew that our clients expected magic, and we had to provide the magic. Obviously, no rabbits were getting conjured from black hats but surely there was a problem to be solved: a reasonably accurate score of a candidate’s English Proficiency within reasonable time as a candidate speaks, writes, reads, and listens.  

The other challenge was that our test, with an ability to automatically evaluate a candidate’s proficiency, would be compared against other manual variants of it. Unlike in some cases, wherein the prefix of AI is used as a marketing gimmick, we genuinely wanted to increase the trust of our Hiring Managers, Recruiters, Learning & Development Leaders, and other stakeholders in AI. We also wanted to ensure that AI actually helps them cut down their hiring or upskilling assessment time.  

We started breaking the of a mammoth task at hand. While one team of experts started deep-diving on CEFR Compliance, another worked on the content of the initial variation of the test. The product team worked on the UX and candidate experience; and we, the Engineering team, started brainstorming the AI techniques, Reporting Engine, APIs, and the overall Integration and Architecture 

Being an English Proficiency Test, surely the major department of AI/ML we were to use was the NLP (Natural Language Processing). However, that didn’t cut down on the brainstorming efforts.  

AIENGLISH-PRO-1

The initial mornings started with alternatives. The questions that rose were should we go with models of our own, should we use services of Cognitive Service Providers like Microsoft. And if we go with our own models, where do we build, train, and test the models. 

Furthermore, we had multiple enabling skills against which we were planning to assess the candidatesFor example, we started with Oral Fluency, Vocabulary, Grammar and Spelling, and Sentiment Analysis for speaking. For writing, we started with Vocabulary, Grammar and Spelling, Sentiment Analysis, and Writing Fluency with road-map items of Pronunciation Check and Context Check Analysis. 

We decided to do a mix and match for our beta version. The plan was to use our own models for few evaluation points; and for few, check with third party libraries or pre-built cognitive services.  

We knew that the technology stack, the models, the architecture, etc. would keep evolving. Hence, we decided to start with very basic models in our initial beta version, gathering data on the beta version of the tests, and using that data to train our models for subsequent phases. 

So we started with VADER (from the NLTK library) for Sentiment Analysis, a third party library for grammar and spelling check, heavy pre-processing and analysis using NLTK and spaCy, the most popular NLP libraries for other evaluation points. The initial check for vocabulary was through a NLP concept called Named Entity Recognition, only to later realize that it definitely doesn’t come close on its own 

AIENGLISH-PRO-2

This basic selection of models or third-party libraries helped with creating a very initial framework of our NLP Engine, integrate it with our test platform as the upstream and reporting Engine as the downstream, containerize, deploy on our Microsoft Azure subscription, and initiate the beta version of the test to some of our friendly clients.

This initial beta version proved to be the key to our success. It not only helped with validating our end-to-end integration flow, test our exception and error handling, test our architecture, test the look and feel of the reports, candidate experience, etc. but it also helped us correct our assumptions on the data. Post the initials assessments, we went back to the whiteboard for few weeks.  

The outcomes were many, the four prominent ones being: 

  1. We were able to replace VADER with our own model based on BERT and Binary classifier 
  2. We successfully replaced the NER based scoring for vocabulary with frequency distribution analysis of one of the largest corpus 
  3. We attempted to replace the third-party library without own model again based on BERT 
  4. And finally, multi-label classifiers were tested on GEC 

On the architecture front, there were heavy changes as well. For example, the underlying messaging medium was changed, container configuration, changes to the base docker images, etc. The UX changed, the content of the tests changed a bit, and a lot more changes were done on candidate experience. 

nlp-ARCHITECTURE

Coming back to the AIwe started thinking more and more on the ML Ops, dataset curation, hyper parameters tuning, heavy training, validation and testing of our models, etc.  

However, one amazing thing that happened during these efforts was that we got connected with the Microsoft’s specialists and engineering team. Detailed brain-storming with Microsoft engineering team and architects helped us change our direction slightly. We started looking at Microsoft Azure Machine Learning Services to build, train, and test our models in order to help with ML Ops mainly. Parallelly, we also started looking at Microsoft Cognitive Services offerings. The Microsoft team, by this time, well understood our use case; and because of our frequent calls followed by POCs and tests, we were able to close down the use of Azure Cognitive Services like Text Analytics, Speech Services, Custom Text, Azure Video Indexer, and other offerings for the phase two of the product 

We were then on a mix of our own models, third party libraries, NLTK, spaCy, and Azure Cognitive services. It seemed perfect.  

We rolled out the phase two changes to some clients of ours. Though the results were great, there was a lot of calibration required. Our main clientfor AI-EnglishPro are based in the USA and India, and hence we had to recalibrate keeping both audiences in mind. This exercise took few weeks, each of us were taking multiple tests ourselves, regeneration of reports post each change in logic and comparing if the results of the logic are acceptable, taking client feedbacks, going over candidate feedbacks from the tests, etc.  

Fast forwarding, our improved version of the test is now actually helping some of our clients assess their candidates on Business English, and the efforts seem to have paid off based on our client feedbacks. However, we are not stopping here for sure. We are constantly learning and improving 

We assess the data and how our models and Azure Cognitive Services religiously on a daily basis, and any discrepancies are fixed in our models or sent to the Microsoft Engineering team for their inputs.  

Moving to the roadmap, one of the key question our clients ask is if they can customize the test. We understand that our clients have various use cases, for example, checking the English Proficiency in a support role, in a back office role, in a management role, and hundreds of other use cases. Due to the requirement of data set and training and testing of the models for each of this use case, we surely cannot open hundreds of these use cases; however, we are in the process of opening some of these key use cases.  

We also have been asked by our clients about our pronunciation check. However, AI-EnglishPro is CEFR compliant and, as per the changes in the Phonological Control aspect of CEFR 2018, which stresses on intelligibility rather than ascent and pronunciation, we have decided to make another product that checks pronunciation of a candidate against a reference text. 

C:\Users\Vishal Madan\Pictures\tempsnip.png

The results of this undertaking have been overwhelming to say the least; we have completed the initial 1000 assessments for a client of ours in Edu-Tech and the results are great, but a lot needs to be done still 

We love to see our clients happy and we love to see them astonished by the magic. We haven’t been able to bring out a rabbit through AI; however, I believe we surely have been able to magically bring out a right unbiased assessment of proficiency of their candidates.  

You can learn more about AI-EnglishPro here.

Vishal Madan
Vishal Madan
Head of Engineering, iMocha, Vishal is a senior techno-functional leader, who takes pride in being fearless, decisive and works only with one motto in mind – 'Value Addition' in any task at hand, be it the primary portfolio in Delivery, Managed Services or Program Management or proactively taken secondary portfolio(s) which helps the organization succeed.
Find me on:

Topics: Tech Recruitment, Product Updates

Related Posts

Top skills and technologies required for software development job roles by 2025

As per the iMocha-EY Tech Skills Transformation: Navigating the Future of Work Report, technology skills permeate every job role regardless of industry and function. In this blog, we will walk you through the top 47 skills required for performing different software development job roles, including application development, UI/UX designing, system administration, etc. So, why don’t we get right into it?  Table of Content 1. Computer language-related skills 2. Web-development skills 3. Native app development skills 4. Back-end framework skills 5. API development skills 6. Data engineering skills 7. Container skills 8. CI/CD skills 9. UI/UX skills 10. Software architecture skills 11. Software testing skills   

Skills Report: Top 5 Power User Skills for the Future

As the times change and the world is continually exposed to new technological advancements, conventional tech skills are also evolving and transforming into "power" skills.

How To Meet The Increasing Demand For App Dev & Business App Skills in a Limited Talent Pool

Introduction With our increasing dependency on digitization, tech skills have become crucial for all organizations, regardless of their industry. As more and more organizations are automating their workflow to meet the demands of the ongoing technological disruptions, tech skills are also evolving at an unprecedented rate and introducing new skills and job portfolios into the mix. At iMocha, we partnered with EY to understand this ongoing paradigm shift and answer three critical questions: What would be the in-demand tech skills across domains in 2025 and beyond?  What would be the strategic and functional impacts of tech skills transformation?  How are organizations responding to tech skills transformation? In our survey, we interviewed 50 HR and business leaders and conducted secondary research on over 26 million profiles to discover that 76% of these professionals & organizations have witnessed an increase in the demand for ‘application developers’ in recent years, while 62% of them have witnessed a similar surge for “business application power users” and “power developers.” But with a limited talent pool and shortage of required talent across the Indian, US, and European markets, how are organizations aiming to meet the increasing demand for application development and business power development skills? The answer can be organizational readiness through skills intelligence! In this blog, we’ll explain how building a skill taxonomy can help you overcome talent shortages and skill gaps and acquire future-proof application development and business application skills for your organization. Let’s get started!   Table Of Content - Three ways to tackle the shortage of qualified Application Developers & Business App Users Understanding changing job roles Investing in developing real-time visibility Assessing your organization’s overall skill proficiency - Emerging Application Development skills and how you can develop them - Final Thought