SmallTalk Speaking Test
SmallTalk Speaking Test is the online test of assessing candidate’s speaking abilities according to international standard by the artificial intellect (AI). After a few years of tests and researches SmallTalk team have been able to create an English speaking test that could evaluate the candidate’s full language abilities.
In order to align our test to international standards, we’ve managed the test structure and its assignments in the way to unlock the potential of the candidates from level A1 to level C2 without making it more complicated to the participant. That means that the test is suitable for all English levels, nationalities and backgrounds.
Aligning SmallTalk Speaking Test assessment to international standards
Assessment of the SmallTalk Speaking Test is aligned to the Cambridge English exams, which at the same time, are aligned to the Common European Framework of Reference for Languages (Council of Europe 2001)*, the international standard in measuring language ability.
The Cambridge English Scale covers a wide range of language proficiency and is aligned to the Common European Framework of Reference (CEFR).
The Cambridge English Scale provides clearer and more detailed information than was previously available, showing where the candidate’s performance falls within each CEFR level and grade. This gives candidates better information about their performance and helps them to identify areas for improvement.
The same scale is used across our exams, so it is also easier to understand the relationship between them.
SmallTalk Speaking Test criteria
SmallTalk Speaking Test criteria is aligned to University of Cambridge Oral Assessment criteria Grid: https://www.cambridgeenglish.org/Images/22649-rv-examples-of-speaking-performance.pdf
Candidate's test result is calculated by individual performance in each criterion:
1.Vocabulary
A1 - Has a very basic repertoire of words and simple phrases related to personal details and particular concrete situations.
A2 - Uses basic sentence patterns with memorized phrases, groups of a few words and formulae in order to communicate limited information in a simple everyday situation.
B1 - Has enough language to get by, with sufficient vocabulary to express him/herself with some hesitation on topics such as family, hobbies and interests, travel, and current events.
B2 - Has a sufficient range of language to be able to give clear descriptions, express viewpoints on most general topics, without much conspicuous searching for words, using some complex sentence forms to do so.
C1 - Has a good command of a broad range of language, allowing him/her to select a formulation to express him/herself clearly in an appropriate style on a wide range of general, academic, professional or leisure topics without having to restrict what he/she wants to say.
C2 - Shows great flexibility reformulating ideas in different linguistic forms to convey finer shades of meaning precisely, to give emphasis, to differentiate and to eliminate ambiguity. Also has a good command of idiomatic expressions and colloquialisms.
2.Grammar
A1 - Shows only limited control of a few simple grammatical structures and sentence patterns in a memorized repertoire.
A2 - Uses some simple structure correctly, but still systematically makes basic mistakes.
B1 - Uses reasonably accurately a repertoire of frequently used “routines" and patterns associated with more predictable situations.
B2 - Shows a relatively high degree of grammatical control. Does not make errors which cause misunderstanding, and can correct most of his/her mistakes.
C1 - Consistently maintains a high degree of grammatical accuracy, errors are rare, difficult to spot and generally correct when they do occur.
C2 - Maintains consistent grammatical control of complex language.
3.Fluency
A1 - Can manage very short, isolated, mainly pre-packaged utterances, with much pausing to search for expressions.
A2 - Can make him/herself understood in a very short utterances, even though pauses, false starts and reformulation are very evident.
B1 - Can keep going comprehensibly, even though pausing for grammatical and lexical planning and repair is very evident, especially in longer stretches of free production.
B2 - Can produce stretches of language with a fairly even tempo, although he/she can be hesitant as he or she searches for patterns and expressions, there few noticeably long pauses.
C1 - Can express him/herself fluently and spontaneously, almost effortlessly. Only a conceptually difficult subject can hinder a natural, smooth flow of language.
C2 - Can express him/herself spontaneously at length with a natural colloquial flow, avoiding or backtracking around any difficulty smoothly.
4.Pronunciation
Pronunciation criteria is not aligned to University of Cambridge Oral Assessment criteria Grid: https://www.cambridgeenglish.org/Images/22649-rv-examples-of-speaking-performance.pdf
All in all, SmallTalk team found essential to include these criteria to final assessment grid. In order to be able to assess pronunciation by AI, we’ve added to our tests a few reading assignments, which are spread between levels A1-C2. With this specific kind of tasks we’ve been able to track where the candidate struggle with the pronunciation of particular words and align these mistakes with their CEFR level. In this case, we could predict, which level of words the candidate could not pronounce correctly.
5.Interaction logic
We’ve found interaction criteria the most difficult to assess in online testing by AI system. The solution to this problem was achieved by specific structure of the test, allowing participant interest in a multiple variety of ways. With the specific structure we could trace how the candidate among other things, could answer and make questions and follow up a conversation.
A1 - Can ask and answer questions about personal details. Can interact in a simple way but communication is totally dependent on repetition, rephrasing and repair.
A2 - Can ask and answer questions and respond to simple statements.
B1 - Can initiate, maintain and close simple conversations on topics that are familiar or personal interest.
B2 - Can help the discussion along on familiar ground confirming comprehension, inviting others in, etc.
C1 - Can select a suitable phrase from a readily available range of discourse functions.
C2 - Can interact with ease and skill, picking up and using non-verbal and intonational cues apparently effortlessly.
6. Coherence
A1 - Can link words or group of words with very basic linear connectors.
A2 - Can link group of words with simple connections.
B1 - Can link a series of shorter, discrete simple elements into a connected, linear sequence of points.
B2 - Can use a limited number of cohesive devices to link his/her utterances into clear, coherent discourse, though there may be in some “jumpiness" in a long contribution.
C1 - Can produce clear, smoothly flowing, well-structured speech, showing controlled use of organisational patterns, connectors and cohesive devices.
C2 - Can create coherent and cohesive discourse making full and appropriate use of a variety of organizational patterns and a wide range of connectors and other cohesive devices.
To sum up, we’ve divided these 6 criteria into 2 different groups: essential and subsidiary.
Essential group: Vocabulary, Grammar. Fluency and Pronunciation
Subsidiary group: Interaction and Coherence
The main difference between these two groups is that currently AI assessment of interaction and coherence is not that accurate to be an independent criterion, but very important in the final level decision, especially if the candidate achieved different levels in each essential criterion.
