The Challenge of Vocabulary Testing
There is a fundamental problem with many, perhaps most, vocabulary tests which is rarely acknowledged. It’s not always clear what the problem is because intuition tells us that testing knowledge of a word should be easy and the result should be unambiguous; either one knows a word or does not. But the reality is much more complicated than that.
A Simple Comparison
Here are two examples at Merriam-Webster’s website. Granted, these are promoted as two different vocabulary quizzes, but exactly how are they different? Are they just tests in different formats but measuring the same thing? Or are they testing different aspects and degrees of word knowledge? Let’s take a closer look.
The Visual Vocab Quiz
In the Visual Vocab Quiz (link to original), a simple illustration is presented and the objective is to select a sequence of letters from a list of four options which can be used to name that concept within fifteen seconds.
The Vocabulary Quiz
In the Vocabulary Quiz (link to original), a single word is presented and the objective is to select the best single-word definition from a list of four options within ten seconds.
Similarities
Both tests assign more points for faster responses as well as correct responses for more difficult words. So both are testing the degree of fluency or strength of the word knowledge (faster responses) as well as breadth of knowledge (knowing more difficult words implies that easier words are also known).
Both tests are also focused on measuring receptive word knowledge. That is, the answer is given and the task is to indicate which option is correct so the one does not need to actually produce the word in order to answer the question. Therefore, it’s not really necessary in either test to be able to pronounce or spell the word correctly. In fact, it’s not necessary to even be able to recognise the word in an aural context because the tests are completely visual. So to be more specific, we can say that both tests measure written receptive word knowledge.
Differences
Despite the similarities, there are some subtle, but important, differences between the two formats.
The task in the Visual Vocab Quiz is primarily to select the correct referent (a string of letters which one may or may not know how to pronounce) for a given concept (in this case, a physical object depicted in an image). The concepts depicted in the illustrations are always concrete and physically durable. They are not fleeting or intangible concepts such feelings or ideas, but rather physical things. So essentially, it is a vocabulary test which measures one’s knowledge of the relationship between a concrete noun’s written form and it’s meaning.
In the Vocabulary Quiz, however, the task is primarily to find the closest synonym of the target word. To complete the task successfully, one must demonstrate knowledge of two words because it’s impossible to answer correctly if either the target word or its matching synonym are unknown. One needs to understand the concept of both words well enough to conclude that they belong in the same category.
For example, one question presents the word obscure and the following list of options:
- creepy
- real
- dated
- mysterious
In order to select the correct option, one must know that obscure can be both a verb and an adjective. If all of the options are already known as adjectives, then the task becomes slightly easier, but if one only knows the verb form of obscure, it might be quite difficult to select the correct option because the primary concept behind the verb form of obscure has to do with making less visible or less clear which doesn’t really match any of the options listed above.
But what if one knows that obscure can be an adjective, but has only encountered the word in the following contexts:
- She was born in an obscure town in the countryside with a population of less than 100.
- An obscure researcher announced a ground-breaking discovery.
- He was charged under an obscure and rarely-enforced law dating from the late 1800s.
- The formerly obscure artist suddenly became the talk of the town.
In these examples, some or even all of the options could replace the word obscure, albeit with some humorous twists in meaning.
The adjective form of obscure has a few senses, most of which are essentially antonyms of clear or obvious. WordNet lists the following senses:
- not clearly understood
- marked by difficulty of style or expression
- difficult to find
- not famous or acclaimed
- not drawing attention
- remote and separate physically or socially
That information is not enough to select the correct option though. One must also know that mysterious has two core meanings:
- difficult to understand or explain, as in a person’s mysterious past
- beyond normal understanding or experience, as in mysterious powers
The second meaning is more closely related to magic, but the first meaning is more closely related to one of the meanings of the adjective form of obscure which makes mysterious the correct answer in this case.
So you can see that the format of this second vocabulary test is actually tapping into not only a deeper level of word knowledge than the Visual Vocab Test, but also a subtly different aspect. Whereas the first vocabulary test taps into a simple mapping of referent to label, the second requires a knowledge of the concept of difficulty in understanding and the words which can be used to discuss that concept.
Aspects of Word Knowledge
The differences discussed above show that the vocabulary tests differ not only in the depth of word knowledge they measure, but they also illustrate the idea that there are different aspects of word knowledge which can be tested. The flaw with many vocabulary tests alluded to at the beginning of this essay is that the test designers either fail to acknowledge which of the many aspects of word knowledge they are measuring with the test or assume that all of the aspects develop in parallel such that measuring one is the same as measuring all.
An honest test of vocabulary must acknowledge which aspects of word knowledge it is measuring and to what degree it is being measured. There is certainly nothing wrong with measuring a student’s ability to select the correct sequence of letters used to name a particular object, but that doesn’t imply that the student knows everything there is to know about that word!
A Word Knowledge Framework
Nation (2001)1 has proposed a framework for discussing and measuring word knowledge based on three domains: form, meaning, and use. Within each domain are three aspects of word knowledge, each of which is further classified as either receptive knowledge (also called passive knowledge) and productive knowledge (also called active knowledge). These aspects can be summarised in the following table:
| Form | Meaning | Use |
|---|---|---|
| spoken form | form and meaning | grammatical functions |
| written form | concept and referents | collocations |
| word parts (affixes, roots, etc) | associations | constraints on use |
Although there is no strict order to the course of word knowledge acquisition, a general rule of thumb is that knowledge develops from the upper left to the lower right. That is, knowledge of a word’s spoken and written form is more likely to be learnt before, and consequentially more developed than knowledge of a word’s collocations or constraints on use. This is particularly true for second and foreign language learners who often find the bottom right section particularly difficult.
Differences in Vocabulary Size Tests
It should be clear by now why different tests of vocabulary size can give wildly different results. Subtle differences in the underlying assumptions, construction, and format of vocabulary tests can result in different aspects of the word knowledge which the test measures.
Therefore, one of the goals of VocabularySize.com is to assist researchers in piloting, validating, and distributing vocabulary tests which measure a range of different aspects of word knowledge. Eventually, correlations will be found among them and their courses of acquisition will be better understood and characterised well enough to begin combining tests to measure more than one aspect at the same time.
A test of English word parts (affixes) is available for users with teacher account which attempts to measure one aspect from each of the three domains in the table above. Over time, it will be refined further and, if all goes well, one’s performance on the word parts test will be used to adjust the final score on the Vocabulary Size Test.
Another test which focuses strictly on receptive written form knowledge is in the works which will, again hopefully, help describe how the degree and depth of knowledge varies for each aspect of word knowledge.
What is a word?
This brief overview of creating vocabulary tests to measure vocabulary knowledge did not even begin to address the even more fundamental issue of how the term word is defined. This topic deserves its own post to deal with the myriad issues involved in operatationalising the units of measure in a vocabulary test, but it bears noting here that measuring single items, sometimes called 1-grams or unigrams, is not sufficient to fully characterise one’s vocabulary knowledge, especially for the aspects of word knowledge in the lower right of word knowledge framework table above. A simple example should make this clear: knowing the basic and frequent meaning of stand, to be or become upright, does not give immediate access to the meanings of stand in, stand for, nor stand out.
-
Nation, I. S. P. (2001) Learning Vocabulary in Another Language. Cambridge: Cambridge University Press. ↩
Loading...