Testing Language in the NATO STANAG Context

NATO STANAG 6001 test papers in Lithuania correspond to the components of the Standard Language Profile, namely Listening, Speaking, Reading and Writing. Test papers with labels ‘Grammar’, ‘Vocabulary’ or ‘Language in Use’ are not designed. This article focuses on the role and the ways of how vocabulary and grammar in the Lithuanian STANAG 6001 test are assessed. As it is seen from contemporary researchers’ works, language proficiency involves not only language knowledge, but discourse, sociolinguistic knowledge and strategic competence. Therefore, while assessing proficiency of language, assessing language knowledge is only one part of it. Both knowledge and the ability to use it are essential parts for communication. While assessing the test takers’ language ability inferences on the person’s language knowledge and strategic competence are made. Knowledge of vocabulary, grammar, and spelling are essential in language proficiency. Without elementary knowledge of language one will not be able to show the other competences or skills in the target language. On the other hand, test takers with higher level of language competence are able to compensate for a certain lack of other competences by making use of their language resources. And finally, it is indicated that there is no need to design separate papers for testing Vocabulary, Grammar or Language Use, as they are incorporated and assessed in tests of Listening, Speaking, Reading and Writing.


Introduction
In Lithuania STANAG 6001 test papers correspond to the Standard Language Profile -i.e.Listening, Speaking, Reading and Writing.Papers with labels 'Grammar', 'Vocabulary' or 'Language in Use' are not designed.Such papers are not relevant because according to NATO Standardization Agreement (STANAG) Language Proficiency Levels' Document promulgated on 21 October 1976 a candidate's language is tested in "oral proficiency (listening and speaking) and written proficiency (reading and writing)" (1976, p. 1, 2).
From this document it is obvious that papers for testing 'Grammar', 'Vocabulary' or 'Language Use' explicitly seem to be not necessary.While assessing proficiency of language, assessing language knowledge is only one part of it.Both knowledge and the ability to use it are essential parts for communication.
In this article the role of language in the Lithuanian STANAG 6001 test will be explained.Although papers for testing grammar, vocabulary or language in use are not designed in this test, they are taken into account when giving a level score for Listening, Speaking, Reading and Writing, thus they are tested implicitly.
The goal of this article is to show what impact knowing or not knowing of usage of language makes on its use, as in the communicative approach to language proficiency both are indispensable to achieve the communication successfully.Moreover, it will be indicated whether there is need to design separate papers for testing Vocabulary, Grammar or Language Use in a proficiency test of the second language.

Theoretical Background
The assessment of second language vocabulary and grammar has been affected by changing perceptions of the importance of these skills in language acquisition.The construct of vocabulary and grammar is still considered by many to be an important aspect in the measurement of an individual's overall performance in a language.Grammar and vocabulary items are quite easy to write and in some tests they are regarded as an efficient substitute for the reading and writing comprehension test.J. Read (2000) points out that as a result of the shift to communicative test formats, less emphasis on vocabulary assessment is observed, still, if vocabulary is tested in language proficiency tests it should be concentrated on vocabulary in use.
The idea of 'language in use' became common during the period in which linguists were dealing with the notion of communicative competence and communicative language teaching.The first to make a distinction between 'language in use' and 'language usage' was H. G. Widowson.He suggested that "knowledge of use must of a necessity include knowledge of usage but the reverse is not the case" (Widowson, 1978, p. 18).P. Rea-Dickins reflecting on 'language usage' notes that our knowledge about grammar testing practices and nature of the construct is incomplete.She observes that with the communicative orientation of language teaching the role of grammar teaching has diminished, which was reflected in some tests by omission of "grammar testing".J. Heaton (1991) admits that writing skills are varied and complex, language use being one of five main skills necessary for writing good prose.The same opinion is shared by professionals in testing of reading, listening and speaking (Alderson (2000), Buck (2001), Weigle (2002), Grabe (1991)).
Communicative approach to language proficiency proposes that language ability is made up not only of multiple areas of language knowledge but also of strategic competence, language knowledge, grammar and vocabulary being a cornerstone of it.Assessing vocabulary and grammar in present day communicative testing still remains a challenge for many linguists, researchers and specialists working in testing.

Subjects
The subjects for this study are military and civilian people working for the Lithuanian Ministry of Defence (MOD) and the Lithuanian Armed Forces, as well as personnel of the Ministry of Internal Affairs who are planned to take part in North Atlantic Treaty Organization (NATO) and Partnership for Peace (PfP) exercises, operations, international training or work within multinational staff, at levels required by NATO/PfP Partnership Goals.The answers of 20 test takers who took the same language proficiency test, which was designed by the Lithuanian MOD testing team, are analyzed.The subjects were classified into Levels 1, 2 or 3 after they had taken the STANAG 6001 test.

Materials
The materials for this study consist of the English Language Test, in full compliance with NATO STANAG 6001 (STANAG 6001 -the language standard throughout NATO) Edition 2 promulgated on 11 June, 2003, designed to assess the English language proficiency of the individual in listening, speaking, reading and writing skills for professional use.It is not based upon a particular language programme, but is a proficiency test.The test level corresponds exactly to STANAG 6001 descriptors (Edition 2) June 11, 2003.To assess Reading, Writing and Listening proficiency of test takers at Levels 1, 2 and 3 they are given a multi-level test designed for Levels 1, 2 and 3. To assess Speaking a guided interview is used, which depending on the test-taker's language proficiency varies in tasks.STANAG 6001 test papers correspond to the components of the Standard Language Profile -i.e.Listening, Speaking, Reading and Writing.According to NATO STANAG 6001 Language Proficiency Levels' Document "Language proficiency will be recorded with a profile of 4 digits indicating the specific skills in the following order: Listening, Speaking, Reading, Writing" (1976, p. 1, 2).
Tests are designed and administered by the MOD testing team.Examiner's background -Bachelor's or Master's degree in the English language or language testing plus experience in testing.

Discussion
In the following parts of this article the testing team's efforts to assess vocabulary and grammar in one of the Lithuanian STANAG 6001 test suits are analyzed.The qualitative comparisons of the test takers' answers, as well as, empirical item analysis are chosen to reveal the importance of grammar and vocabulary in the measurement of an individual`s language proficiency.

Test of Listening
The testing team defined the construct of Listening according to G. Buck (2001).As Buck indicates "it makes more sense to put the emphasis on testing language competence rather than testing strategic competence" (Buck, 2001, p. 105).
According to Buck, every aspect of language competence includes grammatical knowledge.
The Listening Test consists of two parts and two different test tasks.Task 1 is filling a gap with a required word.The vocabulary chosen for Task 1 is more common, with more high-frequency words, no complex grammar, and utterances are comparatively short, consisting of one or two simple sentences.
It is assumed that items 1-10 of Task 1 intend to test vocabulary knowledge, as the answers required are mostly high-frequency words, such as 'table', 'black', 'small', etc.There are a few words, which might not be so common.Moreover, in some parts vocabulary is condensed, tested words are very close to each other.Due to this fact and because the vocabulary is in speech, test takers should demonstrate not only vocabulary knowledge but other listening sub-skills to answer the items correctly.
Task 2 involves items of sentence completion, which requires more skills.There are 30 texts of different length with an item or two to every text, and the test taker is required to understand linguistic information in a variety of texts on a variety of topics.Test takers listen to information only once, which adds to the task difficulty.To answer some items correctly test takers have to show understanding of inferred meaning and show the ability to use the vocabulary, which is not heard in the text.Discourse skills are important for the listening construct and wherever possible and appropriate tasks should require listeners to process more than just short utterances.Longer text engages some aspects of pragmatic knowledge and strategic competence.
Variety and complexity of texts use a wide range of vocabulary and grammar structures to help measure different skills and abilities but at the same time they represent the construct of the discussed listening test, which stands for construct validity.For the test's face validity, vocabulary is very important too.For example, in the situation where military are tested, choosing text with military vocabulary would add to the test's face validity.These are only some glimpses on the Listening test's validity.
Vocabulary and grammar in the listening test plays a very important role.In fact, everything depends on the construct of the test.One solution would be to restrict the construct to grammatical and vocabulary knowledge, but this is only one part of listening ability.In this test a broader understanding of listening is chosen to be tested, which includes grammar, vocabulary and at the same time tests other skills characteristic of listening.As Buck states "If resources allow, a better strategy would be to test grammatical knowledge, discourse knowledge and quite a lot of pragmatic knowledge, and most of what is unique to listening" (Buck, 2001, p. 202).

Test of Reading
Language skills and linguistic competence are closely connected and although it is claimed that grammar or vocabulary are not assessed in the Reading Test, the correct answers to the questions of the test are the result of having grammatical and vocabulary knowledge and not only having the right reading skills.As Alderson points out "In studies of readability, most indices of vocabulary difficulty account for about 80% of the predicted variance.In short, vocabulary plays a very important role in reading tests" (Alderson, 2000, p. 99).
It is widely believed that language competence is needed in order to be able to read in a second language.Vocabulary is important for comprehension of the text and this has a direct connection with the test performance.However, it is assumed that testing reading measures something different from linguistic competence; mainly, it tests reading skills but at the same time grammar and vocabulary are being tested, as reading inevitably involves them.
The construct of reading in the Test Specifications indicates that reading is understood as having skills of understanding a wide range of given information using a great variety of mental and linguistic strategies.The reading construct according to C. J. Weir (1993), which is taken as the theoretical framework for the Reading Test, states that knowledge of grammar and vocabulary plays a considerable role in it.For example, to be successful in reading one must be able to locate a specific phrase, a word or a number in a text; understand the syntactic structure of a sentence and clause, understand lexical or grammatical cohesion; understand lexis and deduce meaning of lexical item from morphology and context.All the reading skills mentioned above include vocabulary and grammar knowledge first of all.J. C. Alderson (1984) investigated the connection between the first-language reading and the second-language reading skills.He asked the question whether the ability to read transfers across languages and whether a good first language reader also is a good second language reader.He arrived at a conclusion that "knowledge of the second language is a more important factor than first-language reading abilities" (Alderson, 2000, p. 23).
Thus, the conclusion would be that the second-language knowledge is more important than the first-language reading abilities and a certain linguistic threshold must be crossed before being able to cope with reading tasks in the second-language.This linguistic threshold is very important for the discussed Reading test.It is not absolute, it varies by task: the more demanding the task, the higher the linguistic threshold.For example, to answer Question 31 correctly a test taker has not only to make propositional inference, but has to understand and connect such words as 'illegal immigrants' and 'stowaways'.To answer Question 29 correctly a test taker should not only have lexical knowledge and reading skills, but should possess grammatical knowledge, too.To make the correct inference he should know that 'unless' means 'if not'.Questions from 14 to 21, 27 and 32 mainly test the reading skill closely connected with vocabulary, i.e. looking through a text to locate a specific phrase, word or number.To answer Q32 one should firstly, locate the phrase 'it will weigh more than 80lb and cost a fortune to replace', then make the connections between words 'weigh' and 'weight', make inference that 'problems' are connected with 'replace' and then the test taker should locate the missing words of the stem 'cost a fortune'.
To answer Q22 the test taker should not only possess lexical knowledge to locate a specific phrase 'withdrawn from firearms duty' but he should possess grammatical knowledge in order to paraphrase it in words 'use the gun' or 'be on duty' to fit the answer into the stem.Otherwise being able to locate the phrase and simply lift it from the text will not enable him to get a point for a correct answer, as it will not make a sensible or understandable sentence.So here alongside with vocabulary and reading skills apparently testing of grammatical competence is incorporated.
On the other hand, one can never be sure which skill the test taker is using.It still remains to be explored what other processes happen in the reader's brain which help him to come to the right answer.However, vocabulary and structural knowledge skills are indispensable in reading and they are tested implicitly in reading, as they are one of the elements of fluent reading process.

Test of Speaking
The construct of Speaking based on Saville and Hargreaves (1999) covers quite a wide range of language competences.The structure of the Speaking Test is a guided interview.It is one of the most common techniques of oral tests and it is somewhere in between of such techniques as discussion/conversation and question/ answer.Compared with discussion/conversation, an interview is structured, which helps the interviewer to get the answers to certain questions without losing the control over the point the interviewer is interested in.At the same time the test-taker is free to develop his comments, opinions and show his language.Compared to question/ answer technique, an interview is more authentic and it could be adjusted to different flows, topics and depth of speech.N. Underhill (2001) in his book "Testing Spoken Language" points out that "at higher levels usual mark categories fail to discriminate well.The tightly-controlled interview … will not easily elicit the learner's best language performance" (Underhill, 2001, p. 56).
The interview during the described oral exam lasts for about 10-15 minutes and depending on the test-taker's language proficiency it varies in tasks.At a Level 1 the main tasks are answering simple questions, maintaining simple, short conversation.It is a short, simple sentence level.At a Level 2 it gets more complicated as the testtaker is supposed to produce narration in present/past/ future, as well as description and comparison.The testtaker's speech should be at a simple paragraph level with basic grammatical structures being typically correct.At a Level 3 in addition to what was mentioned above, abstract topic, supported opinion and hypothesis are added.Requirements for accuracy and vocabulary are also higher.From tasks described above and from the Speaking Rating Scale it is obvious that the interview is not a grammar or vocabulary test, although grammar and vocabulary are very important factors which are taken into account when giving a score level.In the Speaking Rating Scale these factors occupy nearly 50% of the scale.
For example, at a Level 1 the test-taker's performance reveals a shortage of vocabulary.He does not know the word 'far' and cannot answer the simple question 'How far is your work from home?' The word 'church' sounds unfamiliar, but in this situation the test-taker manages to use circumlocution 'god's house'.He has not enough vocabulary to speak about the job.However, speaking about family, hobbies, free time the test-taker seems to find the necessary words.The answers are short sentences or simply one or two words.Only one tense (Present Simple) is used and erratic word order is everywhere if they try to give a longer answer.Thus, looking at the vocabulary and grammar is fairly enough to determine that the test-taker's language proficiency is not higher than elementary.
The performance of the test takers at a Level 2 is different.They are able to give the narration in past.In many cases they use Past Simple correctly, of course, with exceptions.One more thing which is immediately obvious is the length of the test takers` answers.They are in a simple paragraph level and this is their vocabulary and grammar knowledge, which enable them to produce the language.However, they make too many mistakes in word order, structures and tenses to be given a higher level.One more interesting fact is that while speaking about their childhood, native town, work they seem to have good vocabulary, but when given an abstract topic about the recent elections of the president, they seem to be short of words, cannot express thoughts clearly and their speech becomes clumsy and distorted.It indicates that the test takers have reached the 'ceiling' of their possibilities in oral language proficiency and as in the Level 1 case, grammar and vocabulary are the main impediments for them to reach a higher level.
The test-takers, whose oral proficiency was assessed at a Level 3, nicely cope with tenses, word order, comparison of adjectives, relative clauses, etc.Their flow of speech is natural and easy.They use low-frequency vocabulary which is adequate for all situations and it together with language structures enable them to cope with tasks which are set at a Level 3.They do not lose the control of situation whatever question they are asked.They can support their opinion, speak about abstract topics, and make predictions not losing the quality of their speech.
As it is known, oral speech is not characterized only by grammar and vocabulary.Such features of speech as pronunciation, comprehensibility and fluency are very important factors while forming the opinion about the test-taker's oral language proficiency.However, when the speech is transcribed and when it is impossible to hear the above mentioned factors, grammar and vocabulary remain the main distinction between levels of the test-taker's language proficiency and in the oral test they undoubtedly play a very important role.After all, the rating of speech sample is not made from the transcript.In addition to the testing teams' effort it would have been interesting to ask an expert judge or two to tell the testing team what distinguished (in their view) the three performances.

Test of Writing
Current scholarship in the field of components of language ability seem to have arrived at the opinion that "the ability to use language to achieve genuine communicative function consists of interactions between aspects of language knowledge on one hand and strategic competence on the other…" (Weigle, 2002, p. 42).
It is assumed that learning to write involves much more than simply learning grammar and vocabulary of the language, but, on the other hand, one cannot write in the second language without knowing at least something about grammar and vocabulary of that language.That is why in the Writing Rating Scale such descriptors as 'vocabulary', 'language use' and 'mechanics' are included.Writing is understood as performance assessment.
To fulfill two tasks (a letter and a composition) of the Test of Writing a test taker should show linguistic, discourse and sociolinguistic knowledge.The test takers' ability to organize ideas and express them in their words in writing is a skill essential for real-life communication.As the second task of writing a composition is chosen.As it is concerned Heaton (1991) indicates that "sampling a student's writing skills in this way will appear a far more valid test than any number of objective test of grammar" (1991, p. 144).
As it can be seen from the samples of the test takers' performance writing in the second language may be hindered as the test taker needs to focus on language rather than context because of limited second language knowledge.For instance, Level 1 test takers do not have enough vocabulary even to form simple understandable sentences in the second language; their writing is clearly hampered by the lack of language knowledge without which they apparently are not able to show the other skills in writing if they have such.Level 2 test takers' performance is considerably different.These test takers manage to convey the message with the help of the language given in the input of the task.They write in simple sentences, with different grammatical errors, but the communicative aim necessary for minimal every day survival is achieved.Quite a different situation is observed in the performance of the Level 3 test takers.They seem to be comfortable with language and fulfill the tasks without struggling.Their language contains more sophisticated vocabulary, control of a full range of structures; spelling errors never interfere with comprehension.Obviously, it is not only the vocabulary and grammar that help them to achieve this result.Apparently they have a good strategic competence, discourse and sociolinguistic knowledge as well as linguistic competence.What is more, knowledge of language helps them to reveal all the other language competences they have with ease and flow.

Conclusions
It is obvious from contemporary researchers' works and this study that language proficiency involves not only language knowledge, but discourse, sociolinguistic knowledge and strategic competence.Thus in assessing proficiency of language, assessing language knowledge is only one part of it.Both knowledge and the ability to use it are essential parts for communication.
What is more, while assessing the test taker's language ability, inferences on the person's language knowledge and strategic competence are made.Knowledge of vocabulary, grammar, and spelling are essential in language proficiency.Without elementary knowledge of language one will not be able to show the other competences or skills he possesses in the target language.
And finally, there is no need to design separate papers for testing Vocabulary, Grammar or Language Use, as these are components of language knowledge and they are incorporated and assessed in tests of Listening, Speaking, Reading and Writing.