The guidance lecture(1)

The guidance lectures will cover the following topics:

      About this course

      Objectives of the course

      Reference Books

      Organization of the course

      Key points in this course

About this course

Language testing is a major branch of applied linguistics. The study of language testing involves research areas such as linguistics, psycholinguistics, cognitive science, pedagogy, theories of language acquisition, and educational measurement theories.

It studies the scientific measurement of language skills and abilities. It also studies the washback effect (反拨效应) that tests have on language teaching and learning.

Textbooks used in this course

Objectives of the course

The primary purpose of this course is to enable students to become competent in the design, development, and use of language tests. Competence in language testing involves the following:

An understanding of the fundamental considerations that must be addressed at the start of any language testing effort, whether this involves the development of new tests or the selection of existing language tests; An understanding of the fundamental issues and concerns in the appropriate use of language tests; An understanding of the fundamental issues, approaches, and methods used in measurement and evaluation; The ability to design, develop, evaluate and use language tests in ways that are appropriate for a given purpose, context, and group of test takers; The ability to critically read published research in language testing and information about published tests in order to make informed decisions.

Reference books

      李筱菊（1997）。语言测试科学与艺术。湖南教育出版社，长沙。

      桂诗春（1986）标准化考试-理论、原则与方法。广东高等教育出版社，广州

      Alderson, J.C, Clapham, C and Wall, D. 1995. Language Test Construction and Evaluation. Cambridge University Press. （《语言测试的设计与评估》，外研社，2000）)

      Bachman,L.F. 1990. Fundamental Considerations in Language Testing. Oxford: Oxford University Press. （《语言测试要略》，上外教，1999）

      Bachman,L.F. and Palmer, A.S. 1996. Language Testing in Practice. Oxford: Oxford University Press （《语言测试实践》，上外教，1999）

      Black, H.D. and Dockrell, W.B. (eds.). 1988. New Developments in Educational Assessment. Scottish Academic Press Ltd.

      Embretson, S.E. 1985. Test Design: Development in Psychology and Psychometrics. London: Academic Press, Inc.

      Finocchiaro,M. And Sako, S. 1983. Foreign Language Testing. New York: Regents Publishing Company, Inc. 

      Henning, G. 1987. A Guide to Language Testing: Development, Evaluation and Research. Cambridge: Newbury House Publishers. （《语言测试指南：发展、评估与研究》，外研社，2001）

      Hughes, A. (ed.).1988. Testing English for University Study. Oxford: Modern English Publications and The British Council. 

      Spolsky, B. 1995. Measured Words. Oxford: Oxford University Press （《客观语言测试》，上外教，1999）

      Suen, H.K. 1990. Principles of Test Theories. London: Lawrence Erlbaum Associates, Publishers.

      Wiersma, W. And Jurs, S.G. 1990. Educational Measurement and Testing.

      Wittrock, M.C. and Baker,E.L. 1991. Testing and Cognition. London: Prentice-Hall International Ltd.

      Wood, R. 1993. Assessment and Testing: A Survey of Research. Cambridge University Press. （《评估与测试：研究综述》，外研社，2001）3

Organization of the course

  Introduction to Language Testing

  Approaches to Language Testing

  Classical True Score Model

  Evaluation of Tests

  Statistics in Language Testing

  Item Writing

  Item Response Theory

  Computer and Testing

  Evaluation of Language Instruction Programs

Introduction to Language Testing

An introduction to the basic considerations in the area of language testing, Topics like "testing and teaching", "why test",  "what to test" and "how to test" will be covered.

Approaches to Language Testing

Language tests can be roughly classified according to four main approaches to testing:

1. The essay-translation approach;

2. The structural approach;

3. The integrative approach;

4. The communicative approach.

Although these approaches are listed here in chronological order, they should not be regarded as being strictly confined to certain periods in the development of language testing. Nor are the four approaches always mutually exclusive. A useful test will generally incorporate features of several of these approaches.

Classical True Score Model

The classical theory is the earliest theory of measurement. The classical theory is also referred to as the classical reliability theory because its major task is to estimate the reliability of the observed scores of a test. That is, it attempts to estimate the strength of the relationship between the observed score and the true score. It is also sometimes referred to as the true score theory because its theoretical derivations are based on a mathematical model known as the true score model.

Evaluation of Tests

This section is about the evaluation of tests. When we are faced with the responsibility of having to choose or develop an appropriate test, we should take the following into consideration:

1.      The purpose of the test;

2.      The characteristics of the examinees;

3.      The accuracy of measurement;

4.      The suitability of format and features of the test;

5.      The development sample;

6.      The availability of equivalent or equated forms;

7.      The nature of the scoring and reporting of scores;

8.      The cost;

9.      The procurement and;

10.  The political acceptability of the test.

In our discussion, we shall talk about test reliability and test validity in details, both of which are important points in classical true score model.

Statistics in Language Testing

In this section, issues concerning statistics will be our interest. We shall begin with measurement scales, and then come to data management in measurement. We will cover scoring and interpreting scores. And item and test relatedness is also a very important part of this discussion.

Item Writing

We shall first discuss the topic of avoiding problems at the item-writing stage and then we shall discuss separately the development of grammar and vocabulary tests, listening comprehension tests, oral production tests, reading comprehension tests and writing skill tests.

Item Response Theory

In this section we shall move to the recent advancement of language testing studies. Item response theory is a comparatively more powerful theory than the classical true score theory. Due to its mathematical nature, it is somewhat difficult to understand. We are going to have a very brief introduction to this fancy theory.7

Computer and Testing

Item banking, machine construction of tests and computer adaptive testing are the three main topics in this section.

Evaluation of Language Instruction Programs

One of the most obvious applications of language test development is in the evaluation of programs of language instruction. This section is intended to provide an overview of various methods of evaluation.

Evaluation may be viewed from several perspectives: prior-to-program implementation, during-program delivery, and following-program execution. Focus may be directed to affective and/or cognitive program impact. Concern may be with learning gains, instructional delivery, needs responsiveness, materials quality, cost effectiveness, continuing motivation, and/or the degree of realization of a variety of explicit or implicit objectives.

Evaluation may be longitudinal or cross-sectional, formative or summative, criterion-referenced or norm-referenced.

The guidance lecture(2)

The following topics will be covered in these guidance lectures

1.      About this course

2.      Objectives of the course

3.      Reference Books

4.      Organization of the course

5.      Key points in this course

Key points in this course

1.      What are tests?

2.      What is the relationship between measurement, tests and evaluation?

3.      Why do we need tests? (Purposes of tests)

4.      Approaches to language testing.

5.      Classical True Score Model

6.      Test reliability.

7.      Test validity.

8.      Measurement Scales.

9.      Normal Distribution.

10.  Item Response Theory.

What are tests?

A test may be defined as a task or series of tasks used to obtain systematic observations presumed to be representatives of educational or psychological traits or attributes. Typically tests require examinees to respond to items or tasks from which the examiner infers something about the attribute being measured. These are measurement, tests and evaluation. Measurement is the assignment of numerals to objects or events according to rules that give numerals quantitative meaning. Evaluation is a process that includes measurement and possibly testing, but it also contains the notion of a value judgment.

If a teacher administers a language test to a class and computes the percentages of correct responses, measurement and testing have taken place. The scores must be interpreted, which may mean converting them to values like As, Bs, Cs and son on, or judging them to be excellent, good, fair, or poor. This process is evaluation, because value judgments - either implicit or explicit - are being made. Relationships among measurement, tests, and evaluation.

An example of evaluation that does not involve either tests or measures (area "1") is the use of qualitative descriptions of student performance for diagnosing learning problems. An example of a non-test measure for evaluation (area "2") is a teacher ranking used for assigning grades, while an example of a test used for purposes of evaluation (area "3") is the use of an achievement test to determine student progress.

The most common non-evaluative use of tests and measures are for research purposes. An example of tests that are not used for evaluation (area "4") is the use of a proficiency test as a criterion in second language acquisition research. Finally, assigning code numbers to subjects in second language research according to native language is an example of a non-test measure that is not used for evaluation (area "5").

Why do we need tests?

      Diagnosis

      Selection

      Placement

      Program Evaluation

      Motivation and Guidance of Learning

      Providing Research Criteria

Approaches to language testing

      The essay-translation approach;

      The structural approach;

      The integrative approach;

      The communicative approach.

Classical True Score Model

The classical theory is the earliest theory of measurement. The classical theory is also referred to as the classical reliability theory because its major task is to estimate the reliability of the observed scores of a test. That is, it attempts to estimate the strength of the relationship between the observed score and the true score. It is also sometimes referred to as the true score theory because its theoretical derivations are based on a mathematical model known as the true score model.

Test Reliability

The investigation of reliability is concerned with answering the question, "How much of an individual's test performance is due to measurement error, or to factors other than the language ability we want to measure?"

Reliability is a quality of test scores, and a perfectly reliable score, or measure, would be one which is free from errors of measurement. There are many factors other than the ability being measured that can affect performance on tests, and that constitute sources of measurement error.

In any testing situation, there are likely to be several different sources of measurement error, so that the primary concerns in examining the reliability of test scores are first, to identify the different sources of error, and then to use the appropriate empirical procedures for estimating the effect of these sources of error on test scores.

The identification of potential sources of error involves making judgments based on an adequate theory of sources of error. Determining how much these sources of error affect test scores, on the other hand, is a matter of empirical research.

Within the CTS model, there are three approaches to estimating reliability, each of which addresses different sources of error: Stability estimates indicate how consistent test scores are over time (the correlation between scores on the same test taken twice); Equivalence estimates provide an indication of the extent to which scores on alternate forms of a test are equivalent (the correlation between scores on two or more forms of the same test taken concurrently); and internal consistency estimates are concerned primarily with sources of error from within the test and scoring procedures (the correlation among the items on a single test).

Test Validity

A reliable test is one that yields consistent scores for a student. Whether those scores are useful, whether they are adequate for the purpose for which the test was designed, is quite another matter.

Validity is concerned with the question, "How much of an individual's test performance is due to the language abilities we want to measure?"

There are many terms that have been used for test validity, including content validity, criterion validity, construct validity among others.

Content validity refers to the degree to which the items on the test reflect the intended domain. There are two aspects to this part of validation: content relevance and content coverage.

Criterion validity is concerned with the degree to which the test scores are accurate and useful predictors of performance on some other criterion measure.

The determination of construct validity is essentially a search for evidence that will help us understand what the test is really measuring and how the test works across a variety of settings and conditions. A construct is a trait, attribute, or quality, something that cannot be observed directly but is inferred from psychological theory. Tests do not measure constructs directly, rather, they measure performance or behavior that reflect constructs.

Measurement Scales

Nominal scale (称名量表)

male, female

Ordinal scale  (顺序量表)

1st place, 2nd place, 3rd place...

Interval scale (等距量表)

68, 70, 95, 100…

Ratio scale (比例量表)

0m, 2m, 5m, 10m…

Each of the four properties provides a different type of information, and the four measurement scales are thus ordered, with respect to each other, in terms of the amount of information they can provide. Raw score, z-score, T-score, Normal distribution area proportions... Normal Distribution.

Perhaps the most important characteristic of the normal distribution is symmetry.  This means that each side of the distribution is a mirror image of the other.  Because of this characteristic, it can be shown that matching intervals on either side of the distribution represent the same proportions of persons in the distribution.

A second important characteristic of the normal curve or distribution is closely related to the first: that, for the normal distribution, the mean, the median, and the mode fall at the same point on the scoring axis.

The third property of the normal distribution is that the curve is asymptotic; that is, it never intersects the horizontal axis.  This means that in a theoretical sense there is always the possibility that some one person's score may extend infinitely far to the right or to the left on the horizontal axis.  In practice, however, because we cannot test an infinitely large sample and because tests have a finite number of items, this characteristic is not so important to us as the other two properties of the normal distribution.

Item Response Theory

Weakness of CTS theory:

The common models and procedures for constructing tests and interpreting test scores have served measurement specialists and other test users well for a long time. These models, such as classical true score model, are based on weak assumptions, that is, the assumptions can be met easily by most test data sets, and, therefore, the models can and have been applied to a wide variety of test development and test score analysis problems.

Still, there are many well-documented shortcomings of the ways in which educational and psychological tests are usually constructed, evaluated, and used. For these and other reasons, psychometricians have been concerned with the development of more appropriate theories of mental measurements. The purpose of any test theory is to describe how inferences from examinee item responses and/or test scores can be made about unobservable examinee characteristics or traits that are measured by a test.

Presently, perhaps the most popular set of constructs, models, and assumptions for inferring traits is organized around latent trait theory. Consequently, considerable attention is being directed currently toward the field of latent trait theory, item characteristic curve theory or item response theory.

Any theory of item responses supposes that, in testing situations, examinee performance on a test can be predicted (or explained) by defining examinee characteristics, referred to as traits, or abilities; estimating scores for examinees on these traits (called "ability scores"); and using the scores to predict or explain item and test performance.

Since traits are not directly measurable, they are referred to as latent traits or abilities. An item response model specifies a relationship between the observable examinee test performance and the unobservable traits or abilities assumed to underlie performance on the test.