1. Introduction

OALibJ

Open Access Library Journal

2333-9705

Scientific Research Publishing

10.4236/oalib.1107628

OALibJ-110878

Articles

Biomedical&Life Sciences Business&Economics Chemistry&Materials Science Computer Science&Communications Earth&Environmental Sciences Engineering Medicine&Healthcare Physics&Mathematics Social Sciences&Humanities

Applying Kolmogorov’s Proofs to the Evaluation of Instruction: Generalized and Particular (PART ONE)

John

W. Oller

₁

Institute for Pure and Applied Knowledge, IPAK-EDU.org, Pittsburgh, Pennsylavania, USA

30 06 2021

08 07 1 28 10, June 2021 25, July 2021 28, July 2021

2014

This work is licensed under the Creative Commons Attribution International License (CC BY). http://creativecommons.org/licenses/by/4.0/

Successful sharing of information—positive (actual) knowledge about facts, skills that are imparted, abilities developed and expressed—is the implicit goal of instruction in all its varied forms. It is the goal of training athletes, dancers, and professionals in every walk of life from early childhood to the most advanced level of education. PART ONE introduces mathematical proofs showing that the interactional successes engineered by instructors, other things being equal, must trend toward 100% shared information—mastery of the course of study. Failed efforts trend toward a complete absence of shared information. All this holds independently for the subject-matter, methods of instruction, and the attributes conducive to instructional success. In Part One, the underlying proofs are united by a very simple proof from the theory of true narratives showing that every iota of knowledge that might be shared in any instructional context depends on the kind of representations found in true reports of actual experience. Empirical studies in Part One confirm the predicted agreement in diverse contexts on the elements of good teaching. In Part Two, Kolmogorov’s proofs from 1933 are generalized, amplified, and tested empirically showing successful instruction converging toward 100% agreement on 1) subject-matter, 2) which methods of presentation and assessment work, and even on 3) the abstract criteria for successful instruction. At the same time, as the proofs also show, the cumulative effects of failed communicative efforts must and do trend toward zero shared information.

Evaluation of Instruction Kolmogorov Probability Theory Peircean Logic of Relations Polya’s Central Limit Theorem Pragmatic Information Successful Communication TNR-Theory

1. Introduction

Not so long ago, I teamed up with some colleagues [2] to advocate the tearing down of disciplinary boundaries―to put gates in the walls and bridges across the gaps―that according to tradition are supposed to separate disciplines and sub-disciplines in the departments, colleges, schools, and specializations within them all across the institutions of higher learning throughout the world. In addition to colleges and universities, here I also have in mind the training programs taking place in such contexts as international aviation, the business world, vocational schools, the information and technology sector, sports and the performance arts, and, in fact, in education in general. Today much of the teaching and training of interest is being done through the internet asynchronously―increasingly so since COVID-19. All this to say that here in both parts of this paper, I want to address instruction and training, educational interactions, in a general way, aiming to include them all. My purpose is inclusive, not because of any ambition like that of Pinky and the Brain [3] who wanted to “take over the world”, but out of the desire to show how certain published mathematical proofs apply to all possible instructional contexts.

The kernel of truth to be developed here is implied in the opening quotation from John Dewey [1]. Communication enriches those who engage in it. It changes their efficiency toward an increasingly rich sharing of information much as a slight positive curvature, like the curvature of our planet, as Bernard Riemann showed [4] in 1873 (p. 10), ensures (if it exists) that the seemingly infinite expanse of space is both spherical and finite. Similarly, if there is even a modicum of shared knowledge between the interlocutors engaging in any form of instructional communication, the cumulative effect of their interactions over time must tend to produce a growing agreement. It must, as will be shown in Part One of this paper, and as will be demonstrated in greater depth and detail in Part Two, though progressing in a finite number of steps―and relatively few of them― proceed toward a theoretical limit of unity on three levels more or less simultaneously but quite independently. The three levels can be thought of as three distinct spheres where the outermost, largest, and last to be determined sphere contains the innermost, smallest, and first to be determined sphere, which is contained within the middle sphere which is determined after the first and before the third, and contains the former but is contained by the latter:

1) There must be increasing cumulative agreement on the subject-matter reflected in the scores on any form of valid assessment of the knowledge of the subject-matter of any course of study however it may have been presented.

2) There must also be increasing cumulative agreement on how and why distinct methods of interaction―explanations, demonstrations, directed practice, and so forth―serve to enhance the knowledge, skills, and abilities of the learners/students/trainees over time.

3) And, there must be increasing agreement on whatever abstract attributes of the instructor and the instruction that are required to achieve success in such communications.

Kolmogorov’s proofs about probability, and generalizations of those proofs incorporating the theory of pragmatic information [5] [6] [7] [8] [9] as well as the theory of true narrative representations [10] [11], show that all three of the just mentioned levels of successful instruction tend to progress toward the theoretical limit of complete agreement, or we could say 100% unity, in what can be defined mathematically as shared information. In Part One of this paper besides illustrating key implications of the mathematical proofs for instructional communication, I provide empirical evidence from four intensive studies of the evaluation of instruction showing a very general concurrence of both teachers and learners about the attributes conducive to instructional success. Then, in Part Two, which is practically a self-contained presentation of the mathematical reasoning and its testable empirical consequences, additional experimental evidence is provided from multiple studies showing that tested competencies of students engaged with diverse methods of instruction and in different kinds of subject-matters trend toward the convergences predicted: these trends, as will be shown empirically in both parts of this paper, but especially in Part Two, do go as predicted―toward unity in successful instructional communication and toward zero shared information in random responses to test questions modeling completely failed instructional efforts.

2. Removing Barriers

The hypothetical “barrier” between the “classroom” and the “real world” was a figment. As soon as we take account of the fact that schools, industries, and businesses all seek to instill or enhance real and measurable “knowledge, skills, and abilities” (KSAs) [12] in recipients of instruction, the fictional separation of the instructional process from the “real world” vanishes. Also, as Kim noted in 2004, a more comprehensive look at the whole process of instruction from beginning to end, ensures the trend toward tasks that involve “real-life communication” (p. 1).

Realizing from the start that instructional success must be judged more or less in reverse―by asking what end result would that instruction enable if it were successful (see the research and methodologies being developed at Quality Matters [13] )―our basic argument [2] from 17 years ago is sustained: we were already connecting the end of the road assessment procedures backward to the methods and subject-matter of the instruction. We noted back then that Eisner [14] had commented that “performance assessment [at the end of the road and along the way] is the most important development in evaluation since the invention of the short-answer test and its extensive use during World War I” (p. 2). The sort of assessment Eisner had in mind involved the end performances that students/trainees might be expected to be able to perform after the planned course of study or training. What, at the end of the day, is the instruction supposed to deliver in terms of KSAs? To show that I am not generalizing here in some unusual or grandiose manner, the Wikipedia article [12] on KSAs says:

KSA statements are also known as Evaluation Factors. Other agencies sometimes call them “Rating Factors”, “Quality Ranking Factors”, “Knowledge, Abilities, Skills, and Other Characteristics”, or “Job Elements”.

Kim [15] defined “performance assessment”―based on five decades of prior research―as “any assessment procedure that involves either the observation of behavior in the real world or a simulation of a real-life activity with raters to evaluate the performance” (p. 1).

2.1. Training in an Exemplary High-Stakes Industry

Given that English is the required language for international aviation, our starting example in 2005 [2], was about getting air traffic controllers and pilots in international aviation fully up to speed in their English proficiency. The relevant research showed that tens of thousands of accidents [16], some of them horrendously injurious or fatal [16], were predominantly attributable to breakdowns in communication because the interlocutors were using a language not entirely intelligible to themselves [17] [18] [19] [20] [21]. The research showed that the high risk moments in international aviation arise when communications between pilots and air traffic controllers must take place in one or several distinct dialects of English between persons whose primary language is one of the other 7000 plus languages of the world [22].

2.2. Instruction Is Important

We supposed that the only adequate remedy for the well-documented breakdowns in communication in international aviation would be better English language instruction of the personnel―in particular air traffic controllers, pilots, and others involved in guiding passengers, cargos, and carriers. More specifically, we argued that the course of study/training should address the full range of contexts of communication about aircraft, airports, runways, equipment, traffic patterns, signals, lighting, and in general the subject-matter that interlocutors working in what may be a non-primary language for many of them, need to know at a level approximating perfection. The folks whose lives are on the line need to know that when the plane departs from a gate on one side of the world, for instance, that it is almost certain to arrive safely at the one it is destined to reach possibly on the other side.

Although not all contexts of instruction have the same high stakes as those in international aviation, many aspects of the problems presented in those contexts generalize appropriately. No matter what the stakes may be in any given area of instruction/training, the goal in general is to maximize success and minimize failure. No instructional process aims for failure any more than businesses are built with the intention of losing money rather than making a profit [23].

3. Three Critical Components of Instruction

To accomplish the goal of maximizing success (while minimizing the possibility of failure), it is generally agreed that 1) valid planning with respect to the subject-matter, 2) interesting and engaging communication of that subject-matter, and 3) valid assessment of the uptake of the subject-matter in real life scenarios aiming for optimal performance with multiple checks, backups, and follow-ups are essential. In fact, these three components are generalized, incorporated, and expressed in a multitude of ways in questionnaires for evaluating instruction, or most any training in higher education, the business world, sports, dance, and performance arts coaching, and in human industries in general. According to the recent work in Maryland by the growing community of researchers, teachers, and trainers following what is taking place at Quality Matters [13], the three principal components of any course of instruction must be brought into agreement (they call it “alignment”) with each other.

So we ask, how can the policy-makers and the educators who implement the required procedures of instruction/training ensure with nearly complete confidence that the persons successfully working through some course of study in a given subject-matter will end up having the necessary KSAs for optimal performance at the end? In an industry such as international travel by air, the critical actors―pilots and air traffic controllers―need to be nearly perfect in their performances all the time in all the contexts that can and do arise. Expressing all this in the simplest language of Kolmogorov’s probability theory and extensions of it to be made in the two parts of this paper, designers of instruction, it seems, must set up the instruction in such a manner as to make the likelihood of success as near to unity (perfect) as possible and the likelihood of failure as near to nothing (zero) as possible.

3.1. Questionnaires for the Evaluation of Instruction

The driving force behind the provision of instruction/training in any context is to enable recipient/participants to acquire KSAs shared by one or many persons who have either already achieved them or who know how to do so and are a little farther along the road toward doing so than their students are. The objective is always to increase the powers of students in the direction of those already possessed by the designers of the course of study―the teachers, coaches, trainers, and the like. Students hope to progress to the level of more advanced persons in the professions, businesses, industries, sports, or what-have-you who have already paid their dues and have achieved a desirable and advanced level in the KSAs of the “subject-matter”. The instructors are not expected to be perfect performers but they need to be substantially beyond the level of the student/trainees who are invited, or possibly in some cases at universities and colleges, required to successfully complete certain courses of study. For this reason, almost universally, questionnaires for the evaluation of instruction/training, address the alignment/agreement of the measures or performances required at the end of the course of study with the subject-matter, the methods of teaching/training, and the overall course design used to instill the desired gains in the KSAs aimed at. The respondents may be asked: 1) to agree or disagree with a series of statements; 2) to select from multiple choices a level of agreement or disagreement, or to choose among several alternative answers the one that best expresses the evaluator’s view; and/or 3) there may be open-ended questions asking the evaluator to compose a short response or an essay expressing something judged to be of importance about the course of study. Questions may ask what was liked or disliked, what should be kept or discarded, what worked and didn’t work, etc. Here are some examples.

3.1.1. Evaluating Massive Open Online Courses

Cutting directly to the present tense of the evaluation of thousands of adult-level courses of instruction already reaching millions of people throughout the world, in 2021 Deng and Benckendorff [24] analyzed 8475 ratings and reviews from 1794 distinct Massive Open Online Courses (MOOCs) aimed, according to them at the “social sciences”. They focused attention on positive reviews, which the mathematical proofs to be discussed below show are the proper ones of interest. If we are aiming for success in enhancing KSAs, the only instructional attributes of interest are the ones that enable progress toward success, and we are hardly interested (for reasons to be made entirely clear later in this paper) in all the different ways there may be to fail. Summarizing what has been proved mathematically by Kolmogorov and by generalizations of his proofs, the bottom-line is that there are always uncountably many more ways to fail―e.g., not to construct a lightbulb that works, not to succeed in a running a four-minute mile, not to get all the way to the top of Mount Everest, not to learn how to converse in Navajo, etc.―than there are ways to succeed. The proofs show why agreement on success must generally converge to unity (a probability of 1) precisely to the extent that the course has any success at all (like the slightest positive curvature of space assures us that it must be a finite sphere) while the likelihood of failure, amounting to a complete absence of communication must trend toward a probability of 0 information shared.

From their extensive computer-assisted analysis, Deng and Benckendorff distilled six critical components of successful MOOCs (loosely summarized in the “thematic map” of Figure 1): some of their criteria are generalizable to all courses of instruction aimed at advancing any learner’s KSAs in any given subject-matter. I will sum these up in my own words:

1) realistic representation of the subject-matter,

2) requiring performance by students in real-life tasks demonstrating mastery of the subject-matter,

3) making the course experience interesting, engaging, so as to capture and hold the student’s attention,

4) investing sufficient resources and energy in the course design (video lectures for the MOOCs at issue in the study by Deng and Benckendorff) so that presentations are of high-quality,

5) illustrating difficult concepts and principles in intelligible and interesting ways in the presentations (video lectures, etc.),

6) inviting participation by incorporating comments and questions from students enabling them to become integral contributors helping instructors to make things clear to all the students.

One inevitable constraint on the computer-assisted analysis performed by Deng and Benckendorff is that their software performed its analysis mainly on the lexical items and phrases appearing in the questionnaires and responses of the hundreds of students included in the analysis summed up in Figure 1. Nevertheless, their analysis implicitly includes the three major types of agreement and at least suggests that even moderately successful instruction does converge toward 1) agreement on whatever the subject-matter is, 2) agreement on whatever methods of presentation successfully get the subject-matter across, and, finally, 3) agreement on the attributes conducive to instructional success in general.

3.1.2. “12 Amazing Course Evaluation Survey Questions”

Next we come to an organization online that identifies itself as the “QuestionPro” [25]. The website I am referring to at this link offers “Survey Software” among other instruction-related services, some of it for free, and some with differential pricing for collection and storage of more than 1,000 responses to any one of their constructed surveys. The authors of the site begin by telling visitors why colleges and universities “must… conduct evaluation surveys”. They group their questions into three categories: those pertaining to the instructor, the course material and structure (methods) of presentation, and general satisfaction of the respondent with the course. Their “12 Amazing Course Evaluation Survey Questions” [25] consist of statements followed by an invitation to choose from a list of five choices expressing a relative frequency ranging from “almost always” to “almost never” or “strongly agree” to “strongly disagree”. Here are the statements verbatim followed in brackets by [the type of scale used, and the evaluation target the authors of the website believe the item addresses]:

1) The instructor was well prepared for the class. [frequency, instructor]

2) The instructor showed an interest in helping students learn. [frequency, instructor]

3) I received useful feedback on my performance on tests, papers, etc. [agree-disagree, instructor]

4) The lectures, tests, and assignments complemented each other. [agree-disagree, instructor]

5) The instructional materials (i.e., books, readings, handouts, study guides, lab manuals, multimedia, software) increased my knowledge and skills in the subject matter. [agree-disagree, course materials/methods]

6) The course was organized in a manner that helped me understand the underlying concepts. [agree-disagree, course materials/methods]

7) The course gave me the confidence to do more advanced work in the subject. [agree-disagree, course materials/methods]

8) The examinations, projects measured my knowledge of the course material. [agree-disagree, course materials/methods]

9) I believe that what I’m being asked to learn in this course is important. [course materials/methods, agree-disagree]

10) I would highly recommend this course to other students. [agree-disagree, general satisfaction]

11) Overall, this course met my expectations for the quality of the course. [agree-disagree, general satisfaction]

12) The course was helpful in progress toward my degree. [agree-disagree, general satisfaction]

The authors then go on to mention different types of questions that may be included in course evaulation surveys and questionnaires. They differentiate “closed-ended” and “multiple choice” questions (which are really a single forced-choice category), from questions about preferences (which can be either forced-choice or open-ended questions eliciting short answers or longer essays about what students like or don’t like), and, finally, they suggest questions in which respondents are asked to rank different options (possibly statements about the quality of instruction) in the order of their judged importance. Again, it is easy to see that the QuestionPro designers have in mind the three independent levels of agreement already introduced on subject-matter, successful methods of presentation, and the abstract qualities of instructional success.

3.1.3. Example from Laupper, Balzer, and Berger

For reasons that will become very clear later in Part One of this paper, the next study I want to examine closely compares the old-school paper and pencil surveys typically filled out in a classroom setting―or after the conclusion of a course of study to be returned to the college or training facility―with the contemporary online survey more widely used since COVID-19. The research to be reviewed from Laupper, Balzer, and Berger [26] is important for various reasons but among them is the possibility―if the methods should prove to be equivalent in their statistical properties―of disposing of the claim that research with offline questionnaires is not relevant to the online type, and vice versa. In fact, Laupper, Balzer, and Berger focused on the specialized agreement between obviously distinct methods of collecting data about the quality of instruction. Empirically, in Figure 2, and in their discussion of it, they have demonstrated precisely the sort of convergence that is logically required by the generalization of Kolmogorov’s proofs, as discussed later, here in Part One of this paper.

Laupper, Balzer, and Berger [26] tested the convergence expected empirically and required mathematically, by applying a confirmatory factoring method. They collected responses to seventeen survey items from 232 respondents to an online version in 17 courses and from 231 similar respondents to an offline version in

16 courses at a Swiss (German) institution of higher education. All 33 of the courses they examined were focused on the same sort of vocational and professional training. The specific items were these (in their own words):

1) The course content was adequately covered.

2) I consider that the impact of this course on widening my skills is low. (reversed)

3) The information provided is very useful to me.

4) I succeeded in meeting the objectives set in the course programme.

5) The documents provided proved helpful in understanding and learning.

6) During the course, there were a number of elements I did not understand. (reversed)

7) The methods used in the course were suited to the content.

8) I met my personal objectives for this course.

9) The course objectives were clear and understandable.

10) The lecturers were able to link theory to practice.

11) The lecturers explained the subjects in a clearly understandable way.

12) The lecturers seemed competent in their field.

13) The lecturers made the course lively and convincing.

14) Where there were several lecturers, their various interactions went smoothly.

15) I am satisfied with the course venue and infrastructure.

16) I am satisfied with the follow-up by the SFIVET [Swiss Federal Institute for Vocational Education and Training] Secretariat.

17) I am satisfied with the Click&Book registration process.

The confirmatory factor analysis summed up in Figure 2 shows that both the online and offline questionnaires resolved to three major components as expected by the designers of the questionnaire and which I paraphrase as follows:

1) Subject-matter addressed in… “nine items related to how adequate, informative, helpful, understandable, and goal-oriented” the course was judged to be;

2) methods of presentation by “lecturers” addressed in “five items” about “lecturers’ competencies in linking theory with practice and creating a stimulating learning climate”;

3) overall personal satisfaction with the course design, the evaluation process itself (“follow-up”), and the “registration process”.

More importantly, the two methods of collecting data from students about their evaluations of instruction proved to be equivalent in all of the ways tested by Laupper and colleagues. Though superficially different in their response mode―in their deep structure from the point of view of the discursive meanings considered by intelligent human evaluators―the two modes of course evaluation appear to be equivalent for all practical purposes. Among other conclusions, it appears that research with paper and pencil, offline questionnaires and surveys of the quality of instruction in higher education, remains as relevant as it ever was in spite of the fact that educational institutions all over the world are proceeding more and more into the post-COVID-19 world of remote instruction via the internet. It appears that the “factor structure and relationships between the dimensions within it” as summed up in Figure 2 from Laupper, Balzer, and Berger are sufficiently similar across the two modes as to make them largely interchangeable. Also, those findings suggest that research into the evaluation of instruction relying on either of the two modes of eliciting responses from students should largely be generalizable to the other.

3.1.4. Quality Indicators Ranked by Persons of Increasing Experience

With all the foregoing in mind, but especially the findings from Laupper, Balzer, and Berger, I want to look back to an unpublished research project completed three decades ago. In the spring of 1991, I obtained permission from the University of New Mexico Institutional Research Board for a research project funded by the Public Service Company of New Mexico Foundation through their Distinguished Educator Award Program. It was titled: “Improving Instruction through On-Going Evaluation”. At that time, the head of the University of New Mexico Committee on Teaching Enhancement, Charles Beckel, a Professor in the Department of Physics and Astronomy there, asserted that about 95% of UNM faculty see instructional quality as a low priority in terms of budget, rewards, etc., but more than 95% hold the quality of instruction to be one of their highest personal priorities¹.

Our charge was to address teaching quality at our institution from various angles. The committee eventually settled on administering a university-wide survey to give faculty and students a chance to rank the most common criteria for evaluating the success of university level courses as distilled from extant instruments for student evaluation of instruction in higher education. Members of the faculty were also invited to write an essay on what instructional excellence consists of and why it is a priority in their thinking as members of the university faculty. A cash prize was offered for the best essay as judged by the aforementioned subcommittee and plaques of recognition were to be offered for the 10 most interesting runners-up in the essay writing contest.

In the final phase of that research project, the questionnaire in Figure 3 was distributed to faculty and students asking for age, sex, rank if a faculty member, and level if a student (Table 1), as well as a “yes” or “no” about whether course evaluations should be required and if so whether results should be published. Also faculty and students ranked the “possible elements of good teaching” and judged how well faculty peers, students at various levels, and the instructors, could rate those particular elements of successful instruction.

As seen in Table 1, 52% of the student respondents were undergraduates in the 18 - 24 age range, the bulk of them in their junior or senior year of undergraduate work, and 66% of them were on the main campus, with 24% at the Medical School and 2% at the Law School. The majority of faculty respondents were in the age range from 41 - 50 with 36% at 51 or older. No faculty respondents were in the 18 - 30 age range. There were almost as many Assistant Professors as Associate Professors, and only a handful of Instructors. For that reason, the calculations for the questions about mandatory evaluation and publication of the results were limited to Assistant, Associate, and Full Professors, though all student respondents were included in that portion of the table. Of the 327 faculty respondents who addressed our first question, 56% said “yes” to mandatory course evaluation and a slightly different group of 321 faculty members would favor publication of the results if such a mandatory course evaluation policy were in place. Students were even more positive in their responses to both questions with 86% of 848 respondents saying “yes” to both questions: students want courses evaluated and they want the results to be published. Also, there was an interesting trend of apparently increasing agreement across time for both faculty and students.

References 1

Dewey, J. (1929) Nature, Communication, and Learning. In: Hayden, D.E. and Alworth, E.P., Eds., Classics in Semantics, Philosophical Library, New York, 265-296.

Badon, L.C., Oller, S.D., Yan, R. and Oller, J.W. (2005) Gating Walls and Bridging Gaps: Validity in Language Teaching, Learning, and Assessment. Studies in Applied Linguistics and TESOL, 5, 1-15.

(2021) Pinky and the Brain [Internet]. Wikipedia.

Riemann, B. and Clifford, T.W.K. (1873) On the Hypotheses Which Lie at the Bases of Geometry. Nature, 8, 14-17, 36-37. https://doi.org/10.1038/008036a0

Weinberger, E.D. (2002) A Theory of Pragmatic Information and Its Application to the Quasi-Species Model of Biological Evolution. Bio Systems, 66, 105-119. https://doi.org/10.1016/S0303-2647(02)00038-2

Gernert

,et al. (2006)Pragmatic Information: Historical Development and General Overview Mind and Matter 4, 141-167.

Oller, J.W. (2013) Pragmatic Information. In: Marks II, R.J., Sanford, J.C., Dembski, W.A. and Gordon, B.L., Eds., Biological Information: New Perspectives, World Scientific Publishing Company, Singapore, 64-86. https://doi.org/10.1142/9789814508728_0003

Maleeh, R. (2014) Pragmatic Information as a Unifying Biological Concept. Information, 5, 451-478. https://doi.org/10.3390/info5030451

Zhao, M., Liu, T. and Chen, F. (2018) Automatic Processing of Pragmatic Information in the Human Brain: A Mismatch Negativity Study. Neuroreport, 29, 631-636. https://doi.org/10.1097/WNR.0000000000001009

Oller, J.W. (2010) The Antithesis of Entropy: Biosemiotic Communication from Genetics to Human Language with Special Emphasis on the Immune Systems. Entropy, 12, 631-705. https://doi.org/10.3390/e12040631

Oller, J.W. (2014) Biosemiotic Entropy: Concluding the Series. Entropy, 16, 4060-4087. https://doi.org/10.3390/e16074060

(2020) Knowledge, Skills, and Abilities [Internet]. Wikipedia.

(2021) Quality Matters [Internet].

Eisner

E.W.

,et al. (1999)The Uses and Limits of Performance Assessment Phi Delta Kappan 80, 658-660.

Kim

,et al. (2004)Task-Based Performance Assessment for Teachers: Key Issues to Consider Studies in Applied Linguistics and TESOL 4, 1-5.

Ritter, J. (1996) Transcript of Crash Shows Controller Error/Review Reveals Poor English “Over and Over”. USA Today, 7A.

Day, B. (2002) Proposed ICAO Proficiency Requirements in Common English. Part I: The Need for English Language Proficiency Requirements and the Role of the International Civil Aviation Organization. 8th International Aviation English Association Seminar, Warsaw, September 2002, 44.

Tajima, A. (2004) Fatal Miscommunication: English in Aviation Safety. World Englishes, 23, 451-470. https://doi.org/10.1111/j.0883-2919.2004.00368.x

International Civil Aviation Organization (2004) Language Proficiency (Presented by the Secretariat). In: Fourteenth Meeting of the APANPIRG ATM/AIS/SAR Sub-Group, International Civil Aviation Organization, Bangkok, 5.

Yan, R. (2009) Assessing English Language Proficiency in International Aviation: Issues of Reliability, Validity, and Aviation Safety. Lambert Academic Publishing, Koln.

Yan, R. (2013) Assessing the English Language Proficiency of International Aviation Staff. In: The Companion to Language Assessment, American Cancer Society, New York, 484-496. https://doi.org/10.1002/9781118411360.wbcla050

Eberhard, D.M., Simons, G.F. and Fennig, C.D. (2019) Ethnologue: Languages of the World [Internet]. 22nd Edition, SIL International, Dallas.

Broudy, D. (In Press) Transgressing the Logic of the New Tyrannical Normal: An Essay on the Struggle for Reason and Freedom. International Journal of Environmental Research and Public Health.

Deng, R. and Benckendorff, P. (2021) What Are the Key Themes Associated with the Positive Learning Experience in MOOCs? An Empirical Investigation of Learners’ Ratings and Reviews. International Journal of Educational Technology in Higher Education, 18, 9. https://doi.org/10.1186/s41239-021-00244-3

(2021) 12 Amazing Course Evaluation Survey Questions [Internet]. QuestionPro.

Laupper, E., Balzer, L. and Berger, J.-L. (2020) Online vs. Offline Course Evaluation Revisited: Testing the Invariance of a Course Evaluation Questionnaire Using a Multigroup Confirmatory Factor Analysis Framework. Educational Assessment Evaluation and Accountability, 32, 481-498. https://doi.org/10.1007/s11092-020-09336-6

Gnedenko, B.V. and Kolmogorov, A.N. (1968) Limit Distributions for Sums of Independent Random Variables [Internet]. Addison-Wesley, Boston.

Ryabko

B.Ya.

,et al. (1986)Noiseless Coding of Combinatorial Sources, Hausdorff Dimension, and Kolmogorov Complexity Problemy Peredachi Informatsii (Problems of Information Transmission) 22, 170-179.

Ryabko, B., Astola, J. and Gammerman, A. (2006) Application of Kolmogorov Complexity and Universal Codes to Identity Testing and Nonparametric Testing of Serial Independence for Time Series. Theoretical Computer Science, 359, 440-448. https://doi.org/10.1016/j.tcs.2006.06.004

Petoukhov, S.V. (2018) Structural Connections between Long Genetic and Literary Texts [v2]. https://doi.org/10.20944/preprints201812.0142.v1

Bayes, T. and Price, R. (1763) Essay towards Solving a Problem in the Doctrine of Chances. Philosophical Transactions of the Royal Society of London, 53, 370-418.

Haertel, E.H. (2013) Reliability and Validity of Inferences about Teachers Based on Student Test Scores. Center for Research on Human Capital and Education, Research and Development, Educational Testing Service, Princeton, The National Press Club, Washington DC. https://www.ets.org/Media/Research/pdf/PICANG14.pdf