Despite the concern that has been expressed about potential method biases, and the pervasiveness of research settings with the potential to produce them, there is disagreement about whether they really are a problem for researchers in the behavioral sciences. Therefore, the purpose of this review is to explore the current state of knowledge about method biases. First, we explore the meaning of the terms “method” and “method bias” and then we examine whether method biases influence all measures equally. Next, we review the evidence of the effects that method biases have on individual measures and on the covariation between different constructs. Following this, we evaluate the procedural and statistical remedies that have been used to control method biases and provide recommendations for minimizing method bias.

Sources of Method Bias in Social Science Research and Recommendations on How to Control It

Preface Chapter 1 The challenge Chapter 2 The nature of the evidence: A synthesis of meta-analyses Chapter 3 The argument: Visible teaching and visible learning Chapter 4: The contributions from the student Chapter 5 The contributions from the home Chapter 6 The contributions from the school Chapter 7 The contributions from the teacher Chapter 8 The contributions from the curricula Chapter 9 The contributions from teaching approaches - I Chapter 10 The contributions from teaching approaches - II Chapter 11: Bringing it all together Appendix A: The 800 meta-analyses Appendix B: The meta-analyses by rank order References

Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement

generalize create, solve unseen problems, extrapolate to unknown domains 22831.indb 124 6/15/11 2:11 PM Designing intended learning outcomes 125 • the verb at the appropriate level of understanding or of performance intended; • the topic content the verb is meant to address, the object of the verb in other words; • the context of the content discipline in which the verb is to be deployed. The ILOs for the course The Nature of Teaching and Learning illustrate these points: 1 Explain why a particular course topic is important to teaching. 2 Apply a course topic to your own teaching. 3 Refl ect on your teaching in terms of a working theory you have gained from the course. 4 Evaluate a situation that has gone wrong and apply a solution. The fi rst refers to declarative knowledge: the students have to reach a level of understanding that requires them to explain something, not just describe or list it: the latter only display multistructural levels of understanding, but explaining requires students to be able to relate the topic to the context of teaching and is at a relational level of understanding. The second is a functioning knowledge example also at the relational level as it requires a level of understanding that enables the student to apply the topic to teaching. The other two are also about functioning knowledge and should be at the relational to extended abstract level of understanding, depending on the originality of the student’s response. The content in (3) is the student’s own working theory and the context the student’s own teaching, and in (4), the content is the theory used in evaluating and the context the problematic situation in teaching. As a note on the number of ILOs per course, we stated earlier that there should be no more fi ve or six ILOs for any course, even though there may be up to ten topics that need addressing. The answer is to write integrating ILOs that address several topics, or, as in ILOs (1) and (2) above, the ILO allows the student to select just one topic for demonstrating ability to achieve the ILO. Another thing to watch out for are redundant ILOs, such as ‘Describe and explain . . .’. ‘Describe’ is redundant because if the student can explain the topic, he or she can certainly describe it. The other matter one should keep in mind at this stage is that desirable but unintended outcomes, or outcomes unforeseen by the teacher, may emerge. This is the nature of extended abstract responses by the student, and they will be accounted for in the normal assessment, but others may simply be things that the student sees as important and relevant learning. This matter becomes a practical issue during assessment, and we address it in Chapter 10. You should now be in a position to design and write your own ILOs for a course you are teaching (Task 7.1). 22831.indb 125 6/15/11 2:11 PM 126 Designing constructively aligned outcomes-based teaching and learning Task 7.1 Writing course ILOs Take a course that you are teaching. Consider the course aim and write the course ILOs by identifying: a the kind of knowledge to be learned (declarative or functioning). b the content or topic to be learned. c the level of understanding or performance to be achieved. d any particular context in which the outcome verb is to be enacted. The following grid may be a useful framework to help you think. Kind of knowledge Level of Content topic Context Declar/function understanding (outcome verb) ______________________________________________________________ ______________________________________________________________ ______________________________________________________________ ______________________________________________________________ ______________________________________________________________ ______________________________________________________________ Now go across the rows and write out the course ILOs by stating the intended level of understanding or performance (outcome verb), topic and the context in which the verb is to be enacted. There is no need to include the kind of knowledge in the ILO as that is defi ned by the verb(s) you use. To recap an example of a course ILO from our course The Nature of Teaching and Learning : Students should be able to: Refl ect (level of understanding and performance) on your teaching (context) in terms of a working theory you have gained from the course (content). Now write your course ILOs. Students should be able to: ILO1: _______________________________________________________ ILO2: _______________________________________________________ ILO3: _______________________________________________________ 22831.indb 126 6/15/11 2:11 PM Designing intended learning outcomes 127 Aligning ILOs at three levels: curriculum mapping Now that we have written the course ILOs, we have the task of checking to see that the three levels of intended outcomes, graduate, programme and course, are aligned. We can achieve this by curriculum mapping (Huet et al. 2009), which is a systematic means of ensuring alignment between programme ILOs and graduate outcomes, and course ILOs and programme ILOs. Graduate outcomes and programme ILOs Table 7.4 shows a simply way of checking the alignment between graduate outcomes and programme ILOs. The table is a device to ensure that the match between programme ILOs and graduate outcomes has at least been considered. Programme ILOs should not be forced to match graduate outcomes that don’t belong in the programme. Because of the different natures of different disciplines or professions, different programmes may have different emphases in addressing the graduate outcomes. It is not necessary that every programme should address all graduate outcomes to the same extent because some may not be relevant to the programme. Programme ILOs are simply the reasons that the programme is being taught, which is a matter of professional and academic judgment. However, university policy will prevail on this. Task 7.2 parallels Table 7.4: it asks you to align programme ILOs with the graduate outcomes of your university, if it has any. If the programme ILOs ILO4: _______________________________________________________ ILO5:_______________________________________________________ ILO6:_______________________________________________________ Review the ILOs to see whether: a the kind of knowledge, content and level of understanding or performance are relevant to achieve the course aim. b they cover all the main reasons for teaching the course. c they are clearly written, especially in identifying the level of understanding or performance to be achieved by the students, and the context (if appropriate). d the number is manageable for designing aligned teaching/learning activities and assessment tasks. How does this new set of course ILOs compare to your existing course ‘objectives’? Does the existing set need to be rewritten? 22831.indb 127 6/15/11 2:11 PM 128 Designing constructively aligned outcomes-based teaching and learning haven’t yet been articulated, discuss them with the programme coordinator and derive a set, then match them with the graduate outcomes. This should give you a clearer idea of how graduate outcomes can suitably be addressed in your teaching. How does your attempt gel with your university’s policy on this? Gelade and Fursenko (2007) also describe a tool for systematically mapping courses and programmes for graduate outcomes. Task 7.2 Aligning programme ILOs with graduate outcomes 1 Take a programme in which you are teaching and either list the programme ILOs if they are already articulated or, if they are not, sit down with the programme coordinator or programme committee chairperson and fi rst write the aims of the programme and a list of programme ILOs that meet those aims. 2 What are the graduate outcomes of your university? List them in the left-hand column in the grid below. 3 In the right-hand column list the programme ILOs that would address the graduate outcomes. Are all graduate outcomes addressed somewhere? Which are not? Does it matter?’ Graduate outcomes Programme ILO 1 2 etc. Table 7.4 An example of aligning programme ILOs with graduate outcomes Graduate outcomes Programme ILO Competent in professional Analyse and apply principles to real-life practice accounting situations Communicate effectively Communicate as a professional with clients and colleagues in real-life accounting situations Teamwork Operate effectively and ethically as a team member in real-life accounting situations Ethical professional As above 22831.indb 128 6/15/11 2:11 PM Designing intended learning outcomes 129 Programme ILOs and course ILOs The next level of alignment is between the programme and the course ILOs. As each programme is served by its constituent courses, it is important that, when aligning course ILOs to the programme ILOs, the course ILOs in total address all aspects of the programme ILOs. Often a programme ILO will be addressed by several courses, from different and increasingly more complex angles. You may attempt this in Task 7.3. Task 7.3 Aligning course ILOs with programme ILOs For individual teachers 1 List the programme ILOs of the programme. 2 List the course ILOs of the courses that you are teaching in a given programme. 3 Consider what programme ILO(s) each of the course ILOs addresses in the following table. Programme ILOs Course 1 ILOs Course 2 ILOs Course 3 ILOs

Teaching For Quality Learning At University

This is an introductory report for the Brief Symptom Inventory (BSI), a brief psychological self-report symptom scale. The BSI was developed from its longer parent instrument, the SCL-90-R, and psychometric evaluation reveals it to be an acceptable short alternative to the complete scale. Both test--retest and internal consistency reliabilities are shown to be very good for the primary symptom dimensions of the BSI, and its correlations with the comparable dimensions of the SCL-90-R are quite high. In terms of validation, high convergence between BSI scales and like dimensions of the MMPI provide good evidence of convergent validity, and factor analytic studies of the internal structure of the scale contribute evidence of construct validity. Several criterion-oriented validity studies have also been completed with this instrument.

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S0033291700048017

The Brief Symptom Inventory: an introductory report.

The purpose of this article is to examine how the field of mixed methods currently is being defined. The authors asked many of the current leaders in mixed methods research how they define mixed methods research. The authors provide the leaders' definitions and discuss the content found as they searched for the criteria of demarcation. The authors provide a current answer to the question, What is mixed methods research? They also briefly summarize the recent history of mixed methods and list several issues that need additional work as the field continues to advance. They argue that mixed methods research is one of the three major “research paradigms” (quantitative research, qualitative research, and mixed methods research). The authors hope this article will contribute to the ongoing dialogue about how mixed methods research is defined and conceptualized by its practitioners.

/pdf/toward-a-definition-of-mixed-methods-research-3l8bm02uz7.pdf

Toward a Definition of Mixed Methods Research

The traditional conception of validity divides it into three separate and substitutable types—namely, content, criterion, and construct validities. This view is fragmented and incomplete, especially because it fails to take into account both evidence of the value implications of score meaning as a basis for action and the social consequences of score use. The new unified concept of validity interrelates these issues as fundamental aspects of a more comprehensive theory of construct validity that addresses both score meaning and social values in test interpretation and test use. That is, unified validity integrates considerations of content, criteria, and consequences into a construct framework for the empirical testing of rational hypotheses about score meaning and theoretically relevant relationships, including those of an applied and a scientific nature. Six distinguishable aspects of construct validity are highlighted as a means of addressing central issues implicit in the notion of validity as a unified concept. These are content, substantive, structural, generalizability, external, and consequential aspects of construct validity. In effect, these six aspects function as general validity criteria or standards for all educational and psychological measurement, including performance assessments, which are discussed in some detail because of their increasing emphasis in educational and employment settings.

/pdf/validity-of-psychological-assessment-validation-of-2zgxanmue5.pdf

Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning.

The traditional conception of validity divides it into three separate and substitutable types – namely, content, criterion, and construct validities. This view is fragmented and incomplete, especially in failing to take into account evidence of the value implications of score meaning as a basis for action and of the social consequences of score use. The new unified concept of validity interrelates these issues as fundamental aspects of a more comprehensive theory of construct validity addressing both score meaning and social values in both test interpretation and test use. That is, unified validity integrates considerations of content, criteria, and consequences into a construct framework for empirically testing rational hypotheses about score meaning and theoretically relevant relationships, including those of both an applied and a scientific nature. Six distinguishable aspects of construct validity are highlighted as a means of addressing central issues implicit in the notion of validity as a unified concept. These are content, substantive, structural, generalizability, external, and consequential aspects of construct validity. In effect, these six aspects function as general validity criteria or standards for all educational and psychological measurement, including performance assessments, which are discussed in some detail because of their increasing emphasis in educational and employment settings.

https://onlinelibrary.wiley.com/doi/pdf/10.1002/j.2333-8504.1994.tb01618.x

Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning

Authentic and direct assessments of performances and products are examined in the light of contrasting functions and purposes having implications for validation, especially with respect to the need for specialized validity criteria tailored for performance assessment. These include contrasts between performances and products, between assessment of performance per se and performance assessment of competence or other constructs, between structured and unstructured problems and response modes, and between breadth and depth of domain coverage. These distinctions are elaborated in the context of an overarching contrast between task-driven and construct-driven performance assessment. Rhetoric touting performance assessments because they eschew decomposed skills and decontextualized tasks is viewed as misguided, in that component skills and abstract problems have a legitimate place in pedagogy. Hence, the essence of authentic assessment must be sought elsewhere, that is, in the quest for complete construct repre...

https://www.jstor.org/stable/info/1176219

The Interplay of Evidence and Consequences in the Validation of Performance Assessments

Because both score meaning and the value implications of scores as a basis for action are central issues in test validation, a unified view of validity is required that comprehends both the scientific and the ethical underpinnings of test interpretation and use. This unified concept of validity integrates considerations of content, criteria, and consequences into a construct framework for testing rational hypotheses about theoretically relevant relationships, including those of an applied as well as of a scientific nature. The essence of unified validity is that the appropriateness, meaningfulness, and usefulness of score-based inferences are inseparable and that the unifying force behind this integration is the trustworthiness of empirically grounded score interpretation, that is, construct validity.

Meaning and Values in Test Validation: The Science and Ethics of Assessment

Washback, a concept prominent in applied linguistics, refers to the extent to which the introduction and use of a test influences language teachers and learners to do things they would not otherwise do that promote or inhibit language learning. Some proponents have even maintained that a test's validity should be appraised by the degree to which it manifests positive or negative washback, a notion akin to the proposal of 'systemic validity' in the educational measurement literature. This article examines the concept of washback as an instance of the consequential aspect of construct validity, linking positive washback to so-called authentic and direct assessments and, more basically, to the need to minimize construct under- representation and construct-irrelevant difficulty in the test.

/pdf/validity-and-washback-in-language-testing-3n00hw8mlu.pdf

Samuel Messick

Papers

Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning.

Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning

The Interplay of Evidence and Consequences in the Validation of Performance Assessments

Meaning and Values in Test Validation: The Science and Ethics of Assessment

Validity and Washback in Language Testing.