For a better experience, click the Compatibility Mode icon above to turn off Compatibility Mode, which is only for viewing older websites.

Instruments and Methods

Asessment Instruments & Methods

Assessment of student learning can be conducted using a variety of available instruments and methods. Many experts believe that a combination of assessment approaches can be the most effective way to measure student learning. Fortunately, many departments on campus and at other institutions have acquired some experience with many of the more commonly used instruments. Faculty in a variety of academic programs at large and small research universities have tested and used a wide range of assessment methods to determine whether students were attaining prescribed educational goals. In this section, many of these assessment approaches will be presented providing handbook users with information that can simplify the development of assessment strategies.

Assessment Instruments & Methods to Assess Student Learning

Direct Measures

Indirect Measures


Course

  • Course and homework assignments
  • Examinations and quizzes
  • Standardized tests
  • Term papers and reports
  • Observations of field work, internship
  • performance,service learning, or clinical experiences
  • Research projects
  • Class discussion participation
  • Case study analysis
  • Rubric (a criterion-based rating scale) scores for writing, oral presentations, and performances
  • Artistic performances and products
  • Grades that are based on explicit criteria related to clear learning goals
  • Course evaluations
  • Test blueprints (outlines of the concepts and skills covered on tests)
  • Percent of class time spent in active learning
  • Number of student hours spent on service learning
  • Number of student hours spent on homework
  • Number of student hours spent at intellectual or cultural activities related to the course
  • Grades that are not based on explicit criteria related to clear learning goals

Program

  • Capstone projects, senior theses, exhibits, or performances
  • Pass rates or scores on licensure, certification, or subject area tests
  • Student publications or conference presentations
  • Employer, Co-op and internship supervisor ratings of students’ performance
  • Focus group interviews with students, faculty members, or employers
  • Registration or course enrollment information
  • Department or program review data
  • Co-op supervisor surveys
  • Job placement
  • Employer or alumni surveys
  • Student perception surveys
  • Graduate school placement rates

Institutional

  • Performance on tests of writing, critical thinking, or general knowledge
  • Explicit self-reflections on what students have learned related to institutional programs such as the co-op experience
  • Locally-developed, commercial, or national surveys of student perceptions or self-report of activities (e.g., National Survey of Student Engagement)
  • Transcript studies that examine patterns and trends of course selection and grading
  • Annual reports including institutional benchmarks, such as graduation and retention rates, grade point averages of graduates, etc.

Direct Indicators of Learning

Capstone Course Evaluation

Capstone courses integrate knowledge, concepts, and skills associated with an entire sequence of study in a program. This method of assessment is unique because the courses themselves become the instruments for assessing student teaching and learning. Evaluation of students' work in these courses is used as a means of assessing student outcomes. For academic units where a single capstone course is not feasible or desirable, a department may designate a small group of courses where competencies of completing majors will be measured.

Capstone courses provide students with a forum to combine various aspects of their programmatic experiences. For departments and faculty, the courses provide a forum to assess student achievement in a variety of knowledge and skills-based areas by integrating their educational experiences. Also, these courses can provide a final common experience for student in the discipline.

Many research universities are currently using capstone courses in a variety of academic disciplines including general education programs and other academic units. Departments at other research institutions using this instrument to gather information about student learning in the major include many general education programs, chemistry, political science, physics, music, religious studies, theatre, history, and foreign languages.

Course-Embedded Assessment

Assessment practices embedded in academic courses generate information about what and how students are learning within the program and classroom environment. Course-embedded assessment takes advantage of already existing curricular offerings by using standardized data instructors already collect or by introducing new assessment measures into courses. The embedded methods most commonly used involve the development and gathering of student data based on questions placed in course assignments. These questions, intended to assess student outcomes, are incorporated or embedded into final exams [most common], research reports, and term papers in courses. The student responses are then evaluated to determine whether or not the students are achieving the prescribed educational outcomes and objectives of the department. This assessment is a separate process from that used by the course instructor to grade the exam, report, or term paper.

There are a number of advantages to using course-embedded assessment.

  • Student information gathered from embedded assessment draw on accumulated educational experiences and familiarity with specific areas or disciplines.
  • Embedded assessment often does not require additional time for data collection, since instruments used to produce student learning information can be derived from course assignments already planned as part of the requirements.
  • The presentation of feedback to faculty and students can occur very quickly creating a conducive environment for ongoing programmatic improvement.
  • Course-embedded assessment is part of the curricular structure and students have a tendency to respond seriously to this method.

Tests and Examinations

In most cases, a test will be one part of a fully developed assessment plan. Tests are commonly used in association with cognitive goals in order to review student achievement with respect to a common body of knowledge associated with a discipline or program. Departments have traditionally used tests in assessment programming to measure whether students have acquired a certain process- and content-related knowledge.

Using this approach, there are two primary testing alternatives; first, locally developed/ faculty generated tests and examinations, and (2) commercially produced standardized tests and examinations. Locally developed testing and examinations are probably the most widely used method for evaluating student progress. For assessing the validity of an academic program, examinations designed by the instructors who set the educational goals and teach the courses is often the best approach. Cost benefits, interpretation advantages, and quick turnaround time all make using locally designed tests an attractive method for assessing student learning.

Tests designed for a specific curriculum can often prove more valuable when assessing student achievement than commercial instruments. These tests focus on the missions, goals, and objectives of the departments and permit useful projections of student behavior and learning. A well-constructed and carefully administered test that is graded for the specific purpose of determining program strengths and weaknesses remains one of the most popular instruments for assessing most majors. Departments at other research institutions using locally designed tests and examinations include mathematics, physical education, psychology, and English.

Commercially generated tests and examinations are used to measure student competencies under controlled conditions. Tests are developed and are normed nationally to determine the level of learning that students have acquired in specific fields. The ETS Biology Major Field Testm[currently in use in the Biology Department] would be such an example. Tests such as this one are widely used and can assist departments in determining programmatic strengths and weaknesses when compared to other programs and national data. Compilations of data on the performance of students who voluntarily take national examinations such as GRE and MCAT enable faculty to discover useful data that often leads to programmatic improvements.

When using commercially generated tests, national standards are used as comparative tools in areas such as rates of acceptance into graduate or professional school, rates of job placement, and overall achievement of students when compared to other institutions. In most cases, standardized testing is useful in demonstrating external validity.

There are a number of advantages for using commercial/standardized tests and examinations to measure student achievement;

  • Institutional comparisons of student learning are possible.
  • Very little professional time is needed beyond faculty efforts to analyze examinations results and develop appropriate curricular changes that address the findings.
  • Third, in most cases, nationally developed tests are devised by experts in the discipline.
  • Fourth, tests are traditionally given to students in large numbers and do not require faculty involvement when exams are taken by students.

As part of their assessment efforts, many institutions and programs already use a multitude of commercially generated examination and tests. Some of the more commonly used national tests include:

  • ACT - COMP (College Outcome Measures Program): This is an assessment instrument that measures knowledge and skills acquired by students in general education courses. Administered by ACT, Iowa City, IA.
  • GRE (Graduate Record Examinations): The GRE is widely used by colleges, universities, departments, and graduate schools to assess verbal and quantitative student achievement. Also, many discipline-specific examinations are offered to undergraduate students in areas such as Biology, Chemistry, Education, Geology, History, Literature, Political Science, Psychology, and Sociology. The GRE is published and administered by Educational Testing Services, Princeton, New Jersey.
  • Major Field Achievements Tests: Major field examinations are administered in a variety of disciplines. They often are given to student upon or near completion of their major field of study. These tests assess the ability of students to analyze and solve problems, understand relationships, and interpret material. Major field exams are published by Educational Testing Services, Princeton, New Jersey.

Portfolio Evaluation

In assessment as in other areas, a picture can be worth a thousand words. As an evaluation tool portfolio assessment has become widely used in higher education as a way to examine and measure progress, by documenting the process of learning or change as it occurs. Portfolios extend beyond test scores to include substantive descriptions or examples of what the student is doing and experiencing. Fundamental to "performance assessment" in educational theory is the principle that students should demonstrate, rather than tell about, what they know and can do. Contents of portfolios (sometimes called "artifacts" or "evidence") can include drawings, photos, video or audio tapes, writing or other work samples, computer files, and copies of standardized or program-specific tests. Data sources can include a variety of personnel who know the participant or program, as well as the self-reflections of participants themselves. Portfolio assessment provides a practical strategy for systematically collecting and organizing such data. Portfolios used for assessment purposes are most commonly characterized by collections of student work that exhibit to the faculty the student's progress and achievement in given areas.

Information about the students' skills, knowledge, development, quality of writing, and critical thinking can be acquired through a comprehensive collection of work samples. A student portfolio can be assembled within a course or in a sequence of courses in the major. The faculty determine what information or students' products should be collected and how these products will be used to evaluate or assess student learning. These decisions are based on the educational goals and expected outcomes.

Portfolio Assessment is most useful for:

  • Evaluating programs that have flexible or individualized goals or outcomes.
  • Each student's portfolio asseessment could be geared to his or her individual needs and goals.
  • Providing information that gives meaningful insight into behavior and related change. Because portfolio assessment emphasizes the process of change or growth, at multiple points in time, it may be easier to see patterns.
  • Providing a tool that can ensure communication and accountability to a range of audiences who may not have much sophistication in interpreting statistical data can often appreciate more visual or experiential "evidence" of success.
  • Allowing for the possibility of assessing some of the more complex and important aspects of many constructs (rather than just the ones that are easiest to measure).

Portfolio Assessment is not as useful for:

  • Evaluating programs that have very concrete, uniform goals or purposes.
  • Allowing you to rank participants or programs in a quantitative or standardized way
  • Comparing participants or programs to standardized norms. While portfolios can (and often do) include some standardized test scores along with other kinds of "evidence", this is not the main purpose of the portfolio

Pre-test/Post-test Evaluation

Pre-test/post test assessment is a method used by academic units where locally developed tests and examinations are administered at the beginning and at the end of courses or academic programs. These test results enable faculty to monitor student progression and learning throughout prescribed periods of time. The results are often useful for determining where skills and knowledge deficiencies exist and most frequently develop.

Video and Audio Evaluation

Video and audio devices have been used by faculty as a kind of pre-test/post-test assessment of student skills and knowledge. Disciplines, such as theatre, music, art, and communication, might consider this resourc as assessment tools.

Rubrics

A scoring rubric is a method of classifying and categorizing identified criteria for successfully completing an assignment or task and to establish levels for meeting these criteria. Rubrics should be used to assess essay questions, projects, portfolios and presentations and given to all faculty that are conducting and scoring the assessment. A well designed rubric will describe the definitions of each characteristic being assessed and descriptions of the best, worst and unacceptable characteristics of the identified criteria.

A rubric is an authentic assessment tool used to measure students' work. It is a scoring guide that seeks to evaluate a student's performance based on the sum of a full range of criteria rather than a single numerical score. A rubric is a working guide for students and teachers, usually distributed before the assignment begins in order to get students to think about the criteria on which their work will be judged. Rubrics can be analytic or holistic, and they can be created for any content area including math, science, history, writing, foreign languages, drama, art, music, etc.

The rubric is one authentic assessment tool which is designed to simulate real life activity where students are engaged in solving real-life problems. It is a formative type of assessment because it becomes an ongoing part of the whole teaching and learning process. Students themselves can be involved in the assessment process through both peer and self-assessment. As students become familiar with rubrics, they can even assist in the rubric design process. This involvement empowers the students and as a result, their learning becomes more focused and self-directed.

Three Common Features of Rubrics

Rubrics can be created in a variety of forms and levels of complexity, however, they all contain three common features which……:

  • Focus on measuring a stated objective (performance, behavior, or quality).
  • Use a range to rate performance.
  • contain specific performance characteristics arranged in levels indicating the degree to which a standard has been met

Advantage of Rubrics

Many experts believe that rubrics improve students' end products and therefore increase learning. When teachers evaluate papers or projects, they know implicitly what makes a good final product and why. When students receive rubrics beforehand, they understand how they will be evaluated and can prepare accordingly. Developing a grid and making it available as a tool for students' use will provide the scaffolding necessary to improve the quality of their work and increase their knowledge.

Rubrics offer several advantages:

  • Rubrics improve student performance by clearly showing the student how their work will be evaluated and what is expected.
  • Rubrics help students become better judges of the quality of their own work.
  • Rubrics allow assessment to be more objective and consistent.
  • Rubrics force the teacher to clarify his/her criteria in specific terms.
  • Rubrics reduce the amount of time teachers spend evaluating student work.
  • Rubrics promote student awareness about the criteria to use in assessing peer performance.
  • Rubrics provide useful feedback to the teacher regarding the effectiveness of the instruction.
  • Rubrics provide students with more informative feedback about their strengths and areas in need of improvement.
  • Rubrics accommodate heterogeneous classes by offering a range of quality levels.
  • Rubrics are easy to use and easy to explain.
  Beginning 1 Developing 2 Accomplished 3 Exemplary 4 Score
Stated Objective or Performance Description of identifiable performance characteristics reflecting a beginning level of performance. Description of identifiable performance characteristics reflecting development and movement toward mastery of performance. Description of identifiable performance characteristics reflecting mastery of performance. Description of identifiable performance characteristics reflecting the highest level of performance.  

Sample Undergraduate Research Presentation Rubric

Undergraduate research is becoming more important in higher education as evidence is accumulating that clear, inquiry-based learning, scholarship, and creative accomplishments can and do foster effective, high levels of student learning. This curricular innovation includes identifying a concrete investigative problem, carrying out the project, and sharing findings with peers. The following standards describe effective presentations.


Standards

5 - 4
Exemplary

3 - 2
Satisfactory

1-0
Unacceptable

Score

Weight

Total Score

Organization

Has a clear opening statement that catches audience’s interest; maintains focus throughout; summarizes main points

Has opening statement relevant to topic and gives outline of speech; is mostly organized; provides adequate “road map” for the listener

Has no opening statement or has an irrelevant statement; gives listener no focus or outline of the presentation

   

X 2

 

Content

Demonstrates substance and depth; is comprehensive; shows mastery of material

Covers topic; uses appropriate sources; is objective

Does not give adequate coverage of topic; lacks sources

 

 

X 2

 

Quality of conclusion

Delivers a conclusion that is well documented and persuasive

Summarizes presentation’s main points; draws conclusions based upon these points

Has missing or poor conclusion; is not tied to analysis; does not summarize points that support the conclusion

 

 

X 2

 

Delivery

Has natural delivery; modulates voice; is articulate; projects enthusiasm, interest, and confidence; uses body language effectively

Has appropriate pace; has no distracting mannerisms; is easily understood;

Is often hard to understand; has voice that is too soft or too loud; has a pace that is too quick or too slow; demonstrates one or more distracting mannerisms

 

 

X 1.5

 

Use of media

Uses slides effortlessly to enhance presentation; has an effective presentation without media

Looks at slides to keep on track; uses an appropriate number of slides

Relies heavily on slides and notes; makes little eye contact; uses slides with too much text

 

 

X 1.5

 

Response to Questions

Demonstrates full knowledge of topic; explains and elaborates on all questions

Shows ease in answering questions but does not elaborate

Demonstrates little grasp of information; has undeveloped or unclear answers to questions

 

 

X 1

 

Indirect Indicators of Learning

External Reviewers

Peer review of academic programs is a widely accepted method for assessing curricular sequences as well as course development and delivery. Using external reviewers is a useful way of analyzing whether student achievement correlates appropriately with departmental goals and objectives. In numerous instances, recommendations initiated by skilled external reviewers have been instrumental in identifying program strengths and weaknesses leading to substantial curricular and structural changes and improvements. This is a key component of the Program Alignment & Review [PAR] process here at Drexel.

Student Surveying and Exit Interviewing

Student surveying and exit interviews have become increasingly important tools for understanding the educational needs of students. When combined with other assessment instruments, many departments have successfully used surveys to produce important changes. During this process, students are asked to reflect on what they have learned as majors in order to generate information for program improvement. Through using this method, we can gain insight into how students experience courses, what they like and do not like about various instructional delivery approaches, what is important about the classroom environment that facilitates or hinders learning, and the nature of assignments that foster student learning. The university conducts a senior exit survey prior to each commencement.

Alumni Surveying

Surveying of alumni is a useful assessment tool for generating data about student preparation for professional work, program satisfaction, and the relevancy of curriculum. As an assessment supplement, alumni surveys provide departments and faculty with a variety of information that can highlight program areas that need to be expanded or enhanced. In most cases, alumni surveys are an inexpensive way to gather data and for reestablishing relationships with individuals that want to help the program continually improve. The Alumni Survey is currently in development at Drexel.

Employer and Co-Op Surveying

Employer/Co-Op surveys can provide information about the curriculum, programs, and students that other forms of assessment cannot produce. Through surveys, colleges and departments within a university can review employer satisfaction levels with the abilities and skills of both recent graduates as well as co-op students. The advantages in using employer surveys (as well as using student surveys of their COOP experiences) include the ability to obtain both internal (Drexel undergraduate students) and external data (Employers) that cannot be produced on campus, and the responses are often useful to help students discern the relevance of their educational experiences.

Curriculum and Syllabus Analysis

Once a department/program has defined its outcomes, all phases of the curriculum and each individual course would almost automatically cover most of the bases needed to provide each student with the opportunity to learn the essential components of those outcomes. That said, it must also be stated that not every course needs to attempt to cover all the outcomes for the major. The curriculum map or analysis provides a means to chart just which courses will cover which objectives. The chart then provides assurance to the department that, assuming certain sequences are taken by the student candidates for that major, they will in fact have the opportunity to learn those objectives.

Syllabus analysis is an especially useful technique when multiple sections of a department course are offered by a variety of instructors. It provides assurance that each section will cover essential points without prescribing the specific teaching methods to be used in helping the students learn those common objectives.

References

University of Wisconsin, Office of the Provost

Julian, Faye D. "The Capstone Course as an Outcomes Tests for Majors." Assessment in Practice. Banta, Trudy W., Lund, Jon P., Black, Karen E., & Oblander, Frances W., (Eds). San Francisco: Jossey-Bass Publishers, 1996. pp. 79-81.

The Use of Portoflio Assessment in Evaluation, Sewell, Marczak & Horn

The Use of Portoflio Assessment in Evaluation, Sewell, Marczak & Horn

Heidi Goodrich Andrade. "Understanding Rubrics." [Online] 22 October 2001.

Teachervision.com. The Advantages of Rubrics: Part One in a Five-Part Series. [Online] 22 October 2001.

Nancy Pickett and Bernie Dodge. "Rubrics for Web Lessons." [Online] 22 October 2001.