Doing Assessment as if Deep Learning Matter Most
Engaging Faculty and Students in Assessing and Promoting High-Impact Practices

The Inaugural Conference on Learning and Assessment

Materials for the Opening Plenary Session on 10 September 2014

Tom Angelo
Professor of Higher Education
Assistant Provost and Founding Director
Center for the Advancement of Faculty Excellence
Queens University of Charlotte (NC)

Guidelines 1 & 3

Values
________________________________________________
	________________________________________________
_________________________________________________________________
Goal Ranking and Matching Exercise

1.	On the lines below, please list three or four learning goals you hope to achieve–things 
you hope to learn or questions you hope to answer–through participating actively.

Your personal learning goals for this conference  . . . 

	________________________________________________
	________________________________________________
	________________________________________________

2.	Now, rank your goals in terms of their relative importance to you.  
Make the most important goal #1, the next most important #2, and so on.

3.	Next, working with your group of 3-4 colleagues, determine quickly whether you have any first- or second-ranked goals in common. Determine which one or two are most widely shared.

4.	Prepare to report out which goals were shared within your group and to what extent. 
For example, “Three out of four of us hope to learn X.”

Guidelines 2, 4 & 7

Tri-national Background Knowledge Probe
													___

Please answer each question below for the following three countries: The USA, the Ukraine and Syria.  Guessing is encouraged and expected!

				            USA 		 UKRAINE        SYRIA

1.	Population?		__________ 	___________	_________	

2.  	Area in square kms?	__________ 	___________	_________

3.	Capital city?		__________ 	___________	_________

4.	Name of current head
	of national gov’t		__________ 	___________	_________

5.	Name of most likely
next head of gov’t?	__________ 	___________	_________

6.	Corruption ranking?	__________ 	___________	_________

7.	Per capita income?	__________ 	___________	_________

8. 	%-age GDP to Defense? __________ 	___________	_________

9.      %-age child poverty?     __________ 	___________	_________

Additional questions

10.  __________________________________________________________________

11.  __________________________________________________________________

12.  __________________________________________________________________ 

13.  __________________________________________________________________ 

Guidelines 1, 3, 4 & 7

SIX DIMENSIONS OF HIGHER LEARNING OUTCOMES

Approximately what percentage	What percentage of their total
of all the instruction you got	instruction your undergraduate
during your own undergraduate  	students will receive during  
degree program focused on . . . 	2015-2018 should focus on . . . 

			Factual Learning					
				Learning What (Level 1)
					Learning facts and principles
 
			Conceptual Learning				
				Learning What (Level 2)
					Learning concepts and theories

			Procedural Learning				
				Learning How
					Learning skills and procedures

			Conditional Learning				
				Learning When and Where		
					Learning applications

			Metacognitive Learning				
				Learning How to Learn	
					Learning to direct and								                            	    manage one’s own learning

			Reflective Learning					
				Learning Why		
					Developing self-knowledge, 
                                                             cultural awareness, ethics, etc.
________								________	
    100%								            100%
Guidelines 2 & 6

Seven Transformative Guidelines for 
Doing Assessment as if Deep Learning Matters Most

1.  Build shared trust.  Begin by lowering personal, interpersonal and organizational 
     barriers to risk taking and change.

2.  Build shared language and concepts.  Develop a collective understanding of 
     the key concepts (mental models) needed for transformation.
  
3.	Build shared goals and motivation.  Collectively determine goals worth working toward and problems worth solving – and consider the likely costs and benefits.
  
4.	Design backward and work forward.  Design backward from that shared vision and long-term goals to develop coherent outcomes, strategies, and activities.
  
5.	Think and act systematically.  Analyze the opportunities and limitations presented by the larger system(s) within which we operate and seek connections and applications to those larger worlds.

6.	Take a scholarly approach.  Consult relevant valid theory and research. 
	Use what has already been learned about individual and organizational learning, change and assessment to inform, explain, and examine your plans and strategies.

7.  Don’t assume, ask.  Practice what we preach. Make the implicit explicit. 
     Demonstrate the value of assessment by using it ourselves—and on ourselves.
________________________________________________________________________________  
High Impact Practices (Kuh, 2008) 
•	First-Year Seminars and Experiences
•	Common Intellectual Experiences
•	Learning Communities
•	Writing-Intensive Courses
•	Collaborative Assignments and Projects
•	Undergraduate Research
•	Diversity/Global Learning
•	Service Learning/Community-Based Learning/Internships
•	Capstone Courses and Projects

Guidelines 5 & 6
Discussion Draft Questions for an 
Undergraduate Course/Teaching Feedback (not Evaluation!) Form

Questions about yourself              (1= Always, 2=Usually, 3=Sometimes, 4=Rarely, 5=Never, NA= Not Applicable)					
1.	I was self-motivated to learn this course material        	1        2       3       4       5     NA	

2.	I was well-prepared for each class session	1        2       3       4       5     NA	

3.	I asked the instructor for help/guidance when I needed it 	1        2       3       4       5     NA	

4.	I invested enough time and energy to meet/exceed course requirements	1        2       3       4       5     NA	

5.	I participated actively and contributed thoughtfully in class sessions 	1        2       3       4       5     NA	

6.	I attended class sessions and/or individual appointments            		1        2       3       4       5     NA

7.	Overall, I gave my best possible effort to learning in this course        	1        2       3       4       5     NA	

Questions about the course               (1= Always, 2=Usually, 3=Sometimes, 4=Rarely, 5=Never, NA= Not Applicable)

8.  The course was well-organized to help students learn	1        2       3       4       5     NA	

9.   The objectives and criteria for meeting them were made clear     	1        2       3       4       5     NA

10. The assignments contributed to my learning	1        2       3       4       5     NA	

11. The assessments/evaluations were clearly connected to the objectives	1        2       3       4       5     NA	

12. The amount of work required was appropriate to the objectives	1        2       3       4       5     NA

13.  The level of intellectual challenge was high	1        2       3       4       5     NA

Questions about the instructor            (1= Always, 2=Usually, 3=Sometimes, 4=Rarely, 5=Never, NA= Not Applicable)

14.	The instructor clearly connected the course objectives to course
      	    activities, assignments, and assessments	1       2       3       4       5     NA	

15.  The instructor encouraged me to connect my experience to the course	1       2       3       4       5     NA

16.  The instructor provided clear and useful feedback to improve learning	1       2       3       4       5     NA	

17.	The instructor inspired interest and excitement in the course material	1       2       3       4       5     NA

18.	The instructor was available and helpful when asked	1       2       3       4       5     NA	

19.	The instructor communicated ideas and information 
        clearly and effectively	1       2       3       4       5     NA	

20.  The instructor evaluated and graded fairly	1       2       3       4       5     NA	

21.  The instructor treated students and their ideas with respect	1       2       3       4       5     NA	

22.  The instructor used required texts/other required materials effectively	1       2       3       4       5     NA	

Summary Questions: Compared w/ other courses: (1=extremely high, 2=high, 3=adequate, 4=low, 5=very low)

23.  This course increased my desire to continue learning about this material	1       2       3       4       5     NA

24.  If a friend asked about taking this course, my recommendation would be 	1       2       3       4       5     NA	

25.  Overall, I would rate the quality of this course as	1       2       3       4       5     NA	

26.  Overall, I would rate the effectiveness of the instructor as	1       2       3       4       5     NA

27.  Overall, I would rate the amount I learned in this course as	1       2       3       4       5     NA

28.  Overall, I would rate the value of what I learned in this course as	1       2       3       4       5     NA

Guideline 7
Applications Card

DIRECTIONS:  Please take a moment to recall the ideas, techniques, and strategies we've discussed -- and those you've thought up -- to this point in the session.  Quickly list as many possible applications as you can.  Don't censor yourself!  These are merely possibilities.  
You can always evaluate the desirability and/or feasibility of these application ideas later.

Interesting 													Some possible 
IDEAS/TECHNIQUES									APPLICATIONS of those
from this session										ideas/techniques to my work	

																								
Reference:   Angelo, T.A. & Cross, K.P. (1993).  Classroom Assessment Techniques:  A Handbook for College Teachers, 
                       2nd edition.  San Francisco: Jossey-Bass, pp. 236-239. 

A Few Useful References on Teaching, Assessment and Learning

Ambrose, S.A., et al. (2010).  How Learning Works:  Seven Research-based Principles for Smart Teaching.  	        
San Francisco, CA: Jossey-Bass.

Anderson, L. &  Krathwohl, D.R. (Eds.) (2001). A Taxonomy for Learning, Teaching, and Assessment: 
 A Revision of Bloom’s Taxonomy of Educational Objectives (Abridged Edition). New York: Allyn & Bacon.

Angelo, T. (2012). Designing subjects for learning: Practical, research-based principles and guidelines.  In Hunt, L. &   
Chalmers, D. University Teaching in Focus: A Learning-centred Approach. London: Routledge, Taylor & Francis, 93-111.

Angelo, T. & Cross, K.P. (1993). Classroom Assessment Techniques: A Handbook for College Teachers. 
 San Francisco: Jossey-Bass.

Barkley, E.F. (2010). Classroom Engagement Techniques: A Handbook for College Teachers.  San Francisco: Jossey-Bass.

Biggs, J. & Tang, C. (2011). Teaching for Quality Learning at University, 4th Edition. Suffolk, England:  Society for Research into Higher Education.

Bransford, J.D., Brown, A.L, &  Cocking, R.R. (Eds.). (2000).  How People Learn: Brain, Mind, Experience, and School, Expanded Edition. Washington, DC: National Academy Press.

Davis, B.G.  (2009). Tools for Teaching, 2nd Edition.  San Francisco, CA: Jossey-Bass.

Donald, J.G. (2002). Learning to Think: Disciplinary Perspectives. San Francisco, CA: Jossey-Bass.

Doyle, T. & Zakrajsek, T. (2013). The New Science of Learning. Sterling, VA: Stylus.

Freeman, S. et al. (2014). Active learning increases student performance in science, engineering, and mathematics.        
PNAS Early Edition. Downloaded 14 April 2014 from www.pnas.org/cgi/doi/10.173.pnas.1319030111

Hattie, J. A. C. (2009). Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement.
 New York: Routledge.

Huba, M.E. & Freed, J.E. (2000). Learner-Centered Assessment on College Campuses: Shifting the Focus from 
 	Teaching to Learning. Needham Heights, MA: Allyn & Bacon.

Kuh, G.D. et al. (2006).  What Matters to Student Success: A Review of the Literature. Washington, DC: 
 	National Postsecondary Education Cooperative, U.S. Dept. of Education Institute of Education Sciences.

Maki, P.L. (2010). Assessing for Learning, 2nd Edition. Sterling, VA: Stylus.

Pascarella, E. & Terenzini, P. (2005). How College Affects Students: A Third Decade of Research. San Francisco, CA: Jossey-Bass.

Rhodes, T.L. (Ed.) (2010). Assessing Outcomes and Improving Achievement. Tips and Tools for Using Rubrics.
Washington, DC:  American Association of Colleges & Universities.

Suskie, L. ( 2009). Assessing Student Learning: A Common Sense Guide, 2nd Edition. San Francisco: Jossey-Bass.

Svinicki, M.D. (2004). Learning and Motivation in the Postsecondary Classroom. Bolton, MA: Anker.

Walvoord, B.E. & Anderson, V. (2010).  Effective Grading: A Tool for Learning and Assessment in College, 2nd Edition. 
 	San Francisco, CA: Jossey-Bass.

Wehlburg, C. M. (2008). Promoting Integrated and Transformative Assessment. San Francisco: Jossy-Bass.

Wiggins, G. & McTighe, J. (2005). Understanding by Design, Expanded 2nd Edition.  Mahwah, NJ: Merrill-Prentice-Hall