Scholarship of Teaching and Learning (SoTL) Methods

word cloud **
Content Analysis (Quantitative) Focus Groups Interview Descriptive Participant Observation Quasi-experimental Survey Research

Content Analysis: Quantitative Observational Research:*

Steps

1. Define and limit the population – what are you studying? People? Assignments? Courses?

2. Select coding units: Course? Answer? Conversational turn? Test items?

3. Determine classification systems for study: What categories will each unit need to be coded into – inductively or deductively? (Hint: This is an iterative process J)

Example (1):

Use of fonts emoticons (coding online courses for nonverbal immediacy)

Unit: entire class: Categorized based on the majority of pages, headings, documents, and announcements

Categories:  Use of “fun” fonts: less formal and business like; use of emoticons, manipulation of punctuation to indicate emotions (“interesting . . .”; not emphasis for assignments etc.), so multiple exclamations count!!! (single do not); nonverbal vocalizations (“hmmm . . .,  soooo?”); and/or emoticons (as symbols J or created ;).

            Emotion expressive: Often uses fun fonts and/or other nonverbal emotive indicators such as, emoticons, vocalizations etc.(more than half the pages/announcements etc. use some form of nonverbal emotive indicators)

            Emotion low: Sporadic (not consistent) uses of fun fonts and/or uses some other nonverbal emotive indicators
           
            Emotionless: No use of fun fonts or other nonverbal emotive indicators

4. Sample messages: Get your sample

5. Code messages: At least two coders for all or part of the data – determine interrater reliability (Scott’s pi, percent agreement, etc.  - see resources).

6. Analyze the data

Resources:

Content analysis methods guides:

Overview of content analysis: http://writing.colostate.edu/guides/guide.cfm

Quick guide to content analysis: https://www.ischool.utexas.edu/~palmquis/courses/content.html

Flow chart of content analysis process: http://academic.csuohio.edu/kneuendorf/content/resources/flowc.htm

Qualitative content analysis description:
http://www.qualitative-research.net/index.php/fqs/article/view/1089/2385%3E

Calculating interrater reliability:

Excellent website about calculating intercoder/interrater reliability:
http://ils.indiana.edu/faculty/hrosenba/www/Research/methods/lombard_reliability.pdf

Online tool to calculate reliability: http://dfreelon.org/utils/recalfront/

*Source for general information: Reinard, J.C. (2008). Introduction to Communication Research, 4th Edition. New York: McGraw-Hill.

(1) Dixson, M.D., Greenwell, M., Lauer, S., Rogers-Stacy, C., & Weister, T. (2014). Nonverbal immediacy behaviors and online student engagement: Bringing past instructional research into the present virtual classroom. Unpublished ms.

Dixson, Mack @ FACET, 2014

focus groupFocus Groups

Interviewing is a common qualitative research strategy. Group interviewing or focus groups is also a viable option. “With focus group interviews, the researcher is able to collect data from multiple participants and also to observe and record the interactions and group dynamics that unfold” (Lodico, Spaulding, & Voegtle, 2006).
Key points to consider:

During a focus group interview, Davies (2007) explains how it is necessary to begin the interview with an explanation of the task at hand complete with timeframe, guiding topics, and rules to follow. “Two useful ideas are to ask  people to be courteous to each other and to indicate that there is an expectation that all those present will be free to contribute equally” (Davies, 2007). Although you must begin with a set of preliminary questions, a focus group interview needs some flexibility to allow for group conversation. As the interviewer, you will interact as little as possible but interject questions that allow the group members to respond to each other (Lodico, Spaulding, & Voegtle, 2006).

References/Resources

Davies, Martin Brett. (2007). Doing a successful research project: Using qualitative or quantitative methods. New York: Palgrave/Macmillan.

Ely, Margot. (1991). Doing qualitative research: Circles within circles. London; New York: The Falmer Press.

Krueger, R.A. & Casey, M.A. (2009). Focus groups: A practical guide for applied research. Los Angeles: Sage.

Lodico, M.G., Spaulding, D.T., & Voegtle, K.H. (2006) Methods in educational research: From theory to practice. San Francisco: Jossey-Bass

Lofland, John & Lofland, Lyn H. (2004). Analysing social situations: A guide to qualitative observation and analysis. London: Wadsworth

McKinney, Kathleen. (2007). Enhancing learning through the scholarship of teaching and learning: The challenges and joys of juggling. San Francisco, CA: Jossey-Bass.

Saam, Mack @ FACET, 2014

Interview/Descriptive Researchpeople talking

Tracy (2013, p. 132) cites Briggs (1986) for the proposition that “approximately 90 percent of all social science investigations rely on interviews.” The goal of ethnographic interviewing, unlike surveys, is to get participants to tell stories that you can use as data that help you answer your research question. It’s particularly useful to learn how people experience and understand their world.

Structure

Interviews should start with questions that give you context and get your respondents warmed up and comfortable and then build up to questions that go directly to your research question. Some examples of warm-up/context questions include: How would you describe your role here in the firm, What sort of work do you do here, What appeals to you about this type of work, What is there that doesn’t appeal to you about this type of work? (What challenges are there?)

Having warmed up your participant and gotten context, you can next move to questions that go directly to your research question. One way to transition is to say "I'm interested in studying _________. Could you tell me about a time when _________?" Probe as suggested above as necessary to get participants to tell you their story.

Repeat the question-probe cycle as needed to get the participant to tell you more stories.

Phrasing

Because the goal is getting the participants to tell you stories, phrase the questions so they're conversational, using everyday language, and inviting participants to talk.

NOT "What were your instrumental and identity goals?"
BUT How would you describe what you wanted out of this disagreement? How did you want to be seen by [the other person]? What did you do to help [the other person] see you that way?

NOT "Name 5 rules of the org and how you believe they benefit your clients"
BUT "Tell me about some rules of the org that you think benefit the clients"

NOT "whose voice is most prominent in the organization"
BUT ""whose opinion matters most in the organization"

And of course, be sure you're asking the question you really intend to ask

NOT "what do you wish you could talk more about"
BUT "what do you wish you could talk more about with your boss," if that's what you mean

Asking for a specific instance can be a great way to get participants to tell stories

NOT "If the boss says something you do not agree with, what do you think that can teach you and/or be useful to you in your life?"
BUT "Can you tell me about a time the boss said something you didn't agree with, but you learned something from what (she) said?

Probing

Toward helping participants tell you stories, you will likely find yourself probing the answers participants give with questions that aren't on your list (e.g., "can you give me an example," "why," "what happened next")

Expressing interest, as in "that must have been difficult," is a useful way to probe. So is restating or incorporating their words as you probe, as in "what do you mean by ____?"

Analysis

(Way too) generally speaking, the analysis of interview data involves identifying themes that emerge from your interview transcripts, finding connections among those themes, and comparing what you have found to the pre-existing literature.

Some SOTL Examples

Caulfield & Woods interviewed students 9, 24, or 36 months after taking a class to assess its impact

Caulfield, J., & Woods, T. (2013). Experiential learning: Exploring its long-term impact on socially responsible behavior. Journal of the Scholarship of Teaching and Learning, 13(2), 31-48. http://josotl.indiana.edu/article/view/3235/3389

Interviewing can be used to triangulate other research, that is, to use another method to check the results and increase its credibility. Garland and colleagues used the answers to the open-response questions in their survey to develop interview questions. Dole and Bloom used interviews to triangulate data they gathered from course evaluations and discussion board postings.

Garland, D., Vince Garland, K., & Vasquez, E. (2013). Management of classroom behaviors: Perceived readiness of education interns. Journal of the Scholarship of Teaching and Learning, 13(2), 133-147. http://josotl.indiana.edu/article/view/3281/3397

Dole, S., & Bloom, L. (2009). Online course design: A case study. International Journal for the Scholarship of Teaching and Learning, 3(1), Article 11. http://digitalcommons.georgiasouthern.edu/ij-sotl/vol3/iss1/11

While interviews are typically conducted face-to-face, they can also be conducted by e-mail, as did Gordon and colleagues.

Gordon, S., Petocz, P, & and Reid, A. (2007). Teachers’ conceptions of teaching service statistics courses, International Journal for the Scholarship of Teaching and Learning, 1(1), Article 10. http://digitalcommons.georgiasouthern.edu/ij-sotl/vol1/iss1/10

Two Excellent Qualitative Research Books

Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. San Francisco: Jossey-Bass.

Tracy, S. J. (2013). Qualitative research methods: Collecting evidence, crafting analysis, communicating impact. Hoboken, NJ: Wiley-Blackwell

Three Excellent Qualitative Interviewing Books

Rubin, H. J., & Rubin, I. S. (1995). Qualitative interviewing: The art of hearing data. Thousand Oaks, CA: Sage (Now 3rd ed., 2012).

Seidman, I. (1998). Interviewing as qualitative research: A guide for researchers in education and the social sciences (2nd ed). New York: Teachers College Press.

Weiss, R. S. (1994). Learning from strangers: The art and method of qualitative interview studies. New York : Free Press. Mallin, Mack@FACET, 2014

Mallin, Mack @ FACET, 2014

Participant Observation

McKinney (2007) explains that the Scholarship of Teaching and Learning is an inquiry process investigating teaching in your own classroom. Therefore, if you are teaching then you are a participant within your own study. So learning the qualitative processes behind the ethnography approach of participant observation would be beneficial.

Key points to consider:
An ethnographic approach requires you to:

Participant Observers can conceal their research function or explain their research role to participants within the research site (Davies, 2007). Explaining your research role is more commonly used within the scholarship of teaching and learning context. With this explanation, the teacher and students can learn from each other through the research process. With this distinction, you may take on the role of participant as observer or observer as participant (Lodico, Spaulding, & Voegtle, 2006; Ely, 1991). Your level of participation will dictate your role. For example, a teacher participating in the lesson while researching the lesson would be a participant as observer and a co-teacher observing his/her partner in the classroom may be classified as an observer as participant. The main task of a participant observer is to observe and take copious notes of what is seen and heard. Traditionally, notes are then shared with participants to make sure bias was not introduced during note-taking, such as an inference recorded instead of an observation. Organizing and classifying the notes to identify themes would be the next step, the analysis.

References/Resources

Davies, Martin Brett. (2007). Doing a successful research project: Using qualitative or quantitative methods. New York: Palgrave/Macmillan.

Ely, Margot. (1991). Doing qualitative research: Circles within circles. London; New York: The Falmer Press.

Krueger, R.A. & Casey, M.A. (2009). Focus groups: A practical guide for applied research. Los Angeles: Sage.

Lodico, M.G., Spaulding, D.T., & Voegtle, K.H. (2006) Methods in educational research: From theory to practice. San Francisco: Jossey-Bass

Lofland, John & Lofland, Lyn H. (2004). Analysing social situations: A guide to qualitative observation and analysis. London: Wadsworth

McKinney, Kathleen. (2007). Enhancing learning through the scholarship of teaching and learning: The challenges and joys of juggling. San Francisco, CA: Jossey-Bass.

Saam, Mack @ FACET, 2014

Quasi-Experimental

arrow strategy table

Experimental

Quasi-experimental

Strategies for Quasi-experimental:

Triangulation of methods:
Do several different ways and see if results are the same every time.
E.g. Two instructors in same semester. Each do A in one section and B in the other.
E.g. Pre- and post-test in two different sections this semester, with A in one section and B in the other.

SOURCES OF DATA (University)

Triangulate by using several different types of data:
Different sources (students, teachers, parents); different methods (e.g. observations, grades); and/or different times (before and after)

Existing data
(Easiest to gather)

Records

  • Student grades*
  • Attendance
  • Teaching evaluations
  • Retention rate
  • % of students in school-sponsored organizations
  • Standardized test results
  • Demographics

 

 
Archival data

  • Research literature*
  • Local policies, rules, and regulations about education
  • Syllabi and curriculum
  • Accreditation reports

Conventional sources
(Require selection or development of data-collection instruments. Still relatively easy to acquire.)

Behavioral data (best)

  • Teacher journal, or field notes*
  • Number of books read
  • Library use
  • Writing samples
  • Grades on classroom tests*
  • Variety of materials used
  • Teacher observations of student participation, interaction, etc.*
  • Outside observations of classroom$
  • Student journals
  • Minutes from meetings
  • Staff development (hours, types)
  • Audio or videotapes of classroom$

 

Perceptual data (attitudes, opinions)

  • Surveys
  • Simple interviews

 

Inventive sources
(Products or performances. May be  difficult to acquire and can be difficult to evaluate)

  • Exhibits$
  • Portfolios$
  • Expositions$
  • Videotapes$

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

* = excellent source of data
             = may be expensive in time or money to do well
(adapted from Calhoun, 1994)

Resources

Materials from a workshop on quasi-experimental design
http://www.ipr.northwestern.edu/workshops/past-workshops/quasi-experimental-design-and-analysis-in-education/2012/

Good overview of quasi-experimental design
http://en.wikipedia.org/wiki/Quasi-experiment

Research methods for dummies on quasi-experiments
http://psych.csufresno.edu/psy144/Content/Design/Nonexperimental/quasi.html

Another explanation, with diagrams
http://psychology.ucdavis.edu/faculty_sites/sommerb/sommerdemo/experiment/types.htm

A more detailed explanation
http://www.csulb.edu/~msaintg/ppa696/696quasi.htm

A nice but short overview of methods of educational research
http://www.ecs.org/html/educationissues/research/primer/appendixa.asp

Mettetal, Mack @ FACET, 2014

Survey research – a quick guide*clipboard

Choosing/writing questions/items:

Instructions: clear, consider participants in choosing how they respond, pilot the survey

Order of questions:

Types of questions:

Scales (two most common)

Likert

Within this specific course, how well do the following behaviors, thoughts, and feelings describe you? (1)
Staying up on the readings
1                        2                        3                        4                                    5           
Not characteristic of me at all                                                      Very characteristic of me

Semantic Differential

Describe the quality of communication in this conversation (2)
Relaxed             1            2            3            4            5            6            7            8            9        Strained

Resources:

General guides for survey research and design:

http://writing.colostate.edu/guides/guide.cfm

http://www.terry.uga.edu/~rgrover/chapter_5.pdf

https://www.ischool.utexas.edu/~palmquis/courses/survey.html

Scale construction and reliability resources

http://www.personal.kent.edu/~dfresco/CRM_Readings/Clark_and_Watson_1995.pdf

http://www.joe.org/joe/1999april/tt3.php

http://www.statisticshell.com/docs/reliability.pdf

*Source: Reinard, J.C. (2008). Introduction to Communication Research, 4th Edition. New York: McGraw-Hill.

(1) Dixson, M.D. (2010). Creating effective student engagement in online courses: What do students find engaging? Journal of Scholarship of Teaching and Learning, 10(2), pp. 1-13.

(2) Duck, S., Rutt, D.J., Hurst, M.H., & Strejc, H. (1991). Some evident truths about conversations in everyday relationships: All communications are not created equal. Human Communication Research, 18, 228-267.

Dixson, Mack @ FACET, 2014
**worditout.com/word-cloud/402751