Chapter Three: Developing the Coding Scheme

and Selecting the Pool

The Importance of Theory:

No coding scheme can provide a reliable content analysis unless it is grounded in theory. James Potter and Devine Leviine-Donnerstein argue that “the challenge of designing a content analysis can only be adequately met if researchers begin by making decisions about the nature of the content they want to analyze and the role of theory in their study” (259).

Thus, the first step in developing a coding scheme for this project was to identify the content that I was seeking to code and discover if there was adequate theory to guide the search for that content. This step was accomplished in earlier chapters that outlined the theoretical direction this study was taking and which revealed that there is an adequate understanding of the discourse factors that constitute a successful writing conference. Because Laurel Johnson Black, Don Murray, Carolyn Walker and David Elias clearly identify the successful writing conference as one that is student and paper centered rather than teacher centered, my code needed to identify content that illuminated the focus of the conference as it progressed.

Potter and Levine-Donnerstein also emphasize that reliability can not be counted on unless the researchers and coders concentrate where possible on “latent pattern,” which describes the broad characteristics and commonly shared meaning of the “patterns embedded in the content to be analyzed” (260). Coding rules that seek latent pattern avoid the “subjective interpretations” which occur when coders are faced with the difficult and complex task of differentiating among the minute nuances of a micro level “projective” coding scheme. According to Potter and Levine-Donnerstein, a coding scheme that looks at the pattern of content characteristics is more reliable than a projective coding scheme that calls for a technical set of definitions that must be interpreted item by item. Coders looking for latent pattern are not as likely to be forced to “bring their own schema into play” as they are asked to make judgments. “There is an increase in the confidence with which we can construct rules that will guide coders in capturing the essence of that which is being coded,” they report (262). Theory is the basis for capturing this essence, they stress.

“Utterances” as a Unit of Content

Once the theory for this study was specified and understood, it became relatively easy to identify the latent patterns of content that Mikhail Bakhtin terms “utterances.” The process of creating the code and defining the utterances was illuminated by theory, which for the purpose of this study is best articulated by Johnson Black and by Don Murray, who links student-centered conversation with the role that the affective dimension—those subtle and often visual cues which keep a conversation going in a preferred direction—plays in determining the success of a conference. Murray defines the ideal conferences as one that is more reflective and less “deductive” in nature, with the deductive conference being one in which the student expects the teacher the teacher to fill his head with knowledge (Teacher 163), as opposed to the “engaged pedagogy” envisioned by bell hooks which “call[s] on students to be active participants, to link awareness with practice” (14).

Looking at Johnson Black and Murray, I determined there were two very broad categories of utterances in a typical conference: those that fell within Johnson Black’s affective dimension and those that were more deductive in nature. For Johnson black, the affective dimension is so critical because it opens the conversation to the personal, the student’s insights about her own ‘feelings” about the process. Talking about writing, without talking about feelings, “abstract[s] a set of skills and a string of words from what has been a personal process, a human connection,” she warns (123). Alongside the affective topics, then are Murray’s deductive topics, which may lend themselves to a more passive position on the part of the student because they are product oriented, primarily about writing issues. Johnson black calls these “discourse topics” and warns that teachers tend to love them because they are “safe” topics. She writes:

It is common for teachers to take a student’s affective topic and transform it into a discourse topic. We submerge the feelings in something safe, something more clearly about writing or reading or skills and move away from feelings. (132)

Students are lured by discourse topics too, notes bell hooks, because they provide safe ground without the risk of the conflict or tension that topics expressing feelings or which call for critical thinking engender. “The presence of tension—and at times even conflict—often meant that students did not enjoy my classes or love me, their professor, as I secretly wanted them to do,” hooks observes (42).

Affective topics and discourse topics are not binary categories, it should be noted, but sorting the utterances in this fashion seemed useful to the goal of determining difference between traditional conferences and conferences in cyberspace, and because Potter Levine-Donnerstein so closely link theory and reliability, it became essential to clearly articulate the goal of this project. The purpose of recording and creating transcripts was to determine if the affective dimension that Johnson Black and Murray identify as fundamental to the success of the one-to-one writers conference is transportable to a distance learning environment. Because cyberspace is lacking the visual cues—“body language” (Teacher 163)—that for Murray help create the affective dimension of a conference, then the coding scheme clearly needed to identify utterances that served specifically to keep conference power within the student’s control as well as utterances that reflect Murray’s conversational style (148) of conferencing. It also needed to differentiate between utterances that relate to feeling and critical thinking about the process and between those that are more product oriented.

Although earlier I speculated that there might be some unexpected and new forms of discourse between the two modes of conferencing that might create (or destroy) a new manifestation of the affective dimension, a cursory examination of conference transcripts revealed there were no unexpected forms of utterances. Therefore the coding scheme needed to reflect the more subtle differences among utterances, those that could be counted and sorted to measure the “success” of a conference and which could be compared between face-to-face conferences and chatroom conferences. To accomplish this, each category of utterance was broken down into four subcategories to reveal the issues identified by theory as key to a successful conference.

Why “Utterances”?

A basic element, then, of the coding scheme, was to categorize “utterances.” Mikhail Bakhtin defines this term as meaning a unit of speech that is determined by a change of speaking subjects (71). It is a particularly useful unit in the context of conferencing, because of what Bakhtin describes as its responsive nature:

Any utterance—from a short (single-word) rejoinder in everyday dialogue to the large novel or treatise—has, so to speak, an absolute beginning and an absolute end: its beginning is preceded by the utterances of others, and its end is followed by the responsive utterances of others (or, although it may be silent, others’ active responsive understanding, or finally, a responsive action based on this understanding). (71)

Thus the utterance can be used not only to mark a change of speakers, but also to identify a type of response. Theory shows that this responsive nature of dialog, identified by Bakhtin, can be used to judge the success of the writing conference by measuring how responsive the teacher is to the student and her paper.

Johnson Black notes that Bakhtin uses this term in a context larger than just the immediate dialog. “Any concrete utterance is a link in the chain of speech communication of a particular sphere,” writes Bakhtin (91), and hence they are filled with “echoes and reverberations” of preceding utterances from within that sphere. Johnson Black observes that teachers participate in a chain of utterances that becomes a fundamental aspect of a student’s own history of exposure to teachers. Anything a teacher says is heard by the student in the context of all the teacher utterances that have gone before. “That chain of utterances seems to have injured more people than I can count,” reports Johnson Black (54), alluding to the power differential that takes place within a conference. She explains, “It is this power, where even if I have no history with a student, she brings one with her and attaches it to me, that invades conferences.”

Power and the Successful Conference:

Quantifying this invasion of power takes on new urgency as one moves into the cyberconference, where the affective dimension is diminished and the outcome of a conference less clear. Because pedagogy is ultimately about power, and because conferences that put the student more in control of the dialog and of their own paper are identified by Johnson Black and Murray as the most effective, it seemed important to quantify the power differential as a means of objectively defining the success of these conferences. Thus it was clear the coding scheme needed to be able to indicate what Johnson Black calls “control of conference talk” (52).

Developing the Code:

In developing the coding scheme, the following were taken into account:

1. ability of the code to specify and quantify a variety of utterances in a conference;

2. ability of the code to describe the affective dimension;

3. ability of the code to determine control of conference talk;

4. ease of use for the coder;

5. reliability.

Quantifying a conference, of course, is no easy trick, but after conducting some 600 conferences over an 18 month period, I feel they break down into a combination of teacher initiated comments, student responses and student questions and comments that replicate themselves from conference to conference in Bakhtin’s “chain of utterances.” My coding scheme reflects, but does not rank or weight, what I’ve experienced to be the dominant component utterances of a conference. The numbering scheme does not indicate any type of order or occurrence or ranking of importance but rather serves as a means of identifying for sorting and counting purposes the types of utterances I routinely experience in a conference.

Number Codes:

However, there are some distinctions among the utterances that allowed for broad groupings. Johnson Black notes that affective topics—which are the subject of this study—are primarily about feelings, and discourse topics are deductive topics primarily about writing (124). Thus I have numbered and ordered the code to allow it to discover and distinguish between the two. Hence, code items 1 through 6 are designed to quantify the affective dimension, code items 7 through 10 reflect Don Murray’s deductive utterances, and code 11 speaks directly to power issues. Altogether I identified 11 types of utterances which I felt were typical to my conferences. Combined with the four letters codes discussed my coding scheme provides up to 44 separate codes, a number far fewer than the 146 codes used in a 1984 study by Sarah Warshauer Freedman who tracked the topics of conversation in conferences to determine differentials between low achieving and high achieving students and to examine gender and racial issues. Because my study is looking strictly at the differences between what Stuart Blythe describes as “scenes” (25), the number of codes did not seem as important as the coding scheme’s ability to distinguish among different styles of utterances. Blythe’s metaphor of scenes, from Burke’s Pentad, stresses “interpretation” of “actions,” not precise definitions, thus my code is more interpretative in nature—along the lines of Potter and Levine-Donnerstein’s search for latent pattern—rather than minutely descriptive.

The 11 utterances that I describe in chart 3.1 below break a conference down into the components necessary to identify and interpret the differences between a face-to-face conference and an online conference, with the four letter codes per each utterance serving to define the origins and directions of the utterances and to name their source of power.

Letter Codes:

Each code number identifying a type of utterance is followed by a code letter to indicate who is making the utterance and, equally as important, to indicate the “flow” of the control of the conference. Developed along parallel lines to facilitate ease of use, the code reveals this flow of control with code letters that show who made the utterance and describe whether it was an active or passive utterance.

Qualifying utterances as active or passive provides a success indicator of the conference, because passive students, those who do not fully encounter their own papers, are not actively participating in their own learning. As Muriel Harris puts it, “Students who sit passively in a conference are not likely to do a turnaround and actively engage in any substantive revision” (40). hooks, Fife and O’Neill also clearly articulate this need for active engagement by the student if learning is to occur.

Looking at the code horizontally, a and b codes are those of teacher utterances and c and d codes are those of student utterances. All a codes indicate utterances in which the teacher is controlling the dialog by demanding information from the student, by initiating and dominating a topic, or, in the case of code 6, by pushing the conversation along with verbal listening cues. Codes with the letter suffix b indicate a more passive stance in which the teacher is responding to the student. All c codes indicate the student has shown initiative and is actively engaged in the dialog, and d codes reflect passive responses by the student.

Additional Coding:

To further clarify the power issue, which in turn helps clarify the affective dimension and measures “success” of the conference by quantifying how much control students have over their own conference and hence over their own papers, code 11 specifically defines points where the momentum of the conference has been interrupted and forcibly changed to a new direction. Hence, code 11a indicates the teacher interrupted and regained control of the conversation, and 11c indicates the student changed the flow. The b and d codes in this case indicate the power shift went unchallenged.

Summary of the Code:

The coding scheme can be summarized as follows:

· Numbers indicate type of utterance.

· Letters indicate who made the utterance.

· Codes 1 through 6 refer specifically to the affective dimension.

· Codes 7 through 10 are deductive topics.

· Code 11 relates specifically to transfer of power.

· All a and b codes are teacher utterances.

· Add c and d codes are student utterances.

· All a and c codes are active in nature.

· All b and d codes are passive in nature.

· All a and c codes are soliciting information or initiating a topic.

· All b and d codes are responses.

· All a codes indicate the power balance is in favor of the teacher.

· All c codes indicate the power balance is in favor of the student.

Word Count:

Another important indicator of the power issue, of course, is simple word count, which is not part of the coding scheme but is discussed here because the coding scheme is designed to illuminate the distribution of the word count. Johnson Black concedes that a conference in which the teacher does most of the talking may provide the student with a lot of knowledge and give the teacher a certain sense of satisfaction, but she doubts the effectiveness of such a conference:

In real time conferencing, we may sense that this isn’t the close conversation we wanted, but at least we have an interested audience and a bit of give and a lot of take with a student. And so we often settle for that, hoping that at least someone learned something, and we move on. The written product may be better, ultimately, but whether the student is a better writer is debatable. (53)

Johnson Black notes that the issue here is not how much knowledge the student was exposed to or how the student feels about the conference. A student is likely, she notes, to be “impressed that the teacher took the time to talk with her and considered her paper so thoroughly” (53). Rather, the issue points to the student’s ability to utilize that knowledge effectively when it was delivered in such a one-way fashion. “How much will she remember when she has played so passive a role,” Johnson Black asks (53). Johnson Black devotes a section of the appendix of her book to words counts, revealing that in the 14 conferences she studied, teacher talk comprised between 59.6 and 97.6 percent of the conference totals. That means that in some cases students in her study did as little as 2.3 percent of the talking and never more than 40.2 percent. The average student participation given as a share of the total word count among her 14 conferences was 20.2 percent. We will return to this figure in a later chapter.

Word count, of course, was the simplest form of quantification of the conferences. Microsoft Word was used to record all transcripts. After determining a total word count, all teacher utterances were eliminated from the document and another count was made, providing a student percentage of total words.

Other considerations:

In addition to describing and qualifying the utterances, it was also important to ensure that the coding scheme leant itself easily to quantifiable analysis. Thus, the numbering and lettering scheme was designed to be sortable, findable and countable in Excel spreadsheets. An appropriate recording template was also developed in Excel.

In order to assess the reliability of the codes, a draft coding scheme was subjected to an initial run-through of a 279-line transcript from a face to face conference to ensure:

• that the majority of the lines could be coded;

• that the goal of the code—to illuminate power issues and measure the affective dimension—was reflected in the process;

• that the coding could proceed without undo subjective judgment.

With the above three issues in mind, the code was revised and then applied to a similarly lengthy online conference with a similar student. More revisions were made.

After the coding scheme was finalized, four new transcripts were selected for the final coding (see Design of the Study below).

A larger, more scientific project would have included more transcripts, more coders and neutral testing of the codes by two or more coders.

Graphic 3.1 below shows the coding scheme after initial testing.

code		a	b	c	d
	utterance topic	instructor…	instructor…	student…	student…
1	personal/ice-breaking	active	passive	active	passive
2	discussion about process	active	passive	active	passive
3	evaluation of conference	active	passive	active	passive
4	critical thinking about topic	active	passive	active	passive
5	encouraging comments	active	passive	active	passive
6	listening cue	providing		providing
7	discussion of writing issues	active	passive	active	passive
8	evaluative comments	active	passive	active	passive
9	corrective suggestions	active	passive	active	passive
10	procedural issues	active	passive	active	passive
11	control shift	active	passive	active	passive

Graphic 3.1 – coding scheme

Graphic 3.2 presents the results template, which is an Excel spreadsheet that can be counted and sorted. Each line of the transcript was numbered and coded. In the rare event that an utterance ended and a new one began on the same line, the new utterance was used to determine the line code.

line#	number code	letter code
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

Graphic 3.2 – Results Template

Discussion of the Initial Testing:

Roughly two to two and one-half hours were required for coding the initial transcript with the draft code. It quickly became apparent that there were several types of utterances not included in the draft code, including generic comments about procedure and classroom issues, and at first I was tempted to overlook them. However, utterances in which the teacher specifically called for questions and the student either responded or failed to respond seemed impossible to overlook because they clearly reflected the power issue. They were added to the end of the code and not included in the codes for the affective dimension because they measure deductive topics—mechanics of the paper.

It was also apparent that the horizontal codes—the letter codes—were more easily applied that the vertical codes. The active or passive positions of the student and teacher were always easily identifiable, whereas the nuances between some of the number codes were not as easily delineated.

At first this concerned me, but focusing on the goal of the project—to measure the affective dimension and to illuminate power issues—I realized I was not trying to actually dissect a conference into its component parts. Hence, as long as the codes clearly fell within the first half of the code (the affective dimension) or within the second half (deductive and power topics) the results were measurable with validity for this project.

With the above in mind, I compressed the draft code after some utterances did not appear in the initial round of testing and after I realized it was not necessary to make fine distinctions within the groupings of utterances. I was finally forced to add a code in the discourse grouping for procedural issues when I discovered that the chatroom conferences were often focused on procedure. Because a personal goal of this study has been to improve my own conferencing techniques, I also added a code in the affective section after discovering the relationship expressed by bell hooks between critical thinking and classroom pedagogy, hoping to discover just how much critical pedagogy occurred in my own conferences.

I also had to rethink the horizontal terminology of the code. Looking back on the conversational theory articulated by Fife, O’Neill and Straub, I realized after my initial testing that “inviting” a student into the dialog, and then allowing that student to run with the invitation, was not at all a passive response on the part of the student as my draft coding scheme, with its emphasis on descriptive terms such as “soliciting” (in the active column) and “responding” (in the passive column) seemed to imply. To reflect this new understanding of the theory, I had to allow the code to show where the student was in charge of the conversation, even if her response came as acceptance of my own initiative. This issue was particularly evident in the trial coding of an online student, who often simply responded to my invitation to dialogue with a simple “okay,” or “yes.” These one-word responses were clearly passive, whereas if a student took off with my invitation and re-entered Straub’s “chaos of revision,” as the test seemed to reveal was more often the case in a face-to-face conference, it was indeed an active position on the part of the student. With this new realization in mind, I ended up clarifying the horizontal terminology to simply reflect who made the utterance and whether the nature of the utterance was active or passive.

Design of the Study:

Determining the Pool

For the purposes of this study I tape-recorded 31 face-to-face conferences late in the semester. Other conferences were recorded earlier in the semester, but they were not studied because I felt my own conferencing mode matured as the semester progressed, as the students got more comfortable with conferencing, and most importantly, as both the student and I got more comfortable with the tape recorder—earlier I had discovered that initially the tape recorder seemed quite intrusive, but by the end of the semester neither I nor the students seemed aware of the recorder. Of those 31 conferences, 19 students gave explicit permission so that I could study the tapes and report on my findings.

From my online class, I conducted 14 conferences late in the semester. Eight of those students provided their consent for me to study the text transcripts that were automatically (and unobtrusively) recorded in our one-to-one chatroom conference sessions.

Selecting the Participants

In order to determine how many transcripts to study and what type of selection criteria should be used, I turned to other research in the field to see if there were models I could follow. Johnson Black admits, without apology, that her own research is somewhat informal. “Certainly it was a small study of a homogenous group,” she reports (9). “But like most teacher research, it grew from an immediate context,” she adds.

Johnson Black initially studied conferences with six of her own students, all of whom were traditional first-year composition students, and half of whom were male and the other half female. She made no other attempt to differentiate among the students beyond gender. She later expanded her research to include other teachers and other students, studying an additional 14 conferences, but she offered no methodology or justification for arriving at this number.

Jean Ketter and Jonelle Pool, in their recent study of the impact of high-stakes testing in the high school composition classroom, took a more scientific and rigorous approach. Of the 23 students available to them, they chose five students, “identified by the classroom teachers as representing a range of abilities in writing” (356). That figure represents 22 percent of their available pool. A similar percentage of my largest pool, that of my face-to-face students, would be four students. Including four students from my online classes would increase the total beyond Johnson Black’s initial study and would amount to 29.6 percent of my available pool.

Sondra Perl’s famous 1978 study examining the composing process of low-level college writers dealt with just five students (as did the Ketter and Pool study), so in comparison with existing studies, my sample of eight students—nearly 30 percent of my available pool—seems both adequate and generous (although certainly not as generous as Sarah Warshauer Freedman’s 1984 study which coded and analyzed 48 writing conferences).

Because of the nature of the course sequence I teach and the nature of the University, which is a commuter-campus drawing a wide range of non-traditional college students, I sought less to rely on perceived writing ability to determine my sample and more on the nature of the students themselves and their stance towards writing. From my face-to-face students I picked an older student who was a serious writer but who displayed a lot of concern with correctness, a traditional teenaged student who wanted to get a good grade but showed no real interest in writing and who felt out of place in college, a twenty-something returning student who considered herself a very poor writer, and a traditional teenaged student who considered herself an excellent writer and who liked to write.

No considerations of gender or race were made, for several reasons:

· these issues have been thoroughly covered in other studies (particularly by Freedman);

· I was not aware of the race of my online students;

· gender stereotypes did not match with my experiences in the classroom, particularly in the online setting;

· in trying to determine control factors, categorizing students in terms of their feelings about their writing and in terms of their level of self confidence as a student seemed more important to me than seeking some kind of equity based on social markers;

· this study does not attempt to draw any conclusions based on gender or race.

My selection criteria certainly doesn’t exhaust all the possibilities, but it is one that transferred easily to the online classroom: I was able to pick four students from my online class who roughly matched the criteria established for the face-to-face conference analysis. This ability to match students across the divide of the classroom setting seemed important so that the analysis could compare face-to-face conferencing with online conferencing in an equitable fashion.

Coding the Students

In the analysis that follows, coded labels were applied to the student subjects to provide:

· Anonymity to the student.

· An indication of the classroom setting of the conference.

o F2F means the conference was in a face-to-face setting

o OL means the conference was in an online chat room

· An indication of the type of student.

o student1 is an older, active student anxious about writing “correctness”;

o student2 is a young, passive student concerned only with grades;

o student3 is an older, passive student with low expectations;

o student4 is a an young, active student with high expectations.

For example, if a chart or references indicates F2Fstudent1, this label is referring to a face-to-face, older student who is actively concerned about writing issues. Hence, OLstudent1 would refer to his/her counterpart in my online class.

Limitations of This Study

Because of the deadlines and constrictions of a Master’s Thesis project, this study was limited in the end to the coding and analysis of four transcripts, two from the face to face pool and two from similar students in the online pool. As described above, students 2 and 3 were picked from each pool.

The following two chapters present the results of the coding, with Chapter Four providing discussion about the analysis of the coding and presenting tables quantifying the coded data. Chapter Five interprets the results.

---