ch3.tex

 \chapter{Methodology}
 Knowledge about how students conceptualize has a qualitative nature. For
 qualitative research, methodology varies, but has standard parts: design of the
 study, sources and their selection, data, the process of analysis, the interpretation,
 and the approach to validation. Sample selection is recorded and reported
 so that others may judge transferability to their own context. 
 The kinds of data in a qualitative study include interviews and documents.
 Interviews are
 the principle technique used by phenomenographical research\cite[p. 86]{merriam2009qualitative}.
 %Documents can also be used. 
 Normal conduct of teaching can also provide data that can
 be used, if in an anonymous, aggregate form. Both deductive and inductive
 analysis can be carried out on these data.
 The analysis produces a description of the situation under study.
 This description may include a narrative, often called a thick and rich description, and also specific attributes, such as categories of findings and relationships among these categories.
 
  
  \section{Design of the Study}
  
  Information learned in tutoring and lecturing undergraduates inspired the research questions.
   More specifically, questions asked by the students suggested that they were not learning enough about proof techniques to understand material that appeared later in the curriculum.
   So, it seemed useful to discover what their ideas were, about proofs. 
   We used Bloom's taxonomy of the cognitive domain\cite{bloom1956taxonomy} to subdivide the domain in which we hope to find student ideas.
   Correspondingly we created parts of the study:  recognition and comprehension were grouped together into one part, application was a part of the study, and the third part included analysis, synthesis, and evaluation.
   
   We chose a qualitative approach because
      we seek to be able to describe the nature of the various
     understandings achieved by the students, rather than the relative frequency
     with which any particular understanding is obtained.
  We chose a phenomenographic approach because it is aimed at identifying and expressing student understandings in a way that transforms these understandings into suggestions how to help them advance their learning.
 
    We collected data about recognition and comprehension in interviews, and in group help sessions and during tutoring, and also with a written list of questions, and by incorporating observations of computer science classes.
     We collected data about application on homeworks, and on practice and actual examinations.
     We collected data about analysis in interviews.
     We collected data about synthesis on homeworks, and on practice and actual examinations.
   
  
  %The study was designed to observe undergraduate students as they progressed through the curriculum.
  %Changes in conceptualizations of students as they progressed through the curriculum would be interesting if we could detect them.
  
  %Consistent with a phenomenographic study, the principle data were interview transcripts.
  
  We 
  conducted over 30 interviews.
  The conceptualizations of undergraduate computing students were sought.
  We incorporated into our design, a method of validation, called triangulation.
  To check our findings, we also interviewed faculty and graduate students who had provided courses related to proof.
  Our interview participants were sampled from  a large public research-oriented university in the northeastern United States. 
  
  
  %this paragraph below is good.
  Consistent with a grounded theory approach, we used  interviews conducted early in the study to explore students' notions of proof, adapting to the
  student preference for proof by mathematical induction and incorporating the use of recursive
  algorithms.
  
 
    We used  interviews conducted later in the study to investigate questions that developed from analysis of earlier interviews.
    
    
  We used exams to study errors in application of the pumping lemma for regular
    languages.

   We used homework 
    to observe student attempts at proofs, and
   to observe student familiarity/facility
  with different (specific) proof techniques: induction, construction, contradiction,
  and what students think it takes to make an argument valid.
  We used yet later interviews to discover whether students used proof techniques on their own, and how students ascertained whether circumstances were appropriate for the application of algorithms they knew, and how students ascertained certain properties of algorithms.
  
 
 %\chapter{Design of the Study}
 %This work is a qualitative study, the underlying philosophy is constructivist,
 %the research perspective is phenomenography, as extended to variation theory and structural relevance,
 %and the epistemological framework is 
 %social constructivism, in particular, that mental preparation is influential, readying students to learn from instructors and peer interaction, some material better than other material.
 
 %a layered collection of intellectual disciplines,
 %including
% complexity applied to cognitive neuroscience, and neurophysiology.
 %At the highest level of integration, computer science and mathematics
% reside, supported by studies in memory and attention, including computational
 %complexity applied to cognitive neuroscience, and neurophysiology.

 
 % the focus is on determining what questions
 %would be posed, in the process of continuous curriculum adaptation and improvement
 %the meaning students are making of their specific educational experiences.
 
  \section{Parts of the Study}
  
  The parts of the study reflect the several research questions.
  The parts of the study are organized taking inspiration from Bloom's Taxonomy of Cognitive Domain.\cite{bloom1956taxonomy}
  The first part of the study was about recognition: what undergraduates think proof is, and comprehension: how they go about understanding them, and about structural relevance (part of phenomenography\cite{marton1997learning}): why students think proof is taught.
  The second part was about application: How students attempt to apply proof.
  The third part was about analysis, synthesis and evaluation: what students do when a situation might be well-addressed by proof. 
  This part informed us about any structure students used as they pursued proof-related activities, and about what students thought was required for an attempted proof to be valid.
This part also informed us about the comfort level students have about the use of proof, and the consequences students experience, as a result of their choices about application of proof.
  
  These parts are summarized in Table \ref{parts}.
  
  \begin{table}
  \caption{Parts of the Study}
  \begin{tabular}{|p{4cm}|p{10cm}|}\hline
  Part & Purposes \\\hline \hline
  recognition,  & what proof is\\
   comprehension and& how students approach understanding proofs\\
  structural relevance & what proof is for\\\hline
  application & how students apply proofs they have been taught\\ \hline
  analysis, & use of structure\\
  & how is validity attained\\
  synthesis and  & comfort level with proof\\
   & student use of proof\\
  & consequences of not applying proof\\
  evaluation &\\ \hline  
  \end{tabular}
  \label{parts}
  \end{table}
  
 
 
 \section{Population Studied} \label{pop}
 The population of interest is undergraduate students in the computing disciplines.
 In a phenomenographic study, it is desirable to sample widely to obtain as broad as possible a view of the multiple ways of experiencing a phenomenon within the population of interest, according to Marton,\cite{marton1997learning}, who in turn cites Glaser and Strauss 1967\cite{glaser1968discovery}. We studied undergraduate students who have taken computer science courses involving proof. Typically but not always, these are students majoring in computer science. Some of these undergraduate students are dual majors, in computer science and mathematics.  We interviewed graduate students emphasizing those who have been teaching assistants for courses involving proofs. We interviewed faculty who have taught courses involving proof. We have interviewed former students who have graduated from the department. We have interviewed undergraduates who transferred out of the department.
 
 The demographics of the interviewed students is somewhat representative of the demographics of the Computer Science \& Engineering  department.
 The percentage (This has to be checked.) of participants who are persons of from visible minorities exceeds the percentage enrolled in the department. The same overrepresentation is true for persons identifying as female, for Asian persons and for persons documented by the Center for Students with Disabilities.
 
 Every student who signed a consent form was requested to schedule an interview, from an interval including 8AM to 9PM.
  Every student who scheduled an interview was interviewed.
 
  For the benefit of readers wondering to what extent the results might be transferable, demographics of some commonly seen properties of populations are provided: \label{demog}
  
  Number of interviews transcribed:\\
  Number of interview subjects:\\
  Percentage by race:\\
  Percentage by sex:\\
  Percentage by age:\\
  Percentage by language:\\
  Percentage by domestic/international:\\
 
% \section{Chronology of the Design}
% The design of this study began while teaching Introduction to the Theory of
% Computing. While helping students learn the pumping lemma for regular
% languages, and trying to understand from where the several difficulties arose,
% I became curious about the bases of these difficulties. One example was that
% a student felt strongly that a variable, a letter, denoting repetitions in a mathematical
% formulation, could only stand for a single numeric value, rather than a
% domain. Subsequently I have learned that symbolization is a category identified
% by Harel and Sowder \cite{harel1998students}, for students of mathematics learning proofs. Our
% student is a vignette of our computer science student population harboring
% some of the same conceptualizations. As a consequence of this opinion, the
% student felt that showing that a mathematical formulation had a true value
% was equivalent to demonstrating a true value for a single example, rather than
% demonstrating a true value for a domain. Here we see evidence for the category
% Harel and Sowder  \cite{harel1998students} call (is it inductive, perceptual?) where an example
% is thought to provide proof of a universal statement. Later, while helping students
% study the relationship between context free grammars and pushdown
% automata, I learned from the students that many of them did not find inductive
% proofs convincing. Subsequently I have learned that Harel and Sowder  \cite{harel1998students} created
% a category called axiomatic reasoning. In axiomatic reasoning, students
% begin with accepted information, such as axioms and premises, and apply rules
% of inference to deduce the desired goal. This category had not always been
% reached by their students, similarly to ours. As will be seen, later interview data showed,
% some of our students learn to produce the artifact of a proof by mathematical
% induction by procedure. They learn the parts, and they supply the parts
% when asked, but are not themselves convinced. (McGowan and Tall report a similar situation.) This matches with two other
% categories created by Harel and Sowder \cite{harel1998students}, internalization and interiorization.
% Still later, when leading a course on ethical reasoning for issues related to computer
% science, I found that most of the students did not notice that methods
% of valid deductive argumentation were tools that they might apply to defend
% their opinions.
% Thus the idea of exploring the nature of the students' degrees of preparation
% for understanding and creating proofs appeared.
% First, interviews about proofs in general were conducted, with a broad interview
% script. 
 
 %The students almost all selected  proofs by mathematical induction.
 
 
% During analysis of these data, a more elaborate interview script was developed,
% aiming at the ideas of domain, range, relation, mapping, function, the ideas of
% variable, as in programs and mathematical formulations, and abstraction.
% Some students emphasized that mathematical definitions are analogous to
% definitions in natural languages, and that mathematical discourse is carried
% out in the mathematical language created by these definitions.
% The capabilities for expression and care bestowed by these definitions invest
% mathematical reasoning with its persuasive power.
% Thus both the reasoning processes, using concepts and the clearly defined
% mathematical concepts together provide the ability of mathematical argumentation
% to be convincing. Students who appreciated this found it invigorating.
% Other students had different reactions to definitions. Thus, the role of definitions
% and language became another area of exploration.
% The difference between a domain and a single point in a domain can be seen as
% a level of abstraction. If something is true for a single point in a domain, but is
% also true for every single point in the domain, then the point can be seen as a
% generic particular point, representative of the domain. This concept of ability
% to represent is related to the idea of abstraction.
% We saw data in this study that affirmed the observations of others, that students
% do not always easily recognize the possibility of abstraction.
 


 
% \section{Sample Selection}
 
 All participants were volunteers.
 Volunteers were sought in all computer science undergraduate classes involving proofs, and also some that did not involve proofs, so that we could sample  students at different stages in their undergraduate careers.
 
 Graduate student volunteers were also sought. Most of the graduate student interviews were among teaching assistants in courses that taught and/or used proofs. We also included faculty of courses that involved proofs. Graduate student and faculty provided another perspective that was used as triangulation, a validation method in qualitative research.
 
% Students from the University of Connecticut who have taken or are taking the  relevant courses were offered the opportunity to be interviewed.
  The undergraduate students
 who volunteered were mostly male, mostly traditionally aged students. Some students were domestic, and some international. 
 Some students were African-American,
 some Asian, some Caucasian, some Latino/a, some with learning disabilities such as being diagnosed as on the autistic spectrum.
 
 
 \subsection{Proofs by Mathematic Induction}
 This part of the study contributes to recognition and comprehension of  proof, and also to synthesis, yielding insight into consequences of student use (or not) of proof when the situation warrants.
 
 The participants for the study of proof by mathematic induction 
 %We studied students who 
 were taking, or
 % who
  had recently taken, a course
 on Discrete Systems required of all computer science, and computer science and
 engineering students.
 Volunteers were solicited from all students attending the Discrete Systems
 courses.
 Interviews of eleven students were transcribed for this study. Participants
 included 2 women and 9 men. Two were international students, a third was a
 recent immigrant.
 %Every student who signed a consent form was requested to schedule an interview, from an interval including 8AM to 9PM.
 %Every student who scheduled an interview was interviewed.
 

 

 
 \subsection{Purpose of Proof}
 This part of the study contributes to the first part; it is about structural relevance.
 
 Undergraduate students were sought for this study, because we wanted to know what students thought the purpose was while they were taking the undergraduate subjects.
 
 \subsection{Proofs Using the Pumping Lemma for Regular Languages}
   This part of the study contributes the second part; it is about   how students apply proofs they have been taught.
  
  The participants for the study of proofs using the pumping lemma for regular languages were
  forty-two students, of whom thirty-four were men and eight women,
  forty-one traditional aged, 
  %one former Marine somewhat older, one collegiate athlete (a
  %woman),
   there were three students having Latin-heritage surnames, 1/4 of the
  students had Asian heritage, 2 had African heritage, and 8 were international
  students. Each student individually took the final exam. A choice among
  five questions was part of the final exam; one required applying the pumping
  lemma. Half the students (21/42) selected this problem. These were 17 men
  and 4 women. Three quarters of those (15/42) selecting the pumping lemma
  got it wrong. These students, who chose the pumping lemma problem and
  subsequently erred on it, form the population of our study.
  
 
  \subsection{Student Use of Proof for Applicability of Algorithms}
   This part of the study contributes the third part, about student use of proof.
  
  The students participating in this part were mainly those having internships or summer jobs. This changed the ratio of domestic to international students, such that a greater proportion were domestic students. Also, the ratio of women to men students was affected, such that a greater proportion were male students.
 
 \section{Data Collection} \label{divsrcs}
 Our corpus includes interview transcripts, homework, practice and real tests,
 and observations from individual tutoring sessions, and group help sessions. %Interview  transcripts were analyzed with thematic analysis.
 Homework, and practice
 and real tests, from several different classes were analyzed for proof attempts.
 (Incidentally, data from multiple instructors was combined, and no use of information about any specific instructor was used.)
 Data from individual tutoring  sessions and group help sessions were also informative. 
 Aggregations of anonymous data were used.
 Towards the end of data collection, the creation of new codes and categories became very slow. This expected behavior suggests an endpoint to data collection, and is called saturation.\label{satur}
 
 \subsection{Interviews}
 An application to the Institutional Review Board was approved, for the conduct of the interviews. The protocol numbers include H13-065, H14-112 and H15-022.
 
 %Consistent with phenomenographic studies, we wished to sample widely, so we sampled not only current students of courses involving proof, but also teaching assistants, faculty and former students associated with these courses. Some students are strictly CS/E majors, others are dual majors or minors in CS/E and math. Some students are not CS/E majors. Some former students are professionally employed in development, and others have left the major.
 
% All interview participants were volunteers.
 
The audio portion of all interviews was collected by electronic recorder and subsequently transferred to a password protected computer. From here the interviews were transcribed, and names were redacted. 


 \subsubsection{Student Conceptions of What Proof Is}
 Interviews were solicited in class by general announcement, and by email.
 Interviews were conducted in person, using a voice recorder. No further
 interview script, beyond these following few questions, was used. The interviews
 began with a general invitation to discuss students' experience with and
 thoughts on proofs from any time, such as high school, generally starting with
 
 \begin{itemize}
 \item ``Tell me anything that comes to your mind on the subject of using proofs,
 creating proofs, things like that.''
 \end{itemize}
 and then following up with appropriate questions to get the students to elaborate
 on their answers.
 Additional questions from the script that were used when appropriate included
 \begin{itemize}
 \item ``Why do you think proofs are included in the computer science curriculum?'',
 \item ``Do you like creating proofs?''
 \end{itemize}
 and, after proof by induction was discussed,
 \begin{itemize}
 \item “Do you see any relation between proof by induction and recursive algorithms?”
 \end{itemize}
 
 Almost every student introduced and described proof by mathematic induction as experienced
 in their current or recent class.
 

 
 \subsection{Documents}
  
  
  
  \subsubsection{Proofs Using the Pumping Lemma for Regular Languages}
  The study was carried out on both real and practice exam documents. The interpretation was informed
  by the events that occurred in the natural conduct of lectures,
  help sessions and tutoring.
  One method of assessing whether students understood the ease of application
  of the pumping lemma to a language to be proved not regular was offering a
  choice between using the Myhill-Nerode theorem with a strong hint or using
  the pumping lemma. The pumping lemma problem, which could very easily
  have been solved by application of the Myhill-Nerode theorem, especially with
  the supplied hint, was designed, when tackled with the pumping lemma, to
  require, for each possible segmentation, a different value of $i$ (the number of
  repetitions) that would create a string outside of the language. The intent was
  to separate students who understood the meaning of the equation's symbols,
  and the equation itself, from those students engaged in a manipulation with at
  most superficial understanding.
  

  \subsection{Observations from Tutoring and Help Sessions}
  We also compared the results from this with information obtained in tutoring and larger help sessions.
  These were noted down, at the conclusion of the help session or tutoring session,   for incorporation into manuscripts under preparation at the time.
  
  \subsection{Other Sources}
   We consulted faculty, who had experience with teaching this material, and who had experience with students who were supposed to have learned this material in prerequisites.
  
  \section{Method of Analysis}
  
  The phenomenographic approach to analysis has been written about by Marton and Booth\cite{marton1997learning}.
  This method works on interview and other data, and aims to produce a set of categories with relationships among them. Moreover, these categories and relations are used to infer critical aspects, which are ideas that are critical for developing to a more advanced conceptualization from a less advanced conceptualization.
  
  The process by which this transformation of data occurs has been further clarified by  Marton and Booth\cite[p.103]{marton1997learning}, who have written that an analyst should apply ``the principle of focusing on one aspect of the object and seeking its dimension of variation while holding other aspects frozen'' is helpful.
  
  One example of applying this principle is the analysis directed to the question of what students think about why proofs are taught in the curriculum. Using the terminology of Marton and Booth\cite{marton1997learning}, ``structural relevance'', we consider structural relevance to be an aspect of proof in the curriculum. Students should learn about proof for reasons that are connected with other material in the curriculum. For example, proof by mathematic induction is relevant for understanding the explanation of why context free grammars generate the languages accepted by non-deterministic pushdown automata. We focus on the idea of the students' conceptions of why proof is taught. We look for a dimension of variation: some of the students' ideas about why proof is taught will contain more of the reason underlying the presence of proof in the curriculum. Using this single dimension we can sequence excerpts of student interview transcripts, student utterances, according to how little or much of this reason they recognize. This exercise is provided as an example in Table \ref{exemplar1}.
  
   % % % structural relevance
   
   Excerpts of student transcripts were selected on the basis of being related to this question. A dimension of variation emerged from the data, such that the excerpts seemed readily organized along this dimension.
   
   \begin{longtable}{|p{7cm}|p{8.5cm}|}\hline
   \caption{Phenomenographic Analysis of Reasons for Teaching Proof}\label{exemplar1}
   \endfirsthead
   
   \multicolumn{2}{c}%
   {{\bfseries \tablename\ \thetable{} -- continued from previous page}} \\
   \hline \multicolumn{1}{|c|}{\textbf{Category}} &
   \multicolumn{1}{c|}{\textbf{Representative}}  \\ \hline 
   \endhead
   
   \hline \multicolumn{2}{|r|}{{Continued on next page}} \\ \hline
   \endfoot
   
   \hline \hline
   \endlastfoot
   
  % \begin{tabular}{|p{7cm}|p{8.5cm}|}
  \hline
   Category & Representative\\\hline\hline
   
    
    some students do not see any point to proof&
   They teach it to us because they were mathematicians and they like it.\\
   
   
   & we didn't see ok why do i really have to know the proof of the theorem to do that right? We didn't see the point, because no one taught us the point, so, that's a very important part that was missing.\\\hline
  
    some students think that it satisfies the curriculum goals, to be able to reproduce a previously taught proof, or follow a procedure to generate a proof, without being personally convinced&
   I was able to get a full score, but I don't understand why a proof by induction is convincing\\\hline
   
   Some students do not see a relationship between a problem and approach&
   When I have to prove anything, I always start with proof by mathematic induction, that was the one they taught the most.\\\hline
   
    Some students are surprised to discover that there is a relation between proof by induction and recursion&
   I never noticed that before, but now that you mention it, I see that they are isomorphic.\\\hline
   
    Some students see the relationship but do not use it&
    Professor (redacted) would be really proud of me that I learned to understand proof by induction quite well. \ldots I understand how recursion matches induction, there's a base case, there's a way of proceeding. \ldots I just couldn't figure out how to program the merge-sort algorithm.\\\hline
   
    some students do not generalize reasons for studying proof  beyond what they are shown in class &
    I would never consider writing a proof except on an assignment.\\
    
    & I understand the proof of the lower bound on comparison sort. \ldots I understand the proof of the upper bound on searching in a binary search tree. \ldots If I had to prove something about termination on a search tree, I don't know how I would do that.\\
    
   
   & I know that recursion has the same structure as proof by mathematical induction. \ldots If I had an algorithm with a recursive data structure like a tree, and I had to prove something like termination about it, I'm not sure what approach I would use, it would depend.\\\hline
   
   Some students see that they could employ proof to explore whether an algorithm can be expected to solve a problem in a given context that includes bounds upon resources that are available for consumption. & mostly design the algorithm first, we had some expectation of what that complexity results would be and then we try to find an approach to prove.
  %\end{tabular}
   \end{longtable}
   
   
  Marton and Booth\cite[p. 133]{marton1997learning} note that the phenomenographic method of analysis includes viewing excerpts of student utterances in specific perspectives. They advise ``establish a perspective with boundaries, within which one seeks variation'', and to remember to apply perspectives ``that pertaining to the individual and that pertaining to the collective''. So, when we establish a perspective with boundaries, we set a scope, allowing us to admit student text fragments relevant to that scope, filtering out other remarks. When we sequence or categorize the selected utterances, during which we will be comparing data from difference individuals, we must evaluate the utterances within the context of the interview from which they were obtained. For example, one student might be more prone to exaggeration than another. Also, one student may have more mathematical background than another.
  
   Marton and Booth regard the learning objective as a collection of related aspects, with their relationships; we can observe that a component hierarchy can represent the aspects. Marton and Booth discuss the depth of understanding; we can observe that one consequence of depth of understanding is the development of a generalization/specialization hierarchy. Marton and Booth contrast situations with phenomena, such that phenomena are understandings and situations serve as relatively concrete examples of phenomena, as used in instruction and assessment.
   
   We may search for evidence of recognition of aspects; they might be mentioned by learners. Marton and Booth have observed that in different context, different aspects shift between foreground (consciousness) and background. 
    Marton and Booth advise us to ``assume that what people say is logical from their point of view''\cite[p. 134]{marton1997learning}, citing Smedlund\cite{smedslund1970circular}.
   
    
    
    
     Marton and Booth \cite[p. 133]{marton1997learning} write that completion may be recognized by the achievement of a result, specifically the ability to identify a number of qualitatively different ways in which phenomenon has been experienced.
     
     One approach we have taken, besides the single aspect oriented approach exemplified in Table \ref{exemplar1}, is to apply  basic inductive analysis and deductive qualitative analysis, including axial coding (described later), with the phenomenographic paradigm in mind.
     
     More specifically, when processing interview data, we transcribe the data, we transfer the transcribed and redacted\footnote{Names of people are removed.} data to a website-based tool named Saturate\footnote{ saturate's url}. %We have used both the current Saturate application and the previous version. The previous version has features we prefer to the more recent version.
     
     We use the Saturate application to select contiguous fragments of text the capture meaning in our judgment. Each selected fragment is labeled. These labels, which are also called codes, can be reused, thus collecting together multiple fragments, as synonymous. A process, called constant comparison, \label{constcom} begins at this level of aggregation of the data. A code, representing the synonymous fragments, is chosen, either from among the fragments or not. The fragments sharing a code are compared with one another, to ascertain whether the group with that code is internally cohesive, to such a degree that fragments in any one group are relatively distinct from fragments in other groups. A summary description (called a memo)\label{memo} of each code is written, and fragments are checked for compatibility with the code's description. 
     
      Data were analyzed using a modified version of thematic analysis, which is
 in turn a form of basic inductive analysis.\cite{Merriam2002,Merriam2009,braun2006using,fereday2008demonstrating,boyatzis1998transforming} Using thematic analysis, we
 read texts, including transcripts, looked for ``units of meaning'', and extracted
 these phrases. Deductive categorization began with defined categories, and
 sorted data into them. Inductive categorization ``inferred'' the categories, learned them, in
 the sense of machine learning, which is to say, the categories were determined
 from the data, as features and relationships found among the data suggested
  more and less closely related elements of the data. A check on the development
  of categories compared the categories with the collection of units of meaning.
  Each category was named by either an actual unit of meaning (obtained during
 open coding\footnote{Open coding is so-called because it occurs at a time when the analyst is the most open-minded about what the meanings being found in the data might be. \cite[p. 178]{merriam2009qualitative}}) or a synonym (developed to capture the essence of the category).
 A memo was written to capture the summary meaning of the category.
          
     
     
     Then, with a set of codes, we again perform grouping. This time we group codes into categories. Each category is reviewed to check whether the codes contained are relatively cohesive within a category, and relatively distinct from codes in other categories. A memo is written for each category.
     
     Categories at this point in the analysis are also called initial themes. %Themes are used in the process of axial coding. The word axial refers to the hub and spoke placement of the categories, as, one at a time, each category takes its place as a hub in a diagram, with one spoke for each of the other categories, that are ranged around the hub in a circle. Each spoke is labeled with a pairwise relationship between categories. After the relationships have been inferred by considering each pair of categories, the relationships are themselves compared. Participation in multiple strong relationships distinguishes a category, promoting it to a (what's the adjective, something like principle) theme. 
      Next a process called axial coding, found in the literature on grounded theory,
                \cite{strauss1990basics,kendall1999axial,glaser2008conceptualization} was applied. This process considered each category in turn as a central
                hub; attention focused on pairwise relations between that central category
                with each of the others. The strength and character of the posited relationship
                between each pair of categories was assessed. On the basis of the relationships
                characterized in this exercise, the categories with the strongest interesting relationships
                were promoted to main themes.
     
     Attending to the phenomenographic paradigm, we seek dimensions of variation. These are delineated by the appearance of multiple categories that are usually related to each other by including more aspects (component parts) of an idea that is a learning objective.
     The basic inductive analysis with axial coding described above should make more evident relationships in the data that are of the nature of a dimension of variation. Thus we see that basic inductive analysis with axial coding is compatible with phenomengraphic analysis, in that it can be directed towards achieving the goals of phenomenographic analysis insofar as one or more dimensions of variation can emerge.
     
     Phenomenographic analysis proceeds beyond the identification of categories and relationships to infer critical aspects. These are differences between related categories, such that discernment by students of the ideas differentiating those two related categories are thought necessary for the students' depth of understanding to develop into the more inclusive category. 
     
     A diagram showing the main
      themes and their relationships, qualified by the other, subsidiary themes and
      the relationships between the subsidiary and main themes was prepared to
      present the findings. Using the process of constant comparison, the structure
      of these relationships was reviewed in the light of the meanings of the categories.
      A memo was written about each relationship in the diagram, referring
      to the meaning of the categories and declaring the meaning of the relationship.
      A narrative was written to capture the content of the diagram. Using the
      process of constant comparison, the narrative was reviewed to see whether it
      captured the sense of the diagram. Units of meaning were compared with the
      narrative and their original context, to see whether the narrative seemed to
      capture the meaning. The products of the analysis were the narrative and the
      diagram.
      
     The diagram and the memos are used to write the report, the description of context in which, should be detailed enough (thick and rich enough) that a reader can decide whether results obtained in that context are applicable to the reader's own context.
     
     Member checking \label{mchk}(i.e., asking for feedback on the report, from the population the report is allegedly about) of the summary report is used to estimate validity.
      
     
      \subsection{Analysis of Interviews}
      
      The redacted interview   data was analyzed and managed  within Saturate\cite{sillitoapp}.
      Though a newer version than the one we use exists, the ease with which individual codes can be assigned, and collected into categories, in the former version, won our preference.
      The support for memos at multiple levels in Saturate is very useful.
      Not only do memos document the generalization, by holding the (inferred) definition of the category, they also support comparison between that definition and the member codes of the category, but there is yet more. Memos capture the audit trail, a means of supporting a claim of validity according to Merriam and Richards \cite{merriam2009qualitative,richards2014handling}.
      We wove together contemporaneously collected interview data on the several research questions, and considered the relevance of material primarily related to one research question, for other research questions.
      
      We used constant comparison, that is, we looked to higher levels of generalization, as in the process of creating a code from meanings and checked a category from codes, and checking whether specializations of these generalizations were consistent with our data and our general sense of the students' conceptualizations.
      Constant comparison helped build our sense of the plausibility of our interpretation.
      It is a means by which we derive validity from the consistency of our interpretation with our data (see\cite{merriam2009qualitative}).
      
      The analysis took multiple perspectives: data from the students' points of view generated some speculative categorizations, and data from the instructors' points of view was compared with these speculations. Related literature was examined, and it also provided illumination and cross-checking\label{xchk} of our inductions. The component model (described by Marton and Booth \cite{marton1997learning}) of the learning objectives, the lesson to be taught, was built up from student conceptualizations. 
      This inductive approach was beneficial because, when we subsequently apply a deductive approach based upon what textbooks attempt to convey for the same lesson, the parts of the intended lesson, not found in student conceptualizations, are more obvious.
      Had we not bracketed off the deductive perspective, we might have been searching for, and extracted from random, insignificant words, material for populating those deductively obtained categories.
      
      
       
      
      Items excerpted from interviews for analysis should be analyzed in the context of the specific interview and also in the context of the ensemble.\cite{marton1997learning}.
      
      Data were analyzed multiple ways. Both an orthodox phenomenographic analysis, and a modified thematic analysis were carried out.
      
      \subsubsection{Traditional Phenomenographic Analysis}
      
      In the traditional phenomenographic analysis of interviews, the transcriptions are printed, and text fragments corresponding to units of meaning are cut out (as, with scissors). These pieces are then grouped (making copies if necessary) according to a sense of similarity. During a stage in the process, categories are learned, as researchers sense of features that distinguish categories evolves. During this stage, text fragments are moved from one category to another. After this category development phase, researchers, look into each category, to recognize and describe each category. Subsequently the perspective is shifted so that relations between categories are sought. Thus the categories are arranged relative to one another, and pairwise relations, where they exist, are identified and described. This produces a graph. From the graph, critical features of the learning objective are inferred.
      
      \subsubsection{Modified Thematic Analysis}
       
       Data were analyzed using a modified version of thematic analysis, which is
       in turn a form of basic inductive analysis.\cite{Merriam2002,Merriam2009,braun2006using,fereday2008demonstrating,boyatzis1998transforming} Using thematic analysis, we
       read texts, including transcripts, looked for “units of meaning”, and extracted
       these phrases. Deductive categorization began with defined categories, and
       sorted data into them. Inductive categorization “learned” the categories, in
       the sense of machine learning, which is to say, the categories were determined
       from the data, as features and relationships found among the data suggested
       more and less closely related elements of the data. A check on the development
       of categories compared the categories with the collection of units of meaning.
       Each category was named by either an actual unit of meaning (obtained during
       open coding) or a synonym (developed to capture the essence of the category).
       A memo was written to capture the summary meaning of the category.
       Next we performed axial coding.
       %a process called axial coding, found in the literature on grounded theory,
       %\cite{strauss1990basics,kendall1999axial,glaser2008conceptualization} was applied. This process considered each category in turn as a central
       %hub; attention focused on pairwise relations between that central category
       %with each of the others. The strength and character of the posited relationship
       %between each pair of categories was assessed. On the basis of the relationships
       %characterized in this exercise, the categories with the strongest interesting relationships
       %were promoted to main themes.
        A diagram showing the main
       themes and their relationships, qualified by the other, subsidiary themes and
       the relationships between the subsidiary and main themes was prepared to
       present the findings. Using the process of constant comparison, the structure
       of these relationships was reviewed in the light of the meanings of the categories.
       A memo was written about each relationship in the diagram, referring
       to the meaning of the categories and declaring the meaning of the relationship.
       A narrative was written to capture the content of the diagram. Using the
       process of constant comparison, the narrative was reviewed to see whether it
       captured the sense of the diagram. Units of meaning were compared with the
       narrative and their original context, to see whether the narrative seemed to
       capture the meaning. The products of the analysis were the narrative and the
       diagram.
       
    
       
       \subsection{Analysis of Help Session and Tutoring}
       
       Help sessions for Introduction to the Theory of Computation were scheduled weekly; attendance was optional. Typically six to twelve students would participate.
       Originally these were called help sessions, but the demographics of the attendees did not represent the enrolled students.
       Subsequently the name was changed to consultation sessions.
       This change had the desired effect, that the population attending better reflected the enrolled students.
       At these sessions, students would raise topics about which they had questions.
       Frequently the student would be requested to work at the white board, and leading questions were asked, and problems of very small size were posed, to urge the student along the right path of development of a solution.
       Occasionally these suggested paths were met with resistance from the students, which is to say, misunderstandings were encountered and discussed.
       Ideas mentioned in these discussions that were relevant to manuscripts in process at the time were noted, anonymized,  into the manuscripts.
       
      
       
       Due to attention being focused on interacting with students in the normal course of teaching, these field notes are incomplete. 
       
       One use of such data is that they can give evidence that categories of conceptualization of proof already created in the mathematics literature can be found also in computer science students. This is similar to a deductive rather than inductive process, in that we are aware of the categories created by Harel and Sowder\cite{harel1998students} and by Tall\cite{tall2012cognitive} and student utterances that seem well matched to those categories draw our attention to those categories, validating them for students of computer science.
       
       Help session data were more complete in a sense than interview data, because help sessions normally involved a successful improvement in experience that student had of the meaning of a concept.
       This in turn offered validation of the newly discerned concept as a critical factor, at least for that student.
       
       Taking the researcher only activity, dealing with transcripts, conjecturing categories, and inferring critical factors as a base for comparison, analysis of help session data analysis involves conjecturing (based on dynamic student utterances) an experience of meaning, inquiring to discover about partial and/or superficial knowledge, seeking incorrect ideas, forming a conjecture about a critical factor and posing a question to the student that calls upon, calling attention to, that missing resource. Encouraged into taking a helpful perspective, students often become aware of a gap in their knowledge, and are then ready to learn. This, when it occurs, confirms the utility of the proposed critical factor. Thus the main differences between analysis of interview transcripts and help session data is the number of participants, and availability of rapid feedback about utility.
       
      %  The study proceeded using prior interview experiences to suggest further investigation.
       %  Originally asking about proofs and what they were for, we received answers about proof by induction and found out not all students contemplate why the curriculum contains what it does.
         % (reference Guzdial on students trusting that whatever curriculum they take, is very likely to qualify them for a job).
         %When providing leading questions about why, interview data indicated that not all students apply proof techniques with which they have been successful.
         %So, when application of proofs developed as a part of the study.
         %Consequently, what clues there may be that prompt recall of proof and proof techniques to application to a problem became part of the study, involving structural relevance.
         %Generalization is related to the presence of one situation evolving a response that was learned in the context of a different situation prompted the view of teacher teaching from context of generalization hierarchy present and situation as example and homework situation as another example.
         %What opportunities to foster generalization do students notice?
         
        % \subsection{Within What is a Proof?}
         %The study was devoted to proofs, a subject that can be subdivided.
         %Part of the study was aimed at the idea of domain, directed at the concept that
         %though a variable could identify a scalar, it might also represent a set.
         %Part of the study was aimed at the activity of abstraction, because some students
         %exhibited the ability to operate at one level of abstraction, not necessarily a
         %concrete level, yet the ability to traverse between that level of abstraction and
         %a concrete level seemed to be absent. Other students claimed to be able to
         %understand concrete examples with ease, but to encounter difficulty when
         %short variable names were used within the same logical argument.
         
         %\subsection{Order of Exploration}
         %The order of exploration was data driven, thus the material was sought sometimes
         %in reverse order of the curriculum, almost as if seeking bedrock by starting
         %at a surface, and working downwards.
         
         
       
        
     
      \subsection{Example: Application of Phenomenographic Analysis to What Students Think Proof is For}
      
      The analysis for the research question ``What do students think proof is for?'', which was approached as `Why do you think we teach proof?'' exemplifies a phenomenographic approach. One aspect of the phenomenon of proof is its utility. We set the scope of our perspective to be specific to usefulness. We selected student verbal productions related to the use of proof. We considered them in the context of their own interview, and we compared them to data from other interviews on the same theme.
      
      We applied phenomenographic analysis by focusing on the aspect of relevance of proof for learning computer science and practicing as a software developer. In this case we had already identified the dimension of variation to be the depth of understanding of why we teach proof. Thus we could select fragments of student utterances and rank them according to depth of understanding. We then presented them in a sequence by rank.
      
      \section{Method of Creating Summary Report (for Member Checking)}\label{conjectures}
      All participant data were considered.
      Using Marton and Booth's\cite{marton1997learning} component model of the learning objective, we take note of what components of the idea of proof are seen to be 
      missing in some of the conceptualizations of the students.
      Then we infer a conjecture about what student perspective might result in this
      conceptualization.
      Then we test the conjecture to see what results we might expect from that perspective.
      If the perspective predicts the student utterances we have heard, or the student
      behaviors we have seen (e.g., incorrect negations), that lends confidence to our conjecture, and text to our thick and rich report.
      
      
  \section{Method of Addressing Validation}
  
  Our methods of addressing validation are itemized in Chapter 6, which refer back to specific sections in this chapter.
  
  Triangulation is a technique for increasing the confidence that the results of analysis are reliable.
  
  
  \section{Method of Presentation of Results}
  
        
        The product of analysis in a phenomenographic study is a set of categories, and relationships \textit{among} them. These categories and relationships are often depicted in a graph. This product may be accompanied by a ``thick and rich'' narrative description of the categories and relationships. This narrative must be consistent with the individual text fragments, excerpts from transcriptions, field notes or documents obtained for the study. 
        
        Marton and Booth\cite[p. 135]{marton1997learning} state ``in the late stages of analysis, our researcher [has] a sharply structured object of research, with clearly related faces, rich in meaning. She is able to bring into focus now one aspect, now another; she is able to see how they fit together like pieces of a multidimensional jigsaw puzzle; she is able to turn it around and see it against the background of the different situations that it now transcends.''
        
        This tells us that the narrative should describe the categories of composition hierarchies found in the students' understandings. The faces or facets of the learning object have their importance and relationships as envisioned by the teacher. The students' conceptualizations may be less complete, contain superfluous items, and differ as to the relationships of the parts, especially by lacking profundity in understanding of relationships.