\chapter{Validity and Reliability} strategies for trustworthy, valid, reliable what about generalizability? (e.g., to people with ASD) \section{Interviews} software programs that were used, to manage, organize data\\ analyze as we go\\ inductive and comparative\\ precisely how the analysis was done\\ thorough explanation of any strategies, such as discourse analysis \section{Documents} \section{Validity and Reliability in Proofs Using the Pumping Lemma for Regular Languages} Some support for the validity of the results comes from seeing several variations of each proposed error type. We found in our data multiple versions of unwarranted restrictions: choosing $x$ to be empty or choosing the length of $xy$ to be $p$, and others. We found in literature warnings against attempts to prove statements with universal qualifiers true by means of showing the existence of examples \cite{devlin2012mathematical,Franklin}. These warnings suggest these errors have occurred before. In textbooks \cite{epp2010discrete,rosen2003} we find single counterexamples for showing such a statement false, and the method of exhaustion for showing a finite universal statement to be true. Proof by contradiction for the purpose of showing such a statement true, i.e., that any particular tentative counterexample contained an inherent contradiction, is not itself universally accepted, due to not necessarily being constructive \cite[p. 2]{bridges2007did}. We also found in our data, several versions of misunderstanding inequalities. We found support in literature for errors of misunderstanding how to work with inequalities, by students of this level\cite{Mattuck}. \section{Proof by Induction} We were encouraged by the overlap in description among interview participants. The interviews were certainly not the same, but common elements, specifically that there is a form to proofs by induction, appeared. Some students referred to this form as steps, others as a procedure, or framework. Moreover, degrees of understanding filled in a spectrum, from joyful deep understanding to admissions of not understanding why the steps of proof by induction prove anything, and conceptions in between. These included a supposition why a proof of an induction step would, in combination with an established base case, constitute a proof by induction, that its originator characterized as weird''. \section{Domain, Range, Mapping, Relation, Function, Equivalence Relation in Proofs} \section{Definitions, Language, Reasoning in Proofs} \section{Equivalence Class, Generic Particular, Abstraction in Proofs} Using an analogy, I claim, is saying there is a set of relations among things $a_i$ that we agree upon, furthermore, I might wish to teach that there is a corresponding set of relations among things $b_i$. I might wish to say, use the relation we agree upon for municipalities provide addresses for homes that can be used for surface mail, and I might wish to teach that there is a corresponding provision of addresses for items a computer programmer might wish to use for storage and recall. We can note that addresses make use of a hierarchy of place names, countries, states, cities, streets, street numbers, apartment numbers. We can note that structured data types can correspondingly make use of instances and fields and indices that can be arranged in a tree. In the absence of abstraction, the surface mail address hierarchy might not pose much more difficulty, but the data structure might, because the fields therein are more subject to change than municipalities. In the absence of abstraction, the comparison between one hierarchical arrangement with another would be more difficult, because it is the structure of the abstraction itself, namely, the choices of features regarded as significant throughout the tree, that is to be recalled and used as a scaffold for the new information. An alternative pathway towards abstraction involves recognizing an analogy between two structures in different domains, which then focuses one's attention on the abstract structure they share. This new abstraction then becomes a ’concrete’ concept that one can study'' \cite [p 449]{}. \section{Vertical Integration and Explanation} It is accepted that, in discrete math, it is helpful to work problems. We may inquire, what is it about working problems that helps? We can, at the neurophysiology level, expect that long term potentiation of synapses, for the synapses collocated with the long term memory for the concepts in the problem, is being carried out, as the thinking about the problem occurs. We can recall that a sense of reward, as might be gained from success at a problem, or pleasantness in a problem statement, helps consolidate the memory for the ideas that have been gained. We can recall that depression resulting from avoidance of sadness at failure to solve a problem reduces the ability of the hippocampus to support the formation of long term memory. We can, at the cognitive neuroscience level, expect that the opportunity for like structures to be recognized occurs, and analogies, are made, and the abstraction hierarchy of concepts related to the problem is remodeled, extended, to more closely mirror the mathematical definitions being used. We can, at the phenomenography level, suppose that fine distinctions between concepts will be more likely to be noted, because the mental structures that support these are forming. We can, at the computer science education level, consider how to bring activity into lecture, that poses the analogies and distinctions we wish the student to gain, by varying the examples of the concepts such that a representative example is contrasted with a non-example, in an ambience that fosters curiosity and rewards progress. \section{ Validation} We draw a connection between epistemology and validation. Epistemology is why we believe what we believe. Thus, it makes sense to apply our epistemological framework to explain why we believe what we believe about our results and interpretation. Our epistemology is informed by the work of others over a wide range of disciplines: Computer science education, mathematics education, and education generally, especially phenomenography. Cognitive aspects, including memory and attention, are shared by and form a bridge between phenomenography and cognitive neuroscience. Neuroscience provides interesting relevant information. Ira Black, MD, in [?, ?, [p 40] A satisfactory mechanistic description of any well-framed cognitive process requires that we simultaneously explain it at multiple levels of analysis. Different levels provide complementary insights to characterization and causality that are unobtainable from any single line of analysis.'' \subsection{Validation at the Level of Computer Science Education} Interviews with computer science educators, both instructors and teaching assistants have provided a diversity of viewpoints, and generally support the interpretations we have given. For example, one instructor, when asked what students thought proofs were, said some kind of magic incantation'', and teaching assistants have said students really struggle with this''. \subsection{Validation at the Level of Mathematics Education} The literature of mathematics education includes work on students' learning about proof. Our work with computer science students has had the benefit of some students who are dual majors of math and computer science, which has allowed us to trace the similarities and differences of these cohorts of students. The significance of definitions, the necessity and utility of proof, the role played by interest in forming procedures and functions, the difference between functional and procedural programming have differed in these three cohorts, in so far as we have been able to examine. We did not explore the interest in developing procedures/functions or procedural or functional programming in mathematics majors who were not also computer science majors. \subsection{ Validation at the Level of Phenomenography/Variation Theory} Variation theory supports our observation that comparing and contrasting fine distinctions in material being taught aids the process of learning. We used the difference between assignment and equality testing, manifest in the java expression of =='' vs. =''. We compared a software procedure representation with a mathematical formulation (the latter using only =''), for comprehensibility by students. This helped us to see that barriers to student understanding exist, for some students of computer science, at the level of formulation. It also helped us see that the barrier between the internalization and interiorization of Harel and Sowder might be less of a barrier in students of computer science who are routinely conscious of the need to analyze procedures. \subsection{Validation at the Level of Cognitive Neuroscience} Many students have expressed an interest in learning from examples, and researchers studying students' acquisition of the ability to prove have observed a category for concepts called perceptual'' where students mistakenly believe or hope, that examples constitute a proof (of universality). Valiant points out two cases where examples are very effective in learning: when on may employ elimination when the concept to be learned is a conjunction and an exemplar exhibits a variable in negative form, it is clear that that variable is not needed for set membership. When a concept is a disjunction any variable that appears in positive form in a negative example is shown to be insufficient to guarantee membership. Valaint [p. 171] Humans do not argue readily from the contrapositive. P.C. Wason 1983 Realism and rationality and the selection task. Thinking and Reasoning: Psychological Approaches Evans, ed., Routledge Valiant, in Circuits of the Mind [] that an important class that is not currently known to be learnable is disjunctive normal form (or DNF for short)\ldots This appears to be a most natural generalization of simple conjunctions from the viewpoint of modeling human concepts. It can express the idea that examples of a concept fall into a number of somewhat distinct categories, each corresponding to noe of the conjunction. When discussing inductive learning we have a \textit{hierarchical} context in mind. If we wish to learn DNF formulae, but do not have an algorithm for learning these direction, we can nevertheless attempt to learn these in stages. For example, to learn $x_1x_2 \land x_2x_3$ we could first learn the simple conjunctions $x_1x_2$ and $x_2x_3$ separately in some fashion. Having learned these we can learn the DNF when learning hierarchical in this way more is required of the teacher or environment than in the simplest case of learning by example. Somehow the subconcept $x_1x_2$ must be learned separately in supervised or unsupervised mode. In the former case, for example, a teacher may have to teach the name of this subconcept in unsupervised memorization mode and then identify positive or negative examples of it so that it is learned in supervised mode inductively. Alternatively, this subconcept may be learned in unsupervised mode either by memorization or be correlational learning.  Valiant has written\cite{this is in a separate pdf} that the hippocampus is likely to be the location where the allocation of new memory locations is carried out. Valiant has observe that, given a set of concepts that can be hierarchically related, in the absence of hierarchy, when instead the concepts are flattened out, it is more unwieldy to make analogies, such as $A^B$ is analogous to $C$. This is supportive of our interpretation of student data, in which we suggest that students who find abstraction challenging, will in turn find remembering and using definitions more challenging, and will be at a disadvantage in terms of advancing to definition-based axiomatic reasoning. \subsection{ Validation at the Level of Neurophysiology} (Says who?) suggests that memory for events that are observed through one sensory modality are stored near the nervous tissue that process the input for that modality. This is supportive of our interpretation of student data, in which we suggest that students who learn proof by mathematic induction, represented symbolically, and recursive algorithms, represented pictorially, do not always see'' the analogy right away, because the memory traces are not activated at the same time. When considering the two topics at the same time in discussion, both ideas are recalled, corresponding to metabolic activity in the memory (possibly tow different, separate, regions) facilitating formation of connection between the two ideas. From Chapter 18 Migration in the Hippocampus, of Cellular Migration and Formation of Neuronal Connections: Comprehensive Developmental Neuroscience Vol 2, 2013, Elsevier the proliferative subgranular zone located at the border between the granular cell layer and the hilus, which serves as the major site for persistent neurogenesis in the adult hippocampus'' one of the more unique aspects of hippocampas development is the formation of the dentate gyrus, which involves formation of a specialized neural stem cell niche.'' hippocampus harbors neural circuitry essential for learning and memory {Lisman 1999} relating hippocampal circuitry to function neuron'' Cajal-Retzius cells \ldots somewhat penetrate the boundary but tend to avoid invading into olfactory cortex of olfactory bulb Bielle et al. 2005'', maybe signalling pathways \ldots regulate the redial glia-guided migration in the neocortex'' p.335 (not only during development) 0.339 nice graphic for germinative layers (was that supposed to be p.339?) persistent neurogenesis in adult hippocampus Altman and Bayer 1990 Migration and Distribution percursors Comp Neurol subgranular zone Li et al. 2009 neurogenic zone Li and Pleasure 2005 Morphogensis of Dentat gyrus