ASEE-DELOS_Cooper.tex

\documentclass[12pt]{article}
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{subcaption}
\usepackage{longtable}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{etoolbox}
\usepackage{changepage}
\usepackage{titlesec}
\usepackage[parfill]{parskip}
\usepackage[margin=1in]{geometry}
\usepackage{times}
\usepackage{float}
\usepackage[numbers,super]{natbib}

\titleformat*{\section}{\normalsize\bfseries} % Makes section titles 12 pt font


%----------------------------------------------------------------------------------------
%  TITLE SECTION
%----------------------------------------------------------------------------------------
\title{Project-based engineering competition in upper-level engineering laboratory} % using \large makes the title
% Author info isn't included for the Annual Conference but some regional
% conferences might request it.
\author{}
%\author{\normalsize Author Name\\
%\normalsize email@example.com\\
%\normalsize Name of Your Department\\\
%\normalsize Your Institution Name}
\date{} % This leaves the date blank.

\makeatletter % This gets the margins for the title set.
\patchcmd{\@maketitle}{\begin{center}}{\begin{adjustwidth}{0.5in}{0.5in}\begin{center}}{}{}
\patchcmd{\@maketitle}{\end{center}}{\end{center}\end{adjustwidth}}{}{}
\makeatother

%----------------------------------------------------------------------------------------

\begin{document}
\raggedright
\maketitle
\thispagestyle{empty}
\pagestyle{empty}

%----------------------------------------------------------------------------------------
%  PAPER CONTENTS
%----------------------------------------------------------------------------------------
\section*{Abstract}

In this paper, I discuss novel features in an upper-level engineering course
that have been used to enhance technical writing and problem-solving skills.  I
redesigned the course in Fall 2018 to prepare students to make engineering
decisions and accomplish design goals.  My short-term objectives were to prepare
the students to start their capstone projects senior year and improve technical
writing.  The laboratory course includes a number of novel features:
specifications grading, interactive Jupyter lab handouts, and problem- and
project-based learning. Problem-solving skills were evaluated with six
problem-based learning (PBL) laboratories and a Project-based learning (PjBL)
contest that had a cash prize. The technical writing skills were improved using
specifications grading in all seven laboratories. Students were given a detailed
rubric with a pass-fail threshold. Reports that did not meet the specification
for pass, were revised and resubmitted. The specifications grading provided a
method for students to learn from failure. Over 50\% of students increased
technical writing quality. The Jupyter notebooks helped to close the gap between
rational and empirical design. In project-based
learning, the students designed their own set of experiments including finite
element analysis and experimental procedures. The students were graded upon
their approach to the problem and quantification of uncertainties in measured
and predicted values. Using the 2019-2020 senior capstone students, I found a
statistically significant increase in preparation for engineering design from
taking the lab course with PjBL. I discuss the impacts of specifications
grading, project-based learning competition, and detail the measured
improvements in technical writing throughout the semesters in Fall 2018 and Fall
2019. The impacts were measured based upon a standardized rubric and qualitative
assessments. 

%In conclusion, this course included a number of novel features: Problem- and
%Project-based learning (PBL and PjBL), interactive lab handouts via JupyterHub,
%and specifications grading. PBL and PjBL increased student motivation and
%confidence when beginning senior capstone projects. The PjBL competition was a
%welcomed success by students. Most lab groups excelled in rational and empirical
%design processes for the competition. Groups that did not meet expectations
%revised their work and continued to improve technical writing quality. The
%specifications grading provided a method for students to learn from failure and
%over 50\% of students increased technical writing quality. Access to interactive
%notebooks increased the variety and use of the lab handouts. Using Jupyter
%notebook handouts created a medium that mixed background information, data
%processing, and simple engineering models. The Jupyter notebooks helped to close
%the gap between rational, thinking design and empirical, hands-on design. The
%project-based upper engineering lab course redesign has been a success. Using
%the 2019-2020 senior capstone students, I found a statistically significant
%increase in preparation for engineering design from taking the lab course with
%PjBL. 


%------------------------------------------------

\section*{Introduction}

Engineers are expected to create models, take measurements, make predictions,
validate models and communicate difficult concepts.  The most important ABET
outcomes ranked by practicing engineers, employers, and recent graduates are
1-problem solving and 2-communication\cite{passow2017,evans1993}.
Problem-solving comes in two main forms, rational design including: mathematical
models, computer models, and propagation of error and empirical design
including: measurements, curve-fitting, and statistical models. An upper-level
engineering course is the ideal place to combine these rational and empirical
design approaches.  As academics, we often favor rational design e.g. Newton's
laws, differential equations, and thermodynamics.  Students are often drawn to
engineering for its empirical appeal e.g.  learn by doing, hands-on creation,
and create and measure approach.  Rationalists and empiricists
have fought for centuries, marked especially by the conflict between David
Hume\cite{hume1739} and Immanuel Kant\cite{kant1781}.  The divide between
rational and empirical thought creates skepticism in both design methods. I see
the divide between rationalism and empiricism as the same division between
engineering professor and engineering student. Despite skepticism between
rational and empirical approaches, engineers are expected to build innovative
designs with both rational models \emph{and}  empirical
measurements.  We relate quantitative, rational models to
quantitative, empirical measurements through statistical quantities e.g.
confidence intervals and safety factors.  Engineers have to communicate rational
and empirical ideas to accomplish goals. 

Technical writing is crucial to communicating model predictions and measured
results. Despite the necessity for strong writing skills, students struggle to
meet professors'\cite{lillis2001} and employers'\cite{conrad2017} expectations
for quality writing. I use specification grading\cite{nilson2015} to allow
students to learn from failures and respond to feedback.
Specification grading introduces pass-fail grading of the lab reports similar to
competency-based education or mastery learning\cite{bloom1971, kulik1990}.
Students are given a detailed rubric and a minimum standard for passing the
course. Failed assignments can be revised by using a token
system\cite{nilson2015}. Specification grading is meant to decrease the time and
effort spent on individual assignments; this time is spent providing critical
feedback\cite{nilson2015,blackstone2019}. Technical writing is a skill that
every practicing engineer uses to communicate ideas and findings. 

The role of an upper-level engineering laboratory is to teach the connection
between rational and empirical design and technical writing. Technical writing
cannot be taught in isolation from technical context\cite{passow2012}. It is
important for an upper-level engineering class to emulate engineering design as
much as possible. The combination of rational and empirical design and technical
writing fits into the general approach of problem-based and project-based
learning, (PBL and PjBL, respectively). The difference between PBL and PjBL is
that in PBL the instructor specifies tasks to be performed in basic steps. In
contrast, PjBL specifies a greater task and the students create strategies and
approaches\cite{burguillo2010}. Both PBL and PjBL have shown to be effective in
higher education\cite{carlile1998,morrison2004}.  Students
search, solve, create, and share approaches\cite{awang2008} using math models
and measurements, then sharing is done with technical documents or graphs. PjBL
can have a positive effect on students' attitudes towards the
course\cite{bell2010}. Competitions in PjBL helps motivate students to approach
more difficult concepts in the classroom\cite{burguillo2010,michieletto2018}. 

The goals of this upper-level engineering project-based laboratory are to
improve problem-solving skills and technical writing skills.  The
problem-solving skills are evaluated with six PBL laboratories and a PjBL
contest that with a cash prize. Rational and empirical design principals are
presented in Jupyter notebooks that combine background information, data
processing, and modeling.  The technical writing skills were improved using
specifications grading in all seven laboratories. 


%------------------------------------------------

\section*{Methods}

The course focuses on problem-solving and technical writing. The laboratory
schedule is shown in Fig.~\ref{timeline}. At the University of Connecticut
department of Mechanical Engineering department, we had 215 students in
Fall~2018 and Fall~2019 enroll in this course, ME3263-Introduction to Sensors
and Data. In the course, Labs \#0-4 and 6 are PBL activities
where students are given basic steps and asked to write technical documents.
Lab \#5 is a PjBL activity; I specify that the class needed to measure the
mass of an object using a vibrating beam. Lab \#0 is used to introduce
statistical significance in measurements. We relate discussions of rational
models and empirical measurements with statistical analysis. All students work
with the same data set and submit reports graded with the rubric in 
Appendix A. Lab \#1 asks students to quantify differences in machining
methods between band saw and computer numerical control (CNC) parts.  Labs \#2-4
ask students to quantify differences between rational predictions using
rational models and empirical measurements for static and
dynamic cantilever beams. In the PjBL activity, the Lab \#5 competition, the
students are given the task to create a design of experiments, create a
predictive model, and use engineering judgment to measure the mass of an object
on a vibrating beam. The final Lab \#6 included a combination of rational
predictions and empirical measurements using lumped-mass assumptions, finite
element analysis, and thermocouples. 

\begin{figure}
  \centering
  \includegraphics[width=5in]{./lab_schedule.png}
  \caption{Laboratory schedule for the 14-week semester in upper-level
  engineering course. Each box represents an assignment that includes
  measurements, statistical analysis, and lab report. The ``Mass Measurement
  Contest'' asks students to use a combination of methods from weeks 1-9 to
  predict the mass of an object attached to a vibrating beam. The final two
  weeks are used to measure a first-order convective heat transfer problem,
  incorporating statistical uncertainty, finite element analysis, and
  verification. \label{timeline} } 
\end{figure} 

The laboratory course includes a number of novel features: specifications
grading, interactive lab handouts, and a PjBL competition with \$150-prize.
I use specifications grading for lab reports \cite{nilson2015}. Each lab report
is graded based upon a pass-fail criteria and a standardized grading rubric. Lab
groups of two students are given the opportunity to revise failed lab reports
with tokens. Initially, each lab group has two tokens with the opportunity to
earn more during in-class discussions or extra credit assignments. Specification
grading is geared towards meeting a minimum set of standards, but allowing the
teaching assistants and myself to offer technical writing criticism. The goal is
to help the class improve technical writing skills or at least maintain a
reasonable quality for professional engineers. 

The lab handouts are hosted as interactive Jupyter\cite{kluyver2016} notebooks.
Students access a server to process example test data, enter their experimental
data, and plot results of rational predictions and empirical analysis.  The
background information is rendered as html with links to resources such as
Student's 1908 ``The Probable Error of a Mean''\cite{student1908}, animations,
or Wikipedia articles.  The goal is to combine rational and empirical design.
Thus, providing resources for capstone engineering projects and ultimately for
professional engineering projects.

The project-based competition asks lab groups to measure the mass of an object
attached to a vibrating beam. In weeks 10 and 11, the students create a design of
experiments, take measurements, and create finite element analysis models. The
competition does not have calibration weights, so the students have to rely on
rational predictions and engineering judgments.  The
competition ends with the submission of their best estimate of object mass with
a propagation of error and the lab report's Methods section. The lab group with the most
accurate measurement is awarded a \$150-prize. After the prize is awarded, the
actual object masses are announced. The lab groups use week 12 to revise
their approach and submit the lab report. The goal is to encourage students to
create, design, and evaluate. Then, the teaching assistants and myself give
clear feedback on the final error in the predicted results. 

%------------------------------------------------

\section*{Results and Discussion}

% Added F-value from repeated Anova          
%Anova
%======================================
%       F Value Num DF Den DF    Pr > F
%--------------------------------------
%report 23.7442 4.0000 1776.0000 0.0000
%======================================

The course focuses on problem-solving and technical writing.  In Fig.~\ref{quality}(a),
the scores of each lab group is fit to a linear model to measure average increase in
grade per report between Labs \#0-4. The goal was to have the entire class in the green
``continuous improvement''-area. In Fall~2018, 56\% of the class continually improved and
in Fall~2019, 59\% of the class continually improved their scores. The ``maintain
quality'' area represents students that write reports of high quality initially, but do
not improve during the course of the class. In Fall~2018 and Fall 2019, the students that
maintained quality accounted for 43\% and 36\%, respectively.  The remaining 1\% and 4\%
of the class did not improve or meet specifications for lab reports, in Fall 2018 and
2019, respectively. The F-value in a one-way repeated Analysis of Variance of lab report
scores, using the Python package statsmodels\cite{seabold2010}, was 23.74 between labs 0-4
with 445 students indicating that there was a statistically significant affect on lab
report grades. The grades from Labs~\#5-6 are shown in Fig.~\ref{quality}(b).  Lab~\#5 was
the PjBL contest and marked a significant increase in expectations.  The results of this
study, suggest that students are able to incorporate feedback from teaching assistants and
myself and show improvements in technical writing. The Labs increased in difficulty, so
even the groups of students that maintained their grade at the specified level show marked
improvement in communicating difficult concepts. 

I found specifications grading in technical writing to be an effective method of
evaluation. The grades are normally distributed with the class mean increasing from 80
to 85~points. One argument against 
specifications grading is that students may not be motivated to increase their
grade because once the grade is above passing there is no incentive to improve.
I find a clear increase in grades throughout the semester, and the students
that were in the ``maintain poor quality'' regime did fail and redo lab reports.
The students that did not improve found great difficulty in Labs~\#5-6, most
failing those assignments and revising their work. The specifications grading
also has the most noticeable effect on under-performing students. The students
that failed Lab~\#0 had an average grade increase of 5~pts/report. This increase
would result in a score of 85-90 on these students Lab~\#6 reports, if the
progress was sustained and labs did not become more demanding. 

% 2018 2 s = 2/52 maintain poor qual
% 2018 0.56 improve
% 1-0.56-2/220=43%

% 2019 10 o = 10/83 maintain poor qual
% 2019 0.59 improve
% 1-0.59-10/228 = 36%

\begin{figure}[ht!]

  \begin{subfigure}[t]{0.5\textwidth}
    \begin{centering}
    \includegraphics[width=3.5in]{./track_progress/report_quality.png}
    \caption{}
    \end{centering}
  \end{subfigure}
  \begin{subfigure}[t]{0.5\textwidth}
    \begin{centering}
    \includegraphics[width=3in]{./track_progress/report_scores.png}
    \caption{}
    \end{centering}
  \end{subfigure}

  \caption{Plotted above in (a) is the average change in lab report grade as a function of
  the first Report~\#0. The
  specification for passing Report \#0 is shown as a red line at 70 points.  The
  green area above the ``Linear model change in grade''=0 shows the students that
  continuously improved their report grades throughout the semester. The dark red
  section in the lower-left, that has no student data, would be students that
  performed poorly and continued to decrease quality. The light-red section
  between 70 and 100 are the students that decreased quality to the point of
  risking failing Report~\#6. The yellow section between 70 and 100 above the
  orange risk section are students that decreased quality, but maintained high
  enough marks to not risk failing lab reports.  There are three populations of
  students from Fall 2018 $\square$~markers and Fall 2019 $\circ$~markers: Red indicates
  students that failed Report~\#0, but their scores increased throughout the
  semester, Green indicates students that passed Report~\#0 whose scores continued
  to increase throughout the semester, and orange are students that passed
  Report~\#0, but their scores decreased throughout the semester. The orange marks
  in the red sections, ``maintain poor quality'' were at risk of failing other lab
  reports. In (b), box plots of the scores from 2018 and 2019 on reports 0-6 are
  plotted. The median is shown by a horizontal line, the notches indicate the
  confidence interval, the whiskers denote the range of scores, with outliers
  marked as circles, and the upper- and lower-quartiles are shown by the boxes
  above and below the median lines. The red-dashed line indicates the
  specification for a passing grade on the reports. \label{quality}} 
\end{figure} 

The PjBL Lab~\#5 activity results are plotted in Fig.~\ref{contest}. The
histogram of errors based upon reported results demonstrate the range of
effectiveness of each lab group's experimental work. In Fall~2018 and Fall~2019,
the average and standard deviation in error to measure a 32-g object was
18.3$\pm$32.8~g and 11.4$\pm$26.7~g, respectively. While top three most accurate
reports had errors less than 4\%. The competition provides specific
feedback to lab groups, and a non-grade-based metric to evaluate
student effort and learning. 

This PjBL Lab qualitatively had the highest enthusiasm and participation from the
students. Student SET responses included, ``I liked the mass measuring contest!'', ``I
liked using ANSYS and the competition.'', ``I liked the competition where the answer was
unknown. I think that was the most beneficial thing we did and I think more of those labs
would be helpful.'' Attendance to announce winners of the contest was not mandatory, but
over 90\% of the class was present. Students compared answers, studied methods, and
results. After the object masses were given to the class, they revised their methods one
more time to reduce errors in their data collection and processing. These competitions
work best when the learning happens whether or not the group wins\cite{burguillo2010}. The
benefit of the contest was the increased enthusiasm in studying beam dynamics and finite
element methods. Even students that had very high errors demonstrated finite element
models convergence and fast fourier transform analysis of natural frequencies of
cantilever beams.  

\begin{figure}[ht!]
\centering
  \includegraphics[width=5in]{./track_progress/mass_measure.png}
  \caption{Plotted above is a histogram of the reported errors from Fall~2018
  and Fall~2019 for the mass measurement contest. The average error in mass reported in
  Fall~2018 and Fall~2019 was 18~$\pm$~33~g and 11.4~$\pm$~27~g, respectively with
  error reported as standard deviation. The actual mass measurements were
  32~$\pm$~2~g. The histogram is the error=(reported value - the actual value). \label{contest}}
\end{figure} 

I polled the 2019-2020 senior capstone project teams that took this
project-based upper-level engineering lab course in either Fall 2018, 2019, or
not at all. Students' comments about the course included ``Was a great and
helpful class'', ``Great class! Very helpful for senior design'', and ``ME3263
was a great course for technical writing.'' The students were asked how useful
each skill that was introduced in this course is in relation to accomplishing a
senior capstone project. Over 50\% of the class of 270, agreed that all eight
skills were useful and 50\% of the class considered technical writing to be a
\emph{crucial skill}. The last question in the survey is: ``How prepared did you
feel starting senior design with your background from this course?'' Of the
students that took the course in Fall 2018 and Fall 2019, over 45\% felt
prepared and students that hadn't taken the course less than 30\% felt prepared.
Using a one-way analysis of variance on the responses (0:unprepared-4:very
prepared), 121 students from Fall 2018, 24 from Fall 2019, and 17 N/A, the
f-statistic=2.2 with a p-value of 0.11 between all three. While, considering
just the difference between Fall 2018-Fall 2019, the f-statistic is 0.01 and
p-value of 0.93. There is a statistically significant difference between
students that took the PjBL course and those that did not. This measurement
gages the students' perceived preparation for the senior capstone project. 

\begin{figure}[ht!]
\centering
  \includegraphics[width=6in]{./track_progress/survey_prep.png}
  \caption{Plotted above is a histogram of the responses from senior capstone
  project students that either: took the project-based laboratory course
  concurrently with capstone, in the previous year, or not at all. The students
  were asked to rate the necessity of eight problem-solving and technical
  writing skills that were introduced in this project-based laboratory course.\label{contest}}
\end{figure} 

%------------------------------------------------
\section*{Conclusions and Future Work}

In conclusion, this course included a number of novel features: Problem- and
Project-based learning (PBL and PjBL), interactive lab handouts via JupyterHub,
and specifications grading. PBL and PjBL increased student motivation and
confidence when beginning senior capstone projects. The PjBL competition was a
welcomed success by students. Most lab groups excelled in rational and empirical
design processes for the competition. Groups that did not meet expectations
revised their work and continued to improve technical writing quality. The
specifications grading provided a method for students to learn from failure and
over 50\% of students increased technical writing quality. Access to interactive
notebooks increased the variety and use of the lab handouts. Using Jupyter
notebook handouts created a medium that mixed background information, data
processing, and simple engineering models. The Jupyter notebooks helped to close
the gap between rational and empirical, hands-on design. The
project-based upper-level engineering lab course redesign has been a success. Using
the 2019-2020 senior capstone students, I found a statistically significant
increase in preparation for engineering design from taking the lab course with
PjBL. 

Some ongoing work will be to evaluate the effectiveness of individual changes in
the course. Specifications grading is a novel way to asses engineering students'
technical writing skills. I believe the process of revising reports provides
much-needed practice for students, but it would be interesting to see what
fraction of the class has measurable increase in writing quality without this
process. I assume the PjBL competition is a big motivational and preparational
tool, but there may be other sources of motivation and preparation. Some future
work is to compare results between a competition-based PjBL and
PjBL component with no competition and to incorporate senior design grades into
the analysis of the effectiveness of the course. 

%----------------------------------------------------------------------------------------
%  REFERENCE LIST
%----------------------------------------------------------------------------------------
\vspace{4\baselineskip}\vspace{-\parskip} % Creaters proper 4 blank line
                                          %spacing.
\footnotesize % Makes bibliography 10 pt font.
\bibliographystyle{unsrtnat} %Can use a different style as long as it is one
\bibliography{ASEEpaper}

%----------------------------------------------------------------------------------------
\pagebreak

\section*{Appendix A}

\begin{longtable}[]{@{}llllllll@{}}
\begin{minipage}[b]{0.10\columnwidth}\raggedright
Section\strut
\end{minipage} & \begin{minipage}[b]{0.10\columnwidth}\raggedright
Category\strut
\end{minipage} & \begin{minipage}[b]{0.10\columnwidth}\raggedright
Unacceptable (0)\strut
\end{minipage} & \begin{minipage}[b]{0.10\columnwidth}\raggedright
Acceptable (½)\strut
\end{minipage} & \begin{minipage}[b]{0.10\columnwidth}\raggedright
Good (¾)\strut
\end{minipage} & \begin{minipage}[b]{0.10\columnwidth}\raggedright
Excellent~(1)\strut
\end{minipage} & \begin{minipage}[b]{0.10\columnwidth}\raggedright
Weight\strut
\end{minipage} & \begin{minipage}[b]{0.10\columnwidth}\raggedright
Score\strut
\end{minipage}\tabularnewline
\endhead
\begin{minipage}[t]{0.10\columnwidth}\raggedright
Introduction\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
first sentence is interesting and grab's reader's attention\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
2\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
Problem or hypothesis is stated clearly\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
6\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
Physical principles are stated clearly\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
6\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
other applications are discussed\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
4\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
reason for experiment is stated clearly\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
4\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
Methods\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
Relevant experimental details are discussed\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
4\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
Materials\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
equipment and environmnet discussed"\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
4\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
Results and Discussion are not mentioned\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
4\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
Results and Discussion\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
Output of experiments are presented\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
8\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
principles from Introduction are applied to data\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
8\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
analysis/model/theory is presented\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
8\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
Conclusion\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
Summarizes report\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
2\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
relationship between results and analysis (or model or theory) is
discussed\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
4\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
Most important results are summarized\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
4\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
consequences of results are discussed\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
4\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
reservations or limitations of study are discussed\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
4\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
References\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
significant previous work/textbooks are cited\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
4\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
references are complete\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
4\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
Figures\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
Figures are easy to interpret\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
8\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
Overall\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
Spelling/grammar\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
2\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
significant digits are correct\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
2\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
flow\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
2\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
Appendix\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright
contains essential material that would interupt flow\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
2\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
100\strut
\end{minipage} & \begin{minipage}[t]{0.10\columnwidth}\raggedright
\strut
\end{minipage}\tabularnewline
\end{longtable}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}