ASEE_paper.tex

\documentclass[12pt]{article}
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{subcaption}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{etoolbox}
\usepackage{changepage}
\usepackage{titlesec}
\usepackage[parfill]{parskip}
\usepackage[margin=1in]{geometry}
\usepackage{times}
\usepackage{float}
\usepackage[numbers,super]{natbib}

\titleformat*{\section}{\normalsize\bfseries} % Makes section titles 12 pt font


%----------------------------------------------------------------------------------------
%  TITLE SECTION
%----------------------------------------------------------------------------------------
\title{Project-based engineering competition in upper-level engineering laboratory} % using \large makes the title
% Author info isn't included for the Annual Conference but some regional
% conferences might request it.
\author{}
%\author{\normalsize Author Name\\
%\normalsize email@example.com\\
%\normalsize Name of Your Department\\\
%\normalsize Your Institution Name}
\date{} % This leaves the date blank.

\makeatletter % This gets the margins for the title set.
\patchcmd{\@maketitle}{\begin{center}}{\begin{adjustwidth}{0.5in}{0.5in}\begin{center}}{}{}
\patchcmd{\@maketitle}{\end{center}}{\end{center}\end{adjustwidth}}{}{}
\makeatother

%----------------------------------------------------------------------------------------

\begin{document}
\raggedright
\maketitle
\thispagestyle{empty}
\pagestyle{empty}

%----------------------------------------------------------------------------------------
%  PAPER CONTENTS
%----------------------------------------------------------------------------------------
\section*{Abstract}

In this paper, I discuss novel features in an upper-level engineering course
that have been used to enhance technical writing and problem-solving skills.  I
redesigned the course in Fall 2018 to prepare students to make engineering
decisions and accomplish design goals.  My short-term objectives were to prepare
the students to start their capstone projects senior year and improve technical
writing.  The laboratory course includes a number of novel features:
specifications grading, interactive Jupyter lab handouts, and a project-based
competition with \$150-prize.  Students spent the first 9 weeks of the course
following experimental procedures and writing lab reports.  In the project-based
competition, the students designed their own set of experiments including finite
element analysis and experimental procedures. The students were graded upon
their approach to the problem and quantification of uncertainties in measured
and predicted values. I awarded a cash prize to the most accurate mass
measurement. I discuss the impacts of specifications grading, project-based
competition, and detail the measured improvements in technical writing
throughout the semesters in Fall 2018 and Fall 2019. The impacts were measured
based upon a standardized rubric and qualitative interviews. 


%------------------------------------------------

\section*{Introduction}

Engineers are expected to create models, take measurements, make predictions,
validate models and communicate difficult concepts.  The ABET outcomes rated
with highest importance from practicing engineers, employers, and recent
graduates are problem solving and communication \cite{passow2017,evans1993}.
Problem-solving comes in two main forms, rational design including mathematical
models, computer models, and propagation of error and empirical design including
measurements, curve-fitting, and statistical models. An upper-level engineering
course is the ideal place to combine these rational and empirical design
approaches.  As academics, we tend to favor rational design e.g. Newton's laws,
differential equations, thermodynamics.  Students are typically drawn to
engineering for its empirical appeal e.g.  learn by doing, create and measure
approach \cite{bot2005}.  Rationalists and empiricists have fought for
centuries, marked especially by the conflict between David Hume\cite{hume1739}
and Immanuel Kant\cite{kant1781}.  The divide between rational and empirical
thought creates skepticism in both design methods. I see the divide between
engineering professor and engineering student as a skepticism between
rationalists and empiricists. Despite skepticism between rational and empirical
approaches, engineers are expected to build innovative designs with both rational
models and validate \emph{and}  empirical measurements and insights.  We relate
quantitative, rational models to quantitative, empirical measurements through statistical
quantities e.g.  confidence intervals and safety factors.  Engineers have to
communicate rational and empirical ideas to accomplish goals. 

Technical writing is crucial to communicating model predictions and measured
results. Despite the necessity for strong writing skills, students struggle to
meet professors'\cite{lillis2001} and employers'\cite{conrad2017} expectations
for quality writing. I used specification
grading\cite{nilson2015} to allow student to learn from failures in their
writing and respond to feedback. Specification grading introduces pass-fail grading of
the lab reports similar to competency-based education or mastery
learning\cite{bloom1971, kulik1990}. Students are given a detailed rubric and
a minimum standard for passing the course. Failed assignments can be revised by
using a token system\cite{nilson2015}. Specification grading is meant to decrease
the time and effort spent on individual assignments, so that time can be spent
providing feedback\cite{nilson2015,blackstone2018}. Technical writing is a skill that
every practicing engineer uses to communicate ideas and findings. 

The role of an upper-level engineering laboratory is to teach the connection
between rational and empirical design and technical writing. Technical writing
cannot be taught in isolation from technical context\cite{passow2012}. It is
important for an upper-level engineering class to emulate engineering design as
much as possible. The combination of rational and empirical design and technical
writing fits into the general approach of problem-based and project-based
learning, (PBL and PjBL, respectively). The difference between PBL and PjBL is
that in PBL the instructor specifies tasks to be performed in basic steps. In
contrast, PjBL specifies a greater task and the students create strategies and
approaches\cite{burguillo2010}. Both PBL and PjBL have shown tobe effective in the
classroom\cite{carlile1998,morrison2004}.  Students
search, solve, create, and share approaches\cite{awang2008} using math models
and measurements, then sharing with technical documents or graphs. Project-based
learning can have a positive effect on students' attitudes towards the
course\cite{bell2010}. Competitions in PjBL helps motivate
students to approach more difficult concepts in a
classroom\cite{burguillo2010,michieletto2018}. 

The goals of this upper-level engineering project-based laboratory are to
improve and evaluate problem-solving skills and improve technical writing
skills.  The problem-solving skills were evaluated with six problem-based
learning (PBL) laboratories and a Project-based learning (PjBL) contest that had a
cash prize. The technical writing skills were improved using specifications
grading in all seven laboratories. 


%------------------------------------------------

\section*{Methods}

The course focuses on problem-solving and technical writing. The laboratory schedule
is shown in Fig.~\ref{timeline}. Labs \#0-4 and 6 were PBL activities where
students were given basic steps and asked to write technical documents. Lab \#5
was a PjBL activity; I specified that the class needed to measure the mass of an
object using a vibrating beam. Lab \#0 was used to introduce statistical
significance in measurements. We relate discussions of rational models and
empirical measurements with statistical analysis. All students worked with the same data set and
submitted reports graded with the rubric in the appendix A.1. Lab \#1 asked
students to quantify differences in machining methods between band saw and
computer numerical control (CNC) parts.  Labs \#2-4 asked students to quantify
differences between rational predictions using analytical and numerical models
and empirical measurements. In Lab \#5, the students were asked to perform a
design of experiments, create a predictive model, and use engineering judgment
to measure the mass of an object on a vibrating beam. The final Lab \#6 included
a combination of rational predictions using lumped-mass assumptions, finite
element analysis, and empirical measurements. 

Two of the labs
included finite element analysis to create numerical experiments. The numerical
experiments were compared to measured values for validation and analytical
values for verification. Lab \#2 included digital image correlation to measure
the full kinematic deflection of a beam under static load. 

\begin{figure}
  \includegraphics[width=5in]{./lab_schedule.png}
  \caption{Laboratory schedule for 14-week semester in upper-level engineering
  course. Each box represents an assignment that includes measurements,
  statistical analysis, and lab report. The "Mass Measurement Contest"
  asks students to use a combination of methods from weeks 1-9 to predict the mass
  of an object attached to a vibrating beam. The final two weeks are used to
  measure a first-order convective heat transfer problem, incorporating
  statistical uncertainty, finite element analysis, and verification. \label{timeline} }
\end{figure} 

The laboratory course includes a number of novel features: specifications
grading, interactive lab handouts, and a PjBL competition with
\$150-prize.   I use specifications
grading for lab reports \cite{nilson2015}. Each lab report is graded based upon a
pass-fail criteria and a standardized grading rubric. Lab groups of two students were given the
opportunity to revise failed lab reports with tokens. Initially, each lab
group has two tokens with the opportunity to earn more during in-class
discussions or extra credit assignments. Specification grading is geared towards
meeting a minimum set of standards, but allowing the teaching assistants and
myself to offer more criticism. The goal was to help the class improve technical
writing skills or at least maintain a reasonable quality for professional
engineers. 

The lab handouts are hosted as interactive Jupyter\cite{kluyver2016} notebooks.
Students access a server to process example test data, enter their experimental
data, and plot results of analytical predictions.  The background information is
rendered as html with links to resources such as Student's 1908 ``The Probable
Error of a Mean''\cite{student1908}, animations, or Wikipedia articles.  The
goal was to provide resources that prepare the students for capstone engineering
projects and ultimately for professional engineering projects.

The project-based competition asks lab groups to measure the mass of an object
attached to a vibrating beam. In weeks 10 and 11, the students create a design of
experiments, take measurements, and create finite element analysis models. The
competition does not have calibration weights, so the students have to rely on
rational predictions and engineering judgments.  The
competition ends with the submission of their best estimate of object mass with
a propagation of error and the Methods section. The lab group with the most
accurate measurement was awarded a \$150-prize. After the prize was awarded, the
actual object masses were distributed. The lab groups used week 12 to revise
their approach and submit the lab report. The goal was to encourage students to
create, design, and evaluate, then give clear feedback on the final error in the
predicted results. 

%------------------------------------------------

\section*{Results and Discussion}

The course focused on improving technical writing and making measurements. In
Fig.~\ref{quality}(a), the scores of each lab group is fit to a linear model to
determine the change in report grade per report between Labs \#0-4. The goal was
to have the entire class in the green ``continuous improvement''-area. In
Fall~2018, 56\% of the class continually improved and in Fall~2019, 59\% of the
class continually improved their scores. The ``maintain quality'' area
represents students that write reports of high quality initially, but do not
improve during the course of the class. In Fall~2018 and Fall 2019, the students that
maintained quality accounted for 43\% and 36\%, respectively. The remaining 1\%
and 4\% of the class did not improve or maintain report grades, in Fall 2018 and
2019, respectively. We show the grades from Labs~\#5-6 in Fig.~\ref{quality}(b).
Lab~\#5 was the PjBL contest and marked a significant increase in expectations. 
The results of this study, suggest that students were able to incorporate
feedback from teaching assistants and myself and show improvements in technical
writing. The Labs increased in difficulty, so even the groups of students that
maintained their grade at the specified level show marked improvement in
communicating difficult concepts. 

Regarding the effectiveness of specifications grading in technical writing,
there is still a normal distribution of grades with the class mean between 80
and 85~points and grades increased throughout the semester. One argument against 
specifications grading is that students may not be motivated to increase their
grade, because the set point does not change.  I find here a clear increase in
grades throughout the semester, and the students that were in the maintaining
poor quality regime did fail and redo lab reports.  The students that did not
improve found great difficulty in Labs~\#5-6.  

% 2018 2 s = 2/52 maintain poor qual
% 2018 0.56 improve
% 1-0.56-2/220=43%

% 2019 10 o = 10/83 maintain poor qual
% 2019 0.59 improve
% 1-0.59-10/228 = 36%

\begin{figure}[ht!]

  \begin{subfigure}[t]{0.5\textwidth}
    \begin{centering}
    \includegraphics[width=3.5in]{./track_progress/report_quality.png}
    \caption{}
    \end{centering}
  \end{subfigure}
  \begin{subfigure}[t]{0.5\textwidth}
    \begin{centering}
    \includegraphics[width=3in]{./track_progress/report_scores.png}
    \caption{}
    \end{centering}
  \end{subfigure}

  \caption{Plotted above in (a) is the average change in lab report grade as a function of
  the first Report~\#0. The
  specification for passing Report \#0 is shown as a red line at 70 points.  The
  green area above the ``Linear model change in grade''=0 shows the students that
  continuously improved their report grades throughout the semester. The dark red
  section in the lower-left, that has no student data, would be students that
  performed poorly and continued to decrease quality. The light-red section
  between 70 and 100 are the students that decreased quality to the point of
  risking failing Report~\#6. The yellow section between 70 and 100 above the
  orange risk section are students that decreased quality, but maintained high
  enough marks to not risk failing lab reports.  There are three populations of
  students from Fall 2018 $\square$~markers and Fall 2019 $\circ$~markers: Red indicates
  students that failed Report~\#0, but their scores increased throughout the
  semester, Green indicates students that passed Report~\#0 whose scores continued
  to increase throughout the semester, and orange are students that passed
  Report~\#0, but their scores decreased throughout the semester. The orange marks
  in the red sections, "maintain poor quality" were at risk of failing other lab
  reports. In (b), box plots of the scores from 2018 and 2019 on reports 0-6 are
  plotted. The median is shown by a horizontal line, the notches indicate the
  confidence interval, the whiskers denote the range of scores, with outliers
  marked as circles, and the upper- and lower-quartiles are shown by the boxes
  above and below the median lines. The red-dashed line indicates the
  specification for a passing grade on the reports. \label{quality}} 
\end{figure} 

The PjBL Lab~\#5 activity results are plotted in Fig.~\ref{contest}. The
histogram of errors based upon reported results demonstrate the range of
effectiveness of each lab group's experimental work. In Fall~2018 and Fall~2019,
the average and standard deviation in error to measure a 32-g object was
18.3$\pm$32.8~g and 11.4$\pm$26.7~g, respectively. While top three most accurate
reports had errors less than 4\%. 

This PjBL Lab qualitatively had the highest enthusiasm and participation from
the students. Student SET responses included, ``I liked the mass measuring
contest!'', ``I liked using ANSYS and the competition.'', ``I liked the
competition where the answer was unknown. I think that was the most beneficial
thing we did and I think more of those labs would be helpful.'' Attendance to
announce winners of the contest was not mandatory, but over 90\% of the class was
present. Students compared answers, studied methods, and results. After the
object masses were given to the class, they revised their methods one more time
to reduce errors in their data collection and processing. The benefit of the
contest was the increased enthusiasm in studying beam dynamics and finite
element methods. Even students that had very high errors, had finite element
models with demonstrated convergence, fast fourier transform analysis of natural
frequencies of cantilever beams. These competitions work best when the learning
happens whether or not the group wins\cite{burguillo2010}. 

\begin{figure}[ht!]
  \includegraphics[width=5in]{./track_progress/mass_measure.png}
  \caption{Plotted above is a histogram of the reported errors from Fall~2018
  and Fall~2019 for the mass measurement contest. The average mass reported in
  Fall~2018 and Fall~2019 was 18~$\pm$~33~g and 41~$\pm$~27~g, respectively with
  error reported as standard deviation. The actual mass measurements were
  32~$\pm$~2~g. The histogram is the error=(reported value - the actual value). \label{contest}}
\end{figure} 

I also polled the current senior capstone project teams that took this
project-based upper-level engineering lab course in either 2018, 2019, or not at
all. Students comments about the course included ``Was a great and helpful
class'', ``Great class! Very helpful for senior design'', and ``ME3263 was a great
course for technical writing.'' The students were asked how useful each skill
that was introduced in this course was in relation to accomplishing a senior
capstone project. Over 50\% of the class of 270, agreed that all eight skills
were useful and 50\% of the class considered technical writing to be a
\emph{crucial skill}. The last question in the survey was: ``How prepared did
you feel starting senior design with your background from <this course>?''
Of the students that took the course in Fall 2018 and Fall 2019, over 45\% felt
prepared and students that hadn't taken the course only less than 30\% felt
prepared. Using a one-way analysis of variance on the responses
(0:unprepared-4:very prepared), 121 students from Fall 2018, 24 from Fall 2019,
and 17 N/A, the f-statistic 2.2 with a p-value of 0.11 between all three.
Considering just the difference between Fall 2018-Fall 2019, the f-statistic is
0.01 and p-value of 0.93. There is a statistically significant difference
between students that took the PjBL course and those that did not. This
measurement gages the students' perceived preparation for the senior capstone
project. 

\begin{figure}[ht!]
  \includegraphics[width=5in]{./track_progress/survey_prep.png}
  \caption{Plotted above is a histogram of the responses from senior capstone
  project students that either: took the project-based laboratory course
  concurrently with capstone, in the previous year, or not at all. The students
  were asked to rate the necessity of eight problem-solving and technical
  writing skills that were introduced in this project-based laboratory course.\label{contest}}
\end{figure} 

%------------------------------------------------
\section*{Conclusions and Future Work}



%----------------------------------------------------------------------------------------
%  REFERENCE LIST
%----------------------------------------------------------------------------------------
\vspace{4\baselineskip}\vspace{-\parskip} % Creaters proper 4 blank line
spacing.
\footnotesize % Makes bibliography 10 pt font.
\bibliographystyle{unsrtnat} %Can use a different style as long as it is one
\bibliography{ASEEpaper}

%----------------------------------------------------------------------------------------



\end{document}