\documentclass[msp,twoside,print,titletoc,title]{nuthesis}
%Note: twoside could be taken out. Then binding is just left.
\setcounter{secnumdepth}{3}
\usepackage{hyperref}
\usepackage{graphicx}
\usepackage{amsthm}
\usepackage{tikz}
\usepackage{amsmath}
\usepackage{caption}
\usepackage{setspace}
\usepackage{appendix}


%BEGIN JSON STYLING

\usepackage{bera}% optional: just to have a nice mono-spaced font
\usepackage{listings}
\usepackage{xcolor}

\colorlet{punct}{red!60!black}
\definecolor{background}{HTML}{EEEEEE}
\definecolor{delim}{RGB}{20,105,176}
\colorlet{numb}{magenta!60!black}

\def\checkmark{\tikz\fill[scale=0.4](0,.35) -- (.25,0) -- (1,.7) -- (.25,.15) -- cycle;}

\lstdefinelanguage{json}{
    basicstyle=\normalfont\ttfamily,
    numbers=left,
    numberstyle=\scriptsize,
    stepnumber=1,
    numbersep=8pt,
    showstringspaces=false,
    breaklines=true,
    frame=lines,
    backgroundcolor=\color{background},
    literate=
     *{0}{{{\color{numb}0}}}{1}
      {1}{{{\color{numb}1}}}{1}
      {2}{{{\color{numb}2}}}{1}
      {3}{{{\color{numb}3}}}{1}
      {4}{{{\color{numb}4}}}{1}
      {5}{{{\color{numb}5}}}{1}
      {6}{{{\color{numb}6}}}{1}
      {7}{{{\color{numb}7}}}{1}
      {8}{{{\color{numb}8}}}{1}
      {9}{{{\color{numb}9}}}{1}
      {:}{{{\color{punct}{:}}}}{1}
      {,}{{{\color{punct}{,}}}}{1}
      {\{}{{{\color{delim}{\{}}}}{1}
      {\}}{{{\color{delim}{\}}}}}{1}
      {[}{{{\color{delim}{[}}}}{1}
      {]}{{{\color{delim}{]}}}}{1},
}

%END JSON STYLING


\begin{document}

\theoremstyle{definition}
\newtheorem{example}{Example}[section]

%% Start formatting the first few special pages
%% frontmatter is needed to set the page numbering correctly
\frontmatter

\title{A RESTful framework for writing, running, and evaluating code in multiple academic settings}
\author{Christopher Ban}
\adviser{Rick Halterman}
\adviserAbstract{Rick Halterman}
\degree{Master of Science}
\major{Computer Science}
\degreemonth{December}
\degreeyear{2017}
\doctype{Project Defense}

\maketitle
%%
%% You have a maximum of 350 words for your abstract, which includes
%% your title, name, etc.
%%
%% Required
\begin{abstract}
In academia, students and professors want a well-structured and implemented framework for writing and running code in both testing and learning environments. The current limitations of the paper and pencil medium have led to the creation of many different online grading systems. However, no known system provides all of the essential features our client is interested in. Our system, developed in conjunction with Doctor Halterman, offers the ability to build modules from flat files, allow code to be compiled and run in the browser, provide users with immediate feedback, support multiple languages, and offer a module designed specifically for an examination environment.

\end{abstract}


%% The ToC is required
\tableofcontents
%% Uncomment these if need be
\listoffigures
%\listoftables
%%
%% ``Real'' beginning of the document.
%% mainmatter is needed to set the page numbering correctly
%%   mainmatter is needed after the ToC, (LoF, and LoT) to set the
%%   page numbering correctly for the main body
\mainmatter

%% Thesis goes here

\chapter{Introduction}
\label{sec:Introduction}
	In academia, students and professors want a well-structured and implemented framework for writing and running code in both testing and learning environments. In a test or quiz setting, students prefer to type and have the ability to execute code. In an instructional medium such as a handout or online textbook, students would prefer to try out code snippets right in the browser without the need to install or switch to another program. Some on the other hand would like the ability to build exams and quizzes without multiple or repetitive steps, possibly leading to a decrease in the creation of these activities (which Cheang et al.~\cite{Cheang} has shown can be crucial.) Finally, automated and near-immediate grading and feedback remains a huge advantage to both parties.  Our client, Doctor Halterman of Southern Adventist University, has recognized this call for a framework that satisfies these needs.

	Programming tests and quizzes today often are given through the paper and pencil medium. This medium is not desirable as it does not allow for immediate feedback, which can frustrate both teachers and students. Students must wait for results and teachers must invest large amounts of time in grading students' answers. Another problem that arises due to this medium is the inability to type and format code. For students in a high-pressure environment such as a quiz or test there is not always time to neatly write code. Erasing or revising written code can further reduce readability. As a result, teachers and students are forced to read and work with messy answers. Finally, this medium does not allow for the possibility of compiling and running code. The ability to execute code allows students to fix small mistakes such as syntax errors and other problems that do not necessarily have any implication on the testing content.

	Online textbooks and other related learning environments suffer from similar problems (i.e., no immediate feedback, and readability issues), with the addition of the need to switch to a different application completely. Using these media require students to leave the browser, have a compiler or IDE installed, and copy or write code snippets in order to try concepts out on their own.
	

	These issues have led to the use and/or creation~\cite{courseMarker, dougMoodle, dougMoodleInterpretive} of two separate products: Learning Management Systems (LMS), and automated evaluation systems. LMS' like Moodle~\cite{moodle, dougMoodle, dougMoodleInterpretive} and Edomodo~\cite{edmodo, edmodo1, edmodo2} both provide teachers with a web interface for building instructional articles, exams, and quizzes and provide students with a common interface to access those activities. An LMS can also serve as a secure environment for testing and quizzes, and offer features such as multiple attempts, instant feedback, and question randomization. LMSs also have the capability to integrate add-ons which can provide the ability to compile and run code written by students, such as Moodle's plug-in feature~\cite{moodlePlugins}.

	Modifying or building quizzes and exams via an LMS requires multiple, repetitive steps that are more time consuming than editing a simple flat file. In addition, current LMS do not not give statistics on test and quiz properties, such as time spent per question. Existing add-ons for an LMS such as Moodle do not address this issue, and the compiler features do not contain some of the requested properties mentioned above (e.g., immediate evaluation and feedback).

	As for automatic evaluation systems, students can type and execute code, run tests, and receive immediate feedback. However, widely used solutions simply support assignments and code challenges, excluding examination environments and general practice environments.

	Therefore, based on our client's interests stated above, we believe no application exists which allows users to quickly build learning modules, exams, or quizzes via structured text files for students to use, that also give them the ability to write and run code and receive immediate feedback on submitted work.

	This project addresses these issues by creating a framework which provides a medium with the following goals:
\begin{itemize}
\item Allow professors to quickly build activities via simple flat files
\item Allow students to be able to type code in an environment that provides syntax highlighting
\item Allow students to compile and run code in a controlled environment
\item Allow for the ability to build activities specifically for an examination environment 
\item Allow students and professors to get immediate statistics (i.e., time spent per question/exam, student written code, and data for each test case run for teachers, and grading data for both)
\item Allow for the support of multiple programming languages
\end{itemize}


	Given this framework, professors can quickly and easily create quizzes, tests, and other learning environments, and students can edit and compile code and receive immediate feedback. We believe this will help professors limit time spent on building and grading tests and quizzes, and help students focus more on actual test content, rather than on formatting and cleaning up answers.

	For the remainder of our paper, we first cover the background in Chapter~\ref{sec:Background}. Chapter~\ref{sec:Project} and Chapter~\ref{sec:Evaluation} will cover our project architecture/design and testing respectively. Finally, Chapter~\ref{sec:Conclusion} presents our conclusion on the research found and our future work on the Master's Project.


\chapter{Background}
\label{sec:Background}
	Currently, there are virtually no systems in widespread use that provide automated feedback tailored for programming. However there are a limited number of automated grading systems (as shown in a survey paper by Kyrsti M et al.~\cite{survey}) and to move from niche to mainstream it will be necessary for these automated grading systems to exist and incorporated in LMSs because schools are moving to such systems as a standard platform on their campus~\cite{edmodo1}.
	
	The most popular systems in use in academics today are Learning Management Systems (LMS). These products provide teachers and students with a familiar digital based medium in which to access assignments, grades, and even tests and quizzes. These systems also allow for modules to be added in order to provide new features developed by third parties. In addition to these systems there exists more focused projects that attempt to provide similar services, specifically tailored toward programming in education. In this section we explore the abilities of popular LMSs, and open source programming evaluation systems which attempt to provide similar functionality.
 

\begin{table}[h]
\begin{tabular}{|p{7cm}|r|r|}
\hline
\textbf{LMS} & \textbf{Customers} & \textbf{Users} \hspace{4cm}	\\
\hline
Edmodo & 350,000 & 58,000,000  \\ \hline
Moodle & 70,569 & 89,237,532 \\ \hline
SuccessFactors & 6,000 & 45,000,000 \\ \hline
Blackboard & 16,000 & 24,000,000  \\ \hline
Cornerstone & 2,600 & 27,200,000 \\ \hline
SkillSoft & 6,700 & 19,000,000  \\ \hline
Instructure & 3,000 & 20,000,000 \\ \hline
Saba Software & 2,000 & 33,000,000 \\ \hline
Schoology & 2,000 & 20,000,000 \\ \hline
Litmos & 3,500 & 6,000,000 \\ \hline
Latitude Learning & 14,299 & 4,218,001 \\ \hline
Edsby & 10,800 & 1,910,000 \\ \hline
Collaborize Classroom & 57,600 & 420,000 \\ \hline
Brightspace & 2,000 & 15,000,000 \\ \hline
WizIQ Inc & 5,000 & 500,000 \\ \hline
NEO LMS & 8,700 & 970,000 \\ \hline
Absorb LMS & 720 & 6,200,000 \\ \hline
TalentLMS & 3,132 & 1,552,000 \\ \hline
Educadium & 9,300 & 75,000 \\ \hline
DigitalChalk & 4,090 & 908,440 \\ \hline
\end{tabular}
\caption{\small The top 20 most popular LMS packages, measured by a combination of their total number of customers, active users, and online presence (customer and user data shown in figure above) as of November 2017~\cite{Capterra}.}
\label{tab:LMSPackages}
\end{table}

Our background analysis is broken into two different sections, Learning Management Systems in Section~\ref{sec:LMS} and automated programming evaluation systems in section ~\ref{sec:APES}, taking into consideration their strengths and weaknesses as follows:

\begin{itemize}
\item Test and quiz building
\item Code formatting and compiling ability
\item Answer evaluation and feedback
\end{itemize}

These three topics will provide a perspective on how the system functions and how well it addresses our goals.

\subsection{Learning Management Systems}
\label{sec:LMS}
In the subsection below we evaluate the LMS based systems selected for analysis. Table~\ref{tab:LMSPackages} shows the most popular LMS software packages~\cite{Capterra}, from which we select two mature LMS platforms (Moodle~\cite{moodle, dougMoodle, dougMoodleInterpretive} and Edmodo~\cite{edmodo, edmodo1, edmodo2}) for analysis along with two LMS modules for Moodle called the Virtual Programming Lab~\cite{vpl}, and Javabrat~\cite{javaBrat}.


\subsubsection{Analysis: Moodle}
	Moodle~\cite{moodle, dougMoodle, dougMoodleInterpretive} is a free, open-source LMS for producing modular internet-based courses that support a modern social constructionist pedagogy. Because it is open-source, Moodle has a large community contributing many modules that extend its feature set.

\paragraph{Test and quiz building}
Moodle has a quiz activity module which allows the teacher to design and build quizzes consisting of a large variety of question types, including multiple choice, true-false, and short answer questions. To make a new quiz/test assignment in Moodle, the teacher may either import a quiz from a range of different flat file formats~\cite{moodleImport}, or manually add a quiz module to the course. When creating a quiz manually, the teacher must first create the quiz activity. To add a new question to the quiz's question bank, the teacher must click 'Add' and then '+ a new question'. From the next screen, the teacher must choose the specific question type. Once the question form is filled in with the question and any necessary options are added the question can be saved and added to a question bank shown in Figure~\ref{fig:MoodleQuizCreate}. The teacher must then add questions to the quiz from the question bank. 

With all of the steps in manually building quizzes, teachers must invest time in adding items to question banks on top of formulating the actual question content. While importing questions from flat files is useful in avoiding this overhead, the type of questions available to be asked are still limited by Moodle's standard questions types~\cite{moodleQTypes}.

\begin{figure}
  \centering
  \includegraphics[scale=.95]{fig/moodle_quiz_create.png}
  \caption{\small A simple Moodle quiz module.}
  \label{fig:MoodleQuizCreate}
\end{figure}

\paragraph{Code formatting and compiling}
Moodle quiz activity modules can accept multiple choice and text input. However, these input methods do not allow students to properly format code, or show syntax highlighting. Similarly, students do not have the ability to execute code and see results produced by the compiler which limits the types of questions that can be presented (e.g., we cannot ask a student to compile and debug based on the compiler output.) 

\paragraph{Answer evaluation and feedback}
Moodle quiz activity modules provide the ability to evaluate and grade answers immediately and give students feedback, which is another improvement on the pencil and paper medium. Unfortunately, due to question limitations mentioned above, there is no ability to run test cases on student defined code and grade submissions automatically.


\subsubsection{Analysis: Edmodo}
	Edmodo~\cite{edmodo} is a free social learning platform for schools. It provides teachers and students with a secure way to share classroom materials, and access homework, grades.

\paragraph{Test and quiz building}
Edmodo has a quiz activity that allows teachers to create online quizzes for students and receive instant feedback on grade results. Unlike Moodle, all Quizzes and Quiz questions must be created within Edmodo and cannot be imported~\cite{edmodoQuiz}. The steps to create quizzes are comparable to Moodle's, and incurs similar overhead. To create a quiz in Edmodo, a teacher must create a quiz activity, and manually select a question type. Once the question form is filled in with all necessary information the question can be saved and added to a question bank.

Similarly with Moodle, Edmodo is limited by the building overhead and the types of questions available to be asked (Multiple choice, True/False, Short Answer, and Fill in the blank).

\paragraph{Code formatting and compiling}
Again, very similar to Moodle, Edmodo quiz activities can accept multiple choice and text input, which allow students to type code. However, these input methods do not allow students to properly format code, or show syntax highlighting. Nor do students have the ability to execute code and see results produced by the compiler which limits the types of questions that can be presented, similar to Moodle.

\paragraph{Answer evaluation and feedback}
Edmodo quiz activities provide the ability to get feedback immediately. Due to question limitations mentioned above however, there is no ability to run test cases on student's code and grade submissions automatically in that manner.


\subsubsection{Analysis: The Virtual Programming Lab}
	The Virtual Programming Lab (VPL)~\cite{vpl} is an activity module for Moodle that manages programming assignments and whose main features are:
\begin{itemize}
\item Students can edit program source code in the browser
\item Students can run programs interactively in the browser
\item Students and teachers can run tests to review the programs
\end{itemize}


\paragraph{Test and quiz building}
This activity module for Moodle is unfortunately tailored for programming assignments. Because of this, it is not possible to create an assignment in a test or quiz format. 

\begin{figure}
  \centering
  \includegraphics[scale=.50]{fig/vpl.jpg}
  \caption{\small A simple VPL module programming assignment~\cite{vpl}.}
  \label{fig:vplAssignment}
\end{figure}

\paragraph{Code formatting and compiling}
VPL allows files to be edited from the browser using the code editor component, as shown in Figure~\ref{fig:vplAssignment}. It also has the ability to supply initial skeleton code. This added functionality allows students to run and evaluate programs without requiring them to install compilers or IDEs. The code editor component provides formatting as well as syntax highlighting, which is useful for students both reading and writing code.

\paragraph{Answer evaluation and feedback}
VPL allows for scripts to be set to evaluate every submission, which enables students to receive feedback on their code based on test cases. However, this takes more time to set up than simply defining test cases and is geared more towards programming assignments instead of quizzes and tests.


\subsubsection{Analysis: Javabrat}
	Javabrat~\cite{javaBrat} is a web-based grader useful for students learning Java and Scala languages. It is primarily in the form of a Moodle plugin that facilitates instructors to grade Java assignments.
\begin{itemize}
\item Students can edit program source code in the browser
\item Students and teachers can add problems to be solved
\item Students and teachers can run tests to review the programs and automatically grade them
\end{itemize}


\paragraph{Test and quiz building}
Similar to VPL, Javabrat is tailored towards programming assignments. Because of this, it is not possible to create an assignment in a test or quiz format. 

\begin{figure}
  \centering
  \includegraphics[width=\textwidth]{fig/javaBrat.png}
  \caption{\small A basic Javabrat assignment example~\cite{javaBrat}.}
  \label{fig:javaBratAssignment}
\end{figure}

\paragraph{Code formatting and compiling}
Javabrat allows for code to be typed directly into the brower, but does not support syntax highlighting, as shown in Figure~\ref{fig:javaBratAssignment}. Similar to VPL, Javabrat has the ability to provide an initial skeleton code for students to build on. This setup allows students to type out code and evaluate results without having to leave the browser.

\paragraph{Answer evaluation and feedback}
With Javabrat, program results are evaluated immediately. When a student completes a solution, it can be evaluated in the browser by running pre-defined test cases. Once the evaluation is done, Javabrat sends back an HTML report file generated by the back-end as shown in the lower part of Figure~\ref{fig:javaBratAssignment}.


\subsection{Automated Programming Evaluation Systems}
\label{sec:APES}
The following subsection will evaluate the automated programming evaluation systems selected for analysis. From our research we select three popular automated programming evaluation systems: Web-CAT~\cite{webcatip, webcat2}, Turing's Craft~\cite{turingsCraft}, and CourseMarker~\cite{courseMarker, foxley}.

\subsubsection{Analysis: Web-based Center for Automated Testing}
	The Web-based Center for Automated Testing (Web-CAT)~\cite{webcatip} is a flexible automated grading system designed to process computer programming assignments. Its main features are:
\begin{itemize}
\item Provide immediate feedback based on test-cases and code coverage
\item Enable flexibility and extensibility by way of a plug-in based architecture
\item Provide a secure and portable product
\end{itemize}


\paragraph{Test and quiz building}
Similar to VPL, Web-CAT is tailored for programming assignments. Because of this, it is not possible to create an assignment in a test or quiz format. 

\begin{figure}
  \centering
  \includegraphics[scale=.50]{fig/web-cat.png}
  \caption{\small A simple Web-CAT module assignment~\cite{webcat}.}
  \label{fig:webcatAssignment}
\end{figure}

\paragraph{Code formatting and compiling}
Web-CAT does not allow files to be edited from the browser. It does however have the ability to supply initial skeleton code. Without this functionality, students must use their own compiler or IDE. In Web-CAT, students are required to upload their source code to the correct assignment as shown in Figure~\ref{fig:webcatAssignment}.

\paragraph{Answer evaluation and feedback}
Web-CAT uses a composite scoring system, which enables students to receive feedback on their code based on their code correctness, test completeness, and test validity. These three measures, taken as percentages, are then used to form a composite score~\cite{webcatScore}. This formula ensures that no aspect of the approach can be ignored, or the student's score will suffer. Code correctness is based solely on student written tests, unlike the other two scores. For test completeness, instructors must choose what level of coverage should be used for testing completeness, and for test validity, instructors must set specific reference tests to ensure the student's tests are accurate.


\subsubsection{Analysis: Turing's Craft CodeLabs}
	Turing's Craft CodeLab~\cite{turingsCraft} is the commercial equivalent to WebToTeach~\cite{turingsCraftwtt}, developed to provide fully interactive, hands-on practice environments for learning computer programming. The supported topics start with the imperative programming core of Python, C, C++, and Java and then go on to address procedural and object-oriented programming. The exercises are targeted at a typical CS1 syllabus.

Its main features are:
\begin{itemize}
\item Students can edit and run source code in the browser
\item Provide immediate feedback based on test-cases in the browser
\item Provide teachers with a selection of exercises ranging from short and focused to more complicated problems intended to be used as teaching aids
\end{itemize}

\paragraph{Test and quiz building}
Because Turing's Craft is targeted towards programming assignments and challenges, it does not support quiz or exam formats.

\paragraph{Code formatting and compiling}
The Turing's Craft engine is hosted and supported by Turing's Craft and the CodeLab runs in the browser, so students are able to write and run code directly in the browser. Unfortunately, it does not support syntax highlighting.

\paragraph{Answer evaluation and feedback}
Turing's Craft CodeLab provides student with immediate feedback on submitted solutions. With each submission, the instructor's roster is automatically updated regarding solution correctness. As a teaching aid, Turing's Craft research found that CodeLab works best when required as 5-10\% of a student's grade.


\subsubsection{Analysis: CourseMarker}
	CourseMarker~\cite{courseMarker} was developed for the assessment of Java programming skills of CS students. CourseMarker has been tailored to  student needs by teacher and student suggestions, building off of an older automated assessment tool called "Ceilidh"~\cite{foxley}. Its main feature is to allow students and teachers to run tests and receive in-depth metrics and assessments immediately. CourseMarker must be installed instead of accessing it through a browser.

\paragraph{Test and quiz building}
Similar to VPL~\cite{vpl} and Web-CAT~\cite{webcat2, webcatip}, CourseMarker is tailored for programming assignments. Because of this, it is not possible to create an assignment in a test or quiz format.

\paragraph{Code formatting and compiling}
CourseMarker does not allow files to be edited from the client, but it does have the ability to supply initial skeleton project files. Without this functionality, students must use their own compiler or IDE.

\paragraph{Answer evaluation and feedback}
CourseMarker provides grading results immediately once code solutions are submitted. Students receive feedback on typography, scalability, and test cases. With typography, CourseMarker checks the program layout, indentation, and usage of comments. Regarding scalability, CourseMarker attempts to use different sized datasets when possible. Lastly, CourseMarker evaluations solutions by checking for exercise dependent features defined by the teacher. \\

\begin{table}
\tiny
\begin{tabular}{| l | c | c | c | c |}
\hline
\textbf{ } & \textbf{Exam environment} & \textbf{Code formatting} & \textbf{Immediate feedback} & \textbf{Multi-lang support} \\
\hline
  Moodle & X &  & X &  \\ \hline
  Edmodo & X &  & X &  \\ \hline
  VPL &  & X & X & X \\ \hline
  Javabrat &  &  & X &  \\ \hline
  Web-CAT &  &  & X & X \\ \hline
  Turing's Craft &  &  & X & X \\ \hline
  CourseMarker &  &  & X & X \\ \hline
\end{tabular}
\caption{\small A visualization of how many of this project's goals are met by each solution evaluated above.}
\label{tab:solutionEvals}
\end{table}

In summary, as seen in table~\ref{tab:solutionEvals}, each of the popular solutions reviewed met different parts of our client's overall goals but none met all of them. While a few such as Web-CAT~\cite{webcat2, webcatip} came close, many of the widely used solutions do not offer everything this project is attempting to accomplish.


\chapter{Project Implementation}
\label{sec:Project}
Our system creates a RESTful web service that provides the following services:
\begin{itemize}
\item A quiz/test module generated from a quick and easy flat file which defines activity settings, question and answer values, grade-point values, and estimated difficulty levels, in a medium that allows for code formatting
\item An interface to compile, execute code and automatically evaluate output against test cases
\end{itemize}

Seven external libaries were utilized in the development of this system, and the system contains over 2,000 lines of non-library code. In this chapter, we discuss the development approach, task delineation, system process workflow, and final deliverables for this system.


\section{Development Approach}
\label{sec:devApproach}
We implemented our system on an Ubuntu unix VM, using Node.js and jQuery in order to take advantage of modules such as ExpressJS~\cite{express} which provided robust sets of frameworks and features for web applications. These requirements are detailed in Appendix~\ref{app:requirements}. While some synchronous functions were used for purposes such as writing data to file and handling compiler input/output, we generally utilized Node's asynchronous nature and Javascript's Promise objects to minimize blocking. We also made use of the Model-View-Controller design pattern in order to organize our code and system structure. 

Since this system's intent focused on an exam environment, we considered security a high priority. With regards to client-side execution, since all scripts sent from the server to be executed client-side exist in a specific namespace, interaction through the console window cannot reach the code or any variables during normal execution and therefore data integrity is ensured. The design of the system ensures that no exam solutions ever touch the client side and that the system evaluates all submissions on the servver. Section~\ref{sec:Security} covers security in more detail.

We have utilized two internal Node.js modules (fs, child\_process), five external modules, and two jQuery plugins in order to speed up development. All modules and plugins were licensed under the MIT license~\cite{mit}, with one module licensed under the BSD 2-Clause~\cite{bsd}. Table~\ref{tab:libraries} lists each of these modules and their purpose in the system.

\begin{table}
\tiny
\begin{tabular}{| l | l |}
\hline
\textbf{Module/Plugin} & \textbf{Purpose} \\
\hline
  CodeMirror & A text editor implemented in JavaScript used for styling and editing code \\ \hline
  Express & An MVC-based web framework for Node.js used to handle client/server interactions\\ \hline
  Body-parser & Node.js body parsing middleware used to parse JSON files\\ \hline
  Connect-busboy & Connect middleware for busboy used to handle file uploads\\ \hline
  Deasync &  JavaScript wrappers of Node event loops used to keep from blocking threads\\ \hline
  Fs & Simple wrappers of standard POSIX functions used to provide File I/O\\ \hline
  Uglify-js & JavaScript parser/compressor used to minify scripts \\ \hline
  directory-tree & Creates a JavaScript object used for displaying a directory tree. \\ \hline
  child\_process & A module which spawns a shell and executes commands within that shell\\ \hline  
\end{tabular}
\caption{\small Node.js modules and JS plugins used in this system}
\label{tab:libraries}
\end{table}


\section{Task Delineation}
\label{sec:taskDelineation}
Development was broken up into six major categories for testing and development. Table~\ref{tab:taskDelineation} shows an overview of major tasks, including work hours. 

\begin{table}
\begin{tabular}{| l | c |}
\hline
\textbf{Task} & \textbf{Work Hours} \\
\hline
  System design and architecture & 70 \\ \hline
  Front-end Examination module & 85\\ \hline
  Server-side Examination evaluation & 80\\ \hline
  Administration module &80  \\ \hline
  Server-side code execution and evaluation & 120 \\ \hline
  Final changes, bug fixes, testing, and documentation & 90 \\ \hline
\textbf{Project Completion} & 525  \\ \hline
\end{tabular}
\caption{\small A visualization of this project's goals and rough estimate of hours taken to complete}
\label{tab:taskDelineation}
\end{table}



\section{Architecture}
\label{sec:Architecture}
Our system's architecture was distributed across both the server and client. The interactions are described in this section. The system design and interactions are illustrated in Figure~\ref{fig:archDiagram}, and the complete process flow is detailed in Section~\ref{sec:sysWorkflow}.



\begin{figure}
  \centering
  \includegraphics[scale=.80]{fig/ProcessFlow.png}
  \caption{\small A diagram illustrating the high-level process flow.}
  \label{fig:archDiagram}
\end{figure}



\subsection{Server-side}
Node and the ExpressJS~\cite{express} framework together which handled routing REST calls acted as the Controller in this Model-View-Controller (MVC) design. The majority of the Model's business logic provided functionality such as compiling code, building and evaluating exams, and formatting/serving data such as the core client facing business logic (minified~\cite{minify} and sent to the client to execute).


\subsection{Client-side}
The View component utilized HTML and jQuery to populate the exam and provide all View related functionality. An AJAX request made by the client returns scripts to execute which initiate the rest of the exam. The Controller component processes these AJAX calls throughout the exam period, manipulating the Model and View accordingly. An example of this interaction is a simple compile, where the client requests the server to compile code and the response updates the View.


\subsection{Datafile Design}
We use a simple flat file to generate exam content, following the Javascript Object Notation (JSON) format. The flat data file is uploaded to the server, parsed to form a JSON object, and stored. This JSON object has the following structure:\\
\begingroup
\begin{lstlisting}[label={fig:dataStructure},language=json,firstnumber=0]
{
  "0": {
      "questionType": "code",
      "language": "c++",
      "problem": "Write a C++ function called print that accepts an integer and prints it out.",
      "skeleton": ["#include <iostream>\nusing namespace std;\n\n%READ_IN%\nint main() {\n   string x = \"printed\";\n   print(x);\n}\n", "%READ_IN%", "//Write your code here\nvoid print(string x){\n   cout << x;\n}"],
      "input": ["1"],
      "output": ["1"],
      "points": ["15"],
      "difficulty": ".5"
    },
    "prop": {
        "closeDate": "6-15-2018",
        "closeTime": "13:30",
        "allowMultiple": "false",
        "time": "15",
        "warn": ["10", "5"],
        "whitelist": ["0421291", "0421292"],
        "access": "true"
    }
}
\end{lstlisting}
\renewcommand\lstlistingname{Data-structure}
\captionof{lstlisting}{Example JSON datafile object structure}
\endgroup


Within this structure, "questionType" can hold one of two values, "code" or "mchoice". The value of "questionType" defines whether the question is a coding question or a multiple choice question with randomized answers. I.e., questionType determines what result is produced by a given code snippet. Each field serves the same function for either type of question, excluding the "input" and "output" fields. The "language" field defines the programming language used, "problem" defines the question, "skeleton" defines the initial skeleton code, "points" provides the point value, and "difficulty" defines a difficulty level which translates into minutes (value * 10 = suggested time) used to produce suggested completion times. The "skeleton" field also includes an optional usage which holds a "hidden skeleton", a replace token, and a "visible skeleton" for questions that may want to hide some details from the user, instead of just the standard skeleton. We have shown this optional format in line five of the example data structure above.

For coding questions, the "input" field utilizes an array that holds different test-cases. This input value can represent any input a keyboard can provide in a command-line setting. Each element in the "input" array is a different test case that will be run against student's submissions. The "output" field represents the expected output produced after running each test case using inputs (if any) from the "input" field.

Multiple choice questions on the other hand, use the "input" field differently. With this question type, the "problem" field describes the overall instructions and the "skeleton" field describes the corresponding code snippet, identical to coding questions (except it is non-editable). However, "input" utilizes a multi-dimensional array, with each internal array holding a sub-question and an array of options from which students can choose. The "output" field dictates the indices of the correct options.

The structure also contains a required "prop" field which can hold both required and optional information about the activity. The required fields "time" and "warn" are used to control how long an exam will last before automatically submitting. The optional fields "closeDate" and "closeTime" denote when an exam will restrict access, "allowMultiple" defines whether students can take an exam multiple times, and "whitelist" and "access" control access to the exam. Note that the server's internal time-zone may not match your own, which can cause date specific properties to work unexpectedly.

Since our system utilizes the JSON format, code cannot contain line breaks. Instead, line breaks must be replaced with a Line Feed ("\verb|\n|") which can cause the creation of datafiles to become an arduous effort. To address this issue, skeleton code wrapped with predefined tokens (start: "\_<<\_'', end: "\_>>\_") can instead be used with the server automatically converting line breaks to Line Feeds. An example of this format can be seen in Figure~\ref{fig:uploadAdmin}.

\subsection{Administrative Portal}
Our system includes an administrative side where both datafiles and test results can be seen. These files are organized by course and exam. For datafiles, authenticated users can upload datafiles, edit them, and remove them. Result files can also be viewed, but not uploaded, edited, or removed. However, exams have the option to generate a report by walking through each of the result files and displaying it in a printable dialog window. These interfaces can be found in Appendix~\ref{app:screenshots} as Figures~\ref{fig:treeAdmin}, ~\ref{fig:reportAdmin}, ~\ref{fig:printAdmin}.



\subsection{Testing Security and Integrity}
\label{sec:Security}
Security is important for any sort of testing module. Since the AJAX call's response calls the script containing the necessary logic for the client, it exists only in that namespace. Because of this namespace, interaction through the console window cannot reach the code or any variables during normal execution. In order to manipulate anything, a malicious user would have to use browser developer tools such as breakpoints, and even then it would not accomplish anything beyond using their allotted time and distort the presentation of questions. No exam solutions ever touch the client side and all submissions are evaluated server-side.

As an added layer of protection, any variables created are local variables and locked using the Object.freeze()~\cite{ojbFreeze} Javascript method. This method prevents malicious users from adding new properties to the frozen object; prevents existing properties from being removed; and prevents existing properties, or their enumerability, configurability, or writability, from changing. In essence, the object is effectively immutable. This method is mainly used to lock the answers given from the multiple-choice part of the exam once students begin the programming section in order to keep students from changing their answers in the multiple-choice section.

Regarding server-side code execution, students could attempt to take advantage of this system's architecture and submit malicious code. We decided on using both Unix filesystem privileges as well as process monitoring to help protect the system.  

Lastly, we suggest that a proctor be used in addition to these features as we have no control over student collaboration and communication. Similarly, products such as Respondus' LockDown Browser~\cite{lockdownbrowser} can also provide additional security by way of a custom browser that locks down the testing environment. Together, these layers of security limit the ways in which students were able to collaborate or cheat in these tests.



\section{System Process Workflow}
\label{sec:sysWorkflow}
The system utilizes a Request-Response model, defined as a series of HTTP requests and responses between the client and server. This process is broken up into four main stages: 
\begin{itemize}
\item Module selection
\item Module execution
\item Code compilation
\item Final evaluation
\end{itemize}
We have shown these interactions in Figure~\ref{fig:archDiagram}.

\subsection{Module Selection}
This first step initializes the system. For an exam or quiz, the client will first post a request to the server's /getModuleSelector endpoint. This endpoint minifies and sends a script that provides the user with a dialog window and some additional logic that allows the user to request the specific quiz or exam. This information includes student ID, course ID, and exam ID.

Once submitted, the server parses this information from the requested JSON datafile, inserts those values into Hyper Text Markup Language (HTML) templates, and sends the HTML code along with a minified script that holds the client facing business logic. This templating allows the system to build and serve a module quickly.


\begin{figure}
  \centering
  \includegraphics[scale=.50]{fig/test_example.PNG}
  \caption{\small A screenshot displaying an exam coding question built from a datafile.}
  \label{fig:examImage}
\end{figure}


\subsection{Module Execution}
At this point in the system's workflow, the client will now have the full code which will execute and append the HTML code to the specified Document Object Model (DOM) element. Once the student enters the exam and selects 'Begin', the exam and timers will start. When the exam's timer elapses, the exam will then automatically submit all available data to the server for evaluation. Figure~\ref{fig:examImage} shows an example exam built through this process.


\subsection{Code Compilation}
\label{ssec:compile}
Once the exam or learning module has started, users can compile and run code. The response will send the language (currently we support C++11 and Python 3.X), code, and any inputs separated by newlines to be read in as standard input. It should be noted that for the quiz and exam modules, compilation can only occur once a student has completed the multiple-choice section. The system will record and freeze all answers from the multiple choice section once any compilation is initiated.

When compiling, the server creates a temporary directory by concatenating the current datetime and a random integer ranging from 0 to 9999. Once the server creates the directory, the code compiles (if necessary) and exectues, piping in the inputs (if any) as standard input. The response returned will hold the generated output or error.


\subsection{Final Evaluation}
This section of execution marks the final part in the system's workflow and only applies to the quiz and exam module. Two conditions can initiate this process: reaching the exam's time limit or the user clicking the submit button. Once the server responds, a modal window displays the calculated score.

The server processes the submitted exam data and writes the results to a result file (identified by the student's ID). For grading multiple-choice questions, comparing the submitted answers' indices with values from the datafile indicate correctness. Similarly for code questions, comparing the outputs of each testcase to the expected outputs from the datafile determine correctness. For coding problems, all test cases must pass to award points.


\section{Summary}
This chapter described the project solution's architecture, design, and process flow. It detailed the business logic and syntax necessary to understand the overall application and create well-formed datafiles and therefore exams.


\chapter{Testing and Evaluation Results}
\label{sec:Evaluation}
This chapter reviews the software requirements, the evaluation process to test those requirements, and the results of that evaluation. Our evaluation relies on a simple pass/fail process. 

We verified component requirements through user testing, integration testing and visual inspections of the system and show a check sheet that lists the requirements completion status. Figure~\ref{fig:EvaluationChecklist} sums up the results of the testing performed.

\section{Testing}
Our testing approach consisted of both manual inspections and automated test cases via Selenium~\cite{selenium}. Of the more popular browsers, we selected Chrome~\cite{chrome}, Safari~\cite{safari}, and Firefox~\cite{firefox} as our main platforms to test. With Selenium's ability to automate browser interactions, we were able to consistently verify both specific components and end to end processes successfully. These interactions included resource loading, error reporting, and code compilation.

We were able to consistently verify that modules were created and served as expected using Selenium to automate interacting with the activity. As Selenium walked through different sections of the test activity by simulating keystrokes and clicks, the question and answer values were verified as visible and error reporting was displayed when expected. However this only proved the expected elements and values existed in the DOM and were correct, therefore manual visual inspections were utilized to further confirm the appearance and validity of the generated modules.

With regards to compilation, Selenium was also used to test the functionality and correctness. With this approach, we were able to automatically select specific questions, enter predefined code and submit for execution. Results returned were then read and compared to expected results for verification. Compilation was also tested via HTTP requests posted directly to the \/compile endpoint in order to test that functionality directly.

Full integration testing was used in order to verify exam evaluation and reporting. This testing again utilized Selenium's automation to simulate the taking of an exam and comparing the final score received to the expected score. Observation of these processes assisted in verifying the results.


Coordinating with Dr. Halterman, Southern Adventist University's CPTR-124 students also used the system for both practice and actual exams. Through this user testing, the system evolved based on feedback and performance observations.


The following summary describes the testing results and feedback received:
\begin{itemize}
\item Practice exams: The feedback received was positive enough to warrant testing the system in an actual exam. No problems encountered.
\item Actual exams: The feedback received was positive and very useful as users' experience revealed a few bugs such as runaway processes, and quality of life adjustments such as warnings and extra datafile customization.
\end{itemize}

\begin{figure}[htb]
  \begin{tabular}{|p{\linewidth}|}\hline
  \begin{description}
    \item[\checkmark{\hspace{0.5cm}}] Modules are served correctly and consistently across platforms.
    \item[\checkmark{\hspace{0.5cm}}] Questions, sub-questions, and answers are all generated and displayed correctly.
    \item[\checkmark{\hspace{0.5cm}}] Server receives, compiles, and runs submitted code and reports results correctly.
    \item[\checkmark{\hspace{0.5cm}}] Submissions are received and results are reported successfully.
    \item[\checkmark{\hspace{0.5cm}}] All question specific test-cases run and evaluate successfully.
    \item[\checkmark{\hspace{0.5cm}}] Exam results are immediately reported to user and stored successfully.
  \end{description} \\
  \hline
  \end{tabular}
  \caption{\small Evaluation checklist.}
  \label{fig:EvaluationChecklist}
\end{figure}


\section{Evaluation}
User and integration testing and feedback paired with visual inspections show that our solution meets the requirements and goals set for this project. From creating the module datafile to grading submissions and waiting for test results, time spent was noticeably reduced. After implementing user testing feedback, performance evaluation showed that the system was able to handle loads with a negligable performance hit. 


During the first testing session, an interaction between Node's child\_process.exec timeout functionality and infinite loops caused timeouts to fail and the CPU usage to rise to 99\% as these programs kept running. After the cause for these run-away processes was found, we implemented a monitoring solution that kept these programs from continuing infinitely. The peak performance impact after the fix was implemented is shown in Figure~\ref{fig:performance}.




\section{Final Deliverables}
Once the project was completed, we delivered the following materials to the client:
\begin{itemize}
\item \href{https://github.com/chrisban/cs-eval-fw}{Application source code}
\item Access to project source control
\item Documentation on setup, installation, and datafile formatting
\item Final project report
\end{itemize}


\begin{figure}
  \centering
  \includegraphics[scale=.48]{fig/performance.png}
  \caption{\small Graphs detailing the peak performance impact during user testing.}
  \label{fig:performance}
\end{figure}


\chapter{Conclusion}
\label{sec:Conclusion}
Our client wanted a well-structured and implemented framework for writing and running code in both testing and learning environments. The current solutions reviewed in Table~\ref{tab:solutionEvals} cover most of the goals of this project in some form, however we did not find one that met all of the goals according to our client's specifications stated in Chapter~\ref{sec:Introduction}.

In this thesis, we implemented a separate framework to remedy the above issues. Based on our research and testing results, our solution will help both students and teachers in many aspects of programming examinations as described in Chapter~\ref{sec:Introduction}. The opportunity to type and format code, along with the ability to compile and execute code is helpful to students and allows them to focus on actual test content and non-trivial syntax or formatting issues. This solution also can help to dramatically cut down on the time it takes to build and grade exams due to the simple flat file format and automated grading. Therefore this solution will be of great benefit to the student as well as the professor, saving them both time and effort.

For future work we suggest adding support for other learning environment modules such as online textbooks, and a more intuitive way of adding support for new languages. We also suggest further research into work done by Singh et al.~\cite{corrections} in order to create and provide more meaningful feedback such as potential corrections to failed test cases. In regards to furthering security, we suggest an approach similar to the algorithm created by Chen et al.~\cite{maliciousRegex}, which used sandboxing and regular expressions to filter out malicious code.


%% Start the correct formatting for the appendices
\appendix
\appendixpage
\addappheadtotoc
\chapter{Environment Requirements}
\label{app:requirements}
This project was developed on an Ubuntu 16.04.2 x64 Linux server.
The following packages must be installed:
\begin{itemize}
\item Node (v8.9.1)
\item Python (v3.5.2)
\item g++-5
\item Forever (Optional Node Package Manager (npm) package, used for ensuring that a given script runs continuously)
\end{itemize}
\chapter{System Screenshots}
\label{app:screenshots}
\begin{figure}
  \centering
  \includegraphics[scale=.45]{fig/startExam.png}
  \caption{\small The start screen of an exam.}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[scale=.40]{fig/sectionExam.png}
  \caption{\small After using the compiler functionality, answers selected in the previous multiple-choice section cannot be changed.}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[scale=.40]{fig/submittedExam.png}
  \caption{\small Exam results reported back. In this case, the datafile specified not to allow multiple attempts, but did allow access. This is reported to the user in case of misunderstanding.}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[scale=.40]{fig/uploadAdmin.png}
  \caption{\small The upload interface, showing the alternate datafile format which supports newlines.}
  \label{fig:uploadAdmin}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[scale=.40]{fig/treeAdmin.png}
  \caption{\small A tree representing the file structure of both the exam datafiles and results, sorted by class and exam.}
  \label{fig:treeAdmin}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[scale=.40]{fig/reportAdmin.png}
  \caption{\small A styled report of all submitted results for the selected exam.}
  \label{fig:reportAdmin}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[scale=.40]{fig/printAdmin.png}
  \caption{\small A pintable exam report of all submitted results for the selected exam.}
  \label{fig:printAdmin}
\end{figure}

\backmatter

%% Bibliography goes here
%% BibTeX is your friend
%% \nocite{*}
\bibliographystyle{IEEEtran}
\bibliography{./thesis}

%% Index go here (if you have one)
\end{document}

\endinput
