CSE 1729 Mimir Testing Scripts
This repository contains testing scripts to automate grading of CSE 1729 labs and problem sets.
Setup
To setup a new lab or problem set:
- Prepare a
referenceSolution.scm
file and put it at the same directory level as the files in this resository. - Zip up the contents of this folder (including the
referenceSolution.scm
file to theFiles (optional)
section of the test case under Advanced Options. - Add the following to the Bash Script section with your customizations:
export testFunction="fact" export testArguments="5" export testAssertion="(equal? student reference)" ./genericTester.sh
- Uncheck Show Expected Output to Student, Show Bash Script to Student, and Show Bash Script OUTPUT to Student. These typically show extraneous information that confuses students.
Options
The genericTester.sh
reads the following shell variables. They need to be export
ed by the parent bash script.
Shell Variable | Explanation | Example |
testFunction * |
The name of the function being tested. This does not have to be a string. It can be a lambda expression as well. |
|
testArguments * |
Arguments to the function separated by a single whitespace. |
|
testOutputTransform |
Before the student and reference arguments are passed to testAssertion , the test runner will pass it through the provided function. The testOutputTransform function is given one argument, which was the output of the test function. The output of this function against the student code is written to DEBUG . |
|
testAssertion * |
A Scheme boolean expression. This expression has access to student and reference , which are expressions returned from calling the testFunction on the student and reference source files. |
|
Important Note: The testAssertion
boolean expression is evaluated after both the student and reference source files are loaded, so be careful lazy objects (created with delay
) in the student code don't improperly reference object/functions in the reference code. This almost always happens when testing stream objects. To prevent this, use testOutputTransform
to ensure student
and reference
become fully evaluated (or eager) objects before the assertion runs.
In addition to the shell variables above, the genericTester.sh
bash script takes the following flags:
Flag | Explanation |
--show-reference-output |
Passing this flag causes the output of the reference function to be written to the DEBUG file. This is useful for debugging. |
--check-pledge |
Pass this flag to immediately fail submissions that don't contain the student plagiarism pledge. |
--show-test-call |
Write out the tested function and its arguments to DEBUG . At the moment, this outputs an apply expression since displaying an argument list without parenthesis is difficult in PLT R5RS. You may not want this flag enabled if arguments display strangely and end up confusing students. |
Common Scenarios
I'd like to display a message to students
Add the following before the call to ./genericTester.sh
echo "I'm testing $testFunction with arguments: $testArguments" > DEBUG
I need to test a variable
Make testFunction
a lambda function that returns that variable. For example, if the student code is:
(define pi 3.14159)
Then the test should be:
export testFunction="(lambda () pi)"
export testArguments=""
export testAssertion="(= student 3.14159)"
./genericTester.sh
Examples
Testing a factorial function
export testFunction="fact"
export testArguments="5"
export testAssertion="(equal? student reference)"
echo "Testing 5!" > DEBUG
./genericTester.sh --check-pledge --show-reference-output --show-test-call
Testing numbers with some tolerance
export testFunction="pi-approx"
export testArguments="100"
export testAssertion="(ref-approx-=? student reference 0.01)"
./genericTester.sh --show-reference-output
Testing a stream
More care is necessary when testing lazily-evaluated objects.
export testFunction="primes"
export testArguments=""
export testOutputTransform="(lambda (x) (ref-str-to-list x 20)"
export testAssertion="(equal? student reference)"
./genericTester.sh --show-reference-output
Testing object-oriented constructors
export testFunction="make-bank-account"
export testArguments="500"
export testOutputTransform="
(lambda (x)
((x 'withdraw) 300)
((x 'balance)))
"
export testAssertion="(= student 200)"
./genericTester.sh --show-reference-output
Turing-complete test assertions
Expressions in Scheme are extremely powerful. The below example shows how to use a named let to ensure a student output is a list of integers.
export testFunction="something-that-returns-an-integer-list"
export testArguments=""
export testAssertion="
(let is-integer-list? ((lst student))
(if (null? lst)
#t
(and (integer? (car lst))
(is-integer-list? (cdr lst)))
"
./genericTester.sh --show-reference-output
Of course, for this example, it'd be better to use map
, apply
, and and
as an assertion. The above is more complex for demonstration purposes.