CSE 1729 Mimir Testing Scripts
This repository contains testing scripts to automate grading of CSE 1729 labs and problem sets.
To setup a new lab or problem set:
- Prepare a
referenceSolution.scmfile and put it at the same directory level as the files in this resository.
- Zip up the contents of this folder (including the
referenceSolution.scmfile to the
Files (optional)section of the test case under Advanced Options.
- Add the following to the Bash Script section with your customizations:
export testFunction="fact" export testArguments="5" export testAssertion="(equal? student reference)" ./genericTester.sh
- Uncheck Show Expected Output to Student, Show Bash Script to Student, and Show Bash Script OUTPUT to Student. These typically show extraneous information that confuses students.
genericTester.sh reads the following shell variables. They need to be
exported by the parent bash script.
||The name of the function being tested. This does not have to be a string. It can be a lambda expression as well.||
||Arguments to the function separated by a single whitespace.||
||A Scheme boolean expression. This expression has access to
Important Note: The
testAssertion boolean expression is evaluated after both the student and reference source files are loaded, so be careful lazy objects (created with
delay) in the student code don't improperly reference object/functions in the reference code. This almost always happens when testing stream objects. To prevent this, use
testOutputTransform to ensure
reference become fully evaluated (or eager) objects before the assertion runs.
In addition to the shell variables above, the
genericTester.sh bash script takes the following flags:
||Passing this flag causes the output of the reference function to be written to the
||Pass this flag to immediately fail submissions that don't contain the student plagiarism pledge.|
||Write out the tested function and its arguments to
I'd like to display a message to students
Add the following before the call to
echo "I'm testing $testFunction with arguments: $testArguments" > DEBUG
I need to test a variable
testFunction a lambda function that returns that variable. For example, if the student code is:
(define pi 3.14159)
Then the test should be:
export testFunction="(lambda () pi)" export testArguments="" export testAssertion="(= student 3.14159)" ./genericTester.sh
Testing a factorial function
export testFunction="fact" export testArguments="5" export testAssertion="(equal? student reference)" echo "Testing 5!" > DEBUG ./genericTester.sh --check-pledge --show-reference-output --show-test-call
Testing numbers with some tolerance
export testFunction="pi-approx" export testArguments="100" export testAssertion="(ref-approx-=? student reference 0.01)" ./genericTester.sh --show-reference-output
Testing a stream
More care is necessary when testing lazily-evaluated objects.
export testFunction="primes" export testArguments="" export testOutputTransform="(lambda (x) (ref-str-to-list x 20)" export testAssertion="(equal? student reference)" ./genericTester.sh --show-reference-output
Testing object-oriented constructors
export testFunction="make-bank-account" export testArguments="500" export testOutputTransform=" (lambda (x) ((x 'withdraw) 300) ((x 'balance))) " export testAssertion="(= student 200)" ./genericTester.sh --show-reference-output
Turing-complete test assertions
Expressions in Scheme are extremely powerful. The below example shows how to use a named let to ensure a student output is a list of integers.
export testFunction="something-that-returns-an-integer-list" export testArguments="" export testAssertion=" (let is-integer-list? ((lst student)) (if (null? lst) #t (and (integer? (car lst)) (is-integer-list? (cdr lst))) " ./genericTester.sh --show-reference-output
Of course, for this example, it'd be better to use
and as an assertion. The above is more complex for demonstration purposes.