CSE 1729 Mimir Testing Scripts

This repository contains testing scripts to automate grading of CSE 1729 labs and problem sets.

Setup

To setup a new lab or problem set:

• Prepare a referenceSolution.scm file and put it at the same directory level as the files in this resository.
• Zip up the contents of this folder (including the referenceSolution.scm file to the Files (optional) section of the test case under Advanced Options.
• Add the following to the Bash Script section with your customizations:

  export testFunction="fact"
  export testArguments="5"
  export testAssertion="(equal? student reference)"
  
  ./genericTester.sh

• Uncheck Show Expected Output to Student, Show Bash Script to Student, and Show Bash Script OUTPUT to Student. These typically show extraneous information that confuses students.

Options

The genericTester.sh reads the following shell variables. They need to be exported by the parent bash script.

Shell Variable	Explanation	Example
testFunction*	The name of the function being tested. This does not have to be a string. It can be a lambda expression as well.	fact (lambda () pi)
testArguments*	Arguments to the function separated by a single whitespace.	5 (list 1 2 3) (list 4 5 6)
testOutputTransform	Before the student and reference arguments are passed to testAssertion, the test runner will pass it through the provided function. The testOutputTransform function is given one argument, which was the output of the test function. The output of this function against the student code is written to DEBUG.	(lambda (x) x) (lambda (x) (ref-str-to-list x 20)
testAssertion*	A Scheme boolean expression. This expression has access to student and reference, which are expressions returned from calling the testFunction on the student and reference source files.	(= student reference) (= (car student) 25)

Important Note: The testAssertion boolean expression is evaluated after both the student and reference source files are loaded, so be careful lazy objects (created with delay) in the student code don't improperly reference object/functions in the reference code. This almost always happens when testing stream objects. To prevent this, use testOutputTransform to ensure student and reference become fully evaluated (or eager) objects before the assertion runs.

In addition to the shell variables above, the genericTester.sh bash script takes the following flags:

Flag	Explanation
--show-reference-output	Passing this flag causes the output of the reference function to be written to the DEBUG file. This is useful for debugging.
--check-pledge	Pass this flag to immediately fail submissions that don't contain the student plagiarism pledge.
--show-test-call	Write out the tested function and its arguments to DEBUG. At the moment, this outputs an apply expression since displaying an argument list without parenthesis is difficult in PLT R5RS. You may not want this flag enabled if arguments display strangely and end up confusing students.

Common Scenarios

I'd like to display a message to students

Add the following before the call to ./genericTester.sh

echo "I'm testing $testFunction with arguments: $testArguments" > DEBUG

I need to test a variable

Make testFunction a lambda function that returns that variable. For example, if the student code is:

(define pi 3.14159)

Then the test should be:

export testFunction="(lambda () pi)"
export testArguments=""
export testAssertion="(= student 3.14159)"
./genericTester.sh

Examples

Testing a factorial function

export testFunction="fact"
export testArguments="5"
export testAssertion="(equal? student reference)"

echo "Testing 5!" > DEBUG
./genericTester.sh --check-pledge --show-reference-output --show-test-call

Testing numbers with some tolerance

export testFunction="pi-approx"
export testArguments="100"
export testAssertion="(ref-approx-=? student reference 0.01)"
./genericTester.sh --show-reference-output

Testing a stream

More care is necessary when testing lazily-evaluated objects.

export testFunction="primes"
export testArguments=""
export testOutputTransform="(lambda (x) (ref-str-to-list x 20)"
export testAssertion="(equal? student reference)"
./genericTester.sh --show-reference-output

Testing object-oriented constructors

export testFunction="make-bank-account"
export testArguments="500"
export testOutputTransform="
  (lambda (x)
    ((x 'withdraw) 300)
    ((x 'balance)))
"
export testAssertion="(= student 200)"
./genericTester.sh --show-reference-output

Turing-complete test assertions

Expressions in Scheme are extremely powerful. The below example shows how to use a named let to ensure a student output is a list of integers.

export testFunction="something-that-returns-an-integer-list"
export testArguments=""
export testAssertion="
  (let is-integer-list? ((lst student))
    (if (null? lst)
        #t
        (and (integer? (car lst))
             (is-integer-list? (cdr lst)))
"
./genericTester.sh --show-reference-output

Of course, for this example, it'd be better to use map, apply, and and as an assertion. The above is more complex for demonstration purposes.