README.md

# CSE 1729 Mimir Testing Scripts

This repository contains testing scripts to automate grading of CSE 1729 labs and problem sets.

## Setup

To setup a new lab or problem set:

- Prepare a `referenceSolution.scm` file and put it at the same directory level as the files in this resository.
- Zip up the contents of this folder (including the `referenceSolution.scm` file to the `Files (optional)` section of the test case under *Advanced Options*.
- Add the following to the *Bash Script* section with your customizations:
  ```sh
  export testFunction="fact"
  export testArguments="5"
  export testAssertion="(equal? student reference)"

  ./genericTester.sh
  ```
- Uncheck *Show Expected Output to Student*, *Show Bash Script to Student*, and *Show Bash Script OUTPUT to Student*. These typically show extraneous information that confuses students.

## Options

The `genericTester.sh` reads the following shell variables. They need to be `export`ed by the parent bash script.

<table>
  <thead>
    <tr>
      <td>Shell Variable</td>
      <td>Explanation</td>
      <td>Example</td>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>testFunction</code>*</td>
      <td>The name of the function being tested. This does not have to be a string. It can be a lambda expression as well.</td>
      <td>
        <ul>
          <li><code>fact</code></li>
          <li><code>(lambda () pi)</code></li>
      </td>
    </tr>
    <tr>
      <td><code>testArguments</code>*</td>
      <td>Arguments to the function separated by a single whitespace.</td>
      <td>
        <ul>
          <li><code>5</code></li>
          <li><code>(list 1 2 3) (list 4 5 6)</code></li>
        </ul>
      </td>
    </tr>
    <tr>
      <td><code>testOutputTransform<code></td>
      <td>Before the <code>student</code> and <code>reference</code> arguments are passed to <code>testAssertion</code>, the test runner will pass it through the provided function. The <code>testOutputTransform</code> function is given one argument, which was the output of the test function. The output of this function against the student code is written to <code>DEBUG</code>.</td>
      <td>
        <ul>
          <li><code>(lambda (x) x)</code></li>
          <li><code>(lambda (x) (ref-str-to-list x 20)</code></li>
        </ul>
      </td>
    </tr>
    <tr>
      <td><code>testAssertion</code>*</td>
      <td>A Scheme boolean expression. This expression has access to <code>student</code> and <code>reference</code>, which are expressions returned from calling the <code>testFunction</code> on the student and reference source files.</td>
      <td>
        <ul>
          <li><code>(= student reference)</code></li>
          <li><code>(= (car student) 25)</code></li>
        </ul>
      </td>
    </tr>
  </tbody>
</table>

**Important Note**: The `testAssertion` boolean expression is evaluated after both the student and reference source files are loaded, so be careful lazy objects (created with `delay`) in the student code don't improperly reference object/functions in the reference code. This almost always happens when testing stream objects. To prevent this, use `testOutputTransform` to ensure `student` and `reference` become fully evaluated (or eager) objects before the assertion runs.

In addition to the shell variables above, the `genericTester.sh` bash script takes the following flags:

<table>
  <thead>
    <tr>
      <td>Flag</td>
      <td>Explanation</td>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>--show-reference-output</code></td>
      <td>Passing this flag causes the output of the reference function to be written to the <code>DEBUG</code> file. This is useful for debugging.</td>
    </tr>
    <tr>
      <td><code>--check-pledge</code></td>
      <td>Pass this flag to immediately fail submissions that don't contain the student plagiarism pledge.</td>
    </tr>
    <tr>
      <td><code>--show-test-call</code></td>
      <td>Write out the tested function and its arguments to <code>DEBUG</code>. At the moment, this outputs an <code>apply</code> expression since displaying an argument list without parenthesis is difficult in PLT R5RS. You may not want this flag enabled if arguments display strangely and end up confusing students.</td>
    </tr>
  </tbody>
</table>

## Common Scenarios

### I'd like to display a message to students

Add the following before the call to `./genericTester.sh`

```sh
echo "I'm testing $testFunction with arguments: $testArguments" > DEBUG
```

### I need to test a variable

Make `testFunction` a lambda function that returns that variable. For example, if the student code is:

```scheme
(define pi 3.14159)
```

Then the test should be:

```sh
export testFunction="(lambda () pi)"
export testArguments=""
export testAssertion="(= student 3.14159)"
./genericTester.sh
```

## Examples

### Testing a factorial function

```sh
export testFunction="fact"
export testArguments="5"
export testAssertion="(equal? student reference)"

echo "Testing 5!" > DEBUG
./genericTester.sh --check-pledge --show-reference-output --show-test-call
```

### Testing numbers with some tolerance

```sh
export testFunction="pi-approx"
export testArguments="100"
export testAssertion="(ref-approx-=? student reference 0.01)"
./genericTester.sh --show-reference-output
```

### Testing a stream

More care is necessary when testing lazily-evaluated objects.

```sh
export testFunction="primes"
export testArguments=""
export testOutputTransform="(lambda (x) (ref-str-to-list x 20)"
export testAssertion="(equal? student reference)"
./genericTester.sh --show-reference-output
```

### Testing object-oriented constructors

```sh
export testFunction="make-bank-account"
export testArguments="500"
export testOutputTransform="
  (lambda (x)
    ((x 'withdraw) 300)
    ((x 'balance)))
"
export testAssertion="(= student 200)"
./genericTester.sh --show-reference-output
```

### Turing-complete test assertions 

Expressions in Scheme are extremely powerful. The below example shows how to use a [named let](https://www.gnu.org/software/mit-scheme/documentation/mit-scheme-ref/Iteration.html) to ensure a student output is a list of integers.

```sh
export testFunction="something-that-returns-an-integer-list"
export testArguments=""
export testAssertion="
  (let is-integer-list? ((lst student))
    (if (null? lst)
        #t
        (and (integer? (car lst))
             (is-integer-list? (cdr lst)))
"
./genericTester.sh --show-reference-output
```

Of course, for this example, it'd be better to use `map`, `apply`, and `and` as an assertion. The above is more complex for demonstration purposes.