![[Picture of an egg]](egg.jpg)
Macro based unit-testing facility
Peter Keller
chicken 2.225+
(require-extension test-infrastructure)
This extension can currently only be used with the syntax-case
(highlevel) macro system.
This extension provides a macro based unit testing facility based upon expectations concerning evaluations of expressions. These functions return tagged lists which contain the results of the test package, test case, or expectations evaluated (there are a few other types of results dealing with the gloss, todo, and skip macros detailed below). These result lists are wired together during evaluation to form a large hierarchical tree in memory. This result tree is then passed to either user defined functions which traverse the tree manipulating it in any way desired, or passed to a supplied function which manipulates it in a simple way usually producing human readable or html generated output. API functions to deal with the result types are supplied and the representation of the result is black boxed to the user. It is a violation of encapsulation to inspect the representation directly, and it may change unpredictably in the future.
The term MESSAGE below means the name given to a test
element. While this can be any Scheme object #f is
interpreted as a missing name. Further, #f is assumed when
a name is missing. Thus, the name of all un-named test elements is the
same - #f.
The Immediate Test Macro API
This macro will evaluate in a left to right fashion the clauses
inside it and output the results. The clauses are a sequence of
(Expression Expectation) forms. A clause is evaluated as
(expect-equal 'Expression Expectation Expression). For use
in a REPL.
The results are not kept and no other forms are allowed in the body.
test-itEXPRESSION any scheme expression.EXPECTED any scheme expression, the expected result of
EXPRESSION.The Quick Test Macro API
This macro will execute the test procedure and send the results to a stylist procedure. The specifiers can be used to select and de-select test packages, groups, cases, and expectations.
Do not de-select all, or omit to select least one, test packages or groups. Otherwise the result will be ill-formed.
The results are not kept.
test-applySTYLER is a single argument procedure taking the test
result object, output-style-minimal when missing.SPECIFIER is (skip MESSAGE ...) or
(take MESSAGE ...), where MESSAGE is a test element name as
below. The skip specifier causes the named test elements not to be
performed and the take specifier is the opposite. Do not use these forms
together, they override each other. Only named test elements can be
selected. Unnamed elements are always performed. Also, (skip CLAUSE ...) forms,
see below, still apply.THUNK is a zero argument procedure returning a test
result object.The Test Message Introspection API
Provides information on the tree structure of a test.
test-named-structureTHUNK is a zero argument procedure returning a test
result object.thunk, package,
case, expect) and the value is the message
(which will be #f when the element is not named). The
thunk element value is the symbolic form of the
THUNK.THUNK argument is a procedure variable,
and not a procedure object. walk-test-named-structurePROCEDURE is a three argument procedure. The first argument
is the hierarchy level, counting up from 0. The second argument is the
type of the test element, as above. The third argument is the name of the
test element.NAMED-STRUCTURE association list,
invoking the PROCEDURE for each element. The hierarchy level value
is the tree depth at the point of call.The Simple Test Package Macro API
This macro will evaluate in a left to right fashion the clauses inside it. Clauses can only be certain things, detailed below. All of the clauses are executed, except of course if you bail out of the test group with the escape procedure mechanism. Note that a test group is a test package.
The destructor object name is destructor. The escape procedure name
is escape. Each nesting will lexically shadow the previous such
definitions of an outer test group.
test-groupMESSAGE can be any scheme object, though usually it is a string.BINDINGS are let-style bindings that you may create and exist in
the lexical scope of the test package.CLAUSES are uses of (test-group ...),
(test-package ...), (test-case ...),
(expect* ...) along with (gloss ...),
(todo ...), (skip ...), and (terminate
...) macros. If the expectation fails, the test group macro will
continue evaluating until all clauses are evaluated or the escape
procedure mechanism is activated. This is different than a test-case
macro where upon discovery of a failed expectation, evaluation stops
immediately.The Test Package Macro API
This macro will evaluate in a left to right fashion the clauses inside it. Clauses can only be certain things, detailed below. All of the clauses are executed, except of course if you bail out of the test package with the escape procedure mechanism. Test groups and packages may nest indefinitely.
test-packageMESSAGE can be any scheme object, though usually it is a string.DESTNAME is an unquoted symbol for an automatic destructor
object that gets called when the test package completes for any reason.
This symbol is bound to a destructor object and is available to you in
the CLAUSES section of the test package. See below for the description
of the destructor object interface.TERMNAME is an unquoted symbol for an escape procedure
available in the body of the test package, usually, this escape
procedure is passed to (terminate ...) which calls it
for you and performs other tasks. It is not recommended to call the escape
procedure directly.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the test package. The
warn function name is actually a syntax reserved word
in the macro.BINDINGS are let-style bindings that you may create and exist in
the lexical scope of the test package.CLAUSES are uses of (test-group ...),
(test-package ...), (test-case ...) macros
along with (gloss ...), (todo ...),
(skip ...), and (terminate ...) macros. While
you may use the (expect* ...) style macros directly in a
test package, doing so is not recommended. If the expectation fails, the
test package macro will continue evaluating until all clauses are
evaluated or the escape procedure mechanism is activated. This is
different than a test-case macro where upon discovery of a failed
expectation, evaluation stops immediately.The Test Case Macro API
This macro will evaluate in a left to right fashion the clauses inside it, stopping at the first failed expectation. Clauses can only be certain things as detailed below. You may also stop the execution of expectations if you bail out of the test case with the escape procedure mechanism. Test cases may NOT nest.
test-caseMESSAGE can be any scheme object, though usually it is a string.DESTNAME is an unquoted symbol for an automatic destructor
object that gets called when the test case completes for any reason.
This symbol is bound to a destructor object and is available to you in
the CLAUSES section of the test package. See below for the description
of the destructor object interface.TERMNAME is an unquoted symbol for an escape procedure
available in the body of the test case, usually, this escape
procedure is passed to (terminate ...) which calls it
for you and performs other tasks. It is not recommended to call the escape
procedure directly.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the test case. The
warn function name is actually a syntax reserved word
in the macro.BINDINGS are let-style bindings that you may create and exist in
the lexical scope of the test case.CLAUSES are uses of (expect* ...) macros
along with (gloss ...), (todo ...),
(skip ...), and (terminate ...) macros. It is
important to note that upon discovery of a failed expectation,
the test case stops its evaluation and returns with the previous
successful, and including the failed, expectations. This behavior can be
changed using the test-continue? form. This behavior reverts to
default at the start of each test-case.The Expectation Macro API
An expectation at its core simply evaluates its arguments and check to see if it matches the expectation. The positive or negative result is encapsulated, along with other things such as the unevaluated expressions being checked and some messages supplied with each expectation into a particular type of black box object that one can query with the appropriate API calls (detailed below).
Expectations all have a descriptive message that can be bound to them, along with an optional warning syntax detailed below. A design decision was made to supply expectation macros for the usual types of expectations a user needs because this reduced the abstractness of an expectation into something more manageable.
Expectations are evaluated under an exception handler. An unexpected exception
is treated as expectation failure. Note that only the CLAUSE expression
evaluation traps exceptions. Any exceptions in setup or teardown expressions will not
be caught by the test-infrastructure.
expectMESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.KIND is a string stating the kind of test.PREDICATE is a procedure of one argument to perform the test.CLAUSE is a single expression which should return a value meeting the
conditions of the predicate. expect-zeroMESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.CLAUSE is a single expression which should return a exact or
inexact integer. expect-nonzeroMESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.CLAUSE is a single expression which should return an exact or
inexact integer. expect-true#t.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.CLAUSE is a single expression which should return #t. expect-false#f.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.CLAUSE is a single expression which should return #f. expect-success#t, or a non-error exception occured.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.CLAUSE is a single expression which should return #t, or
generate a non-error exception. expect-failure#f, or an error exception occured.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.CLAUSE is a single expression which should return #f, or
generate an error exception. expect-not-false#f.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.CLAUSE is a single expression which should not return
#f, any other value is accepted. expect-equiv(PREDICATE EXPECTED CLAUSE) is
true.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.EXPECTED is a single expression which is evaluated and represents
the value the CLAUSE must be equiv? to in order for this
expectation to return a positive result.CLAUSE is a single expression which, when evaluated must return
an object where an equiv? of this result and the EXPECTED expression
is #t.KIND is a string stating the kind of test.PREDICATE is a procedure of two arguments to perform the test.CLAUSE expression as a field, but not an unevaluated EXPECTED
expression. expect-eq(eq? EXPECTED CLAUSE) is true.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.EXPECTED is a single expression which is evaluated and represents
the value the CLAUSE must be eq? to in order for this
expectation to return a positive result.CLAUSE is a single expression which, when evaluated must return
an object where an eq? of this result and the EXPECTED expression
is #t.CLAUSE expression as a field, but not an unevaluated EXPECTED
expression. expect-eqv(eqv? EXPECTED CLAUSE) is
true.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.EXPECTED is a single expression which is evaluated and represents
the value the CLAUSE must be eqv? to in order for this
expectation to return a positive result.CLAUSE is a single expression which, when evaluated must return
an object where an eqv? of this result and the EXPECTED expression
is #t.CLAUSE expression as a field, but not an unevaluated EXPECTED
expression. expect-equal(equal? EXPECTED CLAUSE) is
true.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.EXPECTED is a single expression which is evaluated and represents
the value the CLAUSE must be equal? to in order for this
expectation to return a positive result.CLAUSE is a single expression which, when evaluated must return
an object where an equal? of this result and the EXPECTED expression
is #t.CLAUSE expression as a field, but not an unevaluated EXPECTED
expression. expect-valuesPREDICATE.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.PREDICATE is a procedure taking a single list argument and returning #t or #f.CLAUSE is a single expression which has a multi-valued result. expect-values-eqMESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.VALUES is a list of values.CLAUSE is a single expression which has a multi-valued result. expect-values-eqvMESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.VALUES is a list of values.CLAUSE is a single expression which has a multi-valued result. expect-values-equalMESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.VALUES is a list of values.CLAUSE is a single expression which has a multi-valued result. expect-near(< (abs (- EXPECTED CLAUSE)) TOL))) is true.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.EXPECTED is a single expression which is evaluated and represents
the value the CLAUSE must be ``near'' to in order for this
expectation to return a positive result.CLAUSE is a single expression which should return an inexact or
exact number.TOL is a single expression which, when evaluated must return a
tolerance value (usually a small inexact number like .0001).CLAUSE expression as a field, but not the unevaluated EXPECTED
or TOL expression. expect-positiveMESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.CLAUSE is a single expression which should return an inexact or
exact number. expect-negativeMESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.CLAUSE is a single expression which should return an inexact or
exact number. expect-exceptionEXN.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.EXN is a specialized logical expression describing the expected exception, a condition
as defined by SRFI-12.CLAUSE is a single expression which should signal an exception matching EXN.
EXN : COND-EXN
| SIMP-EXN
| ()
SIMP-EXN : KINDKEY
| (property KINDKEY PROP-EXPR ...)
| (SIMP-EXN ...)
COND-EXN : (and EXN ...)
| (or EXN ...)
| (not EXN)
PROP-EXPR : COND-PROP-EXPR
| SIMP-PROP-EXPR
| ()
SIMP-PROP-EXPR : PROPKEY
| (PROPKEY ...)
| (PROPKEY VALUE)
| (PREDICATE PROPKEY VALUE)
COND-PROP-EXPR : (and PROP-EXPR ...)
| (or PROP-EXPR ...)
| (not PROP-EXPR)
An example:
(test-case "exn1" d e (
[exn1 (make-composite-condition
(make-property-condition 'abc 'cbs "pbs")
(make-property-condition 'foo 'bar "zip"))])
(expect-exception "1" (foo abc) (signal exn1))
;; These are the same test, the 2nd has an explicit predicate
(expect-exception "2" ((property foo (bar "zip"))
(property abc (cbs "pbs"))) (signal exn1))
(expect-exception "3" ((property foo (equal? bar "zip"))
(property abc (equal? cbs "pbs"))) (signal exn1))
)
expect-ec(PREDICATE EXPECTED CLAUSE) is
true for every state produced by the set of QUALIFIER.MESSAGE can be any scheme object, though usually it is a string.(warn MESSAGE) allows you to specify a warning object, usually
a string, that gets associated with the expectation. The
warn function name is actually a syntax reserved word
in the macro.CLAUSE and EXPECTED are expressions.PREDICATE is a two argument procedure returning a
boolean, equal? is used when missing.QUALIFIER is a SRFI-42 <qualifier>. An eager
comprehension implementation, as described by SRFI-42 must be
provided to use this expectation. expect-set!EXPR under an expect-success. When successful
a test:set! is performed of the EXPR to ID.
The EXPR is therefore evaluated twice. expect-successful-failure(expect-success "Failure" EXPR).Testeez-like API
A suite of synonyms for the expect-* macros. Note that
the (warn WARNING) argument form is not supported. Here
DESC is the same as MESSAGE, and
EXPR is the same as CLAUSE.
test/zero test/nonzero test/true test/false test/success test/failure test/positive test/negative test/not-false test/equiv test/eq test/eqv test/equal test/near test/values test/values-eq test/values-eqv test/values-equal test/exception test/ecResult Object API
Expectations, test cases, test packages, and helper macros (gloss,
todo, etc) all return an object that contains the results and other various
aspects of the action performed which ultimately get wired together to
form the result tree. This collection of functions forming the rest of the
test infrastructure API allows manipulation of these results in an abstracted
way as to allow changing of the representation in the future.
Test Package Result Object API
If any of these API functions, except test-package-result?, are passed something that isn't a test package result object, they will return 'not-a-test-package-result.
test-package-result?RESULT is a result object from the invocation
of a test package macro, then this function will return
#t. Otherwise, it will return #f. test-package-result-result-refRESULT
object. test-package-result-message-refRESULT
object. test-package-result-exps-refRESULT object. test-package-result-warning?#t, otherwise it will be #f. test-package-result-warning-refTest Case Result Object API
If any of these API functions, except test-case-result?, are passed something that isn't a test case result object, they will return 'not-a-test-case-result.
test-case-result?RESULT is a result object from the invocation
of a test case macro, then this function will return
#t. Otherwise, it will return #f. test-case-result-result-refRESULT
object. test-case-result-message-refRESULT
object. test-case-result-expectations-refRESULT object. test-case-result-warning?#t, otherwise it will be #f. test-case-result-warning-refExpect Result Object API: Single Clause Style Expectation
These expectations all take the form of passing a single expression to them to see if they match some a priori expectation. If any of these API functions, except expect-result?, are passed something that isn't a single clause style expectation result object, they will return 'not-an-expect-result.
expect-result?RESULT is a single clause style result object from
the invocation of an expectation macro, then this function will
return #t. Otherwise, it will return #f. expect-result-result-refRESULT object. expect-result-specific-ref(expect-zero? "foobar" (- 1 1))
expectation, then the ``specific'' field of the expectation
result object will be the string "zero". Here is a
table describing what the ``specific'' fields are for each kind
of single clause style expectation:
| Single Clause Style Expectation | Associated Specific String |
|---|---|
| expect-zero | "zero" |
| expect-nonzero | "nonzero" |
| expect-true | "true" |
| expect-not-false | "not-false" |
| expect-false | "false" |
| expect-positive | "positive" |
| expect-negative | "negative" |
| expect | KIND |
expect-result-message-refRESULT object. expect-result-unevaled-ref expect-result-evaled-ref expect-result-warning?#t, otherwise it will be #f. expect-result-warning-refExpect Result Object API: Equivalence Style Expectation
These expectations all take the form of passing a two expressions, the ``left hand side'' and the ``right hand side'' to them to see if they match some a priori equivalence. The left hand side is that which you expect the right hand side to be equivalent. If any of these API functions, except expect-equivalence-result?, are passed something that isn't a single clause style expectation result object, they will return 'not-an-expect-equivalence-result.
expect-equivalence-result?RESULT is a comparison style result object from
the invocation of an expectation macro, then this function will
return #t. Otherwise, it will return #f. expect-equivalence-result-result-refRESULT object. expect-equivalence-result-specific-ref(expect-equal? "foobar" 0 (- 1 1))
expectation, then the ``specific'' field of the expectation
result object will be the string "equal". Here is a
table describing what the ``specific'' fields are for each kind
of equivalence style expectation:
| Equivalence Style Expectation | Associated Specific String |
|---|---|
| expect-eq | "eq" |
| expect-eqv | "eqv" |
| expect-equal | "equal" |
| expect-exception | "exception" |
| expect-equiv | KIND |
expect-equivalence-result-message-refRESULT object. expect-equivalence-result-lhs-evaled-ref expect-equivalence-result-rhs-unevaled-ref expect-equivalence-result-rhs-evaled-ref expect-equivalence-result-warning?#t, otherwise it will be #f. expect-equivalence-result-warning-refExpect Result Object API: Tolerance Style Expectation
This is a specialized expectation which accepts three expressions and checks to see if the ``right hand side'' is within a ``tolerance'' of the ``left hand side''. There is only one expectation in the tolerance style currently. If any of these API functions, except expect-tolerance-result?, are passed something that isn't a tolerance style expectation result object, they will return 'not-an-expect-tolerance-result.
expect-tolerance-result?RESULT is a tolerance style result object from
the invocation of an expectation macro, then this function will
return #t. Otherwise, it will return #f. expect-tolerance-result-result-refRESULT object. expect-tolerance-result-specific-ref(expect-near? "foobar" 100 .01 100.001)
expectation, then the ``specific'' field of the expectation
result object will be the string "near". Here is a
table describing what the ``specific'' fields are for each kind
of tolerance style expectation:
| Tolerance Style Expectation | Associated Specific String |
|---|---|
| expect-near | "near" |
expect-tolerance-result-message-refRESULT object. expect-tolerance-result-lhs-evaled-ref expect-tolerance-result-lhs-tol-evaled-ref expect-tolerance-result-rhs-unevaled-ref expect-tolerance-result-rhs-evaled-ref expect-tolerance-result-warning?#t, otherwise it will be #f. expect-tolerance-result-warning-refVarious Helper API
These upcoming macros and functions allow the author of the test suite to better control both the execution flow of the test suite and ``decoration'' of the test suite with important information like things yet to do, or just plain documentation.
Termination API
When executing in a test package or a test case, one might discover some catastrophic failure of such proportions that it is utterly impossible to continue executing the test case or test package. When that happens you can use the termination facility to exit the test case or test package. Of course, no more expressions will be evaluated in the scope of the termination. It is recommended that you use this method of terminating the test case or test package evaluation since it wraps some contextual information up into the termination result so you can figure out what happened (and where) later when analyzing the result tree.
When using the manipulation API for a terminate result, if you pass a result to one of these function that is not a terminate result, it will return 'not-a-terminate-result.
terminateTERMFUNC is the name of the termination procedure that you
specified in a test case or test package. You may pass any test package
or test case termination function available to you in the lexical scope
in which you call this function. The termination will take effect in the
scope of the created termination function.MESSAGE can be any scheme object, though usually it is a string. terminate-result?RESULT is a termination result object from the
invocation of a termination function, then this function will
return #t. Otherwise, it will return #f. terminate-result-result-refRESULT object. This is currently hard coded to
be #f. terminate-result-scope-refMESSAGE
parameter supplied to the test case or test package associated with the
TERMFUNC. terminate-result-container-refTERMFUNC was associated. terminate-result-message-refRESULT object.Destructor Object API
The destructor object allows for you to create helper functions which clean up for you usually in case of aborting of a test case or package. For example, suppose you are testing whether or not file writing to a file works correctly in a test case, so, you'd perform an expectation to open the file, and then queue a function in the destructor to remove the file, and then perform the expectation of the write. If the write (or subsequent) expectation fails, then the test case will automatically invoke the helper cleanup function specified in the destructor object that removes the file.
NOTE: This API is still a little experimental in the sense that eventually he destructor object should return a typed result that contains the success of the individual destructor calls. But for now, it is functional for what it does. Also, be VERY CAREFUL that you specify the arguments to these API calls correctly since due to lambda functions not being comparable, this API cannot guarantee that a true destructor object name had been passed to it. So if you call one of the following API calls incorrectly, the behavior will be undefined.
destructor-atexit!FUNC with the
supplied ARGS ... into a queue in the DESTNAME
destructor object. Multiple invocations of this API call will
continue to queue up (FUNC ARGS ...) promises indefinitely.
This function returns a special ignore type that is ignored
by the test infrastructure system. destructor-activate!clear itself. This function returns
a special ignore type that is ignored by the test
infrastructure system. destructor-clear!DESTNAME. This function
returns a special ignore type that is ignored by the
test infrastructure system. destructor-dumpTodo API
The purpose of the todo API is to allow the author of a test suite the ability to record into the result tree for later analysis that something still needs to be done. This way you can count/manipulate this information at a later date. Todo macro invocations can occur inside of test cases or test packages.
todoMESSAGE can be any scheme object, though usually it is a string.(warn WARNING) allows you to specify a warning object, usually
a string, that gets associated with the todo. The warn
function name is actually a syntax reserved word in the macro. todo-result?RESULT is a todo result object from the
invocation of a todo macro, then this function will
return #t. Otherwise, it will return #f. todo-result-message-refRESULT object. todo-result-warning?#t, otherwise it will be #f. todo-result-warning-refGloss API
The purpose of the gloss API is to allow the author of a test suite the ability to record messages into the result tree purely for documentation purposes. Gloss macro invocations can occur inside of test cases or test packages.
glossMESSAGE can be any scheme object, though usually it is a string.(warn WARNING) allows you to specify a warning object, usually
a string, that gets associated with the gloss. The warn
function name is actually a syntax reserved word in the macro. gloss-result?RESULT is a gloss result object from the
invocation of the gloss macro, then this function will
return #t. Otherwise, it will return #f. gloss-result-message-refRESULT object. gloss-result-warning?#t, otherwise it will be #f. gloss-result-warning-refSkip API
The purpose of the skip API is to allow the author of a test suite to completely skip evaluation of a set of expressions. Skip macro invocations can occur inside of test cases or test packages.
skipMESSAGE can be any scheme object, though usually it is a string.(warn WARNING) allows you to specify a warning object, usually
a string, that gets associated with the gloss. The warn
function name is actually a syntax reserved word in the macro.CLAUSES can be more than one expression (as in a lambda form)
that does NOT get evaluated at any time. skip-result?RESULT is a skip result object from the
invocation of the skip macro, then this function will
return #t. Otherwise, it will return #f. skip-result-message-refRESULT object. Hopefully, it was stated why this
set of clauses had been skipped. skip-result-warning?#t, otherwise it will be #f. skip-result-warning-refSide Effect API
This section of the API is considered a little experimental for now.
The side effecting evaluates all of its arguments as in a (begin
...) form, it returns a result that is completely ignored by the
system and unavailable to the output analysis code.
side-effect(set! ...) or with complicated situations where a lot of setup
work must happen for an expectation to be performed. test-continue? test-timing? test:set!Miscellaneous API
This section contains a few functions whose purpose is to simplify certain kinds of manipulations of result objects.
*-result?#t of RESULT-OBJ is any kind of an
evaluated result object. Otherwise, it shall return #f. *-result-refRESULT-OBJ
passed to it if it is indeed a true result object. Otherwise, it
shall emit an error to the current port, and return #f. *-result-timing-values-refRESULT-OBJ
passed to it if it is indeed a true result object. Otherwise, it
shall emit an error to the current port, and return #f.#f when no timing was performed, for the test
corresponding to this test-result-object.(6 1) indicates 6 milliseconds spent in user code and
1 millisecond spent in system code. all-testpackage-results-true?(test-package-result-result-ref PACKAGE-RESULT-OBJ) and returns
#t if every single contained result in that list had been true, or #f
otherwise. all-testcase-expectations-true?(test-case-result-result-ref CASE-RESULT-OBJ) and returns
#t if every single contained result in that list had been true, or #f
otherwise.Analysis of the Result Tree
Once a result tree has been evaluated and constructed in memory, what do you do with it? Well, you can do anything you want with it if you choose to write the analysis or output generation functions and pass it the evaluated result tree, and this is, in fact, encouraged. However, usually you just want to print out the tree in a human readable format so you can check things before the serious analysis code gets written. So, to save work for the test suite designer, a tiny API has been written to produce human readable output given a tree of results rooted in a single test package. Of course the single test package may have many other test packages and test cases embedded within it.
Statistics Generation API
(stat-compute-statistics RESULT-TREE) (stat-packages-ref STATISTICS) (stat-package-warnings-ref STATISTICS) (stat-packages-passed-ref STATISTICS) (stat-packages-failed-ref STATISTICS) (stat-packages-timing-ref STATISTICS) (stat-packages-terminated-ref STATISTICS) (stat-cases-ref STATISTICS) (stat-case-warnings-ref STATISTICS) (stat-cases-passed-ref STATISTICS) (stat-cases-failed-ref STATISTICS) (stat-cases-timing-ref STATISTICS) (stat-cases-terminated-ref STATISTICS) (stat-all-expectations-ref STATISTICS) (stat-all-expectation-warnings-ref STATISTICS) (stat-all-expectations-passed-ref STATISTICS) (stat-all-expectations-failed-ref STATISTICS) (stat-all-expectations-timing-ref STATISTICS) (stat-single-expectations-ref STATISTICS) (stat-single-expectation-warnings-ref STATISTICS) (stat-single-expectations-passed-ref STATISTICS) (stat-single-expectations-failed-ref STATISTICS) (stat-single-expectations-timing-ref STATISTICS) (stat-tol-expectations-ref STATISTICS) (stat-tol-expectation-warnings-ref STATISTICS) (stat-tol-expectations-passed-ref STATISTICS) (stat-tol-expectations-failed-ref STATISTICS) (stat-tol-expectations-timing-ref STATISTICS) (stat-equiv-expectations-ref STATISTICS) (stat-equiv-expectation-warnings-ref STATISTICS) (stat-equiv-expectations-passed-ref STATISTICS) (stat-equiv-expectations-failed-ref STATISTICS) (stat-equiv-expectations-timing-ref STATISTICS) (stat-todos-ref STATISTICS) (stat-todo-warnings-ref STATISTICS) (stat-skips-ref STATISTICS) (stat-skip-warnings-ref STATISTICS) (stat-glosses-ref STATISTICS) (stat-gloss-warnings-ref STATISTICS) (stat-terminations-ref STATISTICS)Output Generation API
These procedures help in the printing of test result reports.
THUNK is a zero argument procedure to print a report.
INDENT is the number of spaces to left indent. Use
test:write-object when printing a data object.
(test:output-apply THUNK [PORT|STRING|#t]) (test:write-object OBJECT) (test:display-objects [OBJECT ...]) (test:display-objects-newline [OBJECT ...]) (test:display-indent INDENT) (test:display-indented-objects INDENT [OBJECT ...]) (test:display-indented-objects-newline INDENT [OBJECT ...])Output Style API
These functions will display a rendering of the
RESULT-TREE and return the toplevel package result. The
default destination is the (current-output-port). Otherwise
the destination specifies how the output is targeted. A
PORT destination will display to the port. A
STRING destination will display to the specified file.
Should the file exist the result is implementation dependent. A
#t destination will return the output as a string.
(output-style-human RESULT-TREE [PORT|STRING|#t]) (output-style-html RESULT-TREE [PORT|STRING|#t]) (output-style-compact RESULT-TREE [PORT|STRING|#t]) (output-style-minimal RESULT-TREE [PORT|STRING|#t])Example Usages of the Test Suite Infrastructure
This section contains some simple examples of how to author test suites.
Here is a simple example:
(let ((result
;; output-style-human function requires a single test package
;; to encapsulate everything.
(test-package "Arithmetic Operators" pd pe
(test-case "Testing '+'" d e
(expect-equal "Adding two positive numbers" 2 (+ 1 1))
(expect-equal "Adding two negative numbers" -2 (+ -1 -1))
(expect-zero "Adding positive and negative" (+ -1 1)))
(test-case "Testing '-'" d e
(expect-zero "Subtracting two positive numbers" (- 1 1))
(expect-zero "Subtracting two negative numbers" (- -1 -1))
(expect-equal "Subtracting positive and negative" -2 (- -1 1))))))
(output-style-human result))
The above example, when evaluated, will produce some human readable output
and the a #t value as the result of the top level package. The result
variable contains the tree of evaluated expectations, test cases, and the
package arranged in a hierarchy extremely similar to the nesting of the
above macros. If you desire to manipulate the result tree yourself, you
may use the various APIs to manipulate the various results. Please see the
implementation of (output-style-human) in the
test-infrastructure source to for an example of this.
The variables: pe, e are the escape functions you may use with the
(terminate) procedure if you wish to abort the above code
somewhere.
The variables: pd, d allow use of a "destructor" object which allows
you to run side effecting functions at the "finishing" point of the evaluation
of the test case or test package. These functions are run no matter if the code
succeeded correctly or not, so be careful to manage the destructor object
carefully so you don't perform unwanted side effects. The names of the
destructor objects you supply are lexically scoped in the bodies of the
test case or test package.
Now, here is some example output that (output-style-human) might
generate with the above testing code:
Test Package: Arithmetic Operators
Test Case: Testing '+'
Expectation: Adding two positive numbers
Expect equal
Expected: 2
Unevaluated: (+ 1 1)
Evaluated: 2
Pass: Adding two positive numbers
Expectation: Adding two negative numbers
Expect equal
Expected: -2
Unevaluated: (+ -1 -1)
Evaluated: -2
Pass: Adding two negative numbers
Expectation: Adding positive and negative
Expect zero
Unevaluated: (+ -1 1)
Evaluated: 0
Pass: Adding positive and negative
Pass: Testing '+'
Test Case: Testing '-'
Expectation: Subtracting two positive numbers
Expect zero
Unevaluated: (- 1 1)
Evaluated: 0
Pass: Subtracting two positive numbers
Expectation: Subtracting two negative numbers
Expect zero
Unevaluated: (- -1 -1)
Evaluated: 0
Pass: Subtracting two negative numbers
Expectation: Subtracting positive and negative
Expect equal
Expected: -2
Unevaluated: (- -1 1)
Evaluated: -2
Pass: Subtracting positive and negative
Pass: Testing '-'
Pass: Arithmetic Operators
#t
output-text-compact by Patrick Brannan.
Copyright (c) 2000-2005, Peter Keller - All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following
conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following
disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of the author nor the names of its contributors may be used to endorse or promote
products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS
OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.