Authors tended to come to critiquing with a clear picture of which pieces of evidence they thought were right and a set of mistakes they thought students would make (usually only those in the demo path). An example of the most complex pre-tool constructions were those found in Rembrandt.
A single section from the evidence for one of Rembrandt's scenarios,
Young
Woman at a Half Door can be found in
Figure 6.2. We see that the authors have gone to a great
deal of trouble to outline what is an acceptable correct answer for
this claim. The student must have two or more points from the first
numbered set and two or more from any of the second five sets.
Rules and conditions reflected this structure fairly easily, but failed to help authors with how the client would respond if this evidence were missing. For instance, if the student failed to list points from any of the available lists in the second part, what would the museum curator say?
This kind of lack of planning was typical for authors, and it seems to be a symptom of the lifecycle of GBSes--authors need only a list of tests and one example of a mistake students might make to build a convincing demo path, and rarely are prepared to deal with more complex problems.
In the case of Rembrandt, one author entered all of the rules, and she took a fairly ad hoc approach to what the curator says in response to mistaken claims. When the application was beta tested, testers quickly found that the scenario rules were very incomplete--not only did the rules fail to correctly anticipate the kinds of problems students had, they gave such unpredictable and unstandardized responses that testers were very frustrated and confused. The remediation more served as an impetus to explore the rest of the application than as a chance to learn about their mistakes.
Nutrition has taken the same ad hoc approach as Rembrandt--although authors had a clear picture of what it was students needed to succeed, it remains to be seen if their set of remediations is useful or merely mercurial. Both the Nutrition and Rembrandt teams have chosen to allocate a significant amount of time (2 months) to the revision and testing of their critiquers.