Wolff Daniel Dobson
The Institute for the Learning Sciences & Department of Computer Science
Northwestern University
Evanston, IL 60201
+1 847 491
3500
wolff@cs.nwu.edu
Christopher K. Riesbeck
The Institute for the Learning Sciences & Department of Computer Science
Northwestern University
Evanston, IL 60201
+1 847 491
3500
riesbeck@ils.nwu.edu
INDIE, developed and used at ILS, is an attempt at such an educational-software-building tool. The systems INDIE builds have enough intelligence to be able to offer the student a variety of choices and to critique those choices. The topic of this paper is the problem of building tools for authors that scale easily from simple interface interactions to a well-represented domain model without significant re-implementation at each stage of design and without significant programmer involvement. Both of these issues, as demonstrated with the INDIE tool, tend to be difficult problems when building educational software.
Figure 1: Lake temperature results in Volcano Investigator
As the student works on the problems presented, the system tracks what's happening. As needed or requested by the student, the system uses experts captured on video to guide, coach, critique and give real world examples of similar situations. A GBS is not a typical classroom setting; it is supposed to be like learning on the job, only better because experts are available all the time for help and review.
The INDIE tool builds a certain subclass of GBS learning environments that centers around diagnosis tasks. Students are given an urgent problem that needs diagnosing (such as a patient with an unknown illness), and then given tools to gather information about the case (tests to run on the patient, documents to inspect, and so forth). After investigating the case, students make a claim about the case and build a report from the evidence they have collected to prove it.
Along the way, students can ask for help and browse a richly-indexed multimedia hypertext network called an ASK system[1]. The central idea in ASK systems is that all information is linked to other information via follow-up questions. For example, if an expert in a video clip mentions how volcanoes are caused by magma being forced to the surface during earthquakes, follow-up questions might be, "What is magma?" and "What causes earthquakes?"
After hearing an introduction, students can run six different tests on the volcano. Each one provides evidence, in the form of itemized points, that appears in the notebook to the side of the screen. Figure 1 shows an example of a student seeing the result of a lake temperature test.
After gathering this test-based information, students can submit their suggestion (evacuate now, don't evacuate, or evacuate soon) to the mayor of the town. They choose a claim, then drag points from their notebook to a report to support this claim. The mayor comments on the proposal, and then the students' work is critiqued by an on-screen co-worker. The critiquing mechanism (the "critiquer") comments on whether or not the claim is well-supported, looking for missing, irrelevant, or conflicting evidence points in the report (Figure 2).
Figure 2: Submitting a report in Volcano Investigator
More importantly, the model should allow for smooth scale-up. If a program is just a collection of screens with buttons connecting them, each addition of a scenario or a test would require linearly more information. Instead, we'd like development times where building the first scenario or test takes the longest, and building each additional scenario or test costs asymptotically less.
With this definition of INDIE and its intent, we can now answer the question, how do INDIE projects get built and who builds them?
1 Pre-tool. Authors collect information about their domain from experts and books in preparation for doing a full design. At this point, they usually work on white boards or notebooks.
2 Storyboard: Authors concretely envision what the program will look like. They use paper or programs like Powerpoint or Director to make a linear sequence of screens, each screen showing one or two mouse clicks in the demonstration. Although the introduction to the scenario is usually complete, tests and remediation are usually extremely sketchy.
3 First Walkthrough: Authors animate their storyboard, this time doing a higher-fidelity mockup of the interface. For non-tool projects, this can be built in Director or Supercard, with working buttons, some movies, and functional art. The path through the program is still mouse click to mouse click and extremely linear. Some expert video is shot but most of it has not been indexed into the ASK system.
4 Prototype of first scenario: A nearly-complete scenario is built. The conceptually hard parts are designed (such as all the tests and the structure of the critiquing), but the application often supports only two or three different paths through the content (often corresponding to different demo lengths for different audiences). The application is implemented on the final delivery platform.
5 Shrinkwrapped system: This is a delivery-level system. Full content is available, all tests are running, the critiquer is complete, and enough supporting material has been entered such that all claims can be successfully supported or disconfirmed.
Applications will go through several iterations within stages and across them.
An important thing to note is that frequently projects never get past the first few stages. Projects developed by students, depending on the project size, last one to five months, which generally isn't enough time to build shrinkwrapped systems. Often prototypes are built as research projects or proposals, and after the proposal is demonstrated, the project never gets any further.
Investing time at the beginning of a project in model infrastructure can pay off later when finishing the program. However, if the project isn't going to be finished, why spend the time on infrastructure? This problem was key to the failure of our first tool, INDIE 1.0.
* The Problem Selector: Students choose a scenario to work on-in Volcano, it was which mountain to work on.
* The Lab: Students run tests and talk to experts in the field, collecting information.
* The Argument Constructor: Students make claims to the clients in the simulation, suggesting different courses of action or diagnosises (i.e. "The volcano is going to erupt tomorrow.")
* The Critiquer: Students hear feedback from the client, telling them what is wrong with their submitted claims and letting them ask questions about them.
Each one of these modules had a default interface layout-one or two "screens" for each, and then specific buttons for each kind of interaction: one button per scenario, one button per possible thing to say back to the client, and so forth. If authors wanted to add a scenario button, they would edit the problem-selection module to add another scenario, and a button would be automatically added to the scenario-selection screen. Our goal was to automatically supply the interface so that authors could concentrate on the content.
Authors found INDIE 1.0 very restrictive for several reasons. First, the tool built applications that didn't look the way they wanted them to look, and changing that "look" required the INDIE programming team to reprogram the interface.
Also, and probably more importantly, building the modules' model representations didn't work well. Authors, who had been drawing screens on paper and Powerpoint in the late stages of the idea phase, were suddenly forced to understand the INDIE model representation, and construct objects in it. For example, to make a specific test show a specific result, authors had to create:
* a test (such as finding the content of the gases emitting from a volcano)
* a set of scenario facts (such as that the volcano is about to erupt)
* a prop that could be tested (such as some volcanic gases)
* a table that specifies the results that occurs when the student runs this test on this prop in this scenario
Unfortunately, all that the authors wanted to say was, "When this button is pressed, play this movie." Defining interactions in the model (such as the results of tests, the relationship between available help and the tests, and getting critiquing running) required a fair amount of time, yet authors frequently rebuilt them due to design changes. In other words, rapid prototyping was difficult. Design iterations took full-time consulting from the INDIE programming team and much planning by authors.
On one project slated to use this version of INDIE, project-assigned programmers built an interactive walkthrough in Supercard rather than INDIE. The walkthrough had no model. They did this because, like all teams, they had a limited amount of time, and it was easier and faster to mock something up in Supercard than to use INDIE.[1]
The major issue for product delivery is to consider the total length of time on the project. If the project is completely re-implemented in a new tool, a very large portion of the time spent building the mock-up will be wasted, thus increasing the total project length. If authors started building on the final delivery platform from the beginning, the time spent on walkthroughs isn't wasted; the walkthrough becomes the basis for the shippable application. However, authors couldn't afford to spend the time up front due to short deadlines at the beginning of the project.
Even worse from the perspective of final delivery, authors are likely to (and did) design interface and model interactions that are difficult or impossible to implement for real as they felt no restrictions from their mock-up tool. Authors, especially non-programmers, who aren't familiar with the tool's strengths and limits will construct almost anything.
It was clear that we needed to get the authoring of storyboard-level interfaces into the hands of authors and out of the hands of programmers. At the same time, we had to preserve the authoring of the model information that the full-scale application would require.
* Authors had trouble getting the "look" they wanted while manipulating the model.
* Rapid prototyping of ideas was difficult and time-consuming.
* Authors would duplicate effort between building prototypes and their final products.
Essentially, we wanted authors to have free rein to prototype, but have their application scale-up to a completed application without having to lose the work they did on their prototype.
Figure 3: The INDIE tool visual editor
In this example, the author has created a "Go to map" button and is editing its attributes. The attribute editor is accessed simply by double-clicking the widget. Widgets can be grouped into "layers," which can be shown or hidden as a unit, giving authors the ability to make reusable screens and sub-screens.
* Show or hide a screen or some subportion of a screen, or else switch windows entirely.
* Play a movie.
* Make one or more ASK questions visible on screen as clickable buttons.
Examples of these from the Volcano project include:
* Show the result of a lake temperature test (show a picture and play a movie explaining it, then add a list of follow-up questions),
* Respond to the press of a button that says "Go to the lake" (show a group of widgets, then play a guide movie).
We supported this by allowing authors to create event response lists. Entries in a system-wide table associate an event (which can be a model- or interface-level event) to a list of screens, movies, and help questions that can appear. Each line in the table, which is a combination of event and a response to it, is called a "trigger." Any time an event occurs, the system looks up in the table for a matching event pattern and then executes what it finds in the response list. These triggers are constructed with a fill-out form that can be seen later in the paper.
The trigger-editing form is a simple visual programming system that most authors are able to grasp quickly. Authors don't have to learn any syntax or construct a list of commands from scratch; they can just fill in the blanks. The blanks are the same as those listed above: play a movie, show this list of questions, and show or hide some group of widgets on screen. If an author wants to show a sequence of responses (such as several introductory movies in a row), he or she can chain together several triggers at any length for very long interactions.
Figure 4: An ASK Modget
All INDIE applications have this sort of just-in-time ASK system. It consists of a list of clickable question buttons and an area where answers appear. When a question is "asked" (i.e. the button is clicked on), a media item (movie, text, or picture) response appears in the box above the list of questions. After the media item plays, a list of follow-up questions to it appear below it. These follow-ups have their own answers with follow-ups, which can lead students into a vast web of interlinked questions and answers.
Authors could build each question as a separate button widget whose trigger shows the appropriate movie and then shows other widgets, but this would become extremely cumbersome since there are usually hundreds of ASK questions in an INDIE project.
Instead, authors drag out a pair of widgets from the palette. The first is called an "answer-viewer." Answer-viewers show answers (i.e. any kind of picture, text, or movie). Any answer that is "shown" is played in the answer-viewer widget. There can be many different viewers and authors can, for each answer, choose in which viewer the answer will play.
The second widget is a "question-viewer." It is resizeable and supports different looks and styles of question button. Authors can specify multiple question viewers, and then associate each question with a viewer.
Answer-viewers and question-viewers are called in INDIE parlance "modgets," which is a term that combines "model" and "widget." Modgets are widgets that have a predefined interface interaction with the underlying model. They show what is happening in the model for certain kinds of objects: questions and answers, in this case.
When a student clicks on a question button, the answer for the question the button represents is declared shown. The answer's follow-up questions are declared visible. An answer-viewer then shows the correct media item while the buttons for follow-up questions appear in the appropriate question-viewers. This is shown in Figure 5.
Figure 5: Model/interface interactions when asking questions
Questions and answers are model objects that are authored separately with a database-like tool.
This is a big win for authors-they only have to author two widgets, yet they can scaleably add hundreds of questions to their ASK systems. This sort of scaleability is why we have a model in the first place, and what differs INDIE from Supercard and its ilk.
Notebook modgets (which represent working notebooks as well as reports) work in a similar manner to question-viewer modgets-each point is associated with a notebook, and when a point is made visible, a draggable text item is added to the viewer. Notebooks, aside from having editable properties like font and size and repositionable, automatically support dragging and dropping between them.
There are also glossary modgets, trash modgets (where students drag points when they want to remove them from a notebook), and argument-status modgets that show whether or not students can go on to the next scenario. All of these modgets indicate the state of the model, and are quite similar to model-view-controllers [2].
Some parts of an interface do not lend themselves easily to modgets. Modgets work best when there is a small section of the screen where some homogenous action will take place. It would be difficult, for instance, to come up with a modget that would represent tests. A test can be anything from looking through a microscope to feeling every part of a dog for injuries. Conceptually, tests are model objects with results that are based on which scenario the student is currently working on, which is a structure found in most domains. However, the look of the test is quite varied. For these, authors tend to depend on combination of widgets and conditional triggers.
As the program grows in complexity, authors need to make the interface respond conditionally to clicks, but one of the goals from the INDIE 1.0 experience is to make sure authors have to duplicate effort as little as possible.
There are several different kinds of conditionality in INDIE:
* Show different introductions and conclusions when scenarios start or end.
* Show different advice/critique movies depending on what is missing, conflicting, or irrelevant in the students' reports.
* Show different test results and help questions depending on what scenario the student is in.
At the first walkthrough stage, the buttons that would normally handle this sort of conditionality are "hardwired": "On CLICKED: `Run lake temperature test' button, play movie #45 and show these 3 follow-up questions." A scaleable authoring tool should make turning this interaction conditional a smooth process and require little or no re-authoring.
Currently, the "test" is in a push-button arrangement, as so:
Figure 6: A "hardwired" test
The author would like to instead have the internal structure look like this:
Figure 7: A test that uses the model
To do this, the author needs to do the following steps:
1. Create a temperature test object.
1. Create a scenario object, "Mt. Andrews Scenario."
2. Create a "Lake Temperature" test object.
3. Define a result object for the "Lake Temperature" test for the Mt. Andrews scenario called "15.2 [[ordmasculine]]C".
4. Make the trigger that plays the 15.2 [[ordmasculine]]C movie execute when the result created in step 3 is generated.
5. Make the button's trigger run the test.
Figure 8: A hardwired trigger
To make this test conditional on the scenario, the author would first make a scenario object. Scenario objects only consist of a name.
Then he or she would add a test object:
Figure 9: Test editor
The author then specifies a result. Results are named, such as "Temperature 15.2 [[ordmasculine]]C" and "Temperature 19.0 [[ordmasculine]]C". Each result object has an editor like this:
Figure 10: Result editor
For a result, authors specify what scenario will produce this result. Here, the "Mt. Andrews" scenario will cause the 15.2 C[[ordmasculine]] result to appear.
Scenarios, tests, and results are model objects that don't appear on screen nor do they have associated modgets. Authors make students aware of changes in these model objects through triggers with special kinds of events. These model events include:
* Test-result generated event
* Scenario chosen event
* Critiquing-rule fired event
In this example, the author can re-author his or her old trigger to execute when the 15.2 [[ordmasculine]]C test result is generated. The author opens the trigger editor, and with a visual editor selects the event "TEST RESULT: 15.2 C". The resulting trigger now reads:
Figure 11: Test result trigger
Finally, the author needs to connect the clicked button to the running of the test. Authors edit the trigger for the "Run temperature test" button event by pressing the "Edit Trigger" button again on that button's attribute editor. Given that currently there is no trigger for this button, one is automatically generated. The event slot is already filled out, but the rest of the trigger is blank. To fix this, authors go to the "Model Actions" section of the trigger editor:
Figure 12: Trigger with model action
A model action is a command that allows the interface to connect to non-modget model objects. In this case, authors would click on the "Create/Add" button and visually select the model action "RUN TEST" and, through a few dialog windows, choose the test object they have created. Now, pressing the "Run temperature" test will act conditionally depending on which scenario had been chosen.
Although in this example, the author has built a fair bit more model structure than the author had before, no work was duplicated. This author used his or her old trigger, and now adding a new scenario would only require authoring a new result and result trigger. Authors of multi-scenario applications reported they frequently made use of this flexibility.
We restrict all conditional behavior to occur through the model. Adding different kinds of conditional behavior to an INDIE application is then gradual and only done as needed, reducing the up-front load on authors' time, both for construction and for learning how the model works. And, since there is usually only one way to add such behavior, authors are forced at the early stages into building software that will scale up well.
* Immunology Consultant: Medical students are in the role of an immunologist with a difficult case to solve, and they use tests and interviews to figure out what is wrong with her.
* Is It A Rembrandt?: Art history students have to determine whether each of three paintings is painted by Rembrandt or is a forgery by inspecting the style of the painting, its provenance, the signature, and so on.
* Nutrition Clinician: Medical students have to determine what nutritional deficiencies each of 3 patients has, what the medical implications are, and what needs to be done to remove the deficiencies.
Authors of these projects were experienced ILS staffers who had worked on one or more GBS projects before starting with INDIE, but had no experience with the INDIE tool. They had no previous programming experience (most had liberal arts degrees) but had strong skills with applications such as spreadsheets and word processors. They tended to contact the INDIE developers frequently when starting their projects and getting familiar with the tool, and then around once a week once they were comfortable with the tool.
All of these systems have at least 15 different screens, ranging from introductions, tests, interviews, ASK zoomers, ASK browsers, report-building, and feedback. Each project took from 5 to 10 months to complete.
This table summarzies the relative complexity of each project:
Immun. Volcano Nutriti Remb.
on
ASK 217 150 600 620
nodes
Trigge 162 187 612 334
rs
Points 120 36 1000 514
Table
1: Relative sizes of four INDIE projects.INDIE application sizes are dominated by graphics and video. Rembrandt, for example, has around 60MB of pictures (largely uncompressed) and nearly 4 gigabytes of video consisting of nearly 500 clips of experts. Nutrition has 3 gigabytes of video.
INDIE 2.0 has additionally been used by four groups of graduate students, both Ph.D. and masters, in course projects. These projects tend to go through the third stage, First Walkthrough. Volcano Investigator was the most successful of these student projects, built by first-year masters students in an intensive project in 4 months. Others include KERMIT (ecology), Car Repair, and Clinical Monitor (drug testing).
Through the first few projects in INDIE, the INDIE programming team built many extensions to the tool as authors explored the "space" of diagnosis-style GBSes. Encouragingly, INDIE 2.0 has reached a fairly stable point, where there are enough options to satisfy typical needs and enough examples to show how to use those options.
* Most teams (usually groups of 3) had one person use the tool to build the interface, with the other students working on contacts with experts. Frequently other authors on the team would use the tool briefly for some targeted task, such as ASK system constructing and linking.
* Outside of using Powerpoint or paper for proposal-level storyboarding, all four teams used the INDIE tool as their primary development environment. This was one of our stated main goals of this project. Authors said that they liked INDIE precisely because it is so flexible in comparison to other more model-driven GBS tools available to them.
* It took teams from four days to two weeks to get comfortable with the tool to the point where they were able to construct their demopaths for presentations late in the Storyboard stage. This indicated that authors rapidly gained familiarity with the tool. Authors emphasized how easy they felt it was for them to get started with the tool.
* Two INDIE projects, Rembrandt and Nutrition, have been scaled up from initial demopaths to full-fledged multi-scenario shippable applications. These projects will be used by Northwestern University students beginning fall of 1997. Neither team reported significant re-authoring while moving from a single scenario to several scenarios (outside of a technical point about background images).
A general lesson can be extracted from our experiences with INDIE that applies to tools for authoring intelligent (or at least model-based) applications. The problem in such tools has always been balancing the long-term representational needs of the applications with the short-term needs of the authors. The authors want to prototype their interfaces quickly and easily, but typical interface builders let authors envision interactions way beyond what current intelligent systems can support. On the other hand, if you force authors to do up-front knowledge representation before interface prototyping, they will use other tools first and use your tool late and inefficiently, if at all.
Two techniques were used in INDIE 2.0 to resolve this conflict. First, by representing surface-level interface actions with event-based response triggers, authors can author interface actions as easily as in other systems. Later, as needed, they can incrementally refine these hardwired responses into model-based responses. Second, with modgets, authors can quickly and directly incorporate certain commonly used model-based widgets, such as evidence notebooks and lists of questions, ready for instant scale-up.
This work has been supported in part by the Defense Advanced Research Projects Agency, monitored by the Office of Naval Research, under contracts N00014-90-J-4117 and N00014-91-J-4092. The Institute for the Learning Sciences was established in 1989 with the support of Andersen Consulting.
2. Goldberg, A. Information Models, Views, and Controllers. Dr. Dobb's Journal (July 1990), 54-60.
3. Schank, R. Goal-based scenarios: A radical look at education. Journal of the Learning Sciences 3, 4(1994), 429-453.
4. Schank, R., Fano, A., Jona, M., and Bell, B. The design of goal-based scenarios, Journal of the Learning Sciences 3, 4(1994), 305-345.