Building good GBS tools

Now that we know what INDIE is and what it does, how can we evaluate it? Given the goals stated above, we suggest that INDIE should be:

• Flexible: The tool should work for more than one domain and for domains with different interactions. Immunology had a certain script by which the students collected information, ran tests, then collated those results. Rembrandt had another one, and Volcano Investigator had a completely different one centered around geological tests. INDIE should have flexible interfaces and a model capable of representing many different domains.

  Metrics for measuring flexibility are:

  • Is INDIE used in projects in many and vastly different domains without extensions or re-programming?

  • Does the interface support interactions that look and feel different?

• Complete: It is easy to devise a tool that works well for very visual demo paths and can't be used for shippable applications. INDIE needs to support start-to-finish development, so that authors can build their first storyboards in INDIE, and then the delivered system is entirely in the same tool without requiring vast reimplementation along the way.

  Metrics are:

  • Do authors have to translate their data from one tool to another across every stage in GBS development (e.g., one tool for storyboards, one tool for prototypes, one tool for completed systems)?

  • Do authors use any other software tools besides INDIE for their demonstrations because INDIE is too slow or inefficient for their needs?

• Scalable: It should be easy for authors to incrementally add information to their programs as they construct it. The application should work when there is very little authoring done (such while authors are still in early design stages), when there is enough for a full demonstration, and when the program is complete.

  Metrics of this include:

  • Did authors have to create organizing information outside the tool before entering data during each stage of development? For example, if authors need to build a critiquer in a prototype, must they structure the entire critiquer in Excel or on a whiteboard before entering it into the tool?

  • Did authors ever have to backtrack and re-enter data because each new piece of data they create requires them to restructure their old data?

• Interface independent: Authors need to be able to construct a working version of their model without having their interface implemented. This implies a clear separation of representation and interface.

  A good metric is:

  • Are authors are working in parallel, e.g. some group of authors and artists can work on the interface while another group of authors works on the content for the model?

• Efficient: To get their content running in an application, authors should only have to do as much work as is commensurate with their level of design. This means that if they have a few pictures and a few movies and nothing else, they can enter these into the tool and easily see them shown in some defined order easily without constructing the entire GBS. At the other extreme, authors should not have to do lots of repetitive work to complete their GBSes--if they have several hundred questions that will someday appear on buttons, they should not have to individually construct and position several hundred question buttons with a standard button editor.

  To measure INDIE's success on this attribute, we are essentially looking for rocky places in GBS development such as:

  • Did adding any one kind of model object (like, say, questions), cause authors to have to build a like amount of other, peripheral objects (such as buttons to represent the questions)? If so, the tool is not efficient.

  • Was there any terribly repetitive task authors had to do that could have been automated?

• Accessible: A GBS tool's target audience is made up of largely
  non-programmers, which means authors with probably no formal programming education (or, as we discovered, knowledge about formal logic). With INDIE, our audience was experienced non-programming authors--authors who had several months or years' experience building GBS-like applications and had some knowledge of design.

  The main metric is:

  • Who is using the tool, programmers or non-programmers? Are these authors required to write code?

• Effective: A tool might have all of the previous attributes but lack an important one--that the applications it builds be of high quality. This can include (but is not limited to): student satisfaction, intelligent and responsive interfaces, and fulfillment of educational goals. A GBS should mix learning, student-centered exploration, and fun together seamlessly, and a GBS tool should make that possible.

  Metrics are:

  • Do students like INDIE applications?

  • Are they learning stuff as well or better than they would in other situations?

  • Does INDIE allow authors to build very complex systems as fast or faster than other tools that could be used for the same purpose?