User evaluation

We deployed INDIE to many teams of authors, and carefully tracked the progress of five teams. Each team was made up of:

• One to three content specialists (non-technical staff employed by ILS or else graduate students),

• One or more artists (professional staff employed by the ILS), and

• zero or more professional video staff (frequently, students and staff did their own video without professional support).

Although all of these users were experienced with using programs like word processors or even other GBS tools, only a handful of graduate students had any programming experience, and those students never wrote any code or extended INDIE in any way.

We tracked these teams' progress from numerous sources, including:

• The files they created with INDIE: As authors incrementally added information to their tool, they created numerous versions of their application. We combed this data for clues to how authors used INDIE.

• Exit interviews: We conducted interviews of the teams that built INDIE as they reached an end point (usually the majority of the staff leaving the development team to work on new projects).

• Daily or weekly interaction with the teams as mentors and teachers: Many of the authors, as mentioned above, were students in the graduate program at ILS. The INDIE development team worked with them in a teacher/student relationship. In professional projects built by ILS staff, the INDIE development teams took a consulting role as technical advisors. These two kinds of relationships made it possible to get a first-hand view of authors' interactions with INDIE.

How did INDIE fare? Four major projects were implemented in the various versions of INDIE, summarized in Table 1.1. Authors were able to leverage the different parts of INDIE to build very complex systems that have had some success in the real world, most notably Is It a Rembrandt?, which was deployed to art history students at Northwestern University.

 

Table 1.1: Development, sizes, and lengths of projects in INDIE.

Project	Duration	Status	Num. of objects
Immunology	9 months	1 scenario	1318
Volcano	6 months	Demonstration	939
Rembrandt	1 year	3 scenarios	3145
Nutrition	1 year	3 scenarios	5863

 

These GBSes had thousands of different parts, and extremely complex interfaces. The two projects that were shipped to parties outside of ILS (Rembrandt and Nutrition) had 3-4 gigabytes of video.

Certainly, we have fulfilled one of our major goals, which was to get GBSes built without programmer intervention. With only a part-time team of three programmers doing tweaks, small extensions, and fixes, our non-programmer authors were able to build extremely complex GBSes in six months to a year.

The projects in Table 1.1 are listed in chronological order from top to bottom. Inspection shows that the projects are getting increasingly complex per unit time spent on them; Nutrition is an order of magnitude more complex than Immunology for roughly the same amount of development time.

Despite this gain, we are still not seeing an overall reduction in the amount of time it takes to fully implement a GSB. Authors aren't able to produce a GBS of the size of Immunology in three months. Learning how to compress GBS development time instead of enriching productivity will be one of the major focuses of future work in INDIE's field.