The AMUC (Associated Motion capture User Categories) project consisted of building a prototype sketch retrieval client for exploring motion capture archives. High-dimensional datasets reflect the dynamic process of motion capture and comprise high-rate sampled data of a performer's joint angles; in response to multiple query criteria, these data can potentially yield different kinds of information. The AMUC prototype harnesses graphic input via an electronic tablet as a query mechanism, time and position signals obtained from the sketch being mapped to the properties of data streams stored in the motion capture repository. As well as proposing a pragmatic solution for exploring motion capture datasets, the project demonstrates the conceptual value of iterative prototyping in innovative interdisciplinary design. The AMUC team was composed of live performance practitioners and theorists conversant with a variety of movement techniques, bioengineers who recorded and processed motion data for integration into the retrieval tool, and computer scientists who designed and implemented the retrieval system and server architecture, scoped for Grid-based applications. Creative input on information system design and navigation, and digital image processing, underpinned implementation of the prototype, which has undergone preliminary trials with diverse users, allowing identification of rich potential development areas.
Formulated in response to the AHRC–EPSRC–JISC joint call for e-Science pilot demonstrators, the AMUC (Associated Motion capture User Categories) project was a collaborative interdisciplinary initiative focused on the application of e-Science technologies in the arts and humanities. As a corollary, the project aimed to provide technology developers from computing and bioengineering with fresh insights into such critical e-Science issues as metadata processing, non-text query systems and possible linkages between parallel taxonomies. By concentrating on the concrete demands of prototype tool implementation, AMUC sought to strengthen relationships brokered across distinctive practices and research domains. At the more specific level of motion capture database use, AMUC sought to engage a diversified end-user community in a collaborative design process to open up new approaches to data search and retrieval requirements.
The refinement of Grid computing environments to make best use of distributed affordances and to operate readily across discrete disciplines largely depends on the creation of new kinds of query languages. These must be sufficiently catholic to embrace divergent user cultures, yet sufficiently tuned to specialist requirements to be effective for database search activities. Text-based query languages are loaded with tacit values embedded in semantic systems that need prior contextualization and ‘translation’ before they can offer equally accessible frameworks for users from multiple disciplines. These constraints have prompted rising interest in image-processing techniques to automate feature selection (pattern recognition, change detection, scalable algorithms, clustering and association rules, etc.), notably for exploring data streams. Parallel to advances in computing science, and as pivotal to e-Science thinking, is the emergence of humanities studies that tackle the ways in which digitally processed dynamic data challenge notation-based recording systems by enabling continuous capture or scanning (Rotman 2002).
The merits of closed or open systems to describe corporeal movement underpin choreographic notation debates dating back hundreds of years (Hutchinson Guest 1989; Louppe et al. 1991). Sixteenth century tablatures by Thoinot Arbeau, presented as a canonical grammar differentiating discrete bodily movements that, as parts of speech, we learn to master and expressively combine (Norman 2006), contrast with dynamic affordances provided by modern imaging techniques, where discrete logics are supplanted by continuous scanning and sampling. The AMUC project drew on historical and contemporary perspectives to respond to the challenge of optimizing data retrieval by dynamic feature extraction and adaptive query processing.
Motion capture recordings produce high-dimensional datasets that are continuous through their reflection of a dynamic process, and discrete through their division into sampling intervals or rates. Because multiple kinds of information can potentially be culled from the tracking and processing of markers on a moving body, this area of research represents particular interest for the invention of novel index-based retrieval tools, where annotated data features can be identified in response to multiple, composite and potentially adaptive query criteria. Moreover, because motion capture is employed for a broad spectrum of activities, a prototype retrieval tool potentially allows engagement with a richly diversified population, ranging from artists integrating real-time tracked data in live performances to bioengineers needing reproducible data to analyse and model body movements for diagnostic purposes.
(a) Overview and cross-cutting work methods
During the six-month project time frame, discipline-led inputs became increasingly intertwined. Methodological information is provided below for work packages encompassing: motion capture data collection and processing; file parsing; review of feature recognition software and sketch-based graphical user interfaces (GUIs); creation of string matching algorithm; specifications for Grid architecture; integration of software, sketch and display system; and implementation of a preliminary evaluation framework.
Interdependence of tasks heightened awareness of differences in languages and methods, e.g. in ways of describing and expressing motion (art form, mechanical action, mathematically defined process, etc.). To enhance understanding of these differences, documents drawn up in lay terms and using illustrations (diagrams, photographs and sample files) articulated otherwise tacit knowledge by describing technical motion capture recording procedures, biomechanics processing steps for capture data, Grid specifications for feature extraction on stored data, and logic steps governing search and retrieval actions. Hard- and software requirements were specified for each task, providing an inventory of resources to integrate into plans for future use of the prototype. Video, photographic, freehand sketch and dance notation traces of motion sequences, and textual accounts of recording sessions, were collated with project status reports (figures 1 and 2) Documentation was kept on a wiki used as a shared resource, making sector-specific imperatives easier to assimilate across the team.
(b) Motion capture-recording sessions
Recording sessions involved performers from dance, music, circus and martial arts. Alongside data harvested from a Vicon optical capture system (v. 4.4 with six infrared cameras) by a bioengineering researcher working in tandem with the Vicon operator (also a digital film specialist), records were established using synchronized reference video, photographs, textual accounts and freehand sketches. Comparison of keywords used to describe, store and access movement data raised annotation questions, basic bioengineering references such as standard marker emplacements being queried by artists less interested in normative anatomical templates. Similarly, trajectory labelling preferences differed markedly: movement traces derived from capture points provided performers with useful displays of gesture quality, whereas this visual information was considered superfluous if not obtrusive by bioengineers.
While only a small sample of recorded data was integrated in the sketch-based retrieval prototype, research tuned to language differences encountered across the wider community informed thinking on distributed, interoperable data resources for diverse users. Findings displayed by the pilot retrieval tool consisted of pre-indexed reference video sequences, but parallel records established during capture sessions were seen as informative elements for future database integration—dynamic Vicon-generated sequences, sketches, photographs and automatically processed dance notation (figures 3 and 4). Discussion about the inclusion of such resources in turn raised questions about designing the Grid architecture to make available a raft of associated software packages (notation, image processing, etc.). Ideally, this was viewed as a platform allowing user clusters to gradually aggregate digital tools likely to enhance their interactions with and explorations of the core motion capture database.
(c) Biomechanics data processing
Following data collection, the Vicon workstation was used to reconstruct marker trajectories. A custom-written marker file was attached for a given capture subject, and marker positions were annotated using the reference anatomical template. Labelled data were filtered and gaps automatically interpolated by the Vicon system. Data were then processed using custom-written code from Bodybuilder (v. 3.55), a biomechanics software package that enables data manipulation to create kinematic and kinetic models, model body segments and joints, and calculate angles, forces and moments. Bodybuilder can be used to look at graphs of trajectories, edit files in C3D format, display results and generate macros (e.g. for missing marker substitution or segment visualization). After processing to locate joint centres and define bone embedded frames for each limb segment, model outputs can be exported to text files for further analysis and graphical visualization (figure 5). This affordance, which is decisive for motion capture exploitation by the bioengineering community, can be optimized in a Grid environment by database links to an array of analytical and display tools.
(d) GUI and software design
Processed datasets were indexed and integrated into the test archive, and automatic graphic recognition was developed to match sketch input against searchable features in the database. The proposed search process consists of identifying a meaningful stream or channel of data, then identifying features in the input and mapping them to features in the dataset, where useful data channels include position, velocity, acceleration, symmetry, potential energy, etc. Features are identified by the indexer, which detects, for example, a rapid change, a peak or inactivity, allowing quick searches to rule out having to process all the data for each enquiry. In summary, the retrieval tool allows the user to: (i) input a sketch with a mouse/pen/on-screen tablet, (ii) choose which data streams are searched by means of a filter, (iii) choose which indexer is used (i.e. which features of the data are searched), (iv) activate a search against the existing (pre-indexed) dataset, (v) call up results ordered by closeness of match, and (vi) view a video sequence corresponding to each match (figure 6). This sequence of operations was kept very simple to enhance legibility and generic uptake for an unusually wide user group motivated by diverse data retrieval imperatives.
(e) Specifications for Grid architecture
In order to support potentially large-scale acquisition, storage and search of motion capture data, investigation of Grid architectures focused on questions of data storage and feature indexing for retrieval purposes. In traditional environments, data captured on a PC are stored locally or uploaded to a shared network drive, whereas Grid-enabled data storage offers a service-oriented system to upload data files and store them in a resource designed to authorize feature extraction on the data, key features thus identified being placed in an index database.
The AMUC data retrieval process uses as input a gesture-based search pattern consisting of a trace across a touch pad, for which time and position parameters are recorded. The search gesture is passed through a feature extraction/identification algorithm similar to those used to index the captured data. This generates a set of features contained within the search gesture, which is used to directly query the index database generated when the original motion capture data were uploaded. Here, the Grid architecture is designed to break down and distribute components of the indexing task among multiple compute nodes, and to organize the indexing database workflow so that updates feed into future feature-based searches. Potential integration of specialist programs likely to enrich and extend use of the tool (bioengineering analysis, choreographic notation, animation software, etc.) was discussed as part of the Grid design process, as well as during data-gathering operations (see §3b).
4. Preliminary evaluation and results
While the scale of the AMUC pilot ruled out a standard functional software evaluation, results are based on heuristic participatory design-based tests implemented to elicit feedback to inform discussion and a framework for future iterations. Trials with stakeholders representing the different project sectors, held in Culture Lab and at external dissemination events (All Hands National e-Science Conference, Nottingham, September 2007; EVA–JISC Workshop on New Directions in e-Science and Visual Perceptions, London, July 2007), yielded predictable variations in terms of constraints and related notions of research applications and priorities. For example, artists keen to integrate motion capture sequences retrieved on the fly for live performances posed different data management and interoperability issues from human–computer interaction developers interested in signature-type reproducibility related to sketch-based querying, or bioengineers seeking automated annotation to compare specific motions. Diversity of responses was foreseeable, given the interdisciplinarity of the research group; at the same time, the evaluation provided a useful additional layer of shareable information about work environments and requirements, pragmatically grounded in use of a single cross-sector tool.
Technical insights gleaned from preliminary tests pertained to the following issues. (i) Quality of the initial data: as this conditions indexer effectiveness, it is crucial in an open Grid-based system where manual motion capture data clean-ups may not be as readily performed. (ii) Nomenclature for defining queries grounded in multiple disciplines: the naming of indexers is determinant for user engagement in the absence of common, pre-existing vocabulary. (iii) Query input system: the sketch interface provides just one input device, which might be enhanced by adding text indexers, physical devices with direct joint articulations, machine vision-based techniques, etc. (iv) Legibility of the algorithmic process matching queries and data: optimal uptake requires deeper user understanding of how queries are represented, and the possibility to gauge accuracy of retrieved data against input queries (figure 7).
Overall, the tool was ajudged refreshingly engaging, if not playful: despite the abstract nature of data string matches governing input/retrieval correlations, the ingenuousness of a process whereby graphic strokes call up moving images of live performers proved immensely appealing. Users spontaneously investigated the system, carefully varying and reproducing sketch inputs, and modifying indexer and filter options. By generating visual, graspable correlations between indexer terms and retrieval data, the tool promoted dialogue between diverse end-users from the arts and sciences, offering a platform for ‘epistemic action’. Defined by Klemmer et al. (2006) as the manipulation of artefacts to better understand a task context, epistemic action here consisted of developing novel approaches to the prototyping of interdisciplinary structured query language systems.
5. Applications and research perspectives
Tests by a diverse range of practitioners led to the projection of an equally diverse set of potential applications, along with more generic development proposals. Distribution of the database over multiple physical data stores linked via a Web query service is a possibility that appeals to all user categories, as does the notion of incremental data indexation by each data store (e.g. for Web 2.0 motion capture uploads). For both bioengineers and performing artists, these applications are likely to enrich the database and inspire new search modalities, and thus are of keen interest. Similarly, creation of an application programming interface to allow authoring of indexers, which can access multiple data channels and return further indexable features, is seen as a way to enhance a shared resource. Customizable indexer development would broaden possible applications and client groups, as would the generation of complex search patterns using conjunctive queries and previous matching history data. Such features offer valuable incentives for distributing a novel resource among disparate users. More importantly, they can elicit new shared behaviours among previously dissociated groups. While offering a practical means of exploiting motion capture information, mapping the dynamics of freehand sketching across to visual sequences, AMUC focuses on how we constitute and interrogate databases. By assuming that there is no single or optimal data retrieval method for a heterogeneous population, and overriding formal specialist query languages to build commonalities at the basic level of search behaviours, this tool combines pragmatic and epistemological strengths.
More specifically within the arts and humanities user community, integration of multiple sketch types into the GUI (e.g. sketch notation systems) and diversification of input devices (e.g. making the data seeker's motion-captured gesture a way to explore archived material) would enrich the tool and client base. A further and more radical step to incite arts and humanities e-Science uptake would consist of developing applications for experimental art making that reach beyond conventional archival and documentation uses. Thus, for example, situated expressive gesture from a rehearsal or performance could be captured to a Grid-enabled data source, and artists could use this material in combination with a Grid service provided by bioengineers to select fragments with particular mechanical properties to drive a piece of digital art or artefact, or allow gesture-based search of the activity through sketching.
AMUC enriched our understanding of the complexities of database indexing and querying issues. Use of a sketch-based retrieval system allowed engagement with and of a broad user spectrum, opening up novel perspectives for devising shareable annotation and mapping approaches. Our methodology and results emphasize the often underestimated power and transferability of tacit knowledge, yet we also learned the importance of recognizing specialist knowledge not amenable to interdisciplinary transfer: ostensibly, the simple features that one user seeks to extract from a database may be complex or irrelevant for another. As an experimental platform, the AMUC tool favoured the initial encounter that is a prerequisite to the eventual shared shaping of work languages and methods. While the prototype can be considered as an eminently tangible system in its implementation of an algorithm to match input signals against those of indexed data streams, and of a novel interface for visualizing this input and related output, it additionally and importantly offers significant conceptual value by elucidating the context that determines information retrieval. Viewed in this light, it can be seen as forming part of the iterative thinking through prototyping approach that is a hallmark of innovative interaction design.
We are grateful to our AHRC–EPSRC–JISC funders who supported this project as part of their joint initiative to develop e-Science pilot demonstrators. Our thanks to the performers who contributed to motion capture data recordings: Tamara Ashley, dancer–choreographer and Laban notator; John Ferguson, improvisational musician; Heart Performance Company, improvisational dancers; James Mooney, musician and composer; Sally Jane Norman, kobudo (traditional Japanese sword); Rick O'Shea, juggler and acrobat; Gretchen Schiller, dancer/choreographer; and Peter Wiegold, conductor and composer.
One contribution of 16 to a Theme Issue ‘Crossing boundaries: computational science, e-Science and global e-Infrastructure II. Selected papers from the UK e-Science All Hands Meeting 2008’.
- © 2009 The Royal Society