One of the ways in which the SEMIA project team has sought from the start to further develop its key concerns and objectives, is through the organization of a number of workshops involving representatives of the three ‘user groups’ the project addresses.
The first workshop, which took place on 7 February 2018, brought together media scholars and heritage professionals, along with a number of artists who, as part of their creative practice, repurpose moving image materials in different ways. The workshop’s objective was twofold: on the one hand, it sought to discuss the current ‘state of the art’ in visual analytics research for (archival) moving images, as well as to share amongst project members and other participants some basic knowledge of the underlying mechanisms for (automated) feature extraction and image analysis. On the other, the workshop sought to establish, through discussions with participants, possibilities and limitations in the use of such methods, specifically for media history research and heritage presentation. Our aim was for those discussions to provide input for the remainder of the SEMIA project, specifically in going forward with feature extraction (the first, but iterative phase of the tool development work).
The programme consisted of a combination of presentations, followed by responses, and two series of discussions with participants (first in smaller groups, then plenary). The choice of presentations was determined on the basis of a prior selection of image features identified as particularly relevant for the project – either because of their alignment with existing research interests, or for more pragmatic reasons (such as, the labour-intensiveness of extraction and analysis in relation to the project’s length). Those features (colour, specific kinds of shape, specific dimensions of movement, texture) and tasks, introduced at the beginning of the workshop, varied in status: from essential to the project to desirable (but to be tackled only if time allowed); or, from aligning with prior research to more experimental. Speakers and their respondents were chosen in function of their different takes on matters of extraction and analysis: some were scholars, some artists, and some had extensive experience of using extraction and analysis methods (e.g. in media history research) while others took a more exploratory or even speculative stance. The purpose of their interventions was to assess whether choices made prior to the event, made sense – of if not, to generate ideas as to how to tweak them.
Project leader Giovanna Fossati opening the workshop
The workshop was productive in different ways. First, the combination of presentations by Nanne van Noord (SEMIA project member) and Matthias Zeppelzauer (former collaborator of the Vienna Digital Formalism project), benefitted our understanding of the principles behind, and history of, visual analytics, and the differences between working with purpose-designed, task-specific algorithms (feature engineering) and deep-learning techniques. Zeppelzauer’s contribution also provided more insight into computer vision thinking about visual features, and how such features can best be conceptualised – also for the purpose of determining tasks in the retraining of existing networks for deep-learning. (A post elaborating on these definitions and distinctions, and their relevance to SEMIA, will follow in due course.) Second, the presentation by Adelheid Heftberger (likewise, of the completed Vienna Digital Formalism project) prompted us to consider the importance of consensus among project members about the vocabulary they use in communicating about project objectives and procedures – an advice that has since proven very helpful. Third, the presentations by Barbara Flückiger and Gaudenz Halter (of the Film Colors: Technologies, Cultures, Institutions project at the University of Zürich), Frank Kessler and Danai Kleida (involved in a small-scale film movement annotation project at Utrecht University, in collaboration with the Media Ecology Project at Dartmouth College) and once again, project member Nanne van Noord, brought us up to date on recent initiatives in (manual and/or automatic) feature extraction and analysis, as part of ongoing media, resp. art history projects with related concerns. And finally, the presentation by media artist Geert Mul above all proved inspirational in terms of the sort of revelations visual analysis techniques can engender, and taught us that it might make sense to toggle in the project between conceptually separating out the features that extraction and analysis centre on, and considering them in their mutual relations.
The group discussions later on in the day, focusing on one or more features each, further confirmed this. Aside from revealing a great deal about the working habits and research interests of participants, they also taught us about differences in understanding of the nature of, and relations between, image features. Most importantly, they gave us a better idea of what potential users, specifically researchers, are after, in gaining access to digitized collections. Most crucially, such users want to be taken to places (objects, analogies, patterns) they hadn’t tought of before – even if this means that the relations established in the process cannot necessarily be pinpointed exactly, or explained with recourse to empirical evidence. This strengthened us in our assumption that there is indeed a need for more exploratory ways of navigating through collections, if only just as a trigger for more traditional form of search later on.
– Eef Masson