International Symposium: Sensory Moving Image Archives – Visualization, Exploration and Reuse of Moving Image Data

We are very happy to announce that SEMIA will be organizing a two-day international symposium on visualization, exploration and reuse of moving image data next year in February, at the University of Amsterdam. The call for papers can be found below and is downloadable in pdf format here.


CfP: Sensory Moving Image Archives – Visualization, Exploration and Reuse of Moving Image Data

Date: 25-26 February, 2019

Location: Doelenzaal, University Library, University of Amsterdam, Singel 425.

Registration: Entrance is free but seats are limited, so please register by emailing c.g.olesen_at_

The international two-day symposium Sensory Moving Image Archives: Visualization, Exploration and Reuse of Moving Image Data comes out of the research project The Sensory Moving Image Archive: Boosting Creative Reuse for Artistic Practice and Research (SEMIA, 2017-2019). Working with diverse archival material from Eye Filmmuseum and the Netherlands Institute for Sound and Vision, SEMIA develops alternative ways of exploring moving image archives by analyzing data concerning visual features – such as colour, movement and shape. To achieve this, SEMIA has trained deep-learning networks for extracting sensory data from moving images and developed an interface for exploring the results. The project’s outcomes will support alternative ways of accessing collections and facilitate new forms of reuse based on sensory data.

In pursuing these goals, the project seeks to challenge the limitations of traditional text-based search. In recent years, three groups in particular have expressed a need for this – and it is these groups the symposium targets. They are scholars and scientists (including media historians, museum and archival scholars, digital humanities scholars and computer scientists), artists working with moving image or cultural heritage collections, and archivists (including media archivists and cultural heritage professionals more broadly). Taking SEMIA’s results as its starting point and test case, the symposium offers a platform for exchange between perspectives from those different groups, and the fields they represent. The program will consist of invited presentations and papers accepted through an open call. Scholars, professionals and practitioners from all groups are strongly encouraged to submit proposals.

The programme committee for the symposium invites contributions in the following areas (but will also consider other relevant topics):

Media Historiography and Digital Humanities

  • Data-driven visual methodologies for the analysis of visual culture (Rose 2013), in different areas of the humanities
  • Examples of the exploration of moving image and cultural heritage data, for instance in the lineage of  Exploratory Data Analysis (Tukey 1977) or Cultural Analytics (Manovich 2009)
  • Approaches in film studies engaging with data visualization as a form of  deformative criticism (Ferguson 2017) or new cinephilia (Habib 2015)

Heritage Studies

  • Interfaces for heritage collections that challenge text-based search and retrieval, for instance “generous” interfaces (Whitelaw 2015) or forms of humanistic interface design (Drucker 2013)
  • Processes for and experiences in designing exploratory interfaces for heritage collections, specifically prototyping and user testing

Computer Science

  • Deep-learning and/or feature engineering for visual analysis of moving images
  • Computer science approaches tailored to the analysis of subjective attribute data (rather than object recognition or scenes)

Media Art

  • Media art projects and practice-based research exploring the affordances of non-evidentiary algorithmic approaches to moving image data analysis and visualization
  • Practices of found footage, expanded cinema and moving image archive appropriation involving data analysis and visualization

Media Archiving

  • The integration of computer-generated (sensory) moving image data in media asset management systems and/or moving image archive databases
  • Reuse of (sensory) moving image data for TV production and in journalism

Submission guidelines:

Please submit an abstract of 300 words and a short bio of 50 words, in pdf format, by emailing before 23 November.

Posted in Uncategorized | Leave a comment

Second SEMIA Workshop: objectives and short report

What do people look for, as they access large collections of digitized films, and how do they want to conduct their explorations? Those were the central questions of the second SEMIA workshop, that took place at the offices of project partner Studio Louter, on 28 March 2018.

Compared to the first workshop, this second one took a much more hands-on approach. The abovementioned questions had to be translated into creative assignments, that would be appealing for a diverse audience of media scholars, media artists and creative industry professionals, and professionals concerned in their daily practice with the interests of museum visitors. Participants for the workshop were selected from those four groups.

In the first assignment of the day, the attendees, sat in small groups and surrounded by thousands of film stills, began combining images they felt visually belonged together. Individually, they created one or two collections, chosen out of the 500 sets of stills on their tables. Subsequently, they wrote down on post-its the characteristics which they had centred their collection around, and then associated those labels with the categories of ‘colour’, ‘shape’, ‘movement’, ‘texture’ (all particularly relevant to the SEMIA project) or ‘other’. This way, they manually produced the ‘features’ which extraction in the first stage of the project revolves around, but focussing on those that held particular appeal or relevance to them as (potential) users. Oftentimes, participants decided on highly abstract characteristics (often in terms of the emotions images represented or elicited), which ended up being labelled as ‘other’. Among the four pre-determined categories, ‘form’ was the most chosen one.

In the next time slot, the attendees were asked to highlight their preferences among a wide range of interface types. Here, they could choose between tangible interfaces, pointing device interfaces, motion tracking interfaces, text-based interfaces, touch & gesture interfaces, mixed interfaces – AR, mixed interfaces – VR, speech/sound interfaces and toggle user interfaces (or combinations thereof). All groups of participants were particularly attracted to the tangible interface types, which enable users to interact with the information displayed through their physical environment.

For the next series of assignments, seven groups of 3 to 4 people had to put their preferences into practice. Armed with markers, paper and scissors, the groups created their own interfaces on two posters. On a first one, they designed an interface that enabled users to explore a film collection. On a second, they made suggestions as to how an interface might present the results generated in the course of image analysis, on the basis of the users’ explorations. Once again, a number of patterns emerged in the participants’ suggestions. First, all groups designed interfaces that produced unexpected results, or in other words: that embraced serendipity. Second, many of the prototypes proposed were spatial installations. Third, a great many groups saw potential for new ways of exploring film collections in 3D, rather than in 2D set-ups. Finally, almost every prototype involved a control device that somehow linked to the user’s senses and/or emotions.

The last assignment of the workshop had the participants individually choose one ‘input’ poster (i.e., one chosen from those that made proposals for specific principles of exploration) and one ‘output’ poster (that is, chosen from among the second type, focussing on the presentation of exploration results). The most often-selected ‘input’ poster showed an interface that favoured unintended encounters: it had a person entering an immersive experience, where the interface reversed everything he or she did or gave expression to – their movements or emotions, for instance – by displaying films representing or eliciting the opposite or inverse. Within this proposed set-up, users could however ‘take back control’ by using an app on their mobile phones. The most often-chosen ‘output’ poster showed an interface whereby the result of a search was an actual film, rather than a graph (so, the analysis result as translated back into an image selection, rather than the analysis as an abstraction).

The workshop’s outcomes will be taken into consideration during a first creative brainstorm session with the SEMIA partners at the Amsterdam University of Applied Sciences, who will be developing prototypes for an interface that gives users access to the image analysis tool developed by the University of Amsterdam’s Informatics department.

The clip below gives a visual impression of this most inspiring day.

– Simone Slagboom

Posted in Artistic Research, Data Visualization, Interfaces, Moving Image Archives, Prototyping, Workshop | Leave a comment

Learning Visual Representations of Style

On May 16, 2018 Nanne van Noord defended his PhD thesis on learning visual representations of style at the University of Tilburg. He conducted his PhD research as part of the REVIGO project, prior to joining the SEMIA team. Below, he provides a short summary of his PhD thesis; a digital version of the thesis can be found here. The insights he gained carrying out this research feed into the feature extraction and analysis portion of the SEMIA project.

An artist’s style is reflected in their artworks. This style is independent from the content of the artwork, two artworks depicting two vastly different scenes (e.g., a beach scene and a forest scene) both reflect the artist’s style. By recognising the style of the artist, experts, and sometimes even laymen, can tell the same artist created both artworks. Replicating this ability for connoisseurship in a computer, and potentially even giving a computer the ability to produce artworks in the style of the artist, is the main goal of the work I did for my thesis.

To analyse artworks with a computer we can use techniques from the field of computer vision. Traditionally, these techniques rely on handcrafted features to describe the characteristics of an image. However, in recent years the field of computer vision has been revolutionalised by the emergence of deep learning models. These are models which learn from the data a feature representation optimised for a task.

Deep learning models have been shown to be enormously successful on a wide range of computer vision tasks. Therefore, in my thesis I explore the application of deep learning models for learning representations of the artist’s style. By learning the representation we might discover new visual characteristics of artists, enriching our understanding of the artist’s style. Moreover, such a representation might enable new and novel applications.

Posted in Feature extraction, Stylometrics | Leave a comment

First SEMIA Workshop: objectives and short report

One of the ways in which the SEMIA project team has sought from the start to further develop its key concerns and objectives, is through the organization of a number of workshops involving representatives of the three ‘user groups’ the project addresses.

The first workshop, which took place on 7 February 2018, brought together media scholars and heritage professionals, along with a number of artists who, as part of their creative practice, repurpose moving image materials in different ways. The workshop’s objective was twofold: on the one hand, it sought to discuss the current ‘state of the art’ in visual analytics research for (archival) moving images, as well as to share amongst project members and other participants some basic knowledge of the underlying mechanisms for (automated) feature extraction and image analysis. On the other, the workshop sought to establish, through discussions with participants, possibilities and limitations in the use of such methods, specifically for media history research and heritage presentation. Our aim was for those discussions to provide input for the remainder of the SEMIA project, specifically in going forward with feature extraction (the first, but iterative phase of the tool development work).

The programme consisted of a combination of presentations, followed by responses, and two series of discussions with participants (first in smaller groups, then plenary). The choice of presentations was determined on the basis of a prior selection of image features identified as particularly relevant for the project either because of their alignment with existing research interests, or for more pragmatic reasons (such as, the labour-intensiveness of extraction and analysis in relation to the project’s length). Those features (colour, specific kinds of shape, specific dimensions of movement, texture) and tasks, introduced at the beginning of the workshop, varied in status: from essential to the project to desirable (but to be tackled only if time allowed); or, from aligning with prior research to more experimental. Speakers and their respondents were chosen in function of their different takes on matters of extraction and analysis: some were scholars, some artists, and some had extensive experience of using extraction and analysis methods (e.g. in media history research) while others took a more exploratory or even speculative stance. The purpose of their interventions was to assess whether choices made prior to the event, made sense – of if not, to generate ideas as to how to tweak them.


Project leader Giovanna Fossati opening the workshop


The workshop was productive in different ways. First, the combination of presentations by Nanne van Noord (SEMIA project member) and Matthias Zeppelzauer (former collaborator of the Vienna Digital Formalism project), benefitted our understanding of the principles behind, and history of, visual analytics, and the differences between working with purpose-designed, task-specific algorithms (feature engineering) and deep-learning techniques. Zeppelzauer’s contribution also provided more insight into computer vision thinking about visual features, and how such features can best be conceptualised – also for the purpose of determining tasks in the retraining of existing networks for deep-learning. (A post elaborating on these definitions and distinctions, and their relevance to SEMIA, will follow in due course.) Second, the presentation by Adelheid Heftberger (likewise, of the completed Vienna Digital Formalism project) prompted us to consider the importance of consensus among project members about the vocabulary they use in communicating about project objectives and procedures – an advice that has since proven very helpful. Third, the presentations by Barbara Flückiger and Gaudenz Halter (of the Film Colors: Technologies, Cultures, Institutions project at the University of Zürich), Frank Kessler and Danai Kleida (involved in a small-scale film movement annotation project at Utrecht University, in collaboration with the Media Ecology Project at Dartmouth College) and once again, project member Nanne van Noord, brought us up to date on recent initiatives in (manual and/or automatic) feature extraction and analysis, as part of ongoing media, resp. art history projects with related concerns. And finally, the presentation by media artist Geert Mul above all proved inspirational in terms of the sort of revelations visual analysis techniques can engender, and taught us that it might make sense to toggle in the project between conceptually separating out the features that extraction and analysis centre on, and considering them in their mutual relations.

The group discussions later on in the day, focusing on one or more features each, further confirmed this. Aside from revealing a great deal about the working habits and research interests of participants, they also taught us about differences in understanding of the nature of, and relations between, image features. Most importantly, they gave us a better idea of what potential users, specifically researchers, are after, in gaining access to digitized collections. Most crucially, such users want to be taken to places (objects, analogies, patterns) they hadn’t tought of before – even if this means that the relations established in the process cannot necessarily be pinpointed exactly, or explained with recourse to empirical evidence. This strengthened us in our assumption that there is indeed a need for more exploratory ways of navigating through collections, if only just as a trigger for more traditional form of search later on.

Eef Masson

Posted in Artistic Research, Feature extraction, Media history, Workshop | Leave a comment

SEMIA’s Objectives: short elaboration

In the SEMIA project, we investigate the affordances of visual analysis and data visualization for exploring large collections of digitized moving images. In doing so, we try to accomplish a number shifts in relation to other Digital Humanities projects of the past few years that use similar technologies. As we mention in the project description on our About page, our efforts to develop, and repurpose, concepts and tools address a wider audience than just scholars: we also target artists, and users in the creative industries. But there’s other ways in which SEMIA differs from such image analysis and visualization projects, both past and on-going. Below, we list some of the most crucial ones.

First, SEMIA is distinctive in its overall objectives. The project seeks to develop concepts and tools, not in order to meet very specific film or media studies requirements, but rather to enable users to intuitively explore visual connections between database objects. In the past, and today still, image analysis was often applied – and the tools for it developed – with an eye to answering specific research questions. (For instance, in the context of digital film history, questions about developments in film style. Within our European network, such questions have been leading, among others, in the Vienna Digital Formalism project, now-completed, and the Zürich Film Colors: Technologies, Cultures, Institutions project, still on-going.) While there is certainly a kinship between SEMIA and projects that take such an approach, and while we build on results obtained in the process, we simultaneously try to move on from them. Ultimately, the explorations we seek to enable may lead to concrete research questions; however, we always act in the assumption that the users’ explorations will serve to trigger, rather than to answer those (compare e.g. Masson 2017, 33-34). Therefore, we also do not seek to generate tools for the analysis of specific bodies of work, but rather more robust ones, that allow users to navigate within large corpora of (visually) extremely diverse materials.


(Human-made) selection of images and image series with visually similar features


Second, SEMIA does not make use of semantic descriptors in the same way, or to the same extent, as other projects do. In developing concepts and tools, we don’t rely as much, either on metadata ingested with the archival objects themselves (e.g. as catalogue entries) or on the sorts of manually produced annotations that are key to other projects involving image analysis. On the one hand, because we see this project as an effort to find out how collections can be explored, precisely, in the absence of metadata – archival assets that are both scarce, and very time-consuming to produce. In other words, we actually hope that SEMIA will generate ideas on how users can circumvent the general problem of scarcity of metadata in navigating archival collections. On the other hand, and as implied above, we do so because we want to be able to explore collections in other ways than have been tried so far, and on the basis of other connections between discrete items than those that underpin the sorts of search activities that we commonly perform in navigating them. Connections, indeed, that inevitably involve some form of interpretation of the discrete objects, and that are heavily inflected by the dominance in metadating of either classic filmographic or technical information (often added manually or through crowd-sourcing), or identifications of semantic (often object) categories (increasingly done automatically).

A third shift our project seeks to accomplish is to rely more radically on deep-learning techniques – and so-called ‘neural networks’ – as opposed to feature engineering. We intend to release another post on this topic soon, but in essence, this means that the SEMIA Informatics team will not design task-specific algorithms in order to extract pre-defined features from the images in our database (whether image sections, single images, or strings of images/fragments) and subsequently compare them, but rather tweak, and repurpose, networks that have been trained with techniques for automatic feature-learning (that is, the learning of data representations). (The aforementioned repurposing – and retraining – is necessary, because the networks we can rely on were built for different tasks, and different corpora, than we intend to use them for. Once again, we shall elaborate on this in a future post.) Overall, this means that in developing tools for analysis also, we rely to a lesser extent on human intervention. But we do not seek to avoid such human intervention altogether – if only because we need to be able to minimally understand and evaluate the sorts of similarities the system relies on in making connections between discrete objects. Moreover, as it is our aim to support users in their (unique, or at the very least individual) explorations of collections, we would like our tools to adapt to their specific needs and behaviours. Here, human intervention in the form of man-machine interaction will still be desirable.

Finally, SEMIA also explores new principles for data visualization (but ‘new’, in this context, in relation to such practices in the Digital Humanities more broadly – rather than image analysis specifically). Considering the abovementioned goal of enabling users to freely explore collections and make serendipitous connections between objects – yet potentially as a first step toward asking highly precise research questions – the representation of the data we extract in the analysis process need not directly serve an evidentiary purpose. (For more on this topic, see Christian Olesen’s blog post on Moving Image Data Visualization.) This means in turn that we can take inspiration also from artistic projects in image data visualization – an area of practice that is currently gaining a lot of traction. At the same time, users will need to be able to assess what it is they see, if only to be able to determine what they can or cannot use it for (for instance, in the context of their scholarly research). As this is a key area of attention for SEMIA, we will be sure to reflect on it further in later posts, as our team discussions on the topic evolve.

Eef Masson


Masson, Eef. 2017. “Humanistic Data Research: An Encounter between Academic Traditions”. In The Datafied Society: Studying Culture through Data, ed. Mirko Tobias Schäfer and Karin van Es, 25-37. Amsterdam: Amsterdam University Press.

Posted in Data Visualization, Feature extraction, Moving Image Archives | Leave a comment

SEMIA and Moving Image Data Visualization: an overview and brief introduction

Since the SEMIA project started last year we have had several research meetings between the involved project partners and are gradually developing work routines. We have also organized two workshops involving the different user groups we work with – artists, heritage partners and media scholars – in order to determine the user requirements for the software and interface(s) we aim to develop. We will post more about these events and the outcomes on the blog later.

In this first longer post on our research blog, I will offer an overview and some introductory remarks on examples of current visualization formats for moving image data in film and media studies and their intersection with media art. With this overview I hope to offer insights into the starting point for the SEMIA project while emphasizing the project’s urgency. This overview is in no way intended to be comprehensive (admittedly, it focuses mostly on examples from film studies). Yet, on the other hand, as not a lot of work is currently being done in this field, it can be considered fairly extensive, and I have tried to include a few examples that are not often part of the conversation on moving image data visualization.

An important focus of the SEMIA project is to nurture artistic reuse and exploration of archival moving image collections, using, specifically, data extracted from parts of both EYE Filmmuseum’s and the Netherlands Institute for Sound and Vision’s collections as a basis for visualization and interface navigation. In order to do this, EYE Filmmuseum has made primarily silent film material available to us – in large part from the Jean Desmet Collection – while the Netherlands Institute for Sound and Vision has chosen NOS news broadcast footage dating from the 2000s. How exactly we will analyze this data and allow for exploration of it are aspects which we have sought to determine in a dialogue between artists and scholars in the workshops mentioned above. In this process, our ambition has been to bring together groups – media historians, heritage professsionals, computer scientists and artists – which seldom discuss artistic, exploratory approaches to archival moving image collections at a cross-section between artistic and scientific practices.

Before we began this conversation and started the project, there were however several examples we were inspired by and whose work we build on, which I would like to point to in this overview. I have divided this overview into three sections; respectively Stylometric analysis of films and data visualization, Exploratory Data Analysis of Films and Artistic Data Visualization, discussing what characterizes each below while detailing some of the ways in which they inform the SEMIA project.

Stylometric analysis of films and data visualization

The emergence of film data visualization in media studies is closely intertwined with the development of stylometric analysis of moving images, or – as it is commonly referred to in film studies –  statistical style analysis or cinemetrics. The methods associated with such approaches tend to answer specific research questions empirically and in a deductive fashion to determine the characteristic stylistic features of films, for instance to solve issues of authorship attribution or to date films, by creating statistical profiles of directorial styles, entire periods or genres.

SEMIA is working towards aims that are fundamentally different than traditional stylometric research. The aim of our project is not to do for instance authorship attribution or to study genres and periods, but rather to visualize patterns within moving image collections that may lead to serendipitous discoveries of similarities in different image features, for instance colour, shape and movement. Yet, we are inspired by traditional stylometric research to the extent that it has experimented with visualizing film data and given shape to it in various types of graphical representations in different – often quite inventive – ways, bridging the fields of media studies, quantitative data analysis and visualization.

Statistical approaches to film analysis have existed for almost a century and primarily center around montage and cutting rates as the central features of their analysis. Psychologist Hugo Münsterberg’s The Photoplay: A Psychological Study (D. Appleton and Company, 1916), which endorsed the practice of measuring the cutting rates of films as a basis for comparative film analysis and for understanding the psychological impact of film upon spectators is often referred to as a beginning point. Along similar lines the article “Bildschnitt” (1926) by German film historian Georg Otto Stindt for the journal Die Filmtechnik has been considered an early, pioneering effort in formalizing the systematic study of film’s cutting rates.

Since film studies became an increasingly institutionalized field of research in the 1960s and 1970s, stylometric research practices have slowly flourished and become gradually more refined. In Germany, especially the work of Herbert Birett – German geophysicist turned silent film scholar – from the 1970s onwards stands as one of the most rigorous attempts at developing formalized, scientific procedures for studying the development of film style. Birett’s work was in its time quite unique within film studies because it drew on advanced approaches in computational and quantitative linguistics at an interdisciplinary junction seldom explored in these years. Aspiring to analyze and discern patterns in films in a similar way as in language systems it reflected the quantitative methods of literary computing, as a way to date films and do authorship attribution.

More widely known from this period is the work of Barry Salt, whose classic 1974-article “Statistical Style Analysis of Motion Pictures” published in Film Quarterly is foundational to statistical style analysis. The article presented a general critique of film historiography up till then, which it argued lacked a solid empirical basis and critiqued it for making arbitrary judgements with regard to the discernment of stylistic developments and styles of directors. Instead, it suggested that scholars should count and measure shots – and later also shot types and camera movements – to create a stronger empirical fundament for their observations. Salt’s work has contributed to film studies by propagating especially the calculation of Average Shot Lengths (ASLs) – that is: counting all the shots and shot lengths of a film and calculating  their average length – as the key parameter for comparing films.

Building on the theory and practice of Barry Salt’s research, the database and community of the web-based Cinemetrics project – initiated by film historian Yuri Tsivian at the University of Chicago – is perhaps the foremost site for the continued development of statistical style analysis today. The database counts info on approximately 19000 titles, consisting of a very hetegenerous corpus of film titles, television shows and for instance trailer and music videos. Beyond this, Cinemetrics has also hosted important discussions around key concepts of statistical style analysis, debating for instance the different advantages of calculating Average Shot Length versus Median Shot Length in the production of statistical data (a discussion which I will not detail here, but which is nicely introduced by Yuri Tsivian here), thus refining the epistemological foundations of stylometric research.

So, how may visualizations of such moving image data look like?

Style analysts and cinemetricians have experimented with a great variety of standard graphical representations for film data available in both open source and proprietary visualization software such as Matplot, Matplotlib, ImageJ and R, as well as with custom-made forms of visualization.

The image below shows the custom-made Cinemetrics-graph which Yuri Tsivian made for Charlie Chaplin’s City Lights (1931). For a full view of the graph in relation to its associated data and remarks by Yuri Tsivian see here. This is a good example of a very frequently used format which visualizes film data in a reduced, abstract graphical representation.


Yuri Tsivian’s Cinemetrics-graph of Charlie Chaplin’s City Lights (1931)


Cinemetrics scholar Mike Baxter has taken such work in different directions, exploring especially the potential of the open source software R, for making differently styled graphs. Baxter’s extensive work in this area can be explored in his book-length study Notes on Cinemetrics (2014) which is available here.

Exploratory Data Analysis of Films

Other scholars who have worked extensively with cinemetric approaches have taken visualization of film data in different directions working not only with visualizing film statistics but also with frames extracted from digitised films (sometimes entire films), using the open source medical imaging software ImageJ. Especially media scholar Lev Manovich has explored this software as part of his Cultural Analytics framework, which among many other things aims to visualize patterns in big data image sets – for instance digitised heritage collections or Instagram images. ImageJ was applied in the Digital Formalism project in collaboration with Lev Manovich. In this project, film scholar and archivist Adelheid Heftberger meticulously annotated the shots in a corpus of Dziga Vertov films using the open source software Anvil making it possible to visualize structures in them using different kinds of scientific, visual analytics (in addition to ImageJ also MatLab was tested). The results have circulated widely and are often brought up when scholars point to the exciting avenues which the digital humanities may open for media studies. The visualization below shows an ImageJ montage visualization – this is what the visualization type is named in the software – which puts together a frame from each shot in Dziga Vertov’s Odinnadtsatyy (The Eleventh Year, Soviet Union, 1928) as a way to discern changes between shots.


ImageJ montage visualization of Dziga Vertov’s Odinnadtsatyy (The Eleventh Year, Soviet Union, 1928)


ImageJ allows for making many different visualization types in addition to the montage visualizations, for instance also so-called summary visualizations which scholars have used to find average colour values by layering film frames onto each other (for instance a sample of every x number of frames of a film or all frames from a film). This visualization type has been used to a great extent by media scholar Kevin L. Ferguson in the US to study patterns in genre films – for instance westerns or gialli – or by film scholar Miklos Kiss in the Netherlands. The example below is Ferguson’s ImageJ sum visualization of Sergio Leone’s Once Upon a Time in the West (Italy/US/Spain, 1968).


ImageJ sum visualization of Sergio Leone’s Once Upon a Time in the West (Italy/US/Spain, 1968)


In a previous project based at the University of Amsterdam – Data-driven Film History: A Demonstrator of EYE’s Jean Desmet Collection (2014-2015) – we also experimented with both ImageJ montage and summary visualizations for visualizing colour patterns in films from the EYE Filmmuseum’s Jean Desmet Collection. The results can be viewed here.

These latter examples build on stylometric research which it regards as a hermeneutical antecedent but also presents its visualization work as a form of exploratory research. For instance, Lev Manovich has cited John Tukey’s Exploratory Data Analysis (EDA) as an interesting reference point for visualization work in the humanities, which allows media scholars to use visualizations in a descriptive manner as a basis for generating new questions about the corpora they analyze, rather than regarding them as firm, scientific evidence. Such an attitude may also be said to resonate in Ferguson’s work who, rather than considering his visualizations evidentiary uses them for supporting film theoretical reflection on composition, space and volume. In this respect Ferguson, in particular, has been very clear about situating his work at intersection of media research and artistic visualization. As he writes in his article “The Slices of Cinema: Digital Surrealism as Research Strategy” (2016, p. 279):

“While not always drawn together, there is a tradition of experimental work such as mine that balances between both new media art and digital humanities scholarship.”

It is this intersection which SEMIA also explores, by trying to develop new approaches – for instance using deep learning methods (as will be discussed in upcoming blog posts). In our ambition to explore this intersection we have been inspired by various artistic visualizing practices, while we will not be applying the exact same approaches. Below, I discuss a few illustrative examples.

Artistic Data Visualization

In addition to stylometrics’ visualization formats there exists a small group of art works – media scholar Kevin Ferguson has even referred to these works as forming a “mini-canon” of data art works – which intersect and inform visualization practices in media studies. While working with different coding languages and visualisation software, the results produced are sometimes remarkably similar to what media scholars have produced with ImageJ in recent years and tend to explore especially the summary and montage visualisations as visual tropes of data-driven analysis. Strikingly, several of these works precede what media scholars are currently doing, some of them being made 10-15 years before media scholars began taking up similar methods. In SEMIA the aim is not to replicate the formats of such works, but – among other things through data visualization – to encourage artists to create new ones which may boost this small field of artistic production in new ways.

A series of early works which fall into this category is San Francisco-based artist Jim Campbell’s Illuminated Average Series. Initiated in 2000, the series includes a few film data visualizations, for instance of Alfred Hitchcock’s Psycho  (USA, 1960) – see below – and a scene from Orson Welles’ Citizen Kane (USA, 1941) – which resemble the ImageJ summary visualization type a great deal. Campbell has exhibited these images in several museums in luminous display boxes which allows visitors to contemplate them and discover beautiful details in them.


Jim Campbell’s Illuminated Average Series: Alfred Hitchock’s Psycho (2000)


Media scholar and artist Jason Salavon’s work has also deployed a broad range of visualization types in order to explore patterns in various media types and genres, spanning from film data to music videos to Playboy magazines. For instance, Salavon has made visualizations and video installations on the most succesful films at the box office in his The Top 25 Grossing Films of All Time-works. The single-channel video installation version of this work The Top 25 Grossing Films of All Time, 2×2 (2001) – of which an excerpt can be viewed below – presents colour patterns in blockbuster films in a dynamic visualization of over-sized pixels. In the companion piece The Top Grossing Film of All Time, 1×1 (2000) – made a year earlier – Salavon presented a grid reminiscent of a montage visualization of the top grossing film at that time – James Cameron’s Titanic (1997) – in which each field represents a predominant colour in one of the films serving as source material. An impression of this work can be viewed here. Salavon’s works invite critical reflection on the formal characteristics and cultural conventions of mainstream culture, offering a view on the material they process which may engender critical perspectives.


Jason Salavon’s The Top Grossing Film of All Time, 1×1 (2000)


Another early example is the Cinema Redux project of British artist and designer Brendan Dawes created in 2004. As the subtitle and blurb of the project – both available on Dawes’ website – make clear, the purpose of the project was that of “Creating a visual fingerprint of a movie” which shows: “…a single visual distillation of an entire movie; each row represents one minute of film time, comprised of 60 frames, each taken at one second intervals”. Dawes did this by presenting wall-sized prints of montage visualizations which have been exhibited in numerous museums and media art exhibitions, first in 2008 in the MoMA’s Design and the Elastic Mind exhibition. Dawes work may be seen as being quite close to the fingerprinting which is a strong component of stylometric research but also invites us to contemplate data visualization’s beauty by exhibiting them in a gallery setting rather then using them as the basis for scholarly hypothesis testing.

Finally, one may also expand this reference frame with a few examples which are not often mentioned. For instance, one may consider a few works by Netherlands-based media theorist and artist Florian Cramer as belonging to this category. Concerned with making films for and transferring films to floppy discs – using a disc format allowing for a maximum capacity of 1.44 MB – the Floppy Films project explored, among other things, the aesthetics produced from compressing feature-length films which were nominated for an Oscar in 2009 – for instance Slumdog Millionaire (Danny Boyle, UK/France/USA, 2009), The Wrestler (Darren Aronofsky, USA/France, 2008). A description of this project written by Florian Cramer can be read here, and some of the resulting videos can be viewed here. One may consider these works in the lineage of a tradition of experimental film and video which investigate the material transition of films between different carriers to highlight – among other things – diferent media specificities.

In addition, I would also like to highlight the inventive and highly intriguing work being done by New Zealand-based silent film scholar and audiovisual preservationist Katherine Nagels. A silent film enthusiast, Nagels has written scripts with which she creates colourful rainbows of silent films – below an example showing the rainbow of Die schwarze Kugel oder Die geheimnisvollen Schwestern (The Black Ball; or, The Mysterious Sisters, Germany, 1913) – as a way to explore the colour palettes of silent cinema. Nagels has written a blog post on her work which can be found here.


Katherine Nagels’ rainbow visualization of Die schwarze Kugel oder Die geheimnisvollen Schwestern (The Black Ball; or, The Mysterious Sisters, Germany, 1913)


Data Extraction and Visualization in The Sensory Moving Image Archive

With this post I have given a few examples of the scholarly and artistic work which has preceded SEMIA in the field of film data visualization and which in different ways has offered a starting point for us. In our own work, we will however be taking things in quite a different direction, by working with new methods and approaches which have not yet been applied to archival moving image material – for instance deep learning. Our project’s focus is also quite different from previous work in that we wish to explore different image features than the ones discussed above, while nurturing artistic approaches to a greater degree, by facilitating a greater degree of serendipity and exploration. In the past few months we have had extensive discussions about this in our team and with a great number of scholars, artists and heritage professionals both in the Netherlands and abroad. Stay tuned for more details in the upcoming posts here on the blog!

Christian Olesen


Birett, Herbert. 1988. “Alte Filme: Filmalter und Filmstil.” Diskurs Film: Münchner Beiträge zur Filmphilologie, no. 2 : 69-87.

Ferguson, Kevin L. 2016. “The Slices of Cinema: Digital Surrealism as Research Strategy.” In The Arclight Guidebook to Media History and Digital Humanities, ed. by Charles R. Acland and Eric Hoyt, 270-299. Sussex: REFRAME Books.

Nagels, Katherine. 2017. “A Rainbow of Silent Film” (blog post). Silents, Please! 21 August.

Tsivian, Yuri et al. [N.d.] Measurement Theory. “Articles.” Cinemetrics.

Posted in Artistic Research, Cinemetrics, Data Visualization, Floppy Discs, Moving Image Archives, Rainbows, Stylometrics | Leave a comment

SEMIA at Smart Culture Conference 2017

The research in the SEMIA project is in full swing and we look forward to sharing our first results with you. Meanwhile, we would like to announce an upcoming conference participation which will offer an opportunity to learn more about the project. For scholars and artistic researchers in and close to Amsterdam it will be possible to get an introduction to the project and see a first glimpse of the results at the Smart Culture Conference, organized by NWO in a few weeks – on November 23 and 24 to be precise – where Project Leader Giovanna Fossati and Postdoctoral Researcher Christian Olesen will be speaking. The conference will take place at De Rode Hoed, Keizersgracht 102 in Amsterdam. The conference program and details on time and location can be found here.

Posted in Conference Paper, NWO, Smart Culture | Leave a comment

Welcome to the SEMIA blog

Welcome to the research blog and website of the research project The Sensory Moving Image Archive: Boosting Creative Reuse for Artistic Practice and Research (2017-2019), funded by the Netherlands Organization for Scientific Research (NWO).

On this blog we will be posting about the project’s development and research process. A detailed introduction to the project will follow in a later post. At this moment you can find more info about the overall aim of the project and its members in the About and Who We Are-pages.

Posted in NWO | Leave a comment