Dr Santiago Rentería

Dubbing Point Processes

Thu, 13 Jun 2024 00:00:00 +0000

In this project I play with the implications of sampling and dubbing tape recordings from the John Hutchinson’s sound archive with stochastic processes. Hutchinson, a self-taught field recordist, began capturing these unique sounds in 1959 during his work with the Department of Agriculture. As a result, this archive offers a captivating journey through time and diverse regions of Western Australia. Given the sheer volume of recordings (over 130 hours!), a manual exploration (solely by ear) proved impractical. Instead, I adopted a hybrid data-driven approach to sonic foraging, implementing a sampling mechanism to extract “aural summaries” from the most intriguing parts of the archive. Imagine creating a sonic “thumbnail”, but instead of compressing images we are compressing entire archives by creating representative mosaics with their most salient one-second sound fragments (see a spectrogram representation of the latter in the picture shown above). For this purpose, I used stochastic point processes to select fragments with high diversity. This ensured a representative selection, fast browsing, and avoided the redundancy of (uniform and independent) random sampling. Of course, what counts as representative depends on the task at hand. To address this issue, I relied on Wavelet scattering networks. These are like deep learning networks but with filters defined a priori in terms of Wavelet functions instead of filters approximated from data. Finally, in order to showcase the potentials of this technique, I programmed a bespoke interface to mix the fragments live. The resulting aural exploration was presented at the Data Visualization Institute, University of Technology Sydney.

The interface has a granular sustain. Each fragment has its own track with a send to FX. The balance of effects can be controlled before the master output. The number box next to ‘T’ is for transposition. Loop start and end are shown in samples. Sample # represents a draw from the archive using DPP as described . Since actual files from the archive are retrieved (as opposed to being synthesised), it is possible to use the retrieved fragments as entry points to previously ‘unlistened’ regions (ie sonic foraging).

Aural summaries: Unveiling diversity through stochastic sampling

Imagine exploring a vast sound library. How do you quickly grasp its essence without listening to everything?

Let’s put it in a different way, how do you read through the pages of an infinite book of sound without losing your mind? You can’t use bookmarks because the pages fall through your fingers.

I tackled this challenge (and kept my sanity in return) by employing a method called Determinantal Point Processes (DPPs). DPPs are mathematical models originally used to represent the behaviour of fermions (such as electrons), which naturally repel each other (in theory they cannot occupy the same quantum state). This same principle can be applied to sampling sound files! Turns out DPPs can help us to select sub-collections of short audio snippets that are as sonically diverse as possible. Think of it like drawing sounds from a bag but the smart way, you’re guaranteed to pick contrasting textures, avoiding redundancy. Along these lines, a recent paper proposed DPPs to address data exploration in very large audio recordings. This is of great value in ecoacoustics, where audio recordings are made longitudinally for years to understand environmental changes through persistent patterns in soundscape dynamics.

So, how does it work? First, a wavelet scattering network (or for the pros, an adequate feature extractor) is used to analyse each sound fragment, essentially creating a unique “fingerprint” (or feature space) that captures sonic characteristics of interest (see the picture below for a 2D representation of such fingerprints). DPPs then leverage this fingerprint to select audio fragments that differ as much as possible (ie repel each other) in the feature space. Unlike other methods which sample fragments uniformly at random, DPPs do not get stuck repeating similar sounds. Intriguingly, this property is given by a mathematical structure from which the processes derive their name: the probabilities of random subsets are assigned according to the determinant of a function. In our case this function is expressed as a matrix of numbers, the output of the Wavelet scattering network for all sound fragments.

Overall, with this approach between high-mathematics and experimental sound sampling, I wanted to push the boundaries of curation and soundscape composition. What a better way to do it than to improvise with a whole archive of wild sounds? These days most generative systems are based on prompts and do not reference their own archival origins. In contrast, with this experimental method of sampling, I advance a new form of creating sound mosaics through archival listening. No longer passively prompting with words, but dubbing and remixing fragments from the archive in real time ala musique concrète. A hot pot of sounds, a mishmash of Wavelets turned Sukiyaki for the ears!

In more media-archaeological sense. which is a particular mode of inquiry into how media works, I performed the role of a machine learning model. By listening to a batch of 20 windowed 1-second fragments from a whole archive, I attempted to produce soundscapes via improvisation. A genre reflecting the trial-and-error nature of such models. Therefore, the main points put forward by this performance are:

Most machine learning models do not implement smart-sampling methods like DPP to create training batches (see this for an exception). In this case it was me who curated what DPP had already sampled from the archive.
Machines ‘listen’ through standardised filtering or data-driven feature extraction and not by tapping into the fully-embodied experience of a human listening and performing before an audience.
The live-mixing interface displays the potentials of hybridising human and machine capabilities in archive-based soundscape composition. In cultural theory, these cybernetic chimeras appear have been denoted assemblages or social machines.

Credits and event documentation

Special thanks to Andrew Burrell and Zoë Sadokierski for their support and including my proposal in the programme.

Dadamining Datamining

Wed, 18 Oct 2023 00:00:00 +0000

Data mining methods for animal sound are used to produce a multichannel soundscape. Sound fragments of a Western Australian archive are subjected to two Machine listening algorithms. The first one assigns numeric addresses to soundscape regions by signal similarity (ie neighbouring sounds have similar sound texures). The latter segregates the products of algorithmic composition, a monophonic recording, into six audio channels. As a whole, the resulting soundscape displays a mundane industrial process of datamining sound with no analytic purpose but a dadaist impulse of listening to found fragments. Click here for more information on the scientific use of this sound archive.

In my arts-based research, machine listening is not a double of ear-bodied listening experiences, nor a continuation of computation in the domain of digital audio. As a regime of automation, machine listening constitutes an archival force. Digital sound memories are industrialised with algorithmic addressing schemes and filtering protocols which are never fully autonomous as the myth of artificial intelligence suggests.

If sound ecologies are open-ended and more-than-human, how can the (re)generative performance of machine listening subvert reductive technical desires (ie anarchive) what has already been fully pre-empted and automated in the archive?

This work was premiered in Melbourne (Australia) at the festival NONSTOP WKND: Planetary Auditions.

Binaural recording available here.

Birdsong Phrase Classification With Siamese Neural Networks

Wed, 01 Jul 2020 00:00:00 +0000

As part of my masters thesis I developed a “Shazam” for birdsong based on siamese neural networks, a few-shot machine learning technique capable of recognizing Cassin’s Vireo song elements. Below you will find the recorded live stream of my thesis defense.

Abstract: The process of learning good features to discriminate among numerous and different bird phrases is computationally expensive. Moreover, it might be impossible to achieve acceptable performance in cases where training data is scarce and classes are unbalanced. To address this issue, we propose a few-shot learning task in which an algorithm must make predictions given only a few instances of each class. We compared the performance of different Siamese Neural Networks at metric learning over the set of Cassini’s Vireo syllables. Then, the network features were reused for the few-shot classification task. With this approach we overcame the limitations of data scarcity and class imbalance while achieving state-of-the-art performance.

Disecciones Sobre Planos

Sun, 01 Dec 2019 00:00:00 +0000

Dissections over planes. Essay(s) from Tlatelolco is a transmedia project directed by Pablo Martínez Zárate that explores one of most emblematic architectural sites in Mexico City. Through a combination of a webdocumentary, a book a VR Installation and a live cinema performance, incisions are made over Tlatelolco as an element in Mexico City’s landscape. I developed the VR experience using various media, including digitized analogue recordings, drawings and photographs of a model of Tlatelolco. Pablo composed a soundscape that was part of the VR version.

To explore the web version please visit: dsctlatelolco.net.

Posters

Award

The project was awarded by Festival Internacional de Cine con Medios Alternativos (FICMA).

Noise City

Wed, 01 May 2019 00:00:00 +0000

This work is under development. One of the objectives is reflecting on the psychological effects of the noise in Mexico city. Up to date we have a Max application that distorts video by using the audio signal. In this way acoustic noise is translated into a visual experience. Stay tuned. I want to take this idea to an augmented reality platform.

El Cuerpo es un Archivo

Tue, 01 May 2018 00:00:00 +0000

El cuerpo es un archivo is a 360º video montage with multichannel sound inspired in the Mexican Movement of 1968. It is part of the permanent exhibition at Centro Cultural Universitario Tlatelolco in Mexico City. During the performance dancers interact with historical photographs. In response, a dance unfolds leaving the viewer immersed and inviting the audience to engage with the archive (of Tlatelolco Massacre).

The project was directed by Pablo Martínez Zárate and involved an indisciplinary team. My job here was setting multichannel audio and mapping a 360º video over a cylindrical screen using 6 short throw projectors. Below you can see the process of wrapping the image to the curved surface with the MadMapper software.

Format: 8,16, 35 y 120 mm + HD video
Duration: 15:00 min

Process

Final work

Credits

Director, screenwriting y photography: Pablo Martínez Zárate.
Choreography: Sociedad de Carne y Hueso (Aura Arreola, Teresa Carlos, Raquel Salgado, Marcela Vásquez y Rodrigo López).
Illustration: Santiago Moyao.
Assistant director: José Luis Rangel.
Postproduction: José Luis Rangel, Pablo Martínez Zárate, Hernán Perera.
Cameras: Pablo Martínez Zárate, Hernán Perera, Leonor Castro Guerra, José Luis Rangel, Pablo García, Luis Suárez.
Audio: Pablo García.
Programming: Santiago Rentería.

With the support of Universidad Iberoamericana, Biblioteca Francisco Xavier Clavijero and Laboratorio Iberoamericano de Documental del Departamento de Comunicación.

Bahidora Festival

Thu, 01 Mar 2018 00:00:00 +0000

Spacetime, a multidisciplinary architecture and design agency, invited me to collaborate at Bahidorá Festival with two art installations.

The first one, named Colormancy, involed the interplay between code, divination and chance. By mapping peoples’ names to particular colors custom ambiences were created out of light and smoke. Eventually a quote based on song lyrics fragments was shown as a personal fortune in a screen. A Raspberry Pi and a touchscreen were used for controlling the whole interaction: receiveing users’ input, automating multicolor LEDs and smoke machines via DMX and electronic relays.

The second installation used no electronics but played with basic shapes translated in space and time.

Migrante

Fri, 01 Dec 2017 00:00:00 +0000

Migrante is a stage performance that describes the journey of a character (that you can be yourself) in six stages. Migration, from the metaphorical point of view, is a search in which all human beings are immersed either for need or will. From this phenomenon emerges a sense of migration from the awareness of change, that allows us to question our mission in life, as we assume to be free to chart our own course and write a personal (his)story.

The project was directed by Bernardo Rubinstein with the goal in mind of generating an interdiscipline of dance, emotions and sound. While its nature remained physical it also involed theoretical reflection around the concept of migration and its historic meaning.

I consider Migrante as one of the most challenging experiences in my life, because it involved embracing my body as a canvas for emotional expression. A task not very usual for a musician.

If you want to read the essay I wrote for the performance please click here.

Poster and Credits

The Impact of Melody on Short-term Memory

Fri, 01 Dec 2017 00:00:00 +0000

During the class of Music and psychoacoustics I studied with my team the impact of melodic components in memory. For this purpose we conducted two experiments: The first one consisted in studying the effects of melody in short term memory by evaluating the recall of sung and spoken letter sequences. The second used two groups of sung melodies (structured and chaotic) to study how the regularity of melodic and rhythmic features impacts memorization. Both experiments were carried out with students between 12 and 16 years old. The following video explains the results and methodology in detail.

Recording and Mixing

Fri, 01 Sep 2017 00:00:00 +0000

Thanks to the class of Music Production and Recording Techniques I had the opportunity to collaborate with various musicians and learn from Juan Switalski, a friend and talented recording engineer.

Orchestra recording

Recording and mixing of Escuela Superior de Música Orchestra at Centro Cultural Coyoacanense.

Decca Tree and AB Recording Techniques

AKG 414 Mic

Juan Switalski and some colleagues

Studio recording

The Risin’ Sun

SantiagoRenteria The Ballad (of Being A Man) - The Risin Sun

Why don’t you do right (Peggy Lee Cover)

SantiagoRenteria · Why don’t you do right - Peggy Lee (Cover)

Binaural Recording

Please use headphones for best experience.

SantiagoRenteria · Acoustic Intro. (Live Binaural Recording) - Contemporary Guitar Ensemble

SantiagoRenteria · Asturias (Live Binaural Recording) - Contemporary Guitar Ensemble