CONCEPT
Contemporary multimedia art is in a state of continuous exploration for new forms and formats, a process actively fostered by the digital environment, its evolution, and the emergence of new technological capabilities. Thanks to artificial intelligence, there is a fundamental convergence of sound and image, which has become a key direction in the work of multimedia artists. Neural networks allow sound not just to accompany or illustrate visual content, but to generate it directly, acting as a universal conduit between the acoustic and the visual.
This phenomenon elevates long-standing artistic explorations in the field of «visual music» and synesthesia to a new level. Another crucial feature of the neural network is its operational principle: it interprets sound into a mathematical structure, thereby generating a multimedia visual component that projects the patterns it has learned. Previously, research was based on the relationship between sound and image—as physical imprints (e.g., Ernst Hladny’s «sound figure» [7])—where sound oscillations were directly translated into a visual projection. However, this process excluded «processing» via neural connections.
Contemporary multimedia art actively explores the synthesis of sound and image using neural networks. This defines the relevance of the topic—how artificial intelligence transforms sound into visual forms within modern installations.
The research material includes the works of key artists—from the pioneers of media art to contemporary practitioners whose projects demonstrate the evolution of audiovisual synthesis.
The visual exploration reflects the evolution of audiovisual synthesis: the starting point is historical examples of synesthesia, and the focus then shifts to modern installations where neural networks play a crucial role.
The theoretical foundation draws from the work of philosophers such as Friedrich Kittler and Christoph Cox, who help explain the transformation of the relationship between sound and image.
The main question I explore in my visual study is: How are neural networks changing audiovisual synthesis? The research is based on the hypothesis that neural networks create not merely a translation, but a complex interpretation of sound into image, generating a new type of synesthesia where art emerges from the very logic of computation.
INDEX
Chapter 1: The Evolution of the Relationship Between Sound and Image in Multimedia Art
1. Synesthesia Analogues in Precursors 2. Analyzing the Intersection of Auditory and Visual in the Pre-Digital Art Era 3. The Hegemony of Image Over Sound
Chapter 2: Analysis of Contemporary Installations and Practices
1. Audiovisual Synthesis: Sound as the Primary Generator of Visual Imagery 2. Neural Networks as a Catalyst for a New Era 3. Synesthesia or «Mythologizing»
Conclusion
Bibliography Image Sources
Chapter 1: The Evolution of the Relationship Between Sound and Image in Multimedia Art
- SENSORY CROSS-MODALITY IN PRECURSORS
Classic examples of «synesthesia» in the art of the late 19th and early 20th centuries are A.N. Scriabin and V.V. Kandinsky. Their goal was not merely to transfer sensations, but to find a universal language.
«Kandinsky believed that the color blue was analogous to the sound of a flute, blue—the cello, dark blue—the double bass, deep blue—the low notes of an organ, green was the color of serenity, white—the color of the universe from which emanates great silence (like a pause in music), and black—dead nothingness, eternal silence without future or hope. Specifically, A. Scriabin empirically developed a parallel table of musical and color tones, convincingly applying this principle in „Prometheus“ (1911).» [1]
Jim Hodges, «I dreamed of a world and called it Love, ” Gladstone Gallery, New York, 2016
Christoph Cox: «Antisynesthesia.» Cox contrasts traditional synesthesia with what he terms «antisynesthesia.» He argues that sound and image are inherently heterogeneous: rather than attempting to synchronize them, art should reveal this radical difference. The works of Jim Hodges or Christian Marclay are striking examples. [3]
- ANALYSIS OF THE FEATURES OF SYNCHRONIZATION BETWEEN AUDIO AND VISUAL IN THE PRE-DIGITAL ART ERA
Jim Hodges: Sound and image in one installation do not explain or duplicate each other. They coexist, creating a tense and multifaceted experience precisely because of their independence.
Christian Marclay: In his famous work, «The Clock, ” he physically cuts individual frames from the film strip, creating a visual rhythm that directly and materially influences the sound track. [5] Here, the connection is not metaphorical, but indexical and causal: the visual change directly impacts the auditory.
- THE HEGEMONY OF IMAGE OVER SOUND
Despite the vast diversity of approaches to sound—ranging from Kandinsky’s spiritual synthesis to Markley’s material experiments—one fundamental imbalance remains common to all: the visual hegemony over the aural. Sound rarely functions as an equal partner; far more often, it serves as an illustration, supplement, or decoration for a dominant image.
Composer and theorist Michel Sion writes: «It is the screen that always radiates power and spectacle, and the image—that focal point and magnet for auditory impressions—is what is adorned with sound in all its untamed splendor.» [6]
The analysis of pre-digital art reveals: despite the goals and approaches of the artists discussed above, a fundamental imbalance persisted—the visual dominated, while the auditory remained subservient, functioning merely as an «ornament» or illustration, as Michel Sion precisely characterized.
Christoph Cox suggests looking at the problem differently: as two independent streams. They run parallel, interact, but do not subordinate one another.
This approach shatters the hierarchy and gives rise to truly hybrid art. In it, sound and image coexist in a complex relationship where neither is a simple illustration or translation. This offers a new freedom for creativity.
Chapter 2: Analysis of Contemporary Installations and Practices
- AUDIOVISUAL SYNTHESIS: SOUND AS THE PRIMARY GENERATOR OF VISUAL IMAGERY
Contemporary technologies are radically transforming media art, creating a new paradigm for audiovisual synthesis. Neural networks and generative systems are becoming «co-creators,» as they enable the translation of sound into imagery at the code level, leveraging their «knowledge base.» This technological shift opens up novel opportunities for artistic exploration.
Pioneers of the Electronic Age: A New Wave of Media Art
In this section of the analysis, I examine multimedia art—from a brief look at the pioneers of the electronic age to contemporary installations. Specifically, the work of the «innovators» Nam June Paik and Alva Noto (Carsten Nicolai). Additionally, Ryoji Ikeda (Japanese Ryōji Ikeda), Robert Henke, Nonotak Studio, 1024 Architecture, and Dreamlaser, as contemporary studios and artists creating installations.
1. Nam June Paik and his work TV Buddha (1974) 2. Installation. Nam June Paik (1993) 3. Alva Noto (Carsten Nicolai)
Joon Pike and Noto
Work by Jun Paik Registration and the sound and image as modulation of electronic signals revealed possibilities for a more direct relationship between the auditory and the visual than previously seen. The electronic connection allowed sound to alter the image, and vice versa, in real time. [3] — p. 274
Tracks that fail in this configuration generate raster patterns… revealing the electronic conditioning of video as a medium… white horizontal bands rise and fall, merge, and diverge on the black surface of the television.
Noto’s work «telefunken» (2000) demonstrates a fundamentally new approach to multimedia. The artist uncovers the relationship at the level of electronic signals. His approach changed the very nature of multimedia art—the auditory and visual generate one another.
Ryoji Ikeda
Ryoji Ikeda: Test Pattern [100m Version], 2013 An example of «post-digital» aesthetics: technology becomes the main actor and the core meaning of the artistic statement. The sound is generated from the same data as the image, creating a complete synthesis of media. Speaking of synesthesia, one can ponder that in this installation, both the image and the sound originate from a single source—the digital code. The artist «re-coded» the data using neural networks.
