Translating Through the Senses: How can intersemiotic translation facilitate literary translation?

Such an inclusive interpretation of ‘language’ means that the process of translation can occur between different sign systems, for example between text and image, music and text, image and dance, etc. This idea is nothing new and is usually traced back to 1959 when Roman Jakobson published ‘On Linguistic Aspects of Translation.’ In his article, Jakobson distinguishes three kinds of translation. ‘Intralingual translation’ is ‘an interpretation of verbal signs by means of other signs of the same language.’ As implied in its name, this type of translation is conducted within one language. For instance, J.R.R. Tolkien produced intralingual translation when he rendered ‘Beowulf’ from old into modern English. The second type of translation, which Jakobson calls ‘interlingual translation,’ is ‘an interpretation of verbal signs by means of some other language.’ An example of this most common kind of translation are renditions of McEwan’s novel Saturday from English into other languages, such as Dutch, Hebrew, German, French, Spanish, Polish, Russian or Japanese. Finally, ‘intersemiotic translation’ is, in Jakobson’s words, ‘an interpretation of verbal signs by means of signs of nonverbal sign systems.’ In this type of translation words are rendered into images, music, dance, etc. An illustration of intersemiotic translation, as Jakobson defined it, is Jeff Wall’s elaborately staged photograph, in which the artist recreates a scene from the prologue of Ralph Ellison’s novel ‘Invisible Man.’

Jakobson’s unidirectional (verbal to nonverbal signs) definition was extended over time to include the translation from nonverbal into verbal signs and multimodal works (such as comic books, films, computer games or artistic performances). As a result, intersemiotic translation has come to signify any translation from one mode of expression into another. It covers the rendering of opera into ballet, music into poetry, a film into a novel, a graphic novel into a theatre performance, a computer game into a film, a text into emoticons, a poem into the tactile mode of Braille, etc. In fact, such a comprehensive definition, as Gideon Toury astutely observes, does not always involve a change of semiotic systems. Therefore, he argues that a division should be made between ‘intersemiotic translation,’ which entails transcoding, and ‘intrasemiotic translation,’ which occurs within one system of signs. Toury’s differentiation seems a resounding echo of Jakobson’s intra- and interlingual distinction. At present, however, most explanations of this term embrace the instances of both intra- and intersemiotic translation.

Despite some anxious voices, such as the one raised by Marta Kaźmierska, that repeat – and rightly so – the warning that overly inclusive terms carry little methodological value, it seems inevitable that the tendency to broaden the scope of intersemiotic translation will continue as new technologies, new forms of communication and new multimodal texts evolve.

However, this inclusive approach to ‘intersemiotic translation’ is also a strength and constitutes a great opportunity: the wide framework allows us to explore a plethora of dimensions and applications of the concept. It enables the analysis of ‘intersemiotic translation’ as an activity, process and product. This broad scope of analysis provides a space in which one can conduct experiments in order to gain better knowledge and understanding of theory and practice of intersemiotic translation.

One of these experiments was a series of workshops that Joanna Kosmalska, a translator and scholar, and Tomasz Wochna, a graphic artist and art director, designed and conducted for M.A. students of literary translation at the University of Łódź, Poland, in April and May 2022. The guidance provided by Joanna and Tomasz functioned merely as signposts; the decisions whether to follow a path or go off the beaten track were made by each individual, and more precisely, by Anna, Daniel, Dominika, Emilia, Jakub, Julia, Krzysztof, Mikołaj, Natalia and one participant who wishes to remain anonymous. It was their lively discussions, perceptive observations, liberal imagination and exuberant creativity that filled the workshops’ framework with intellectual content and vivid colours.

The book was a starting point for our workshops whose nucleus became Jerzy Jarniewicz’s ‘Make-up.’ The selection of this poem – which is intimately connected with pop-art and may be analysed as an ekphrastic poem – was a deliberate manoeuvre that adds yet another dimension to the study of links between literature and visual art.

Make-up equips people with masks which, just like actors in Ancient Greek Theatre, they put on to perform different roles. Make-up is an external mask worn as part of a performance played for society. It represents a plethora of roles that society imposes on the individual and a variety of frameworks into which each person is squeezed. Behind the mask there is a face, a true self. What intrigues Jarniewicz is the tension between the mask and face.

In the poem, the character applies foundation with his/her fingers in order to avoid streaks and reach the areas that cannot be accessed by the sponge. The person starts with the forehead and gradually moves downward to the chin. She/he dabs the face with her/his palms gently, covers up the dark circles under the eyes with concealer, then applies face powder, contours the face with blush, outlines the eyebrows, creates black lines on the eyelids with the cake eyeliner to enhance ‘a sharp and expressive look.’ The text reads like a clear set of instructions on how to apply make-up. Rather than poetic, the language of the poem seems very technical: it is studded with words and phrases borrowed from the make-up jargon, such as ‘concealer,’ ‘powder,’ ‘blush’ or ‘the cake eyeliner.’

In the end notes, Jarniewicz discloses he relied heavily on the press and internet for the ‘language of make-up art’ and citations. The poet uses make-up jargon and quotations – not because they carry some specific meanings or because they render his ideas with precision – but because he finds their form intriguing. He experiments with the language for its own sake. The poem is thus a disguised exercise in intertextuality: the words borrowed from magazines and the internet are reused in the poem. This method shows that the poet does not have to produce new phrases but can operate on the existing ones and defamiliarise them in some way, for example, by placing them in a new context.

What the poet does with the language is reminiscent of how pop-artists create their works. Not unlike Andy Warhol, who takes the publicity photograph of Marilyn Monroe, remodels it and displays it in a gallery, Jerzy Jarniewicz finds excerpts of articles in magazines and on the internet and presents them in the form of a poetry book. The poem may, therefore, be interpreted as an example of pop-art: mass-produced, ready-made and borrowed from popular culture phrases are remodelled and displayed in a new context. As a result, the set of instructions becomes fragmented, shortened, cut into lines. This has the effect of forcing the readers to pause and read the poem in a different way than they would read an article in the beauty magazine. Also, the technicality of the language stands in direct contrast with the poetic imagery employed by Jarniewicz in the closing lines of the poem, where the character bites his/her lips and ‘the blood/ flows in a gentle wave/ over the cheeks yet to be written.’ While make-up is a mask that the character puts on to play out the assigned role, the blood is symbolic of the character’s authentic selfhood and his/her attempt to break free of social, institutional and linguistic frameworks. If blood is something genuine, authentic, real, then make-up is its apparent contradiction. Jarniewicz’s project of transforming the informative text into a poem, which celebrates each person’s uniqueness, seems a resounding echo of the methods employed by pop-artists.

Another way of interpreting ‘Make-up’ is to look at the character in the poem as the painter who is creating a (self)portrait. Such reading of the poem reveals important questions implicit in Jarniewicz’s work: Can we paint with words? What can a poet learn from an artist? What can poetry filch from art? How can literature, which uses a different medium of expression than art, represent a work of art? What happens with the image when it gets depicted in a literary work? Most people would say that nothing extraordinary: the writer looks at an object and describes it. But in fact, rather than simply recreating the image, the author translates it intersemiotically, or experientially. By using language to depict something that is visual and can exist outside of linguistic structures, the writer transforms the work of art. Jarniewicz, who is also an experienced translator and gifted scholar, explores this relationship between the linguistic and the visual in a series of his brilliant lectures broadcast by the online library Wolne Lektury.

The poet’s reflections on language-image relations laid the basis for our workshops. Having discussed the obvious functions of make-up (such as concealing something, emphasising our identity, stating our belonging, taking on a role, expressing our gender, etc.) and a variety of contexts in which we use make-up (everyday life, TV industry, fashion, religious events, tribal cultures and the like), we interrogated the links between make‑up and a mask.

Our first  encounter with the text of the poem was the bracing game of Pictionary, in which the cards contained words and phrases from the make-up jargon that were used in Jarniewicz’s poem. The game was an instrument with which to investigate the language of make-up art, and the ensuing realisation was a discovery of the technical nature of this discourse.

Following Jennifer Liddy’s tips on translating poetry, we pondered on the rhythm of the poem, its pace, the pauses, the beats and the swirls of energy. Although this activity was skipped for the lack of time, the rendering of the poem into musical sounds would offer a helpful tool to identify those features. Our subsequent challenge was to grasp the meanings and identify the problems in ‘Make-up.’ This task was approached by acting out the text and videorecording the performance with a mobile phone.

This way the poem was at once translated into body language and digital film.The prelude to this exercise was an animated discussion in which the following questions were raised: Can we touch a poem just like we touch a table or fabric? Can we use touch to analyse a poem? Can we read a poem with our body? No sooner was that confirmed than other questions suggested themselves: Can we watch a poem just like we watch a film or a painting? Can we examine a text by translating it into images or film?

The latter questions were taken up by Tomasz Wochna who widened our angle of vision and encouraged us to analyse the issue from the perspective of a graphic artist. Through a series of hands-on activities, he showed us how to use the basic means of artistic expression, such as point, line, plane, colour, value and contrast, to produce meaning. The experiments with different kinds of composition (open/closed, static/dynamic, symmetrical/asymmetrical) led us to an intense debate about the relationship between form and content in art and literature. Equipped with new knowledge, we responded to the challenge of translating abstract concepts, such as ‘perfection’ or ‘togetherness,’ into images. What followed was a smooth transition to the introduction of pictograms, that is, graphic signs which convey their meaning through their pictorial resemblance to physical objects. Examples of pictograms include scientific signs, brand images/logos, street signs, signs at the airport, images of superstition, emoji, etc. To illustrate different ways in which pictograms can function, Tomasz adduced the example of a pictogram that consists of two black and two yellow squares placed diagonally towards each other. The pictogram seen on its own is simply a sign. When the same image is placed on a yellow car in New York, it becomes a signal for passers-by that this is a taxi. If this black and yellow pattern is used on the promotional materials of Yellow Cab Pizza Co., an international restaurant chain, it turns into a symbol.

The demarcation between ‘signs,’ ‘signals’ and ‘symbols’ is based on an insightful study of pictorial representations by Adrian Frutiger, a Swiss typeface designer. In his 1989 book Der Mensch und seine Zeichen: Schriften, Symbole, Signete, Signale (Signs and Symbols: Their Design and Meaning), Frutiger repeats the popular notion that at the root of progress and growth of human civilization is the need of people to communicate with their communities and improve this communication. He traces how before the invention of the Latin alphabet, the ‘illiterate’ population communicated with pictures, symbols, signs, and signals forming a kind of ‘tribal handwriting.’ But with the spread of script in the 15th century, Frutiger argues, people lost their ability to understand pictures and signs and use them for communication to the same extent as before. It seems, however – and Frutiger also notes the beginning of this tendency – that the development of television, the internet, online communication and an increasing production of multimodal texts have prompted a new wave of interest in the symbolic content of signs and images.

Following Frutiger’s mode of thought, which assumed that a message can be conveyed duly by means of signs and images, we made an audacious attempt to render our readings of Jarniewicz’s ‘Make-up’ into pictograms and images. While we were conducting the intersemiotic translations described above, we were simultaneously working on improving our linguistic Polish-to-English translations of the poem. The intersemiotic and interlingual translations were completed side by side in order to investigate in what ways, and if at all, intersemiotic translation can facilitate interlingual translation.

The subsequent discussion, in which both positive and negative sides of intersemiotic and interlinguistic translations were hotly debated, led us to a few consensual conclusions. One of them was that the performing of intersemiotic translation has given rise to a surge of creativity and self-confidence among workshop participants. While translating the scenes presented in the poem into images, body language and film, we grasped the ideas and techniques employed in ‘Make-up’ almost effortlessly. We also noticed the apparent ease with which we identified aspects of the text (such as undisclosed gender of the character, make-up jargon or citations taken from beauty magazines) that were likely to pose translational challenges and found some satisfactory solutions to the seemingly insolvable problems. Another observation was that intersemiotic translation helped us to examine the poem from many different perspectives, and hence allowed for a truly comprehensive analysis of the text.

We investigated a plethora of possible readings of ‘Make-up’ and explored the connections between the poem and art. Our third reflection was that translating a literary work into another medium of expression is helpful in detecting its tone (whether the text is happy, sad, nostalgic, etc.) In The Eight Stages of Translation, Robert Bly notes that translators ‘need the ear […] turned inward toward the complicated feelings the poem is carrying. Each poem has a mood. Harry Martinson remarked that to him a poem is a mood. A poem did not come to him out of an idea, but a poem marked a moment when he was able to catch a mood.’ While sharing our intersemiotic translations of Jarniewicz’s ‘Make-up,’ we formulated an awareness that there was a useful clue to the tone of the poem in the colours we had used in our visual renderings. Another realisation was that intersemiotic translation would work well as an analytical device with which new digital, multimodal literary texts, such as e-poems, could be examined. Finally, intersemiotic translation turned out to be a thoroughly enjoyable experience for all of us. By engaging our senses of sight, hearing and touch, it unleashed our latent energies.

These energies have yielded results that are displayed in the brochure titled ‘Translating Through the Senses,’ which you can download below. Part one of the brochure presents our linguistic and visual translations of Jerzy Jarniewicz’s ‘Make-up’ against the original poem. Part two shows selected stills from Tomasz Wochna’s animation ‘Intersemiotic Space-Time Composition,’ which explores the process of translating poetry into visual language and which was shown at the Experiential Translation exhibitions in Ledbury and London in July 2022. The predominant technique here is a graphic synthesis of basic mediums of expression such as a point, line and plane. In the closing section of this booklet, there is a set of pictures that highlight some of the major stages of our workshops.

Joanna Kosmalska and Tomasz Wochna owe huge thanks to the workshop participants: Anna Kozieł, Daniel Marcinkowski, Dominika, Emilia Mamrot, Jakub Kuciel, Julia Maryniak, Krzysztof H., Mikołaj Kret, Natalia Banaś, Tomasz Dobrogoszcz and one participant who wishes to remain anonymous. This project would not succeed without their intellectual brio, lively engagement and opinionated, thought-provoking commentaries. We are very grateful to Jerzy Jarniewicz for permission to deconstruct and reconstruct his brilliant poem ‘Make-up’ and his unsparing and kind encouragement in our endeavour. We would not take up this project if it was not for Ricarda Vidal and Madeleine Campbell who invited us to be part of their ground-breaking Experiential Translation Network. The workshops, symposia and meetings they organised have been a constant illumination and their kindness and support over the past few months have been invaluable. We are very thankful to Alison Landy for her professional and friendly help in language editing. A deep debt is also owed to Mariusz Chałubiński, Wioletta Wochna and Halina Kosmalska whose technical and organisational assistance made this project possible.

Ellison, Ralph (1952). ‘Invisible Man.’ New York: Random House.

Frutiger, Adrian (1989).‘Signs and Symbols: Their Design and Meaning,’ trans. Andrew Bluhm. New York: Van Nostrand Reinhold.

Hall, Stuart (1997). “The Work of Representation”, in: ‘Representation: Cultural Representations and Signifying Practices,’  ed. Stuart Hall, London: SAGE Publications & Open University, pp.15-69.

Jakobson, Roman (1959;2000). ‘On Linguistic Aspects of Translation,’ in: The Translation Studies Reader ed. by Lawrence Venuti. London: Routledge, 113–118.

Jarniewicz, Jerzy (2009). ‘Makijaż.’ Kołobrzeg: Biuro Literackie.

Jarniewicz, Jerzy (2021) ‘Po co obrazom słowa, a wierszom – obrazy?’ in: ‘Mistrzowska Szkoła Poezji Wolnych Lektur,’ access 3 August 2022, < https://www.youtube.com/watch?v=EMZMKQmUblU&gt;

Kaźmierska, Marta (2018). ‘From Intersemiotic Translation to Intersemiotic Aspects of Translation’ in: ‘Przekładaniec,’ special issue ‘Word and Image in Translation,’ pp.7–35. DOI:10.4467/16891864ePC.18.009.9831

McEwan, Ian (2005). ‘Saturday.’ London: Jonathan Cape.

Tolkien, J.R.R. (2014). ‘Beowulf.’ New York: HarperCollins.

Toury, Gideon (1986). ‘Translation. A Cultural-Semiotic Perspective,’ in: ‘Encyclopedic Dictionary of Semiotics’ ed. T.A. Sebeok, vol. 2, Berlin: Mouton de Gruyter, pp. 1111–1124.

Wall, Jeff (2000). ‘After “Invisible Man by Ralph Ellison, the Prologue”’. New York: Museum of Modern Art.