Photographs are made by cameras. But what if an image can be made from the memory of a human being? A research and design studio is using AI to do just that.
Real photos are great for recalling memories but when no photographer was present, recollections have no hard imagery. The Synthetic Memories project by the Domestic Data Streamers aims to change that with the help of AI image generators.
The team recently worked with an 84-year-old woman from Barcelona called Maria. Maria has vivid memories of peering out from her balcony as a child to try and catch a glimpse of her father who was incarcerated in a prison opposite where they lived.
These childhood memories only existed inside Maria’s mind, but the AI researchers worked with her to bring these reminisces to life by describing the place and the historical context (Maria’s father had been jailed by General Franco).
“It’s very easy to see when you’ve got the memory right because there is a very visceral reaction,” Pau Garcia, founder of Domestic Data Streamers, tells MIT Reivew. “It happens every time. It’s like, ‘Oh! Yes! It was like that!’”
How Does it Work?
An interviewer and a prompt engineer will sit down with the subject whose memory they are trying to retrieve, as the person recalls a specific event or place the promptographer will input the descriptions into an AI image generator and what follows is a bit of back and forth to get the image right.
“You show the image generated from that prompt to the subject and they might say, ‘Oh, the chair was on that side’ or ‘It was at night, not in the day’,” explains Garcia. “You refine it until you get it to a point where it clicks.”
Synthetic Memories have turned dozens of people’s thoughts into AI images; working with refugees who fled their homes and exploring whether the technology could be used to help people with dementia. When working with older people, the researchers found that they connect with the image better when it’s printed out rather than on a screen.
Older AI Image Models Work Best
The researchers found that earlier AI image models that produce flawed images actually work better for this project. Memories are nebulous and the dreamy, unrefined artifacts produced by AI models such as DALL-E 2 and early versions of Stable Diffusion are better in this use case.
“If you make something super-realistic, people focus on details that were not there,” Garcia tells MIT Reivew. “If it’s blurry, the concept comes across better. Memories are a bit like dreams. They do not behave like photographs, with forensic details. You do not remember if the chair was red or green. You simply remember that there was a chair.”
Garcia says the imperfections are a positive feature for the Synthetic Memories project but he’s worried that companies like OpenAI might retire its earlier models.
“I’m really scared that OpenAI will close DALL-E 2 and we will have to use DALL-E 3,” he adds.
Image credits: Synthetic Memories