Have you ever wondered what a painting would sound like? American artist Shane Guffogg has a special ability called synesthesia that lets him “hear color.” This means when he looks at different colors, he hears musical notes in his head.
Everything you need to know:
✓ Shane Guffogg, an artist with synesthesia, works with AI and musicians to make music from his paintings.
✓ They created a visual alphabet that matches musical chords to colors in Guffogg’s art.
✓ AI-generated music based on Guffogg’s paintings surprises and delights audiences, showing how technology can help artists be creative.
Guffogg loves to listen to music while he paints. It helps him be creative and make beautiful art. But he always wanted to know what his own paintings would sound like if they were turned into music.
So, Guffogg decided to ask some experts for help. He talked to an AI expert named Radhika Dirks. Dirks told Guffogg that there wasn’t an AI program that could turn his paintings into music yet. But she had an idea.
Read more: The 8 best (free) AI music generators in 2024
Guffogg created a visual alphabet to turn paintings into music
Dirks suggested that Guffogg make a special alphabet that matched musical chords to different colors. Chords are when you play several notes together on an instrument, like a piano. Guffogg loved this idea and thought it could help him be even more creative.
To make his color-chord alphabet, Guffogg worked with a very talented pianist named Anthony Cardella. Together, they looked closely at Guffogg’s paintings and figured out which musical notes went with each color. Guffogg would point to a color and say what note he thought it was. Then, Cardella would play the note on the piano until they found the perfect match. They did this for 40 different colors and made a chart to keep track of all the chords.
AI unlocks the music hidden in Guffogg’s paintings
Next, Guffogg met an AI researcher named Jonah Lynch. Lynch was really excited about Guffogg’s idea and wanted to help. He made a special computer program that could “read” Guffogg’s paintings and turn them into music.
To make the program work, Lynch watched videos of Guffogg painting. He paid close attention to how Guffogg’s hands moved and which colors he used. Then, Lynch taught the AI program to match each color in the painting to the right chord from Guffogg and Cardella’s chart.
Screenshot
But Lynch’s program did something even cooler. It had learned about the history of music played on keyboard instruments, like pianos and organs. Using this knowledge and Guffogg’s color-chord matches, the AI program started to create brand new sheet music.
Sheet music is what musicians read to know which notes to play. The AI program was making its own sheet music based on Guffogg’s paintings. It was like the computer was dreaming up new songs inspired by the colors and chords.
When the music was ready, Guffogg listened to it for the first time. He was so amazed and happy that it brought tears to his eyes. Even though it wasn’t perfect yet, Guffogg could hear the same things he heard in his head while he was painting.
Over time, Guffogg, Cardella, and Lynch worked together to make music for many of Guffogg’s paintings. They even put on concerts where Cardella would play the AI-made music live while people looked at Guffogg’s paintings. The audiences were surprised and delighted. Many of them said they could see how the colors and notes matched up, and that it was a totally new way to experience art.
Some people are a little bit afraid of AI and worry about what it might do in the future. Guffogg understands this and thinks it’s important for humans to make sure AI is used in a good way. But he also believes that AI can be a powerful tool for artists.
Thanks to AI, Guffogg was able to turn his paintings into music and share his special synesthesia ability with the world. It helped him be even more creative and find new ways to make art that no one had ever seen or heard before.
In the end, Guffogg’s project shows us that when artists and technology work together, amazing things can happen. It opens up a whole new world of possibilities for the future of art and music.
