Client Progress: Reconfiguring Songs with Juice
This spring I began working with a new client building a social media platform for music tastemakers. I have focused on VFX that add a dimension found in video games.
It is no secret that I have interest and experience in creating audio based tools, instruments, and digital experiences. So, when Máuhan the former head of innovation at Spotify contacted me about building a more active experience to music listening, I was extremely intrigued. He has great ambitions to expand the market for music creators and tastemakers: by breaking apart a song into Lego blocks, we can experience music in a way that is uniquely digital. In order to achieve this, the team at Máuhan’s company Shibuya built a versatile sandbox environment that anyone can easily hop in, listen to music, and play around with it. When I joined, there was a level of polish or delight that the team wanted to enhance. I took this on by relying on concepts from the video game industry.
The core concept we looked to the video game industry is called juice1. Essentially, it is the confluence of feedback (visual, audio, etc.) at moments of player interaction to enhance the experience for the player. Many visual effects you see in games aim to serve this purpose. I started with adding particle effects to different parts of the experience. Below is an example of when you change an instrument. As you drag the object into different quadrants of the grid, the model (and audio which is silenced in this video) change. To enhance this moment, I made a particle emitter that creates white sparks in the direction you are moving. It is further highlighted by a one frame shot of a white glow. The visual contrast draws your attention to the boundary reinforcing the moment you change instruments. The team also added haptics so the phone vibrates a bit each time a change happens. The summation of these feedbacks add to the juice of the application.
Other effects add to the distinct style of the application, which sits somewhere between a traditional mobile application and a video game. Below I created an effect known as a shader that implements a domain warping algorithm. This technical speak means any video can have a textured look without impacting the device’s processing time too much. These types of effects are powerful because they have a small set of parameters (which I am changing in the video to alter the effect) that the team can use as they like. This effect can make many different videos (live-action, animated, slow, fast, etc.) feel in the style of Shibuya’s application. The versatility of the parameters and the uniformity the effect brings to different styles present more opportunities for juice. When should this video effect change? Like the previous example, it could change when you change instruments to further accentuate that moment. It could equally be used in a different setting to draw attention to a different interaction. Perhaps when you speed up or slow down the song.
Deciding when and how to link different effects to moments of interaction is an interesting rabbit hole for me. While there is not a right answer, the process deeply reminds me of two ideas. The first is the concept of transcoding found in media art practices which I researched extensively in my thesis during graduate school at Paris College of Art.2 The second is the design process for both user experience and branding. The Shibuya team spends a lot of time testing these builds with users to see if the experience is intuitive. Sometimes tweaking small things creates large impact (both positive and negative) with a test group. When users “get it” it is rewarding and enhances the feel of the game. But, this juice equally imparts a sense of what the brand or identity of the application is. Because we have decided what rules and moments are linked to interactions, we are offering our vision of what the brand is all about without needing a logo or traditional design element. Does the application feel playful? Serious? Neutral? All of these interactions reinforce an identity.
As this is a progress report and not a completed project, the decision process is fluid. The amount of juice is all a consistent point of debate within the team. Regardless of the state, there is a lot of content on juice: talks, videos, books, articles. There are a few on how it intersects with design3. It is a fascinating concept that has resonant implications for the design process. It has been a joy to tap into the world of juice.
Here is to more juice.
—Jono
Secrets of Game Feel and Juice. Game Maker’s Toolkit.
Charting Artistic Rigor. Jono Brandel.
Serious play. Andy Allen at Figma’s Config 2024.
Feels juicy!