❔ About

With this post you'll see how I started my first full artwork creating a bridge between:

📜 Data
🎙️ Sound design
🤖 Generative Text To Speech
🖌️ Video artwork
📢 Digital contents streamline
📈 Social networks and content embedding

💡 Inception

What triggered this creation is the following tweet:

Onus Radio

@onusfm

🤫

18:32 PM - 20 Apr 2023

... I immediately started to think:

"... and if I could create a fully digital, multimodal Customer Experience that would be ready to be shared on social platforms ?"

☝️ Also, people are talking a lot about about AGI like Midjourney, DALL-E... but very much less about Generative AI for TTS (Text to Speech).

♾️ " Voice prompts," aka. "History prompts"

As all others AGI, suno-ai/bark makes no exception : it relies on "PROMPTs".

Luckily, the bark's community is very active and share their voices prompt (and tags) discoveries :

🔁 Creative workflow

Here is the current workflow I could experiment:

Create & release a SDK to get the data
Imagine a customer experience at restaurant
Develop & tune the data driven script and build soundtrack
Create an avatar and scene for the waitress
Put together soundtrack & avatar into video

🧰 Tools

Here are the open source tools I used for now:

📦 auptitcafe package to get the data
🎨 bluewillow.ai : "AI Artwork Generator" for avatar creation
🐕 suno-ai/bark to build effective soundtrack
🤏 pydub to compress wav (~ 20 Mo) to mp3 (825.65 kB) & webm (1.68 MB)
🎥 OpenTalker/SadTalker
🦾 T4x2 GPU from Kaggle

🍿 Demos

Below are the demos:

🤓 How it's built (author's words)

🎙️ Soundtrack

Output soundtrack with bark:

🎞️ Movie

Then put the sound into an avatar with SadTalker:

🤔 Ideas for "later"

Automate:

Video creation
Video upload on dedicated cloud services for further optimal collaboration, digital marketing,...
Avatar creation so video is totally code driven... and makes content more original (and funny) on each release thanks to one time generative prompt (prompt design required)

↩️ Conclusion

The more I think about designing - and achieving - such experiences, the more I find evident the core of this kind of project is:

🎯 Get a clear idea and be strongly focused on what you want to achieve (ie. you don't get lost in your creative journey)
🔗 Design a clean linear workflow that focus on tasks (not tools) so you can adapt it easily as AI projects are evolving at an amazing pace (I mean every week there are new tools)

🔖 Resources

🔭 Tools to prototype

HackerNoon | Learn Any Technology

@hackernoon

Turning any kind of input into any kind of output using AI - hackernoon.com/this-ai-can-tr… #ai #artificialintelligence

03:00 AM - 27 May 2023

😋 AGI (bark 🐶) Smart waitress 🎙️