WORKSHOP | From text to sound: Building multi-modal AI agents

A recent demo from OpenAI saw a hand-drawn website on a napkin shared with a version of GPT-4 that could process images. In less than a minute, a photo of the napkin was translated into HTML and JavaScript for a functioning website. Multimodality in machine learning refers to the ability of AI systems to work with and across several different communication ‘modes’ – images, sounds, smell, and text for example.
This workshop will explore rapid advances in multimodality in AI and how “chains” of decision-making and behavior can increase the artistic potential of these systems. We will look at recent work from UKAI Projects and jointly explore generative language models such as LangChain, a framework for developing applications powered by language models. The most interesting applications going forward will connect across various data sources (a chain) to interact with environments without the need for ongoing human monitoring.

Concretely, participants will:

1. Try out multi-modality in large language models.

2. Practice making agents data aware, by connecting them to an audio classification system that is itself connected to ambient noise in the city.

3. Set their agents free to interact with their aural environment and with each other.

No coding experience or familiarity with AI is necessary. While sharing laptops is possible, bring your own to fully benefit from the workshop.

This workshop is presented in collaboration with UKAI Projects, a Toronto-based research collective prototyping digital futures.

During the event, no transcription or interpreter will be present.