Machine Learning for Musicians and Artists (Kadenze)

Overview

Have you ever wanted to build a new musical instrument that responded to your gestures by making sounds? Or create live visuals to accompany a dancer? Or create an interactive art installation that reacts to the movements or actions of an audience? If so, take this course! In this course, students will learn fundamental machine learning techniques that can be used to make sense of human gestures, musical audio, and other real-time data.

The focus will be on learning about algorithms, software tools, and best practices that can be immediately employed in creating new real-time systems in the arts.

Specific topics of discussion include:

• What is machine learning?
• Common types of machine learning for making sense of human actions and sensor data, with a focus on classification, regression, and segmentation
• The “machine learning pipeline”: understanding how signals, features, algorithms, and models fit together, and how to select and configure each part of this pipeline to get good analysis results
• Off-the-shelf tools for machine learning (e.g., Wekinator, Weka, GestureFollower)
• Feature extraction and analysis techniques that are well-suited for music, dance, gaming, and visual art, especially for human motion analysis and audio analysis
• How to connect your machine learning tools to common digital art tools such as Max/MSP, PD, ChucK, Processing, Unity 3D, SuperCollider, OpenFrameworks
• Introduction to cheap & easy sensing technologies that can be used as inputs to machine learning systems (e.g., Kinect, computer vision, hardware sensors, gaming controllers)

Syllabus

Session 1: Introduction

What is machine learning? And what is it good for?

Session 2: Classification

This session will cover fundamentals, how to use Wekinator for classification, and an introduction to classification algorithms: kNN, Decision trees, AdaBoost, SVM.

Session 3: Regression

In this session we will discuss the fundamentals of regression, using Wekinator for regression, and neural networks for more complex types of models.

Session 4: Dynamic Time Warping

In this session, you will learn what dynamic time warping is and what it’s useful for, as well as how to use Wekinator for dynamic time warping.

Session 5: Sensors & Features Part I: Basic Signal Processing For Learning

This session will cover retrieving data from devices: Streaming data vs events; Smoothing noisy signals; Throttling, downsampling, and upsampling; First and second-order differences; Buffering & chunking.

Session 6: Sensors & Features Part II: Intro To A Few Fun/Popular Types Of Sensors & Sensing System.

This session will introduce Kinect, Leap, and basic physical computing sensors such as accelerometers, gyros, FSRs, ultrasonic distance sensors, and photosensors.

Session 7: Wrap Up

This session will provide a wrap-up for the course and will discuss practical tools, books, and resources students can access for furthering their work in this field.

Teacher

• Rebecca Fiebrink