How piano apps use Artificial Intelligence to listen to your playing

Gone are the days of having to get a 500 pound piano into the house, tons of sheet music books to store in the piano bench or on a shelf and a weekly lesson with a local teacher. Piano teaching apps have arrived and streamlined everything, including complementing the electronic piano perfectly.  Sheet music appears within the apps, saving thousands of trees, and you can maintain your minimalist life-style while you become a pianist.

What is a piano app?

There are many piano apps these days that can teach you or your child to play the piano right from the very beginning to quite advanced.  Like any app, you download it onto your device and sign up.  Most have a free trial to start you off, so you can try several if you want to see which one you like best.

How do piano teaching apps work?

Piano teaching apps have a mix of videos that show you concepts about musical theory and techniques, and a large catalog of songs.  Songs are usually presented in bite-size lessons and demonstrated either by a live player view or with an animated video.  They take you step by step through the basics and get you playing songs very quickly, often with a backing track so you feel like you’re playing with a band. 

This is a very different experience from learning the traditional way with a teacher, often learning to play scales and boring warm-up pieces that were no fun.  Now you can be playing the songs you love and yearn to play really quickly, along with a virtual band.  Because it’s more fun, you’re likely to play more and improve faster.

Some piano apps “listen” to what you’re playing in order to guide you and give you feedback.

How does it hear what you’re playing?

This is where AI (Artificial Intelligence) comes in.  The app uses AI to “listen” to what you play on the piano or keyboard – taking the place of a live teacher.  It then uses internal algorithms to compare what it “heard” to the notes in the song – i.e. how it should sound.  If it doesn’t match, it will signal that a note was wrong by flashing red.  If it matches exactly, it will signal that it was correct by flashing green.

The piano learning app Skoove uses AI, so we’ll use that as an example.  In certain lessons, Skoove will pause and wait for you to play.  This is because the AI is in operation and has not detected any note being played.  Once you start playing, it might detect that you played a wrong note or played it on the wrong beat.  It will let you know by flashing a red or green line on your device, over the note that was incorrect.

How does AI help you learn?

Learning with an app runs the risk of the student making mistakes and being unaware of them.  An app that “listens” can give real-time feedback to the student, alerting them to the fact that they’ve missed a note or played the wrong note – or perhaps played the correct note, but on the wrong beat.

By flagging these errors along the way, it makes the student read the music more carefully than they perhaps otherwise would, as well as making them listen more intently to the notes they are playing.  By doing this on a daily basis, it drives the student on toward a higher skill level than if they had no real-time feedback.

What do you need in order to use it?

With powerful AI programmed into the app, all you need is either an acoustic piano (a normal, non-electric piano) or an electronic piano or MIDI keyboard.  The AI will listen to the sound of an acoustic piano, as well as listen to a sound produced through the speakers of an electronic piano.  For an even better experience, you can plug your electronic piano or keyboard into your device so that the sound is delivered via MIDI.  This results in an even faster response from the app so you can move along quicker.