How to Use AI and Music to Boost Creativity with Percy Liang, Computer Scientist and Classical Pianist

0:00 Finally, I get to think about how a I especially with the recent advances and generative Ai and generative models, how you can actually have a program that learns to generate music, what fuels, a multi passionate life, I'm Jessica wine. And in this podcast, I interview people who straddle two, completely different worlds, I call them ampersands and we are collectively designing. The Purse and Manifesto. I'm delighted to be talking with Percy of the um computer scientists and classical pianists.

0:43 Percy is an associate professor of computer science at Stanford University. He got his bachelor's from MIT and PhD from UC Berkeley. His research is on artificial, intelligence, more, specifically Foundation models that can generate text images and music. Percy has been playing piano since he was eight years old. He's won several contests including the Phoenix Young Musicians competition, the MIT concerto competition and kdf sees classical Star Search. He enjoys giving both solo piano recitals and playing chamber music, Percy, welcome to the show. Thanks Jessica for having me. I'm really excited to be here.

1:24 When you think back to Childhood, what were some early inklings that pointed toward a future life in computer science and music. Isaac I started playing piano when I was eight and I at the time I liked science and math and eventually started getting into programming. So the to piano and computer science started quickly becoming integral part of my life. I would say that in the beginning piano was I got excited when I first started to learn because I was getting better and then there was a slump I change teachers moved around. That and then starting in eighth grade, I started getting more momentum and since then I've been really trying to keep up with piano up until just a maybe a few years ago I feel like I was improving improving of course things get a little bit harder when you have a full-time job and kids and so on but I majored in computer science in undergrad and at the time I was also actively involved in music. I Would say that obviously professionally, I'm a computer scientist.

2:36 So in terms of actual number of hours, I spent a lot more time working doing computer science but piano has always been to integral part of my life. How do you see piano and computer science connecting with each other or even helping you learn about? The other thing superficially it looks like the two are very different parts of my life. I work hard candy. Our science and research, and then I go play piano. But there are some interesting connections that I found throughout the years because it is one brain up here at the house to manage both of them.

3:17 In one of the aspects is when I had a teacher, Mel, 10 years ago, who really resonated with me, he was actually a scientist, but then he became a professional pianist and then a teacher and I think there was this a Aspect of really architecting performance and thinking about the big picture because often piano pieces, especially if they're technically challenging to have just a lot of notes. And it's very easy to get wrapped up in how to play the individual notes but thinking about the Arc of the entire Tire piece was something that I really learned from him.

3:57 And this idea is actually very useful for for Research when I write papers when I write programs to think about the big picture and the essence of what you're trying to do. And so that's sort of an abstract connection. The more recently, there's been a tighter connection. So, for a while, but I've always wanted to combine the two in some capacity, but never really got the chance to do to apply for another career things. But more recently, I've actually had two postdocs in my lab who were Were applying machine learning to, to music. And this has been actually really exciting because finally, I get to think about how a I especially with the recent advances and generative Ai and generative models, how you could actually have a program that learns to generate music. There's actually, a sort of a personal reason for this, which I'll rewind a little bit.

4:59 So, for in grad school, I was really interested. Then doing this counter competitions. I spend a lot of time away from research, trying to prepare myself. And, you know, I realized that I could never really compete with the people who are doing this full-time. If you practice eight hours a day, I just can't afford that kind of time. But on the other hand, I felt like I had these ideas of how I wanted to express things, but it doesn't have the time to actually execute cute them. So I was looking to machine learning and AI as a way to extend, or augment my capabilities in some way, and I haven't got there yet.

5:45 So maybe one day, I would love to be able to have some sort of way of figuring out how to express my musical ideas. But not have to, I mean, I know this sounds like, oh, I don't have to practice but, but given the limited time, Actually be able to create something with AI that I'm happy with this is absolutely fascinating. I think that we hear so much about AI being competitive with human work, the human creative aspect, but what you're really talking about is how a I could help you create more and create better.

6:30 That's what I'm hearing. Yeah. So there's a little long tradition of AI, which is artificial intelligence versus IA, which is intelligence augmentation, which was something that the human computer interaction Community, really embraced. And so these two communities really worked in parallel, in some ways where a, I was focused on building these independent agents that could mimic human intelligence and operate independently. Whereas I tend to think of a lot more about AI, as IA as tools that augment and help us as humans because after all, we should be helping ourselves in humanity and Society in some way.

7:20 I think that for the piano aspect, this is really in some ways personal because music is so, you know, it's part of my expression. And anyway, I can use To augment and improve that. I would be very excited about what would a brilliant outcome? Look like, what would the augmented Percy be able to achieve? So I have no immediate plans to do this but I would say that I would still like to be able to play the piano. I mean, I could imagine people wanting to just have an alternative Her face, for example, describe it in words or play something very simplified but I do find joy and actually trying to execute and there's a trade-off.

8:12 I feel like when I'm performing can be play with more abandoned and there's more energy and creativity and risk-taking but you play wrong notes and sometimes you mess things up. But so often you have to pull back a little bit and keep your head on your shoulders and And make sure that it's sort of coming out correctly, but maybe there's a way to have your cake and eat it too. If you have something that could fix the mistakes, we see these things in, you know, auto-tune for example, for her, for singing. But I think it goes beyond just fixing mistakes.

8:54 I think these guys could create new suggestions of things that you might not have thought about. I think this could teach us about how we might want to interpret things or express ideas because the way that these as are built they are exposed to such a vast array of different data and experiences that they could capture a lot more than any individual human. Being of course, doing its own limited alien way in some ways. But nonetheless, it could bring new ideas. In. So ideally I would like to learn.

9:34 I would like to better myself. In addition to creating more polished output, we're going to share an excerpt from your recording of Franz Liszt's Piano, Sonata in B. Minor, this is one of the most challenging pieces of piano music ever. It is 30 minutes straight and there are so many theories about this Sonata ranging from being a portrait of the Faust Legend to certain biblical Theory. Is to just being artistic expression, having worked on this piece extensively. How do you interpret it? It's one of these pieces that you learn and then you come back to and you play and each time you come back you see it in a slightly different light and What attracted me to this piece is it has a few motifs in it and these motifs get transformed over and over again and you See them in different ways and it's just a kind of a very coherent cohesive piece of work.

10:42 So it is interesting to see how list was able to take a very simple motive rum pum pum pum pum pum pum and transform it in all these different variations lyrical. Staccato, old gentle and that sort of creativity with. When is very inspirational. And, of course, it's a very Monumental work and it's extremely challenging, especially being able to play for 30 minutes without stopping. So, everytime I play through it. It feels like, you know, there's a sort of a physical and emotional reaction after having gone through. It's a journey.

12:01 Now, being a father of young children and having more life experience, how does this Sonata resonate with you in this particular season of life? I suppose. One thing about raising children is that everything is constantly moving and changing just when you think that you've landed and Certain routine things change and it's very capricious unexpected, and a lot of this is reminiscent of the list. Sonata where everything keeps on moving, it involves over time.

12:45 There are some things refrains where you come back to the same ideas but sometimes I'll be in a different key and it will be slightly different and it's it suggests that life. Raising children is, is always moving to a new place. It's never exactly in stasis. Let's take a short break and pause for a moment of reflection. What in your life is moving to a new place outside of Hosting this podcast. I partner with leaders in the workplace as they rise up, perhaps you've gotten a promotion. Should've landed a new role or taken on a lot more responsibility and you need a trusted sounding board to support you as you support your team. I coach individuals and Leadership teams to rise to the challenge with my a and blend of analytical and creative. If any of this intrigues, you reach out to me at Jessica one.com.

13:50 Now, back to the show, you have mentioned that you Were really, really into competitions. You are currently working on learning all 24 Chopin, etudes. It seems like this yearning for Learning and achievement has kind of been with you throughout your life. How do you decide what that next thing is that you want to work towards for the last 20 years. I've consistently tried to set a goal of giving a recital and I find that This is good because it allows me to share music with other people, but it also gives me a focus point.

14:36 I learned in you to learn a certain amount of music to, to be able to play and not embarrass myself. And so this is led to steady progress and I think there is something really exciting and about growing when you're exposed to new music. Obviously, many of these these pieces, like, let's not Anna you could play every day for the rest of life and you'll see something new but there's so much interesting and mesmerizing music out there. I'm way more than any individual human kid Explorer so just trying to do as much of it. I think is one of the driving factors and for this, this latest project of learning that 24, etudes, I've played my first Etude in.

15:26 High School. Okay, the number number 3, Opus 10 and over time I've learned more and more and about six years ago I decided Well I'll set a goal of Performing the first 12 of them. And I did that which was in itself, a quite a, quite a heavy lift and then so I decided that I would try to, you know, I'm halfway there, I might as well continue and so I'm working on it. It's through the pandemic and having kids, it's definitely a lot, a lot slower and I'm not sure I would perform all 24. I don't know if the audience would want to sit through an hour Chopin etudes or whether my fingers would fall off at the end. But but nonetheless, I will learn them and maybe I'll record them.

16:18 Both piano and computer science are heavily finger Reliant and you've talked a little bit about how you keep your fingers. In shape how do you keep your fingers healthy you know over a long period of years decades of playing piano and also as a computer scientist for piano obviously it's much more physically demanding. Like you're pressing the keys, you're going around and I've been fortunate enough. I think in my career not to have serious risk problems. Knock on wood there have been a few moments. Why I feel like I was at the cusp like I was typing in a lot or playing a lot where it would it would hurt a little bit that was a sign. I should probably hold back as and try to give it some rest. They're also been times where it's not so much the wrist but the actual fingers touching the keyboard over and over again that they get fairly bruised up.

17:19 And I remember once in a kind of competition going and pulling my thing and then I realized that Blood on the piano. So I apologize profusely to the next contestant and try to wipe it up as much as I could. Yeah, it could get really intense. I think actually more important is like the back and the shoulders, if you're into something whether it be at a computer or in front of a piano, I sometimes forget about my physical body until I'm reminded by it. When my shoulder start hurting and That's something that requires obviously taking breaks sometimes, even going between different activities sitting in front of a computer. And I'm going to a piano and going back there. Both can be demanding but there may be demanding, a slightly different ways and that can give some muscles or part of the brain.

18:18 A bit of a break. While the other part as active, what has fueled you to keep going as both a computer scientist And as a musician. So for computer science and AI researcher the idea of having computers that can think and reason and do various tasks that humans are we were usually in the human scope has always been kind of a dream. It's shocking how much progress there is in the field and it's just very exciting part of it is the Research and the ability to find new things and discover on the music side.

19:02 I feel like it's something that my body just needs in a in a way. So why do you why do we sleep or eat or do exercise? It's rejuvenating and it's keeps me balanced, so there's that aspect of it. And for me there's another aspect which is I am curious about how what you can do. So I play a lot of classical piano where the score is written, but there's so much interpretation. And if you look listen to Beethoven Sonata by Richter or five Horowitz or Daniel barenboim, they're all very different and I think developing my own voice and figuring out what I want to say in this music. I find that Very appealing and I guess part of it is also related to that aspect of research.

20:02 How do you say something generally new, right? Because everyone has played many of these these pieces. But what can what can I bring to the table? That's saying something fresh and new. And so in that way, that's a bit similar to the way we think about research. What advice do you have for people who are pursuing or Thinking of pursuing, an ampersand lifestyle, it's challenging, to be honest to have two activities that you both love and want to keep up and you often have to make trade-offs.

20:42 But I think it's extremely rewarding and I think from a practical perspective, I think the the way to make it work is to figure out. How to interleave them, in a way in sort of, in time and mental capacity. Because they do tap into different aspects. And there are complementary, which means that it's actually in some ways, perhaps easier to do the two things compared to doing one thing. But twice as much, for example, after a long day of doing research, I actually really feel rejuvenated to go and practice. But if I were practicing all day long, I would probably not want to practice or my fingers would tell me, not to do practice. So there's there's kind of gaps, I guess it's a in computer science.

21:41 You can call pipelining where there's there's overlapping in in some way because your body has the ability to multitask in a way. What do you think should be in the a saint Manifesto? You know, there is a sense of self that I feel like I am me. And the to the extent that all my activity stem from this single source of Direction, I think makes it sort of easier in a way, right? So I don't think, even though playing classical piano and doing research are distinct activities.

22:24 For the most part, I feel like the motivation and the way that they fit together in my life is is really a synergistic. And I think maybe this is the thing that makes it work for me. It's not that I have two lives. I'm Jekyll and Hyde and they're just like two voices in my head too kind of running around, but it's really one me. And the one me sees both of these is contributing to my personal growth, like professional growth. And as I've mentioned, how the Quest for novelty, the ability to see the big picture, there is a lot of ideas that get transferred over between the different skills in machine learning, we there's a notion of transfer learning where you learn on one task You train a model on one task and it then it can help you on other tasks.

23:27 And this is I think is especially true of humans. But it's interesting from that vantage point to think of the role that music plays in computers science and vice versa. I love this idea of transfer learning and thinking about the many sides of a human, you know, Feeding into the others Percy. It has been fascinating talking with you today, folks. You can check out Percy's work at CSD stanford.edu backslash tilde P. Liang, that's pli. Ang if you liked this show hit, like, subscribe and share with your friends and fellow ampersands. I got a chance to reflect on my interview with Percy, there were so many nuggets of wisdom so I'm going to try my best to distill down to my top 3 1.

24:26 This whole notion of AI vs IA was fascinating to me as a creative. There's so much talk today about chap, GPT replacing humans, for tricking humans, but the way Brucie is approaching this work is around how we realize our full potentials as humans to create and how Can a I be a support system for us to tap into our greater or most fullest ability. There is an aspect in here about The Human Experience of creation and how precious and irreplaceable that is. So, is there a way that technology can actually help us get to the joy of creating faster to Percy talked about pieces that you've come back to over and over again? And how they affect you in different life stages? I have felt felt this so much in my life and not just with music, I perform.

25:21 But with the Arts that I consume one of my favorite movies is the sound of music. When I was a kid. I identified with Gretel, my teenage years. I saw my struggles in the easel. I am now fully in My Maria gears, and maybe even identifying more and more with mother abbess. So, yes, go Climb Every Mountain. Three this theme of one brain. One main came up. Several times in our conversation, many of us a Sands use our Two Worlds to bolster the other and even if externally, they look like different worlds internally, they're driven by the same brain and the same person Sort of implicit in a lot of the framing, as well, you know, I have my professional life primarily our computer scientist.

26:11 And I have this this other life that keeps me sane, but in some ways, maybe that's not quite right. And in fact, it's a two-way street