Logan: I want to talk about like building a culture of iteration, AB testing, continuous improvement, all that stuff. I kind of have an [00:01:00] opinion that in some companies or some markets. There's an opportunity that opens up and it's just a debate of which company is going to run through and build the equity value that comes out of that.
Logan: I'm not sure if you agree with this, but my sense is that you guys were in the right place, right time with regard to mobile. And you were right about your thesis around language, but it was actually the culture of. incremental improvement and just continuously getting better with the product that has led to the distance, how big of a business you've been able to build.
I don't know if you agree with that, that thesis that it wasn't necessarily inevitable that there would be an 8 billion public company. built around language learning.
Luis: Yeah, I, I don't think it's necessarily the case that a, that an 8 billion company needed to exist in language learning. Um, it's true that we were at the, at the right time with mobile for sure. And we capitalized for that, [00:02:00] uh, capitalize on that. But I really think we were able to create a product that, um, got people very engaged and such a thing didn't exist for language learning.
And I'll tell you. But the biggest step that supports that, 80 percent of our users in the United States were not learning a language before Duolingo. Just, we basically massively grown the market in certain countries. Not in every country, but in certain countries, we've massively grown the market.
Luis: And it's, I think it's just because I mean, the main reason for it is because for the first five years of the company, the only thing we did was work on making Duolingo have higher user retention.
And that's not paid user retention. That's just free. We, we, we were not making any money for the first five years of Duolingo. We make zero dollars. We were not making any money. And because we were not making any money, we also weren't spending any money on marketing. So there was kind of nothing to do. I mean, we.[00:03:00]
We weren't sending any money on marketing. So there was not much to do there. We weren't making any money. So there was not much to do in terms of like, you know, working on the monetization of dueling or anything like that, the only things we could work on are teach better and keep people engaged. And so that's all we did.
So five years, we simply made the product stickier and stickier. And it really made a big difference.
Logan: How do you build a culture of like A B testing in iteration and people willing to take chances, uh, knowing that you might make mistakes and that you can undo them if so?
Luis: It's not like when we started, we thought we're going to build a company around A B testing. This is not, you know, the way it started is we started, we started with the product and we, we would make some changes to it. And, um, we would make like whatever the changes were like, okay, we're going to add a, uh, a screen at the end of a lesson that has fireworks or whatever, random changes.
[00:04:00] And two things were happening. Number one. We would add the screen and we couldn't tell if the thing was better or not, because, you know, we would, we would see that the metrics were like, either kind of flat, but if they were flat, we always found an explanation like, oh, it's because it's because this week is Thanksgiving or whatever it is.
And that's why normally we would have gone down. But because we, uh, you know, added the screen, we didn't go down or something like that, like, whatever it is, couldn't tell if things were actually helping or not. And in addition to that, we couldn't tell which idea was good and which wasn't because we're like, well, if we couldn't tell if that was a good thing or not, we, you know, I don't know, was that a good idea or not.
And so at some point.
Luis: Um, it occurred to us, you know what, we should actually run one AB test and figure out if, if that thing was actually good. Um, and we run, I remember we ran a test. I don't remember exactly what the feature was, but I remember there were like four options [00:05:00] and we, each one of us bet on which the option was going to be best, and we thought of ourselves as very good product people, and it turned out.
None of us were right. And then after that we thought, oh crap, I guess we really had no idea. You know, we ran the AB test. The thing that won was not the thing any of us thought was was gonna win. And then for the next few weeks we ran a number of other AB tests and we always would say, try to predict what was going to win.
And we realized we were actually quite crappy at predicting what was going to win. And that's when we really convinced ourselves, well look. We don't really know what is better for a product. I mean, we have some idea, but it's not, you know, it's better than 50%. They're not much better than 50%. So, you know, at that point we kind of started this rule that every change up to a link was an AB test.
And so we built. A pretty sophisticated infrastructure, which was way ahead of its time by now, I think this was more than 10 years ago. It was way ahead of [00:06:00] its time in terms of how good it was also for the size of company. Maybe Google at the time has such a good infrastructure, much probably better than ours.
But for a small company, you know, it was kind of unheard of to have this infrastructure, but we built an infrastructure that really allowed us to make everything an AP test. And I think that that really made a made a big difference.
Luis: The other thing that I think in terms of a culture, you know, we make We make it clear that it's totally okay to run A B tests that don't succeed.
And, you know, we have the stats and, you know, I don't know off the top of my head anymore, but I mean, we run something around 500 A B tests per quarter. And look, something like 60 percent of them succeed and the other 40 percent fail. That means hundreds of A B tests that we run per quarter and it's okay.
Everybody's everybody's done that. And I also like to tell the story that some of the best features of Duolingo, uh, have been things that I've been vehemently against at first.
Logan: Most notably, the, the streak, I think, was one that you said [00:07:00] you originally pushed back on, is that
Luis: No, the streak actually was not the streak. I was, you know, you don't really remember anymore, but I'm
Logan: Who, who came up with that originally?
Luis: sure I was the one who brought it up, but there are, there's more than one person that will tell you that they're pretty sure they're the one who brought it up. So
Logan: Success has many fathers, right? Or mothers,
Luis: I'm like, I'm not, the thing is, I'm not even lying to myself.
Like, I actually have a memory. Remember, you better remember that, but there are other people who claim that too. So, and I believe them. So we don't really know who actually, so I was not against the streak, but I was against our leader boards. We have this league system in our leaderboards. I was against that because we have tried leaderboards many time groups war and they have failed And this was like the fourth attempt and somebody came and they said, we're going to do leaderboards.
I'm like, no, don't do that. We've tried it. We've tried it many times. It's just not going to work. But they were very, um, committed to it. And it turns out that one of the best things we've ever done, the league system at Duolingo.
Logan: How do you think about like listening to [00:08:00] people's feedback? I've heard you use the example before of like people thought it was too bright. The green
was too bright and they all complained about it. And then you said you changed it and they said, okay, wait, nevermind. It's not too bright, even though nothing actually happened.
How do you think about like trusting the data versus listening to the users and the balance between that and product design?
Luis: But a couple of things to say, the good news about Duolingo, what makes it a little easier about Duolingo is that. We can be the users too. There are some products that you design where you're not the user. For example, for a while, I was, you know, we, we, we have a Duolingo ABC, which is an app to teach you how to read.
I'm not the user. I know how to read. I just, I'm just not the user. I'm also not like four years old. Um, but for Duolingo, I am actually the user. And so for that, we, we do take advantage of that. I mean, everybody that is in the product team at Duolingo. We asked them, and you kind of have to really be a big user of it, and that makes a big difference.
So, [00:09:00] that's one thing, you listen to yourself. Um, when we talk to users, we have to take everything they say with a grain of salt. Um, we just, we have found, that doesn't mean don't listen to them, but we have found definitely do not build features. Based on the things that you've asked. I mean, usually there's a deeper reason for why they're asking for a feature.
So what, what we're interested in is not what's your idea for a feature. We're not very interested in that. We're interested in what, you know, where you're having, where you're confused or where, you know, where you're having trouble or something like that. And, and if you have an idea, we want you to tell us why you have an idea.
What, why is that you're suggesting that? Because that usually gets at what the actual user problem is. Um, so, you know, we, we do, we do use the research, we try to hold that, but ultimately I think what works best for us is we have a group of people at Duolingo, which fortunately has grown over time, but it's a group of people that started with one or two people who [00:10:00] just, because we ran so many AB tests, we had a really good idea of what worked and what didn't.
And by now we have a group of, I don't know how many colleagues, 15 to 20 people in the company who are just. Excellently, um, attuned to what works and what doesn't with an A B test, just they're very good. Um, and I think that's what what's worked the most.
Luis: We just when you have under your belt when you have hundreds of reps where you're like, I tried that and you know, did it work?
No, it didn't work. I tried that and you know that it worked. Didn't work. And you do that hundreds of times. You just get a really, really good nose. And I think that's the thing that has worked the most for us. Having having this group of people,
Logan: I've heard you say that when you find a metric that you want to optimize, you'll actually form a team that focuses on the optimization. How does that is that a cross functional team across product engineering, marketing, like what does a team look like? Maybe give an example of a metric that you thought about, like how it actually comes to be.
Luis: yeah, I'll give you some examples. I mean, [00:11:00] one, for example, is, um, um, metric that we call time spent learning. A while ago, a few years ago, we decided, you know, we should have people spend more time in the app because the more time they spend, the more they learn. It's very simple. Uh, so let's form a team around it.
And what a team usually is, is we have, um, usually two team leads, sometimes three. Uh, usually there's an engineering team lead, and then there's a product manager team lead. Sometimes there's a design team lead, and sometimes there's a, um, a learning scientist team lead. Um, but usually it's two. And then we put in that team, we put designers, we put engineers.
We may put one or two more program, uh, product managers, depending on how many, how big the team is, um, and then they work and they can touch any place in the app after a few months, [00:12:00] they, um, they figure out what moves that metric in the case of time spent learning, what moves the metric is the leaderboards.
And so the time spent learning team mainly touches the leaderboards. That's what they do.
Logan: And as you think about the, uh, the unintended consequences of setting these metrics, I think I've heard the example of the streak. Uh, people were just signing into the app just to keep their streak going, maybe speak a little bit about that or the Groupon example of when a channel gets too saturated and the balance of local optimization versus long term value creation.
Luis: Yeah, this is the tyranny of metrics. I mean, we have to watch out for that a lot. Our teams are so good at optimizing metrics that sometimes they lose sight of what's actually important. They're just, it's just a machine to optimize the metric. [00:13:00] Um, and they're very good at, and by the way, the people that we hire, you know, are usually these people that have optimized their entire life.
I mean, they were probably. The best student in their preschool, just kind of optimizing absolutely massively good at optimizing and they'll optimize a lot of something. And so we have to really watch out and the way we watch out for that is by, we have this process for product review where, you know, every change of the app to the app gets reviewed by a senior group of people who are kind of.
Overseeing the whole thing. So they have to keep in mind, you know, they're the ones who stop spamming us and stuff like that. But, you know, when you just let teams just fully optimize a metric, this leads to spammy behavior. And, you know, I, I, this, we have a story that we use inside the company a lot, um, which is, comes from Groupon from the founder of Groupon, who told us this story, um, who said, look, originally with Groupon, we have this rule that we can only send one email per day per user.[00:14:00]
Because who wants to receive more than one, you know, kind of marketing email per day from a company. And, uh, at some point, some product manager said, well, can we try sending two? And, you know, the founder said, no, we only send one. And the product manager used this amazing argument, which is. Are you against knowledge?
We just, let's just, let's just see. Let's just see. Can we, and of course that's a very hard argument to argue against. No, of course I'm not against. So it's fine. You try it, but there's no way in hell that we're going to launch two. Like we're just going to see what it does. So they, they tried it. They saw what it does.
It turned out the metrics were way too good not to launch it. It would have been irresponsible not to launch it. Like my God, this preached everything so much. And so, you know, of course they, they, they launched it. Um, and, uh, then, you know, I was like, well, we tried to, what about free? They tried three, three was better.
They launched it and they had AB tested their way [00:15:00] to a larger number. I don't remember the exact number, but like seven emails per day. And then immediately after that, the whole channel died. Because everybody started marking their emails as spam because it's too much. And then they just AB tested their way to killing the channel.
And the reason for that is because they didn't have full information about what was going on, all the metrics they could see were good, but they actually couldn't see how many people were marking their emails as spam. And that's always true. You never have full information about what's going on in the user.
You just have some metrics and from what you can see, it's good, but you're not seeing, for example, uh, that maybe you're, you're getting. People to spend more time in the app, but you're actually frustrating them. You're not seeing their frustration. You're just seeing the time and they're like, okay, well, that's good.
And so. You know, we, we combat that by basically, you know, it is okay. I do look at there. So maybe it does. We just don't run. Even if, when people say, are you against knowledge? I'm like, yep, I actually am against knowledge. I don't want to know what happens when you send 30 notifications in one minute. [00:16:00] I don't want to know, sorry.
And so, because, because in general we do things and that's why it's good that we're users of the app. It's like, would you want to receive seven emails per day? Uh, that are all marketing emails. Would you want to do that? No, I wouldn't. Okay, then don't do that. And I think that has really helped us, um, do the right thing here and not drive our product to the ground.
I think a lot of problems, particularly games drive themselves to the ground when, you know, they're just like, well, and we're just going to monetize more and monetize more and monetize more. And it works for a while until at some point, it's just like, you You know, I, by the way, I wouldn't know how it is with Facebook anymore, but at some point kind of Facebook, the Facebook app kind of just had so many ads that even I think we kind of ran into the ground to, um, I mean, I haven't used in a long time, but this is, you know, if they started with like one ad per whatever, 10 posts at some point, it was, I don't know how many ads, you know,
Logan: Do you set constraints on, I mean, push notifications and emails is a very obvious one, [00:17:00] but are there constraints about things that you won't change with In the app, and those are non starters, or is is push notifications really a one of one thing because it can actually be marked as spam.
Luis: no, no, it's not just, um, we have a lot of channels in the app. That our product managers get very good at. So, for example, when you finish a lesson on Duolingo, there is a sequence of screens that we call session end. So this course should be one that says you finished and here are your points, but we start adding other ones.
For example, you increase your streak. Good. You increase your streak. That's another screen, but we have other ones too that are just like, Hey, uh, by the way, do you want to subscribe to a super dueling or we add another one? I've met. Oh, by the way, have you tried our podcast? It's basically an app. We just have decided to start adding things in there and it works every time you add another session and screen, it does what we want, but you can't [00:18:00] have 30 session in screens after you finish one lesson, right?
So this is just like at some point, if you finish one lesson, they have 30 screens that you have to click through the channel guys. I mean, it just, it just makes the whole thing. So, for example, we have limits, you know. We just cannot have more than a certain number of screens at the end of a lesson. Uh, and, and then there's a whole pretty sophisticated system that tries to, it's, it's kind of like an auction.
It's not exactly an auction, but there's, there's a pretty sophisticated system that tries to figure out, okay, I can only show three, which three should I show? Um, that's the type of thing that we then, and there's a lot of channels like that in the app.
Luis: And you know, I'm still red dots. Turns out if you put a red dot somewhere in the screen, people are going to tap on it.
Well, after a while, when we, when that was discovered by, uh, enterprising product managers, the app had like 30 red dots all over the place. And then the channel dies. Cause if there are 30 red dots, you don't click on any of them. There's just 30 red dots. Um, so that that's the type of thing that, that, you know, we use the Groupon story a lot for[00:19:00]
Logan: And then how do those things roll up to a decision? Like, is it does it ultimately land on your desk or someone else's that like, Hey, maybe it might be penny wise pound foolish on on these types of tests. And this is a little hacky or gimmicky to approach it this way.
Luis: every Tuesday and Thursday for two hours. And we do this thing called product review, which is I am there. There's usually three reviewers. There's me, there is somebody from product manager and somebody from design. And every change to the app gets 15 minutes locked and we decide whether it's okay or not.
And after product review, uh, we spend, uh, what we call debriefs for that day. And we just go through everything that we approve or didn't approve. And we make decisions about like. Hey, um, this seems like a, like a more general thing. Let's let's add on. We, we call, we, we internally have been calling them fatwas, which had like, don't do that anymore.
So that's the type of stuff. There's a lot of feedback [00:20:00] in, in this where we, we try to make ourselves a better product team. So we spend, yeah, I mean every Tuesday and Thursday, multiple hours kind of discussing what is okay and what is not okay. In terms of, um, features to add
Logan: If you were an entrepreneur or an executive at a company and we're trying to copy or mimic elements of just how iterative of a culture you all have built, I sort of think like consumer apps, maybe gaming is the furthest extreme of just what a nice edge it's on in which you need to iterate and do as much as you can to keep users and then enterprise software is probably on the far other side.
But if you're trying
Luis: three years ago.
Logan: Yeah, yeah, exactly. Exactly. If you were trying to inject some more of this DNA into a company, is there any tactics or anything that outside of what we've spoken about here or philosophies that you would espouse to a founder that's [00:21:00] thinking about this?
Luis: you know, one thing you did say right is. I don't believe that this iterative approach, it's what's best for every single product. It happens that with Duolingo, a lot of things came together that made it so that it works really well. So, for example, we don't have to pay licensing fees for stuff, but we don't license music that doesn't allow us to play this song in that country or whatever, we don't have to worry about that.
So we don't have to worry. We don't have to worry about that type of stuff. We don't, we don't have, we're also don't run something where it's like life or death, where something changes, somebody is going to, you know, they're going to lose their job or whatever. We can easily just change the app at any point in time.
Um, so there's just a lot of things. That that came together so that it makes it so that we really can just Continually improve the product and what the other thing is that we can deliver the product, [00:22:00] you know We can change the app often when we change the app once a week There's a new version of the app once a week so we can change the app often There's a lot of this that makes it so that we really can change Very fast and can iterate and because we can and we do it And and if what's what that's what you want to do Um, I would say the most important things are make it so that, uh, there's no shame in failing, uh, for the A B tests.
That's very important. Make it so that you can, um, iterate pretty quickly. So, uh, I mean, that's, those to me are probably the biggest things.
Logan: Shifting gears a little bit to marketing, I'm not sure I would have guessed that a professor at Carnegie Mellon starting a company would be the one, especially computer science background, right? It would be the one to build a world class marketing organization and [00:23:00] brand. And it was not intuitive to me.
Um, can we talk a little bit about how you think about marketing and where that fits into Thank you. Duolingo strategy, including maybe brand and paid user acquisition, influencer, organic, all that stuff.
Luis: Yeah, marketing is, has had a long history at Duolingo. I think, um, you know, the first several years. We had a mark marketing budget of zero, so can't spend any money. Just can't. The reason for that was because we were making no money. I thought it was silly to spend money acquiring users. And by the way, you know, when we were launching, everybody was, that's kind of when performance marketing was getting like a, you know, really growing.
We're like, Oh my God, you can just go to Google and pay them five bucks and they give you a user. It's amazing. Just do that. Um, I thought it was silly to do that, you know, pay Google five bucks or however many dollars. [00:24:00] For a user when we were going to make no money from a user.
Luis: So that just seemed like throwing money away So this is why we never got on the performance marketing boat.
We just didn't But what we did, you know, at first marketing for us meant PR. And it actually, when you were small first, when you were small and be 10 years ago, PR was actually quite a strong channel. I think it's a much less strong channel now because you know, it's kind of social media has come taking over a lot of generally news.
Outlets don't have as much traffic as they used to, but that was quite a big channel for us. Um, and so we had a good story for that and that helped. The other thing that helped us is that we had this green owl that was kind of weird. And I think people started taking to it. And so you would see like posts on Reddit or posts that would go viral, but it would be, you know, kind of once every few months, post on Twitter or post on Reddit, or one of these would go viral.
And it had to do [00:25:00] something with the owl or something like that. Um, we became significantly more professional with marketing, uh, when we hired. Um, our first CMO who was Cammy Dunaway, um, Cammy had been the CMO for, um, you know, very fancy companies. She was in the heyday of Yahoo. She was the CMO for Yahoo.
Then she was the head of all marketing at Nintendo, which is just crazy, crazy marketing budget. And then she came to us and then, you know, I said, well, we have no marketing budget. And then she said, obviously I can't operate that way. You gotta give me something. And what she convinced me, and I don't know, I don't remember anymore, the first year she may have convinced me for something like 30 million dollars a market, which to me seemed like a.
My God, what are you doing? And for her, it must've been the smallest marketing budget she ever had in her entire life. Um, but she was really quite a trooper for it because she was like, okay, well, I'll operate with this. [00:26:00] And she started trying a lot of stuff and she hired a really good team. Um, and she started trying a lot of stuff.
The majority of it didn't work. We tried stuff like, Oh, you know what? We're going to try a radio ad for Latinos in the U. S. who want to learn English, stuff like that. Kind of, a lot of it wasn't working, but what we did start noticing was that more and more there were these memes kept popping up relating to our owl.
And we weren't doing much. For that, we were just, you know, we were trying our own marketing stuff, whatever it is we were trying is not that it was working zero, but it was not home runs, but more and more, we started noticing that like, Oh my God, there is another, another mean of some person, uh, that decided to say that they're scared of the dueling while, because it's going to kidnap their family.
And that's when, you know, Kami and a few people in the team and me and all of us decided, you know, what, [00:27:00] why don't we lean in on this? That works, like a lot of the stuff we're working on, like we're doing, you know, it's, it's half ass working. It's not, not working like that. So we started leaning in on it and we had a group of people that she had hired that were just really creative and really had their finger on the pulse.
And then we started, you know, I remember we did one video, which was an April Fool's campaign. Where we hired, um, the idea for the proposed campaign was that we were going to make a video of one of these really cheap looking lawyers, lawyer videos, where they're like, you know, I will get you millions of dollars.
And we thought, okay, let's make a video just like that, like the crappy video saying, uh, for a lawyer that is looking for people who had been kidnapped. By the dual and go out and you know, there was so many good things, by the way, really interesting thing is we actually went to try to get one of the companies that make these crappy videos.
And we talked to them and [00:28:00] we're like, Hey, we want to like, like crappy, like the videos you make. And they're like, what do you mean crappy? Like we make excellent work. Okay. And they kind of didn't. And in the end we didn't hire any of them. We actually hired a very, very fancy company and told them to make it crappy.
And they actually understood how to do that. And it was very good as opposed to tell age.
Logan: Just do your normal thing. Just make it
Luis: Now, that wasn't working so well. We ended up hiring a very fancy company, um, to make a very crappy looking video
Logan: What year is this that you guys are doing this?
Luis: maybe that was 2019, I think,
Logan: So there were some precedent, like I sort of think of, um, dollar shave club and their video back in whatever, 2013, 14, that's kind of the
first. Satirical brand dish video out there. So you guys had some precedent. They did a great job with it.
Luis: We, we did, but we weren't thinking about that. We were, I mean, I do know about that video and everything, but we just went, we were just like, look, this is what, this is what people are saying anyways. So we put out that video and it [00:29:00] went really quite viral.
Logan: And the video, by the way, to be clear, I mean, it's a very, uh. Inside joke
of okay, so so the push notifications you have which I want to talk about in a second people uh made memes that they are aggressive, uh in nature or passive aggressive in nature and so that the owl is going to come And then you made a derivative joke about that, which is if you're a lawyer, uh, you're a lawyer looking for the people that the owl actually stole. So it's like a very inside joke.
Luis: actually made multiple, we made multiple derivative jokes for it. I mean, we made another one where we also, um, as a joke product, we said, you can, you can actually pay us to send the owl to light. Stand in front of you until you do your lesson and stuff like that. Um, so yeah, we leaned in on that and that started working really well.
Logan: and by the way, working [00:30:00] really well as a percentage of how do you even think about the, you guys are so data driven. How do you think about the ROI or the, could you just see signups?
Luis: much, we didn't have much ROI at first. All we could tell was that the videos. Or the things we were posting were being watched a lot. That's the only thing we could tell. Eventually, we ended up getting data because, um, One of the marketing people decided that they were gonna, um, That they were gonna do a, a, you know, We added a screen at the beginning when you sign up to Duolingo, How did you hear about us?
And we did a lot of Um, number tension there, and we figured out there's really a very high correlation. People actually answer pretty truthfully there. It's not 100%, but it's pretty truthfully, and we are able, we use that data, and we're now able to really figure out where things come from, and that really works.
Luis: Um, but at first, all we knew is that a lot of people were watching our stuff. Um, and we weren't paying for it, which, which was great. I mean, people were watching our stuff, and we weren't paying for it.
Logan: How did you, I mean, this is something innately a founder can do, I guess, [00:31:00] but you're, you're taking a chance by making your brand and the icon of the owl, uh, into. A meme even more than just the Internet's already doing, right? And so how did you think about, like, clearly you must have a good sense of humor to find this funny yourself, but there's a risk associated with that, right? You're a learning app. You don't want to be taken as a joke necessarily.
Luis: and we discussed it, there was a risk and we discussed it, but in the end, you know, it, it was a risk, but it wasn't, it didn't seem like a existential risk at first. It's like, well, I just published that video. What's the problem? And, you know, it kept working and we kept doing it more and more.
I think that was, we kind of boiled the frog into it. I mean, it just didn't seem like that big of a risk at first. Um, and I did learn something pretty big here. Um. We hired, um, a person [00:32:00] who, uh, was straight out of college and Kami hired, uh, this person, a woman named Zaria, who's amazing. Um, who, um, you know, I remember, I remember her hiring meeting.
Um, you know, Kami was trying to explain that we should hire Zaria and, you know, I saw her resume, I saw the whole thing and I'm like, Why are we hiring this person? I don't understand. And Cami's like, trust me, look, you have to hire some young people because they understand young people. You don't anymore, Luis.
And, um, just trust me on this one. I'm like, okay, Cami. And Zaria came on and pretty quickly, she said, you know what, you know, where young people are on TikTok. I'm like, my God, TikTok, fucking people dancing. They have, I don't understand it. And, uh, it's like, look, let, let me make this video. They made a video.
They showed it to me. I'm like, I. I don't even understand what this is and I have no idea what this is. In fact, it's not going to work because nobody's going to understand this. This is what I told them. But they said, well, [00:33:00] will you let us? And sure, carry on. I don't think it's going to work. Anyways, they posted a couple of videos and they went crazy viral.
Like, I don't know how many, I lost track, 20 million views on crazy viral video. And I still didn't understand why, why it was even funny. I just did not get it. And one of the things that I understood, which I'd be a major shift, you know, most of my time at Duolingo and even before Duolingo, I spent designing products.
When you design a product, it is important that every single one of your users understand what you're doing. It needs to be understandable. And I was applying that lens to our marketing. I thought, well, everything we put out has to be on this, at least understandable by the world. This thing you put out makes no sense to me.
In fact, most people want to resend it. And what I learned that with marketing, that's okay, especially with things like social media that targeted to the people, you know, somehow TikTok targeted it to the people that understood it. And that was fine. [00:34:00] And so, you know, that, that worked out and we started really leaning in on TikTok quite a bit, and then we started getting worried that we were too reliant on TikTok, at least for our marketing.
Um, uh, so we started, uh, figuring out how to, how to get better at that. Um, we started, we tried the same thing on YouTube shorts and it works on YouTube as well. So by now our TikTok provide about the same number of new users to Duolingo, which is awesome. The other thing is. We started replicating the same formula in multiple countries.
So now we have accounts, TikTok accounts and YouTube accounts on, you know, uh, in Germany, in Japan, in, um, Brazil, in Spanish speaking world, and, and, you know, a bunch of different countries in Vietnam and they've all succeeded and they all use the same formula. And, and so we're, you know, it's, it's worked out really well, but, but the formula, you know, [00:35:00] there's a number of things, but I think the biggest thing about the formula is.
If you look at our TikTok stuff and our marketing stuff, it does, it's not selling to you. It's just funny crap. And like, there's no button that says download now or buy now or anything like that. It's just, we now have built a personality for the owl and, um, people love it. And I think that, that really gets.
A lot of people, you know, it's basically free marketing. We don't, we don't pay for that. It's not true that we don't pay for any marketing, but that, that we don't pay.
Logan: Was there a session in which you needed to come up with what the personality of,
of the owl was? It's just, it's iterative.
Luis: It's, it's, it's evolved.
Luis: And, um, it really didn't mean, I would say the first glimmer of a personality came, this was a many years ago from the passive aggressive notification. This one was one I added and it was, it was as follows, but I didn't think it was going to be this, this big or anything.
It was, and it was because, [00:36:00] you know, we've always. Stop sending notifications if you stop using Duolingo. So we, you know, if you don't use Duolingo, we stop after like, it used to be about five days. We're like, okay, after five days, you know, whatever you've given up, we're not going to spam you forever. And it occurred to me, you know what, if we're going to stop, we may as well tell people that we're stopping.
Why not? So the last notification on the fifth day that we that we started sending was, uh, these reminders don't seem to be working. We're going to stop sending them for now or something to that effect. And it turned out this we didn't expect. It turned out that that really got people to come back because they, you know, it's this passive aggressive thing that they, you know, they felt bad.
That Duolingo was giving up on them. And that is, that is the first glimmer of a personality that was just like, Oh, this owl really wants me to be there and he's got to be passive aggressive.
Luis: And that's, I think that's kind of what gave rise to a lot of the memes, but I mean, you know, we didn't, and, and so the personality of the owl has really evolved over time [00:37:00] and it's been kind of co created by a lot of different Duolingo employees, but also by the community because they've made a lot of this stuff.
And so.
Luis: There hasn't been no, what we have had meetings about, especially for some of the videos, you know, with a lot of the videos, we, you know, we are learning brand after all. And so we don't want to do things that are lewd. But with a lot of the videos we are, um. We're unhinged, and so, um, we do have, uh, a system in place for not posting things that we're going to regret later.
The system has failed a few times. Um, but we do have a whole approval mechanism, uh, for not posting really crazy crap.
Logan: what do you do when you, uh, when it feels it's crossed the line and, uh, and something just offended people in a way that you didn't expect? Do you just take it down and, and move on?
Luis: We usually take, I mean, we haven't, that hasn't happened too many times. Um, [00:38:00] we do take down, by the way, I don't know, good or bad, usually the most offended are the bilingual employees. Um, so we, the, the times we've taken stuff down is more often than not, it's because somebody in the company or a group of people in the company are like, how could you do that?
Um, and so that's, uh, that's what has happened, but, uh, you know, but there was only one time that where there was a major, major backlash publicly, which was when we commented. On, um, the, you know, I couldn't have told him this was going to be that bad.
Luis: There was a, our account, our TikTok account left a comment on a video about the Johnny Depp, Amber Heard thing.
And the comment said something to the effect of like, You guys think Amber Heard is on TikTok or something like that. I think that was the comment and there was a huge backlash because somebody posted saying that [00:39:00] Duolingo does not care about, um, uh, like spousal violence or something like that. And there was a huge backlash to the point where our, um, social media team had to, um, Make their, their personal accounts private because they were getting death threats.
Logan: Wow.
Luis: Um, so that's when we learned to stay away from, you know, there's certain things we just stay away from. Even, even if our comment, even if our comment is, you know, I don't think that comment was particularly incendiary. But, you know, there's some things we just stay away from.
Logan: we had one one time. We have a, uh, we'll joke on TikTok and Instagram reels and YouTube shorts. And we had one one time that was, um, it was when the market had corrected and, uh, it was VCs six months ago versus VCs today. And the joke was VCs six months ago were a Spend as much as you want. Like, [00:40:00] are you, are you really burning as much cash as you can?
We can give
you more money, blah, blah, blah. Then VCs today were, are you sure we need a marketing team? Is that
something? And the joke was that the VCs were flippant in their advice, right? Uh, people took it as we were making a joke about layoffs of a marketing department. Right. And it was. The joke was about us and VCs, but I
understood where they were coming from.
So we got a big, that was a big outrage. And you know, uh, it was something that we ended up, I think we apologized or something, but you just never, if you're going to take chances at these lines, inevitably you're going to touch one and you're not sure. I stand by that was just a good joke. And we were making fun of VCs and not. Founders, but,
Luis: to tell, it's impossible to tell what somebody's gonna, you know, third mentality takes it, you know, they kind of go against you. [00:41:00] We haven't had, that was the last time, this was maybe like a year and a half ago or whatever. We haven't had a big problem since then.
Luis: Um, and I think it's important, by the way, if you're running a marketing team like ours, which I think this is a major home run machine, um, to be okay with it.
So, you know what I did when that happened?
Luis: I mean, they thought I was going to fire a few of them, and I actually walked to the desk of, in fact, the person that posted it, and I said, it's completely okay.
Logan: to stand out is to risk being polarizing in some way. And so inevitably there's going to be some people that don't find it funny. And
inevitably there'll be some percentage of people that are offended. And it's hard to say where it's going to come from. So no, that's, that's great that you went over it and said that that's.
Yeah, the, the, the checks and balances are important in trying to get enough diverse eyes on the things so that you can understand where the totality of your audience, maybe he's going to react to it, but it's the cost of doing business in some ways that inevitably you're going to get hit with some of these things.
Luis: and yeah, we're going to [00:42:00] continue. I mean, and, and by the way, we've had a number of things that we've had to pull. I mean, the most recent one, we had a hilarious video that the marketing team just made. It was hilarious. Um, that we were going to post close to Halloween. We had gotten we had gotten a famous person, like, in there.
It was, it was great, but it had kidnappings. And, uh, the Israel-Palestine conflict erupted right around that time, and so we didn't post it. We were so sad, but we couldn't, I mean, obviously there was no way we should have posted that. But, um, in other times, this would have been an amazing video.
Logan: what about being scrappy on the brand marketing side?
Logan: I'd heard a story about like you guys, uh, Just hanging around waiting for billboard inventory to pop up and then being able to take advantage of it Can you can you tell that story as well as like other examples of just sort of being? scrappy about opportunities that present themselves.
Luis: You know, because historically, our [00:43:00] marketing budget has been pretty small. I mean, it's still pretty small marketing budget. Now it's about 50 million a year, which for a company with a revenue is this, it's about, it was about a fifth of what it should be. Um, you know, we don't, we are very scrappy. I mean, there's this story with, you know, with the billboards, we, and the billboard was going to be a funny billboard.
It was, um, this was a billboard, not necessarily so much for marketing duolingo, but it was for. Getting employees, um, so we just put a billboard in, in the one on one in San Francisco that said something or other, like, uh, own a home, work in tech, move to Pittsburgh, and it just said Duolingo or something like that.
Um, yeah, I mean, we only got that because we got a really good deal. We just hung around for a long time until we got a really good deal. Um, and, but we used that, we used that same idea in a lot of ways, by the way, that is exactly. What we do with our performance marketing. So we do performance marketing.
[00:44:00] We don't do very much. We do, you know, on the order of 10 million bucks a year on performance marketing, a little more than that, but not much more than that. And that's what we do. Um, and what we do with performance marketing is we, we noticed we were in a really good position, which is we don't particularly care which country our users are coming from.
Uh, we don't particularly care at what time we acquire them. We don't, we don't need to acquire them. You know, there's some, some companies that need to acquire them. You know, for a specific date or for something we just don't care. And so one of the things we do with our performance marketing is we just have a system that basically looks at the whole world.
And so if, for example, right now, ads are too expensive in Italy, we just don't do anything in Italy. We move all our budget to France, um, or, or, or whatever, if it's too expensive in France, we move it to Vietnam or something. So it just, we're always shifting budget a lot between countries because we are, we're in this great position where we.
You know, [00:45:00] operating every single, single country in the world, uh, or have users in every single country in the world. And we are agnostic as to where they come from. Um, and so that, that's the type of stuff we do. We're, we're, we're very, um, very scrappy with our budget.
Logan: I heard when you were starting Duolingo, uh, you had no experience in teaching languages, and so you read a bunch of books and they all contradicted each other, or they, they had mixed messages on how to learn, how to teach. I, I don't know if that's true, but if so, like, how did you go about figuring out what the core nucleus is of Teaching language and how to start
Luis: We didn't, I didn't know how to teach languages. I mean, I knew, I knew Spanish because that's my native language, but I had never taught Spanish. Um, Severin, my co founder knew German because that was his native language and he'd never taught German. And so we, yeah, we, we read a bunch of books. We, you know.
There were some commonalities about what all the books said, how to best teach a language. And we kind of took those. [00:46:00] Unfortunately, the books didn't agree with each other. They all had their own method and their own, you know. Um, so we took the commonalities and then we did what we thought made a lot of sense.
By the way, we made a bunch of dumb mistakes that only computer scientists would make. So for example, this was a, this was done. Um, and we thought, hey, this, this, this kind of. Approach to deconstruct everything. Language is just words. So all we need to do is teach you words and how to use them. And if you know all the words and how to use them, you learn the language.
Done. And so, we're lucky though, so that's what we're gonna do. And so, the first, one of the first versions of Duolingo had just, we just were teaching you a list of words ordered by frequency. So you know what? The first word, the most frequent word is the, teach you the first. Then the second most common word, whatever the hell that is, teach you that word, etc.
Turns out that's a very bad way of teaching. Um, and, uh, you know, it's also not true that because you've learned enough words to cover the [00:47:00] first like 30 words or whatever, cover something like 20 percent of the language. It's not true, you know, 20 percent of the language by knowing the first 30 words.
That is just absolutely not true. Um, but that's kind of how we thought about it. So this is the type of stuff that we were doing. Um, over time, we learned that we could run A B tests to figure out how to teach better. And so that we started getting better at that. So we started kind of figuring some stuff out ourselves.
Um, and then, but then we got significantly more pro, um, when we hired our first person with a PhD in second language acquisition, um, right? That was, uh, we should have done that earlier. Um, yeah, we got a lot more pro, uh, and, and that, that has really helped. And now what's interesting is when we got these people with PhDs from second language acquisition, it's not like they know how to teach with an app.
They didn't because they know how to teach in a classroom setting. Um, but we knew how to teach with an app. Now, we had our own wonky ways of teaching with an app, and they had their own [00:48:00] very professional ways of teaching in classroom settings. Um, what has been really amazing is kind of, you know, they've learned how to take everything they know, and some of it applies to an app.
Some of it doesn't. They've learned how to make that kind of closer to the app. And we with the AF and with the AB testing and everything have learned how to go more closer to them. And so at this point, you know, I can, I can say that compared to 10 years ago, Duolingo teaches, I know, way better. Um, but we've understood that actually there's just all these skills that you need to learn and that they don't carry to each other.
Like, for example, you can be amazing at reading a language and have no idea how to speak it, and we just didn't understand that at first. Um, so now, you know, we try to teach you all the different skills that are required to teach. To, to learn a language and, you know, we, we understand that the way to teach better is at first you really need to teach things about how to, you know, uh, how to talk about concretely about things and greetings and stuff like that.
These are not the most popular word [00:49:00] or the most kind of high frequency words at all. Uh, but they are very useful to get started. And so that's, um, I mean, we've learned that.
Logan: in the, in the early days when it was just you and Severin, uh, You know, I think we, we talked about this a little bit before we started recording, but neither of you actually Uh, retained as users of, of the app necessarily and trying to learn new languages.
Logan: Was there a, was there a single product unlock or an aha that you realized about how to retain users or teach people through it?
Or was it just a continuous process of these little tests to get incrementally better every day?
Luis: So what happened with, this is before we launched Duolingo. Um, you know, I made the first version of the Spanish course. And Severin made the first version of the German course, and we were going to learn each other's language. I didn't know any German, and he didn't know any Spanish, and we were going to learn each other's language.
And we ran into this problem that neither of us wanted to do it. I [00:50:00] mean, we would come into the office, and we would say, Hey, did you do your Spanish lesson? And you'd be like, No, man. So, so boring. And I, I thought the same thing. It was so boring to learn, um, uh, German. Um, and that's when, that's when we realized that the hardest thing about learning a language by yourself is staying motivated.
And so that's when we started like adding, um, you know, adding all kinds of game mechanics and like progress bar at the top. At the end of a lesson, we give you points. The lessons should be short. There's all this stuff. And this is before we learned. So we weren't A B testing anything, but we added a bunch of stuff so that at least we.
Swallow taking a lesson. I mean, it's like, okay, well, at least it doesn't feel like a complete chore. Feels, yeah, kind of fun. And that's, you know, we launched with that. Um, and so by the time we launched, it was kind of fun. It wasn't as fun as it is today, but it was kind of fun. We had done no A B testing.
It was just, at least with us. We had made it, you [00:51:00] know, kind of engaging and, but, but the, the learning remained, which was just, um, making us engaging as possible. And so that continued.
Logan: What did you think this could be? Be when you were starting the company, you had been independently successful selling recapture to Google. And so you were able to be a little bit more mission driven in your pursuits, I think, which
my understanding
Logan: was, that was an important consideration in the early days.
Did you think that this would be? A commercial potential public company or was it, Hey, this is just a good service and maybe it's a good business along the way.
Luis: I never thought this could be a public company. Um, I thought. You know, I thought we could do something that would help teach people, mainly English. Um, I thought that that, that was going to be there. I mean, I didn't know how big it was going to get. Um, when we were launching, um, [00:52:00] Rosetta Stone was like huge.
We thought at the time it was huge. Um, it's not even clear that we thought we were going to get bigger than them. We just thought, well, We're free and we're at least going to let people, you know, learn that can't afford to pay and any recession was super expensive. It was like 500 bucks or something like, okay, well, uh, we're going to let people learn that, uh, you know, when they don't have to pay us 500 bucks before we launched, I thought if we ever get to about 100, 000 monthly active users, um, that'll be success.
That's what I thought. And, um, I mean, that's funny. I mean, we get, we get way more than that new users per day now.
Logan: There's a great YouTube video. I think it's around the launch of, uh, of Duolingo, uh, whatever, uh, 14 years ago or 12 years ago or whatever it is, but I believe it's on that video. You make reference to this, this concept of the cap on human [00:53:00] coordination, uh, that historically had been around 100, 000 people.
I think that was true of the pyramids. Panama Canal, getting a man on the moon, that like, that's been the, the, when, when, when physical objects were involved, that was the most we could scale to as, as humanity. Um, can you talk a little bit about that and how you, how you thought that as an influence of starting Duolingo?
Luis: So we always wanted Duolingo to be free and it was one potential way to monetize it. We ended up not monetizing this way, but there was one potential way of monetizing, which, which was, um, look, maybe we can get a lot of people learning a language on Duolingo. And at the same time, while they're learning, they're helping us translate stuff because at the time computers were just not as good today, they're essentially perfect, but at the time they weren't as good and they are translating stuff.
So we thought we could, rather than charging people to learn a language or using ads, we thought we could use their work, um, to help like [00:54:00] translate stuff on the web and then we could make money from that. So, for example, the idea was going to be something like, well. If you write a news piece in English, like if you're CNN and you write a news piece in English, you can send it to us.
And our users, while they're learning English, they could help translate it into Spanish because they're native Spanish speakers, and that's part of their learning. And then once it gets translated by our users, we send it back to CNN and then we charge them for the translation or something like that. We thought that was a potentially a good business model because it, you know, no, no money was involved, uh, at least for the users.
Turns out this is a very bad business model. So we ended up not doing that, but that was, that was one of the original ideas with Duolingo to try to make money. While keeping it free, it turns out a much better way to make money while keeping it free is to use a freemium model, which is you can learn as much as you want, but you may have to see some ads you don't want to see ads.
You can pay us. It turns out that works really well. That was not clear back then, but this would, but this would work so well.
Logan: Yeah. So what year was it when, uh, so, so Layla Sturdy from Capgee [00:55:00] invested and, and told you, hey, You've got to figure out how to make some money here at some point, right? What year was that
Luis: She invested in 2015. I believe we, at that point, Duolingo, she invested a valuation of about half a billion dollars.
Logan: with,
with, with roughly how many users, zero revenue, plus or
Luis: If zero revenue, we must have had, yeah, I don't remember anymore. Call it, I'm going to make up a number, call it a single digit. May one, 2 million daily active users,
Logan: One, 2 million data there.
Luis: actives. And if you looked at monthly actives, it was probably. Uh, call it, uh, I don't know, six, 7 million monthly actives, something like that.
Logan: So, so a rich valuation, uh, if we're certainly for, uh, uh, those times, uh, maybe
Luis: Yeah, we were, the good news is we're growing a lot. So we had just, you know, we, the previous year we had like 10 X. So it looks like we're growing a lot, which was good. [00:56:00] And, but you know, she said to me, look, um, great product, great team. Um, you got me money. And, uh, yeah, she, she took me out to drinks after I started my third drink.
She said, look, you're not going to find, uh, she's used exactly this word. You're not going to find a bigger fool than Google. Like that, this is it. Like, you, you know, we invested, we like you, this is great. We believe in you, but you're not going to find them. You know, you're not going to go get a valuation hike from anybody else.
You're not going to do that unless you make some money. So get it going. And then, um, I gotta start some
Logan: Yeah.
Well, and so now
Logan: at this point, you have a very mission driven company, right, that are very aligned with your democratizing access to learning languages and all that. And so you come back to them and say, Hey, you're a little hungover. After a couple drinks with Layla, he started
noodling it. And you say, Hey, we've got [00:57:00] to figure this out.
I assume not all of the company was, uh, was excited to start showing ads or charging subscriptions. Is that a fair, fair assumption?
Luis: Yeah, no, they really were not. They were, in fact, um, entirely against it. Um, most everybody that joined Duolingo joined because of the mission to really deliver education to everyone. Um, and so You know, I said to everyone, you know, we got to, we got to, you know, somehow pay for this whole operation and what everybody, you know, first question I said is why, well, you got to pay for your salary and you've been paying our salaries this whole time.
What's the what's the problem? You know, that's when I kind of told him kind of what Laila had said, you can't raise venture capital for it. Um, and, uh, you know, people finally were like, okay, okay, fine. I thought we still. Months for people were like, fine, you got to make money. How are we going to
Logan: By the way, when you have that, I mean, a lot of our companies have dealt with different versions of this in [00:58:00] particular now trying to get people back into. Office or whatever it is, right? Different social issues or the mission driven nature of a company and trying to corral it in some way, shape or form.
I think as we're recording this, uh, there's this, there's this large a business called open AI. That's maybe going through some, some
elements of, of this, like, how did you actually go about building consensus around, uh, getting people on board with this? It sounds like you didn't do it by edict. And say, hey, listen,
Luis: No, no, it had to be consensus. It wasn't generally a consensus driven company. It had to, I mean, the way that works, we were, I don't know how many employees we were at that time. Call it 150, maybe, give or take, um, you know, there was clearly a spectrum of how much people cared and also a spectrum of how influential they were.
And, um, you know, I spent time with each. With each individual who either cared a lot or was very influential, like one on [00:59:00] one, and some of them were easy to convince there were a few stragglers who were just like. Really resisted the idea. Um, but, you know, it took, it took a period of, I don't know how long it was.
A couple of months of, you know, people being, you know, uh, upset at, at this. And then when they started seeing the mocks of how, they became even more upset when they started, the first time somebody saw a build of Duolingo with ads in it. People were like, what is this? That's awful. And it's because, yeah, it's true.
I mean, the ads, the truth is the Duolingo product experience is quite polished. And then you put an ad, which is look like, sorry, but most of these ads that you play on apps look like crap. Um, so yeah, it, it, it took, it took quite a bit to, to figure that out, to, to, sorry for people to get used to it. Um, but you know, the explanation that really resonated most with everybody is this.
Luis: [01:00:00] If we make money, we can use it to really speed up our own mission. And it was a hundred percent true. As soon as we started making money, we started growing the employees a lot. Instead of having like four separate product pipelines. Now we have, I don't know how many product pipelines we have. We have like 50 separate.
You know, kind of lanes where people are, are just improving the product. Um, and that's all because we, you know, we make revenue now,
Logan: How did you think about the line between subscription, which my understanding is that drives is a very small percentage of the users, but a high percentage of revenue,
and then advertising, which is the opposite. When you were thinking about this philosophically, how did you come up with where the lines of demarcation were?
Luis: We didn't have too much philosophical. It was a lot of A B testing. The only thing that we decided to do is we're going to put an ad on the free book that we're going to do at the end of a lesson, we will put one ad that, you know, after five seconds, it's capable. [01:01:00] We're going to do that. And we decided we were going to do, and then we added an edict that said.
We're not going to degrade the free user experience further than that. That is it. We're going to degrade it by putting one ad at the end of a lesson. Don't degrade more. Um, we didn't know whether ads or subscription was going to give us. You know, not much more money in retrospect, I think, I think subscription businesses are a lot more well understood now than they were at least digital subscriptions like that for apps than they were a while ago in retrospect, this would have been obvious, but at the time, it was not clear to us that ads or subscriptions, you know, which one was going to be the killer one, but after running it for about a while, You know, a few months it became clear that subscription was so much of a better business.
Um, and so yes, up to a, a, give or take, you know, 8 percent of our, um, monthly active users are paid subscribers and 92 percent use it for free yet for a revenue, give or take. [01:02:00] 80 percent comes from, uh, the subscription.
Logan: And you guys philosophically will not charge people for content, right? Like that's, uh, you don't want to become Rosetta Stone and gatekeep content too much.
Luis: Correct. We don't charge people for content. The only things that are okay for us to charge for are things that we have to pay for. So, for example, um, we now have a tier in Duolingo, which is Duolingo Max, which has some features that have, um, large language model usage. We have to pay for that because we don't have our own large language model.
So we have to pay for that. And because we have a variable cost on it, we do put it behind, um, um, you know, you have to pay for it. Um, because otherwise that would be a major money loser for us.
Logan: There's a rule I've heard you guys have that if, uh, if five people or less can build something as well, you won't charge for it. Is that, is that still true? And
how philosophically do you think about that?
Luis: This is, you know, it's one of those internals [01:03:00] she called the five Germans and a dog. Five Germans and a dog can build it. Uh, why are, what, why, why can we charge it? Because, you know, the reason we decided not to do it is because that, you know, that's how you, that's how, honestly, that's just how we beat Rosetta Stone.
I mean, we weren't German, but it just made us think that was free. And it was not made by 300 people, it was made by a very small number of people and very quickly we took them over in terms of usage, not in terms of revenue, because we're making any revenue, but in terms of usage, we took them over very quickly.
Um, and so, yeah, we're going to get disrupted if we build something and start charging for it, that can be built, you know, so easily, some company that has a small amount of funding, I mean, for, you know, 5 people, you need 5 million bucks of funding or less than that. We can build it, I mean. Eat your business.
So that, that's been the rule. Um, we're getting less worried about that these days, just, [01:04:00] you know, that was more of a worry a few years ago when, when our dominance was not as big. I mean, at this point it's just, we have a lot of moats and it's, it's become pretty hard to compete with us. So we're a little less worried about that, but that was a worry.
That was a pretty major worry. I'd say, you know. Six years ago.
Logan: So now you're actually branching outside of languages. Uh, how do you think about doing that? And then what's the hardest part? Is it, is it getting the brand?
Logan: Is it the content? Is it rebuilding elements of the product? What's the hardest part of doing that?
Luis: It's creating the whole learning experience. Um, you know, we're branching out to math and music. Um, there's some differences in how you teach math and music to languages. And so we're creating the learning experience. That's to me. That's the hardest part.
Logan: What's the most interesting difference of how someone would learn a language versus math or music?
Luis: Well, math needs more explanation. Then languages. Um, it's also the exercises and there's a lot more graphical elements to it. It's just a lot of math. I [01:05:00] mean, some of it is just numbers, but a lot of it, it just, especially when you're going to teach geometry and stuff like that, it's a lot more graphical stuff and, and, and you have to start really, you know, you have, there's a lot of things you have to kind of.
Figure out that we've now just figured out with languages. I mean, we figure out, for example, with languages, I'll tell you, it took us many years to figure this out. And in retrospect, it's obvious, look, if you're going to teach a language, you should align yourself with this thing called the common European framework of references called the CEFR.
Just do that. Don't deviate. Um, there's no such thing for math. For math, there's something like it's called like the common core, but it's not clear to us that, that the common core is as good as, uh, CEFR. And so that's the type of stuff that we just don't know. And also for math, um, we're not entirely sure who our audience is.
There's our audience we want, but there's an audience that we may end up getting that is not what we want. What we want is everyone. We want to do for math what we did for languages, which is, [01:06:00] you know, 80 percent of our users were not learning a language before Duolingo. We want it to be the case that.
Anybody who's 25 years old or 30 years old or however, or if they're eight years old, they just used Duolingo to learn math and they get better at it. We want that to be true. It may end up being that the only people we get are the people who are forced to learn math because we just can't make math fun enough.
Uh, but what we would like is, is to get to everyone. And with math, this is a real open question. Can we get it? So that your average 35 year old decides to spend five to 10 minutes a day, getting better at math. Can we do that? I don't know. That is what the product team is tasked with. And we have a really good product team, but I don't know. They'll succeed.
Logan: Because it's interesting, and you alluded to this, of getting to spend five to ten minutes a day, but Uh, you don't view your competition as Rosetta Stone or a private language tutor or, uh, whoever. It's like, how do you think about your [01:07:00] competition and, and what you're going up against for that individual?
Luis: at competition is time. Um, and, and, you know, if you want to be more precise, it's time on the phone. So a real, you know, when we talk to our users and we ask them, you know, the ones who like, stop using Duolingo, what would you stop? I don't know what we've done. I'm sure they exist, but I don't know. I don't know.
We've done any user research where they've told us they've gone to a competitor, like a, like a language learning competitor. That's not what people say. I mean, the most common thing that people say, well, the first thing they say is I haven't stopped, which they're just lying about once they, once you convince them that, yes, you know, that they have stopped, they'll tell you what to be honest, I'm just spending more time on Tik TOK or Instagram or whatever the hell it is.
So our real competition. Is, is, you know, the apps, games, uh, whatever it is, the, the things that take up your time on your phone.
Logan: Is there an optimistic point [01:08:00] about Humans and their desire to learn embedded in that that they're willing to waste time on TikTok or Instagram, which presumably I think most of that is, uh, or spend it learning a language. If the language experience is comparable or enjoyable or easy. Like, how do you how do you think about
Luis: I mean, I believe that. Obviously, we haven't gotten there. I mean, obviously, TikTok has more users than Duolingo, but I do believe that, um, as long as you're close to us, as far as,
As long as you're close, I think we have a good, good shot because afterwards, the good news about dueling was afterwards, you don't feel like you wasted your time. You at the very least got somewhat better at Spanish. Um, whereas if you spend 30 minutes, you know, scrolling on Instagram, no, you may learn a couple of things, but really mostly time wastage.[01:09:00]
Um, and so I, we think that as long as we're as close, as long as we're close to how engaging that is, we're in a good spot. And, and again, I think we've gotten somewhat close, but.
Luis: We're not there. Uh, you know, we have, I don't know the latest numbers. We released something like 24 million daily active users. We don't have 500 million daily active users. We got to get there.
Logan: Yep, there's room to grow, uh, in terms of artificial intelligence, you all have been fairly early in terms of using this. Can you talk about your original desire to maybe build some of this stuff internally? And then, uh, where you all are in the evolution of using tooling and what that's going to mean from a user experience standpoint.
Luis: Yeah, for artificial intelligence. Um, you know, we've always from the beginning for duolingo. The idea was we're not going to use humans to teach you. There's a reason for that. And because humans are [01:10:00] expensive. Um, you know, if we want to teach you literally, whatever, hundreds of millions of people, you got to find a lot of teachers and they're expensive and you've got to pay for them.
And we're just not going to use them. Basically, we're going to teach you with With a computer. Um, so that has always been the idea. And if you're going to do it with a computer, you know, you got to make it so the computer can teach as well as a one on one human tutor. And so we've been working really hard to use AI to teach as well as a one on one human tutor.
Um, there was a major shift, of course, in the last year, call it. Because of large language models, I, and I don't think almost anybody else could have told you five years ago that large language models were going to be this good. Um, turns out they're extremely good. So, you know, immediately when they came out, um, we started, you know, applying some of our AI efforts into that.
That's not the only place where we've used AI. We use AI for a lot of different things. Um, [01:11:00] but at least for conversational practice, this is the thing we were never able to do, uh, without large language models, we just could not build a very good conversational partner. But today, uh, we can build a really good conversational partner with a large language model.
Logan: How has, uh, going public made you all operate differently? Has it been a good experience? What would you recommend for, for people maybe thinking about it?
Luis: For, for us, I think it's been an excellent experience. Um, for me, it has been a really good experience. I think there's a couple of things I think we, you know, I heard this from. Um, Rich Barton, uh, who started Zillow and a number of other, you know, amazing companies. He said, look, going public is like joining, uh, the major leagues.
Like everything just operates at a higher level. And I believe that. I mean, just everything started operating on a higher level. You just are forced to do all these things. You didn't have to do before, so you don't do them. It's like, you know, when you get no visitors in your house, you're not really forced to clean.
But if you're going to have visitors [01:12:00] every day, you keep a pretty clean house. Um, it's the same thing. Like everything just, we operate a lot better, I think. For me in particular, I've been very I'm lucky that we have a really good CFO and finance team, so I haven't had my job hasn't changed all that much because a lot of the public company stuff it's done by our finance team, and I'm other than just regular doing regular earnings calls, um, um, hasn't changed all that much for me in particular, but I think I've had a fortunate experience for a number of reasons, but I think the part of the biggest reasons reason why we've had a fortunate experience is that, um, we Got very good at forecasting and we don't over promise.
And I think we've historically haven't, we haven't just been one of these companies that completely over promises. And so I think that that has helped.
Logan: Why do you think there there haven't [01:13:00] been more successful consumer companies as we look today, you all are at your all time high, despite having gone public, you know, everyone else that sort of has gone public in the 2021 vintage or whatever, are definitely below their all time high. I would say there aren't that many businesses that have been created between five and 20 billion, 25 billion in equity value and within consumer, do you think there's, there's some innate cap that you either get super big, like a Facebook or a Google or a Snapchat, or it's, it feels like there's, there's just not a lot that exists at that level versus enterprise software.
There's a ton. Do you have a perspective on that?
Luis: I mean, we asked ourselves this question, um, that there does seem to be something to, once you get kind of big enough, you just immediately turn into like a Facebook, um, There does [01:14:00] seem to be a little bit of that. I think another thing that has happened certainly for a lot of kind of a smaller consumer things.
I, I think. I think a lot of them were fooled by performance marketing. I, I think, um, we were very fortunate, not because we were so smart. It's not that we were so smart, it's just we just, because we were not monetizing, we didn't do performance marketing. But performance marketing is such a drug. It's such an addiction, and And once you're in it, you can't stop it.
And the problem with it is that really cuts into your margins to the point where you just, it's very hard to, it's very hard to ever become, you know, very profitable when you're, when you're relying on a lot of these consumer apps that are not, uh, Instagrams of the world. If you look at where their users are coming from, it's overwhelming with performance marketing.
When you look at like 80 percent of their users are acquired or something like that, you're going to have a shitty margin. That that's that. And so, and then, then yeah, investors are going to think you're a shitty business too, [01:15:00] because you're a shitty margin, for us. We never did rely on performance marketing.
And so our margins are really good because we don't do that. Um, and I think that's it. I think, I think in retrospect, I can understand that again, it's not because we were so smart back then, but in retrospect, I can understand. Look, the truth of the matter with performance marketing is that companies like Facebook or like Meta or, or Google, um, are excellent at extracting all the value for that.
Like, you know, if there is more value to extract, if there's more margin to extract, they'll take it, uh, because that's what they're good at. And so I think that's another big reason, I think,
Logan: Interesting. So Duolingo today has approximately how many people, uh, LinkedIn says 1700. Does that sound about
Luis: no, that's a lot of, um, um, I don't know why that is that there's something, uh, contractors there and who knows, who knows what else randomness there is, uh, we have about 750 employees,
Logan: Seven 50. Okay. [01:16:00]
Logan: Uh, I had heard you or Severn, your co-founder have been or were in every hiring meeting. Is that
still the case?
Luis: both of us, both
Logan: both of you all 750 people you've met
Luis: Well, we've hired more than that because we've also lost some people, um, but
Logan: I guess I
Luis: everything. And by the way, not only that, we've also turned down a lot of people. So, yes, we've been in the hiring meetings of thousands of people,
Logan: How do you how do you make that work time wise? Uh,
and how is that a
Luis: every week. We just do it every week. There's a meeting every week that lasts about half an hour where they present us. You know, the hiring manager comes and presents the case for the person that they're uh, getting the country for sending roles, you know, this goes super fast, particularly for more kind of junior roles.
Sometimes these meetings take a long time and, and, you know, we may go through a lot of, um, back and forth, but. You know, it works, it's not like, I mean, again, it's not like [01:17:00] the majority of the people that apply for Duolingo get rejected much earlier than that. The only ones that get to us are the ones that where everything looks pretty good until they get to us.
So it's not that many per week, it's
Logan: Got it.
So it's not that you've met you don't meet every employee before they join, but you
sign off on
Luis: to, no, we used to meet every employee. That was, that was a while ago. I mean, that probably stopped five years ago. Um, at this point, it's it's really just the case gets made to us, by the way, this is very important. Um, and it's because of this. Everybody else but us. And it's not because we're special people.
I'll just tell you what it is. Everybody else but us has a true incentive to fill the role. The person from the talent acquisition team just wants to fill the role because they're being paid to fill the role. The high end manager has a hole in their organization, so they need to fill the role. So everybody just has an incentive.
So what happens is they're just like. Yeah, I guess this person is good enough. And so they come in with an incentive to hire, whereas we have an incentive to hold the line. [01:18:00] And by the way, yeah, again, it's not because we're such great individuals with such great, it's just a matter of incentives, because we do the same thing.
Because I've seen when Severin wants to hire somebody, he also has this problem. And when I want to hire somebody for my team, I also lower the bar. And so we try to, you know, It's, it's simply, it's not because we are such great judges of talent or whatever, it's simply, I mean, it's, it's a matter of incentives and that's why we, we, we keep
Logan: It's interesting. So it's kind of human nature if it's yours, then you see the pain of having to restart it and so
Luis: And that's exactly right. I mean, when I'm hiring for my executive team, I'm just like, my God, I've been looking for six months. Look, this person's pretty good. They're pretty good. They're pretty good. And then if I show it to somebody who's way more impartial, they're like, really? Come on. And so that, I think that's what it is.
It's simply that.
Logan: Beyond just being a good person, what's a, what's a non negotiable for someone joining Duolingo? And maybe, maybe to the good person example, if you could also tell [01:19:00] the car ride, uh, of what, what you all do. If people hear this, maybe they'll know now to be nice
Luis: Well, we do a bunch of things and, you know, um, we do that, you know, for us, it's important that people are kind. Um, and one of the places where that shows up a lot is how you treat people who are, uh, lower in the organizational structure than you are. Yeah, everybody's pretty nice to me when they meet me, uh, and they want to work for Duolingo, but, um, you know, we definitely see how people, uh, treat, um, some, and this is usually for, for, um, more senior roles, um, where we look at, you know, how they treat.
Much more junior people and you know, I mean, this is one of, you know, we've, I've told this story before where, um, you know, we were looking for a CFO for a long time and, um, one person had everything look good, everything look good. [01:20:00] They did not treat their driver well. And the driver is a person that has worked with us.
You know, not, it's not a duolingual employee, but it's the same, you know, we have a few drivers that we work with, but we send people to the airport, you know, to, to pick them up and, you know, we asked them, was this person good? And they're like, actually, no, they were pretty nasty to me and we didn't hire them.
And that was, it was painful for me not to hire them because I wanted to hire this person because I had been looking, this is an example I had been looking for a long time. At that time, I probably had been looking for like nine months. And this was a very smart person, uh, very competent, but I guess they were not nice to try to, and my sense is if that happens, you know, they're probably not going to be very nice to a level one, you know, finance associate or whatever.
Logan: Beyond that, beyond being kind to people, are there certain traits that you look for, uh, that are non negotiables or is every role kind of specific beyond
Luis: It depends a lot on the role. No, it's not [01:21:00] the role, but more the function. So, you know, in the case of, um, uh, design, you know, we really look at people's portfolio and we really care about polish, you know, some design organizations are okay. That some of the designers don't have as much polish as others for us, every designer that we hire, we want to have polished.
So we look at their portfolio a lot. It just kind of depends on the function. Uh, but I would say company wide, you know, we want people to be kind and obviously really good at their craft. Um, that that's just, you know, kind of obviously. So there's one other thing that we do look for. It's it's it's which which we try.
We really want people who are, uh, in one way or another exceptional and, and we, we look for that quite a bit and exceptional can mean different things. I mean, in some cases, you know, exceptional is the usual kind of the company definition of exceptional, which is like, went to Stanford, got a 4. 0, um, et cetera, et cetera.
And that's pretty exceptional. Honestly, [01:22:00] very few people were able 0 from Stanford, so that's pretty exceptional, but we look for a broader definition of exceptional. First person in their family to go to college. It's pretty exceptional. Um, comes from a country, from a very poor country. Actually grew up in a very poor country.
And somehow managed to, you know, go to the university in France. That's pretty exceptional. Uh, so stuff like that. Um, you know, we, we, we, we try to make it so that everybody we hire has something exceptional.
Logan: What have you learned about, I guess, both leading and managing over the course of running Duolingo?
Luis: definitely I've learned how to not become, how to not be a micromanager. I actually believe that if you're a small company, you know, under 20 employees. It really is actually a good thing for you to be a micromanager, like, why not? Um, after a certain number, you just can't [01:23:00] be, and then after you start hiring very senior people, you piss them off.
Um, so I've learned not to be a micromanager. Um, you know, there's a number of other things I've learned. That ultimately when there's a fuck up, not to point fingers to anybody, but yourself, and we actually have made that pretty good part of the culture of bullying, especially in some functions where, you know, there's fuck up something every day.
Um, and we're, we've gotten pretty good at, you know, it's just like, okay, well, keep going. And as a leader, I think it's really important to be very, very ready to blame yourself. Um, I think a lot of leaders. I've seen it with, you know, even with people who were in over the years report to me, were they really trying not to get blamed for something like, no, no, no, somebody else did it.
I think people respect that a lot more when you say, look, ultimately I'm responsible for it. [01:24:00] And, you know, by that I'm responsible for all the Duolingo fuck ups and we've had hundreds, maybe thousands.
Logan: It's probably safe to describe yourself as a, uh, accidental entrepreneur. Is that, is that fair?
Luis: I did not grow up thinking I would be an entrepreneur.
Logan: And then Captcha, then Recaptcha, then Duolingo, I guess, um, when Recaptcha sold, how many people did you have?
Luis: Oh, not a lot. Um, that doesn't,
Logan: Got it. What, what, what part of the entrepreneurial journey, uh, do you innately enjoy the most?
Luis: I don't know if it's the entrepreneurial journey that I enjoy it more. And I enjoy doing things that have impact that part. I enjoy. So I enjoy, I'm enjoying a lot what I'm doing right now with Duolingo, but it's because I'm still allowed to touch the product. Um, the day I'm no longer allowed to touch the product.
I, my job satisfaction will go down quite significantly. [01:25:00] Um, so I like that. I mean, I like tinkering. I like being able to have a lot of, um, say in what In, in, in what goes into the end product of a, of a consumer product.
Logan: I've heard you say that working out has been a life changing thing
for you. When did that start? Uh, and, and how, how has it actually changed your life?
Luis: I mean, it started out, it started about right around when I started Duolingo. Um, I just think it was a streak. I decided to have a workout streak and, um, I haven't, I haven't lost it. Actually. It's been, you know, however many years, 13 give or take years,
Logan: Every, every day or Every
Luis: day, every day, every day. I have not lost it.
I did cheat a couple of times as follows. These flights to Japan, there's a problem with the date there.
Logan: Yeah.
Luis: I did cheat by working out twice the day before. And, uh, you know, something like that. [01:26:00] But, but yeah, I haven't lost my streak of working out every single day. Um,
Logan: What's the most bare bone thing you've done to, uh, to make it work?
Luis: uh, yeah, the New York Times seven minute workout. Um, look, because you can always say you can go running, but sometimes it's like horrible weather, you're in some random hotel that has no gym or whatever, and all you got is You, and yeah, you do that seven minute workout a few times and that's all you do.
Logan: How has it changed you? Is it, is it the energy from it? Is it the
Luis: energy. I, you know, I used to get sick quite often, which is funny, which I, now I am sick, but I get sick once every like two years. Um, I used to get sick very often before that. That's one thing. I think the other thing is it really allows me to regulate my own emotions quite a bit. But I think I, you know, I'm generally a pretty emotional guy, but I think [01:27:00] that, um, My wife says I'm, I'm Latin American, Latin American men, but I think, uh, um, uh, you know, it allows me to regulate my emotions.
I think,
Logan: Someone that maybe struggles with that or wants to get going. Is there, is there any advice you would have to them on like how to just start making progress? Is it just get a streak going in a similar way?
Luis: uh, yeah, yeah, yeah, yeah, yeah. And by the way, that's what matters the most. Just get a streak going with anything. I mean, this is, this, by the way, it's the same with Duolingo. Look, a lot of times people are like, no, no, no, no, there's a, there's a better method for learning a language. And it's like, look, ultimately that's like saying the treadmill is better than the elliptical.
It probably is. But you know what matters 95 percent is that you do something. That's, you know, did you do it with a treadmill, with an elliptical or one of those weird climbing machines or whatever? Sure. Probably there are some differences, but between not having done anything and having done even the dumbest one of these, it's better to do the dumbest one of these.
I mean, and that's the [01:28:00] thing with Duolingo, we're optimized for keeping you going.
Logan: Is there, is there another habit or anything you've picked up on maximizing productivity and balancing your, your, you know, work life balance and all of that stuff?
Luis: I mean, I'm pretty efficient at most everything I do. And I think that's, you know, a lot of people comment on that. I'm, I do most everything very fast. I eat very fast. So I, I don't know if it's a habit that I picked for some reason. I'm just very fast at most everything I do.
Logan: I'm the same way eating, by the way. I, uh,
it's always been a problem.
Luis: think that I've been to jail,
Logan: Yeah.
Luis: people think that I've been to jail because I like eat so fast, I eat so fast, but the truth is, I, you know, I love food, but I, I don't savor it in my mouth for five hours,
Logan: Yeah. That's funny. Um, one of the, uh, one of the things I'd be curious about is, um, recapture was sold in, what was it? 2008. Nine ten
Luis: nine,
Logan: nine. Okay.
Logan: Uh, and at that point in time, I don't think it's ever been [01:29:00] disclosed But you didn't raise any outside money and sold for I for you said tens of millions of dollars. Uh, and so Um, and now duolingo is an enormous success. I'm sure we can look up in the s1 how uh, You know your ownership at time of ipl but for safe to say, uh you, you, uh, you've made more money than you can probably spend between both of these things, unless you want to go buy a sports team or something. Uh, but I assume, I don't know if that's in the cards for you, but
Luis: depending on the sports team, depending on the sports team.
Logan: Yeah, Well, in that case, then then there's a lot of stuff. There's a lot more that you can keep going, uh, to buy the sports team. What would be the sports team of choice? If you could.
Luis: I mean, I don't know, I like soccer, but I'm not, I'm not that crazy. I mean, I like soccer, but I'm not
Logan: You're, yeah. It's not going to be the Steelers or the, yeah, one of the Pittsburgh teams. Okay. Um,
Luis: the pirates would be cheap, but
Logan: that's right. That's right. How, um, [01:30:00] how has that most changed your life or how you view the, the, the world, uh, having that level of, of success? It seems like you, you still live in Pittsburgh, presumably a similar-ish life, but is there any way that it's change to the positive versus the negative on stuff.
Luis: Yeah, I mean, you don't worry about some of the stuff that you used to wear. I mean, I remember when I was a graduate student, you worry about like, are you going to have enough money to pay the rent and stuff like that? Um, you know, those, those worries entirely go away. And that, I think, you know, that's, that's quite a nice thing.
I mean, I just don't, and not only that, the other thing is just not having to spend any mental energy on like, Um, you know, does this toothpaste cost more or less than that other toothpaste, et cetera, just, just, I, you know, I basically learned just don't think about money on less, again, unless you're going to go doing some crazy thing, like buy a sports team kind of just doesn't matter.
Uh, and, you know, at least I don't spend the, um. Mental energy and that now for me, I'm, [01:31:00] uh, you know, it's not like I go around spending crazy amounts of money. Um, so it hasn't changed that much how I live my life. I mean, I have, I have like three pairs of jeans. That's it. It's kind of what I have. Um, Yeah, it hasn't changed that much.
I mean, I did, you know, I, right now I'm in New York, I'm in New York and you know, I did buy a pretty expensive house here, but other than that, I haven't done all that much.
Logan: I've heard you say something about like to truly understand something is to be able to explain it simply, or
some philosophy like that. Can you espouse or expound on that a little bit? Uh, for, for
why you
Luis: learned that from my PhD advisor, Manuel Blomi. I mean, it is an amazing thing and it's hard to explain unless you've actually felt it. Um, once you actually understand something, you can give an amazing, clear, simple explanation of it. [01:32:00] Um, and I used to think about that. I just didn't know that. Um, and, you know, I learned that, um, from my advisor, and it was as follows.
It was early on in my PhD, you know, I had to explain scientific paper time. And he said, go read this paper, explain it to me, because I haven't read it. I would come, you know, I came in for a one hour meeting and I said like the first statement and he said, I don't understand. And I'm like, how can you understand?
It's just a simple, a simple statement that I could make you, but I just don't get it. Why did you say that? Why did you start with that? Why? I don't get it. And I spent the whole hour and I couldn't even go after, past the first line of the paper. Um, I came back the next week and it was the same thing. It took me six months to finally get to a point where he said, Okay, I understand.
Meeting him weekly. It turned out he was, he was pretending the whole time, actually. That paper, he was a real expert in, [01:33:00] um, but it took me really six months. And it was true. I mean, the explanation that I had at the end of the six months was just because I really understood it. It was just crystal clear.
And I could explain something that at first I was planning on taking like 40 minutes to explain it to him. I was able to explain the idea in like a minute. There's like, no, this is the main idea of the paper. Blah. And, um. Yeah. But that only happens once you really understand something. And that's, I mean, that's the genius of my PhD advisor.
He's, he, he really understands things. Well,
Logan: I think today there's more people learning languages in the U S. through Duolingo than the education system combined. Do I have that fact right?
Luis: correct. There are more people learning languages on Duolingo in the U S than there are people learning languages across all us high schools combined.
Logan: But, um, I, I heard you say recently that you're not anti the U. S. education system. Uh, and in fact, maybe pro of it. Having learned so much [01:34:00] about How people learn and built a business around it, at least for languages and now increasingly music and math, if you could wage wave a magic wand. Is there anything from a teaching standpoint or how we go about the educational system in the U.
S. That you would change if there was one thing that stuck out?
Luis: It's very different from what we do at Duolingo, but I mean, what matters the most in the educational system are the teachers by far what matters the most. And, um, there are some excellent teachers, but there's also quite a high variance. And, you know, what I would waive the most is what I would waive or want to is make sure that every teacher that is out there is amazing.
Um, but it's pretty hard to do, uh, given a system that doesn't pay teachers very well, given a system where, um, you know, there's essentially equivalent of 10 year in some of these places where, you know, you [01:35:00] basically can work and not do all that much because of teacher unions. So there's a lot of stuff that is.
Working against that, but the thing, you know, you want better education, better teachers, that's, you know, again, very different from Duolingo, uh, Duolingo, we don't use humans, but the truth of the matter is that for the foreseeable future, uh, humans are going to be needed for, you know, the educational system.
Logan: Can you tell the story of Bill Gates? So Bill Gates called you once to try to personally recruit you to, uh, to join Microsoft. What can you give that backstory? I picked it up in something I was researching and I was like, well, I have to get that story, uh, or a little bit more detail into it. Clearly it didn't work.
You never went to go work at Microsoft, right?
Luis: Yeah. And so when I graduated from my PhD, I, um, There's this one thing that happens when you kind of the winner takes all kind of thing. I was the kind of most sought after candidate after I graduated from my PhD. Every university wanted me to become a professor there and, um. [01:36:00] Usually when you graduate from a PhD in computer science, you go become a professor in computer science at our university, but there's another path, which is you could also join Microsoft Research.
And I was the most sought after candidate, you know, every university, every computer science department wanted to hire me. Microsoft Research wanted to hire me, too. And Microsoft Research, I guess, at the time, I don't know, they had a thing where the person they wanted to hire the most that year, um, they would get Bill Gates to call you.
Sorry. Personally pitch you for as long as you wanted to. So I ended up talking to him for like an hour, um, mainly not because I had, you know, I, I love a lot of the stuff that they've done at Microsoft and certainly Microsoft research, but I, I wanted to become a professor. So I had no, no desire, but I wanted to talk to Bill Gates for an hour.
So I was happy to do that.
Logan: Yeah. Well, that's good. I,
if he, listen, Bill Gates ever calls me, I will gladly take his call and I'll spend
an hour
Luis: a smart guy. He's such a smart guy. It's crazy when you enter a room and you're like, he's clearly smarter than me. Yeah, you think [01:37:00] that with him, like
Logan: you, do
Luis: I've been in a room multiple times with him for different things, different than every time you're like, yeah, that guy's smarter than me
Logan: do you remember what you talked to him about or what questions you asked
Luis: at that time. No, I don't too much. Um, but I've talked to him about a number of things. I mean, he's, you know, he's, uh, we've talked about dueling one stuff a number of times again. He's such a smart guy.
Logan: Yeah. Now, uh, last one, I'll let you hop.
Logan: What's the, what's the story of you setting up a honeypot website to catch people that were cheating on your, uh, on your tests when you were a
professor.
Luis: Yes, I was a professor of computer science. Um, and, um, uh, it wasn't tests. I, you know, we were, I was assigning, um, you know, math problems. It was basically discrete math, but it was, um, math problems. We had this rule, which was a dumb rule, but it was a rule that I inherited from the guy who taught the class before me, that was you can't use, you can't Google for solutions.
Um, and so we tried every time that we never, because there's only so many [01:38:00] really awesome math problems. And, you know, we tried to disguise them a little bit, but, you know, we thought, well, probably people are Googling for solutions. And then it occurred to me, well, I'll teach them not to do that. And so I made up a problem.
I specifically made up a problem and then I made a website for that problem. That there's a fully unique problem in a website that had the problem, et cetera. And I own the website. And what happened is I assigned the problem. And I said, of course, you know, as usual, you can't Google or anything. If you Googled.
That problem. You got to my website and I could track who you were on the website. And so, um, 30 percent of the class. This was a class of 300 kids. So about 100 of them fell on the little website and, um, I found them all. And then, um, The next lecture, you know, I didn't say anything. I didn't contact in the next lecture.
I just went in. I stood in front of the class. I projected my screen. [01:39:00] I typed, you know, the query to get to that problem. I got to the website and everybody's just like, and then I said, I did a, you know, to, to know who owns a website, you just do who is and the name of the website. I typed in, in the prompt there, who is.
The name of that website, and then it said that it was owned by me. And then you see everybody's jaw just drop. And then I'm, then I, then I, then I, you know, I'm using, using Linux and showing them, you know, the, the server logs and just everybody's IP address. And I'm like, and everybody was just, they were, um, they were peeing their pants.
Um, and so.
Logan: What'd you do with it? Did you, did they confess or what happened?
Luis: I mean, I knew who they were. Um, so I just gave them a zero, all of them in that assignment. I said, look, I'm not going to ruin your life. I'm just going to give you a zero in that assignment. You can still get an A in the class, but I was not interested in ruining anybody's life. I just wanted to have them not do that anymore.
Logan: Yeah, that's a [01:40:00] good lesson. Well, thank you for doing this. This is fun. I hope,
hopefully you.
feel better powering through here. Uh, yeah. So thanks for doing this. It's a pleasure. And, uh, yeah, this is great.
Luis: Thank you. Same. Bless your mind. Thank you.