EP 10 — Dustin Lehr: How Fivetran Builds Empathy Between Developers and Security

Dustin: It's great to be here, Harshil. Thank you for inviting me.

Harshil: Fantastic. Dustin, before we go too far down the conversation, I would love for you to just briefly introduce yourself. What do you do nowadays, and a little bit about your background.

Dustin: Yes, happy to. So my background is actually mainly in software engineering. Spent about 13 years as a software engineer plus application architect before I officially got into security. And currently I am Director of Application Security at Fivetran. I also do application security consulting on the side. Previously, I was the AppSec leader at Staples.

Harshil: Fantastic. So that makes you one of the few guests that we've had who come into security from a software engineering background, which in my experience is a little bit different than people who've been doing security for almost all their career. What do you think? Does that give you a different perspective if you're coming as a software engineer into security?

Dustin: I think so, because I have seen the other side of it and experienced the other side for many years before, I would say, getting into it to help change it, you know? It wasn't an opportunity to join security because I loved everything security was doing. It was a little bit more of I haven't had great interactions with the security team over the years and I believe this is important to get right. I've always had a quality focus in general, always felt that security should be part of that equation and really wanted to join the security industry to make a difference and to bring that perspective, right? Bring the developer perspective, help folks understand the right way to interact with developers.

Harshil: That's phenomenal. And that is so important these days, especially a lot of professionals in the industry, they don't really recognize how important it is to have that empathy and to have that other perspective. Because at the end of the day, this is software security, right? It's security of the software that other people are building. So we have to look at it from both those perspectives. What do you think you do differently as compared to other people just because you have that background?

Dustin: I do think there's a lot of focus on empathy in my approach, and partnerships, and sort of building strong relationships. You know, it's not just about, “Here's the rulebook, why don't you just follow the rules?” I think security teams can and should work a little bit harder to make things more accessible for people outside of security.

Harshil: Right.

Dustin: I would say that's the biggest difference. It's really putting yourself in the shoes of developers and thinking about how this is going to impact them, recognizing that they're busy already. So what's the best and easiest and most efficient way that you can interact with them that's not going to bog them down with just one more thing that they have to think about on top of everything else?

Harshil: Right. So if you can, I would love to see an example of something that you actually did in one of your previous roles. Because when I was hearing people like yourself talk and give their advice and insights, I get it, but I don't know how to tactically implement those things. Maybe if you can give me just one example of what you have done with that perspective in mind.

Dustin: Yes, certainly. So I would say trying to focus on providing clarity in terms of the specific actions requested from development teams. Okay, so it's one thing to say, “Hey, here's a fancy dashboard, go do stuff. Like, does this concern you? Like, here's a dashboard, here's how many highs are in your area or that are vulnerability specific for your team”. Isn't that enough? I don't think so. I think that we need to spend more time adding context, prioritizing being very clear with SLAs, and also helping teams understand how they're doing overall because they don't necessarily… you know, development teams, again, are busy, right? They have a lot of stuff going on. The harder that we can work to make that easy for them to consume, “Here's your prioritized list of findings. This one's first. Focus on this first. Here's the date that we would expect it to get done. We've already done the heavy lifting and the hard work to understand the context of that vulnerability. So that's why we came up with that date. Now we're asking you as the engineering team to fix it”. And I think that clarity is lacking in a lot of cases. The approach I see again just to kind of recap is more of a, “Hey, we just installed a really fancy security tool, go check it out. You know, we think it's cool because we're on the security team. Don't you think it's cool too?”. And I just don't think that's the right approach.

Harshil: So wait, you're not sending 55 page PDF reports to developers and asking them to fix everything?

Dustin: Haha. Exactly. No, of course. Bite sized, right? One step at a time. It's kind of a baby step approach.

Harshil: Yeah. Do you also have a security champions program or something similar where you're bridging those gaps together from a people perspective?

Dustin: Absolutely. And I think creating that partnership and that culture shift is just so important. So I'm a big believer in Security Champion programs. I think that they have a lot of benefits for scaling the AppSec operation. I also think with an emphasis on culture shift that they're extremely important to be a catalyst or instigate that change. I also think one of the biggest benefits of having that Champion program is you have representatives, Security Champions on the various development teams and they're shifting culture from within and they already have the respect of the reps of their team, in theory. And that means when they say something, when they bring up something like, “Hey, this is important, I think this is worth looking at”, their team is more likely to listen to them than an external security team. “Dustin the security guy is talking about security again. Whatever.”. You know, it's like easy to ignore versus somebody that they already trust saying the same thing.

Harshil: Yeah, that's a good point. I think the commonly understood things to do when you're running a security champions program is enroll people in your program, incentivize them, train them. Those are all standard things. Have you come across anything unique or different that you have seen work really well in terms of keeping those champions engaged or motivated or really be more impactful?

Dustin: Yeah, that's a great question. I have seen a benefit leaning into not just appealing to their intrinsic motivation, but more their extrinsic motivation as well. What that means is realizing that people don't necessarily naturally care about the security issues so much as having some sort of reward or benefit for them as well, which could be as simple as recognition. Like if they put a little bit of extra effort into fixing a bug or security issue, or reporting a security issue, etc. Trying to create a system that rewards them properly for that type of activity, I think can be very motivating and increase engagement. I'm talking specifically about gamifying your security champion program, thinking about ways to do things like you've heard of Belt systems or Ninja Belt systems or that kind of thing where people can actually gain a level based on the activities that they're doing. I think those programs designed that way can be very effective.

Harshil: Love it. That's a great way to think about it. Motivating with not just intrinsic, but also intrinsic motivations for keeping the engagement up. That's a great point. You've been doing this type of work, you've been in AppSec for long enough. You were at some very mature companies previously as well. Have you seen any shift in the challenges to AppSec that are making successful AppSec programs previously, five or ten years ago versus now? What has changed?

Dustin: Yeah, I think one of the biggest things that's changed that I think all of us as AppSec professionals need to probably spend more time on is the fact that people are moving toward creating cloud native applications. Okay, you know it used to be you create an application, you can run it here, you can run it there, run it in Nginx, run it in Concat, whatever it is. You can lift and shift, essentially. But now the cloud providers have lots of features, services, et cetera that you can use in their native environment that has an impact on how you should design your application. And this comes out in threat modeling, for instance. You know, it's not enough to just simply look at it from kind of an isolated application perspective, but you have to take those cloud infrastructure components into account as well. Are there existing compensating controls in the cloud environment? You know, that kind of stuff. So I would encourage all AppSec professionals to study up and become very familiar with what the cloud service providers offer today, and start to account for those things when you threat model and when you review architecture.

Harshil: Yeah, I love that perspective and I think I can relate to it so much because in a lot of the conversations and even the previous podcast recordings, almost every single episode we talk to people and their roles as AppSec people have shifted. Now, a lot of times they're using product security instead of application security. And that's exactly aligned with what you are saying, which is not just software, but also how the software interplay with infrastructure and cloud. Because at the end of the day, cloud infrastructure configurations, it's also stored in GitHub. Kubernetes configurations, also stored in GitHub. It's everything as code now. So the scope of AppSec has expanded to much beyond the traditional definition of AppSec. And that has implications on what a good AppSec engineer looks like, a good AppSec professional looks like. What are the expectations from you as an AppSec professional. You’ve got to understand not just the traditional software, but also how the software is released, deployed, where it is deployed, what's the environment associated with it. Everything becomes relevant in this case.

Dustin: Yup. And developers are thinking that way as well, right? There aren't hard lines that are drawn like there used to be. Like, “Oh, I'm just a software developer, I don't deal with infrastructure and deployment and all of that”. I mean, we've seen DevSecOps come to the forefront and I think the mindset on the engineering side is the same. They do need to understand not just how their application works, but how it's deployed and how they can utilize effectively the cloud native features as well.

Harshil: Right. And Dustin, I know you've been a big person driven by data and metrics. How have you shifted some of these metrics or what you communicate and how you communicate to account for this broader scope, not just AppSec, but also other things? Like do you talk to different people or do you show different things when you're reporting these overall risks, I guess full stack risks?

Dustin: Yeah, I think it comes down to what I was saying before, around when you're analyzing a system, an application, etc, you should take into account some of the infrastructure components and I think that does affect the metrics as well. You're talking about infrastructure as code and the fact that everything is moving toward being code means that your metrics have to expand to account for that as well.

Harshil: Right.

Dustin: So it's not just how many bones do we find within the particular app code base, but take into account the container image plus some of the infrastructure components as well, for that full picture of your total risk score or landscape.

Harshil: Yeah. I believe you and I had a conversation some time ago where you were talking about a system that you had built in maybe one of your previous companies which would build that centralized posture for applications or teams. Does that ring a bell?

Dustin: Yeah, absolutely. In my previous company, I built out what I called an inventory system. It was essentially an asset inventory system. And what made it extremely useful was we were able to calculate the priority based on several factors of the services across the company, and we essentially built context around it, right? So we took into account the data classification, the business criticality, whether it's public facing, and we actually modified the CVSS scores that were from the tools to be more accurate based on those additional pieces of information. So we essentially added context, and that helped us better prioritize the services that we had to support to help us plan for risk assessments and threat modeling and all of that. It also helped inform us if there was some sort of security concern, what are the systems that we need to put the most resources into protecting? So it took a lot because there's obviously a lot of different moving pieces there in order to collect that information and then map it to services correctly, including things like are your repositories somehow mapped to your services? Like, do you actually know what code supports what specific products in your environment? That's not a mapping that just falls out at you from the data. A lot of times it needs developer input, etc. And then ownership is also another, I would say, challenge for the industry today to properly map, but it's completely necessary because, let's say there's some sort of out of date library, etc., can you connect the dots? Can you say, “Hey, I know this library belongs to this repo, which belongs to this service, which is owned by this team”, right? And that kind of relationship mapping is difficult but it's necessary to get right today.

Harshil: Right. Yeah, and that was one of the things that was most painful during the whole Log4J initially, right? Because finding out usage of Log4J, of vulnerable versions of Log4J, that was not so difficult. The difficult part operationally was who needs to do something about it, right? And who needs to own that? And it's really really hard to maintain that ownership and inventory, especially when you're a small security team outnumbered by developers with a ratio of 1 is to 100, 1 is to 200, something like that, right? It's really hard, you can't keep up. And dev teams constantly keep changing, especially the more agile they are, the more change that you see in the team formation. So it definitely is a critical problem to solve.

Dustin: Yeah, I was thinking a lot of Log4J when I was giving you that example. Imagine being able to type a search for Log4J, get a list, see which teams own it or utilize it, and then just send them an email like, within 5 minutes. It would be nice to have that kind of reaction time to something like that, but it takes a lot of moving parts to put all that stuff together to create a system. I will say that that kind of system doesn't just benefit the security team either.

Harshil: Yeah.

Dustin: This is where I get excited being on the security side because a lot of times we're stakeholders and we prompt and encourage the organization to do things better in general.

Harshil: Right.

Dustin: That sort of repo to service mapping that I was talking about could very much benefit the engineering work as a whole. Like any kind of look up that they want to do.

Harshil: Yeah, I think this is related to the same age old problem that we've had about asset mapping, right? Back in the day, like, forget about software development, but in the world of IT asset management, everyone wants asset management, but nobody wants to own it, right? Like the IT team comes to security, the security team comes to IT, and whoever is good at their job, they end up creating this awesome asset inventory and they become the de facto owner of assets. But that's not how it should be. It's like one of those non sexy problems that everyone wants to solve but nobody actually wants to solve it themselves.

Dustin: Yeah, I would agree. That's exactly right.

Harshil: Awesome. So looking forward, the way the technology landscape is evolving, multi cloud, multi tech stack, different types of frameworks that developers are using, a lot of concerns around open source and supply chain issues and a lot of new buzzwords coming into the industry. How do you foresee key challenges over the next three to four years? What do you think are the big things that as an industry we have to solve for?

Dustin: I think despite all of the new technology, especially as we mature on the security side, in terms of the ability to scan, respond, et cetera, it's still going to come down to people and the fact that we need people's help to ultimately affect our security posturing, right? What good is a fancy tool that's going to show you all the vulnerabilities in your environment if nobody knows what they mean and they can't respond to them, right? So going back to what we were talking about in terms of security champion programs and finding a way to motivate people, I actually think that's the future. I think that's where the emphasis needs to be in this dream that I think we all have where everybody as part of their daily habits take security seriously. How do we actually create cultures where that's reinforced? So that's where I think we're going. I think that with AI obviously there's going to be advancements with tooling and our ability to detect and find issues, but those tools are not going to fix the issues for us, right? It's always going to come down to us designing as people better systems and responding accordingly to what the tools tell us.

Harshil: Yeah, that's a good one. It's a problem that is so fundamental, but it's not technology related. Impacting people and influencing people, it's a really hard problem. I remember a few years ago I had given a presentation, a conference talk at one of the conferences in San Francisco, and the example I used was at that time it was super relevant, which is fitness trackers. So when you use a fitness tracker, I used to use it at that point, it has really strong influencing mechanisms built into it, whether it's the three rings on an Iwatch or a leaderboard that you can add your friends, and what I noticed was a lot of my friends who are into that, they would get motivated just seeing their name on a leaderboard because there was this device that was keeping a constant track of identifying your performance onto that category of things like a fitness tracker, counting the number of steps or number of calories you've burned and so on and so forth. That is such an impactful motivator. Of course, at some point you lose track of it and you stop caring about it and whatnot. But there are other ways to manage it. The point I'm trying to make is if you can influence people, whether it's an intrinsic motivation, they're just passionate about it, or in some other way drive them towards achieving certain outcomes, that is such a powerful thing to have.

Dustin: Yeah, 100%. I do think that's the future and I would say that's where I'm placing my bets because I've actually spent quite a bit of time studying gamification, motivational design, even human thinking fallacies and that kind of stuff to understand what actually does motivate people. Because when you're faced with that on the security side, you walk in the door as an abstract person and your problem is, how do I get people to do things? I need to get people, I need to get the engineers to fix this issue. I need to get people to pass the system, whatever it is. So how do we do that effectively, right? Without just saying it's part of your job. Can we come up with other creative ways to motivate people and understand that human element? You mentioned people get tired of certain experiences. I actually think there are ways to keep people engaged anyway. If the device that you're talking about was designed a little bit better and frankly accounted for later stage experience, it would have been able to keep the interest of folks. There are methods to do that. And at this point I'm way far deep into learning all that. I can share my secrets, but yeah, I don't know. That might warrant a separate conversation.

Harshil: Haha. I can imagine a Slack bot keeping your personal fitness trainer, keeping you up to date on security, things that you need to do. That could be pretty awesome. I read a book a while ago. The book's name is “Hooked” and the author is Nir Eyal. If I remember correctly, that book was all about how to tap into people's motivations, especially how social media companies do it. They're so effective at engaging people and there's just so much to be learned in terms of how to get people to do things for security. If you're thinking about how to get developers or engineers to do anything about security, I think if you're interested in that topic, you definitely need to read that book.

Dustin: Yeah, I have a few other books on my mind too. One of them is called “Actionable Gamification”, which goes beyond just, “Hey, should we gamify things?”, and goes into more details as far as how you should implement these types of techniques and methods and what to do, what not to do. Because if you design them incorrectly, you can actually demotivate people. So let's go back to the leaderboard. Let me just share an example here because I'm excited about being able to share this kind of thing. Leaderboards can work, but you can do them wrong, okay? If you splash up a leaderboard and the person who sees the leaderboard is actually in position 1,000 and there's no way that they're going to catch up to the top five anytime soon, that can actually lead them to be demotivated by the whole experience, right? So in general, leaderboards will help with the top few to keep them competitive. Everybody below them gets a demotivating thing. So how do you design a leaderboard correctly then? One of the techniques is to just show your relative position. I don't care about the leaders at the top. Show me who is above me and how many more points they have. So I can see if Harshil’s above me and it only takes ten points for me to beat them. I'm now motivated, right? “Oh, hell, no, Harshil can't be there. I can do this”, right? So that's just one example where you can actually take these gamification elements and implement them in a way that's not going to produce the desired effect.

Harshil: Yeah, there's definitely… this is a field of study in itself. There's quite a bit of work that can be done in this space. I haven't seen a lot being done in security other than the security awareness training type things, which kind of scratch the surface of this, but not really. I think there's a huge scope of doing more around this within our space of cybersecurity.

Dustin: Yup, I think so, too. And I think metrics and reporting has a lot of opportunities there too. Because instead of just showing people the number of vulnerabilities and that kind of stuff. If you can move it into some sort of a score or a medal I actually don't really like scores. Like grades or even colors. Like green, red, yellow. But more of like an Olympic medal type thing. You know, like silver, gold. That sort of thing. And then what you can do is you can compare the different areas in your organization by VP, by whatever it is, however your org is organized. And a lot of times that can be motivating too. Like, “Oh, I'm only silver in maturity, what does that mean? How do I get to gold?”, right?

Harshil: Yeah. It's a positive reinforcement, right? It's fantastic.

Dustin: It's a little friendly competition.

Harshil: Right. Talking about people and motivating them, one of the amazing things that I believe you've done is you run this community, “Let's Talk Software Security”. Tell our audience a little bit more about the community.

Dustin: Yeah, absolutely. So it's been running for over a year now. About a year ago I noticed that there's a lot of podcasts which are nice, there's a lot of presenters that have kind of like a webinar format, right? I'm going to invite a bunch of people, spread the word and then have a speaker present, which is great. I'm not knocking those. I think they're great, right? However, what I felt that was lacking in the industry was some sort of format where we just get together and talk and everybody's heard, right? So I created this meetup called “Let's Talk Software Security”, and it's in that format. So what we do is every month we choose a topic, we send out invites, we tell people we're going to talk, and then when people join, it's an open discussion. Anybody can join in, can pipe in, can share their opinion, can ask a question. We get all kinds of different positions, from CISOs on the security side, to security engineers, to even developers, to people in other industries completely who are just fascinated and curious about cyber security, and we all talk about the topic and everybody learns a lot and it's just a good old time.

Harshil: That's fantastic. I've attended a couple of those and I loved the brutal honesty and the transparency. I mean, you're not talking about confidential secrets, but also what it actually takes to be in the front lines of building and scaling AppSec. And every single topic being sourced from within the community is just so relevant to that time. And there's an active Slack channel as well, I believe, right? So if any of our audience members wants to join, how would they look you up, look up the community and how can they join?

Dustin: Yes. You can search on meetup.com for “Let's Talk Software Security”. I post about it every month, a couple of times a month, just to get the word out that we're having an upcoming discussion. So feel free to connect with me. Find me and you'll get those updates. That's probably the best way to find it. I wonder if you could probably search on LinkedIn at this point too, and find the post on it. So “Let's Talk Software Security”, just search for that and you should be able to find us.

Harshil: Fantastic. Dustin, this has been such an insightful conversation. Loved our topics and again, you're doing amazing work with this “Let's Talk Software Security” community, I highly encourage our audience members to look it up, join and contribute to the community. Thank you so much, Dustin, for being on the podcast.

Dustin: Thanks, Harshil. It's great to be here. Great chat.