Ep 22 — How to Find the Right Balance Between Compliance and Security with KnowBe4’s Senior Director of Product Security, Bradley Petzer

Harshil: Welcome to another episode of the Future of Application Security. Today I have Bradley Petzer, the Senior Director of Product Security from KnowBe4, as a fantastic guest today. Bradley, welcome to the show.

Bradley: Hi, Harshil. Thank you so much for having me.

Harshil: Bradley, this is going to be an exciting topic. I've known KnowBe4 for several years now. You guys have done really really well. And I was actually super excited that KnowBe4 has somebody senior like yourself in charge of Product Security, Application Security and things like that. This is great news. I am very interested in the topics today, we were just discussing earlier before this podcast. I would love to dive deeper into some of those things but before we do that, it would be fantastic if you can just briefly introduce yourself to our audience. What do you do, where you work, and other things like that.

Bradley: Sure. So, as Harsil mentioned, I'm Bradley Petzer. I'm the Senior Director of Product Security at KnowBe4. KnowBe4, if you don't know, is the world's largest provider of security awareness training and simulated phishing. And my role at the company is I'm responsible for both the AppSec and Cloud Security teams. And just to go over how I got here, being with the company for six years, and I started on the Infosec team early on, I think I was the CISO’s second hire. Really worked my way up from the bottom of the Infosec team, helped build the team to where it is today. And now I'm running a team of myself on the Infosec team, making sure our product that we provide to our customers is secure.

Harshil: That's an amazing journey, Bradley. And you said six years at KnowBe4, right?

Bradley: Yeah, just going on six years.I’ll be at six years in a couple of months.

Harshil: Fantastic. And were you hired in Application Security or Cloud Security or one of those things?

Bradley: No, actually, and this is a crazy story, I actually was looking at using KnowBe4 at my previous company, and I found a position in support and actually came over to KnowBe4 in support. And I worked in support for a couple of months before trying to hop onto the Infosec team based on my previous experience, I was a great fit and I was able to work my way up.

Harshil: Oh, that's amazing. So that's interesting. So you started at KnowBe4 in tech support and you went into security. Is that because you were proactively looking to get in there or somebody identified you as a talent? 02:58

Bradley: Yeah, no, I was proactively looking. I was always interested in getting into cyber security. I feel like a lot of people have a passion to get into cybersecurity. So I was proactive. I went to the CISOand I was like, “Hey, if any positions open up, I don't have direct experience inside security, but I've done a lot of IT experience. I feel like I would be a good fit”. He actually looked for someone with more cybersecurity experience initially, and wasn't able to find anyone so he was like, “You know what, I'll hire you and see how it goes”, and it worked out well for him and myself. 03:37

Harshil: That's such an incredible story. We don't see these things very often, especially because you also rose up the chain fairly quickly, so I'm sure it was the right decision six years ago.

Bradley: Yeah, I mean, the company KnowBe4 has been in hypergrowth since about six years ago, seven years ago. The team, the Infosec team, has had to be in hyper growth too. So you had to grow with it.

Harshil: So tell me about when you originally started in cybersecurity in your first role after tech support. I'm guessing there would be a lot of new things for you to learn in that domain. I mean, security is pretty deep and also wide in so many different ways. Where did you start? How did you learn the things that you had to do in your first security job?

Bradley: So, yeah, I actually was working on a cybersecurity certification from Cisco initially. I also had Security+, so I started working on getting some certifications in the industry to build my understanding of what's needed in cybersecurity. A lot of it is researching, finding out, learning by doing. I feel like that's the best way to actually learn cybersecurity is by doing.

Harshil: Fantastic. And it sounds like it was also the right timing and the right place for you to be in, so the company supported you to learn those things on the job, which is also not very frequent. A lot of times, people, unfortunately, they just end up looking for senior people with experience, and they keep looking and looking, and it's hard to find those people, but it's great. If there was somebody who is now in a position similar to what you were at several years ago, just looking to start in cybersecurity, what suggestion would you give them?

Bradley: Be proactive. Go out there and get the cybersecurity certifications. You want to make sure you stand out, right? You want to go ahead and participate in bug bounties in cybersecurity programs. A lot of people are doing these, so you want to make sure you stand out from the rest. Don't be afraid to take a position where you start at the bottom and you can work your way up.

Harshil: That's interesting. I've always heard different opinions about certifications itself. A lot of times, if you're just entering into the field, then it gives you exposure to some credibility, but I've also seen many practitioners who also say certifications don't actually matter. I'm not saying either way, but I'm curious to see why you recommend certifications as a way to get started.

Bradley: Yeah, I mean, I believe the certifications type of security certifications definitely help to a point, right? They help you with the foundational knowledge on the topics that you'll be working with. Some certs even have good practical parts to them where you can actually learn, but that helps you get started at a certain point. If you don't actually do and practice and start creating your own tools or be in AppSec. Go ahead and create your own web application. If you know how to do what the engineering teams are doing, you will go way further than just having that knowledge without having that actual experience.

Harshil: Right, right. That makes a lot of sense. I think, just showing that you are willing to roll up your sleeves, learn before even actually getting the job, that shows a lot of proactive mindset and initiative, which are also important skills to have in cyber because we end up having to learn new things all the time anyways right? So we have to protect ourselves and the companies against new vectors, new attack patterns, new things, new technologies.

Bradley: You have to constantly be staying current with all of the threats landscape out there. With Application Security, there's vulnerabilities coming out every day, and with cloud security, there's all these new challenges with running services in the cloud, and AWS is always coming out with new services so staying current with that, it's very important.

Harshil: Love it, love it. So another thing that piqued my curiosity, and I've had this conversation with a few other guests as well. In terms of your team and your title, which is focused on Product Security. Now you spent some time as an AppSec engineer, Application Security engineer as well. How do you see those two things as different, if you do see them differently?

Bradley: They're very closely related, right? On product security, we are focused on the security of our products that we provide to our customers, our products, our applications, so we need to make sure we have the AppSec side of things covered, but we also have to make sure where those applications are running. They're running in the cloud. We're running those in AWS, so if we're not protecting our cloud, those applications are beginning to be just as vulnerable, so they tie very closely together. And so that's why I run both the application and cloud security side of the things.

Harshil: Got it. So in your team, do you have engineers who are one group with core AppSec skills and another group with Cloud Security skills, or is that blending in together?

Bradley: Yeah, we do have separate skill sets. I have an AppSec Engineer who is focused on making sure the code, the dependencies, all of those pieces are secure. And then we have someone dedicated who knows AWS really well, who's familiar with cloud security posture management tools, who's familiar with IAC and Terraform. There's two separate skill sets there that I feel like that would be difficult for a single person to know really well of both, so I expect both sides of the team to know a bit about each other, but to specialize in both AppSec and CloudSec we have separate members for that.

Harshil: Got it. That makes sense. Now if you think about product security and your mission or your charter being to secure the entire end to end product that you're selling to your customers, what are the key challenges that you see that product security teams face today?

Bradley: So one of the big challenges I feel like we're facing today is when it comes to the risk with compliance versus actual security risk, right? Everyone is familiar with the Venn diagrams of like oh yes, security and compliance, they just overlap where we have these audits that are saying that, such as SOC2, FedRAMP, ISO that are saying, “Hey,, if we're doing these, we are secure”. Is that really the case? Does that actually make us secure by getting these audits? And I feel like our mindset is not on track with that. Where our focus is on too much of the compliance side, we're taking away from actually doing what matters. Compliance tells us to do annual pen tests where if we're doing only annual pen tests, we're not going to have full coverage of apps. I know at KnowBe4 we're deploying code changes every week, multiple code changes every week. If we're doing annual pen tests, we're never going to get full coverage of that so trusting that compliance and audit are going to go ahead and actually say that an organization is secure is a fallacy. 12:19

Harshil: Right. It's interesting that there's a lot of different compliance frameworks that are putting more and more focus on the application security or the product security realms, right? Earlier it was very superficial in a way, but now I think there is more and more scrutiny starting with the government regulations, right? We've seen FedRAMP do a lot more in application security or even infrastructure security. We've seen some of the recent White House administration guidance around SBOMs and building more supply chain security. We've seen NIST build out SSDF and putting more focus on how software and applications are being built and deployed. I think that over the next few years, in my opinion at least, there will be more and more compliance inroads into the domains of AppSec and ProdSec, which wasn't the case up until a few years ago. So you're right, I think as practitioners will have to figure out, how do you maintain that balance, right?

Bradley: Yeah, for sure. We want to make sure we're prioritizing what we're doing in the AppSec realm. If we're focusing on outdated audit requirements, then we're not putting our focus where it needs to be. So I'm hoping that mentality shifts, it seems like we are trying to get on the right pathway with those stands coming from the government, but we'll just have to see where that takes us.

Harshil: Right, right. And I've seen several cases where it just goes to the level of craziness that's unmanageable. So, for example, I remember talking to someone and they were really struggling with this FedRAMP requirement that said you have to scan your code using a static analysis tool, and you have to respect the severity that the tool gives you. You cannot change it. Well, if you follow that to the definition and you use pretty much any commercial static analysis tool, you're going to get stuck with hundreds of thousands of findings that you'll have to dedicate so many resources just to triage and prioritize. If you can't even change the severity of the findings, that's a nightmare situation for a lot of not just security teams, but also for the engineering teams who have to deal with all that data. And there's got to be a reasonable position somewhere in the middle. Now, I don't know if that's a real requirement. It was just as a part of the conversation with somebody, but I'm kind of curious to see what you think is happening in the world of FedRAMP requirements and the intersection of AppSec, ProdSec.

Bradley: Yeah, I mean, there are these requirements, right? And we all know how reliable SAS tools are. There's a lot of false positives that come in from those. So if we aren't able to prioritize those results correctly, we're going to give our engineering teams hundreds and hundreds of security fixes to do. And if we keep doing that, they're not going to trust us with security. They're going to be like, “Oh, these aren't actually important”. Everything's a high priority because of compliance, because compliance told us it is. Also the same with CVEs, with all of the dependencies out there's all of these that are coming in high credit level. Last year, I think there were 25,000 plus CVEs published, and we're probably using thousands. If you're a large organization, you're probably using thousands of dependencies in your applications. If everything comes in high end and critical from these, our developers aren't going to be able to trust that security knows what they're talking about if everything's a high priority.

Bradley: To go ahead and actually prioritize those, make sure we're providing meaningful security risk to our engineering teams, that's pretty important so that we can form good relationships with the engineering teams.

Harshil: I'm actually kind of curious, how have you structured your team to be able to do this type of work? The reason I say that is that for most AppSec or Cloud Security engineers that I know, nobody gets excited about working on FedRAMP compliance or any other compliance requirements. right? But the people who are experts in compliance, they're also usually not very technical. Like, they don't understand. I mean, they're not running these security tools. They're not doing architecture reviews or code reviews or threat modeling. And the engineers definitely don't want to deal with all of that stuff, right? So how do you make your company successful when there's an intersection of engineering skills and technical security skills and compliance skills needed?

Bradley: Yeah, I think it's all about the relationships, right? So we have a pretty good compliance team that we work closely with, that we try and make sure that what we're doing is reasonable. Sometimes we just have to bite the bullet when it comes to compliance, but we try and work with them to help make sure we meet in the middle, make sure we're doing what's required of us, but also doing what's reasonable when it comes to security. I think what's fairly important for myself is to make sure we prioritize security risk. We take a look at not only what the impact of that risk is, we take a look at what the likelihood is in the context of our applications of our organization. And we go ahead and submit those to dev based on that to get the true risk. And luckily our company finds tech debt very important, right? So we're not shy of handling our tech debt, even if that comes to security tech debt. So we're pretty good at handling those dependencies and container vulnerabilities, all those vulnerabilities that make their way up. But we set the expectations correctly. And when we do get a high priority, say, from our bug bounty program, it's like, “Hey guys, this is actually the highest priority finding”. Our engineering team knows when I put that in, they don't question it. They drop what they're doing and they get those. Even though we have a two week SLA for critical vulnerability, they'll get it done that day. It's very good to form those relationships so that expectations are set across the board.

Harshil: Right, right. So giving high fidelity signals to your engineering team builds that trust, is one of the components that builds that trust, right? And what I hear you say is that your team is the one that does that upfront work to make sure that you're giving good customer experience to your engineering teams. Now, in terms of compliance, specifically, have you dedicated people in your team who take most of the compliance related things? and I'm talking specifically about FedRAMP or SOC2 or ISO or things like that, or is it distributed across every individual in your team?

Bradley: It's distributed across, right? So we have multiple scanning sources. You have your container scans, you have cloud scans, you have open source scans, fast scans, all of that. So there's quite a bit of a workload. What I've done and I think it's very important to do for an AppSec team is generally, I feel like AppSec teams are pretty small and they have a lot of responsibility in an organization. You need to automate these processes. If you're not able to automate these processes, you are just going to find your team just busy, busy, constantly having to deal with findings all the time. What we KnowBe4, and all the SEC teams, have is we have an automation team who are responsible for automating processes for us. So even though my team can write Python scripts and do that themselves, we go, “Hey, we have this challenge, we have this process that needs to be automated. Can we give it to our automations team?” And they automate that for us, and that helps us tremendously. It saves our AppSec engineers time, saves our engineering team’s time, saves our compliance team’s time.

Harshil: That's amazing. Now you talked about automation that saves time across multiple different teams. If you can share a few examples of automation that have been most effective for your team.

Bradley: Yeah, sure. We use Snyk for our open source container scanning, et cetera. And what we do with that is we have an automation that goes ahead and ties into Snyk, pulls the findings every month and goes ahead and creates the Jira tickets for the dev teams, creates them an epic for each project. And that notifies the dev team like “Hey,, we've got this current list of monthly scan findings. Let's go ahead and automate those”. We're also working on automation now with Snyk has these priority findings. So sometimes a CVE has a higher priority even though the criticality is lower. So we have this automation now that higher priority findings go ahead and notify us straight away so that we can work on those quicker than just relying on a monthly scan to be like, “Hey, we need to go ahead and patch these findings”.

Harshil: That's awesome. Yeah, I think one of the key things is Snyk will tell you the high priority thing, but then you create a ticket, how do you automate the rest of the part, which is who owns this issue? How do you assign it to the right individual, the right team, and then track it to closure? I'm kind of curious if you have built any automation around that piece as well.

Bradley: So that is all handled by our engineering team. For the most part we've got each project and Snyk is mapped out to a project in Jira so when it's created on that side, our project management team knows to go ahead and assign it to you. Our sustaining, we have a sustaining team that goes ahead and handles the majority of those security findings. And then we use Jira for tracking our progress. So we have fields that we, we actually have custom fields that we've built out in Jira for severity, for what the weakness is, where the source is. And what we're working on is getting this to go ahead and map out to the same fields that are required in the FedRAMP POA&M report. I'm not sure if you're familiar, but every month you have to build out a POA&M report, it's an Excel spreadsheet and present that to the federal agencies. We have this whole automation process so that we can provide these reports on a constant basis. And it also helps us provide these findings to our customers. Our customers want to know, what is our current status, what findings we have. We have this all built out in Jira, automations to go ahead and export these reports.

Harshil: Right. That's awesome. I think the related topic that we were talking about earlier is how do you actually deal with this massive volume of open source vulnerabilities, right? You mentioned Snyk. Snyk is phenomenal, but at the same time, if you look at the entirety of all of the things that even Snyk might be giving you, I don't know about you guys, but in most people I talk to it's a lot. It is. Do you have any opinions or thoughts on how you guys want to or are doing prioritization of that?

Bradley: Yeah, it's definitely a challenge. I think that all organizations have to deal with it, and I don't think it's a problem that's been completely solved anywhere currently. We rely on devs to go ahead, and our engineering team to go ahead and actually be self proactive. When they find something, we want them to go ahead and update those dependencies. We need to go ahead and provide them the resources, we need to go ahead and help them self serve security. I know that's the big thing these days with the whole Netflix paved road approach. So providing resources like that, not just relying on the AppSec team to present issues, but having dev teams be proactive in it is also very important. Another issue though with open source in general is they have repo compromises. So it's like we're trusting this third party code into our applications where our applications have all these controls in place such as MFA, merger approvals, code scanning, and then we go ahead and allow this third party code in. So I think that's another major challenge that AppSec teams face and the industry as a whole faces, is with open source packages, we need to find some good solutions to address those issues.

Harshil: Right, right. Yeah, and a lot of times the first step is people start looking at the manifest files and the dependency files in your repos, which is obviously a good start. But if you're using containerized workloads, then you want to look at containers as well because a lot of dependencies get pulled during build time or at later stages of the build deploy pipeline which you may not identify if you're just looking at the manifest files in the Git. So it's definitely a multifaceted problem. It's not that easy to solve for sure.

Bradley: Correct. Yeah, andI think we need to maybe shift the solution to this like a little bit early on. I feel like the devs that are building out these open source packages need to come together with the AppSec community and with the package registry community and work on solutions so that we can do those code scanning, do those merger approvals. Make sure these packages that we are getting and pulling it have had a decent security look at them before they even get approved to get pulled into our applications.

Harshil: Right. And that makes me think that the success of this aspect of security is heavily dependent on the maturity of the engineering processes. Like, if engineering operates as a wild west, then there's nothing apps that can do, pretty much. If there is a well defined process of pulling in new dependencies, keeping them fresh, and sustaining the operational environment, then security can have a much easier, much more effective way of doing things.

Bradley: Yeah, it's very important for security teams to be monitoring what developers are actually building out and where those packages are coming from, for sure.

Harshil: Right, right. So Bradley, as you think about the rest of 2023 or even a few years down in the future, where do you see there will be interesting new things happening in the world of Product Security, Application or Cloud? What's exciting you in this field today?

Bradley: What's exciting to me, there are some great products coming out. I don't know if you're familiar with Wiz, but it's revolutionizing the way we approach cloud security. Having context with the cloud is very important. The cloud scales enormously, how things connect, how applications connect. So that's been a really exciting tool that we recently started using. So more context. It's very important with all of our applications scaling on such a large scale, more tools like that. I hope there will be more tools for open source package registries that we can host, that we can have more context on the packages that we're putting in our applications. And I also hope that we as an industry start focusing on actual security risks, prioritizing on what this truly vulnerability truly means in my organizations, what's the likelihood it's going to affect me and stop relying so much on the CVEs and clients requirements. I think that will help us significantly go ahead and stop more breaches versus requiring FIPS. “Hey, I've got FIPS”. Okay, cool. But I left the front door wide open. It doesn't really matter so much.

Harshil: Yeah, for sure. I mean, I think a lot of this, definitely a lot of the innovation that is happening is also driven by the fact that now we have access to that data from the cloud environment, right.? You can leverage AWS or Azure GCP, whatever. You can leverage the APIs and the metadata from those platforms to build all of that rich context. If you're getting excited about Wiz’s context, you should definitely look at Tramzo. It's kind of similar model where there's a lot of interesting context that you can derive out of GitHub or Gitlab or Bitbucket as well. So your point about context being very important these days, it's definitely applicable to many other realms as well.

Bradley: Yeah. Products that are providing that context is. There's just so much going on nowadays. I think that's super important for us in the Product Security and Application Security realm.

Harshil: Amazing. Fantastic. Bradley, this has been an awesome conversation. Thank you so much for sharing your insights on this. I really hope we can have you again on this podcast and maybe talk about more compliance friendly automation and security. We'll see where it goes, but thank you so much for your time, Bradley.

Bradley: Thanks so much, Harshil. I appreciate it.