EP 13 — Daniel Harvey: How to Shift from AppSec to Product Security

Daniel: Thank you. Harshil. Yes, so I've been in application security for 13 years. Started back when I was at Clayton Homes. And I've really been in most verticals, from manufacturing, to financial, to the product side, worked at Veracode on the application security side, and have a vast array of different experiences at different sizes and focuses of companies.

Harshil: Yeah, that's fantastic. The people who have been in application security for this long, we can tell the transition that you've made, like back in the day when application security used to mean versus now in a more modern agile world that is micro services native, it's a very different challenge compared to what it used to be. But it's great in a way, I think application security is getting much more attention now as compared to what it used to earlier. One interesting thing that I noticed on your profile is Daniel, you obviously started as an application security professional, but one of your recent roles has a title of product security. Tell me a little bit more about what you mean by product security?

Daniel: So I know product security and application security can be very interchangeable in the industry today. It seems you talk to different people and they mean different things. When I started at InVision, I had the application security title, and the focus of the work we were doing there was mainly working directly with engineering teams and trying to make sure they were doing the right security processes. One of the things that I've worked through while I was there was I got us a seat at the table with our product team and became a stakeholder from there. And so just working with the holistic organization, and understanding what our role with the company was, the team was there to secure the product we were selling. And so working with our product team, allowing them to understand what the benefits we provided, and what services that we were really providing to the company is when we decided to make the change from application security to product security. What came along with that was shifting from just using tools, doing code reviews, to also providing what are features that can be added to the product to sell the product to various customers. We had several large enterprise customers that wanted different features, and we would work with our product teams to understand what type of priorities we should put on security features. Not normal product features that they were delivering for the users of the product, but from just an overall company when they're onboarding the product. What are the security teams looking for? Can they access audit logs, can they manage users right? How did they control the permissions on different areas and who accesses what? And so that's why when we made that shift, we started providing those recommendations to product to say this is what you should help prioritize as customers are starting to expect this more and more.

Harshil: That's pretty cool, actually. So you're moving beyond the traditional realm of risk management to actually making product security a business enabler, right? So potentially a competitive advantage for your business. Now as a part of doing that, you mentioned that you would work with product and help them drive prioritization of security features to be built into the product. The actual development of those features, was that under your product security team or was that a separate team?

Daniel: No, that still fell under the traditional engineering teams for who was responsible for those areas of the product and where those features would be implemented at. So we weren't doing any development work in that aspect.

Harshil: Got it. So you would be the eyes and ears, help the product people get a security lens and really understand what would be helpful to customers because you bring that persona to the table. That's awesome. So did you end up getting on calls with your customers to understand what they would need and so on and so forth?

Daniel: Yes, several times I would get on calls with customers going through the purchasing process. Customers really wanted to understand what our security was, how they can do different features. And we weren't setting any time frames on there, but we would go and we would learn what they were asking for. We would read through the questionnaires that they would send over and try to get a good understanding of what customers really wanted in addition to the various experiences that we had as a team, and how do we evaluate the vendors that we are using, and what are we looking for. Because it's hard to sell a product now if you can't sell the security of the product.

Harshil: Right, you're 100% right. I remember in my previous roles we used to do exactly similar things and sort of in our space, we were the first ones to build some of these security capabilities and our sales team loved it. They used to use these security capabilities as differentiators in competitive deals and so on and so forth. One question that we ran into when doing these things, which is building security features within the product that our company was selling was do we charge extra for some of them or we don't. I think we have some precedents from companies like Salesforce, which I believe have a product called Shield which gives additional security capabilities. And now there are a few more as well with extra security capabilities that come on top of it. What's your perspective?

Daniel: I think all default security controls should be included in the product. When you go and you say you need to upgrade just to buy a more secure product, I think that gets into a state where as a company, you're accepting that some of your users may be less secure. And so my preference is you don't have to default to everything locked down, but default to some generally accepted secure settings. And then as far as the features, there can be some differentiations between what features you provide to, say, free users versus company users. At InVision, we had both free and enterprise customers. They require different controls just because they want to manage things differently. And so with an enterprise plan, you should be able to access it more. Should you charge on top of that enterprise plan for additional security measures? I think unless you're providing additional value to the product that you're selling, from a feature standpoint, I don't believe you should charge extra for it. It's more just when you're working with security teams, you can sell them and help get their sign off by having the additional security features. And there are several customers that may choose a company that has all those security features and not have to pay for it versus having to pay extra for it.

Harshil: Right. Yeah, I wish that the broader SaaS industry would come to a consensus that security should come free with the staff platform, right? Now switching gears a little bit, I think we dove too quickly into the deep end of the conversation here. I think it would be super useful for our audience to understand what InVision is, if you're open to explaining that just out of a high level.

Daniel: Yeah. So I know InVision has gone through some transformations, and me not being at InVision anymore. When I started, InVision was a platform for designers, for the design process. So to help designers, engineers, and any stakeholders in the design process to collaborate at different stages. Since then, it shifted to be more of a workspace. So it wasn't just designers, it was trying to focus on the whole industry across the board, whether you're engineers or financial. And they had a whiteboard, and the whiteboard could be used for anything. Imagine drawing on a whiteboard and you can pretty much do that on InVision. And one of the big things is integrating with several different products, whether it's Google Docs, Microsoft Docs, it was trying to bring those into one place to help with just visual collaboration with all different aspects of a remote environment.

Harshil: That's pretty awesome. So is it safe to assume it was delivered as a SaaS service?

Daniel: Yes, it was all a SaaS product.

Harshil: SaaS product, fantastic. Okay. And I'm guessing you would be selling, since you mentioned, to free users, but also enterprise customers as well. So you would have requirements from your customers around security and compliance and things that you would have to comply with.

Daniel: Yeah. So it was a SaaS multi-tenant product. So with that, you're housing customer data in similar environments, and so you had to sell the security, making sure customers couldn't access other users' data. And one of the interesting pieces when it came to how we evaluated vulnerabilities, if you take the traditional, say, CVSS scoring where you check, do you have to be authenticated to access the vulnerability? When you have a SaaS platform that anyone can go sign up for free, everyone can be a free authenticated user. And so we had to work through some of those, and how do we write vulnerabilities differently than how other customers are, that you have to have a paid subscription to have it as an option.

Harshil: That is definitely an interesting way to think about it, which makes a lot of sense because if you have a free tier, then pretty much any interested party can sign up and start using it as compared to if you require an enterprise purchase, then you have some level of trust. Not full, but there's some level of credibility built into the users coming in. That's fascinating. So going back to what we were discussing earlier, when you started your AppSec career 13-14 years ago, the landscape was a little bit different as compared to now. Are there any key things that you see in terms of what has changed? What are the new challenges now as compared to what used to be a decade ago?

Daniel: I'd say a decade ago, some of the biggest things were you were getting a lot of tools coming out at the time and it seemed everyone was really pushing tools. People are still pushing tools and some of it is the same tools whether it’s DAST or SAST. Some of the newer ones are IAST and RASP. But the challenges are with more companies going to more of that SaaS model than what it was, you're getting more data that lives in similar environments and accessible from anywhere. Business logic flaws, I think, are becoming more prevalent because you have the tools that will access and find the majority of, say, your SQL injection and your cross site scripting. And so it doesn't matter how much time and effort you throw at the tools. The business logic flaws and authentication are going to be some of the hardest to find and where most of the resources have to go to now instead of some of those lower hanging fruits that are easier to fit.

Harshil: Does that potentially have something to do with the increased usage of third party dependencies as well? So I read statistics somewhere that 80% of the code is just third party dependencies. So now developers are writing less in-house code, I guess, but focusing more on just the business logic.

Daniel: Yeah, I think with the business logic, depending on what type of application you have, I know with some of the microservice architecture, it's really separating out a lot of that, and different teams are writing their own business logic rules within those. And even though, say, someone, a user in your application should have the same level of access or permissions across different microservices, the business logic may be written differently by each engineering team that's doing it for their area and the product that they're working on. And so you're getting more cases where you're having to make those business logic decisions at more locations than if you had more of a monolith that can have more, say, one place that's making those decisions and not propagating them.

Harshil: Right. Yeah, that makes sense. That brings up another problem also where there are multiple smaller squads or dev teams that are building all these microservices that may or may not be interdependent with each other. But it's definitely a different mindset when you think about security of a monolith versus security of different microservices and multiple smaller dev teams pushing releases very quickly. So the traditional stuff of running scanners and all that stuff, that is fine. They can continue to run scans and CI pipelines, or during commit time or what have you, but after that, how do you think about security beyond scans? Because I'm sure there's a lot that is being missed if you just run scanners and just stop at that point, right?

Daniel: Yes. So with that, I think that brings up some of the biggest challenges I see in the space right now is you can have engineers that understand how to deploy tools and you can get those orchestrated and that automates and scales really well. But with the items such as what I mentioned for the logic flaws and the authentication checks or authorization checks, how do you validate those? And a lot of automated tools can't find those. So there's a couple of different methods that we did. We used CodeQL to help where we can write some rules and identify consistent patterns that we were looking for to help scale our manual reviews. But a lot of our effort was spent on those manual reviews and that becomes hard because manual efforts can only scale so far without costing a lot. And so how do you balance the resource time to scale that? And you have to really determine what is the risk of the company and how do you make those reviews a priority and not slow down the release process of engineering. Talking about the difference between 13 years ago to now, more teams are on the Agile Methodology or some form of Agile, whether it's CI/CD or your traditional agile, where historically things were more waterfall. It's hard to fit security checks and those manual reviews within an engineering cycle because you don't have the time without slowing down the releases. And so you have to have a good collaboration with your engineering teams. Not necessarily train them on how to detect security issues, but make sure they're aware of what you're looking for and come to an area where if they have questions, they raise it to you so you can help them at that point. And you can always be the eyes and ears on everything.

Harshil: You know, that brings up another question, which is you mentioned you have to have better collaboration with an engineering team, which 100% agree with, but does that create a dependency on how mature the engineering processes are? And does that have an implication on how much your security can be?

Daniel: I actually think the more collaboration you have with an engineering organization, the less mature the engineering organization can be. One of the things we did InVision is we didn't do your traditional Security Champions program. We did what we called a squad or a team alignment model. And we had our security engineers embedded in multiple engineering teams. And so even though the engineering teams had all separate processes a lot of times and didn't follow the exact same cadence, we were in their stand ups, we were in their meetings, working in their Jira boards, and so we saw everything that was going on and we became the normal face in there. And so when they actually had those questions, whether it was in a formal process or outside of a process, they were comfortable asking the person that was embedded in their teams to say, “Hey, can you look at this?”, or “How should I do this right?”. And so a more mature organization where they have more formal processes, you can build those and embed processes in there to help with that, but the less mature ones, that becomes harder. And building those relationships, I think is more consistent to have a better secure product.

Harshil: Yeah, that's a great point. In terms of that you can actually collaborate closely. So if you don't have a consistent process, that collaboration backfills some of that lack of consistent process. One of the logistical challenges that we ran into in earlier life was, so we decided on a similar model where our security team, product security team, would attend stand ups from the dev teams, but what we very quickly realized was every team had their stand ups exactly the same time, 09:00 Monday morning. So you can't really attend multiple stand ups, at least in our previous company.

Daniel: Yes, there was definitely that challenge. And at some point if you attend all their meetings, you don't have time to do anything else.

Harshil: Right, exactly.

Daniel: And so working with the team is more balancing and making sure there's some teams that require a little bit more interface time with them, there's some that have some guys that collaborate really well from the start. And so you have to balance your team. The more time you get to spend with them and you learn what teams you're working with, you can set how much or what meetings you need to go to. And I think the pitfall that you can get to with that is you can't attend the same one all the time. You have to spread it out among all the teams that you're responsible for. But yes, that's very much of an issue, especially if you're all located in similar time zones.

Harshil: Right. Did you all end up doing anything interesting for building the relationship between security and dev teams beyond just attending meetings? Like, how do you build a personal one to one relationship?

Daniel: So InVision’s culture was really good with just the one to one relationships in general. It was a fully remote environment. And so from leadership on down, it was built into the meetings to help get those connections from a people standpoint, not just work related. And so just being in normal meetings, you had those chances to interface with them about non work related aspects. As a manager, my weekly meeting with my team, the first part of the meeting every week was not about work. It's about what's going on in life. And talking to some people, they question, “Why are you wasting a meeting for that?”. And if you look at it in office culture, you're going to waste that time near the water cooler, going to or back from lunch. And so in a remote culture, we built that in a way that it allowed us to build those relationships. It doesn't matter if you're in a meeting just with your team, or if you're meeting with people outside of your team. Really, you’ve got to see more than just the work product of that person. You got to see who they were. And I helped build those relationships in general. And so being on those stand ups, a lot of that discussion happened there and allowed the team to really be thought of as a team member of that engineering team, not just an outside security team.

Harshil: That's pretty awesome. Yeah, because the risk of being a security representative in a dev team is that the security person always comes in with more work to do, more backlog items. And it's nice but up to a certain point when it stops being nice, right? So it's definitely a balance of how you navigate those conversations. How much do you push the dev team? Where do you actually stop?

Daniel: And that's something we weren't always just telling them they had more work to do. And one of the things when we were to identify things, we not only tell the engineering teams, we'd also work with product counterparts that was really driving the time frames to say, “Hey, we need to improve this and can we get the engineering team some time and to be able to prioritize it?”. And so we helped that side as well. There were also times where we had some pen tests and we had some outstanding vulnerabilities that were found. The team would go in and write a PR and submit it to the repo to actually fix the vulnerability. It's not something we did often, but when we realized that the team was really being overworked right now with the current demands, we did step up and show that we were really a partner with them at that time. And just doing little things like that, it really built the relationship to allow them to see us as more of their partner, not what I say a lot of security teams are thought of as more of an auditor.

Harshil: Right. Yeah, I think that's a good point. I mean, getting out of that mindset of being an auditor and always pointing fingers at stuff that's wrong and rolling up your sleeves, helping out, even if it's not writing code, but at least figuring out what's the best way to solve problems, helping find the right solution, going into Stack Overflow and trying to send the right links to the developers. All those things help, right? So that's an interesting model of working, and did that scale really well as the dev team grew, as the company grew?

Daniel: Yeah, we felt that scaled really well. When we were fully staffed we had some issues when we had some attrition. And how do you support all the teams when you only have a couple of employees? It becomes really hard at that point, and you have to scale that back to some extent. But it allowed us, being that we build those relationships and there was an expectation on when we should be involved, the engineers knew when to really reach out to us a lot of times, they just didn't know who to reach out to. And so that became the issue and they were asking, who should we reach out to more about the things and so we can get involved. So it did allow us to scale even when we pulled back, because we felt comfortable that the engineers were going to bring the higher risk items to our attention.

Harshil: Right, that makes sense. Just switching tracks a little bit, one of the things that I remember us talking about earlier was a more modern version of code ownership. I think you had some interesting ideas about how to do all of that stuff and how you've done it. And this is not a very common thing that security teams typically solve for, but it's so fundamental to our jobs, to our day to day. Do you have any suggestions or thoughts on in a modern engineering world where there are multiple dev teams, multiple squads and units working with each other? As a security person, what kind of co ownership, asset ownership model is needed for you for a security person to be very effective on a day to day basis?

Daniel: Yes. I think I'm very passionate about the code ownership stuff. And I know we've talked about that some in the past. But I feel when you're evaluating what code you have in your environment and looking at repositories, it doesn't matter how many repositories that you have, knowing who you need to go to work with if you have an issue in one is generally the first battle that you have. In so many places it's hard to say, “Okay, I have this repo, we have this issue in it, who do we talk to?”. A lot of times, if it's legacy code, that team may have been morphed into another team, it may have went away, you don't know who to respond to. And some of that is done because there's not good ways to say to really track who owns what over time and make sure as organizations change that those are getting updated properly. And so we were looking at ways on how we can keep those updated, how can we tie those into what's actually being changed in the company. And so your people team or your HR team always has what the structure of the company is, and whether that's fed into your identity management system or not. If you can tie code ownership to your identity management systems that have those updates and get alerted when things are not matching up, when there's a repo that doesn't have an identified owner earlier in the process, not five years down the road when you realize you have an issue, you can make those decisions a lot quicker after those changes happen, rather than having to wait and then everyone's gone that worked on it previously.

Harshil: Yeah. So just so I understand clearly, what you're saying is identity management systems typically tend to have groups and group members and things like that to map the repo ownership or code path ownership to those groups ideally, if possible.

Daniel: Yes. I think that would be the ideal that I've always looked at is you have people that are updating those and keeping those up to date. So how can you leverage those rather than having to build a whole different model for it?

Harshil: Yeah. How have you seen those mappings being done typically?

Daniel: Yeah, so there's different ways. A lot of them will be mapped directly from, say, your people management systems into your identity management system. And from there, if you're accessing repositories such as in GitHub, there's ways you can map your identity provider in the groups into GitHub. And from that point, it can be an access model that you have, or you can have some files in your repo that identifies who owns what. That is checked and verified every time a PR is submitted, or you have some regularly scheduled job that would go and validate that as well. It really depends on what your tech stack is and what abilities you have. I worked most recently in GitHub, and they've come a long way in what you can do from there. So it's more open to where things are going and you're going to have more of the ability to do that in the future, I really believe.

Harshil: Yeah, that's awesome. I was talking to another product security person and they mentioned this open source project from Spotify, it's called Backstage, which does a lot of this code ownership management. It's a much bigger platform than just code ownership, but I've seen a lot of people use that for managing code ownership, which is pretty awesome, open source software. So let's talk a little bit about what do you see as some of the bigger challenges that we all need to think about. Like, if you're in a company that has developers using modern stacks and agile development methodologies, what are some of the things that you have to keep in mind as sort of newer challenges that you'll have to address sometime soon?

Daniel: Yeah, I think more and more so just the speed of how quickly things are being deployed. How does security fit in that process? Security has not historically been quick in the process. Whether it's tools that take a lot longer than it takes to build the product application, to scan if you're doing scanners, or just a manual review. Manual review is only as good as the amount of time and effort you put into it and the level of experience a resource has. So you have companies that are wanting to do their continuous delivery models and release multiple times a day. How do you scale that? How can you be able to give something a proper rating to say, this release is a high risk release, this release is a low risk release. Because I don't think you're going to be able to get eyes on every line of code. That's just not scalable. And so the number one challenge going forward is where do you classify what a release is and what is the security risk of that? To be able to provide the stakeholders that are releasing it the data to make that decision to release or not.

Harshil: Right. And the same goes for whether it's a software release or if it's an infrastructure release as well, right? It could be deploying Kubernetes environments or cloud environments, and the same could hold true for that as well.

Daniel: Exactly. It covers all areas of releases and whether it's something on the back end systems because most things are tied to something and it could cause a downstream security impact. So you really have to weigh in on how that fits and what the overall impact is.

Harshil: Fantastic. So that is a very very interesting coverage of different topics we talked about. I think we're right on time, and this has been a phenomenal conversation, Daniel. I really appreciate you coming on to the show and sharing your experience and thoughts and insights here. Thank you so much for being here.

Daniel: Yeah, thanks for having me.