EP 14 — Mark Stanislav: How FullStory Measures & Improves Product Security

Harshil: Welcome everyone to another episode of the Future of Application Security. Today we have Mark Stanislav with us. Mark is the VP of Security and Compliance at a phenomenal startup called FullStory. Well, I guess not a startup anymore, you guys have grown quite a bit. Mark, welcome to the show.

Mark: Thank you for having me. Yeah, I'm really excited to talk about AppSec and ProdSec today with you and appreciate you having me on.

Harshil: I'd love to have that conversation. I love your background, you spent a lot of time in this domain especially. I was going through a number of your talks and presentations that are on YouTube and that are available on the internet, pretty fascinating types of things that you've talked about and we'll cover that in the rest of the episode, but before we go too far down the path, I would love to have you just talk about your career journey so far, where did you start, and how did you end up in this position. Because there are people listening to this podcast who are very early in their career journey so I'm sure they'd love to hear your story.

Mark: Yeah, the kind of origin stories are always interesting, in security especially. So I came through into I guess the broader world of security and really product security first as a units administrator. So I did that for many years doing systems administration, really got excited about securing of like FreeBSD and Solaris and all the analytics. And as I learned more about system security, inevitably you start doing more software engineering and as you learn more software engineering you start learning about vulnerabilities and on and on, right? So I kind of had that early stage, just kind of kept finding new areas that I wanted to dive into more and more. Eventually I did my own startup back in 2009, a kind of cryptography appliance when appliances were still cool, and a lot more software engineering that moved into full time security consulting, pen testing, security program development. Was at Rapid7 doing services there, worked at Duo Security very early on in their life cycle and came back as a kind of a boomerang employee to build out their product security program, and just kind of organically over time you just find something that's exciting or interesting, say yes to some opportunities at your employers, and 20 years later we get to have this conversation. So it's been a very long but such a fascinating just like fun career, and I really take a lot of pride and enjoyment of how I've gotten to where we are today.

Harshil: That's amazing. And there's some really good, well known security companies in your list of companies you’ve worked at. I'm kind of curious if that has changed your perspective or influenced your thinking in any way, working in a company that builds and sells security products like Duo Security or Rapid7.

Mark: It really does. And if you're a security vendor, you obviously are trying to sell them the companies and sell your solution to hopefully some problems that are well defined, that need a solution. And when you're a vendor buying security solutions like, you're on the other side. So the fact that I've worked in companies like Cisco and Duo Security, and Rapid7, where we've been providers of security technologies and services, but then building security programs in places like Philips Health Tech or Gemini University that I've worked in, you really get the kind of two sides of the coin, and you get the empathy side of the sales cycle and the buyer and the toil there, you also understand a lot more about defining value propositions. And so I think as a security leader, I've learned a lot more about how to sell investments in security because I've been on the seller's side. Like, you really have to think about, again, persona based selling and opportunities to make solutions maybe not just make sense on paper, but really, how do you derive the value of that solution in practice? So both of those together, I think, have created a lot of perspectives that I'm able to take with me.

Harshil: I love it that you mentioned you've been on the selling side. Now, a lot of building a successful security program is also internal selling. Do you think that working at a product security company or security product company rather, has it helped you sell security better internally?

Mark: I think so, because when you're trying to, whether it's a services deal or a product, get to that point where you start listening. And having done security consulting, going into organizations, doing program analysis, gap analysis, kind of maturity assessments, as a security professional, as a technical professional, we're very good at coming up with solutions. Like, we really like to solve problems, like engineers like to solve problems. And I think when you have to become a seller and or providing services to other people, you really start to understand the context and be able to frame the value in context and not just like, work around the context, obviate the context, you really start to make a relationship where people understand the value. And so I think as a seller, you need to dive deep into the past problems, the concerns, the uncertainty that people have. And historically in security, which I think we're kind of on the other side of this industry mostly, security sales is a lot of fear, uncertainty and doubt. It was to scare people, make them like, “Hey, a breach is going to happen to you”. And sure, maybe it will, maybe it won't, rarely does one single technology start or stop a breach from happening, right? So I think giving people perspective as a seller about why it's going to be additive or decrease risk, but putting that into a healthy framing where you're not doing the silver bullet panacea thing, you're just giving them like, some context on why their actual company will be benefited. And so doing that in terms of a security program at a company that I'm building a program at, similarly to whether our executive team, the board, trying to give them context of why that next six figure number purchase will make sense. Is not just because it's secure or because it adds security but what part of security, how much security, what kind of return will we get on that investment as the idea of return on security investment has been a hard thing in our industry for a long time. So I think, yeah, in concert it's really helped give me an edge there internally as a seller of security programs to a company.

Harshil: Yeah, I think that's where a lot of things become a little bit nebulous, right? As you mentioned, return on security investments is really hard, and even if somebody was to come up with a number, somebody can poke holes in it very easily. Maybe if you can share an example of what you might have done in the current role or a previous role that you would call selling security, or justifying budget, or making people do things for security; if you can share a concrete example of that.

Mark: Yeah, one that I really like, and in the spirit of product security and really thinking product security is not just AppSec or just CloudSec, it really is about the product development lifecycle, partnering with product managers, understanding the product fit and market fit and your TAM, those are things that make a product security team successful. And so one thing that I had done at Duo Security, we actually had kind of a product security enhancement roadmap where this wasn't necessarily vulnerabilities that was like a defect we were remediating, it was actually kind of forward looking, something about the product that we know could actually reduce friction in the sales cycle, that could actually help build confidence in the market or trust with our customers. And we had a few examples of these. One that comes to mind in terms of internal selling, we have historically had a finding on our annual pen test report. There were some nitty gritty details about basically like serialized objects for the number of people on the podcast listening, you know, where the dragons are beyond those, but basically we had worked around the problem enough and it never felt complete, and customers still kind of turn their head a little sideways and said, “Yeah, that's good, but not great”. And I was actually able to get our product team and our engineering managers to kind of commit a couple of quarters to go through the entire code base and eradicate this entire very pervasive, like, hundreds of instances of this thing. And they made the investment because the business relationship there was that sales can sell faster, customers will have more confidence, it's the right thing to do, and we know it's the right thing to do. And being able to get that buy and actually allocate cycles to technical debt or to things that are not crisis level security, that's a culture of an organization that people would buy into that with me and I wouldn't have to threaten security compliance 101. They understood why it was good for our customer and you know, from the buyer's sense. And that's like a great example where if you put that on a roadmap you talk to product people as a product roadmap and you bring it to them in a way that's still time-boxed and still has a value proposition. Reasonable people are going to do their job well, and in this case Duo Security had amazing product managers I worked with and we shipped a lot of those kinds of forward looking or just like better than expected levels of security to our customers which is amazing.

Harshil: That's phenomenal. That's a fantastic story. Especially security adding value to the business, that's a great story. Now coming back to the current times which is now you're leading security and compliance at FullStory - and by the way, we are a very happy customer of FullStory, love the product, it's phenomenal. Can you help our audience understand just a little bit about when you join FullStory and what was your motivation in joining here?

Mark: Yeah, funny enough, my previous employer, we were actually looking at FullStory to purchase ourselves for the same value proposition. Like it’s an amazing product, there's a lot of value to understanding your customer whether on their website or mobile app or otherwise. So I learned about FullStory through being a buyer at my last employer and understanding that FullStory’s business is one really of trust. As the platform operates, basically we are able to understand browser interactions with a lot of rich metadata, a lot of qualitative quantitative analytics. And in order to do that we have to have a position in the end user's browser. And as we all know, the browser is the world. Like every email, every PDF, it's in your browser. And so I really appreciate the mission of FullStory to provide that service securely and to create trust with our customers. And for me, 20 years into my career, I'm always looking for an organization that is as committed on paper as they are in reality. And a lot of companies say “Security is very important to us, we take it seriously”, that sort of byline from the breach press release, but the lead for that it's really a matter of I think with FullStory, I could talk to all the engineering managers, people that have been here for as long as the company existed, and it was palpable. They really, truly wanted to do the right thing for their customers and build a great security program. So it's been about going on five months now that I've been at FullStory, as most people that come in to take, in our case, about an eight year old company, and kind of look at what's there, what needs to be there, where we are on the journey ourselves, what are the customers needs. Yeah, there's lots to figure out, but so far it's been more than so good, we've really made a lot of progress already in just a few months, and I think that goes back to the culture of engineering here. Engineers here love security, they understand security, and partnering with them is a very easy proposition.

Harshil: That's great. I'm sure you're getting a lot of success getting traction from engineering just because of that cultural aspect. It's not very commonplace in most companies. But tell me about, like, when you joined security, I think just a few months ago, when you joined there, did you have to do any sort of lay of the land assessments? And the reason I asked this is because I was going through some of the content that you published around maturity models, application security, products security maturity models. Did you use any of that when you're starting to build or scale this program at FullStory?

Mark: Yeah, maturity models, and there's multiple out there, BSIMM which is from Synopsys, originally from Cigital before the acquisition there, and then OWASP actually has SAMM, which is the software assurance maturity model. And there's pros and cons to both these different maturity models but yeah, within the first few weeks, I started kind of pulling together my draft of what I consider a baseline assessment, almost kind of that first look at the program using SAMM as my maturity model of choice. And what I think that does for me, I mean, quite literally right now, as I'm getting value from it, it's very easy, and this is true whether you're a new leader or an existing leader, it's very easy to not ask the right questions at the right time. You can fall into recency bias, you can say, “Oh, we just had this SQL injection”. And so you'll often see teams start tilting, “Okay, everything's got to be about SQL injection mitigation, primarily queries”. It starts to really snowball because people have that bias, they want to go solve the problem and never feel that pain again. Maturity models, in the case of like, SAMM, if you really kind of pull apart SAMM, it's 300 individual security activities. And so if you do an assessment against that many individual activities, you can't get tunnel vision, you can't get too caught up in recency bias. And for me, as a new employee, at a company that has an established program, established engineering, it's a really good kind of survey and discovery tool because when you ask the questions about each of those items, you meet new stakeholders, you meet new partners, you learn about like, the lore and the history of the company, of why that thing is the way it is. And so it's a really effective way to ask honest questions in a really productive manner and learn a lot about your new employer and their tech stacks and everything else, as well as start that progress towards where do you want to go from where you are today. So it's a highly effective and really efficient kind of way to get into a new company - it’s to start with a baseline and do a gap analysis against the baseline.

Harshil: Right. Yeah, and most often I love your reasoning behind that, which is it gets you to understand how the business operates and what's the lay of the land, right? But a lot of companies typically, just when they want to do a maturity model assessment, they just get some consulting company to do it for them. And in that case, I'm guessing you wouldn't have as much of a learning experience just by somebody else creating a report based on all the questions. They wouldn't really get to learn as much, would you?

Mark: It's a struggle that I have I think because I did do program assessments when I worked at Rapid7. You know, I can [play] devil's advocate, I can say that as an objective party that had no stake in the game, no politics, no salary, I can ask any question I wanted and really dive deep and aggressively towards a conclusion. And I think for many of my clients, it was a great experience and derived a lot of value for them. Simultaneously, doing this maturity assessment myself, one, it really establishes me as the leader of the program. Like, if I'm the person asking the questions, people learn my name, they learn what I'm interested in, what I'm concerned about. And when you start building those relationships at a new company, often people are really excited to help and partner and do what they need to do to enable you to be successful. But we're not great, especially in security, at knowing how to ask for what we want in the right way. And so when you use a maturity model, it's almost like the maturity model is the objective third party and I'm just like the passive participant where I'm asking people questions that could be in other cases, kind of pointed or maybe come off as like, “Well, is he judging me?”. But when it's this maturity model that I didn't create, it's just “I'm just going to build this out and the answers are the answers”. So it diffuses the tension of a new employee to a new company, and lets people give honest feedback. And then when I do that assessment once, I think the other gap here is when I use a maturity model I actually score it every quarter. This isn't once a year, this isn’t once every three years, this is every quarter. And we can get some other details on why quarterly makes sense, but I think from that perspective I don't also want to bring in a third party firm every quarter. That creates a whole other set of problems for me to manage through and cost is also like a reasonable consideration there. So I think there's a lot of pros and a couple cons but overall, for a leader in a security program, I think it's a great investment to do it yourself.

Harshil: Right. And that's interesting that you do it every quarter. How much of a time investment is it?

Mark: I've done this a few times now. This first initial baseline is, I don't know how the hours would have, like, many many days at a time. And it's a lot of time upfront because you know nothing, you have to learn everything. So when you have 300 questions assuredly, like some of them you kind of fill out by touch and feel and you sort of know where things fall. But one of the people on my team who's been here for four years, I let him do an entire draft review with me out of 300 areas he had I think 75 comments. So we had to go comment by comment, like, “Well, why do you think that scoring is high? Why do you think that the context I noted down is maybe incorrect or incomplete?”. So, again, going back to the learning investment, hugely beneficial for me. I've learned so much so quickly, and that's the upside. The quarterly is actually kind of that delta, right? So for instance, I like to use maturity models not just for the maturity model, but I actually associate the identifiers to an individual security activity, to my OKRs, or to my project plan. So I can actually say I'm pulling my shot and we're going to invest in threat modeling this quarter, right? We're going to do some threat modeling related projects. Well, if I associate the maturity model to those projects, when I go to the scoring next quarter, I'm not looking at all 300, I'm going to go zoom into the threat modeling related identifiers I called out, and I'm going to adjust the scores for those. Because the reality is, while you might get lucky, most programs do not mature accidentally. You make investments, you know where you made the investments last quarter, you know where you spent time, so that first baseline is a lot of work, and then after that, it's a couple of hours of work. It's very quick, actually.

Harshil: Yeah. So now that you mentioned you use a maturity model to sort of interrelate or even drive your quarterly objectives, is it driven by a target improvement or target milestone on the maturity model, on specific domains, within specific business units? Or how should someone think about it? Because you could end up spending years and years just getting to higher and higher maturity levels across the entire spectrum. How do you pick and choose where to focus on?

Mark: And it really kind of dovetails nicely into like, how to use a maturity model. I use a maturity model not to tell me what to go do, I use a maturity model to keep me honest to what I'm doing. So in a maturity model, which is different from a compliance checklist or something else, you will assuredly have levels of maturity but the levels of maturity tie back to a stream of topics and those streams of topics tie back to a domain like governance or something. So for me, understanding where we stand and kind of looking at the macro view of where we are is data. But the next thing that I do myself and usually partner again with my team is we go and actually do a prioritization exercise second. I don't want to influence the scores by telling people what my priorities are and I don't want to pick priorities before I know our scores for a baseline assessment, right? We don't want to bias ourselves in either direction, so what we're actually doing literally right now is we're going through the 30, in the case of SAMM, 30 streams of activities which those 300 break into, and we're going to prioritize those. And the cool part about that is, and this is all very simple Excel math or wherever you want to do this, If you have a maturity score for a given set of items for a stream, and you have a priority score for a stream, you can do like a little bit of cell math and you can say, ”This is my highest priority, lowest maturity area. This is my second highest priority, second lowest maturity”. So you don't have to necessarily decide what you're going to work on but the maturity model helps guide you down the path because you said it was a priority and because it's low maturity, it's kind of a foregone conclusion that you might want to think about investing there. So again, it's like, how do you use a maturity model? It's not a checklist, it's not a thing. Like, not everything in the maturity model, you have to go. you still have to be a domain expert, you still have to know where your business needs are. Are you having sales problems? I might have to invest in more governance because we're having problems with our sales cycle because we don't have a lot of materials. So you still have to use logic and still understand, like read the room. But I think the maturity model gives you those guardrails on that journey.

Harshil: Have you used these maturity models for either showcasing products, or your team's capabilities and achievements to somebody outside your organization, outside the security team, or even for things like justifying existing or new budgets and investments in this?

Mark: Yeah, in fact, one of the stories that I relay a lot is going back to Duo Security, and I really credit the Duo Security leadership team, we in fact published the - going back to your call about domains - we had five domains in our maturity model of security for our project program at Duo, and we actually more or less published every single quarterly board deck with the growth of those domains in the actual board deck for our maturity model. So that was more of a talking point. We actually had some metadata about why did it go up or what were some big projects we worked on. But our maturity model was front and center of our board reporting process. So yeah, going back to how do you sell internally, when you have a maturity model and if you walk around your companies that you've ever worked on or people listening, how many parts of your organization have ever built something as robust as a maturity model based program? There are definitely pockets for sure, like I've been in other companies that do, but it's not common. And I think when you are standing on a maturity model and publishing your data and tying it back to OKRs and project plans, it's a pretty easy place to start to say “Hey, invest in us because we know what we're doing, we know what we're focusing on, we know why we're doing it”. It makes the sales cycle for your security teams a little bit quicker because you are showing your work, you're showing your data and you're making a conviction like this is where we want to go and be great at, and it definitely helps.

Harshil: Yeah. And I also like the idea of showcasing due to other teams, it spreads awareness about what it actually is, like how complicated this overall application security domain is, right? It touches so many things and it's just not a technical function. It's got process, it's got governance, it impacts compliance and sales. It's got so many different moving pieces and not everyone outside of security understands a lot of these things. So it becomes a great way to increase awareness, educate and shift knowledge outside of security. And I think that's also one of the other topics that you've talked about a little bit, which is shifting knowledge left. Tell me a little bit about what you mean by that, and when you were presenting on that topic, what was the driver behind talking about that?

Mark: Yeah, that was a really fun presentation for myself and Fletcher Heiseler, who was the successful founder of Hunter2, which was acquired by Veracode a couple of years ago, an education platform for application security. And we enjoyed building that talk for Black Hat because one of the big threads I pulled at Duo and working to pull here at FullStory, security teams - and you can see this pan out in the BSIMM data that's released by Synopsys every year- if you have, like, let's say one software security group AppSec engineer for every 100 software engineers, that's about par for the course, and certainly there are lower and higher. We will never have a one to one relationship AppSec or ProdSec to software engineers. It's just not possible, right? So the question becomes, how do you scale a great security program up? And in the market and in the broader tech space, like very technical sense of text space, we've been doing shift left, which for the most part means building security more or less into your CI/CD pipelines, exposing engineers to security related tests or software composition analysis, risks for vulnerabilities in dependencies were shipping. And so there's a lot of visibility, but I think it doesn't go far enough left. And in fact, if you look at the Microsoft secure development lifecycle, training is the start of the secure development lifecycle. And for me, that's everything. We have to build every engineer into a security engineer. It's not their full time job, It's not the number one thing they know in life, but there is so much value and also interest. Like, have you ever met an engineer who is not fascinated by hacking? They love it. Everyone loves it. It's such a cool thing to learn, you learn so many tricks, but it's career development for them. So for us, in that presentation, it was really about where are all the kind of conceptual gaps on how we're educating people. Like, literally, the tools, the approaches to education, where are they falling flat? And that presentation was kind of just that. What we've seen the market not work, how we can see that education program done right is actually a super high leverage and high value investment in your engineers. And the last thing I'll mention is that when you invest in any one of us, when someone invests in you, what would you do back for them? And so when we were giving education opportunities to engineers at teams I've led, they want to partner, they want to ask you questions, they want to get your opinion on the software they're writing. So it really is about that reciprocation that if you invest in someone else, they'll invest in you. And as a security team, it really hasn't always been a partner program. It's been like a draconian, top down thing. But in the last five to ten years, I think we as an industry are getting a lot more centric on stakeholders, and buy-in, and culture, and empathy, and education is a great way to kind of kickstart all those things.

Harshil: You know, one of my challenges with training, especially at scale, where there's hundreds of thousands of developers, is that we can train all we want. I mean, we can offer or incentivize or make them take secure coding classes and training exercises or what have you. But I'd love to get your thoughts on what is the actionable thing, behavior that you would want a developer to change or even to keep in mind as they are doing their job day to day? Because if I'm writing Python code or Java code or whatever, I might have taken secure Java coding classes six months ago or a year ago, but in my day to day when I'm pushing feature, when I need to meet a deadline, when I need to merge my code today, get it tested, released before the weekend, what is that actionable thing that developers need to keep in mind?

Mark: I think one of the most common red flags that I try to observe and then coach people on when I see it is you'll hear someone say, “Oh, we're going to bring security in, but there's no relevance to security here”. And I think that's one of those things that I try to coach the engineers on. You know, understanding what is relevant or not relevant to security is harder and harder and harder, especially in agile microservice based organization where you might know your piece of the pie and that ecosystem of Microsoft architecture but understanding things like Taint analysis where some input goes through like 35 microservices, into a log store, and then that log store gets shown on a web page, and there’s cross site scripting. So it's one of those things that if engineers stop to consider what is maybe wrong or risky in their software whenever they feel like they're confident that it's not important, please double check with us. Reach out to your AppSec and your ProdSec team, ask a follow up question. We do Office Hours here for product security. Come to Office Hours, just say, “Hey, take a look at this PR, here's the functionality I'm working on. I don't think there's any security problems here. Do you all think there is something here?”. And so more than anything is the engagement. Keep coming back to your apps and your product teams, do not conclude yourself in a vacuum that there's no risk.

Harshil: Right. That's a good insight because the level of depth and understanding that a security engineer has, theoretically a developer would not have the time to go into that level of detail about specific security issues, right? So they're focused on coding, they're focused on quality, performance, reliability, scale, shipping their code, all kinds of other things. They've got too much on their plate, they'll never be the security experts. So I like your idea of them, at least at the minimum, knowing when to reach out to the experts. And obviously security should make themselves available in a collaborative fashion. But yeah, I really like that idea. Now this is also very challenging in the sense that developers are typically under a lot of timing constraints and they have to move faster and deploy faster and code faster. Do you think that we are headed in the right direction as a product security application security domain in the world of modern DevOps and fast agile development lifecycles. Anything particular that you recommend people keep in mind?

Mark: One of the things I ran into more times than not is when I've described things that have worked for me in presentations, grabbing drinks with someone in the space, I hear a lot of repeated patterns like, “Hey, this is the way to do X”. And I think it's interesting because a company like Duo Security, with a very modern tech stack, modern engineering lifecycle always, we really were like a release every two week company. We weren't pushing deploys like 40 times a day. So I built a program like that. I built a program where we didn't block builds, we didn't blow up developer LDEs with like a thousand findings every time they wrote a line of code because we didn't need to. We actually had time on our side that we could slow down, not just smatter all these alerts on people's day. And again, going back to the relationship, one of the things we did very specifically, which I think goes back to even the agile approach, while you could conceivably ship, you know, if you're like an Airbnb or like an Etsy shipping, shipping, shipping all day, you very well could ship a vulnerability. The thing I think a lot of people maybe get wrong about vulnerability management, risk management in companies is just because you ship a vulnerability for most code that could be there for months, years, no one might know. And so I think it's like this idea that we have to stop and break builds and blow up CI/CD pipelines and just show every single possibility, third party dependency that you ever had a vulnerability in I think is an anti-pattern. I think great product security teams should actually treat their work more asynchronously to the engineer lifecycle. I can scan up a product security CI/CD pipeline, run all the tests I want, blow up builds, do static code analysis, all the things. But we should own the triage, we should own the first pass of signal to noise. And I think that's really where a lot of AppSec and ProdSec programs have gone awry, and why a lot of security teams are not loved in their organizations is because you spend so much time plumbing in tools and shipping noise to people. And in the modern agile ship, ship, ship, deploy, deploy, deploy, you could break a build every minute for an engineer trying to ship, and how will you ever create a relationship with that person? It's an uphill battle that's not one I want to fight. So I think we really have to consider what are the ways engineers work, how do we have high leverage, low noise outputs of our work as product security engineers, and how do we only intervene in their day to day work when it's absolutely risk sensitive to the business operations like core security. But I don't think a lot of people are thinking this robustly about the problem. I think they're like “Put the tool in the pipeline, put the tool in the pipeline”. That's a great way to not have an AppSec team that is talked to at your company.

Harshil: Yeah, I 100% agree and I think a lot of it has to do with the way a lot of the vendors in the application security space talk about integrating everything in CI/CD and break the builds. But the reality is not every security issue is also important, not everything needs to be fixed at the same time, and the best people that can make those decisions of what has to be done and when and what's okay to live with in a lot of cases are security people. I love your suggestion which is that as security professionals we need to own the triaging and prioritization.

Mark: I think the other thing that happens is not only are you going to frustrate your engineer, but what happens next? They're immediately going to call a product security engineer and say “Why is this broken?”. And so not only are you creating unplanned work for them, you're creating unplanned work for your ProdSec team. Like, why do you not want to be heads down working on a great new static code analysis module or great new enhancements? You don't want to get yelled at, you don't want to get pulled into a bad bug triage process either. So I think if you flip the model on its head, the incentive model, going back to incentive models being important, if you as the product team own first pass triage, your incentive model is to tune high signal low noise because you don't want to go through a thousand bad results either. If you give it to your engineers and they complain to you, you're not incentivized other than to not have them complain. Like that's not the right healthy model in the first place. So owning your problems, and we see this in DevOps, right? Like you are an SRE for your own micro service, if it goes down you get a page, if it has bad performance you get a page. It changes the incentive model that you're going to create resilient, well tuned, well resourced, thoughtful services if you own the problems as well.

Harshil: Right. So on this topic of security that should behave or should act in more closer alignment with dev teams and SREs and similar things. In your opinion, what does the future of product security look like five years from now?

Mark: You know, space wise we move a lot slower than the vendor space in terms of how we conceive of our job or how we conceive of organizational intersections. It's a pretty slow moving train in our space. Even product security, something that we were talking about briefly at one point, what is product security? Is it the thing you sell? Is it an internal product or solution that your corporate team builds? So I think one thing we have to reconcile is when we say product security are we saying that we are separate from corporate security, or are we simply saying that there's a lifecycle management of software engineering, cloud architecture, privacy engineering, and we can actually expand the scope and agreement of a product security team in our organization. I think that's an interesting topic that you would raise. So that's one kind of question I'm wondering like where are we going to go left or right on that. One thing that I'm noticing and one kind of hope that I have for product security is there is so much technology to manage in a product security team these days that you almost need your own DevOps team or SRE team or Ops team. So many services, you have to run, so many tools, so many tool chains. And we're getting to the point that I think we're going to have to figure out how to be more like a QA team in most organizations. How do we leverage frameworks and facilities and platform components so that we can seamlessly integrate with the lifecycle of things? Because I still feel like ProdSec programs overall, AppSec programs, they're still on the periphery. So I think the evolution in five years is that we learn the lessons of quality assurance. Because if you talk to an engineer that's been around for a while and you say, “Oh yeah, QA, what do you think of QA?”. They're like, “QA is great. They find my issues, they ship good POCs of a bug, they tell us X and Y and how to fix it”. QA teams and ProdSec teams are basically the same team with a different purpose. Like we're looking for bugs, we're shipping patches, we're concerned about impacting the business. And I think most teams love their QA organizations. A lot of product security teams are not loved, and I think it's just about the way we work and the perspective we share. And I think there's a lot of reconciliation in the next five years that we are actually an engineering partner. We are an engineering force multiplier and an enabler. And I think a lot of the tools that we've had historically did not solve a problem of an engineer, it created problems for engineers. And I'm hopeful that the vendor space overall, how do I conceive of rapid risk assessments for my app? How do I conceive of the maturity, the metrics, the reporting KPI dashboards of my software for security? We're not exposing enough data to engineering teams to know if they're healthy or not healthy, the same way they get code coverage tests for their QA team. So I think a lot of it's visibility, I think a lot of it’s relationship building, and I think the other bits are just learning lessons from others that have gone before us and are doing a better job. I think some of those.

Harshil: I love that concept of self reflection and continuous learning from within ourselves and from other teams that have gone through this journey. With that, Mark, this is all the time we have for this podcast. Super, super interesting conversation, we love all these insights that you're sharing with our audience. I'm really hoping we can have you once again sometime very soon.

Mark: I appreciate it. Thank you so much.

Harshil: Yes, thanks for being here, Mark. I really appreciate it.