EP 27 — Mohit Kalra: How Sprinklr Scales Product Security

Harshil: Welcome, everyone, to another episode of the Future of Application Security. Today, I have Mohit Kalra from Sprinklr with us. Mohit, welcome to the show.

Mohit: Hey, thanks, Harshil.

Harshil: Mohit, I would love for you to give a little bit more introduction other than just your name, in terms of what do you do, where do you work, and what are you passionate about in security?

Mohit: Okay, yeah, that sounds great. So let me start from the beginning. I started my career as a software engineer back at IBM, and then I kind of moved to Adobe, where I spent a long and satisfying career. I started as a developer there, managed a lot of PDF technologies, worked on flagship products, and then eventually I started getting into application security and I kick started application security for my group. And then eventually I moved into the more central security team by transforming the team from just being desktop to cloud and mobile. Then I've kind of taken up cloud security teams, cloud security engineering teams. So after having spent quite a few years at Adobe, I kind of joined Sprinklr as the head of product security. So what I do today is like, my team is responsible for ensuring that our product, be it the application, be it the infrastructure, are secure proactively. So my team will undertake security tooling, automation, threat modeling, pen testing, engaging with vendors, sifting through a lot of data to find the needle in the haystack to ensure that we can proactively secure the product and give both tactical and strategic recommendations to the teams out there so that our product is built securely and it's deployed securely as well.

Harshil: That's amazing. I love the use of product security as a function. Tell me a little bit about how that is different from traditionally what we have application security or infrastructure security.

Mohit: I think it's a great question. So the two security streams of cloud security and application security are kind of merging together. So my definition is that if you are going to do anything with the product, if you're going to write code for it at the application layer, or deploy it at the infrastructure, or even build it or source some external components in its supply chain, all of that kind of is in product security. So if you're going to be a part of the product, be it infrastructure, be it the build process, or be it the actual code that is the product itself, all of that is product security, and securing that kind of is an end to end role where you're not supposed to, like you should be able to secure above the network and below the network all at the same time.

Harshil: Right. So basically, because the product runs on the different layers of the stack, that includes the cloud and a lot of the cloud configurations and the infrastructure components that go on it like containers and artifacts and things like that, they're all effectively stored as code or artifacts in a registry, so it makes sense to consolidate those functions under this umbrella of product security, and that brings a new holistic view. And by the way, in my previous role, my team was also product security and that was exactly the reasoning behind it because containers are stored as docker files in GitHub, cloud configuration is stored in GitHub, so I need people who can read, understand code and be able to work with it. So we define this team for product security, which is the traditional AppSec and cloud security combined together as well.

Mohit: Right, exactly.

Harshil: Fantastic. So you've been at Sprinklr. What does Sprinklr do?

Mohit: So Sprinklr is into Customer Experience management. What that means is if you want to talk to your customers through social media, want to listen to them, want to advertise on multiple social media platforms, customers use the Sprinklr platform in order to do that on their social media streams. Also, we're also into customer care as a service. You see pop ups, artificial intelligence based pop ups come up on sites that interact with you or give you voice support, et cetera, Sprinklr runs behind the scene as well. What we do is we try to make our customers happy and we provide that platform that allows us to do that.

Harshil: Fantastic. So for an audience who doesn't understand this in depth, why is AppSec or ProdSec important? Where does that fit into the broader business?

Mohit: So, security is very important for the company because we're a SAS company, we're running in the public cloud and we deal with data. We would deal with data, we are custodians of other people's data and we are responsible for ensuring that security is a scalable program within the company that works in tandem with the engineering and the tech ops teams out here. As any other SAS product, when you are providing the interfaces for the customers to talk via mobile applications, APIs or web applications and then behind the scenes you deal with data, it's paramount that security is important for any company. And the more critical or the larger the data is, the responsibilities managers

Harshil: Right. Yeah, I have spent some years leading security in customer experience management as well so I totally understand exactly what you're saying. There's a very large volume of data that these platforms can consolidate, for sure. Now, since you recently joined and before this podcast, we’re talking about your goals for maturing product security to the next levels, the milestones that you're thinking about, can you talk briefly about how you think about maturity of product security programs? At least from your perspective, where do you want to take this function towards?

Mohit: Right, so I think when you start a security program you are tapping into whatever you can. You're running any tools you can, you're running it on assets that you're able to identify and then you're able to take help from vendors, et cetera, take a security program off the ground. Eventually what you do develop is you develop the right security stack for yourself. You have the right capabilities in terms of tools, you have the right capabilities in terms of process and people that are able to then take the security program to a more scalable level. So from a more immature program you kind of step into a more consistent, reliable and repeatable program that eventually you can execute with and kind of tie it down to the release that the company is making out there. So in a mature program, once you have hired the right people and you have established a strong program and process and you have the right security capabilities, you know, you first start with kind of being able to enumerate the security risk within the company and then kind of pipe it into the engineering processes so that engineering can start working upon remediating those risks. So this is like a step that every company needs to take: the ability to enumerate, have the processes and have the ability to remediate the risk in a timely way. And as you advance, you kind of start investing more into protection such that when the security program sits side by side with the application itself and the product itself and starts protecting the application in real time, such preventative controls are kind of where people strive to be such that security is not a process of enumeration and remediation that happens afterwards, but it's more kind of a real time protection that can happen. But as we go forward I see like the security program is going to evolve into what we do today, more of DevSecOps happening where the issues are given to developers way in advance and also some level of protection at runtime that is going to protect the product from the risk that might not have been remediated yet. So a combination of that is where I see the product security program to go. I also see that it's not going to be much difference between cloud and application. You're treated the same. So protecting at both levels, at the application layer and the cloud layer, is going to be pretty commonplace in years to come.

Harshil: That's one of the best definitions I've heard of a program going from reactive to proactive and you explained it in a very easy to understand way in tangible terms. So going from just enumeration of risk and piping it into engineering to react to the risk that you already have to going more proactive and invest in protection and building proactive, preventive controls. So you preempt the risk that might manifest in production at some point. That's phenomenal. So that's also a very complex topic though, right? There's many layers upon layers into this. So when you just pick up one of the first ones, enumeration of risk, there's multiple parts of it. First is enumeration like what actually do you have in your environment, understanding what do you have in your environment? Does security even know about everything that exists in your dev organization? Understanding the risk of it's not just risk, it’s not vulnerabilities, it's vulnerabilities are a part of it, but what is the true risk of that? And then being able to communicate that effectively, like that by itself, is a very complex project. Let's peel the onion more. Let's talk about starting with the enumeration. Like, what do you actually have in your environment? And the reason I ask that question is typically security teams are central security teams, product security teams are central teams with many many dev teams that keep changing, that keep building and deploying things. They don't necessarily come to the security team to tell them like “Hey, this is what we're doing”. So as a central security team, how do you get that visibility into what is your attack surface? What do you actually have in your environment?

Mohit: Right, that's an excellent question and maybe we can spend the entire podcast on tha, haha.. The first thing is, of course, you're new to the company, you're coming in, you don't know what to secure. You go to your product company's page and try to see what are you selling to your customers? That's the starting point that gives you at least you get the idea of what the product is about and how many sub products there are and what kind of offering that the company has. Within the company, there are lots of ways to learn about the product, not just of course you should talk to people, learn about the product, get hold of recordings and architecture diagrams so that you can better understand the product, but you can also listen to the product as things happen. A few ways of doing that is if you align to the release process of your company, you will all of a sudden be able to listen to all the requirements that are happening. And then when you look at all the requirements, you start understanding the product in more ways than just kind of understanding the marketing page of that product. Another trick I can share is that if you start looking at the main developers in your company and you start looking at their source code activity, you will start seeing the major contributed repos that they're contributing to. So at the end of the day, you're supposed to make an inventory. What are the products? What are the functional solutions underneath it, so that you can make logical attribution when something happens and you want to kind of file a vulnerability against somebody, you need the list of products and also the owners. You need a list of all the repos that your company has. You need a list of all the APIs, the mobile application. So kind of it's just about collecting information, centralizing it, and it takes time sometimes to build and sometimes there are surprising discoveries. But most of the time when you have a handle over what you're securing, then the problem of applying tools to it kind of gives proper coverage. And that's kind of one important step that you do take.

Harshil: Righ, right.. And since the scope of your function is product security, I'm guessing you would also look into the artifacts being deployed, being built and deployed, right? Artifact registries, AWS accounts and things like that, which become also complicated if you're multi cloud, if you have acquisitions with different tech stacks and all of those complexities.

Mohit: Yes, all cloud accounts, all hosts, all containers, all applications, all built artifacts, everything I think is kind of something that you need to have a handle over. But of course, day one, you're not going to secure all of them, so you start strategizing on how to start having incremental capabilities within the company that will start kind of securing the right assets in the right time.

Harshil: Right, right. That's the key part, the right assets at the right time, right? That's where a lot of the risk calculation comes into the picture. So help me understand how you calculate that risk in an organization like yours.

Mohit: Let me also kind of stress that when a security program executes, what you will undertake as a capability will not just depend on risk, but also which is going to be an important part, but will also depend on the needs of the company and the needs of the business. For example, if a customer does require to see some reports that are important to them, you will prioritize that hire. Or if there's a compliance need to support some security functions, you will support that. So the security program, unfortunately, is not going to be very easy to add this capability and then this pretty much the security team is like the orchestrator of capabilities. The decision to take the capabilities depends on where the business is going, who's asking for what the risk is like. So 0nce you have done an inventory, you start kind of firstly taking some of the standard tools and you start applying them. You bring in vendors, you start looking at critical systems in the company and identifying them, and then kind of start doing deeper threat models. And then as you see the companies evolving with its product, you tap into all of these changes and kind of try to do a security requirement review to make sure that if they're doing an integration with another partner or with another API, you just want to know that is happening. And that's when you start kind of inserting yourself.

Harshil: Right. I love the mention of going beyond risk management, but also towards enabling the business, right? Sometimes, even though the risk might be low, you might have to focus on things to enable the business to unlock a deal, to help a customer to unblock an audit. That's the right thing to do, obviously. So you're wearing dual hats, enabling the business, also managing the risk. So you mentioned interesting things around just listening to what is happening within the engineering organization, understanding who is working on what is actually being built, deployed, what is getting the attention from the leadership, from the customers. How do you establish that as a process? Let's say you're looking at quarterly planning or sprint planning, and you're looking at all of those things as a security person, where you don't have the detailed context of what microservice, what code repo belongs to who, and what, how do what is high risk, what is low risk? How do you put that lens, risk based lens, on this?

Mohit: There is going to be like you should be talking to engineering teams and tech ops teams to educate yourself. That is a must so that you can understand. You need to listen into where the business is moving and can automatically see the critical items bubble up automatically. But when you start kind of taking all of the inventory that you've collected, you do need a process to now make the security activities not a random attempt at like running a tool against a random set of assets and be done with it. You always have to think about how am I going to take this security capability that I have in my hands, take the assets that I can see and then kind of create a more repeatable process that is rigorous and of high quality and is able to give me what I'm looking for. For example, you have a tool, you see some assets and you run it once and you're done with it. That's kind of going to be very ad hoc. But if you start seeing that, how can I have the rigor of repeating this process? How can I extract the security backlog that the tool is saying in a meaningful way to the engineering team? How can I have a process with the engineering team to remediate this in a timely way? That is where the process binds the randomness of running things at your own convenience to kind of running them with more consistency and providing value to the business by saying that the risk is actually getting remediated. You need a pipeline between enumeration, which can come from tools, processes, threat models, pen tests, so on and so forth, and then you pipe them into your process, and then the process essentially delivers, creating a security backlog. The security backlog is what needs remediation, for which you need an entire process to remediate in the right amount of time.

Harshil: Right, right. I love the way you're talking about this. It's very structured, and you're rating the risk pipeline to a backlog, and then you have a whole other process to manage that backlog and remediate the risk. One of the foundational challenges that I've seen in many teams struggle with is you have a high level understanding of what the products are that are critical to the business, that's making a lot of revenue, that's used by a lot of customers and is a higher priority. But then on the other end of the spectrum, you have hundreds of thousands of code repos, thousands of containers and AWS assets. How do you bridge that gap between a high level product that sits as a skew on what you sell, what your company sells, with the tactical things that developers are working on day to day? Like, how do what belongs to what belongs to what?

Mohit: So attribution, I think, so the more time you spend with your product, the attribution problems start to solve by itself. What happens is when you take a tool it will give you scale. Tool will give you a scale that it can say, I can scan all your containers in whatever sum out of hours, and I can give you a report that tells me what are your most vulnerable containers, or I can kind of look into your open source libraries and tell you where is the CVSS score that is high.So you have tools, you get a scale, and when you have a scale, and when you see a problem, you're motivated to go back and attribute that to respective teams. It takes time, but product security at the initial days is more like many people blindfolded touching an elephant and then realizing that it's an elephant. You know, that's pretty much how it is. If you become overwhelmed with the number of assets you have and then you want to solve everything on day one, that's not how product security is going to happen. Product security is an incremental process towards making improvements every day. Once you've made an improvement, once you've understood scope, once you've understood priority, then you don't go back and you just make more improvements day in and day out. So over time, yes, you might have missed out on products you shouldn't have, you might have strongly prioritized a product priority, but over time, you recalibrate. So it's not like you don't take a decision once and you stick to it. You always improve, you always iterate over your decisions on is this the most important asset, is this the most important priority? Because security at the end of the day is a problem of prioritization. It's not a problem of just brute forcing assets with tools or with people and then saying that this is what needs to be fixed.

Harshil: Right, right. Yeah, that's fantastic. It's all risk management at the end of the day. Right, so let's walk through that. Now that you have enumerated the assets, the enumeration of the security issues, I'm sure our audience understands that really well through multiple different activities or tools or processes or what have you, let's say you come up with a set of risks that are high priority, that need to be remediated. And this is still, we are still talking about the first step, which is the reactive version of your product security maturity, but you need to pipe that into the engineering processes so they can be remediated correctly. Now, it would be amazing if we could just create a Jira ticket for somebody and people would fix it and we call it a day, right? But obviously that's not how things work. Do you have any tips or tricks that you can share for accelerating that remediation by different teams?

Mohit: Right, so the first trick is that, yes, you should ticket your security backlog because you want to make the process agnostic of the back end tools. You have one tool, you replace it with another. But if the ticketing is consistent, the engineering doesn't have to relearn a tool, so don't bring engineering to your security tooling just yet and have them and you go towards the process that they operate out of. If they're working out of ticketing the right tickets out for them. The other thing is that when you're ticketing, you have to be strategic in how you ticket that. For example, a container can have thousands of vulnerabilities. I'd rather have one ticket per container rather than thousands of tickets out there. You have to be strategic in how you capture the backlog, and it's important to do that. And once you have done that, then labels or some level of data tagging is your friend because these tags will result in important issues bubbling up. You being able to create dashboards, you can sift the noise. Like you can have an entire backlog in a dashboard, but if you have labeled it with some special label, you can have a prioritized backlog that is different from the underlying backlog, and then you can just hand off the smaller version, which is highly prioritized to the engineering team. So the trick is to unify your security backlog to a common platform as opposed to individual tools. Ticket it in a very strategic way such that it doesn't cause too many ticket numbers or just doesn't make any sense. And then once you have it, then you kind of start visualizing or kind of dashboarding the data, and then you kind of go to your engineering teams and now strategize the remediation strategies.

Harshil: Yeah. I'm going to ask you a very opinionated question. Do you ticket these backlog items into a security project in Jira, or do you distribute it across a dev team’s Jira projects?

Mohit: If my team needs to do the work, it goes in a security team project. But if someone needs to do their work, they need to see it in their project because if it's out of sight, it's out of mind. So we need to go to where engineering is operating, to the tool they use, to the queue they are in, and put our security tickets there. You can create a security issue type so that it distinguishes the issue from other issue types, but you should be in the engineering team's queue, not in your own queue.

Harshil: Right. Yeah, that's lovely. Either you create an issue type of separate security team or security issue so it gets flagged. You could also use a label security issue so they can flag it, prioritize it accordingly. In your experience, who do you think are the right group of people to get those things prioritized? Because a lot of times the basic simplistic approach is to talk to the developers to fix them. A lot of times it doesn't work. In your experience, who is the right audience to make a decision whether the security backlog item should be prioritized in this print or not?

Mohit: Right. So firstly, you have to understand why you are prioritizing something. The reasons for that could be because there's risk associated with the issue. The customer is asking for some remediation or you just have to do it because you have a necessity that you need to meet. So firstly you need to understand the underlying risk and firstly the security team needs to internally agree that this is the right priority. You first have to have, let's say if there's a sprint coming up, you need to internally decide that’s what's important for this sprint and why. The why is very important because it's not about plugging in 20 issues or 100 issues, it's about these 20 and why these 20? Like, why not the other 20? I think that is an important aspect of where the product security team partnering with the technical program management within security is able to kind of come with that. Once we have that in play, then we need to work with the engineering team to first, not just say, you go and fix it, we just discuss it, see what their opinions are about it, do they push back? Do they not agree with the priority and kind of reconcile the truly negotiated backlog that we are agreeing upon. And once that happens, that backlog is then tracked to completion like any other project is, because now we have exited the security domain and now we have entered the project execution domain. And that can be treated like any other project that engineering is conducting. It just happens that these tickets are specialized to solve the security problems of the company, which is why it's important to converge into how the company operates rather than bring people to your processes and your tools.

Harshil: Love it. Yeah, I think it's a great piece of advice and especially the handoff between when the security owner not necessarily the ownership, I would say, but the security heavy work ends and when it gets into the realm of execution by the dev teams. And depending on the culture, it could be an engineering manager, product manager, tech lead, whoever needs to look at it, but whatever the culture for the company is, they then take ownership of it.

Mohit: Exactly.

Harshil: Love it. I also like the fact that you mentioned earlier, which is to tag it with appropriate labels or appropriate level of metadata so then you can build dashboards and then you can report on things. Can you share some examples of things that you might be reporting using this metadata?

Mohit: So firstly, we are able to sort it per leader, like, hey, this leader has so much security backlog so they can focus on that. We can look at the overall security backlog in a split as per the severity so that we know the size of the security backlog as well. For example, if we get a pen test done within the company and we want the vendor to come back 90 days from now, we will tag them such that thematically we're able to identify. So what you'll see is like dashboards that either show attribution, either show the priority or show some theme that essentially might be a theme, like for example, we want to take care of our pen test findings by this date and we want to work towards that. So all of these labels essentially allow us to create these dashboards and then we can kind of zoom into one problem rather than sift through like a bunch of issues that might be unrelated to each other.

Harshil: Fantastic. Yeah. In my previous role we used to use the terminology of a remediation campaign which is exactly similar to what you're talking about, which is at some point during annual customer facing audits or pen tests, that would be the number one priority for us, for everyone to focus on. But during other months, other quarters of the business, there would be other priorities. Like we would run a campaign for fixing every single base container image that would be used across the company. So in that quarter that was a priority. We would build dashboards specifically for fixing base images. At other times it would be like a Java focused remediation campaign or JavaScript focus campaign, whatever it is, right? So pick a theme, build dashboards and drive metrics for that particular business objective within that time frame.

Mohit: Right. I think one important thing is like while we're talking ticketing, tickets don't necessarily have to be small issues, they can be strategic remediation items as well. And the security team needs to find a balance between those short wins that will make the issue go away and the long strategic investment that if a company makes might kind of reduce the entire class of vulnerabilities. So while I say ticket and this sounds small, I think the work behind the ticket depends on what's in the ticket, not necessarily a vulnerability, but it could be like an architectural investment that one company needs to make to become better.

Harshil: Yeah, for sure. That makes a lot of sense and I think aligning with how engineering works within your organization, that would be a much better decision. So if engineering is used to structuring types of work in the format, for example, of an epic and task and issues and things like that, then aligning with that would make a lot of sense because that will also convey either the size of work or the priority of work to whoever needs to work on it.

Mohit: Right.

Harshil: Awesome. Now let's talk about being proactive at this. We talked a lot about being reactive, responding to risk being introduced. You mentioned a few things around investing in protection from the beginning where security sits side by side with the deployed apps or the infrastructure and presents real time preventive controls. How do you envision that? Is that just a technology specific thing or is that something more than that?

Mohit: So the way I've seen the evolution happen so far is that the security team takes its own set of tools, the assets, and then runs stuff. Then we start interfering with the developers workflows that you're checking in bad stuff, we will try to break your build, et cetera. And in the protection phase, you're kind of sitting more alongside the application and trying to make intelligent decisions around whether this is supposed to happen or not. In the past, we've done this with infrastructure. If you're going to make infrastructure insecure, the automation would come and make it secure for you. So that if you're making a security mistake infrastructure, we would essentially fix that for you. So I think on the protection side, it depends if you're trying to protect using an intelligent next generation WAF or you're trying to protect your APIs or you're trying to protect your infrastructure, that protection essentially is very specific to the company. And I'm seeing that technologies are springing up in that layer, but it's not equivalent to the enumeration that we can do in security today. So preventative controls are there in some shape or form, but it's the security team that needs to think deeply about what they are trying to protect and then adopt some of them that makes the most sense for them. For some people, it might be the infrastructure policies that are important. For others, it might be the APIs that not just need testing but also need protection.

Harshil: Right, right. Yeah, that's interesting. I want to double click one topic, which is the sequencing of these controls. Now, I'm not arguing with the need for protective controls, but does it have to be sequenced. Actually I don't even know if you said it has to be sequenced after the enumeration and there's left of deployment pipeline controls but is there a world where preventive controls like this RASP, whatever you call it comes in before anything else?

Mohit: In my dictionary I think you're supposed to get more comfortable with your assets, your company, you're supposed to understand it a bit better before introducing preventative controls. Like, just to take an example of let's say APIs, you don't want to protect your APIs, you first want to test them, be comfortable, have an idea where they are, be able to discover new APIs that get introduced. So you kind of want to get a handle over what it is that you have and then eventually move towards protection. I don't see that you protect first and then test later. In my way I think I would kind of progress on start small, kind of get the confidence and then kind of go big because when you're kind of in the product you have a much higher responsibility to not just protect but also the runtime and the uptime of the product is equally important at that time.

Harshil: Right? Yeah, 100% agree to that. One scenario that I was thinking of and I asked that question is back when Struts came out as an issue, I was talking to one of my peers and they were a big Java shop and they were using Struts all over the place within their platform. So effectively for them to remediate Struts would take 15 developers and six months worth of work to get there. But obviously you can't live with that risk for a long time so they had to invest in protective control as a surgical instrument just to manage that specific risk because it was so high, such a big risk that they had to build some controls for it. And this was at the beginning of their AppSec journey. But maybe that's a very specific case where it justifies that investment.

Mohit: No, it's a good point. I think at the end of the day we are here to not do things in a certain order, but to reduce risk. So whatever reduces the risk in the shortest amount of time and keeps the business up and running I think that is the right way to do it. In RAFTS, usually you can bring up a protection but then you should never forget to fix the underlying problem so that eventually maybe RAFTS is abandoned at that time. It's not a full fledged protection service if you will, and then you go back and fix your in this case Struts and then you can kind of enable stronger rules at the end of the day, but whatever works for the business to reduce risk is the right strategy. There is no strategy that is right or wrong in this case. You just need to adapt to your business’ needs.

Harshil: Right, that makes sense. Now Mohit, as you look into the future more, what does the future of product security look like for you?

Mohit: So one thing I forecast is that context will play a very important part in product security in future. Like, the same CVE in two different places today doesn't mean the same thing. And there's a lot of negotiation needed in order to come to that. I believe that a lot of product security will factor in context and then not treat all CVE or CVS scores the same. and depends on today, the approach is a bit simplistic and the approach will become slightly more complex. I also see, like I've said this, I think we'll see more of product and cloud merge, not just from a responsibility perspective, but even from the tooling perspective. The cloud security tool chain will climb up to application security. The application security stacks provided by vendors will climb down to the cloud security space and eventually we'll have companies who are selling security stacks rather than security tools. And that is going to be like one comprehensive solution from the top to the bottom is how I see the industry move towards.

Harshil: That's such a recurring cycle, right? There's always a consolidation and then there’s accreditation. And every few years we go back to consolidation, right? Mohit, this has been such a phenomenal conversation with you. I appreciate you spending time with us on this podcast. Thank you so much.

Mohit: Awesome. Thank you so much.