Real-Time ML Workflows at Capital One with Disha Singla – 606

All right, what’s up everyone? Welcome to another episode of the Tumol AI podcast. I am your host, Sam Charrington. And today I’ve got the pleasure of being joined by Disha Singla. Disha is a senior director of machine learning engineering at Capital One. Before we dive into our conversation, be sure to take a moment and hit that subscribe button wherever you’re listening to today’s show. Disha, welcome to the podcast. Hey Sam, thanks for inviting me here. I’m very excited today. I’m super excited to chat with you as well. I’d love to get started by having you introduce yourself to our audience and share a little bit about how you came into the field of machine learning. Sure Sam. So I joined Capital One earlier this year as a senior director of machine learning engineering. Before Capital One, I have had opportunity to work at some great companies like into it, Sony, Newstar, etc. So my career, I started as a full stack engineer, but I had a deep love or passion for data. So the whole end to end life cycle from collection of data all the way to driving meaningful insights. So that’s how I organically continue to grow in this role. And here at Capital One, it’s a great company. I joined earlier this year where I’m leading a group of very talented data scientists, machine learning and software engineers. A group is called data insights. We are working on something that’s very close to my heart, which is democratizing AI, democratizing ML by making ML available to everybody. So our team has built reusable libraries components and workflows and have created a platform which allow the citizen data scientists to drive meaningful insight using our various offers within Capital One. Awesome. Awesome. Can you share maybe an example of, well, a couple things. First, when you think of a citizen data scientist, what’s the kind of archetype for that person? What are they typically? What’s their day job, so to speak? And then what are some of the things that they have done with the tools that your team is providing? So to me, let me take you back like five to seven years back where everybody wanted to build models, right? But not everybody can formalize. So people grow, they explore data sets, right? Or they want, they know that there is value in the data. So they want to explore that value. They want to use those insights to drive some meaningful decisions. So the way I would like to think is to categorize data scientists are the people who have this formal education of ML, of stats. Whereas the citizen data scientists in my point of view are the people who are analysts, who have done something with data, but they do not necessarily are keen or build models. Or there are engineers who are data engineers or software engineers who want to be able to do ML. That’s how I would like to think about it. So now when it comes to our tooling, if you think about what we are doing is we are building reusable components. What we are trying to do is that let me give you an example, a concrete example and that that ever helped. So the way our team is working on we have built like reusable libraries, workflow components and algorithms, which are in the field of monitoring and forecasting. When I say monitoring, we are talking about time series, time series anomaly detection, change point detection, root cause analysis and time series forecasting. So when we talk about regular data scientists, they want to build these book models. Everybody wants to like quickly spin up a Jupyter notebook and then do some research, do some analysis of data, then they want to build the features, they want to train the model, deploy the model and do a bunch of things and drive the insights. Where we differentiate is our audiences, they want to do something quick, they might not have strong engineering background or data science background. But what they know is where the data is, what is they want to drive quickly using our tools and components. So for example, right, we for, let me give you an example of forecasting. So we have workplace solutions team. They want to help our associates with our hybrid work environment. So they’re using our solution to kind of forecast how many people are going to be returning to work on each day. How much the kitchen stocking has to be done. And now, so that’s kind of forecasting. But now let’s think about the other thing, which we are very proud of is the transactional fraud. So this is, for example, right, a third party, our internal team, which is a third party fraud team, they came to us. They were looking for some kind of solution to identify the anomalies in the fraud and automatically create defensive to mitigate the losses and reduce the customer friction. So they partner with our team, they come and say like, Hey, we want to do a B C D E and what we help is like we help them by creating this workflow, which is under the hood is a dad. And then what they do is they work with us. They say, this is where the data is. And then what happens? Let me give you a little bit more detail. So when a transaction is marked as fraudulent, right, it is then analyzed by our solutions in a batch mode. And then what we do is the segments with the highest anomalies are flag. So anomaly detection algorithm, they say, this is the segment with the highest anomalies. Then another set of algorithms, again, which say that, okay, this is the change point happens, starting when they start noticing the change. And then there are another, then after that, they go and see the root cause analysis. And then what happened? The automatic rules get generated, which then get applied to the real time systems to prevent the future fraud. And Sam, also if you know with any anomaly detection, one of the biggest challenges is minimizing the false positives. So we have built an intelligent there with what helps is all algorithms are open source and appropriate real algorithms. So what these algorithms do is that we have intelligence built in, which also minimized these false positives. For example, right, if I go for a dinner with my husband, right, a nice dinner, and if my credit card gets declined after that, that won’t be a good thing, right? So it’s not happy. So it’s kind of very important that we’re not just having those algorithms built in, but we are, but we are also like helping to use the results of this algorithm, help with the real time systems to mitigate future fraud and also give a good customer experience and reduce the friction because false positives are big concerns. You mentioned some things that I think of as existing in different kind of layers of a stack. You mentioned anomaly detection and forecasting as kind of these higher level, almost primitives that maybe you offer to these citizen data scientists. You also mentioned workflows. Has your team, and you mentioned as well, open source and proprietary algorithms. So as your team kind of curated a collection of these algorithms for forecasting and anomaly detection and provided a platform that allows them to create these workflows and you’re supporting them and building the applications. Is that the way to think about the way your team engages with the client? Yes, you said it very right. So let me also give you so what is important for democratizing ML and how we are helping is we are focused on the key for us is no low to no code. So what we are doing is yes, we have an internal platform which is very scalable built on Kubernetes. We have this sophisticated orchestras, we have like this sophisticated workflow and libraries built in. So yes, our the way I would say is are reusable libraries are sophisticated packaging of as you said these algorithms which are open source and proprietary algorithms which you have built over time by researching using white papers and everything. And then under the hood are the low level tags on which we have an abstraction very dark. So we help our customers through providing a UI where the customer just goes. They just say okay, hey, this is in the s3 bucket or in the snowflake or wherever many data sources we have where the data sets are. Then they provide us some parameters that okay, I want this I want to drive this and these are some of the features which we think are important for us and then this is how often the model needs to be retrained and this is the output sync whether the data has to go to an s3 bucket whether it has to go through snowflake or whichever other capital one data and capital one system data systems are. So we kind of collect all these parameters and under the hood instead of now this person building the whole ML pipeline which is very sophisticated piece of coding and not easy. So we do it under the hood for them and what happens is that we have governance and rules in place. So now what happens is because they just choose the templates they provide the parameters under the hood we have all our algorithms reviewed by our model review office. We have governance, we have everything in place. So the time to market for these solutions for our internal platforms is like very very good. And what roles do you have on your team supporting all this? Do you have everything from data engineering to ML engineering to data science? Yes, so the way I’ll just share something interesting with you, right? So for me what an ML e encapsulates like so an ML engineer is a good mix of data engineer software engineer data science and like business acumen and be able to like cloud skills skillset. So now to answer your questions our team is like data scientist software engineers and machine learning engineers. So they have all the necessary qualifications to build this end to end sophisticated system. Right, right, right. You mentioned that the platform is Kubernetes based. I recently spoke with one of your colleagues Ali Radell whose team, I think from that conversation they’re very focused on kind of low level Kubernetes based environments for machine learning there. Can you talk a little bit about how what your team does and any intersection points or how to approach your team’s take and how they differ? Of course, first of all I heard you I am as I said I’m a huge fan of your podcast. So I did listen your podcast with Ali and it was a great talk by the way. So the way I would like to answer this question is the way I think Ali’s team and our team is doing is very complimentary. What we are doing is we are building a coherent ML ecosystem at Capital One catering to all kind of personas if you will. So the way Ali’s team is working on is building a sophisticated system for the regular data scientists as is done in a lot of companies. They’re providing system for regular data scientists who want to spin a jibberer notebook, who want to do feature engineering, who want to train their model, who want to deploy the model, who want to create an artifact and deploy it, do hyper parameter tuning. And where we add value, our team is adding value is democratizing ML in the niche as I said around monitoring and forecasting. Where we add value is by building the providing reusable libraries and components and workflows. For example, right back in the day when I was a data scientist, what I would do is I would spin up a jibberer notebook or an IDE that I like. I would like to do feature engineering spend like long cycles to feature engineering and then training and then hyper parameter tuning and then creating an artifact creating a Docker image and then deploy it. I think you mentioned bespoke earlier like a very handcrafted process of creating this model. Yes, which is like solving a particular problem, but what we do is our audiences are the people who want to drive quick insights. They don’t necessarily care about building a model. They are looking at, okay, can somebody put a solution for us where we can minimize the transactional fault? Can somebody help us like where we can detect anomalies in this system? So this is where we differentiate. And are the users of your platform are there? So they build these models using the algorithms that you provide and the reusable components, as you mentioned. Is there goal to, you know, production allize them in the same way as the traditional data scientists? Like we think about, you know, the traditional data science or machine learning. Like you want to eventually get a model into production, you know, maybe behind some API or even if it’s batch, but like it’s in production. Whereas more of an analyst role, you know, they want to get the insight. Maybe it’s to regenerate a report as opposed to have a model in a production. Curious how is that distinction significant to your users? It is because at the end of it, they want their own API that they want to hit to drive the insight. So, yes. So there is this desire for productionization of what they’re doing on your platform? Yes. I’m, yeah, we have like many productionized use cases. So whatever examples I was giving you before about the third party card fraud or the workplace team, those are two of the many production use cases that we have. At the end of it, yes, there is this API that they hit or it is hit automatically by what we have built in the pipeline, which is kind of giving them the predictions or that they then use or surface of the stack for the business to make necessary decisions. Okay. Okay. So in a sense, just because they’re these citizens, they’re scientists, they still, they still need a fairly sophisticated set of tooling to support what they want to do because they want to do a lot of the same things. They just don’t necessarily have the expertise or the desire to tweak all the parameters that the traditional data scientists might have. Actually, they need more sophisticated because now the onus is on our team because the contribution from the citizen data scientists, they don’t know if they want to use the dice coefficient versus the binary coefficient, right? So the onus is on us to kind of build that sophisticated offering for them that they get the results if not better than same as if somebody from their team would have created this handcrafted model for them. Got it. Got it. Yeah. Yeah. You mentioned in talking about the fraud aspects of both batch and real time. And you know, when you talk about monitoring and anomaly detection, real time comes to mind there as well. Can you talk a little bit about real time requirements for the kinds of things that your users do and how you accommodate that? Sure. So I’ll give you an example in general also, first of all. So you know how the technology is changing correct. And I represent an average internet user. So if I’m on a website, I don’t get what I need to see in a couple of minutes I believe. And some, we know that there are so many platforms, there are so many e-commerce systems. So machine learning has to be fast. And what that, it has to be fast. For example, right, if I’m on a e-commerce website, right, I need, it needs to tell me like how do they differentiate their products? There are so many vendors on that one, right? And then if AI is an ML is getting smarter, right? If I’m leaving the website, they know they want to intervene to keep me there. They want to upsell a product. So what’s needed is something that is very, very fast. So the way models have to be, it’s not only that they need to be statistically strong models. They need to be fast also. And what happens is like many a times I see that and even when I started in my ML journey, the focus was mostly on building the sophisticated models, right? But a lot of focus doesn’t go into how to make them optimized in the sense like fast. And also it’s a whole stack if you think about it, right? So you have a model which is fast. Then you have a platform which is able to like surface the anomalies or surface the outputs of the model fast. Then another interesting thing that happens is that these things are ultimately like, but if you see the entire stack, you might have a UI layer, you might have a business layer, you might have different layers which are owned by different teams. So you need to have like an overall system which deals with the TPS 99 SLAs and everything so that the time in which the whole stack responds in like few milliseconds. So what am I trying to say here is that over time and coming from the engineering background, the way I like to see a model as we talk a little bit before also is like a piece of software or an API that’s on production. So people need to focus on it. If it’s just a piece of software, right? Forget about the model of my lifecycle. They should at least think about it as a software doll of my lifecycle and it should go through the same engineering and operational radar. Real time makes it more interesting because now we are talking about distributed systems. We are talking about TPS 99. We are talking about SLAs. We are talking about the cost effective cost usage. So because there are so many things in the stack and they are so interdependent, so real time becomes kind of like challenge these days. So what we are doing is in our system is that most of our use cases are batch models but there are few which are real time systems. So what we are doing is like we have put in, so if the data is coming right in the magnitudes of gigabytes or something. This is like mostly for training but when it comes to like real time, we are trying to listen to the event bus so that we can listen to the, if some of our features are coming through the event bus, right? We need to factor that and we need to have our model respond in microseconds or like 100 milliseconds or something. And then we need to make sure that our dad, we are doing like parallelization of systems. We are having our DAX like work in a way that we can parallelize depending on how the feature vectors are created and everything. And then so a lot of focus is right now on across the not just like the model has to be good or the algorithm has to be fast but the whole pipelining that we are doing, it has to be like super fast. And also the teams who are receiving it, they have to work with the same rigor as we are working if they want to have like within product intervention. Are you doing much with serverless technologies for inference? We are doing a lot of work which is serverless but a lot of work which is our own proprietary internal stuff. Okay, okay. And you mentioned event bus so the idea is that you’ve got some model server that is listening for events on an event bus and then hands out to some model for inference when it sees the appropriate things. There are two ways to do that actually. So we can read through the topics in real time. Our second thing is also that’s happening is that if a team has a, our partner teams have sophisticated feature platform. So what happens is that they can read from the topic and put it in the feature store from where we can listen to those features or we can gather those features on the feature store. So both works. Okay. And so I’m trying to get at other, I guess other challenges that you run into when you’re supporting these real time use cases. Sounds like real time in general is something that you expect to grow there. You’re doing a lot of batch but you’re expecting to see more real time over time. What else do you anticipate having overcome as you take on more of those use cases? So what we’re also going to do is that for real time systems, right, we need to have more control on our overall deployment our workflows, right? So what we are also doing is not just for real time but what we are also trying to do is as I said, we are trying to centralize the build and centralize ML ecosystem at capital one. Right. So not necessarily what my team is doing and what Ali’s team is doing. I see at some point we are going to work together on building something like a more sophisticated real time serving, more sophisticated integrated batch serving. So we are coming together. We are trying to do the best practices around like federated learning, automated learning. So as the system is learning, it’s training automatically and then it’s providing better like better outcomes or better predictions. So we are on that journey. There are a lot of things that we are working on and I’m pretty sure that we will be able to be almost real time. I don’t think any company is like fully real time but that’s just me. Sure. It’s always latency somewhere. Yes. And introducing yourself, you talked about a pretty wide variety of organization types that you worked at. Can you talk a little bit about how tackling the kinds of challenges you are discussing here and the use cases that you are discussing here are different at a financial services firm like capital one relative to more of the tech oriented firm, start up. What are some of the things that you need to think about that you haven’t had to think about it other places? Yeah. Very interesting questions. Let me tell you. So as I told you right, I grew from full stack in general to now being a leader in the ML space. So I’ve worked in companies which are like startups to companies which are like capital one who are like highly regulated or into it. So the way I would like to answer this question isn’t two ways. So if you are working in a startup or in any company who are in its initial ML journey, you don’t think about governance or regulation or compute cost. You basically have a whole lot of rules in place. So what you do is like, oh, I need this. Oh, let me get this data set and let me just quickly build something. So you feel like very empowered. You build something quick and snappy. But then when you grow in your ML maturity, that’s when you see like, oh my god, so many issues start surfacing up. Now your cloud pass are out of order. You have like a data scientist who spun off a big cluster of the three GPUs to train a model and forgot to shut it. Right. And then multiple teams who are doing that. Right. So what happens is like your cloud cars start going up and then you end up seeing that different teams. There are 10 teams in the company and they all have their individual ML platforms or pipelines. So then the leadership has to think about and then there is no governance. They are not thinking about the PII SOX compliance, GDPR, CCPA. As you said, I’ve written regulated companies. I can tell you all the names. Right. So then those things kick in. Right. So that’s why so there is this thing that we’re now eventually there is a tech that’s created and then the leadership have to make some RDoS calls around like, hey, we need to throw away work or we need to merge or we need to converge. But now working in companies which are in the ML maturity space, right. And especially the regulated environments. What I’ve seen being a machine learning engineering leader is very important to standardize the tools. It’s very neat. It’s very important not just the tools, your processes, your algorithms. Right. What this does is like now these companies are big companies which have a lot of ML associates data scientist MLs. But what it allows it, it helps them to come with a centralized process where they can identify. They have mechanism. They know how the ingestion pipelines are working. They know how what’s the best way of getting the data. They are scripts or there are like scripts in place where you just cannot spin off an EC2 by going to your control plane. You need to follow some rules and practices. Right. And then as I’m talking about governance and auditing. So what happens in like companies like Capraone which are highly regulated, you need to be able to store your engineering, your training runs, the input and output parameters that were part of the model. Right. So that if needed you are able to recreate the model. You need to be able to tell the customer that if if you get audited why such a decision was taken from them. And then it’s important that you have governance like at Capraone we have our model review office. Right. So it’s very so that’s where I see the difference. But again, I’m not going to shy away from saying that like all these things add like add like a lot of processes in place which sometimes people don’t like. And what happens is like these procedures and practices in place it at time create a lot of dependencies. And then like your time to market or your time to go to production like slows down. But honestly on this one, Sam trust me on this one. It’s much better to be a little bit late than to go to production and then have to deal with the production on compliance issues. It’s it’s not fun to do. Meaning it in your experience it’s better to have those processes integrated into the development of your model as opposed to trying to bolt them on at the end. Yes. Yes. And also please follow the processes because everything looks nice. But if you ever get added because your model behaves a certain way or you have to go through a compliance issue that takes like weeks or months to work on that. You mentioned this the need for reproducibility and the idea that you’re taking all of your training inputs and training data and kind of storing that away so that you can reproduce these models. Is that based on internally developed technology or do you use some external some third party tool to provide for that? No, internal build technology. Internal build technology I can talk a little bit more about it. So in storage costs are not that very, storage is cheap. So without taking name of the company I’ve worked in various companies like where logging is our instrumentation is very very important. You need to instrument what are the features coming and you need to instrument what is the model’s output is and then other things which go in the part of like reprocessing a feature engineering so that you have all that data. So now if you get a customer call saying that hey you said this and I went through this and then maybe that’s not the right decision for that customer then you need to be able to go and see exactly what happened and it helps you by for answering the question of the customer but also helps you reverse engineer better about what went into the what went into it and you know like how expandability and responsible AI and all those things are important. So a good instrumentation of the whole how much ever you can instrument I think it’s never enough. That’s just me. That’s my engineering mindset. On the topic of engineering mindset you kind of alluded to a level of robustness that’s required which makes me think about like testing and the importance of that and that seems like something that over the past I guess year 18 months like we’ve kind of gotten to the state in machine learning where we’re you know trying to apply the same level of rigor that we’ve had for traditional software to model testing you know have whole companies set up now around model observability as one kind of expression of of this desire for testing. Can you talk a little bit about how you approach that for your platforms and teams? Sure so what I want to start by saying is that what I alluded before also so model when it’s on production it’s it should be treated as a piece of software or an API but with more sophistication now because there is the involvement of the data piece right. Today even today even though you’re saying we we are thinking about like okay MLabs and testing and scalable models still I personally feel a big chunk of time goes into like creating a sophisticated model which has like a high statistical efficacy. You want to spend rounds of cycles doing hyper-valent or do you make feature engineering and everything. I think defensive coding and exception handling they are key right there if if anything else if the model is a piece of software it needs to go through the same rigor right it needs to have like end to end testing integration testing unit testing load testing a b testing but what makes a model special is because there is this data component so data quality testing is the key and you know what it’s interesting as data scientists and machine learning engineers sometimes we think oh I’ve spent this humongous cycle of building these features and I’ve dealt with everything now my features are good but that’s where I think the mistake happens because see we are the one who are closest to the data the data scientists and ML engineers so I think the owners is on them to write as part of their code around the defensive coding or data quality test there is a high ROI in deficit coding right or preventive coding as I mentioned different types of coding I think data quality testing is the key for example right some of the data quality testing that we are doing and then I think is value add is checking the data type and the values of categorical data another one interesting one is the categorical is the cardinality shift right where there is a sudden shift in the distribution of categories and bam your model is predicting like it has predicting a particular category data leakage and drift over time and something which is very close to my heart is missing data there are so many papers around how you want to impute data and then what are the techniques well particularly for forecasting and anomaly detection that’s going to be a big deal for you yes but then also we need to know where we draw the line let me talk a little bit more about it right so some you know the feature vectors these days are like highly complicated they are based on myriad data sources right so I think we need to have some guard rails and thresholds on how much in which features should be allowed with missing data how much can be imputed and what we should not be imputing so that’s where the responsibility part comes in also so and at what point the model should just like raise an exception and be like hey this is not adequate for me to make the predictions meaning what I’m hearing there is it sounds like you you’re speaking to examples where okay you you you kind of do the thing that you learn in school right you have missing data we’re going to impute it but if you don’t have if you’re not paying too much attention you could be imputing half of your data and you you really don’t know what your model is basing its decision known it’s the signal the noise is not high enough yes exactly that’s what I’m saying like data science is a privilege but it’s a responsibility also so we need to be able to draw a line around what can or cannot be imputed I’ll give you a very simple example and the reason I’m giving you this example because I have carried multiple times at different places right so I’ll you know demographic models we build a lot of demographic models and most of the times zipboard is one of the important features right from which you can drive you can have like okay short distance from this place long distance or get bunch of other demographic data I’ve seen many a times the model is expecting a five digit zipboard but sometimes the users if they it’s a free text they pass in a nine digit and then the mark phase so what I’m suggesting the people is like hey it’s only one or two lines of this extra code which should be part of the model on the key processing code and it can just save us like bunch of trouble for our use cases right Sam as I said like we we focus a lot on centralizing it engineering and operational regular code standards across chapter one so we have standards for coding we do peer review we do unit testing integration testing end-to-end testing cross validation and like every other company our data quality testing is evolving too so some of the best industry practices that I’ve seen is like creating synthetic synthetic or world and data sets right and then you revisit them when you find the edge cases other thing we do is like scenario based testing to mitigate output or surprises are recently I was also reading about how we people do randomizing or first testing in software engineering that is also becoming becoming like important in the data science or ML world yes so those are some of the best practices we follow and thinking back to kind of your role in enabling these these citizen data scientists imagining that these are things that you want your team thinking about and building guardrails around or implementing but not necessarily you know that’s a lot of cognitive burden to put on a citizen data scientist to think about you know this and that not impute too much data and that kind of thing like are they things that you’re able to kind of isolate as concerns of your platform as opposed to concerns of the end user so what we do is as you said we have Godreels built in place right so where we are today we are doing UI and API based innovation but where we want to be is that which we are working towards is like the user just goes on the API go on the UI just say this is my data press a button and after that we take care of everything right so at every pace we want a Godreel for example let’s just say I’m just going to give an example right let’s just say this user is asking us to frame the model with like some 5060 gigabyte worth of data right so we want to be able to do some testing we cannot do like this whole kind everything but we want to have some and we want the user to tell us like okay for think about like as a regular business use case right in the B-Scope world what happens is the business intelligence the business analyst or the product sees something is happening on a specific page or something is happening they think oh there is definitely a signal let’s go to the data science team now and see if they we can have like an ML intervention or like a date or an insight built in during this journey similarly when people come to us they have already seen oh there has been this fraud or hey how they are being using something based off Excel for the associate student to work the sorry in the hybrid work environment right so what’s happening there right so they have some signals so our ex our hope is like we’ll be at a place where they just go to our UI they say where the data is which are some of the critical features that they need and then we have some reg is based on like our learnings as we are learning every day we go and like okay for this particular feature these are important we put the guardrails that okay this kind of data quality checks are in place this much data needs to be there especially as you said anomaly like time series right what can be imputed what’s the best way of imputing we do some kind of data cleaning wrangling show the features then have the user take a look at it and say like okay press the next button we’re training happens then they deploy whether they want to go on batch versus that we are time so that’s the end state that we are working on can you talk a little bit about you know we’ve talked about use cases and kind of technology thus far you’re in this position where you see all that but you’re also kind of looking the other direction in the organization and towards the business side of things can you talk a little bit about how you think about kind of ROI of machine learning and getting buy-in from executives and you know all the things that you need to think about as a ML leader sure so let me tell you first what is working for us right so when we complete a project for our internal user right internal client right we create detailed documentation around what the problem statement was what were the challenges that the team was facing what is the solution that we provided what were the results of that solution and then what’s a value proposition like what’s a nibbett or like dollars we are helping in like operational efficiencies and then we ask them as they feel okay to write some testimonials and then what are the next steps right so there’s also our learning like what really went well what would be done to get better next time right and then what we do is like we promote our wins with our stakeholders and leaders across the enterprise and then personally as a leader once once we have like some proof points in terms of like ROI and better user experience it keeps the leadership engaged and that also helps us like okay now that we are preventing X million dollars here oh it means like if we can invest more in here to like tackle similar kind of use cases etc so at capital one what we have think what we see is that ROI can be not just money it’s like I personally think it’s three things it’s like improving user experience generating operational efficiencies and driving the top line so those are some of the things because not every model will give you some money directly it could be like oh because we have done that so our analysis time has decreased from a month to now like two days or something right so those are the ways so not everything we can put a dollar back directly and so have you you know through your track record have you demonstrated kind of consistent ROI such that you know the only thing you know you you would do 10 times as many projects if you had enough people to do them like is it is talent the constraint or you know you’re still buying for funding relative to other potential efforts or uses of a resource at the company our team has established ourselves so we are helping drive the ROI by helping other teams operational efficiencies and like better user experience so our leadership like trust us and then we haven’t hit like the road rock where we go and say like we need some resources they understand that but what we also need to do is like we need to have a good justification hey we have done this we are saving this x million dollars if we get these two other folks to help us out we can bring in another five to ten use cases of similar thing because and you’ll be like hey the shawaier you’re saying two people and five to ten use cases because I want to remind you like what we do is we do it in a reusable fashion so yes so that’s the last and so does that is your implication that you’re not so concerned about talent there you you okay no no no what I mean is this see top talent in industry especially in ml is it’s not easy to get it so you need to be able to continuously motivate them right give them work which is challenging let them like people want to work even I want to work on things which I know is impacting my customers right so some challenging statement so what we do is we continue to grow our engineers our ml folks our data scientists right and also there we what we do is like as we are on this cutting edge or bleeding edge of ml right as we are improving our stack what we continue to do is like we invest in our in our in our associates we train them we retrain them right and also we have like a strong pipeline of external candidates for which we have like a very rigorous process built in and then I think you have talked to Bian and Ali before and they might have also talked to you about like how capper one is so invested in like working with universities like MIT UBA and everybody so we have like and especially when it comes to ml right I’m very impressed with all the efforts we are putting in we have like this ml training program we have a product manager ml training program we have like a PhD program so capper one is doing a lot to keep its associates on this beating edge of chemical technology and also we have this DEI like diversity inclusion and belonging kind of culture where we are also trying to attract up talent from the industry but on those on those principles and to be clear my implication wasn’t that you didn’t care about your talent or anything like that Osmoor was the degree to which talent was a constraint for your particular team ml is interesting because it’s not like simple software simple engineering where you learn something new packages like honestly Sam I try to keep myself up with technology by the time I I like carve out time to do something and I’m like yes this weekend I’m going to study that and then then a new technology or new thing is out right I was looking at this 2022 Gartner a paper where they were talking about some friends and everything and I just wrote down like five or six things that I’m going to study in next one or two months and I was then I started talking to my colleagues at capper one and I was like very impressed by some of the things are team and they are already doing which is on the Gartner next to two five years so there is in ml you need to keep yourself like always learning and growing it’s tough moves quickly it’s fun how do you think the way that machine learning is approached there at capital one will evolve over the next I don’t know three years so it kind of goes into what I was saying like the 2022 Gartner study with the hype cycle for data science and machine learning so I as I mentioned I was going through that and what it suggests companies or industries are going to be moving in next two to five years and I’m very pleased to see capper one is pursuing that earlier we talked about like capper one is trying to build this ml coherent ml ecosystem right and what we are trying to do is that how we talked about my peers team and I team we’re trying to serve all personas if you will be citizen data scientist traditional data scientist right and what we are trying to do is we are trying to centralize our platforms we’re trying to democratize we are trying to do reusability of libraries components right so what we are trying to do is like ml available accessible and leverageable for every for all with necessary guardrails and responsibility in place that’s where I see we are growing there is like a lot of emphasis on overall like redo automating the overall machine learning development like cycle vice building robust pipelines for ingestion data ingestion sophisticated feature platforms training and execution platform I’m thinking all the things my peers are doing and my team is saying so I’m thinking that the c i c c i c d platforms the feature and model monitoring right we are investing a lot into ml ops extensively ml observability another interesting thing is like provisioning clean data and govern access to data are important these days cost optimizations scalable multi tenancy platform as you said and like multi region so stabilizing our platforms we are also researching a lot of things around graph ml synthetic data set which I mentioned before ml ops model ops transfer and federated learning and then because we are in governance we are in the governance it’s I would want to say again and again like we have a model review office and then we continue to invest in our standards for governance transparency and explainability for models so I see like capper one is on the beating edge of ml and I’m very proud to be here and I’m very impressed by the work my particular organization and all my people’s organization so I think it sounds like you got a lot going on it’s fun it’s fun it’s an awesome journey we are on awesome awesome with this you thanks so much for taking a time to share with us a little bit about your team and what you’re up to thank you thank you.