Video: Machine Learning Deployment: Eric Siegel and The AI Playbook

Hello and welcome.

I am Steven Ramirez.

I’m the CEO of Beyond the Arc.

And, we are, really happy today to be hosting, Eric Siegel, to talk about his new book, the AI Playbook.

I just want to thank everyone for joining us.

We have people tuning in.

This is going out live over LinkedIn.

So thank you very much all of you for joining us there. broadcast is also live on Twitter and on Facebook.

And just some housekeeping.

I want to let that, you are able to, and we’re welcoming you to add some comments and questions as you go along. you should be able to add a comment just directly on the event page, and we should be able to see it and respond, which we will. without any further ado, let me just give you a little bit of, well, I guess it’s a little bit of a, okay, it’s going to be a little bit of a ado, a little bit of a setup here just in terms of why we’re having this conversation today. so obviously there’s not a topic perhaps that’s more important, to business and technology than AI. it is really center stage of organizations as they try to, develop policy guidance and then practical implementation. how do you actually, how do you actually make it work?

And over the course of the last, five or 10 years, we’ve went from a point of, everybody sort of wandering around in the dark, trying to figure out what are the best practices to, I think now being a, they’re very clearly established sort of protocols that can actually lead to the success of AI. but unfortunately, not all the organizations have learned those lessons, and there’s still, there’s still some huge gaps. so I think we have a lot to talk about in terms of what makes AI successful.

And in this next 30 minutes, that’s exactly where we’re going to go.

All right, so with that, Eric, if you don’t mind just doing a brief, introduction.

Let’s, start there.

Sure.

Eric Siegel, I’ve been in machine learning for more than 30 years, a former Columbia University professor where I taught the grad courses in AI machine learning.

And I’ve been an independent consultant for 20 years and running the, long running machine learning week conference series that Steven’s involved in.

And hopefully we’ll talk a bit about that first week of June in Phoenix.

And I’m the author of the new book, the AI Playbook, which follows up on my previous book, predictive Analytics that covers how machine learning works.

This is on how to capitalize on it.

So as Steven mentioned, what’s the protocol?

What’s the maneuver?

What’s the organizational method?

The full title, the AI Playbook. mastering the rare Art of Machine Learning deployment because indeed, unfortunately, it still is a relatively rare art to successfully get that last mile of a machine learning project, to the point where it deploys and therefore actually captures value.

So, Eric, I sometimes feel with AI that, it’s now gotten to the point where it just seems so simple, why is, isn’t it really straightforward?

Why do companies need to worry about deployment? well, in a nutshell, you need to worry about deployment because it’s a business project.

That means business operations are going to get changed.

And that’s a big deal.

It’s not just a technology project.

The core rocket science is the coolest of science and technology ever, in my humble opinion.

Learning from data to predict.

That’s why most of us got into this, why I did more than 30 years ago. most of us data scientists, but when it comes to actually, getting it deployed, implemented, put into production operationalized so that it changes operations in order to improve them, that’s a whole nother ball game.

And that’s the kind of project, process that is not widely understood and adopted.

It turns out you really need a very particular specialized business practice paradigm playbook in order to get machine learning projects to run successfully through to deployment, reverse planning for that operationalization and deeply collaborating with business side stakeholders.

Yeah, and Eric, I think that is actually pretty deep, right?

I think that, I think that for many years, this, this field of, maybe analytics, predictive analytics, nalytics AI, machine learning, I would say, I think it’s really sort of seen as a technology endeavor, but I think that you are arguing that it’s way more than just the technology.

Yeah, I mean that’s, it’s, it’s a reframing, I’m kind of calling for a reframing when we call a machine learning project, it kind of undersells the point.

It’s a business project, an operations improvement project that uses necessarily machine learning as a key component.

But first and foremost, it’s a business project.

So with that reframing instead of sort of being in loved with this incredible rocket science, right, which we all are, we’re kind of fetishizing it’s the best of technology, it’s amazing, it’s so potent, but there’s a difference between, generating value and capturing it and realizing it by way of that actual deployment where the operations are being changed by way of the predictions, the probabilities that are output by the machine learning model.

And that’s the, that’s the actionable, output you get from machine learning.

Are these predictions, that’s what it comes down to.

Now, are you going to use them?

Are you going to operate on them?

So are there a couple of case studies that have come from the book that you think really stand out?

Yeah. the, there’s a couple from UPS, FICO and a couple prominent dot coms.

So let me tell tell you about, UPS.

Here’s a perfect example.

We have this longstanding organization now more than a hundred years old, well entrenched in their processes.

If they’re going to improve, they’re de their delivery of 16 million packages a day in the us.

Wow.

That means a change to a massive process.

And, a fundamental change to an entrenched process.

They predict tomorrow’s deliveries in order to optimize ’em.

This is the last mile or last several miles from shipping centers in the trucks to their desk to their final destination.

And so what they do is by predicting tomorrow’s deliveries, they can augment the list of known packages, the ones that are physically in their hands.

And it turns out there’s other ones that may be arriving or that they don’t even know about.

And there’s a certain amount of uncertainty, especially when you’ve already gotten into the evening time.

They need to fully plan and then the actual assignment of packages to trucks and then load the trucks all night long so that they’re ready for on time departures in the morning.

They have to do this based on that uncertainty.

They make it less uncertainty.

They less uncertain in the sense that they augment it with ly packages.

Now they have a more complete picture and their, their optimization system could do a much better job.

The bottom line result of getting this implemented, which took a lot of doing, of convincing both up and down the organization executives, Jack Levis was the hero of this, of this story at UPS and above him, the executives and then, and then down in the organization, where you’ve got staff workers on the loading docks being compliant to new prescribed behaviors by getting this actually implemented and getting it to work, in combination with another system that prescribes the actual driving routes. ’cause it’s one thing to say, Hey, look, now I’ve loaded the packages into this truck, for which now the truck potentially has a pretty, pretty optimal delivery route, potentially.

It’s not necessarily going to drive that route unless you really prescribe the driving directions to the instr to the driver.

That makes, made a huge difference.

You put those together and the wind, the wind is tremendous.

185 million miles a year saved by UPS $350 million and 185,000 metric tons of emissions.

And as you described that does not, what, what comes out is not machine learning or algorithms or data warehouses or, anything that sounds a technical project.

Yeah.

So yeah, Jack’s Jack Levi’s, title, he’s recently retired at UPS’s.

Title was Senior Director of Process Management.

And that, if you think that sounds boring, he, he didn’t call it a machine learning or an AI project or anything sexy that.

He called it an operations research project.

But more to the point, his title was about improving processes.

That was his job title.

It wasn’t about the technology.

It wasn’t a sol a solution looking for a problem.

It was let’s get things to work better, right?

Well, this is a tool we need because prediction’s going to help.

Great.

But it, so he was focused on the business value, and that’s sort of the whole point of the process.

I’m espousing, I should also probably mention what that process is by name Biz ml.

Biz ML is, the, is a name I’ve coined for what is, broken down into a six step organizational process in the book, spread across six main chapters of the book.

So, so let’s, let’s extend on that, just a little bit.

So this is not really about having great algorithms.

It’s not having about, great, the best data. how do you, so, so what are the, there’s probably a long list, but, but what are the keys of suc keys to success then? so what, what is it that you really need for this successful, AI project?

Well, I mean the algorithms certainly matter. although there’s probably usually diminishing returns as they get more complex, more important is probably the data.

But outside of the technology, which generally is relatively sound, these things work in the sense that they make models that predict and pertain to new cases never before seen.

And in that sense, have drawn true generalizations from the limited, even, even if it’s a large amount of data from which it learned the training data, it’s still relatively limited number compared to the number of all possible kind of scenarios and situations that may be encountered.

So it’s drawn a generalization for real in this, and in that sense, it’s truly learned.

There’s a good reason we call it machine learning.

So that’s sort of, it can be very well defined and it does it, it does it really well.

Great.

The technology works awesome.

The data is generally sound, it’s, it’s predictive by nature.

We can, we, we can, we can leverage it.

Now we’ve got a model, what are we going to do with it? we’re going to have to change.

I mean, this is the, what we’re talking to contextualize this.

We’re, we’re focused so far in this conversation more around what we could call predictive AI or predictive analytics to differentiate from generative AI, which is the kind of technology you turn to, it’s really one of the last remaining points of differentiation to improve most any of your main large scale processes.

So when we are able to get it implemented and make the change to operations, that’s, that’s the rub making that actual implementation deployment actually changing operations with probabilities. that’s what really what we’re talking about, predictions are essentially probabilities.

Well, I think that you raised a important point here now is you sort of crossed over it into talking about, generative AI underneath the AI brella. and let me first ask you, so does the AI playbook, help you to think about generative AI?

Yeah, it, it does in, in a couple ways.

It helps clarify what, to a large degree, what AI is about. the, the, the antidote to sort of having, a shroud of, of, of vagueness around AI and the antidote to the hype is to focus on concrete value propositions.

And that’s what this book is all about. it also helps understand what that is in very specific terms, those types of predictive use cases versus generative.

Generative is, is largely out of scope of this particular, book and the process, but broadly speaking, it still pertains in the sense that you need to reverse plan for deployment.

What are the operations at the enterprise that stand to be improved?

Exactly how will this technology improve it?

Let’s plan that.

That may be sort of the culminating final step of the project, but at the beginning from the get go, let’s plan for it. and in, in biz ml, let’s literally the first of six steps is what is six steps?

Excuse me, what is step six going to entail?

How are we deploying it?

How are we improving it?

The same concept does apply for generative AI. if you’re using it to create draft writing, because you need to send, write and send out a hundred letters to customers a day or something that. who’s doing that?

What are the parameters?

Which language model you gotta try out?

What’s the metric for success?

A lot of the general concepts do apply for generative AI projects.

So, so if you think about, let’s go down that path, I mean, obviously, that I’m the co-chair of, generative AI world, a new conference to specifically looking at generative AI.

So, that I’ve got, I’ve got some, some deep interest there.

I’m wondering sort of what’s your take?

I mean, so it’s, it’s something that can be planned for something that it can be useful, but I know that you might also be a little bit of a skeptic.

So do you have a, do you, do you have a, you want to opine on that a little bit?

I’m, I’m a skeptic in the sense that I think a lot of the messages out there are hype and that they, they mismanage expectations, but that doesn’t mean that I’m a skeptic, that the stuff’s actually valuable.

I think generative AI is incredible.

I never thought I’d see something that in my lifetime.

Lemme take a step back.

The conference you just RA mentioned.

I, I want to, I want to tell our listeners what that is.

Machine learning week, which I’ve been running since 2009.

I’m the founder previously predictive analytics world, now has a new sister conference, generative AI world.

They’re both first week of June in Phoenix. and Steven has been involved with these conferences for many, for most of these years.

And, is for this two, 2024 event. the co-chair of both, he co-chaired with me machine learning week, and he is co-chairing with another chair, the, generative AI world.

So he’s got his work cut out from, we’ve already programmed machine learning week and the generative AI, list of speakers should be posted very soon. so, I spent six years in the natural language processing research group at Columbia University.

So it’s all edge cases.

I’m this is never going to scale.

I’m, I’m sick of this.

I never thought I’d see in my life what you can get from a la from a large language model.

Yes, I think it’s unbelievably amazing, the fact that it could be any topic, that it’s responds to, in a way that’s often seemingly cohesive and that it’s, kind of any topic, any topic and any turn of phrase, the way we speak in metaphors, it can handle a lot of that with a real seemingly human-kind of aura that you experience. however, as excited and amazed as I am, I kind of think the world’s, at least in order of magnitude, more excited to that than that, which means, in my opinion, that much too excited. because I think the general narrative is that because it’s so seemingly human-it means, it’s a concrete step.

So towards general human level capabilities, people call this artificial general intelligence, a computer that can do anything, a person could do, essentially, let’s call it what it is that’s an artificial human who you can onboard the same as a human employee and let it rip.

They could run a Fortune 500 company as the CEO.

I think that’s a, I don’t think that’s necessarily theoretically impossible, but I don’t think there’s any, concrete steps despite how impressive and seemingly human this is.

I don’t think pontificating on that now is much different than it was in the 1950s. but there’s a big problem when you have that kind of expectation, because if, if there’s a sense that we’re definitely moving that direction, and many even technical experts say it’s going to happen in 10 years, it’s going to happen in 30 years.

We’re, that means all of a sudden maybe we’re only already 5% there.

If you had 5% of a human, that could potentially be really valuable.

But we’re not really measuring that in concrete terms.

I don’t think we’re making a, a a credible case. and I think there’s a real problem with hype mismanaged expectations.

There’s going to be disillusionment, the bath, baby goes out with a bath water.

This is really important.

The antidote is to focus on concrete use cases and value propositions for the organization.

That’s where we kind of weed out the fluff and get actual value.

Yeah.

Eric, I think that is something that has always attracted me to machine learning week.

So, I, I believe in our prior conversations, I found out, I think I attended the first one.

So back in San Francisco.

Oh yeah, I think, I think I was there, I think I was a speaker at, at, at, at conference number one. and one of the reasons why I, remained engaged and really, felt part of this, event is because I feel there’s a community.

I feel there’s people at Machine Learning Week who are really working to solve business problems with technology, and particularly with machine learning.

And that has now been a, many year, many year journey.

And that is what I still get out of the, out of the conference even as a, as a co-chair, is the ability to hear from, people both, sort of on the ground, sometimes, and that’s both sort of practitioners.

So data scientists who are, programming the algorithms, but as well as people who are leading those teams, and to hear from them the business challenge and how they’re thinking about machine learning to be able to address, I, I feel that I’m, I’m really excited looking forward to this June event, about how that conversation will grow to encompass, generative AI.

Because as you said, I think that there’s a, there’s a ton of, there’s a ton of, of hype, but there’s also also a ton of real work that is happening in, in organizations to implement AI. and the, and the essence of that is really being driven by generative AI.

And so, so we ourselves, I mean, we’re working with clients right now to be able to use generative AI technologies to really to, save time, and to be able to, in, in some cases, review information that just would take you, weeks, months, years to be able to really properly synthesize that generative AI can really do that very, very seamlessly.

So I really see this whole, that generative AI is kind of, is really sort of this next evolution of where AI is going, but I think that it is going to be, there are going to be these challenges about, how do you actually implement, right?

So I think that right now, there’s a lot that’s happening at the proof of concept level, but generative AI is going to follow the same path as other types of AI implementations, and ly it’s going to, it’s going to have the same kinds of roadblocks.

That’s at least my, that’s at least my, that’s my hypothesis. what’s your, what’s your take on that?

Yeah, I think the kind, I mean, that’s sort of the purpose of the conference is to both celebrate and bring forward all the successes, and then also talk about all the hard lessons learned and what’s been learned in the best practices. we, unlike most data science conferences, it’s not just the technical stuff, the entirety of track one of machine learning of the machine learning week part is on that business operationalization.

Now, we’re also using the word biz ML in the name of that track.

And a lot of the keynotes are about these case studies that cover in the book.

In fact, in addition to my, keynote at, at the opening of the conference, we have Scott Zdi, the chief analytics officer from Fi CO who’s whose story is featured in, in my new book, Jack Leva, who I mentioned at UPS also as a keynote, and Morgan Vater, who’s the global VP analytics at Unilever.

And she provided a forward for the book.

And they’re all actually keynoting.

We have an expert panel on the business process, but there’s plenty of other kind of case studies and, technology focused tracks. so, right, so that is to say machine learning, doesn’t have, it’s not stuck in, it’s not stuck against a wall.

There’s lots of success cases, but it turns out that most new, especially particularly innovative machine learning projects actually fail to reach deployment.

So we need to increase the track record of success and get those prac track, business practices, well understood in general business practices, business professionals, non-data scientists aren’t really even a rare aware of the need for a very particular customized, business practice in the first place, let alone the name of name of any particular one.

So that’s what I’m trying to do here with the biz ml buzzword is say, Hey, look, let’s brand this thing so people kind of catch on and see that it’s a thing that they should know what it is. and it, it’s not the rocket science part.

In fact, what it, it gets tries to get business professionals involved in the semi-technical aspect, and it lays that out, which is what’s predicted how well what’s done about it.

So not the rocket science, but how to use it.

I think it’s really important to get there.

I mean, I’d love to, if we had more time, I’d love to get in maybe for a minute now with you what you said about gen AI being able to synthesize information.

I have a question.

I know we also want to leave time for listener questions, but maybe we can spend another minute on that so that, I think that kind of brings up the question between reality and hype, and there’s maybe a spectrum, if it’s synthesizing, let’s say I use gen AI, a large language model, large language model, to summarize what should be salient for my purposes over a large number of documents, how do I trust it?

Right?

So that’s the thing. you don’t know what you don’t know.

If you already knew it, you don’t need something else telling you.

And if you don’t know, how do you, so there, the thing is, is that these language models, since they operate on essentially the per word level, they have a human-aura.

They, they make things that sound human-but they, the core, foundation models themselves are, are not designed to meet higher order human goals, being correct.

That’s an, that’s an outstanding research process.

So this idea of sym synthesizing information, I’m sure there’s places where it could be valuable, especially where there’s leniency for errors as there is with predictive AI.

There’s, if, if it says this, if it blocks a transaction as fraud, but it was actually a legit, that’s an error, but one for which there can be leniency as long as it’s tipping the odds and doing that less than a previous method, right?

That’s the win. mm-Hmm.

I don’t know, with gen AI, it tends to be trying to do things that humans do where there’s less leniency for, for that kind of error.

What, what do you think?

I mean, and maybe this is a big part of what you, you’re going to explore?

Yeah, no, I think, no, I think that this is a, I think this is definitely an issue and I think it, it gets to how you design the application or how you design your solution to address these kinds of issues.

Because there, there are, there is just sort of the inherent possibility of, of incorrect, incorrect hallucinations.

I’ll just tell you just a couple of the methods that we’re exploring.

So, so one is, we started with, just kind of a sample test bed. we actually had a human do the task and, and then create, take a set of, of, of text from different documents and do a, and do a, kind of a gold standard test of we think this is a good summary, so that we had something to compare against when we were getting summaries from the, from generative AI for us to say, does this even make sense?

Right?

So, so there was kind of a, and we did that just to be able to, just kind of reality, reality check. and then the other, thing that we’ve done is, is to really think about the design as a human in the middle process.

And so what, what we’re looking at first in this, in these initial steps is, can we, can we identify documents or parts of documents that we think might be of interest to you?

So, so you still make the decision as the user, if this is actually, if it’s actually relevant. so are you, is it actually meeting your needs?

And so we’re giving you more of a menu to choose from for, but for exactly that same reason that we don’t feel we can necessarily rely on a single, on a single answer, right?

Mm-Hmm.

So, so trying to, trying to apply some, I’ll say some data science thinking, to the process of, generative AI development and realizing that it’s all in a very, early stage, early stage of conceptualization.

Yeah.

And I think that I am very optimistic that within well honed topic areas, the history and making of wine, and I’m not a wine expert at all, but take that expertise or a chat bot meant to serve, people on the, in the field fixing washing machines, where in both of these cases, you can, chat in a human-manner, use any kind of expressions in terms of phrase that you might use with a human.

And because it’s limited in scope, in terms of knowledge base and being able to measure that it’s correct when it most of the time, and that it knows the correct answer as often as we would hope from the human expert.

And it’s with that limited scope, I see that as a con, as a concrete and plausible research area where within years, those higher order goals of, of really making sure that it’s reliable and trustable, for broader use, where it could be any topic area, I see that as a much hairier area area and then starts to get to the A GI thing.

So defining the scope, I think is a, is a critical part of it.

A absolutely, and I would say just broader than generative AI, it really goes to, well, I’ll just call sort of anything that is black box AI, right?

If you don’t know how the, how the decision is being reached and you don’t have any transparency, I, I think that does lead to some just inherent just some inherent questions.

And so I do think that, and I, and I know that at the, in, in prior, public addresses and things that I’ve seen, I know that Scott Aldi from from FICOs talk talks about this quite a bit in terms of how do we ensure that AI is trustable?

So look forward to hearing his, his latest talk in this, in this context. but I think it’s something as a field that we have to address, right? as the, as you now are looking at models that have, 80 billion parameters, how do you, how do you sort of trace back and have some, some idea of the, of the outcome? so, so Eric, we’re coming down to our, last few minutes.

I, I had asked our team, beyond the arc, if they had some questions. and so, one question I want to, talk about is this issue about scalability. so let me share with you this question from our team. in what ways does Biz ML contribute to the scalability and wider adoption of machine learning initiatives?

So let’s call it, let’s say adoption is the actual deployment part.

So it’s one thing just to adopt a technology in the sense of green lighting a project.

So the data scientists start to crunch the numbers.

Then once they’ve had a model that could potentially help operations, is it going to go to deployment?

Is it going to be adopted in the full sense of the word?

And the syndrome is that these models are generated with the intention of being deployed, and then they don’t get deployed.

And that’s because the stakeholders get cold feet, and that’s because their hands, if your hands don’t get dirty, then your feet get cold.

And we need to involve them from get end to end of the project and make sure that, this, that actual deployment and change to business operations is very concretely and thoroughly planned for from the get go.

And that along the whole project, we’ve got the business stakeholders in place.

What do we need to make that to happen, to get that as a more, much more common practice than it is now?

And to get us outta that syndrome, that reoccurs so routinely, we, we need, a general understanding that, hey, look, there’s a life cycle, six steps that include, prep the data, train the model, and deploy it.

And those are the three culminating steps.

And then there’s pre-production steps that involves, planning that deployment, what’s predicted and what’s done about it.

That’s what defines, the use case.

And then get much more specific about the what’s predicted part, what’s this model meant to predict For data scientists, it’s the definition of the dependent variable, but there’s a lot of caveats and qualifiers coming from the business side.

We need to get the stakeholders involved so they understand exactly what’s predicted, because that’s the thing, you’re going to be acting on those predictions.

And then finally, metrics.

How good is it?

How well does it work both technically in pure predictive performance and in terms of business metrics profit and ROI in terms of how well it’s going to serve the business, what kind of returns are you going to get depending on how you actually deploy this model so that the stakeholders can get an estimate of what that looks and make an informed decision and not get the cold feet an informed decision to authorize the deployment.

So that’s what, by branding it as biz ml, we’re saying, Hey, look, there’s a lifecycle.

Everyone should be familiar with it. paradigm playbook, what everyone call it.

And the business side, people need to ramp up, need to get their hands dirty, need to ramp up on that kind of semi-technical understanding.

And then, and by doing so, they’ve prepared themselves more than anything else, successful machine learning doesn’t need more technology, but needs you as a business side stakeholder.

Once you’ve ramped up on that semi-technical understanding, that’s the missing ingredient, that participation in that deep collaboration.

Excellent.

Well, Eric, I think that we should probably, leave our conversation there, but, before we wrap, I mean, the book is the AI playbook.

Where do people find it? biz ml.com is the website for the book.

Excellent.

And then if they’re interested in learning more about the upcoming conferences, what’s, where should they go?

It’s, machine learning week.com. and, and the other conference that you’re also co-chairing generative AI world is, is a generative AI events.

I know we have a few domains.

Yes, I think so.

AI events.

Yeah.

Excellent.

Hey, well, Eric, thanks a lot.

Definitely have enjoyed this conversation.

I want to thank everybody for joining us and, really appreciate the time today. hope everybody has a great afternoon.

Thank you.

Great.

Thanks Steven.