(01:15) can kind of hear that um so today’s agenda we’ll jump right into this um is going to be um you know based on generative AI of course everybody knows this is a Hot Topic uh things that we continue to hear about ourselves and so the the title is demystifying generative Ai and the reason for that is is because we want to be able to kind of get through a lot of the noise that’s out there and you really get down to the points for real businesses small mid-market wherever your business is the things that are going to really affect

(01:45) you so that’s going to be the focus and the reason why we brought Ken and Rob in today and we’ll introduce him here in a minute um to be able to kind of have that real business level discussion we’re not going deep into Tech here this is not meant to be a tech talk this is you guys have seen some of the promotional stuff this is this is for more of an executive View and strategy on how these things are used and again like we mentioned before we’ll have some panel discussion questions with with Rob

(02:07) and Ken and um between us and then of course open this up for uh for Q&A this is a four-part series so we’ve got a number of different ways we want to look at this I would really encourage you to um to attend all four there’s these are going to be from a little bit different perspective for each one so it’s very worth um you know your time to be able to show up for these um this first one we’re kind of going to jump right into meat of it but there’s a lot of interesting things we’re going to talk

(02:32) about next so we’ll talk about we’ll introduce the next one um at the very end so just to kind of introduce ourselves so if you’re not familiar TX cxo um is is a technology leadership company um that’s a lot of different things whether it’s fractional leadership uh with with with tech tech Executives that we have on staff or it’s working with tech Executives to become better in their roles um lots of different things that we do with working with companies and I don’t want to spend

(02:57) a whole lot of time on this we have a lot more slides we could show here but we just want to introduce that and and again you know our tagline is technology leadership solved because that’s exactly where we focus on making sure we’re helping solve those problems with the technology leadership we partnered with um provision analytics with Ken and Rob here uh who you’re going to be able to hear more of uh thankfully than me talking um and so these guys have been around and you can see lots of logos

(03:20) there and and I’ll let you guys read this we not going to go into this but um tons of worldw experience and being able to to from the analytic side of things but also um on the uh on the obviously the generative Ai and AI in general so um Ken and Rob both have a ton of experience in real world working with Waste Management uh the last place but I’m really looking forward to hearing more about you know not just not just strategically or thematically some of the things but really hearing some of the real world experience they’ve had

(03:48) with these so looking forward to jumping in that with that being said I’m going to stop talking and get into the the people who really count for this conversation and Rob I think you’re up next I’ll hand it over to you thank you Tim so listen pleased to be joining everybody today and and certainly this is a Hot Topic and by the way it’s been a hot topic for a while uh you know November 30th 20122 is when Chachi BT 3.

(04:14) 5 was released and became by far the most uh quickly adopted technology within UH 60 days two million users which is was just unheard of nothing like it ever before in history and then ever since then I don’t think you can watch an hour of CNBC without people talking about AI so this is uh certainly something that’s hit the World by storm and I think people are still trying to make sense of all the information that’s coming out of and so for today what Ken and I want to do is take some of the mystery out of generative generative AI

(04:40) get really clear about what it is relative to the AI topic more broadly because you know generative AI is a specific application of AI that uh certainly is poised to unlock a lot of value uh by March by the way this isn’t just chat GPT and open AI the whole world of software providers had for years been been working on this technology that’s probably why it feels like it’s exploded so quickly but really this was a decade or so in the making just a few months after Chachi PT 3.

(05:08) 5 hit the market Google and Microsoft and meta and Salesforce and Bloomberg companies that uh people are just now starting to learn about coher and anthropic all of them coming out with these large language models and generative AI capabilities that we’re going to be talking about today uh it’s no wonder that this past July when Gartner released their hype cycle and many of you on this call may be familiar with the gardener Hive cycle it really tracks the trends of any new disruptive innovation uh and technology that comes

(05:38) through and identified that sure enough no surprise to anybody we are absolutely a peak hyp cycle regenerative AI the question is is it Hyperbole and we don’t think it is uh uh we think there is a tremendous amount of value to be unlocked around the technology and just notice that the other thing that Gardner does is looks at the Timeline uh that it takes from something to go from Peak kyp into the plateau of productivity this is when people are actually really getting the meaningful value out of it the

(06:05) promise gets delivered and the timeline for this is pretty short Gardener recommending uh in their research two to five years we know that for the next few years to get there uh it’s going to be a bit of a bumpy ride just like it is with any other technology uh I would argue that we’re probably already in what they refer to as the trough of disillusionment how do we know this we’re starting to hear the stories come out about how this is hard we’re starting to hear some stories around uh could be expensive we’re starting to

(06:31) hear stories about some of the vendors are maybe over promising their capabilities we hear that some people have had some some uh privacy and data loss issues around the technology so listen proceed with caution and that’s one of the things that that Ken and I also want to equip this audience with today how do you accelerate your curve through the hype cycle and accelerate your way into the plateau of productivity so we’re really excited to be joined with you today I’m going to hand it over to Kin at this point to now

(06:58) really break down uh in very clear and easy terms understand what is generative Ai and what’s different thanks a lot rob you know um before we jump into it we wanted to just kind of take a pulse from you all kind of where are you in in this cycle and in your journey uh with generative AI so we put a quick poll here together uh which says you know how have you implemented generative a in your organization are you not yet started are you planning to do it are you in Pilots of this or have you already generated uh develop

(07:30) generative AI in production so if you would just take a moment uh take a look at this poll and and uh and give it your best shot appreciate it I almost want to guess I’m gonna go planning it 25% I’d say higher you think so I would say are we taking are we taking bets here um yeah planning definitely the highest we’ll see I don’t want to I don’t I don’t want to what do we got all right we can close it up now and see what we got oh there you go we have so first of all we have generative AI in production at

(08:22) 25% that’s an exciting uh uh number of be interested to hear more about those all right very good so what we wanted to do is first kind of level set on this term generative Ai and artificial intelligence you know there there’s been a lot of mystery in the term a lot of concern unwarranted concern around um chat GPT and this the intelligence behind it and is this making decisions for us is it ethical U should we be concerned about it learning and growing and taking over and and I think amongst um amongst the UN

(08:55) initiative it can really be concerning and and cause concern for a lot of risk and so on and so it’s important to really understand what what it is how it works uh because that’s the best way to manage it and manage the risks is really a thorough understanding what it is first of all the term AI has been around for actually since the 50s we’ve been talking about artificial intelligence uh for a long time and and if you Loosely Define artificial intelligence as a computer that actually Acts or decides

(09:21) as a human does uh then you know computers have been doing this for several years when you start to get into artificial intelligence making decisions uh in learning then you start to get into machine learning algorithms that we put into place in the in the 80s and the 90s um natural language processing capabilities uh and then in the 2000s we got even smarter and started using neural networks and really modeling these algorithms based on how the brain works uh so we’re using a lot of deep learning uh capabilities for the last

(09:51) several years uh to look at image analysis and to find complex um patterns and and and uh networks and Supply chains and so on so the field of artificial intelligence and being able to really understand a massive amount of information U find patterns that information and make decisions and recommendations based on that has been around for a while and and there’s there’s hundreds and hundreds of different use cases of how AI is help helping companies today what we saw with chat GPT is really the emergence of this

(10:21) new branch of of artificial intelligence what we call generative AI which is B based on this deep learning Network techniques but brought together with language models that allow it to understand context amongst very complex large language documentation and then the magic part is is then by understanding that large language and context and understanding of documentation produce output that seems like it’s been written by a human so generate language generate sentences and paragraphs and summaries uh in in a

(10:57) conversational way that we”ve really never had had before even our natural language processing and all the word processing we’ve been doing historically has been counting words and estimating sentiment and doing some very basic stuff but this generative AI capability now has the ability to understand complex documentation and then communicate back with us it’s almost as if for the first time we’re able to talk and have a conversation with our computers and that branch of artificial intelligence we call generative

(11:26) Ai and and the way it works it’s just like other AI and and predictive analysis and and advanced analytics techniques it learns from data finds patterns and then interacts and specifically with large language models we hear this word large language models a lot in in in the industry and you see several of them down here open AI being a large language model clad um coher being large language models what these large language models are is they’re actually built from massive massive amounts of data off the internet

(11:57) documentation Wikipedia controversial sources like copyrighted information but scanning the basically the web bringing together all that language and all that knowledge and understanding into a language model that now I can speak to it it’s beyond search it’s really understanding context of these these uh these these documents and then what chat GPT did for us is it actually created a nice interface that we could then ask a question so give it a prompt and it would give us a response and so allowed us to talk to this large language model

(12:33) and large language model is an important concept because these are um uh proprietary licensed models there are some open source ones coming out but as we talk about how do you implement large language models for your organization it’s important to know that there are a few out there um that actually are quite expensive and timec consuming to build and maintain and and make accurate and so on but that’s what this large language model is in the middle chat GPT is a interface to those large language models and basically what these large

(13:02) language models are they’re predicting the next word in a sentence or the next string of words in a concept and building up this language building up these paragraphs based on what it’s seeing in its large language models it’s a it seems like a minute point but it’s really predicting like for example here the quick brown fox this is likely to be the right um response based on the predictions of the next words and the next Concepts meaning it’s really not understand understanding and uh thinking

(13:31) and becoming biased and doesn’t have an intent to it it’s just a predictive model that’s predicting words it just happens to do it very very well so some cases of how the G of AI U are put into practice Rob maybe you can share a few of these for us yeah yeah and so there’s thanks for that K there there’s two ways that it’s I think helpful to think about this the first is from the key business processes that can be uh relevant and it can be applied and so there seems to be uh a a strong

(14:02) alignment across you know any of the strategy firms and technology companies that these first five here customer operations Marketing sales software engineering research and development those processes is where they expect about 75% of the value from generative AI uh to be delivered and then the second area is around the think of it from a role perspective all of the different roles that uh you have across knowledge worker landscape um is where the second half overall of all the value is going to come from and so just let

(14:33) unpack those for a few moments think of customer operations we know that call centers are a difficult place not only to staff resources but to get good answers because you got to consolidate all of the information provided in real time uh this is an excellent excellent application for uh for a co-pilot for a service agent or uh you could even just automate uh that process through through generative AI with a completely next level uh chatbot uh that we’ve all started to have some experience with that just don’t seem to be that helpful

(15:05) uh you can expect a significant uh Improvement in those types of capabilities certainly from a marketing perspective content creation uh uh it is exceptional at creating multiple versions of uh of creative content certainly human is in the loop then to you know do the final polish but a lot of content creation work can be done uh all of the the search engines optimization product marketing even Outreach uh that can be done from a marketing perspective or even sales from a sales perspective imagine as a sales rep being

(15:35) able to pull from the best information in the company to put together a best-in-class proposal for instance the amount of time that that might save uh software engineering this is where there’s a lot of expectation that the most significant amount of early value is going to come from uh given that the uh now they say that what’s the most common program and language in the world these days English uh would the winlos twins even have needed Mark Zuckerberg to create Facebook uh now or sometime soon into the future we know there’s a

(16:03) lot of things that can be done with generative AI to accelerate uh and drive productivity uh in software engineering the development and by the way in other areas of it operations as well and then research and development uh again think about drug Discovery uh materials uh uh Discovery think about creating the next kind of aerodynamic car and using the ability of generative AI to create multiple altern naves and then to interact as Ken said we can interact with our computers now hey this is great but I want you to add this feature what

(16:35) would that how would that change the you know the genome sequence in this in this Desy process it’s just an incredible uh tool that can drive productivity in these processes and then by the way just knowledge workers overall uh one of the things that I think is pretty interesting uh November 30th 20122 chat GPT 3.

(16:58) 5 comes out November 1st 2023 here in in just a few days Microsoft will be releasing Microsoft co-pilot powered by open Ai and chat TBT across all of its tools whether that’s word and PowerPoint email even teams uh the amount of productivity and just assistance uh that we’re going to get from these tools moving forward um You probably started to see a proliferation of notetaking tools that you or your colleagues might be using right now there is just going to be an endless amount of ways that these generative AI tools start to improve what we do every day

(17:29) uh it’s uh it’s really Limitless back to you Ken yeah and when you think about all of these use cases it it could could be daunting and I think as as an executive of an organization it’s important to understand how your Workforce is actually leveraging these tools and we’ve kind of broken them down into three three ways that uh your Workforce is going to access these large language models um and they are each important because they have specific um uh levels of security and concerns and and things

(18:03) that you should be aware of as you’re implementing and managing uh these capabilities the first one is is public applications like chat GPT so right now you can and and hopefully already have and your employees certainly are using chat GPT to help them through any number of tasks improving paragraphs um potentially writing poems or researching a topic on the internet and the way that that works is this is free open language model and uh open AI gives you this portal that you can log in and and uh send your question and then get a

(18:38) response back um seems simple enough until your employees are actually putting in let’s say written code from your company and sending sensitive information through this prompt to get back better code or proprietary information or um personal information uh so it’s important to note that as soon as that data goes into this prompt in chat GPT you’re you’re no longer protected with the information that could be leaving your organization um so understanding the potential risks of of chat GPT and then

(19:09) the information that comes back you know how is the information that comes back how has that been vetted for your organization what decisions are being made with the response now are those responses somewhat biased or hallucinatory and and giving you bad information so how are your employees using this publicly available Tool uh for interacting with these large language models um it’s out there now um unless you’ve locked them down your employees are using them and and be aware of what that what that can

(19:38) entail the next way that our employees and Workforce are going to be using chat GPT and these other large language models are through thirdparty applications um Rob just mentioned co-pilots coming out from Microsoft um we have large language models and generative AI capabilities built into Salesforce we have generative AI capabilities built into Google applications and and other sap applications and other Enterprise applications are starting to embed these generative AI model capabilities within their software um and software as a

(20:12) service models so now you have a situation where you’re getting great U let’s say call dialogues or call notes from teams or your call center application but how is that application taking the data from your your Enterprise um calling the same service because at the end of the day we’re all calling the same service in the back end sending that information your your company’s information through their application to the back into large language model and getting a response back but what does that pipeline look

(20:41) like how secure is that pipe um and who has the accountability for the information coming back right so even though your thirdparty vendors are embedding these capabilities into the software it’s just important to know how does that work um so you can control the the potential sensitive information coming back and forth through your organization and then the third method here is building a custom application some of the use cases that Rob showed earlier were companies that are already and and looks like we’ve got a few on

(21:10) the call here that are actually piloting large language models in their organization which is building an application uh and then sending information controlling the information flow to the large language model in back and this gives you great control but of course it’s it’s the development of of an application so these are three primary ways you’re going to see uh L M and and generative AI models coming to coming to your organization so I mentioned a few of these risks potentially associated with

(21:38) large language models again this is important just to note just to make sure that you’re managing the risks there’s certainly tremendous value that these language models are are presenting but a lot of boards and uh cios are concerned about the potential risks and want to make sure that they’re protecting their company as we’re deploying these kind of capabilities so we thought it would be worth at least um talking about a few of these risks we’ brok them down into two types so there are risks to the your

(22:03) organization itself and then of course if you’re following the AI um news and articles and and podcasts and so on there’s a lot of conversation around the the impact of AI on society in general and and and who we are as humans and so if you separate kind of organizational risks versus societal risks and you can see those listed here as well starting on the organizational side um certainly one large topic which is actually part of this four-part series that tag is is sponsoring is around the use of AI in

(22:35) cyber threats and cyber security so this is going to allow for much more intelligent fishing uh capabilities and so how how are we protected against um the the the maturation of cyber threats and cyber criminals and so that’s a very critical important topic one one to stay on top of the other one that’s just as disruptive and potentially risky for an organization is just industry disruption your competitors are leveraging this new capability to become much more efficient finding new ways to to Garner revenue

(23:04) and new ways to reduce costs how how is this disrupting the industry and what do companies do that are falling behind um so this capability will be disruptive and one that um you should lean into to make sure that that you’re not left behind but but some specific operational risks that we kind of alluded to here a moment ago was around data loss so as you send data through to this prompt um Samsung was famous in the news for one of their employees sending over information that had um security codes in it over into a large language

(23:38) model um and so this kind of data loss is very important and one one of the first things you should be concerned about when implementing a large language model hallucinations is really important this is uh the large language model as I mentioned earlier what is just basically doing is predicting words that are likely to happen next so if it’s trained on bad data or trained on incomplete data it’ll do the best it can to kind of create the end of the sentence um but it might be nonsense uh it it’s just going

(24:06) to do the best it can based on the information that’s been given so there’s a lot of uh training that has to be done alongside these large language models uh to fine-tune them to be relevant to your business context uh and and there there are ways and methods to shape the prompts and and load the data in and fine-tune these models so that they’re more relevant but um certainly if you just use a current open capability you might come back with um some responses that look good and and hard to see face value that they might

(24:36) not actually be accurate so training and and validating is is absolutely critical so without reading uh the rest of them those are some critical ones um that are going to hit you right off the gate and then of course there’s staying ahead of the legal and Regulatory um landscape when it comes to how AI should be used in the organization uh there are uh where we’re probably all already familiar with the the California consumer Privacy Act and the gdpr which were launched a couple years back to help protect uh consumer

(25:09) data and consumer privacy around their data and gave certain rights to Consumers to be able to recall their data and and know how their data is being used um the extension to kind of data privacy is now this new wave of regulations coming out around the use of AI and so beyond just the use of data this these are about the use of AI and so the European Union um published their their act their AI act which is going to affect at the end of this year companies need to comply in two more years um but it’s basically saying if you’re using

(25:40) artificial intelligence in your application Suite Andor if you’re leveraging artificial intelligence to do your business uh you are accountable for these things and it’s um it’s interesting to see around the the the fact that you need to demonstrate the transparency of how these algorithms are are actually making decisions so that needs to be made available they needs to be explainable you need to demonstrate that you’ve managed for bias so you know if you’re if you have biased data and you make biased decisions you need to

(26:10) demonstrate that you have you have methods and processes in place to to Monitor and vert bias um you have to have a human in the loop uh in these processes and you have to have your consumers have to have the ability to say listen I don’t want an AI generated decision I want a human gener ated decision they have a right to ask you for that so as we’re building out these AI systems and AI Solutions it’s really important to stay AB breast of these regulations that are coming down haven’t hit the us yet uh the the US has

(26:43) published this AI Bill of Rights which is basically guidelines um which will probably form into regulations uh into next year but certainly important to to be aware of those as we deploy these Solutions thanks Ken so as we think about the path forward you know in the conversations it seems to be one of four things that uh that you hear people talk about um the first is hey we’re just going to play wait and see you know this is this is brand new um we’re not really sure if there’s any there there we’re

(27:13) going to let other folks you know figure it out um you know listen the the strong uh suggestion there is at least proceed uh with caution you know it’s one thing to say I as a company don’t want to be using it but you know you got you know uh your your employees that are out there using at GPT and so there are some risks associated with the weit and see stance which is why I think you know a lot of people are at least saying we’re we’re going to want to take some steps to understand the risk uh to protect our

(27:40) business from data loss from some of the examples that that Ken gave a moment ago uh and start to educate our organization around these tools and so we call that really more the protect mode stance now still uh there’s a lot of folks uh and companies that say Really we see the transformative nature of this and uh as Ken mentioned we see um the risk by the way uh if we’re you know in our industry you know uh behind the first movers and so we want to explore we want to take a thoughtful and deliberate approach to

(28:10) beginning to experiment and learn uh in a very thoughtful and deliberate way from these tools the one that really concerns us is when we’re talking to folks and they’re saying I we’re undecided and we’re not really sure who has uh responsibility for the decision I think our number one uh message that we would invite folks to consider is as Leaders we each have a responsibility to make sure our organizations are intentionally making a decision uh to continue to to uh uh to be undecided is really to play wait and see but without

(28:41) the intentionality of making sure you’re at least aware of the risks you’re really just a drift and so our strong uh uh uh recommendation would be that everybody uh uh who is in the position to toh to be responsible for driving decision in their firm get one made uh in their company and at a minimum take the steps to protect uh the company from data loss uh from some of the risk and begin to educate your organization uh with that I think Tim we’re over to you yes and I appreciate that that perspective um because you know the

(29:16) whole we’re GNA wait and see or do nothing is kind of scary um the whole this whole thing can be a little scary if you’re not familiar with it if you’re not into you’re not a techy person um can kind of be overwhelming but now I appreciate you guys that part of what we wanted to do and the goal of this was is this four-part series is to lay these out and we’ll start digging into these and helping Executives who are facing these things understand where they need to dig into so so they’re not just

(29:42) jumping into this blindly and so they can have some some guidance to be able to understand you know how do they kind of step into this and and and it doesn’t have to be that you all of a sudden jump in this now we’re in and now we’re doing all these things there’s a there’s a path to that and we’ll kind of probably I’m sure talk about that some through some of these questions you guys have um so so let’s kind of start off with some of the panel questions that we had I know we came up with a few please um

(30:04) again as we’re going through this if if there’s any questions from anybody um from the attendees please please put those in the Q&A um we’ve got some that we’ll run through as well to kind of give you a little bit you know more conversation on these um back and forth but now feel free to to have your input as well um the first thing I want to kind of start off with um Rob was very back at the beginning you showed the chart and I love that chart that kind of shows you know how the the partner hype

(30:28) cycle how things have kind of taken off and I can tell you I’ve probably been one of those that’s followed at myself I was you know excited to see it come out went out started trying things and said well that doesn’t really sound like me or you know whatever and you know kind of fall off then you kind of start learning okay well this is how you actually use this um so what would you say you know to those who say this is just hype you’re not going to really replace you know my job with this because you know I’ve seen what it can

(30:49) do um I can do much better than that or whatever what would you say to those people who say this is just a trend that’s that’s going to go away at some point yeah this is this a fat is it just a party trick and and and we we won’t be talking about this in a few months listen I can understand because you know even in my own experience you know you you you you do that first interaction with chat GPT and it really blows your mind right um and as you continue to work with the tool you start to realize

(31:15) that it’s heavily limited you know and by the way it can be wrong and and the other things and so uh so it’s under it’s a certainly understandable that that uh people would feel that things are overhyped right now I think there’s a few things I would point to just to uh uh underscore our conviction around the use of generative Ai and and how pervasive it’s going to be the first is uh compared to the metaverse which was probably the last big thing that was hyped for the last few years um we have

(31:41) by over 10 times the amount of pure reviewed articles over over several years sustained over time around artificial intelligence Ginger of AI um this is very clearly something that uh has been driving a lot of investment leading up to now on a on a whole other scale and by the way I think maybe the canary and the coal mine also is what do you see from the large IT services providers and technology companies may have seen the headlines that Accenture is investing $3 billion dollar uh to uh to get prepared to serve their clients

(32:11) based on what they’re hearing from their clients uh ey as a matter of fact I think came up with that b billion dollars um there is just a an unprecedented level of investment in this capability uh so we know that it’s going to have some lasting power and then you just think of the you know the what is the potential what is the promise um this this may be a little overstated but AR invest which by the way was early on Nvidia which is one of the core Technologies that’s now powering they invented the G GP early

(32:39) the GPU they’ve in a lot of ways what’s powering what’s happening with generative AI right now um Arc investors out there suggesting that there will be more economic value created this decade by 2030 from 2020 2030 from generative AI than the internet has created since it was in conceived now this is crazy talk the these are huge numbers but even if they’re halfway right um excuse me we know that we’re we’re up for for quite a wild ride and at a minimum we know the cats uh out of the bag you you can go to

(33:09) a website right now called there’s an AI for that and there are literally thousands of tools that are out there uh for anything you can possibly conceive of uh to try to help Drive some productivity and things you do from a day-to-day basis so uh so listen the question is is it hyp yes we are at Peak hype right now is it hyperbole we don’t think so we think there’s a lot of evidence to suggest that this is going to have a transformative effect uh significant transformative effect on the way business is conducted yeah and I

(33:37) appreciate that Rob I think I think for me part of my Evolution with it was is viewing this more as a a tool rather than you know just this fat out there that’s going to replace everything that everybody’s doing and just like any tool you have to learn how to use it so like I said when I first got on there you get the whole cool factor of Something’s talking back to me I can type whatever but then when I actually try to use it to you know create some you know an email for a sales cycle or something

(34:00) like that and then I’m looking at it it’s like well that that really doesn’t sound like me that didn’t I mean I can do much better on my own but then you start learning okay this is how you actually use this to make to make it work for me and learn my voice learn my tone there’s a learning process to it and I think once I understand this is this is a tool all that stuff really like you said is is is is comes into play where just like you know the internet change thing so much just like phones and everything all these these

(34:23) big devices I think like you said this is going to be one of them one other thing that you mentioned and Ken I like the way you broke down you know between you the chat gbt the generative II the the actual co-pilots and and you know the the individual applications with inside the companies um but the co-pilots are something I think that we’re now just starting to see come out more and more I be honest with you I have very few applications now that I use that don’t have something that uh that they want you to to tie in chat gbt

(34:50) or something like that into it so when you mentioned you know the the great financial benefits to this number one where do you see that coming from is it just a productivity thing using those tools um is there really going to be that level of financial gain that’s going to be gained from it what are your thoughts kind of from that on on the copilot and all those things that are directly involved in it yeah that’s a great question because if I’m a CFO I’m I you know show me the money like this

(35:15) this is all interesting conversation so let’s let’s let’s talk macro first so um you know you already heard Ark invest uh uh Proclamation but let’s let so McKenzie came out with a study and again they broke down by process and roles and how they could identify economic value that would be generated by these tools and they came up with A4 to6 trillion doll economic benefit from generative AI okay so this is a big background number by the way just to put that in perspective yeah um at uh at at the $

(35:43) four trillion dollar number you would be right there with Japan and Germany as you know T you know buying for the third largest economy in the world these are massive numbers and so you have to ask yourself where is that going to come from from and we’re very early in in this but we do have some uh some studies that are just now starting to come out which are very encouraging so uh Harvard Warden and MIT all three of them came together to work with one of the TR top strategy houses strategy Services firms

(36:16) and had nearly 800 of their uh uh staff uh leveraging chat PT uh uh for productivity enhancement tools and what they found studying this cohort relative to their peers who were not allowed access to chatu PT they got 12% more tasks completed they completed them 25% more quickly and with a 40% higher quality um you can go out and find the study and it’s it’s 50 pages long and quite Illuminating around what we can expect to see coming around the corner even then though um you know you got to ask yourself okay great so that’s

(36:50) productivity but again I’m the CFO where does that show up in the uh in in eida and I think what you then start to say is all right show me some some examples around how this is going to either reduce cost or drive revenue and let me just give you two very quick examples that we’re starting to see emerge we think early on a lot of the cost is going to be uh identified and pulled out of it uh uh from a development perspective so if we can get 20% more efficient in software engineering application maintenance some of our it

(37:18) operations that can be reinvested in supporting these other use cases that we’ve talked about early uh uh reports out of people using the Microsoft GitHub copilot for instance is somewhere between 20 and 40% efficiency there’s a bit of a range but is a material uh measurable productivity impact of using these tools now let’s talk about it from a revenue side so so so now you’ve taken some some cost out of the business where am I going to redeploy that from a generative AI perspective to to drive a

(37:47) Topline uh there’s a really interesting study uh or I shouldn’t say study use case pilot that was done in one of the the the largest IT services companies in the world around proposal generation so imagine you you you take the best of the best proposals from the last 12 months uh you have generative AI do your first draft for you what they’re finding is they’re able to get 40% more proposals out as a result of leveraging this generative AI capability and what does that do for them from an economic impact

(38:17) perspective what they’re finding is it’s those smaller kind of more million dooll level or sub deals that many times it just takes too much effort to respond to but in currently today’s economic environment the fact that they’re getting an uptick and winning those kinds of deals is actually doing a lot to support them as we’re in this bit of a lull as people have seen around digital Spin and so you can easily imagine starting to apply these tools in the areas that that bring some cost and

(38:45) productivity efficiency particularly in your it space reinvest that in use cases that we spoke about before customer operations sales uh R&D Etc to start uh uh driving further economic impact and and just kind of a followup from that Rob you know as far as I mentioned earlier for myself my it was live it was training and helping me figure out how do I use this thing the right way because just because it’s a tool doesn’t mean it’s going to give you everything you want just like any other tool we use

(39:10) out there if I don’t know how to use Salesforce the right way then it’s can be more danger than good do you see how do you see companies now starting to um help their employees not just give them access to it but help their employees make sure they’re using it in the right way to be able to get the right type of results you guys see a lot of training and and and and from a corporate level rather than just from individuals out there um that they’re helping them use it the way that they should be using it

(39:34) within organization listen it’s a great point because what we’ve known forever in technology is you can get a great solution to Market but unless it’s adopted and and leveraged appropriately that’s that’s the last step to extracting the value out of the tool and we’re seeing a few things the first is that there are some trepidation that people have in using these tools chiefly among them too the way these gener of a AI tools work is by taking in all the information by listening in on your

(40:01) meetings by combing through all of your documents so there’s real privacy concern that folks have around using these tools and so there’s some real education I think that’s going to be required over and above what you might normally have in a change management program Beyond just the training and you knowbe making sure people have the tools on that they can leverage them and then of course there’s the concern that other people have around is this going to cost me my job Hey listen I love this idea

(40:25) that you’re going to drive all this productivity does that mean there’s going to be less of us there next year and why should I be rushing to go use these tools and I think the thing that’s important there from a message perspective is listen I don’t hear anybody talking about and nor if you read all of the uh the economic studies coming from uh uh from these these strategy houses and others that there’s going to be Mass layoffs I think what they’re saying is you’re going to be not replaced by AI but somebody who’s

(40:48) proficient using AI because we’re gonna have a higher level of of throughput that’s going to be coming through this team that’s where a lot of the economic value is going to come from and so listen I think there clearly needs to be traditional uh change management and communication programs it is unique it is as much a cultural transformation uh as it is anything else and this is going to be something that you can see spreading through organizations starting November 1st Enterprise wide release Microsoft co-pilot and so you can

(41:16) imagine as companies really unleash the value of those tools it’s going to come with a significant amount of uh training and development and support and education and cultural evolution inside a company yeah yeah and I’ve heard the whole thing if you’re not the one using AI you’re can be replaced by the person who is and and definitely definitely do you know see that just like any other tool that you use if you don’t know how to use the tools well enough you’re not going to have the job to be able to use

(41:39) the tools so makes perfect sense um for you know most the people that that are on here you know this is this is more of an kind of an executive crowd whether you’re you know the CEO level CEO CTO whatever type of level you are um and as we know from working with them you guys as well you there’s they have a long list of priorities that they’re having to manage um along with that and so you know Ken um you know from your thoughts and we showed the slide earlier about how do we go from or how you know the

(42:08) different types whether we’re a take action person we we’re kind of wait and see how do you kind of as an executive set your priority of where you need to be with that if you already hadn’t jumped all in and you’re trying to figure out where should I be how do you think through that process and understand where you should be in that I think the first thing to do uh which we alluded to is is really understand the potential risks of data loss because that’s you you don’t have to use the

(42:31) tool but if you if your teams and your your Workforce are using look for example chat GPT does that represent a potential data loss for your organization and how you’re controlling that you really have two options one option is you can just shut it down um and there are companies that have really just closed the pipes and said when you’re on our Network you’re not you don’t have access to these tools so we’re just literally going to lock it down and for these companies especially companies that are in copyrighted domain

(42:57) spaces like music and and documentation uh that that was a prudent decision to start with so you could lock it down uh the other thing you could do is just monitor it so you could say we’re not going to lock this down but we’re going to monitor it and we’re going to educate our employees right so we’re going to give them um some codes of conduct around how to use these tools and give us some training on how to use these tools um and you you alluded you alluded to training a moment ago in in your

(43:21) question and and if you think about our codes of conduct training you talk about data training we we all hopefully are are giving our employees training on what is personally identifiable information and how do we secure our our clients information and so on it’s the same type of training that we’ll be doing on how to use these AI systems and how to use the results of AI systems and we’re just going to be having to expand our expectations with our Workforce on what is the proper way to use these

(43:49) tools or not if you’re going to allow them to have to have access to these tools um and then keep your eye on regulations I would say monitor these regulations because as they become as they come down and they’re they’re all responsible regulations that are going to protect companies and protect individuals uh they’re going to come with requirements for your own ability to demonstrate transparency and to demonstrate how you’re using these tools so keep your eyes on those because you’re you’re going to be required to

(44:16) take action um for several of those um so lock make sure you’ve either decided to log it down Andor monitor it definitely have a policy created I think at minimum there should be an AI policy within your organization for how you intend to use it get the training to your employees on the proper use of the tool because there’s only so much you can control um and then you know monitor the regulations and then finally keep your eye on your competition as we’ve talked about before um you can stay in this protect mode for so long but just

(44:48) keep your eye on what’s going on in the market yeah and I think that’s it’s almost two phase like said how you protect yourself and then how you use it to you know to to compete and hopefully create a Advantage with it um and and where do you want to be in that cycle do you want to be someone who’s just protecting versus do you want to be moving forward and kind of being on not not even on the front end but you know somewhere in there where you’re actually using it to to provide that that that

(45:11) level of productivity for you so um yeah and I think that’s I think that’s important you know so when you talk about being able to implement this stuff um you know how do you know if I’m a take action person how do I jump in and actually come up with a plan on what I need to be implementing this stuff when and do you have to do it all at once or there you know pra I would which I would assume are there approaches to where you’re more stepping into it piece by piece how you decide the priority from

(45:38) that um and then and then you know what are the approaches how the approach is different if I’m a small versus mid versus a large company um I know there’s kind of a number of things in there but um kind of what what are your thoughts on those yeah well so so if you do decide that there might be an opportunity here to lean in and it’s really a question of selecting use cases that are going to be the most productive for you um and how do you select those use cases it could be on on as Rob mentioned earlier you could decide that

(46:05) hey we have too many people in our call center we want to use generative AI to actually create more efficiency in our call center Andor uh interact with our customers in a self-service way to reduce that inbound need altogether and create a better customer experience you may have contracts reviews that you have to do within your organization that are taking a lot of effort and time and backing up the system you might want to explore how to use of AI in those areas and so on so I think the first thing to do if you’re planning on leaning in is

(46:31) is how do you kind of solicit for these potential use cases uh and then prioritize and select some Pilots because what’s interesting is um this capability is new this generative AI capability is new and so most companies are in experiment mode Andor discovery mode on hey is this possible can this really get me the the the outcomes that I’m looking for So you you’re really going to Pilot and experiment with these cap capabilities and so you should be soliciting for these ideas from your organization yeah now before you solicit

(47:02) these ideas everybody needs to understand what the heck is this so you probably want to start with some general um um education to Your Business Leaders say hey here’s an opportunity for us this is what this thing is we’re soliciting ideas you know um prioritize those ideas based on potential impact and then go through a process to select a pilot and then uh and then get started and and learn fast yeah and and I know you know a lot of again as an executive you know a lot of guys are not really wanting to get too much

(47:32) into the techie side of this but I think in my opinion because my experience as well is is just educating yourself when generally what’s out there helps as well I know for me I do a lot of presentations so you know learning there’s tools out there that I could actually go describe the type of presentation I wanted and it would generate me at least a first copy that I could start from and have cre you know just even knowing that type of stuff is out there I think just takes educating yourself and making sure you know that

(47:56) because otherwise you’re kind of blind blind and then you know competition probably is not um you’re right especially if there’s use case that you’re looking at that somebody else has already solved there’s a lot to learn from that prior work but I think that the the the biggest challenge right now I think is there’s so much noise right every software vendor is now talking about hey now new with generative AI here’s your new toothpaste now with generative AI in it it’s like and how do

(48:19) you make sense of that you’re a CIO or cxo trying to run your business and then all your vendors are coming in your big five consulting firm everybody’s coming in saying they’ve got gen it you know you have to have a PhD in it to stay on top of it so it is a challenge for organizations to to Really weed through this you know what’s kind of just marketing where what’s a real solution and then what does my business really need but again I I would take a very prudent approach don’t worry about

(48:46) the Solutions in the market worry first about your business problem yeah problem trying to solve what’s the solution for that problem if J is the right solution let’s go make it happen and then find the right um way to implement that secondary y you know we did we did I SP about a month ago a a session with cyber security and some of it’s the same thing there’s so much stuff out there if you try to take everything on you’re going to fail I think this is very similar you have to identify you know your your

(49:10) simple way of being able to deal with this your pieces that are most important to you the things that drive your business the most and start looking about how you can apply these to that so absolutely one other thing I wanted to and we mentioned you started mentioning use cases and Rob I know you even did I think in the slide where you broke down some general ones um but can you kind of go a little bit further maybe on some that you guys uh have have worked on in the past use cases that you’ve done and you know also I’m kind of looking

(49:33) through some of the question and answers up here and someone had asked I think it’s a good question um if anybody else uh because there was 25% I think that said that they were already using it already doing something so if anybody else wants to share um a use case in chat or in Q&A feel free to we we’ll we’ll kind of uh we’ll read that for you as well but um yeah what what type of use cases are you guys seeing out there um that are causing the most effect to this yeah so there were listen there

(50:02) were a few um you might have seen the tiles when we were talking about the examples before that might be good to highlight right now so the first is Morgan Stanley which has been uh very public you know a lot of lot of good information out there on on what they’re doing and their idea was to create the super agent so how do they go through the Thousand tens of thousands of documents right that they have their Corpus documents in support of being the best financial adviser a super adviser that you possibly could be with clients

(50:29) and so they are rolling so think of it as a co-pilot right that takes the best so every agent we should equalize uh the insight and knowledge and ability to uh uh to advise and counsel across those agents they’ve gone one step further since then by the way and they are in fact now in client sessions uh recording uh uh H having the the note taking occur which is automating uh the notes and action items and follow-ups coming out so you see a lot uh that’s happening there anything around uh co-pilots for

(51:01) summarization bringing complex subject matter together another example let’s let’s go far away from financial services and let’s go more into like operations so imagine all of the the technical documents around you know heavy equipment that you might run uh in process manufacturing a US steel uh has one of the largest uh uh uh uh Mills in the country and they for all of their heavy equipment are now equipping uh their uh um mechanics excuse me uh and technicians uh with a similar type of co-pilot that can accelerate their

(51:35) ability to uh uh to serve and and get heavy equipment back into into service and so you have that one PWC by the way is uh rolling out I don’t think anybody’s are suits fans here Harvey Spectre there’s this uh there’s this generative AI tool car Harvey um that PWC is is rolling out is going to be rolling out at scale that allows them to do all kinds of contract reviews um imagine going through a large Corpus of documents across your clients there’s a new regulatory uh uh you know need around sustainability and you got to go

(52:06) find and update all of those um so there’s just any number of use cases that are up and running right now um Ken I don’t know if you have uh uh any that you would add to that no it’s just it’s it’s those use cases where there’s large volumes of documentation that you have to call through and then you have to produce kind of summarized content it’s not like how much did we sell last you know yesterday that’s a numeric use case right or predict my financial performance next week that’s a that’s a

(52:35) numeric forecast but if it’s hey read all of these documentations like the Morgan Stanley one read 100,000 documents and tell me what I should advise this customer that’s a really good use case I think I think that’s exactly right and I think the other just think about on the other side too around the the human capital challenge so so listen um we talk about call center uh and you know listen the idea is more and more you’ve got these co-pilots and you also have self-service it is tough to hire people in call centers I um you

(53:03) know typically it’s a it’s a difficult job to fill and so uh I don’t think what you’re going to see is layoffs I you see through attrition we’re just going to have need for less people in a job they really don’t want to be doing so now think of about on the operation side we’ve got over half of the Baby Boomers that have retired they’re taking a lot of knowledge with them uh you know we look demographically about trying to to hire enough people into the labor force one could argue

(53:26) uh that use cases that help you drive labor resiliency uh is a significant focus of use case as well yeah and there was a question asked as well and I think you guys have already kind of covered this so I I just want to call it out though um kind of in the same vein how do you see generative AI helping knowledge workers improve their you know productivity the Morgan Stanley uh I think is a good use case of that where you know people who are making investment decisions based upon all this information they have to be able to go

(53:53) through and probably half the time can’t get get through all of it to even make the right decision how are they how are they able to take this and actually make better decisions from that um I think I think that’s a good example of that I don’t know if you guys have any more you want to add as far as knowledge workers but and that one would definitely fit just imagine 12 months from now or or a year after Microsoft co-pilot has launched for for Enterprise and people start to get proficient for it

(54:17) proficient with it imagine just sitting down and and almost as if you’re you know just voicing out hey I want to do a presentation on this topic these are the five points that I care about or by the way just take uh a a deliverable um that you already have and say give me a presentation off of this it’s going to save you hours and that’s just one of innumerable number of of of examples um uh that that’s going to occur I can tell you this I know that I’m pretty poor about all of the tracking of action

(54:46) items and follow-ups and orchestration and I think as soon as we see that we’re all able to get um you know really an AI based assistant that follows up on all those things that that’s going to save us a lot of time there’s just incalculable ways that this is going to manifest and I don’t think we fully know exactly what that’s going to look like but I’m pretty sure that we can have this conversation a year from now and see that things look and feel different yeah and and that’s spot on because like

(55:09) I mentioned before I I use I’ve used tools for before you know it’s helpful to describe presentation you want like you just said and they will actually do that now now with the c-pod and Microsoft I could see people were just using that directly within PowerPoint I’m create a slide that does this with these type of Statistics that whatever that you know what whatever I want to see um and even myself you know I’m on a lot of different calls so having basically a note taker um that will take notes does one thing for you

(55:35) but it even goes beyond that to describing within that call you know analytics from it who you know who talked the most um you know who what was the sentiment with you know some of the questions back and forth and some of it’s right some of it still takes some work to train but um even these are the follow-up actions I should have from this that type of stuff is I mean for me a huge time saer because I hate to I hate to write and and take notes while I’m talking to someone and and kind of handles that for me and help helps me be

(55:59) more productive um here’s one other question and I’m be honest with you from this while you guys are here um that I wanted to ask where’ It Go um I lost it now oh and I think that’s because somebody answered it Ken I think you already answered it just just to kind of mention it to everybody recent um llm announcements have mounted to now with more tokens what does that mean to 60 if anything um yeah well a token is is uh when you send these words over and then it processes the words and comes back

(56:35) with a a response you’re basically it’s kind of like paying by the minute you’re actually paying by the token which is these word Parts um so it’s not paying by the letter or the words they just like chunks of words so basically right now these pipes have a limitation to how many tokens you can actually send back and forth and all the vendors are expanding that so you’re actually going to get you’re going to be able to send more more information over and get more information back but you’re paying per

(56:59) token so it um it’s just metering your your interaction with these language models yes um one of the other questions that came up was about copyright and I’ve kind of questioned this myself because you know there’s a tool that I’ve used before that you know I can describe an image that I want to create and it will it creates some amazing things now the better you are just go back to training the better you are describing it what you want and the different parameters that’s expecting the better you get but you know how does

(57:24) that work and other things with copyright type of things and that was one of the I forget who that was from there was a question that came out from that yeah certainly something to monitor right because copyrighted M the the the key to this question is what data is inside a large language model how has it been trained that that you’re using for your decisions in your organization if copyrighted material was loaded into this large language model and then you have access to it through this then yes what’s happening right now

(57:53) in the courts they’re trying to say you you don’t have access to this those large language model owners are going to have to Rumer the the copyright owner for for their use so there could be sensitivity there if you’re actually leveraging that information yeah these litigants may come after organizations but if you’re if your heads up on how you’re building these models if you’re training your own model you can control that and create like the Morgan Stanley 100,000 documents those were their

(58:18) documents right there’s no copyright issue there so it’s just a matter of really understanding how the models were built that you’re going to use and it gets more opaque when you’re using these third-party generative AI Solutions because it’s baked into Salesforce for example so you have no idea what they’re using and then litigation could come after you as the end user so it’s just something to watch out for as you’re design and we’re going to dig deeper into some of these conversations and we

(58:43) even have I think one of the subjects we had talked about doing in part of the series was cyber security and you know what are the what are the executives you what are what are their responsibilities for these things so those will be some of the other topics we can dig into I want to close at least the the panel discussion out um with just one for both of you guys and you know pretty much everybody except what’s on here or at the executive level either like said either CEO CTO cxos CFOs somewhere in that realm um you know if you were one

(59:11) of those that’s kind of on here listening or or listening this live or recorded you know what would you guys be doing about generative AI where would you guys be putting your focus yeah well listen I think it starts and we’ve talked about this before what what is the uh the Strategic impact of this capability to your industry and your company and specifically within that uh the differentiator here is going to be who has the data and so if you’re an industry leader and you’ve got the you know the the best-in-class uh uh

(59:46) data around you know for instance if you’re your Waste Management the uh you know the 360 degree sustainability Loop a solid waste and recycling uh what can you do with that uh you know to drive uh impact and and uh you know create service and and uh in those things so so you have a real unique advantage in your data and I think you’re becoming more and more aware of that now than you you ever have before uh what’s going on in your industry and your company how can you take advantage of it would be I

(1:00:12) think the first thing that uh that we’ make sure we were talking about Ken yeah that’s right um at least bring everybody together your executive team together to to align on a decision to either protect or or explore and then move forward from there so you have something to say to your board when they ask you what are you guys doing about gen at least get to the point where you say we’ve understood it we’ve discussed it we’ve made a decision and here’s how we came to this conclusion y when as

(1:00:40) always you know we have guys like Rob and Ken who are out there um working with people every day they’re they more than glad to get in those conversations and and help with this stuff as well so and tag tag loves working with me time flight so um we are I apologize a couple minutes over I’m just now know noticing I do want to at least um Ken can you throw up the slids one last time and I want to at least introduce the next one that we have coming up just so everybody’s aware and please um you know

(1:01:04) like I said this is a series of that we’re going to be walking through a number of different things um as part of this um so the next one we have le have have coming up is how AI delivers margin expansion we talked about that a little bit um in this one but we’re going to again dig deeper into that November 2nd 2023 you all know the year with Ken and Rob again we’re glad to have you guys on here should be about the same time typically we like this time it’s a good time to have people on so you again

(1:01:32) you’ll be getting more stuff uh and materials from this to register and looking fors to if there’s any feedback on some of the use cases you guys are looking at like the question said we love to get you the messages back through Linkedin or whatever Ken and Rob and and I are all on LinkedIn more than glad to to hear that or you’ll hear him follow up from us as well so love to hear those things and any other things that you guys may be concerned about with this um please let know we love love being able to have these

(1:01:56) conversations honestly we could be on here for another um probably two or three hours and still and still be going um so we’ll we’ll split it up into a four-part series so you don’t have to hear us all at once so again Robin Ken thanks for joining us looking forward to November 2nd and uh looking forward to hearing back from more the people that are on here have a great day thanks guys see everybody