Leveraging Tech & Generative AI for Risk-Smart Debt Collection Practices

0:04

Hello everyone. Thank you so much for being part of our webinar. My name is John McKinney. I'm the head of marketing at Cornerstone. We're excited to have Josh Allen, unfortunately not that Josh Allen, and Joe Liffrig with us on today. Josh is the founder and CEO of Tratta, a board member of RMAI and has extensive expertise in financial tech and software product strategy. And then Joe Liffrig is the CTO at Cornerstone Licensing and has developed Cornerstone's world-class Atlas license management platform. And before we get into the discussion, we do have a short disclaimer. Due to the nature of the topics we'll be discussing today, we need to establish that this information is not intended to be legal advice and may not be used as legal advice. Legal advice must be tailored to the specific circumstances of each case. Every effort has been made to assure this information is up to date. It is not intended to be a full and exhaustive explanation of the law in any area. However, nor should it be used to replace the advice of your own legal counsel. With all that out of the way, let's dive into the webinar. And for the audience, please feel free to ask any Q&A throughout. There's an option at the bottom on Zoom for asking Q&A, and we'll try and get to those throughout the discussion. You know, for my opinion, you joined a webinar hoping to get some sort of information about this topic. So, feel free to ask questions as we go. And we want to take this in whatever best direction suits our specific audience.

Josh Allen1:37

So, John, I want to throw out there, John, no question too small or that anyone should feel dumb. I think it's really cool that we have a group here. I'm not a software developer. Josh is not a software developer. We are just normal business people a lot like the people watching this webinar who have taken a couple small steps towards AI maturity. So, bring the questions in. I'm excited to talk about actionable like lower level stuff that hopefully is relatable, not pie in the sky technical stuff. Yep. Absolutely. Great call out. That is one of my favorite segments of one of the podcasts with the Kelsey brothers. They have the no dumb questions very much. No dumb questions, you know. So please ask whatever comes to mind. But Josh, you just released a survey, a big industry survey from Tratta. I'd love if you could kind of kick us off by highlighting a little bit about that and start our discussion off from there.

Joe Liffrig2:30

Yeah, absolutely. So, we decided to run a survey for primarily third party agencies, collection law firms and like the original creditors of course their collections departments. We didn't, you know, we kept it to a smaller group like that, tried to not extend it to BPOs or first party to really see where digital maturity is in this industry from actual operators rather than from what operators may be seeing through marketing channels. We closed it out in late December is when we closed the door and just released it. The I would say it's a 50-page survey. It does have pictures, so it's not that intimidating, but the biggest like call out from it is that it kind of reinforced what a lot of agencies and operators I think are already feeling, which was that compliance maturity is way out there.

Joe Liffrig3:38

Like high on the scale like there's very like high maturity but digital adoption is not and so there is a demand from it operationally but an adoption that is incredibly low and kind of those resistance points come from like lack of proper infrastructure and like compliance roadblocks which does sound very obvious like of course that's why we can't implement this and that but it was broken down over like a widespread of questions in four different areas such as like how involved are your agents with digital you know things like that like are do you have like training rolled out to them but I would highly recommend if the link is invisible now we can share it like taking a look at it and reading it I think that if anything it gives a little bit of comfort that you are not behind and that really no one is we are still on that upward adoption curve there's there just is what seems to be a lot of like foundational things, roadblocks, not necess again, the desire is there, the foundation is not. And so we're kind of at that moment in this industry according to the survey. But it was awesome to do.

Josh Allen4:53

So, thank you for bringing that up. Absolutely. Yeah. And you can find that in the resources section. There's a link to it to download for yourself to dig through all that great data. I know Joe, you were mentioning earlier today that you had a chance to look through that. And so know if there was any highlights from there that you wanted to touch on.

Joe Liffrig5:10

Yeah, I think the thing that jumped out at me the most from all the fantastic data was the focus on integrating and how the biggest gap that Josh's respondents identified was in integrating their existing systems. I think it's easy to think like ooh AI like I need to go invest in a million dollar system and like get something brand new and big and like replace everything where there's really an opportunity as highlighted in the survey but also that I've seen in my experience just to use AI in really targeted small areas and to connect the systems that you already have better and to get a lot of unlocks from that.

Josh Allen5:52

Yeah, that is like without a doubt like I mean you can play that same scenario out in so many aspects of every business and in this one the central hub of everything is usually the most dated and yet everyone's pulled in a direction that doesn't really mesh and so we just create silos on top of silos. And it's very funny because it goes largely ignored externally but then you have something like this that everyone was like yeah where are the problems? It was an anonymous survey so that speaks to the honesty in the responses and that yeah that stood out in a big way. Yeah that called to mind an example for me too that I thought would be perhaps insightful to share. I spent a lot of time over the last couple years seeing like a lot of YouTube videos like build your you build your million-dollar app in an hour. I'm sure if you're following, you know, AI in the business world at all, you've seen plenty of those. 15 minutes to being a millionaire with AI and like they just seem so ridiculous, but like I for a long time just kind of watched the videos and didn't dive into like what are these guys actually talking about?

Joe Liffrig7:09

What are the actual tools? And it took actually at Cornerstone we had a use case that this annoying integration gap between two systems. We have a fantastic kind of front-end marketing piece that John and his team have built where we have a lot of customers saying yeah or prospects come saying like yeah we want Cornerstone to help us with us. This example in particular was we need to get shy bonds. Shy bonds are like a super annoying just kind of compliance thing that everyone needs to get, but they're kind of a commodity and like not sexy, not exciting, right? But we had a lot of people being like, 'Yeah, I need this.' And then Corn has a great system for getting those shy bonds. We have an in-house shy with connections to a lot of underwriting. But there was this little gap of getting the people who are interested to actually getting shy bonds. And so that we just took on the challenge like let's just dig into this and see how we can solve this really quickly and easily. I didn't have space in my road map. Didn't want to get like developer time on it. And so I found an AI coding tool. This is called Replet. There's tons like this. There's base 44. There's lovable. There's cursor. I'm sure you've seen all those headlines, too.

Joe Liffrig8:21

But this one was just dead easy to the point where John, I think I took your prompt here and John didn't develop this prompt on his own and write out all this technical jargon. Like this is AI on top of AI. Like John talked to Chat GPT and said, 'Hey, we've got this gap. Like help me think about how to solve it. Help me make a prompt for a little app to do so.' So, like from probably like two or three sentences out of John and my brain, Chat GPT helped us put together like a pretty decent prompt here saying like what this app needs to do and what it should be. And again, this is a couple paragraphs and AI augmented just helped develop this and put the thoughts together. And from that, we then had like a lot of back and forth of like you have to it's not just a one on one shot thing. You'll give an app like this a prompt and then it'll digest it. It'll do some coding and then like it'll ask you some more questions. So you can kind of see some back and forth, right? But after a couple rounds of back and forth, what we landed on was an app that builds that little integration gap for us. So somebody can come to our bond website now. They can fill in a little bit of data.

Joe Liffrig9:36

I don't know if this is your actual email or anything, but Josh is gonna be my guinea pig. You can choose a few states. You can choose the type of industry you're in. So, we're both collection agencies because that's the topic dour. A couple qualification questions. This was just like sucking in a spreadsheet. We had a spreadsheet in the background. Again, not connected to our system record, not connected to the marketing funnel. And that would basically just building that little bit of logic into this app made it possible to where somebody instead of having to, hey, I know I need an Arizona collection agency bond. Can you just get it for me? Previously, we just didn't have the mechanism. So, it was you have to schedule a call with our bond experts, which is cool. We have nice people to talk to, but like if I just need a bond, let's just make it quick and easy, right? And that puts something like this together. And then this sends an email notification, which was again a simple integration. And I don't know how to do like do code integrations myself, but this tool is like now I need your API key for SendGrid so I can send an email. I'm like step-by-step business level terminology that made it possible for someone like me and John who are not software developers to put together this app and bridge that little gap. So that's the example that came to mind when I read Josh's survey of like this integration gap exists and how simple it can be for somebody to bridge that gap. Solve a problem for your business.

Josh Allen11:01

I can really appreciate that you showed an example because it just made me want to share my screen and just start like whipping out of nowhere. Just to show how easy it is, but to put it in perspective of really how easy it is. What was and you don't have to share specifics, but like say the time investment to like ROI based on like you removed a lot of processes, right? Like intern like little manual ones. And so these are just to like re-emphasize the title of this. This is a risk smart practice. That wasn't diving into like here's a bunch of PII. I'm just going to dump this into like outer space. This is just like a simple process which saved a bunch of time and you probably took you I don't know hour two hours maybe tops 10 hours maybe I should.

Joe Liffrig11:50

Yeah. I the number that come to mind was 10. Yeah. I say we probably spent like two hours with the actual development, but then like all told like we have to go and talk to people, talk to my bonds team, make sure we have the process right, do some testing. We wanted to also have an integration go up to HubSpot so we could track it in our marketing system like all that together like 10 hours of hands-on keyboard work, but like an hour or two of like building the app with the AI. Yeah. So that's awesome that's where the world is moving. And I mean, you didn't have to go tap a developer and pull him off of a project. And then teach them all the subject matter expertise like here's exactly what I want. And again, I think it's much less risky to do that than it is to implement like a AI system of record and just like here you go where you start allowing it to act, which is something that we hear a lot about on the collection side, right? Because we're collection software not the system of record but we do hear a lot about you know dumping in a lot of you know consumer data so where the AI could act and I think that your example was more along my mindset of how AI is a benefit to this industry which is not necessarily consumer facing but operationally like it's an oper it's a it's an enhancement of essentially like your operations team right so there's a joke I made in a business group yesterday. Actually, I think your CEO hit the haha button or something, the laughing one, which was that operation people are now going to have code reviews. Because instead of t, you know, tapping a developer, you can build these little tools inside you know, inside your company to help your job. And of course the big risk there is what if it what if your tool spits out the wrong thing for your own job.

Josh Allen13:46

Like you forget a guardrail and whatnot. But I think what you did is like you're immediately getting traffic and you can test and iterate on that and that's like awesome. That was a really cool example and it looked clean too. Yeah. I'll give John the credit for making it look clean. But I'm curious Josh. Are there other examples that come to mind for you of stuff you guys have done that helps with that like guardrail risk or keeping AI in the places that AI should be and not just like go trying to go too big all at once. Yeah. So I think it's funny. I saw a little meme today which was three or four years ago AI was chat GPT and then it was like AI today is like use this tool for this and this to this and this but these are all still like very siloed tools unless you put them together which is another layer of work. Don't want to dive into that. But the biggest thing is whether you're using it for yourself in the work environment or you're rolling it out to a team is having a base like a file basically that can be shared like this is the guard rails that everyone should use. This is the under this is what our business is. We did this at the very beginning with language but as funny as that sounds like everything every response has to be in like English like US based right like formal American English not this sort of and so like all the responses these are like the little details that you don't think about when you're like doing things like this but you know you may see aluminium instead of aluminum in a response like just because of that.

Josh Allen15:27

And so I think the biggest thing, like again, I'm right now we're looking at rolling how do we roll this out to our operations team, 'cause the developers are using it as far as like code checking in this. We still are definitely hands-on. We're not vibe coding our app at all, but like they're very hands-on. The operations team, of course, who all have like much more independent jobs. I would say it's like okay, let's make sure like if they're going to use an internal scoring model, they all have the same one at the same time opposed to like this, they got a 44 score, why do I have a 75? And so it's like those little things and it's a weird, it's kind of a weird thing. Maybe I'm like a little bit too far down the rabbit hole because I will just say like very plainly, like last week while we were at Armtech, one of our account managers, I got him into this and he built a QBR generator. That was it.

Josh Allen16:20

He can generate a nice PDF, no PowerPoint stuff, nothing. It's just like plugs in some numbers, add some notes, hits the button, QBR generator. Took him like half a day to build and then done. Don't have to go buy custom software for it and it's like fit for our industry and for like our job role, which is like again like incredible when it comes to this. Do you have a framework, Josh, for then that guy or that person had that great idea, spent half a day on it? Is there a mechanism for that tool to then get shared across the org or is it kind of it's that person's tool? So it is that person's tool, but this is where that code review comes in of we are actually rolling out like a governance program type thing internally now, like and I would say as fast as we can because all like this whole thing has been, if you look at the velocity over the last couple years, it just continues. And for everyone watching, I'm not like an AI fanboy. I'm not like someone who's jumped up and down for AI.

17:21

But I do think in the current environment this year in particular the for tools like this it will the velocity will increase. I'm still skeptical of the consumerf facing stuff but rolling out a governance program to where some of the stuff can be shared but not necessarily all of it needs to be. I right like we don't want to report wrong things. Part of the governance program is like no live data, nothing in the cloud if like you're building it yourself. Like our developers and our infosc team are like none of you guys are like putting this in the cloud. Like these are all for you at this point in time, right? To help your job. Like you want to upload a file, it's that file is already on your computer. It stays in your software. But essentially, we're building desktop software, which is the weirdest thing.

18:09

Like it's like a the re like coming back from the cloud and onto desktop software all over again. You can like build yourself powerful desktop software. It's kind of like the mindset behind it. Oh, and what is that to the extent you can share that or that you have already conceptualized like what is the government's framework look like? Like are you adding a new like a governance platform of some sort or is it just policy? So [snorts] we've built out like internal files for specific departments and so like when you start a project you have to use those files at the start of the project. So like you guys mentioned you created a prompt and whatnot like we have that as well like we have like a framework that every one of them has to include. We'll work with them on like touching it up. It's obviously it's a live business like we're a moving business so things may change which means those things have to get updated but that is like the biggest one.

19:04

Because guard like I think as everyone has seen is generative AI can be tricked into like multiple things and so like making it as rigid as possible with guard rails and understanding of the business like our billing cycle it's I built a forecasting app this is like a great example and so in it had a client or a previous client at zero and it's like oh this client's at risk well it's because it didn't understand our billing cycle M and so like having like building it out under that to say like oh this is how the building cycle works. They're not at risk. It's just like a low like you know how they onboard. You know we end up putting like a delay like some like runway delay on the on how the math works in there. And so like little things like that right to fine-tune the accuracy.

19:55

I think the big one of the biggest things is you have to have subject matter expertise. I recently got into collecting coins, like really old ones. I tried building myself a little coin app. I stopped at like the limits of my knowledge, which was like 30 minutes of research. There's no coin app coming out like at all from me at least. And so like I do think that the power is in what you actually know. It's generative AI can't just go do everything. It can't think for you. It doesn't actually think regardless of how good they've made people feel about it. It really like it really only gets better with the like what you can like kind of firm up if you will. I like that insight that obviously it still requires that human expertise for sure.

20:38

Yeah. Yeah. I tried Pokemon one also. It didn't work out too well. So we have a question from the audience I'd love to throw to both of you. From John, how do you see clients approaching redundancy plan provider with so many proprietary solutions out there? Yeah. Not ripping on any of our fellow vendors who are in the ARM tech vendor hub, like there's a lot of booths there that all say we're bringing AI to collections, right? How do you choose between those? How do they work together? And that's like a, this is like a really good question I guess also. Yeah, there's that like there's a lot of noise and also yeah, I mean correct me if I'm wrong in this but I think it's like yeah like if everyone's spinning up their own like personal solutions then like consistency can waver also or even if you yeah that is a good question.

21:41

One thing that comes to mind for me maybe Josh pondering that is the buy versus build paradigm I think has shifted a lot in the last just very short term last couple years where I think two years ago my mindset was if there's you know if there's anything that's remotely related to the use case I need on the market I'm definitely buying it because building something custom is super expensive, buggy, not worth it and that still is often the case but back to that kind of like if it's something small, something connective or something that something finite, consider if you can build it and bridge the gap that way and not have to sort through the redundancy across all the proprietary solutions out there.

Josh Allen22:25

Yeah, the bridge the gap is connect the stuff you have. Yeah, connect the things you have. Yeah, I mean, I do like that question. It's that's a very like real question. So, I don't know. The only use case I can the use cases I can think of are the ones that we experience. And so I'll go back to like my little my forecasting app. The data just goes right to our accountants and it goes into QuickBooks. So I'm not like which is the real source of truth. And they check it and if things don't match up I'm tuning my side. They're not necessarily tuning their side unless there's like a real reason like there's been like whatever like some something's miscategorized. My numbers was really I really started it actually to reformat our reports so that I could give it to them without having to use like pivot tables. So I was like why can't I just like upload my reports all the different reports of this one thing have it come out as one export in one thing and then just give it to them and so the proprietary solution thing it was really like a job that I was already doing and so and taking a lot of time and I just was like I got to speed this up. I think also since we're a software company on our side, what we've used it for on the software side is like live you guys actually use something on your lead gen form and that's awesome.

Josh Allen23:41

And that then but we've used it for like almost mockups like if I'm trying to describe a feature to our CTO, I can now just have like the feature like built if you will. Then I send him a video and then I send him the feature in a zip file. He opens it on his computer and then he's like, "Okay, I understand." And he trashes it. We do not use it. And then they will rebuild it the right way. And so it's almost like think about all that back and forth. If you want something done with someone, you have to make them understand what you're thinking. You have to like paint that picture, do all these diagrams. So I guess going back to that thing, it's it is it's like connector in a way. So it's a great translator.

Joe Liffrig24:18

That Joe you mentioned that you use it a lot for prototyping to then give to your dev team as opposed to all of the back and forth that you're describing Josh of like no actually I wanted the button to be here I wanted the language to say this I wanted you know the flow to be like this like instead you can you don't have to fix all of the business logic on the back end but you can at least have like this is what I generally want it to do and look like and then give to them and then they can actually make it work the right way that you know building it in the way that's you know in line with the code standards and things like that you have at your organization. So it allows you to prototype quickly and then I think of even for like proprietary solutions like if you can like if you're looking for an improvement to a proprietary solution you could essentially go to replet or something like that and basically prototype exactly what you're looking for and be like hey this is the gap we're running into with the solution that we're using from you. I prototype what like what we are looking for and kind of the giving that to people for improvement ideas.

Josh Allen25:26

Yeah, I would be skeptical of what you apply it to always like prototyping is great. I wouldn't recommend someone go build like a whole system of record themselves and things like that and then everyone has their own individual system of record and but I do like the redundancy plan and provider. I do no matter what I think there's always has to be a source of truth. And so if my little forecasting app went down, you know, we always have the old way. And again, it's like kind, you know, we use the we use QuickBooks, so it's still there still is that process. Yeah. One other thought coming to mind from John's question is that we use to evaluate redundancy between providers or who's the best fit is the integrate ability integration ability evaluate how much they're trying to up to what extent it's a walled garden and this proprietary solution is going to say sorry you can only do what's in here versus here's our integration marketplace here's our API documentation because even if you as John RER are not like reading the API documentation like the fact that they have thought about that and they have the ability to then extend and connect prevents you from getting into a situation where you're relying on that one vendor or you know their redundancy you can you have a lot more optionality yeah AI is incredible for like things like documentation and like flat file like reading data so even someone who's non-technical can like drop in an API like documentation like just a link to it and say like hey tell me how this works and you know what I possibly would need to do or pass to a development team to get something done and it could like spell it out in like plain English obviously again if you're non-technical like it has to be always keep a human in the loop on those things but it still is like a great equalizer when it comes to like a communication gaps which is communication is one of the hardest things to master right and so this like generative AI is pretty good at that like you know with what again with what you provide So, a question comes to mind based off of you're talking about the Josh that obviously we talked a lot of things that you can use AI in great ways for, but I'd love to talk about some of the things I would you might avoid or you know things that sound great you know in a demo but tend to break in real world environments or you know like expectations you were mentioning like you know don't build your own system of record. That would be a that'd be a difficult thing to get right and build.

28:01

And so kind of where do you see some of the red flags or the kind of like the wouldn't recommend pushing into areas for AI for kind like the business logic there. All right. So either I'm going to eat my words in a couple months to a year or years from now or I'm going to have enemies like as soon as this ends and people see it. But so how I look at software in general is that software really performs like repeatable functions over and over again. AI still has like so many black holes. And again the stories of like yeah you know we were able to trick the chatbot to give us a free car on the dealer's website you know and now they had to honor it because it took a court. Things like that pretty extreme. Those are early like cases that like did happen. Good for those people for getting, you know, I think they're like a free truck or something. I do think that things that like generative AI, like even chat GPT, just you pull up the voice thing and you can talk to it. They use demo amazingly well, but specific to an industry, there's so many nuances and edge cases and everything is evolving all the time. And so in software like again how I look is like when we see a bug happen it's never a one time bug like even if it only happened one time that might be an extreme edge case out of millions and millions of interactions.

29:30

However that now that can be repeated again and so when it comes to the generative AI that risk exposure expands because it's so fluid. And so I'm very cautious of like outward communication unless it's something like hey generate me some email templates and of course let me add my own stuff because then those become static. The AI is not like going back and forth in email template in real time and modifying it. But I do think and I know this is like the big topic and we saw a ton of it at harm techch is I do think like the AI voice bots like I'm incredibly I'm a big skeptic of that. Because I do think that you can roll those out to an awesome few couple days and then all of a sudden you start seeing the edge cases come up and then how fast can you remedy those before it actually is like a problem and maybe it's not. I do know someone not in the collections industry who created one to field calls but they gave it the only thing was do you need to check your balance or your get your invoice it's a moving company or just talk to a rep. So like you could kind of do that anyway but because it can read information so fast he just gave it access to the invoice database so we could just pull that up and that's it. So I do think that there's the risk in the call it like two-way AI rather than like a one-way messaging push out.

30:48

And when it is up the little insight I like on the end of that is that when it is a two-way or when it is client facing like keeping it very finite very guardrailed and with a human escalation path. If you can bite off those 20% of easy questions great that's awesome but if I think about the support the AI support experience has worked well for me it's ask the question does this article answer your need no okay here's the you know here's the human rep. Yeah, it doesn't mean it can't be like it doesn't mean it shouldn't always be avoided, but yeah, like if especially if it's a vendor, well, you know what? It doesn't even matter if it's vendor or not. Like the LLM models are you know, borrowed models essentially. And so it's so easy to set up that to me is like always a concern because MVPs are very easy to set up and getting early traction on any software that you roll out is really easy because it's usually very like it's not like there's like it's just like one feature or one thing. It's as soon as you like start to expand on that one thing that things start to fall apart. TRA started out as a single transaction report. That was it for reconciling transactions. Really awesome traction right away. Soon as you start adding more things and it becomes its own ecosystem, like you have to like understand it has to keep up with itself somehow.

32:17

And so the like margin of error just there's a lot more I think with when it comes to LLMs like in that sense. And again, my not Yeah. My big thing is like I don't know the whole like bajillions of records and you're talking to one of them and all of a sudden like it's like yeah, oh no, whoops. You're not that person, but I just told you all of their information. My bad. And it's like that's like the whole being pleasant and nice and everything as they're being trained to be. It's like, oh yeah, I'm just going to go back on track, but really like that could be a third party disclosure. Who knows? And so again, that's like my big like warning, if you will. And how do you test it? John, one other example that comes to mind for me is building AI into deterministic systems rather than having to try to be just AI end of the rainbow all on its own. So one thing we've attempted a few times just kind of as like a risk mitigation or as like an exploration is like hey chat GPT go and get me all the licenses I need is a licensing example and like I'm this kind of business doing this thing. And it will get something that looks plausible but if you scratch below the surface it's like it's nowhere close.

33:32

It does something that kind of looks like yeah this would be the list of licenses you need and kind of how to get them but it's nowhere close to that and just on its own like that's it's not useful really. Where we have found a lot of traction is taking that same approach and saying hey AI go and get me for this target industry go and collect all the licenses in all 50 states and the steps to get them and plug that into this deterministic like into this JSON payload this format. So it's telling the exact target you want and then point it into a system where our human experts can then review it and it's a huge accelerant and make sure that our licensing database is up to date in a way that was way faster than having a human go to each website individually.

Josh Allen34:20

But it didn't just say, hey AI, go do the thing for us. It's AI, accelerate the thing for us within the framework, the deterministic framework. So I tried to do the, I did a little experiment with this last week where I said, "Hey AI, can you update this one model for me?" Like just move it from here to here and it would have taken me maybe one or two hours and it took Claude 48 hours. And I actually had to keep interrupting it because it was like going in the most inefficient ways and I was like this is probably never going to work if I'm like, hey just now go here. Like it's way more efficient. Like it was trying to write code in a little box and it was trying to scroll down in the box to write it and then every time it would scroll it would forget what it was above so it would rewrite it. Nope, I wrote it twice and it was getting so I was like okay yeah just like don't do it there just do it outside of there and whatever thing you're, you know, paste it in it was like great idea so it finally finished after 48 hours and then yesterday I checked on it, this is like a little internal just project and I went and I realized like it made up half of it just to get it done and I was like, "Oh, so 48 hours to even get it done." And so I had to go back and correct it all and it was like it was actually really funny to have done. But yes, without like the very specific context, it makes it feel so easy.

Josh Allen35:40

And it's like I don't know sometimes like again the larger application you have the cooler it feels because you're getting so much done faster and then all of a sudden you realize something just ran for 48 hours and actually didn't do anything. But I thought it did. It sure looked convincing. It did convince me and so I did look into it. Yeah, that's funny. So one other question I'd love to get to, you know obviously there's a lot of ways you can use AI, there's a lot of ways that your organization can use AI but I feel like without kind of proper operational guidelines and guard rails, you know you could be using it in a way that's one, you know you're talking about Josh the data governance there of making sure that everyone's using the same types of things but also just like where people are using it, how they're using it. What comes to mind for both of you on how to make sure that the whole organization is doing things kind of in line with what you'd want as opposed to, you know for instance the easy one is always don't use the free version of ChatGPT and put client data in there because it trains the model off of the free version. But outside of those kind of like that basic one, what other one comes to mind for ways to kind of structure those policies internally to encourage the use of AI while still making sure that it is used appropriately.

Josh Allen37:06

So we have another one where so again the biggest thing that a lot of these models are doing now is like interconnectivity. So you can just go in and connect something like connect my Gmail, connect this, connect that. So we create like a fake user in our, we use business Google business suite, whatever the name is they changed it I think recently, so the Google business email one. So we created a fake user in there that's the only way we could accomplish this. And then in that user we created like their Google Drive and put documents in there and that's the only thing that some of these have access to. And so if like so our, you know, people they can't plug into like their own Google Drive, they can't plug into their, they can only plug into this like one fake drive essentially and that's the source of information that they can pull from. Like basically little things like that, they don't make it that easy for other software like Notion. They're like it's all or nothing.

Josh Allen38:09

But that's like one of them I do remember. Yeah, like that's a, that is a big one. And also just like what you can and can't plug it into, what you can and can't use it for. I do think it's just so easy to use it for everything that it's like, I mean someone, one of our guys sent me a game he made yesterday. I think it was called like the collector trail, like the organ trail before. And I was like, well, I was like, yeah, that's not that wasn't really the point of doing this. And it was a terrible game. Yeah. But so like, yeah, like you know, like don't go, don't spend half your day building a game type things. But yeah, like again, the governance policies and what you can put into it and what not. I do think for non-technical people like the form is actually a really good use case. I'm, I got to give it to you on that one. But like data that interacts with and it comes down to like this whole like I think a lot of people like we can build, we can replace HubSpot now, we can replace our system of record and then all of a sudden you like throw that in the cloud and it doesn't have that same security guard rails that you would like if you had like, you know someone looking over everything. So I think having like highlighting, yes, it's easy and fun, but you didn't conquer the world just because it looks nice.

Joe Liffrig39:30

But I think building on that, Josh, one other paradigm shift over the last year or two is that when we were talking about AI internally a year or two ago was like how do we convince our specialists to use AI because it can be such an efficiency gain. And now literally on Friday, I had one of my specialists say like I need to use an AI tool so that I can work efficiently. How should I be doing it? And so I point her towards here's our AI policy that we published like a year and a half ago now. Have to be reminded of that. So John, back to original question. If you don't have an AI policy, you should because people don't have to be persuaded to use AI anymore. They are using AI to say, you know, hey, I've got these ingredients. What should I cook for dinner tonight? They're using AI for, hey, make me a funny game. They're using AI for it's become consumer level and becoming second nature where they're when they're at their desk, they're saying, "Oh, I have this thing I need to do. I'm going to ask AI first. I've outsourced part of my brain there." So, if you don't have a policy, huge liability and like, "Hey, chat GPT, write me an AI policy. Make sure to include these three bullet points." Like, it is not complicated. And that has to be table stakes starting point.

Joe Liffrig40:35

And then also having an answer when they say, "I need my AI tool. What do I do?" For a while, we were like getting individual chat GPT like enterprise accounts for people because it has all the guardrails for data security. Making sure it's not been used for model training. That's great for first step, but like gets expensive kind of quick. It doesn't have quite the level of functionality that we want for like you were talking about Josh like pull enterprise data from these systems and not these or only these sections. So like right now we're currently vetting platforms for what's that what's the one just like operational AI tool that everyone can come to can have the shared memory so that our specialists were asking the same questions can you know can fine-tune those get better we can give them prompt ideas we can put like system prompts in there and help them ask better questions and we can curate the like our company knowledge base that it uses. And just making sure that you have a really rock solid answer to your employees of this is the AI tool for you to use and make sure it's a relatively good experience. So they're not regardless of what your policy is, they're not tempted to go outside and say, "Hey, take this data set full of proprietary information and summarize it for me or something because they are going to do that."

Joe Liffrig41:46

So you have to have a good answer that's streamlined and easy. Yeah, 100%. I think that is very true and yes, having a middle layer of it like the Google Drive we have is great, but actually having a real central layer is better. To where everything could just plug into that and then it's like even another step removed which is good and then you can control it like everything that goes into it like we can't put everything on the Google Drive. And so that's actually yeah that's like a really that's a great point. There was another thing that I had read of how like if you know your employees are using it like it's obvious and a company had mentioned that with theirs it like instead of saying like I know you're using AI and it's like they just say like look if you're going to use AI create a sharable link to include with it and then like filter it through. But if you think about it all of a sudden you're now creating even more work but it's like because the this company in particular wanted to see where are they prompting it to get some of these answers in order to like okay yeah so that like this is why and it was like a it was more of a working type thing to get them to be better at it.

Joe Liffrig42:54

But I've done that a few times I'm like okay yeah I see what you're say because again AI can be very convincing and so all of a sudden you get these debates but you don't know the what they maybe said in the background and so it's you know cite it. And then we yeah we also did like reports weekly reports now with the development team. It's all summarized by AI, but it's just a PDF. So, one of my things was like, hey, can you have it not only summarize the report, but have all the sources that it brought the information from cited so I can just click the link and actually see the real source in case like it's questionable.

43:33

And so I think those are like again the world is developing so fast that those are like not bad simple ones but your the internal brain one is like becoming much more of a reality which is incredible. It's like yeah like it's a crazy world now. Yeah. Like we use SharePoint for our like internal knowledge base. So we have SharePoint and then we have like our apps like SQL database of just not our client's proprietary information but like our proprietary like what licenses are needed where and so like talk to me in a week and I probably have a better answer here but like we're stitching searching either of those is a bear you either have to do SQL queries over on this one or like use SharePoint search which is somehow the worst in the world and terrible to use like and so people always just ask me for answers that are in the knowledge base. We're putting those together, put it into one platform where they can ask them. And then I love your comment about like saving the like what questions are being asked and so if you have a platform they can save the questions and understand for real like I don't know how well any of the listeners would understand like what questions are the people on your team asking and do you have a way to capture those and then like answer them better. That's something I don't have a great answer on. Again, I think in a week or two I'll have a much better answer, but that's something we're focusing on like how do we capture those questions and then answer them better so they're not going out to chat GPD and answering them in the wild unsighted.

44:56

Yeah. So, it's funny actually on that it's even how you use it like beyond the question. So, I am a person who has a million tabs open on my laptop. So, every time I talk with one of these, I'm starting a new conversation. Unless it's like a continuation of one like everything I'm like segmenting it. If I ask like what food to buy my dog and then all of a sudden like I start a new conversation in that same one, I'm like, "Oh no, like I'm lost." So I start new. And yet there's some people on my team I realize, can you send me like that transcript so I can see it? And I realize like it's the same window they've been using for like Yeah. For like a year and a half. And so like the context may like and I'm like okay, you should be like starting over like again you have to like almost provide it like a fresh foundation. I don't think there's a it's just like a totally different perspective and I was realizing that someone was giving an answer and I was like where are these words coming from? Like these are like not our industry words and it was like oh yeah I was like start it over you got to start it over and so the use of it because that's the one thing that they haven't done yet is they don't they haven't done a very good job these companies and I know it will come out of providing like in-depth insight into like what is going on like you do have like ability to get it but like in Slack or Teams for example you can create channels and you can't really go jump into somebody's AI channel, if you will, to look at this. And I could be completely wrong and there could be somebody who's already doing this.

46:23

But I do think that like yes, it's the questions and the how and whatnot. I think all those influence it. Absolutely. Yeah, that those are great ones. Joe, I love the internal brain kind of thoughts there. And one of the other benefits I see of that is that even if you are let's say having each person have their own individual kind of chats if you have the internal brain that you or your operations team get to control then if there are discrepancies you can go fix them by adding like if you are asked this question this is the answer. I know it says this online that this is the answer but this is the real answer. And so you can kind of like set up the like the exact guard rails of what you want it to say when certain specific answer questions are answered.

47:16

And then you just have to do it one time as opposed to you have to, you know, tell people, okay, if it says this is actually what you want to say or like this is how you prompt it properly or something like that. So we're almost out of time, so I'd love Josh and Joe you know and if there's any other questions, please let us know. We can always follow up with you afterwards. But Josh and Joe, if there's kind of one additional point or kind of key takeaway that comes to mind for you on making sure that you know that the technology and especially the generative AI is you know assisting a lot of these debt collection agencies and you know how they can turn these awesome you know this awesome tool into something that really makes a difference towards their business.

48:11

Who goes first? Rock, paper, scissors. Josh, PT, who goes first? Flip a coin. Let me make a quick app that a coin flip app and then coin flip app. So I think the most I do Okay, so I think the takeaway that I came away with even just after this last week, which was the first time we started like pushing the like a cohesive operating team type use was that this is one it's new to everybody and so like no matter who you are, it's just new to everybody. It's new to technical people. It's, you know, it's still like relatively new. We're still on the ground floor. In that less about what the actual application is, I found it fascinating to see who on our team jumped in and adopted it and who on our team said, "Well, I don't have time or I'm this and that." And who like turned away from it. And that was very enlightening to see like the difference in curiosity.

Josh Allen49:19

And like how almost the world would move going forward. Now that's a very meta like way to think about it obviously but those ones who were really engaged in it and who came forward kind of like they built their own little examples like I wouldn't have thought of that we can now share with the rest of the company as kind of like a standard. And so that would be my one takeaway like you know give a small group permission to use something and say here and like I'm not going to lie like our first example they everyone was like what do we do with it and I was like tell it to go organize your download folder tell it to go organize your trash on your computer and just like that's it and don't connect to the internet. That was the other part. And yeah, like there was a massive difference after a week between who took it on for something he would actually use and then who didn't. And again, that would be like my one big takeaway is just like pick three, four, five people non-technical operations side because I do think that is where this will have the biggest impact the fastest.

Joe Liffrig50:28

I've got a similar thought, Josh, but I'm going to take it one step further. Don't well go ahead and delegate to people. Do exactly what Josh said. And also, if you haven't already, do it yourself. The single best thing that I did to understand AI in the last year was to take one little challenge. And if you're if it works in the work context and you have the permissions and whatever, great. If not, do it in the personal context. Spend 50 bucks and get a, you know, find an AI coding tool. Find one problem. Even if it's as simple as build the organ trail of collections game and just go and try to do that building yourself and get from an idea to something where you can publish an app to the internet and share it to share like share with your friends and family. Say here's my organ collections game. You will learn more from that and will take the mystery out of it and take it from being a pie in the sky. I need it to talk to my developers or I need some fancy vendor to sell me a million-dollar solution to AI is cool but it's not that complicated. There's large language models. All they're doing is calling, you know, calling the same model and it can do different things. Do it yourself.

Joe Liffrig51:31

Take the mystery out of it. Go from there. Yeah. And don't get caught up in the hype. The hype will be very expensive in the long run. Sorry I had to say that. That's why you have to go try it yourself. Just to re-emphasize why you should do it yourself to see that it's easier to do it's almost easier than getting caught up in the hype to experiment with yourself and seeing the limits. You just got to the first step in I love that I made a note don't do AI for AI sake was my other takeaways that keep there's an old line in product management like fall in love with the problem or the opportunity. Make sure that you know what problem you're solving or opportunity you're grabbing and not just saying I saw something on Twitter or I saw this cool demo so I'm going to go spend a bunch of money on it because like chances are you already have a solution in house.

Joe Liffrig52:17

There's a solution that isn't AI powered that does it even better or like there's a I can just use chat GPD for this. I don't need to pay you know something extra for it. Yeah. And make sure you follow the links like you said earlier. Don't have it write like a legal argument for you for cases that with backed up by cases that don't exist. Yep. That was that's one it's definitely done. It does hallucinate. As Sam Altman, the CEO of OpenAI, likes to point out, it is not infallible. Please do not think that it does not make mistakes. It makes mistakes all the time. Yep. I've heard of people getting invited to Zoom meetings with it just like this one. And of course, they click the link. It's like, "Oh, nope. Sorry. Actually, I can't actually host a Zoom." Like, what? Where did that come from? But yes, that is a that is actually something that I've heard has happened to someone where it invited them to its own Zoom meeting to help them build something.

53:11

Yep, absolutely. Well, thank you both so much for joining. Please, as attendees, please go download Josh Andrada's excellent survey that was done. It has a lot of great information in it. And please be on the lookout on our social channels for our next webinar in February. It's going to be with some tax professionals helping to answer a lot of questions related to taxes for your business and kind of demystifying that area. So again, thank you so much for joining us and we will see you next time. Thank you.