The fourth edition of the “Real-Life AI” series explores pitfalls, legacy hurdles and unexpected challenges once the technology graduates from experimentation to deployment.
What surprised companies the most after deploying AI?
Responses have been edited for length and clarity.
It is very different from deploying software and the skillsets you need are very different. The big thing is that you have to treat AI like an intern that has a Ph.D., who is new to a role, has a lot of capabilities, could be incredible but we need to spend a little bit of time getting up to speed. As you deploy AI, on day one, it might not be working. Then it makes a lot of the gains very quickly in the first couple of weeks. After about two to three weeks, it will get to a point where it could complete most of a workflow. What surprised us was the need for one of our engineers to sit with customers to understand their workflow and configure agents for those workflows. You literally cannot deploy any smart AI without this process.
In general, we assume that people are excited to use it. I know many people’s change management (resistance) dwarfs the technology adoption. Many companies enjoy 30-plus-year tenured employees that are not easy to convince to click a button to save some time. Many AI solutions work great in a clean, good data environment. In 65-plus-year-old companies with 20-plus different data sources all over the world with spelling errors, the old rule, ‘crap in, crap out,’ still presents a hurdle. At igus, we had to first invest in a new ERP system and global data management solution to be ready for more. That step many companies try to skip and then the results can only be limited.
The ability for Claude to generate visual information, whether it's graphs or being able to analyze multiple sources of data and simplify it. But on the other hand, where you point your AI is so critical. For example, we had 17 tabs on some of our revenue reports. So I said, ‘I want to do a quick analysis for each business unit. Show me the financial variances and where are the key variances driven from.’ But it gave me wrong numerical numbers. It did not know which tab to pick from because there was so much contextual information that is not available to Claude. Unless you point in the right manner, contain your stuff and then do small experiments instead of making it really open-ended, I won't say it's hallucination, but you can get very bad and wrong answers and they are confidently wrong.
I think many companies expect world-changing, revolutionary changes in the way they do business by integrating AI. But the reality is that most functionality today really shows their benefits in incremental gains. I think accepting that and using it to free up time for critical roles rather than forcing AI adoption where it might not make as much sense may be a bit of a wake up call for some.
If you would like to share how AI has impacted your business, contact Nolan Beilstein at [email protected]
