Most manufacturers have a stated AI strategy. Fewer have thought about what happens when AI operates inside a system that doesn't tolerate approximation.
ERP is deterministic by design. Input A produces output B, every time. That logic is why your financial statements reconcile, why cost rollups survive audit, why Monday's production schedule still reflects reality on Friday. Third-party AI is probabilistic. It produces the most likely answer. Those two things are incompatible in the same critical workflow without governance between them, and most manufacturers have none.
Shadow AI Is an Operational Architecture Problem
Most conversations treat shadow AI as a cybersecurity issue. It is that, but it's also quietly building a second operational layer no one controls, audits, or can turn off. Personal AI account usage among employees dropped from 78% to 47% year-over-year, but the average enterprise still saw roughly 223 incidents per month of users sending sensitive data to unvetted AI. Fewer people, more damage each.
The analogy is Shadow Analytics: employees pulling corporate data into personal spreadsheets to bypass IT. Same pattern, higher stakes. A spreadsheet doesn't learn from what you feed it, but an LLM does. When an engineer pastes a proprietary BOM into an external model, that content may enter a training corpus. There is no delete button for training data. Breaches linked to shadow AI in 2025 cost an average of $670,000 more than standard incidents, largely because ingested data is irrecoverable.
What Probabilistic Logic Does to Deterministic Systems
Consider demand forecasting. An AI model might hallucinate an order spike. Without a human-in-the-loop control or deterministic guardrail, that signal triggers automatic purchasing of excess materials. Inventory overstocks, working capital ties up, and the quarter's cost rollup reflects a decision no one can reconstruct. The reverse, a hallucinated demand drop triggers supplier cancellations, stockouts, and failed commitments. The governance rule: if you wouldn't trust an intern to execute unsupervised financial decisions, don't trust a third-party AI tool to execute alone either.
The Case for Bolt-On, Taken Seriously
There's a real counterargument. Specialist vendors often ship faster and build genuinely better models for the narrow thing they do. A CIO choosing best-of-breed for speed is making a defensible bet. But that argument is right about capability and wrong about where risk lives. The real distinction is advisory versus execution. A superior specialist model used as an advisor, behind a human decision and inside a governed boundary, is reasonable architecture. The danger is the quiet slide from advisor to actor that happens when an unvetted tool gets read/write access because someone wanted to remove a slow human step. "Build it natively" isn't the answer either; native AI that executes without guardrails carries the same risk.
Compliance Windows Don't Flex
For food and beverage, pharma, aerospace, and life sciences, audit risk has timelines attached. FDA's FSMA Rule 204 requires complete traceability records within 24 hours of a contamination request. If an AI tool manages lot-genealogy outside the ERP's governed environment, the audit trail doesn't exist where the auditor looks. The 2024 listeria outbreak at a well-known meat producer, which led to nine deaths and an existential recall, was a traceability failure at the moment answers were needed. Pharma's 21 CFR Part 11 requires immutable, repeatable outputs; a probabilistic system producing different decision paths for the same batch approval fails inspection. GDPR Article 30 exposure runs up to 4% of global turnover or €35 million (~$40.39 million USD). There is no flexibility in compliance.
The Attack Surface You Created
Manufacturing has led all sectors in cyberattacks for four consecutive years, and every manufacturer must be vigilant to reduce their exposed attack surface. However, every unvetted tool with read/write ERP access is a new entry point with operational authority. For example, a compromised AI agent with procurement access can poison demand data, sending excess orders out while an attacker sits inside the network for weeks - an unvetted tool with ERP access providing an entry point with uncontrolled procurement authority.
The Architecture That Holds
The pattern is consistent across compliance, financial control and security. Probabilistic recommendations and intelligence is invaluable. Probabilistic judgment and execution with unsupervised authority over a deterministic system is the exposure. The architecture that works keeps intelligence close to governed data and every consequential action attributable, logged and tied to a version, a human identity, and a defined scope of authority.
Your ERP vendor’s approach should build toward both: a native decision layer above the transactional system, and action-auditing that logs activity and decisions executed. The test for any vendor is whether the architecture keeps probabilistic judgment governed and accountable, or lets it execute in the dark. Integration is where the control is, with a protective moat around your proprietary intelligence.
