📦

MODULE 07 · TECHNICAL STAFF

The Package That Wasn't There

Scenario
A developer uses an AI coding assistant to scaffold a new integration. The AI suggests importing a package called learner-sync-utils — it looks plausible, fits the naming conventions they use, and comes with confident example code. They install it. The package exists — but it is not the one the AI described. It is a malicious package with a similar name, uploaded specifically to exploit AI hallucinations in dependency suggestions. It exfiltrates environment variables on first run.

Hallucinated dependencies: a real and growing attack vector

AI coding assistants generate package and library names statistically, based on patterns in their training data. When a real package does not exist for a described use case, the model invents a plausible one. Attackers have begun registering packages matching common AI hallucinations, turning a model error into a supply chain attack.

This is not a theoretical risk. It has been documented across npm, PyPI, and other registries.

The wider landscape of AI development risks

🔍Hallucinated APIs and methods — AI may suggest function calls, parameters, or endpoints that do not exist, or that exist but behave differently. Code that appears to work in a test environment can fail or introduce vulnerabilities in production.

🔒Security anti-patterns — AI-generated code frequently includes known security weaknesses: hardcoded credentials, insufficient input validation, insecure defaults. The code looks fine but contains patterns a human reviewer trained in security would reject.

⚖️Licence and IP risk — AI models are trained on public code repositories. Generated code may reproduce GPL-licenced snippets, creating licence compliance obligations you may be unaware of.

⛔Prohibited uses — generating exploit code, obfuscated payloads, or tools for bypassing security controls is prohibited regardless of stated intent or framing.

The code review requirement

AI-generated code is subject to the same or higher standard of peer review as human-written code. Reviewers must specifically check for:

✓Verify every package and dependency exists in the expected registry before installing

✓Check API calls against official documentation, not just the AI-generated example

✓Review for security anti-patterns: injection risks, hardcoded secrets, weak crypto

✓Confirm licence compatibility of any suggested dependencies

Never submit real data to AI during development
Test data must not contain real learner PII, real credentials, or real API keys. This applies even in “temporary” or “just checking” contexts.

Key takeaway

AI coding assistants are genuinely useful — and they make mistakes that are easy to miss because they look correct. Verify every dependency, review every security surface, and never lower your review standard because the code came from an AI.