AI Hallucinated a Dependency. So a Cybersecurity Researcher Built It as Proof-of-Concept Malware

'Several big businesses have published source code that incorporates a software package previously hallucinated by generative AI,' the Register reported Thursday

'Not only that but someone, having spotted this reoccurring hallucination, had turned that made-up dependency into a real one, which was subsequently downloaded and installed thousands of times by developers as a result of the AI's bad advice, we've learned.'

If the package was laced with actual malware, rather than being a benign test, the results could have been disastrous.

According to Bar Lanyado, security researcher at Lasso Security, one of the businesses fooled by AI into incorporating the package is Alibaba, which at the time of writing still includes a pip command to download the Python package huggingface-cli in its GraphTranslator installation instructions. There is a legit huggingface-cli, installed using pip install -U 'huggingface_hub[cli]'. But the huggingface-cli distributed via the Python Package Index (PyPI) and required by Alibaba's GraphTranslator — installed using pip install huggingface-cli — is fake, imagined by AI and turned real by Lanyado as an experiment.

He created huggingface-cli in December after seeing it repeatedly hallucinated by generative AI; by February this year, Alibaba was referring to it in GraphTranslator's README instructions rather than the real Hugging Face CLI tool... huggingface-cli received more than 15,000 authentic downloads in the three months it has been available... 'In addition, we conducted a search on GitHub to determine whether this package was utilized within other companies' repositories,' Lanyado said in the write-up for his experiment. 'Our findings revealed that several large companies either use or recommend this package in their repositories....'

Lanyado also said that there was a Hugging Face-owned project that incorporated the fake huggingface-cli, but that was removed after he alerted the biz.

'With GPT-4, 24.2 percent of question responses produced hallucinated packages, of which 19.6 percent were repetitive, according to Lanyado...'

Thanks to long-time Slashdot reader schneidafunk for sharing the article.

Read more of this story at Slashdot.