'AI Prompt Engineering Is Dead'

The hype around AI language models has companies scrambling to hire prompt engineers to improve their AI queries and create new products. But new research hints that the AI may be better at prompt engineering than humans, indicating many of these jobs could be short-lived as the technology evolves and automates the role. IEEE Spectrum: Battle and Gollapudi decided to systematically test [PDF] how different prompt engineering strategies impact an LLM's ability to solve grade school math questions. They tested three different open source language models with 60 different prompt combinations each. What they found was a surprising lack of consistency. Even chain-of-thought prompting sometimes helped and other times hurt performance. 'The only real trend may be no trend,' they write. 'What's best for any given model, dataset, and prompting strategy is likely to be specific to the particular combination at hand.'

There is an alternative to the trial-and-error style prompt engineering that yielded such inconsistent results: Ask the language model to devise its own optimal prompt. Recently, new tools have been developed to automate this process. Given a few examples and a quantitative success metric, these tools will iteratively find the optimal phrase to feed into the LLM. Battle and his collaborators found that in almost every case, this automatically generated prompt did better than the best prompt found through trial-and-error. And, the process was much faster, a couple of hours rather than several days of searching.

Read more of this story at Slashdot.