Study finds AI-generated content performs poorly in search

An experiment using generative AI to produce a whopping 2,000 articles and evaluating the results over 16 months found that all that AI content generated a measly 1,062 clicks

When generative AI was still fairly new, an entrepreneurial friend of mine saw an opportunity. He would create a new site, in a niche space, and use generative AI to generate a steady stream of new content for it.

The planned business model? Advertising, of course.

It didn’t last long. The site was unable to produce meaningful traffic or engagement.

>>> Subscribe to receive thoughtful weekly blog posts by email

An experiment across 20 sites

Bogdan Babiak, and the team at the SEO firm SE Ranking, performed a similar experiment, as published in Search Engine Land, on a greater scale. They started by purchasing 20 new domains across 20 different industries, including business, technology and industry.

Importantly, none of these new sites had “backlinks, domain authority, brand recognition, or search history.”

Next, they identified 100 long-tail keywords for the popular “how-to” and used generative AI to generate 100 articles for each site. That’s 2,000 articles in total.

Finally, they added these sites to Google Search Console, submitted site maps and “we left the sites untouched to observe performance over time.”

Short-lived results

In the first month, about 71% of the articles had been indexed by Google Search, which Babiak calls “notable” for domains with “zero-authority.” It generated ~120,000 impressions and 244 clicks – which my math works out to ~.12 clicks per article.

The results aren’t much, but aside from setting up the experiment, the effort wasn’t much either. A well-researched blog post easily requires 4-6 hours of time or more. Generative AI requires a prompt.

That’s a promising start for a brand-new site – especially with no other promotion other than organic search. But it didn’t last. By the six-month mark, the sites had collectively earned 706,328 impressions and 1,062 clicks.

Divided by six months, that works out to about the same level of impressions and clicks each month over that time frame. Overall, it works out to about one click for every two articles.

A year later, and full 16 months after starting the experiment, those sites earned another 300,000 impressions and another ~381 clicks. In total, the sites gained 1,092,079 impressions and 1,381 clicks.

It just goes to show what many of us in marketing, who have been paying close attention to generative AI, have been saying for a while: “good content” may be subjective, but generative AI doesn’t come close to meeting the standard.

Why content generated by AI doesn’t perform

Babiak provides some sound analysis for why the content didn’t perform, including the following:

“No backlinks or external validation.”
“No authors, credentials, or real-world expertise.”
“Much of the content resembled what already exists.”
“No internal linking, topical organization, or clear hierarchy.”

Datelines and bylines have been important trust signals for branded content for a long time. Yet the biggest factor in my assessment is the third bullet.

Generative AI works on probability, so the content it produces is statistically driven. It is not the best content, nor the worst content, but average content.

No one is going to bookmark, share, subscribe or revisit a site with average content. That’s especially since they can prompt AI for themselves and get a more personalized answer, and drill down on areas that are of their interest.

Accelerating the sea of sameness

Marketing and PR professionals working in B2B technology circles know that the “sea of sameness” has been a problem long before generative AI became commonly available. Too many companies look to see how their competitors describe things and rush to match the language.

The result has been a “sea” of content that all sounds the same. No flavor. No distinction. No informed viewpoint. All repetition. The results are telling – prospects and customers can’t understand what distinguishes one company from another.

So, what do they do? They ask trusted colleagues. They look at analyst reports. They stick to brands that are familiar – because familiarity is safer. No one is going to take a chance buying an unproven product from an unfamiliar company that sounds like they do the same thing as everyone else.

A math analogy for marketing communications

I was always capable of performing well in school as a kid growing up. I never had to work too hard to make decent grades.

That changed when I got to high school. Algebra, in particular, was challenging for me to wrap my head around. The teacher was a bit aloof, and I, as a teenager, had a lot of competing interests.

I didn’t fail the class, but I was required to take a similar class again the next year. The teacher was qualified to teach math, but his primary job was teaching music. That meant he had a completely different way of explaining Algebra – and perhaps a better way to connect.

When he explained it, I understood. Suddenly, these math problems that seemed abstract became real and logical. Math wasn’t a mystery; it was a puzzle and a solvable one at that.

That’s what I think humans bring to the table when it comes to developing marketing and communications content. Humans don’t set out to just explain the thing the way AI does; they strive to connect to readers, or content consumers, and bring context that brings ideas to the real world.

An average of the internet

If you ask 10 people the same survey question, you might get 10 different answers. However, if you ask 100 people, you’ll start to see a solid average.

If you ask 1,000 people, the average becomes clear. This is how confidence intervals in surveys work – it’s the confidence that if you ran the same survey, you’d get the same results.

These LLMs have sucked up an internet’s worth of content already. The average is set. It won’t get worse, but it also won’t get better.

There are a lot of credible SMEs that have come to a similar conclusion. Nikita Bier, who is focused on eliminating AI-generated spam on X, recently noted that he thinks the platform will be successful.

Why? “We are very close to approaching the limit of the content being indistinguishable,” he said recently. And you can see from the results of the experiment above that Google is figuring it out, too.

There are no “hacks” or shortcuts to marketing and PR. It takes time, effort, data, consistency, and perseverance. Generative AI is a useful tool for augmenting your marketing and communications team, but it’s not a replacement.

It’s definitely not a replacement for good writing. That’s not my opinion, but a conclusion we can draw on a mounting pile of data.

* * *