One of my greatest lessons in life is to walk towards what worries you.
Knowledge quenches fear-based fires and helps me understand how to approach a situation. Does it work like that for you?
Dousing anxiety with information might sound like madness to some. And for a long time I thought that too. I was a world-class ostrich.
But today, with my feet planted and my value tied to more than one thing, I can be gleefully curious.
Writers Against the Machine is an experiment. A curious exploration of the largely unknown.
The fear? AI is eroding my value as a writer. And what that means for my business.
Stepping back
Maternity leave took me away from the industry as ChatGPT took off. I knew better than to dismiss it out of hand (we wrote plenty about it 2023). But I also didn’t have the time or energy to look into it in great detail.
Returning months later, everything looked different. Was it the lens of motherhood or had the machines really taken over?
Writers Against the Machine was born of an ignorance I knew I had to conquer. And a conversation with Cat, Incredibble’s marketing strategist, that caught alight and presented an opportunity to be publicly ignorant, honest and, hopefully, helpful.
Leaning in
I posted about this first in October 2024 on LinkedIn. Here’s the original post.
This was our idea:
- Write a brief for an article.
- Pay a writer to fulfil it.
- Take that same brief to AI and see what we get.
- Invite all writers to also ‘go against the machine’ and write their own version of the article.
- See if writers come out on top.
The LinkedIn post brought two valuable things:
Excellent questions.
From commenter Catherine Maher in particular (tips hat), who said the experiment ‘felt off’ and asked what was our endgame?
And a speaking opportunity - that made me think more about our methodology.
That’s in January 2025 for Writers Against the Algorithm - a community supporting writers during these fast-changing times - when I’ll be talking about this experiment.
A conversation with its founder Felicity Wild encouraged me to think about the methodology for comparing the outputs.
Let’s take those two in turn.
What is our endgame?
It’s very selfish, in the first instance: I want to understand what AI can do. And I missed months of discourse where this might have been shared on social platforms.
And I also want everyone to see how great writing from creative minds compares to regurgitated writing. And what using AI like-for-like delivers.
In the second instance, it’s about understanding our clients more deeply. What decisions are they making? What choices are they faced with? How can we serve them alongside ChatGPT? If at all.
Let’s sit in that for a moment.
Let’s be our client…
You’re leading a marketing department.
You know content. You understand its value. You respect good writing.
In previous roles, working with a content or copywriter has been a luxury. In this role, it’s a necessity. The team can brief but they aren’t writers. And you want their heads in the data, distribution and execution, rather than in the words.
It’s also about bandwidth. Good writing takes time and you need additional writing resource to deliver your strategy.
It’s also (always) about budget.
And the rise of AI makes you wonder… can I brief it and get an article in under a minute? Will it be good enough? How do I frame that brief? How much can it do? Will it understand our feedback, for example? How far can I trust it? Is it accurate? What about our voice? Our brand personality? What are the ethics?
And so our experiment doesn’t go far enough. It doesn’t answer these questions. It doesn’t even know if these questions are accurate. For that, we need to speak to some real people. And learn more about the opportunities and limitations of AI.
And that’s part two of this curious adventure. More on that later.
Now, back to our current experiment.
What is our methodology?
Initial thoughts? Lit crit 101. Compare and contrast.
But Felicity brought my attention to a different approach that piqued my interest. She flagged this paper in her weekly newsletter and mentioned it when we spoke.
Authored by Nina Begus, a postdoctoral fellow at Berkeley, University of California, and published in October 2024, it uses Pygmalion myth to prompt writers and LLM (large language models) to write stories. And deeply analyses the results.
NB. The Pygmalion myth is an ancient narrative about a man falling in love with his humanlike creation. Think Frankenstein, The Terminator, Pinocchio, Ex Machina.
The experiment sought to do three things:
- outline the scope of contemporary cultural landscape surrounding a specific trope (the Pygmalion myth)
- analyse implicit social biases manifested in storytelling
- [determine] whether the state-of-the-art language models could be more innovative in storytelling than human writers, or are innovative in other ways
Now… I’m not even going to attempt to reach such depths of understanding with our experiment. Not least: time, money, resource. We approached this entirely differently. But I will share some of Nina’s findings in this article because they matter to what we’re doing here.
Our methodology went like this:
We wrote a brief.
We paid an Incredibble freelancer to answer it.
We took that same brief to ChatGPT and Claude.
We did not stipulate tone.
We did not sign into the AI platforms or, if we had to, we used new log-in details. This ensured the AI wasn’t influenced by previous inputs.
We did not give feedback or ask for revisions.
And now we’re looking at the results.
This link takes you to a portfolio that contains the brief and all the articles.
If you’re a writer, submissions remain open. Read the brief and go at it! Contact Incredibble to submit your article to the portfolio.
NB. The platform allows for up to 10 clippings to be added for free. We currently have space for five more submissions.
What are the results?
My immediate thoughts:
- ChatGPT went way over the word count (1000+ words).
- ChatGPT and Claude Haiku both used US English by default.
- Both AI articles had more character than I was expecting, but I prefer a little more edge in my writing and I’d want the same for my clients.
- I enjoyed the humour in Becky’s piece.
- Becky’s article had more authority because she used anecdotes and cited sources.
- I would have had minor edits for both AI articles in how they cited (or didn’t) information.
- Because of this, both AI articles risk sharing inaccurate information which makes me uncomfortable.
- ChatGPT’s interpretation of ‘10x’ was inaccurate. Claude’s was spot-on.
I wouldn’t be totally disappointed to have either of the AI articles pass my desk. Claude’s was better. However, it’s the false truths and lack of authority that make me uncomfortable.
Chat GPT’s parroting was concerning. Its understanding of 10x and Claude’s lack of cited sources risk damaging a brand’s authority.
If I was a busy marketing leader, I would want utter confidence that the content we were producing was accurate, honest and in the best interests of the reader and our brand.
We could spend more time on the brief. Pack it full of data and information so writing the piece requires little more than stitching together the ideas.
Some clients do this when they brief us. I often wonder why, when they’ve put so much detail into the brief, they don’t write it themselves. 1500-word briefs for 1000-word articles will make you question things!
And for these clients, I can see AI as a cost-effective short-cut.
When the brief is more like the one we used here, would I be happy for one of my team to take an AI article and adjust it? Would I suspect such an article would make any busy person lazy, and likely to not check facts as deeply as possible? Would I be certain our brand voice would be lost and tricky to weave back into the article?
The conclusion I’m coming to is that AI is, as we already know, a tool but not a complete writing solution.
I’d be happy for my team to use it to research, generate ideas and get started.
But I’d want opinion, character and personality poured into the pieces in reflection of the brand and its audience.
That’s what I look for in my freelance writers. An edge that brings things to life. We write for B2B audiences, often about dry subjects. We’ve always excelled at bringing interest and engagement to those areas. Never at the expense of brand voice or the sincerity of the topic. Always with the reader in mind.
I want imagination. And art.
And now it’s time to bring in Nina’s research findings. Because she found:
- “GPT’s imaginative landscape is much narrower than that of human writers.”
- “[AI narratives…] offer less imaginative scenarios, settings and rhetoric than human-authored texts.”
- “ChatGPT characteristically leans towards simplistic and moralising resolutions, revealing a much narrower narrative skill set and imaginative scope.”
- “GPT narratives are less colourful and playful than most human-written stories, even though they do not lag behind in clarity and comprehensibility.”
More positively, AI does “exhibit a greater propensity to place female characters in roles traditionally occupied by male characters and to introduce same-sex relationships into the traditionally heterosexual Pygmalion paradigm.”
However… “attributes and descriptions of characters remain biased to an extent, especially regarding physical appearance and feminine traits in female protagonists.”
But, wow! Isn’t this now so much about our clients? About what kind of marketers they are? About their love of words, brand and voice?
Because if you don’t care, if you haven’t invested in these things, AI is going to be a great solution for you.
We’re coming up to part two of our curious experiment: what do our clients think?
For writers at this point I’m wondering: who are your clients?
The shift in our industry might not be in our skills, but in who we choose to work with.
That might shrink the pool. Or it might be an invitation to work with different clients in new ways.
Part two is coming soon. Three interviews with three Incredibble clients. What do our clients think of AI?
Do you want to go against the machine? Access the brief here, write your article and we’ll add it to the platform (there’s currently space for five more submissions).