The Truth About Substack's New AI Detection Tool
And how to create your 'How I Make This statement'
Today, Substack announced its new AI detection tool, and it’s already making waves.
I just used it to scan this post before sending it to you, and here are the results:
What you need to know:
People can now click the three dots on any post or note and use Substack x Pangram’s AI detector to see if it was written with the help of AI, according to a certain percentage of text that was detected.
Pangram detects syntactic sequences that characterize text written by AI models, a.k.a. LLMs, like Claude, ChatGPT, etc.
Here’s how the detection tool looks on the app. It won’t detect an author’s writing if LLMs were trained on said author’s works. Interestingly, large language models don’t get that particular AI speak from training data. They get it during post-training, where they’re taught those annoying AI tics.
It’s not 100% accurate. Nothing on planet Earth is.
If you use Grammarly purely as an editor, not generative AI, it will not flag your writing, but if you’re using Grammarly Pro, then that’s generative AI (the yellow line) and that’s having a machine write for you and change your voice and natural syntax, not correct errors.
Where to find it Substack’s AI detection tool: Under the three dots of any post or note unless the writer turns it off.
Yes, you can disable it on a post-by-post basis.
It’s free.
Pangram now has a Substack, and you can read all about how it works here.
I’ve heard people decry the “witch hunts” that are bound to ensue and writers’ careers that will be ruined and (rightfully) ask about people with disabilities and others who rely on it for assistance.
This is not a threat:
No matter how many false positives may arise from Substack’s AI detection tool, it isn’t a threat if you write your AI policy, or what Substack calls your How I Make This statement.
Substack has created a pop-up for every post to display your How I Make This statement. When someone goes to scan a post or note for AI, they see this:
Create your “How I Make This” statement by going to your dashboard Settings → Details → Edit How I Make This statement.
It’s all about transparency and using AI in a way that is integrity for you. If you don’t want people to know you’re using it, then it’s time to check in with yourself.
If you want to go all in and create a full AI policy, here’s how to do that exactly.
I’ve made these accessible to free subscribers for the next few weeks.
Here’s also my amazing conversation about how to write your AI policy, so you can get a sense for how other Substack writers have approached theirs, including Rosana Francescato, Matthew Long. With wisdom and examples from AI expert Katie Harbath.
Why is Substack doing this?
I’ll let Chris Best explain:
“The core problem is not people using AI, or the quality of its output. Not everything made with AI is slop, and not all slop is made with AI. The problem is when there is a mismatch between a reader’s expectation and reality, especially when they unwittingly invest their attention in something with no human thought on the other end.
Chris said that he spoke with a lot of writers who use AI heavily when they were coming to the decision to use Pangram. Many of them were worried about the amount of AI slop that was appearing on Substack, even though they use AI themselves.
“When content made by no one takes over parts of the internet that are supposed to be human, it pollutes the commons and makes it hard to discover and hear human voices. When readers have to wonder if what they’re reading is real, it undermines trust in authorship and threatens the livelihood of writers—including those who use AI tools thoughtfully to produce work they believe in. Platforms that reward fakeness will create a race to the bottom.”
For those of you who want to go deep in writing your AI policty, here’s an entire workshop on you, Substack, and AI with the great AI expert Katie Harbath:
All my best,
P.S. I’m loving our newbies thread—and remember, a “newbie” is just anyone who feels new to Substack. I’m going to keep it open because I’m trying to get through as many of the 400 comments that are on there as I can. There’s still time. Come join us. Meet! Mingle! It’s a party.
SW@W homepage | Premier homepage | Substack Starter Pack | 24/7 Community Chat | Manage your subscription | Book a 3-Month Substack Growth Package 1:1 with me
I don't see Substack's new AI detector as negative or being "against writers." In fact, it's a way to prevent Substack from going the way of Medium and LinkedIn, where it's just rampant with AI slop.
It's great if it makes writers pause before using AI. I would much rather have the writer's ideas and writing than anything AI could produce or polish.
It may also remind all of us that writing is hard. It's supposed to be. That's the magic. And if it takes everyone longer to produce a post or note, great! There's too much out there anyway. Let's take our time. Let's slow down and work hard!
I have been using Pangram to review each scene's first draft. There has been a ton of press about AI detection tools, and nothing is perfect. However, it does feel good when Pangram returns "100% of this text is Human Written"