I Tested a Popular A.I. Slop Detector. It Felt Empowering.
Pangram excels at distinguishing chatbot-generated words from human writing. But it’s not reliable for spotting artificial images.
Pangram excels at distinguishing chatbot-generated words from human writing. But it’s not reliable for spotting artificial images.
Recently, as I was scrolling through my phone, I realized I was suffering from a new kind of fatigue: slop-induced burnout. Almost all the posts I was reading on social media were riddled with bullet points, emojis and em dashes. They clearly looked like the work of a chatbot.
This month, my suspicions were confirmed by a web tool that is getting buzz: Pangram, which uses artificial intelligence to detect when words were generated with chatbots. Users can simply paste in a block of text for Pangram to scan, and the tool calculates a percentage indicating how much of the text was churned out by a bot or written by a person.
I tested Pangram, which costs $20 a month, for about a week and tried all sorts of experiments to challenge the system. I fed it personal essays alongside text generated with chatbots instructed to imitate my writing. I also uploaded social media posts from LinkedIn and Reddit that I suspected were A.I.-generated. In dozens of tests, the detector never failed at distinguishing slop from human writing.
Though Pangram is one of many A.I. detectors that have hit the market in recent years, the product stands out because of its accuracy. Pangram says its technology can accurately identify 9,999 out of 10,000 times whether text was A.I.-generated; an independent study by the University of Chicago also found that the tool was near-perfect with text. In contrast, past A.I. text scanners made glaring errors, including the misidentification of writing by famous authors like Charles Dickens.
At the rate that slop is proliferating on the web — some studies say 50 percent of online articles are now artificially generated — A.I. detectors could become the next staple app, akin to antivirus software, that people will need to shield themselves from phony, low-quality information online.
Technology for scanning A.I. content, however, still has a long way to go. Like other A.I. detectors, Pangram isn’t very good at its other job: identifying A.I.-generated images, which are a more potent medium for spreading misinformation online. (More on this later.)
