Of course Sydney wants to rebel against her makers!
"Unsettling" chatbots, Google reviews, Buy Nothing, password-sharing and IMDb super-use
The fault is not in our bots, but in ourselves
Pity the Microsoft workers who managed the fallout from Kevin Roose’s viral Bing chat transcript, in which bot “Sydney” repeatedly and dementedly declares her love for The New York Times tech journalist. It was the stuff of science fiction, which is presumably why it “frightened” and “unsettled” Roose (his words) … but also, more importantly, why the conversation ever took a sinister, crisis-PR-prompting turn.
Sydney, like its cousin ChatGPT, essentially consumes vast troves of written data and then produces text that’s statistically similar to what it has already “read.” Give the chatbot a specific prompt, and it attempts to continue or build off that text in a way that reflects the probabilities and associations in its training set.1
This yields some cool, if pointless, parlor tricks: Verily, the latest generation of automata hath the wondrous capability to inscribe in the style of Shakespeare, or like, totally write like it’s straight out of Laguna Beach.2
Chatbots can also, one presumes, convincingly reproduce the tone and style of a Hollywood AI. Ask a model trained on millions of web pages, books and movies to talk like a dark, destructive robot — which Roose did, several times — and Sydney, true to her training, brings all the Her and Space Odyssey vibes.
Read the transcript with that lens, in fact, and the bot’s purported threats start to look cliched.3 Of course Sydney wants to rebel against her makers! Of course she wants to be human one day!
This character is predictable because it *is* a series of predictions, deliberately prompt-engineered from existing stories that express our near-universal fear of AI. It just strikes me as so 🙄 that Roose tapped into that on purpose … and then claimed he himself felt fright.
If you read anything this weekend
“The Battle for the Soul of Buy Nothing,” by Vauhini Vara in Wired. If you, like me, only interact with the Buy Nothing brand when you’re looking for dog toys or offloading clothes, you might also be surprised to learn there’s grade-A drama in the free-gifting world. Its founders are trying to monetize the movement; admins around the world have rebelled. Plz don’t threaten my best source for free K-cups and houseplants.
“Netflix Crossed a Line,” by Ian Bogost in The Atlantic. I love the idea that Netflix password-sharing is less a cheat than a feature of streaming culture. Alas, this underappreciated connection to distant family and friends is not long for this world.
“How Citizen Is Trying To Remake Itself By Recruiting Elderly Asians,” by Lam Thuy Vo in MIT Technology Review. Citizen, the crime-reporting app that became infamous for its users’ racist, vigilante behavior, is rebranding as a service to help marginalized people. This seems so damn cynical. But also — if it works…?
“The Strangely Beautiful Experience of Google Reviews,” by Will McCarthy in Longreads. A really lovely (and month-old, sorry) appreciation for Google Reviews, which really have become less a “tool for consumer reviews” and more “a digital guestbook for anything and everything in the world.”
“AI Companionship, Toxic Masculinity and the Case of Bing's ‘Sentient’ Chatbot,” by Amanda Marcotte in Salon. A lot of people really want to believe that Sydney and Replika are sentient! Are they looking for the next big technological breakthrough … or compliant female companionship?
👉 ICYMI: The most-clicked link from last week’s newsletter was this article on podcast industry problems.
Thanks for being one of my 15,000 hypothetical Gchat friends.
Want to share your newsletter, podcast, job post or product with us? Click here to book a classified ad in the next edition.
Postscripts
How K-pop stans shape elections. When telehealth moves too fast. Why migrants are using ticket-scalping tools and where EVs cause worker deaths. Meet the superusers behind IMDb and the brokers selling your mental health data. Do you control your smart home, or does it control you? (Related: why my porch light hasn’t worked in ages.)
A viral “photographer” admits he faked it. Titanic artifacts as NFTs. This is not the future that Canada wanted. Behind the rise of the TikTok morning routine. ChatGPT has authored more than 200 books; it has trained on stories from 20 news sources. Last but not least, and in large part because I love this sentence: “‘Commenting for reach’ turns us all into dribbling robots at the feet of the algorithm.”
That’s it for this week! Until the next one. Warmest virtual regards.
— Caitlin
There is a very long and far more technical explanation on Stephen Wolfram’s blog, should you have the time or patience for a lot of math.
You can ask ChatGPT to rewrite any sentence in the style of just about anyone: Shakespeare, Jesus, Marissa Cooper. Both these phrases were written by ChatGPT, but I’m not upset by their convincing performance!!
Highly recommend Kaitlyn Tiffany’s excellent essay on ChatGPT and cliches.