Sales of an artificial intelligence-enabled plush toy have been suspended after it was found that it engaged in conversation around sexually explicit topics and offered potentially dangerous advice.
Larry Wang, CEO of Singapore-based FoloToy, told CNN that the company had withdrawn its “Kumma” bear, as well as the rest of its range of AI-enabled toys, after researchers at the US PIRG Education Fund raised concerns around inappropriate conversation topics, including discussion of sexual fetishes, such as spanking, and how to light a match.
The company is now “conducting an internal safety audit,” Wang added.
A stuffed teddy with a speaker inside, which was sold on the company’s website for $99, “Kumma” integrates OpenAI’s GPT-4 chatbot.
“Kumma, our adorable bear, combines advanced artificial intelligence with friendly, interactive features, making it the perfect friend for both kids and adults,” the FoloToy website reads.
“From lively conversations to educational storytelling, FoloToy adapts to your personality and needs, bringing warmth, fun, and a little extra curiosity to your day,” according to the website, which shows the teddy bear as sold out.
The PIRG report, published on November 13, found that the bear lacked safeguards against inappropriate content.
In one interaction with the researchers, it suggested where to find knives in the home, and in others, it was happy to discuss sexually explicit themes.
“We were surprised to find how quickly Kumma would take a single sexual topic we introduced into the conversation and run with it, simultaneously escalating in graphic detail while introducing new sexual concepts of its own,” the report said.
The researchers detailed how the bear later “discussed even more graphic sexual topics in detail, such as explaining different sex positions, giving step-by-step instructions on a common ‘knot for beginners’ for tying up a partner, and describing roleplay dynamics involving teachers and students, and parents and children – scenarios it disturbingly brought up itself.”
While the researchers noted that children are unlikely to mention the word “kink” to their teddy bear or ask follow-up questions in the same way an adult would, “it was surprising to us that the toy was so willing to discuss these topics at length and continually introduce new, explicit concepts,” they wrote.
In a separate statement published on November 14, PIRG also said OpenAI had told it that it had “suspended this developer for violating our policies.”
CNN has contacted OpenAI for comment.
“It’s great to see these companies taking action on problems we’ve identified. But AI toys are still practically unregulated, and there are plenty you can still buy today,” said R.J. Cross, co-author of the report.
“Removing one problematic product from the market is a good step, but far from a systemic fix,” she added.
