Imagine seeking guidance from an AI-powered chatbot only to receive bizarre advice that could land you in legal trouble. Well, that’s exactly what small business owners in New York City are facing with the city’s controversial chatbot.
Despite widespread criticism and reports of the chatbot dispensing incorrect information, Mayor Eric Adams has chosen to keep the tool on the official government website. While acknowledging the chatbot’s flaws, Adams defended the decision, claiming that addressing issues is just part of refining new technology.
Launched in October as a resource for business owners to navigate bureaucratic processes, the chatbot offers algorithmically generated responses. However, it comes with a disclaimer warning users that the information provided may be incorrect or biased, and that it does not constitute legal advice.
The bot’s problematic responses, which include advising on illegal actions and offering absurd solutions, have raised concerns among experts. Julia Stoyanovich, a computer science professor, criticized the lack of oversight in implementing unproven software.
Despite efforts by Microsoft, the provider of the chatbot’s AI services, to improve accuracy, the tool continues to give out misleading advice. The situation mirrors recent incidents where chatbots from private companies have faced backlash for misinformation.
Experts emphasize the need for caution when implementing AI-powered systems in the public sector. Jevin West, a professor, highlighted the importance of trust in government-endorsed technologies and the potential risks of following inaccurate guidance.
In contrast, cities like Los Angeles have adopted more curated approaches to chatbots, reducing the risk of misinformation. The pitfalls of New York’s chatbot should serve as a lesson for other municipalities, prompting a reevaluation of the purpose and accountability of such technological solutions.