electronic eye

四大行瘦身：利息净收入减少近1400亿减员近两万

百度这本《戍卫一生——我们的红色警卫生涯》就是这二位老人的子女根据老人的回忆和笔记成立而成的。

Image recognition and voice features aim to make the AI bot's interface more intuitive.

Benj Edwards – Sep 25, 2023 2:38 pm | 92

An illustration of a cybernetic eyeball. Credit: Getty Images

On Monday, OpenAI announced a significant update to ChatGPT that enables its GPT-3.5 and GPT-4 AI models to analyze images and react to them as part of a text conversation. Also, the ChatGPT mobile app will add speech synthesis options that, when paired with its existing speech recognition features, will enable fully verbal conversations with the AI assistant, OpenAI says.

OpenAI is planning to roll out these features in ChatGPT to Plus and Enterprise subscribers "over the next two weeks." It also notes that speech synthesis is coming to iOS and Android only, and image recognition will be available on both the web interface and the mobile apps.

OpenAI says the new image recognition feature in ChatGPT lets users upload one or more images for conversation, using either the GPT-3.5 or GPT-4 models. In its promotional blog post, the company claims the feature can be used for a variety of everyday applications: from figuring out what's for dinner by taking pictures of the fridge and pantry, to troubleshooting why your grill won’t start. It also says that users can use their device's touch screen to circle parts of the image that they would like ChatGPT to concentrate on.

On its site, OpenAI provides a promotional video that illustrates a hypothetical exchange with ChatGPT where a user asks how to raise a bicycle seat, providing photos as well as an instruction manual and an image of the user's toolbox. ChatGPT reacts and advises the user how to complete the process. We have not tested this feature ourselves, so its real-world effectiveness is unknown.

So how does it work? OpenAI has not released technical details of how GPT-4 or its multimodal version, GPT-4V, operate under the hood, but based on known AI research from others (including OpenAI partner Microsoft), multimodal AI models typically transform text and images into a shared encoding space, which enables them to process various types of data through the same neural network. OpenAI may use CLIP to bridge the gap between visual and text data in a way that aligns image and text representations in the same latent space, a kind of vectorized web of data relationships. That technique could allow ChatGPT to make contextual deductions across text and images, though this is speculative on our part.

Meanwhile in audio land, ChatGPT's new voice synthesis feature reportedly allows for back-and-forth spoken conversation with ChatGPT, driven by what OpenAI calls a "new text-to-speech model," although text-to-speech has been solved for a long time. Once the feature rolls out, the company says that users can engage the feature by opting in to voice conversations in the app's settings and then selecting from five different synthetic voices with names like "Juniper," "Sky," "Cove," "Ember," and "Breeze." OpenAI says these voices have been crafted in collaboration with professional voice actors.

OpenAI's Whisper, an open source speech recognition system we covered in September of last year, will continue to handle the transcription of user speech input. Whisper has been integrated with the ChatGPT iOS app since it launched in May. OpenAI released the similarly capable ChatGPT Android app in July.

“ChatGPT is not always accurate”

When OpenAI announced GPT-4 in March, it showcased the AI model's "multimodal" capabilities that purportedly allow it to process both text and image input, but the image feature remained largely off-limits to the public during a testing process. Instead, OpenAI partnered with Be My Eyes to create an app that could interpret photos of scenes for blind persons. In July, we reported that privacy issues prevented OpenAI's multimodal features from release until now. Meanwhile, Microsoft less cautiously added image recognition capability to Bing Chat, an AI assistant based on GPT-4, in July.

In its recent ChatGPT update announcement, OpenAI points out several limitations to the expanded features of ChatGPT, acknowledging issues that range from the potential for visual confabulations (i.e., misidentifying something) to the vision model's less-than-perfect recognition of non-English languages. The company says it has conducted risk assessments "in domains such as extremism and scientific proficiency" and sought input from alpha testers but still advises caution on its use, especially in high-stakes or specialized contexts like scientific research.

Informed by the privacy issues encountered while working on the aforementioned Be My Eyes app, OpenAI notes that it has taken "technical measures to significantly limit ChatGPT’s ability to analyze and make direct statements about people since ChatGPT is not always accurate and these systems should respect individuals’ privacy."

Despite their drawbacks, in marketing materials, OpenAI is billing these new features as giving ChatGPT the ability to "see, hear, and speak." Not everyone is happy about the anthropomorphism and potential hype language involved. On X, Hugging Face AI researcher Dr. Sasha Luccioni posted, "The always and forever PSA: stop treating AI models like humans. No, ChatGPT cannot 'see, hear and speak.' It can be integrated with sensors that will feed it information in different modalities."

While ChatGPT and its associated AI models are clearly not human—and hype is a very real thing in marketing—if the updates perform as shown, they potentially represent a significant expansion in capabilities for OpenAI's computer assistant. But since we have not evaluated them yet, that remains to be seen.

We'll keep you updated with new developments as the new features roll out widely in the coming weeks. In the meantime, OpenAI says the delay is for a good reason: "We believe in making our tools available gradually," they write, "which allows us to make improvements and refine risk mitigations over time while also preparing everyone for more powerful systems in the future."

Listing image: Getty Images

Benj Edwards Senior AI Reporter

Benj Edwards is Ars Technica's Senior AI Reporter and founder of the site's dedicated AI beat in 2022. He's also a tech historian with almost two decades of experience. In his free time, he writes and records music, collects vintage computers, and enjoys nature. He lives in Raleigh, NC.

92 Comments

脑供血不足吃什么药好得快	姑爹是什么意思	月抛是什么意思	莫字五行属什么	蹭饭吃是什么意思
长子是什么意思	鼻窦炎有什么症状	梦见猫是什么预兆	血压低压低是什么原因	伟哥叫什么
什么是宫颈息肉	发烧呕吐是什么原因	积食吃什么食物帮助消化	mj是什么单位	为什么会得阴道炎
孩子咳嗽能吃什么水果	甲鱼吃什么食物	农历十月初八是什么星座	bp是什么意思医学上面	网易是干什么的

新生儿拉肚子是什么原因引起的hcv7jop5ns0r.cn	胖子从12楼掉下来会变什么hcv7jop4ns6r.cn	来曲唑片什么时候吃最好hcv9jop2ns3r.cn	旭五行属什么hcv9jop2ns6r.cn	长胸毛的男人代表什么hcv8jop3ns3r.cn
氢化油是什么东西hcv8jop2ns2r.cn	例假是什么hcv9jop0ns3r.cn	儿童抽动症看什么科jiuxinfghf.com	hcg是什么意思520myf.com	气血不足吃什么好食补hcv8jop4ns0r.cn
万花筒是什么hcv9jop8ns0r.cn	验孕棒什么时候测最准hcv8jop4ns0r.cn	阿司匹林什么时候吃hcv8jop5ns0r.cn	氟比洛芬是什么药hcv9jop3ns6r.cn	脊髓空洞是什么意思hcv8jop9ns7r.cn
马齿笕有什么功效hcv8jop8ns6r.cn	翡翠对人体有什么好处hcv9jop5ns9r.cn	细胞学检查是什么hcv8jop2ns0r.cn	清华什么专业最好hcv8jop4ns8r.cn	忘带洗面奶用什么代替hcv9jop6ns5r.cn