AI engineer asked ChatGPT for recipes from refrigerator pic. The answers were...

The image of the refrigerator that was fed to visual ChatGPT. (Image: @sudu_cb/Twitter)

OpenAI has announced the launch of its latest language model Artificial Intelligence tool in GPT-4 that has brought together key functions of ChatGPT and Dall-E. GPT-4 is a large multimodal model and unlike ChatGPT, can accept text and image inputs but its outputs will be in text only.

AI engineer Sudharshan on Twitter recently wrote about what happened when he hacked and used an image model (Visual ChatGPT) and fed it with an image of food items from a refrigerator asking for recipe ideas with the ingredients visible in the photograph. GPT-4 is expected to work in a similar manner when it goes live – it will be able to provide text responses from photo inputs as well.

While the user mentions GPT-4 in his first tweet, he later clarified that it was visual ChatGPT and not GPT-4 that has not been launched yet.

Want to make some clarifications!

This is a fun, harmless application, and I didn't think much before posting it. Certainly didn't expect it to blow up. I understand the title could've been better. I chose it, so it's not verbose (I didn't want to include a large explanation… https://t.co/k7mfmgXR4t — Sudharshan (@sudu_cb) March 16, 2023

Visual ChatGPT quickly came up with several recipes from the items that included eggs, fruits, milk and some processed meat products.

Fruit salad, cheese omelette, ham and cheese sandwich, and fruit smoothie were the top choices and for each of the items, the chatbot suggested an easy recipe to whip them up.

He also wrote that since the multi-modal models were not live yet and he hacked the visual chatbot.

“The multimodal models aren't live yet, so I hacked and used image models from the next best thing - Visual-ChatGPT. Why Visual-ChatGPT? - Has good Visual Foundation Models - GPT-4 has been in the works for the past 6 months. Have a very strong hunch that it uses these models,” he wrote.

He added: “Fed the image to Visual-ChatGPT and asked it to describe in detail what's in the fridge.” The chatbot responded: “The image you provided is a refrigerator with food items such as apples, oranges, bananas, eggs, a carton of milk, cheese, and a package of lunch meat and other items."

He even used an appropriate prompt for the chatbot and called it an “expert chef” to which the language model responded with “thank you for the opportunity”.

After 1.5 million views for his tweet, the user had some clarifications.

“This is a fun, harmless application, and I didn't think much before posting it. Certainly didn't expect it to blow up. I understand the title could've been better. I chose it, so it's not verbose (I didn't want to include a large explanation at the very start). In hindsight, I could've elaborated more and been more thoughtful. GPT-4 doesn't have the image to input features yet! And I use visual ChatGPT (Which I mention and elaborate in the thread). Title could've been better,” he wrote.