Travels with ChatGPT
Summary
This summer I took ChatGPT on holiday with me. I used GPT-4o in the Android app on my phone to translate menus, find historical artifacts, and understand art and wine.
ChatGPT CAN SEE!
It’s honestly the closest thing to magic I know of. Here are quick examples of all the ways it helped me.
All the ChatGPT panels are scrollable. You can download any of the pictures, upload them to your favorite Large Language Model, and try it for yourself.
Translate the menu
Translate this menu from French
Here is the translation of the menu from French to English:
MAINS PIÈCE DU BOUCHER Choose from our premium butcher’s cuts according to availability.
DESSERTS
|
The first time I took a picture of a menu and asked ChatGPT to translate it, I did not expect it to work. It was just trying to find the edges of it's ability. I mean, how _could_ it work?!
And yet it does. If that’s all it did, if it had been carefully, precisely built for the one task of translating French menus into English, I would happily pay for it. What boggles my mind is that it wasn’t built for this at all, it just happened. Scroll the panel on the left to see the full translation. |
In Spanish, from a chalk board
|
The emergent nature of this ability means it can do Spanish too, here from a picture of a HAND WRITTEN CHALK BOARD!
And it knows it’s left and right. If you count the items on the chalkboard there are eleven. ChatGPT’s first response had only ten elements, so I told it so. And it corrected itself. |
Translate the column on the left from Spanish
Sure, here is the translation of the left column from Spanish:
Toasts
You missed one
I apologize for that. Here is the corrected translation with the missing item included:
Toasts
|
With some Basque, why not
Translate from Spanish
Sure, here is the translation of the menu: PINTXOS
RACIONES
BOCADILLOS (Sandwiches)
|
I never met a menu it couldn't translate. Here's a final example from a much busier menu, with some less common seafood items, and a mixture of Spanish and Basque.
If you scroll to the very bottom you’ll see I interrupted it as I was only interested in the first (Pintxos) section. Google Lens can do this now, thanks presumably to Gemini. This is clearly a very valuable feature, so much so that Google advertises it on the front page of the Pixel phone’s website: use your camera to translate menus. 
|
Explain the world
|
Having ChatGPT in my pocket helps me understand the world, and so makes it that much more interesting. Here we were walking past a beautiful clock on the outside of the law courts in Paris, which had a Latin inscription. I can’t read Latin, but ChatGPT can! The translation makes sense in the context. It is affixed to the law courts. I later learnt that the clock is important and famous enough that the street is named after it: Le Quai de L’Horloge. This maybe seem the same use case as “Translate the menu” - it’s just “translate stuff”, but to me it’s quite different. You have to understand the menu. ChatGPT made this inscription accessible to me. Without it I would have guessed a few words and moved on. |
Translate this latin inscription
The Latin inscription reads:
“Machina quae bis sex tam juste dividit horas; Justitiam servare monet legesque tueri” This translates to: “The machine that divides the hours so justly into twice six; Advises to preserve justice and to uphold the laws” |
The backstory of a bottle of wine
Tell me about this large bottle
Searched 6 sites
The bottle you have is “Studio by Miraval,” a rosé wine from Château Miraval, located in the Provence region of France. This wine is a blend of Cinsault, Grenache, Rolle, and Tibouren grapes. It is well-regarded for its elegant nose with delicate notes of grapefruit and white flowers, alongside a palate featuring juicy red berries and a hint of crisp texture【6†source】【8†source】. Château Miraval, co-owned by Brad Pitt and Angelina Jolie, partnered with the renowned Perrin family to produce this wine. The estate is noted for its connection to the music industry, having a recording studio where bands like Pink Floyd and AC/DC have recorded【10†source】. The rosé is known for its pale pink color with bluish reflections and a fresh, refined taste, making it a popular choice for its balance and elegance【7†source】【9†source】. |
A restaurant in Barcelona had this large bottle of wine on display. Previously I would not have paid it much attention, presumably it means something to someone. No more! ChatGPT told me all about it, and it does indeed have an interesting back story. |
An art historian in your pocket
|
There are a ridiculously large amount of rennaisance paintings in the Louvre in Paris, and my eyes glaze over rapidly. I picked one completely at random and asked ChatGPT about it. First IT IDENTIFIED IT, with no clues or context. Second it turns out that’s quite an interesting painting. My eyes un-glazed (is that a word?). It’s like visiting an art gallery with an art historian. The title of the painting was right there on the card. I could have looked it up on Wikipedia. In that sense ChatGPT didn’t provide anything I couldn’t do before. I’m including it here because I’m still amazed it identified the painting like that. |
Tell me about this painting
The painting in the image is "The Fortune Teller" by Michelangelo Merisi da Caravaggio. It is an early work of the artist, created around 1594-1595.
About the Painting:
Description:“The Fortune Teller” depicts a young man having his palm read by a gypsy girl. The scene captures a moment of interaction and deception, as the fortune teller subtly steals the young man’s ring. The painting is notable for its realism and attention to detail, typical of Caravaggio’s style. The figures are rendered with naturalistic precision, and the use of light and shadow (chiaroscuro) enhances the three-dimensionality of the scene. Significance:
Caravaggio’s work had a profound influence on Baroque art, and “The Fortune Teller” is a significant example of his early mastery of genre painting and his innovative approach to composition and lighting. |
And so much more
It can do contemporary art too. It identified a Jeff Koons sculpture from a picture with no context.
Desperate to find something ChatGPT can’t do, I asked it to translate some Egyptian hieroglyphics. It didn’t, but it did teach me a lot about hieroglyphics very quickly. I was able to point out details and sound knowledgeable in front of my family. I don’t think they were impressed, but I was!
Level Up
I’ve used ChatGPT via the API and text interface a lot before, and for sure it’s amazing, but it is also limited. Having ChatGPT on my phone, with vision, really changed how I use it. I feel like I have super powers.
I could have used search to find out a lot of this, but that would take time. You have to tap tap tap on your phone scroll scroll scroll, close the “hey did you know we have a newsletter” popups, skip the top five results that are SEO garbage, and so on. That’s tricky to do while you’re actively walking around and / or having a conversation.
With ChatGPT I ask and it answers. I can walk, talk, and know things all at once.
|
Where is Hammurabi's code ?
Hammurabi's Code is inscribed on a stele, which is a large stone monument. This stele was discovered in 1901 by French archaeologists at the site of Susa, in what is now Iran. The stele is currently housed in the Louvre Museum in Paris, France. The inscriptions on the stele include one of the earliest and most complete written legal codes, which dates back to around 1754 BCE during the reign of King Hammurabi of Babylon.
What room in the Louvre?
Hammurabi's Code stele is displayed in Room 227 of the Louvre Museum, which is part of the Sully Wing. This room is dedicated to the Ancient Near Eastern antiquities collection.
|
The Louvre is an overcrowded maze of precious artifacts. At the entrance they give you a map to the five or so things they expect you to want to see (such as the Mona Lisa), and everyone dutifly ticks those things off. Finding anything else is much harder. Unless you have ChatGPT!
I spent a sizeable part of my youth playing Civilization, so I know the importance of Hammurabi’s Code. I suspected it was in the Louvre. ChatGPT took us right to it. |
A restaurant we were in had scultupres of hand gestures on the wall. The waitress explained they were Indian dance Mudras, so I quickly uploaded that knowledge into my brain, and you can too: Indian Dance Mudras Explained
I use ChatGPT all the time to plan vacations, and it does an amazing job. This San Sebastian / Donostia itinerary is pretty much exactly what we ended up doing. The beautiful part of how an LLM works is that this is the most statistically probably itinerary. It’s the most likely thing we were going to do anyway once we put in the effort to figure it out, which I didn’t have to. Three-Day Itinerary in San Sebastian
There are so many more examples. ChatGPT helped me choose wine, understand Spanish beer sizes, summarize books, and on and on.
The closest thing to magic
It really is. How else do you explain this?
|
That's a poor quality picture from a train, with no context. And yet! Check Wikipedia, this is all correct as far as I can tell.
Now some will say “oh that’s easy it read the words and then it Googled it”. Yes! Exactly! IT READ THE WORDS! But only the correct word, not the “SNCF” part. And then it searched the Internet (it uses Bing) and UNDERSTOOD WHAT IT FOUND enough to summarize it for me. How is that not magic? ARE YOU NOT ENTERTAINED?! |
Tell me something interesting about this town
The town shown in the image is Agde, located in the Hérault department in southern France. Here are some interesting facts about Agde:
Agde’s rich history, unique architecture, and scenic location make it a fascinating destination to explore. |
You can’t Google this
|
What type of hard drive adapter is this?
The hard drive in the image is a SATA (Serial ATA) hard drive. You can tell by the connectors, which include the larger power connector on the left and the smaller data connector on the right. The middle connector is for optional features and not used for standard connections. To connect this hard drive to a computer, you would need a SATA cable and a power supply with a SATA power connector.
What about this one?
The hard drive in this image has both SATA and Molex power connectors. On the left, you see the SATA data and power connectors, similar to the previous image. On the right, you see a Molex power connector, which is used in older systems. This means you can use either a SATA power cable or a Molex power cable to power this hard drive, providing more flexibility in connecting to different types of systems. The Molex connector is the larger 4-pin connector on the right side of the image.
What type of hard drive adapter is this one?
The hard drive in this image is an IDE (Integrated Drive Electronics) or PATA (Parallel ATA) hard drive. The wide, flat connector with many pins is the IDE interface, which was commonly used before SATA became the standard. The 4-pin connector to the right is a Molex power connector, which provides power to the drive.
To connect this hard drive to a computer, you would need an IDE ribbon cable for the data connection and a Molex power connector from the power supply. IDE drives are typically found in older systems, and connecting them might require an IDE to USB adapter if you’re trying to connect it to a modern computer without IDE ports. |
On my travels I took apart several old computers and tried to recover their data (it's a completely normal vacation activity, get out of here). I needed to buy the correct adapters for various generations of hard drives.
How would I have searched for this? How do you describe the back of those drives in words? I could have put the pictures in a forum and waited and hoped. Or I could have built a list of hard drive adapters with pictures and tried to match them up. ChatGPT saved me a lot of time and effort. Scroll the panel to see the three different adapters it identified. If you’re thinking Google image search / Bing visual search, that’s just using a different AI model, it’s still magic. ChatGPT is like having an incredibly smart, well informed, but overconfident personal assistant. |
Writing this post was difficult mostly because of the amazing examples I had to delete to avoid turning this into a book.
For example, I took a picture of the Sagrada Familia’s magic square in Barcelona, and said “What is special about this picture”. It identified it as a magic square, and with a couple more prompts it checked the maths by doing the row, column and diagonal sums, and told me exactly where it was and what it signified. That’s incredible.
The magic square is a good example as to how ChatGPT can be wrong and helpful at the same time. It at first told me the magic square had an error because it didn’t sum to 34. I’m not sure why it thinks it should. One extra prompt was all it needed to get back on track.
Confabulations and censorship
|
Here's a more worrying example of it being wrong.
I had just gotten out of the water in San Sebastian / Donostia when I noticed a couple of flags on the beach, and no other swimmers in the water (lots of surfers though). I asked ChatGPT what the flags meant. It told me the two flags meant that the beach was closed becase of dangerous weather conditions AND the presence of dangerous marine life (sharks! jellyfish!). Yikes. We had a good laugh about my narrow escape. It wasn’t until we left the beach that we notice signage saying those flags meant “water sports”, that’s the area the surfers should be in. ChatGPT gracefully admitted it could have got it wrong. How very human. |
What does this flag mean at the beach?
The flag in the image is a red over blue flag, which is commonly used at beaches to signal a specific condition.
A red over blue flag typically indicates that the beach is closed for swimming due to dangerous conditions such as strong currents, high waves, or other hazards. It serves as a warning to beachgoers to stay out of the water for their safety.
And what does this flag mean?
The flag in the second image, which is divided diagonally into red and blue halves, typically signifies "Dangerous Marine Life" in many coastal regions. This flag is used to warn beachgoers of the presence of potentially harmful sea creatures like jellyfish, sharks, or stingrays. It alerts swimmers to exercise caution or avoid entering the water.
Could it also mean "water sports"?
Yes, in some regions, a red and blue flag can also indicate an area designated for water sports. This flag is used to mark zones where activities such as surfing, windsurfing, or other water sports are permitted, separate from areas designated for swimming. The specific meaning of beach flags can vary depending on local regulations, so it's always good to check with local beach authorities if you're unsure.
|
Everything, everywhere, all at once
List the languages in this picture.
The image shows the Lord's Prayer in various languages. Here is a list of the languages visible in the picture:
These languages are inscribed with different scripts and may have slight variations or different transliterations.
Give an example word from the picture for each of the languages
Here are example words from the image for some of the languages identified:
These examples are based on the text visible in the provided image. The languages are not exhaustive, and there may be additional languages on the full panel that are not fully visible in the image.
Are there any full sentences? If yes, list them.
Yes, there are full sentences visible in the image. Here are some examples of full sentences from the Lord's Prayer in various languages:
Continue
Certainly, here is the continuation with the full sentences from the visible languages in the image:
Does a pair of letters stand out?
Yes, in the image, the letters "A" and "G" in the word "QUE CAIGUEM" (part of the Catalan text) stand out in a different color (orange) compared to the rest of the text.
|
This one has everything, let's step through it.
This is a picture of the Eucharist door of the Glory Facade of Barcelona’s Sagrada Familia. It features the full Lord’s Prayer in Catalan and the phrase “Give us this day our daily bread” in 50 different languages. The languages are Albanian, Arabic, Aramaic, Aranese, Basque, Berber, Bulgarian, Catalan, Chinese, Coptic, Croat, Czech, Danish, Dutch, English, Esperanto, Estonian, Finnish, French, Gaelic, Galician, German, Greek, Guarani, Hebrew, Hungarian, Icelandic, Indonesian, Italian, Japanese, Korean, Latin, Norwegian, Polish, Portuguese, Quechua, Rumanian, Russian, Sanskrit, Sardinian, Serb, Slovenian, Spanish, Swahili, Swedish, Tagalog , Tibetan, Ukrainian, Vietnamese, and Wolof ChatGPT found 101 languages. It’s completely incredible that it found much of anything in there. But there should only be 50, so I asked for examples. It seems to have included every language it could be. For example the Romanian is the same as the Moldovan, which isn’t officially on there. Czech (official) and Slovak (unofficial) are the same, and so on. So far ChatGPT isn’t wrong. The words it lists though, are mostly “Our Father”. That’s the start of the Catalan part, but the 50 languages should say “Give us this day our daily bread”. So is that part wrong? Well I’m not sure. If you zoom in really close top right it does say “Our father” in several languages, and probably again top left. It seems like not all languages are covered, but I’m not sure. So I asked for full sentences. It easily finds the large Lord’s Prayer in Catalan. Then it thinks it finds it in Spanish too, which, well I can’t find it. I think that’s a confabulation. AND, OpenAI doesn’t allow it to recite the Lord’s Prayer. A censored confabulation. I don’t know why, presumably a blanket ban on religious things. But not in Catalan. It can’t be persuaded to continue. Finally I ask an easy one, and it has no trouble picking out Gaudi’s initials which will become the door handles. Confabulation, censorship, and mastery of 101 languages. All the wow. |
Conclusion
I’ve been a software engineer for decades, I have a Masters in Artificial Intelligence, and still the things ChatGPT can do completely blow me away. The behavior is emergent. No amount of understanding it’s internals come close to explaining it. What a time to be alive.
I often end my posts with music, so I hired a professional musician used Suno to create an offical darkcoding.net song. Sit on the porch, crack open a cold beer, and enjoy darkcoding.net: Software and Society.