Fuzzy Logic

Ted Chiang wrote an excellent article for The New Yorker this month: ChatGPT is a Blurry JPEG of the Web. In it, he describes the loss of information that happens when you try to save a large set of data in a small storage space. Modern algorithmic models are pretty great at guessing what goes in the gaps, but it is still just a guess.

Instead of just summarizing the Chiang article (we strongly recommend reading it for yourself), we wanted to give a birds-eye-view on what makes it such a great analysis.

The essay starts with a familiar, and famously lossy, technology: Xerox photocopiers. The photocopy then serves as a vehicle to introduce relevant terminology, and as a metaphor for the data loss that occurs in most of our everyday digital filing systems like JPEGs and MP3s.

With that foundation, the reader is primed to understand the relatively new and complex technology of Large Language Models. Chiang is then able to pack in a ton of technical detail about how LLMs work, and make a compelling case:

I’m going to make a prediction: when assembling the vast amount of text used to train GPT-4, the people at OpenAI will have made every effort to exclude material generated by ChatGPT or any other large language model. If this turns out to be the case, it will serve as unintentional confirmation that the analogy between large language models and lossy compression is useful. Repeatedly resaving a jpeg creates more compression artifacts, because more information is lost every time. It’s the digital equivalent of repeatedly making photocopies of photocopies in the old days.

This excerpt is so easy to understand, and offers so much to think about.

Thank you to Ted Chiang and The New Yorker for the great work!