Visual Recipes

Cooking often seems harder than it really is. The literature doesn't seem to help us here either. A lot of modern cooking instruction seems to be written in an overly-narrative style, often emphasizing prose aesthetics as much as practical instruction. Obscure techniques and exotic ingredients can leave one to wonder, what is essential to a recipe? What must be replicated and what can be simplified?

These characteristics of modern cooking instruction make for attractive cookbooks that are enjoyable to read at leisure, but often make work in the kitchen cumbersome and error prone.

As an exercise in anti-narration for the simplification of instructions, this project, codenamed ICONIFY, will explore the possibilities and limits of de-narrating cookbook instruction toward the development of an instructional presentation that is easy to follow in the heat of the kitchen.

While previous lab projects have explored the emojification of social media language, ICONIFY aims at a level of specificity and pragmatism that will help to guide the exploration and implementation of the project. The hope is not merely to present iconified text that is interesting, but that is ultimately useful. ICONIFY aims to present cooking instruction in a way that makes cooking easier, more fluid, and ultimately more enjoyable.

To this end, the ICONIFY team will explore a number of elements related to linguistics and semiotics and their roles in instructional language and cooking. They will use state-state-of the art tools and techniques in the domains of user interface design and natural language processing to design and develop a system for the production of iconified cooking instruction. We’re seeking to build a diverse team, with members who can bring creative input, interest and experience in interface design, and technical knowledge of linguistics, NLP, or AI. Team members should have a passion for cooking and/or instructional design, and should be prepared to bring their own kitchen experiences or cooking-related explorations to bear on the project.

Inspiration / Reading List

  • Simple, The Easiest Cookbook in the World, by Jean-Francois Mallet: Simplified recipes, visual presentation of mise.
  • Where Cooking Begins, by Carla Lalli Music: Emphasis on technique, simplicity, and substitution.
  • Salt Fat Acid Heat, by Samin Nosrat (and her show on Netflix): Classification of ingredients by their role in a dish.

Faculty and Staff Leads

Scott Bradley

Senior Engineer

Mercurial technologist currently focused on electronics and natural language processing.

Project Details

2019 Fall

Important Questions
  • How can structure improve presentation. E.g. creating the initial semiotic of the recipe by showing an iconified ingredients list; separating prep from cook phases and showing the prep's resulting mise en place; color coding things such as ingredients by type?
  • What tools and techniques in the field of Natural Language Processing/Generation, and related technologies, are available for reducing, simplifying, summarizing, etc? To what extent can these tools be applied to automate the iconification process?
  • What can be iconified? What cannot? What is the balance between semiotic and narrative in the presentation of instructional material?
  • The state of the art of NLP is unlikely to be in a place where we can produce consistently excellent automated results within the scope of a quarter. Thus, to what extent does it make sense to explore a semi-automated approach to iconfication? What is the user interface that will create a synergistic process between human and AI for quick and accurate generation of iconified instruction?
  • To what extent can we embrace the fungibility / flexibility of cooking and use this to our advantage in the presentation? Can we build aspects into the user experience that will encourage experimentation, exploration, substitution, and a generally creative approach to cooking? Can the UI and general presentation model serve not only to simplify recipes, but to empower amateur chefs to break the mould of the recipe and to experiment?
  • To what end can the identification of cooking fundamentals be utilized in the simplification process? Techniques are, for the most part enumerable (braise, roast, fry, etc.). Ingredients and food types can be categorized, e.g. by their role in a dish (e.g. salt, fat, and acid). Can identification of these types and taxonomies enable certain approaches to simplification, substitution, variation, etc.?
  • Can analyses of recipes help us determine which ingredients "go together". Can similarities, differences, and patterns help us to identify meta-recipes and variations? To what extent might it make sense to produce an ad-hoc recipe engine vs direct iconic translation of existing recipes?
Sample Milestones
  • Define an expanded semiotic of cooking instruction
  • Determine linguistic components that can/cannot be iconified
  • Explore layout and workflow designs of iconified and mixed (icons + narrative) cooking instruction
  • Explore and develop natural language processes for simplification, summarization, iconification, etc.
  • Explore and design a user interface for recipe iconfication

The team will develop a set of technologies proving out the ideas germane to the core concepts of ICONIFY, including: The development of a domain-specific (cooking) semiotic. The reduction, simplification, and iconification of cooking instructional language. This may be automated or semi-automated, and will utilized combined skills of NLP and of UX design. The design and presentation of simplified cooking instruction.

Apply to Project