2023 reflection: on conviction and building 20+ AI prototypes
Year-end projects recap and my word of the year
This newsletter covers what I’m building and learning in creative AI, as I move towards what I’ll work on next. Today I’ll talk about:
Building creative interfaces for generative AI in the #AIxUIdailies coding challenge.
Reflecting on the year as a whole and my word of the year: conviction.
Projects recap and insights building AI prototypes over past year
Wishing you a warm and restful last few days of 2023, and all the light as we go into 2024.
Think outside the chat box: Exploring new interfaces for generative AI with #AIxUIdailies
I feel like generative AI is a new mysterious material and we’re all still figuring out different ways to extract value from it. The area I see a ton of opportunity is where the technology meets us, at that layer between the models and people. Yvonne and I were discussing the possibilities of new interfaces to genAI models, and this led to us to start a daily coding challenge to build out these ideas. Think outside the chat box!
I was so happy to have completed this challenge. It was intense and also super gratifying. I learned a ton. Here are the prompts, ideas doc, and my prototypes from the week:
App to build interactive sound & visual websites: I made an app that uses AI to make interactive sound and visual websites given a description.
Physical LCM: I made an app to create open ended and tangible interfaces for real time image generation, by using a webcam and Latent Consistency Models.
Multi-user storytelling with AI: What would it be like to make more collab experiences with AI? I made a two person interactive storytelling app, the Adventures of Kiki and Bouba.
Visualizing a latent space of wonder: a.k.a “Wandering things to do to invite more wonder, sometimes when you are alone.” I used embeddings of a list of playful, imaginative activities to explore this latent space of wonder.
Particle system image generation: How might we use UI metaphors that are more amorphous or fluid to express the probabilistic quality of genAI? This prototype uses a particle system to render images generated from AI.
Year-in-review: My year of conviction
For the last several years I’ve picked a word of the year. I love doing this. From my 2023 planning:
My word of the year is conviction. How might I create structures of support to build motivation and focus my energies, in particular, in work and relationship. I want to find work that is motivating to me, and lucrative. I want to find and build a relationship where I can see having kids and a family with my partner.
The year began in a cloud of uncertainty. How would I figure out these foundations of relationship and work? I looked into the unknown, and I created structure to act and learn, to move towards it. I started weekly accountability calls, I read about learning and uncertainty. Gentle sparks formed and lit up a way. What-ifs that felt right became more and more affirming. I took a few leaps with a deep knowing.
I followed a what-if spark back to the bay and visited an old friend in Oakland. Then a few months later we spent a week together in New York. We planned, we talked through scary things, we did the work, we giggled, we imagined, and we made things real. In the autumn, I moved in with Helen. Decisions become clear when you fall in love.
For work, I planned to spend the first half of the year building different ideas, and then pick one to go deep on the rest of the year. When I didn’t feel conviction on an idea, I started looking for full time work. I’ve moved back to the Bay Area, the center of the world in AI. It occurred to me that the best way to prepare for the job I want is to start doing it. So I focused on making demos exploring interfaces and AI, creating videos of them, and talking to people about them. I haven’t found work yet, but this feels like it is bringing me closer.
To begin the year starting to code again, and then now complete the technically and creatively intense #AIxUIdailies challenge was enormously gratifying. It made me feel more confident in what I can do.
Insights:
When you don’t know what to make, make whatever. This will lead to more interesting ideas.
Tiktok-style video is the new “blogging”. If 10 years ago I’d write a post, now it’s about making a video about it. Once I started doing this, my ideas went further.
Pushing just beyond abilities leads to flow & confidence.
Designing for desperation leads to faster learning
A call once a week with an accountability buddy is enough to force structure and get into a daily flow.
Set the intention, let go of the outcome. Put in the effort, consistently, and be open to what happens.
Presence is powerful. Maintaining presence through difficult conversations (while being aware of boundaries and needs) is everything for relationship growth.
Turn towards the creative possibilities that the current circumstances allow for. What unique opportunities does this exact moment provide?
2023 Projects Roundup
I built 20+ AI prototypes over the course of the year. Here’s what I learned prototyping generative AI tools, first published in my Nieman Lab article here.
It’s better to augment skills rather than replace with AI. Using chatGPT to write is okay, sometimes good, but not great. Instead, using it as a place to bounce ideas or get started quickly in a larger workflow is wonderful. I’ve found this for programming too — it’s easier to edit and add rather than start from scratch when coding.
LLMs make text malleable. You can quickly expand bullet points into paragraphs or contract articles into summaries. Given some text, you can shape it for a particular interface: a summary for a newsletter preview, a quick phrase for a notification. To illustrate this, I made an app where you can pan across a slider and see the article length go from a two-word summary to the full text length.
User interface for LLMs is an open field of opportunity. LLMs are powerful and mysterious and the way we interact with them is through chat, which is simple, but not entirely welcoming. You have to learn how to prompt and get a feel for what LLMs are good at. A blank input, with no indication really of what it can do, feels analogous to the command line. The command line as a UI gave way to a more welcoming, intuitive, interactive, and expressive interface: graphical user interfaces. Now expanding the analogy, what will the future interfaces look like for LLMs? To put it succinctly — command-line : GUI :: llm chat :: ???. This is the space I’m most excited about. We have the opportunity to define a new human-computer interaction for this era of LLM computing.
And now for a roundup of my prototypes!
Gas Me Up: I made an app to see me and say nice things. It uses GPT-4V, OpenAI’s TTS to share affirmations.
Zooming in summaries: Zoom in and out of an article to get a quick overview summary and then progressively add more details until you see the full article. (demo and live app)
Quick pep talk: When I’m feeling stuck on a project I use this app to say nice things about what I’m working on.
Doodle & diffuse is a collaborative AI drawing prototype that combines a drawing canvas with StableDiffusion Img2Img. (30s demo, or the more detailed one)
Event flyer → Calendar: I made a GPT that takes in a photo of a flyer and automatically adds it to your calendar. (Link to GPT, demo video, twitter thread with details)
Nieman Lab Predictions GPT: I was invited to contribute to Nieman Lab‘s year end predictions. For my 2024 prediction I shared a few insights building with LLMs over the last year and then downloaded all of the previous year end predictions - all 1,369 of them - and made a custom GPT.
Summaries Tree: Imagine navigating an article as a tree, getting an overview, and then zooming in on a section for more details. This prototype illustrates navigating nested summaries of an article. (demo and live app)
tldr-cl: A command-line script to summarize a web page, coding it with gpt4.
I made a few bots to help reframe certain ways of thinking and support healthy behavior changes:
Creative Coach AI helps me get unblocked and motivated.
Gratitude bot helps me be more aware of things to be thankful for.
Enthusiasm bot helps me get excited about things I want to do but feel hesitant about.
Meditation bot helps me chat and settle my thoughts and become more mindful.
“Get started building with generative AI” is a tutorial for creating a chatbot with the OpenAI chat API. See this code for a streaming version of this app.
I put together a series of clips chatting with this bots for AI care videos, played at Fun-a-Day 2023.
Ally: an AI-powered web app that makes use of LLMs to generate code to make websites more accessible. (tweet and blog post)
Voice to text: This example shows how to record audio from the browser, play it back, and also get a transcription from openAI. The app is built with nextjs.
Image based Q&A: Asking a photo questions, through image based reasoning, by using BLIP-2.
yt-transcript: Script to get a transcript from a youtube video with Whisper.
Drawing games with AI: What it would be like to play a drawing game with AI? For this game, I look at the resulting image and describe it in a new prompt, and then run SD again with the image and the new prompt. (video and blog post)
Building custom GPTs: Creating a range across productivity, creativity, and wellness.
Open/Closed Door Classifier: This was my first app of the year! This involved building and deploying a classifier, and creating a web UI.
Events and Talks
I love throwing events to further explore new projects and things I’m getting into. Here are events I organized, and a couple talks I gave about my work:
Into the unknown together: Creative people getting together and sharing what they’re making, curious about, and planning. A combination workshop and social gathering in LA, organized with fellow creative dynamo Maria Scileppi.
Taking the Leap: a gathering in DC to talk about new projects, next steps, and leaps.
Boston Tech Poetics Show & Tell & Hang Happy Hour: Science fair style event in Boston to meet up and setting up creative tech projects around the space.
Hacking Models: Show and Tell: A space to bring creative sketches, AI experiments, LLM hacks — showing things in progress too, to get feedback, and jam on ideas in the AI space. Hosted in SF at NFX.
I gave a talk at *the intersection of art and technology* #5 sharing about my web synths.
Talk and live demo of doodle & diffuse, taking prompts from the crowd, at Art Party in Oakland.
Thanks for reading!
📚 Check out my AI Resources list. I made this list for myself to stay up to date on AI things and organize resources I find helpful.
📞 Book an unoffice hours conversation: We could talk about something you’re working on, jam on possibilities for collaboration, share past experiences and stories, draw together / make a zine, or meditate.
🙌 Follow what I’m up to by subscribing here and see my AI projects here. If you know anyone that would find this post interesting, I’d really appreciate it if you forward it to them! And if you’d like to jam more on any of this, I’d love to chat here or on twitter.