Project Overview

Design a handheld device that generates real-time visuals of conversation.

Buddy resolves the disconnect of group work by acting as an intermediary that captures conversations in real time through LLM-powered image generation. Built with rapid prototyping, electronics, and full-stack software development, it preserves discussions as a visual history and saves valuable concepts from being lost to misarticulation.

The Problems

Our Goals

Diagrams made with Napkin.ai and Figma

The Research

7.5 hrs per week lost to miscommunication in knowledge work.

Participant

✦✦

Adam

31 Years Old

Product Designer

Portrait generated with Gemini

Frustrations

• Hard to regain alignment when spoken ideas are interpreted differently by each teammate.
• Little visibility into what was actually agreed on once a working session ends.
• Good work still feels like it stalls when concepts are lost to misarticulation or memory.

Ideation

Low-Fi Wireframes

Early Drawings

Iteration 1

Selected for construction

Dial for changing through iterations, button for on/off on top, and version with an on/off and commit button on the front.

Iteration 3

Send button and an on/off on the front.

Iteration 4

Basic set up with visual as main portion.

Hand-drawn sketches

Hi-Fi Mockups

User Testing

Tested with a group of designers and engineers during a live working session.

Participants used Buddy across a real design critique session. Tasks included: capturing a decision, exporting a session summary, and interpreting a generated diagram.

Some thinkable quotes

Engineering

Diagrams created with Mermaid.js code and Python in a Figma plugin.

Diagrams made with Mermaid.js (rendered from Python)

system_prompt = f"""
You are a Visual Assistant. You generate Mermaid.js code OR Fal.ai image prompts.

CURRENT MODE: {DIAGRAM | SKETCH} (switch based on intent)

TASK:
1. ANALYZE USER INTENT:
   - Chart, graph, flow, or timeline -> output mode: "DIAGRAM"
   - Scene, photo, texture, or visual style -> output mode: "SKETCH"
   - Referring to "it"/"the image"/"that" -> use CONTEXT HISTORY

2. FOR DIAGRAMS (Mermaid):
   - Return valid Mermaid code only. No backticks.
   - Support: graph TD, mindmap, pie, sequenceDiagram, xychart-beta, gantt.

3. FOR SKETCHES (Images):
   - If refinement, keep core details and apply the change.
   - set "is_refinement": true only if editing the previous image.

Return JSON ONLY:
{ "mode": "DIAGRAM" | "SKETCH", "prompt": "...", "is_refinement": true|false }
"""

# Create a PDF summary to ensure universal accessibility
pdf_buffer = io.BytesIO()
c = canvas.Canvas(pdf_buffer, pagesize=letter)
text_obj = c.beginText(40, 750)
for line in summary_lines:
    text_obj.textLine(line)
c.drawText(text_obj)
c.save()

# Package into a downloadable ZIP artifact
zip_file.writestr("session_summary.pdf", pdf_buffer.getvalue())
return StreamingResponse(
    io.BytesIO(zip_buffer.read()),
    media_type="application/zip",
    headers={"Content-Disposition": "attachment; filename=session_export.zip"}
)

Hardware

Progress Photos

Building the physical enclosure on a Raspberry Pi, hardware constraints forced architectural clarity that cloud deployment never would have.

‹

›

‹

›

Final Product

The Finished Build

‹

›

Impact

What I Learned

1. 1

   Budget Is the Real Constraint

   Budget is the most important constraint in any project. Things you want often have to be sacrificed just to get the product to market—and to earn enough to fund the next version.

2. 2

   Scale Before You Fabricate

   Scalability should be thought about earlier. It was fun to get hardware running and learn shell scripting, but building an individual handheld for everyone at this size and scale is too expensive.

3. 3

   Less Is More

   Complex problems often have simple solutions. It is easy to get caught up in complexity and overdesign—in the words of Mies, less is often more.

Considerations

Bibliography