The Butterfly Path

The Butterfly Path : Voice-driven AR butterfly discovery on Snap Spectacles

01 · The Challenge

Nature is vast, beautiful, and inspirational — a treasure trove of discoveries for naturalists and researchers of any age. But there are two challenges: naturalists and researchers constantly seek more efficient ways to identify and document species in the field, while learners and educators seek more engaging ways to connect with local biodiversity. We created The Butterfly Path to address both — using butterflies as our example — by simplifying ecological observation and turning discovery into an interactive, gamified learning experience.

We started by talking to the people who live this every day. We interviewed naturalists and researchers who are experts in the whole process of noticing, identifying, recording, and sharing wildlife, and asked how that expertise could be shared with everyone — through the wonderful technology of Spectacles.

A recurring theme was that real curiosity starts with first-hand exposure and close observation. As Professor Antonia Monteiro, an expert in the evolution and development of butterfly wing patterns, told us:

"Experiences that got me hooked into butterfly research, I would have to say that it was plenty of exposure to nature outings during college, where I could observe butterflies first hand at close distance. It was their colors and beautiful patterns that attracted me to them. Then, a book by physicist Fred Hoyle, that mentioned that some patterns, such as eyespots, should not be able to evolve gradually, got me thinking about eyespot evolution. Those two things got me started in the research I do now."

Close observation plus a story that piques your curiosity is what turns a passing glance into a lifelong interest. So we built the experience around that idea — an eager guide that is part enthusiastic, nurturing naturalist, part knowledgeable researcher, designed to encourage curiosity, model Socratic questioning, and share stories that spark interest.

02 · The System

The experience follows a natural discovery loop — Noticing → Identifying → Recording → Sharing → Collecting — all driven by voice and your gaze.

Noticing. A Gemini Live voice agent runs the whole experience hands-free: raise an open palm toward your face and pinch to talk. A Naturalist agent guides you to slow down and observe through Socratic questions ("what do you notice about its wings?"), turning a passing glance into real attention. You can also ask the agent to scan for butterflies, which sweeps the camera view for a few seconds and flags any it finds.

Identifying. Once a butterfly is found, the agent kicks off species identification, then spawns a 3D info card filled with species data + photos, including its conservation status (a Secure-to-Critical gradient) so you understand how threatened it is. AI wing generation turns the identified species into a bespoke wing texture + opacity map applied to a live 3D butterfly that flutters into your space, while an Archivist agent shares facts, migration stories, and context.

Recording. Every sighting (species data, photos, and GPS) is persisted to the cloud via Supabase, gated by your Snap identity, so what you find is documented and attributed to you.

Sharing. A floating AR minimap pins real-world sightings around you (including ones logged by other players) and opens a detail card on tap, turning every identification into part of a shared, persistent sighting world.

Collecting. Identified butterflies fly in and perch on your index finger, and each one joins your personal collection, where you can revisit every species you've found, each rendered as its own AI-generated 3D butterfly. Discovery becomes a gamified, growing gallery you build over time.

03 · The Architecture

Identity and persistence. Supabase on Snap Cloud with Snap OIDC authentication via signInWithIdToken({ provider: "snapchat" }). Each player gets a stable Supabase UUID bound to their Snapchat account. Row-level security sets user_id server-side from auth.uid(), so the client is never trusted with identity.

Storing sightings. When a butterfly is identified, SupabaseDBManager grabs GPS and the Snap display name, uploads the photo and generated wing textures to Supabase Storage, and inserts a structured species row, all in parallel via Promise.all.

Finding nearby sightings. "Butterflies near me" is a two-stage query: a cheap lat/lon bounding box on the database, then a precise Haversine distance filter on-device, sorted nearest-first.

AI-generated wing textures. Heavy generation runs off-device on a Snap Cloud Edge Function: the lens POSTs a reference image to a function using Replicate to run Nano Banana Pro, producing both a clean wing texture and a matching opacity map. The lens decodes them in parallel and applies them to a live 3D butterfly mesh — so the butterfly that flutters in front of you is the same species you just identified, with bespoke AI-generated wings, and none of the model inference runs on the headset.

Voice-driven AI guide. The experience is fronted by a Gemini Live agent (gemini-2.0-flash-live-preview-04-09) handling real-time speech-to-text, text-to-speech, and camera vision over a WebSocket. On top of it, a multi-agent education system: an AgentOrchestrator runs LLM-based routing (no hard-coded if/else) to pick the right agent and tool per utterance, and two agents with distinct teaching styles collaborate mid-conversation — a Naturalist that guides discovery through Socratic questioning and an Archivist that follows up with species facts, migration stories, and context. The same agent layer activates every feature hands-free through a ToolRouter: ButterflyDetectionTool runs a short on-device YOLO scan (deduplicating frame-to-frame hits by IoU overlap) and auto-chains into ButterflyIdentificationTool for species ID, while SpatialTool, LocationTool, and WeatherTool give the agents real-world context to teach from. A priority queue with a depth limiter keeps the two agents from ping-ponging forever. The provider layer is pluggable: Gemini Live is primary, with OpenAI Realtime wired up as a fallback.

Palm-up push-to-talk. Voice input is gated by a deliberate gesture rather than always-on listening. PalmPushToTalk watches for an open hand raised toward the face, pops a hand-anchored mic menu, and streams audio only while a pinch is held. A small activity indicator lights up the mic while recording.

The AR map. Built on Snap's Map Component. NearbySightingManager reads GPS once and drops live pins; a custom sweep-line collision solver spreads overlapping markers apart so off-screen quest markers and labels stay legible.

Info cards. Cards bind data by SceneObject name (case-insensitive), so designers can rearrange the prefab without touching code. A conservation status bar maps IUCN/red-list rating onto a Secure-to-Critical colour gradient with a hover tooltip.

04 · The Challenges

Flaky CDN images. Some species reference images return HTTP 200 but silently fail at loadResourceAsImageTexture. We had to detect and filter those URLs before display so cards never showed broken slots.

Readable AR markers. When sightings cluster, floating labels overlap into an unreadable pile. A sweep-line collision solver spreads overlapping markers apart in 1D (along screen edges) and 2D (in-view).

Wing generation from Spectacles photos. Photos captured directly from Spectacles are often insufficient quality for AI wing generation. We fell back to reference images from the Kindwise insect.id database instead — the trade-off being generated wings reflect the canonical species appearance rather than the specific individual spotted.

Wing generation latency. Running the wing texture through Replicate takes roughly a minute end-to-end. The quality of the resulting texture is substantially better than faster alternatives, so we accepted the wait rather than compromise the visual output.

Training a custom SnapML model. We trained a custom SnapML model for on-device butterfly detection and bounding-box localization. We got reliable detection but couldn't reach accurate per-species classification in time, so we fell back to the Kindwise insect.id API for identification while keeping on-device detection.

05 · The Evidence

A fully shared, persistent sighting world: sightings logged by one player show up as real-world AR pins for everyone else, with photos, species data, and attribution intact. Real geospatial "near me" queries running end-to-end on the headset. AI-generated wings with texture and opacity map applied to a live 3D butterfly mesh. 3D butterflies that land on your index finger, managed by a lifecycle system that tracks, animates, and retains every species collected. A personal butterfly gallery where each species is rendered as its AI-generated 3D model. Gemini Live-controlled workflow that activates tools and features entirely through voice, delivering a true hands-free AR experience. Custom on-device SnapML butterfly detection. Cross-continental team collaboration across a 13-hour Singapore/US time difference with no in-person overlap.

06 · My Contribution

I owned the agentic backend — designing the multi-agent education system (AgentOrchestrator, Naturalist + Archivist agents, ToolRouter), connecting every tool and feature to the Gemini Live voice interface so the entire experience runs hands-free through conversation. I researched educational technology and learning objectives to ground the agent personas in real pedagogy, wrote the Socratic questioning patterns, and structured the tool-routing logic that lets voice commands activate butterfly detection, species identification, spatial queries, and weather lookups without hard-coded branching.

I also experimented with training a custom SnapML classification model for on-device species identification, though we ultimately fell back to the Kindwise API when per-species accuracy wasn't reliable enough in the available time.

I produced and edited the showcase film, focusing it on learning objectives — the explicit ask of this hackathon — and stitched together footage, screen capture, and voiceover into the final narrative. This was also my first time in Lens Studio and on Spectacles.

Built with

Snap Spectacles + Lens Studio 5.x · Snap OIDC auth + Supabase on Snap Cloud · Supabase Postgres with row-level security · Supabase Storage · Supabase Edge Functions (species ID via Kindwise, wing-texture generation via Replicate + Nano Banana Pro) · Gemini Live (real-time voice + camera vision over WebSocket) · Multi-agent education system (AgentOrchestrator, Naturalist + Archivist agents, ToolRouter) · OpenAI Realtime (fallback voice provider) · On-device YOLO butterfly detection · Palm-up push-to-talk (SIK hand tracking) · Snap Map Component · Spectacles Interaction Kit · Spectacles UI Kit · Kindwise insect-ID API

Collaborators

• Joe Chung — Agentic backend (multi-agent system, voice interface, tool routing), educational technology research, video production, and SnapML classification model experimentation.
• Agrika Gupta — API integration for butterfly identification (Supabase Edge Functions), video recording, backend infrastructure, and wiring features for end-to-end product delivery.
• Boon Jun Aw — Nearby sighting system, Snap Cloud data storage and retrieval, AI wing texture generation via Nano Banana Pro, butterfly flight animation, and info display system.
• Nicolletta Rothschild — Custom ML model training for on-device butterfly detection using synthetic and real-world captures.
• Marina — Design.