Agentic Heera

AI-driven 2D to 3D jewelry generation with real-time customization.

Stack (as requested)

• Frontend: React + Three.js + Tailwind CSS
• Backend: Python + FastAPI
• AI Vision: Gemini Vision API (OpenAI-compatible endpoint)
• 3D Generation: InstantMesh (via command adapter)
• Version Control: GitHub

Project Structure

• app/ FastAPI backend
• frontend/ React app with Three.js viewer
• outputs/ generated previews and exported models

Backend Setup

1. Create virtual env and install deps:
   • pip install -r requirements.txt
2. Copy env:
   • copy .env.example .env
3. Set environment variables in .env:
   • GEMINI_API_KEY
   • OPENAI_BASE_URL (default https://generativelanguage.googleapis.com/v1beta/openai/)
   • VISION_MODEL (default gemini-2.0-flash)
   • INSTANTMESH_CMD (optional, command with {input} and {output} placeholders)
4. Run backend:
   • uvicorn app.main:app --reload

Backend runs at http://127.0.0.1:8000.

Frontend Setup

1. Go to frontend:
   • cd frontend
2. Install deps:
   • npm install
3. Run:
   • npm run dev

Frontend runs at http://127.0.0.1:5173.

API Endpoints

• POST /api/generate (multipart image)
• POST /api/customize/{design_id}
• POST /api/export/{design_id}
• GET /health

Generation Flow (Database + Gemini)

POST /api/generate now follows this sequence:

1. Save uploaded image in SQLite (outputs/designs.db).
2. Fetch the saved image bytes from the database.
3. Send fetched image to Gemini Vision (VISION_MODEL) for structured analysis + detailed description.
4. Store Gemini outputs (jewelry_type, components, notes, description) in SQLite.
5. Build 3D geometry using both image parsing and description-guided heuristics.
6. Save generated preview/GLB paths back into SQLite.

The generate response includes a description field with Gemini's detailed jewelry description.

InstantMesh Integration

Set INSTANTMESH_CMD to your local command. Example:

• python C:/path/to/instantmesh/infer.py --input {input} --output {output}

If InstantMesh command is not set or fails, backend uses local procedural fallback mesh generation so the demo remains runnable.

Notes for Hackathon Demo

• First generation returns a pre-populated model (metal body + stones + settings + prongs).
• Customization updates materials/colors/gem type without full regeneration.
• Budget-aware suggestions are returned when estimate exceeds budget.
• Export supports both GLB and STL.

PPT LINK : https://drive.google.com/file/d/1P0Au7ySBqHy2ItmgNbARoMQaC5Av16ph/view?usp=drivesdk

video link : https://drive.google.com/file/d/1w2Nep3LGJwSBW0xxqyk-CWdfsgbRkLNI/view?usp=drivesdk