i love google maps and streetview, documented here.

i've always been fascinated with the world around us. so i decided on building a creative mode sandbox for google earth.

was also inspired by this project which was built with mapbox.

on top of the mapbox graphic library, arcki has a built in model generator. the user enters a prompt, which should be detailed, but stripped down to minimal examples could look like: "authentic japanese temple", "brutalist apartment complex", or "cn tower replica". the generated 3d model can be published to a global model library, where other users can see it, place it into their worlds, and build on top of it.

the challenge was combining three domains that don't usually work together: geospatial data, ai powered image to 3d generation, and a browser based 3d engine.

arcki is a full stack application built on mapbox gl and with a fastapi backend.

client side, arcki renders a comprehensive mapbox globe with react three fiber overlaying generated glb models. indexeddb stores model data locally, and localstorage persists world state: camera position, deleted buildings, weather settings, so scenes are persisted in memory.

on the backend, fastapi orchestrates a multi stage ai pipeline: gpt-4o cleans and enhances prompts, dall-e 3 generates a reference image, and fal.ai's trellis model reconstructs a 3d mesh from it. redis tracks async jobs with ttl based expiration, with a graceful fallback to in memory storage for local development.

the whole pipeline produces a text → image → 3d model output in ~30-60 seconds.

the editor is where everything comes together. it's a 1700+ line react component integrating mapbox gl, react three fiber, and mapbox draw into a single interactive surface.

the toolbar provides four core modes: select, delete, insert, and generate. each mode changes how the map responds to user interaction.

in select mode, clicking a building reverse geocodes its coordinates with mapbox's api to show the building name, address, and dimensions. we use turf.js to calculate real world area and dimensions from geojson polygons, converting degrees to meters with a cosine correction for longitude convergence at the poles.

delete mode uses mapbox gl draw to let you draw polygons. any building within that polygon gets removed from the extrusion layer. insert mode lets you drop custom glb files onto the map. and generate mode opens the ai panel.

once a model is placed, the user can manipulate it with a custom built svg transform gizmo, similar to what you would find in blender or unity.

the gizmo is rendered entirely in svg with colored axes: red for x, green for z, blue for y. it switches between move mode (arrows) and rotate mode (elliptical rings) with real time degree readouts.

the tricky part was mapping 2d mouse deltas to 3d world transformations. horizontal mouse movement maps to x-axis movement and rotation angles, vertical movement maps to height or depth, and the center circle enables unconstrained free movement:

if (dragAxis === "rotateX" || dragAxis === "rotateY" || ...) {
  const deltaAngle = deltaX * 0.5;
  let newRotation = (rotationStartRef.current + deltaAngle) % 360;
  onRotate(rotAxis, Math.round(newRotation));
} else if (dragAxis === "y") {
  onHeightChange(-deltaY * 0.1);  // vertical mouse = height
} else if (dragAxis === "free") {
  onMove(deltaX, deltaY);
}

all of this is screen space delta calculations that get translated into geographic coordinate offsets.

the generate panel is a three stage workflow: input → preview → placing.

users describe what they want, pick an architectural style (architectural, modern, classical, futuristic), and hit generate. the panel shows a 2d preview while the 3d model generates in the background.

a subtle optimization: 3d generation starts automatically as soon as the 2d preview is ready. so while the user is reviewing the image, the model is already being built. if they click "finish & place" immediately, they might only wait a few more seconds instead of the full generation time.

// Auto-start 3D generation when preview completes
useEffect(() => {
  if (previewResult && workflowStage === "preview" && !threeDJob) {
      start3DGeneration(previewResult.job_id);
  }
}, [previewResult, workflowStage]);

the panel also supports minimization. you can close it and the generation keeps running. a minimized pill shows progress ("42% complete") and expands back to the full panel on click. all state is persisted to localstorage, so even closing and reopening the panel mid generation picks up right where it left off.

the editor supports day/night mode and weather effects: rain, snow, clear skies. these can be toggled from the weather panel or controlled through natural language in the search bar ("make it rain", "night mode"). environment state persists across sessions, so the atmosphere is restored on reload.

when a model is ready, the client downloads the glb file and enters placement mode. click anywhere on the map to drop it. after placement, the model becomes fully interactable. select it, drag it with the transform gizmo, rotate it, publish it to the global model library, or delete it. these features, when combined, makes arcki a sandbox.

everything the user does, placing models, deleting buildings, changing weather, and moving the camera, is persisted locally. localstorage handles world state metadata, but glb files are too large for localstorage, so they go into indexeddb:

export interface SavedWorldState {
  insertedModels: SavedModel[];
  deletedFeatures: GeoJSON.Feature[];
  lightMode: "day" | "night";
  weather: "clear" | "rain" | "snow";
  camera: { center: [number, number]; zoom: number; pitch: number; bearing: number };
}

there's a dual storage strategy: models can live locally in indexeddb or be uploaded to supabase for the global shared model library. when a model is published, react three fiber renders its thumbnail so other users can browse it visually.

models with a supabase url are fetched from the cloud; local only models get their glb data restored from indexeddb and converted back to blob urls. world state saves are debounced by 1 second to avoid excessive writes during rapid interactions like dragging.

the search bar isn't just a geocoder. it's an ai powered intent parser that understands natural language and maps it to actions.

every search query goes through gpt-4o to classify intent into one of eight action types:

SEARCH_INTENT_PROMPT = """Parse user queries to understand their intent.
Users may have typos, misspellings, or use informal language.

1. **action**: One of:
 - "navigate": go to a specific location ("take me to Paris")
 - "find_building": find by characteristics ("tallest building here")
 - "set_weather": change weather ("make it rain")
 - "set_time": change time of day ("night mode")
 - "camera_control": adjust camera ("zoom in", "bird's eye")
 - "delete_building": remove a building ("delete the CN Tower")
 - "question": ask about a place ("how tall is the Burj Khalifa?")

CRITICAL RULES:
- use your world knowledge. "6th tallest building in the world"
→ navigate to "Goldin Finance 117, Tianjin, China"
- ALWAYS correct typos. "tke me to Prais" → "take me to Paris"
- For vague queries like "somewhere with good sunrises"
→ resolve to a REAL location: "Santorini, Greece"
"""

what makes this powerful is how it handles typo correction, world knowledge resolution, and vague queries simultaneously. "tke me to the eifel twoer" correctly resolves to "eiffel tower, paris". "bring me somewhere cold asf" could resolve to "tromsø, norway".

for find_building queries like "tallest building here", we fetch real building data from openstreetmap's overpass api with failover across multiple endpoints. buildings are ranked by the query's attribute: height, footprint area, or an "underdeveloped" heuristic that finds buildings with large footprints relative to their height (high area to height ratio = a lot of land "wasted" on a short building).

the delete_building action is fun. it geocodes the building, then generates a deletion polygon around it scaled to the building type. towers get small, precise polygons; stadiums and arenas get larger ones since they're wide.

building selection uses nominatim for reverse geocoding. we wrote a display name shortener that strips long addresses into readable labels:

def shorten_display_name(full_name: str, address_details=None):
  # Full: "CN Tower, 301 Front Street West, Toronto, Ontario, Canada"
  # Short: "CN Tower, Toronto, Canada"

  if address_details:
      parts = []
      for key in ["tourism", "building", "amenity"]:
          if key in address_details:
              parts.append(address_details[key])
              break
      # ... city, country
      return ", ".join(parts)

this prevents addresses like "cn tower, 301 front street west, entertainment district, toronto, golden horseshoe, ontario, m5v 3t6, canada" from cluttering the ui. it strips down to "cn tower, toronto, canada".

rendering 3d models on a mapbox map in the browser required careful optimization. heavy components like the toolbar, search bar, generator panel, and transform gizmo are wrapped with React.memo() to prevent rerenders when the map moves. modals and the tutorial are loaded with next/dynamic and ssr: false to keep the initial bundle small.

one subtle issue: mapbox event handlers are registered once during initialization. without refs, they capture stale state from the initial render. so every piece of state that mapbox callbacks need is synced to a ref:

const activeToolRef = useRef(activeTool);
useEffect(() => {
  activeToolRef.current = activeTool;
}, [activeTool]);
// ... same for deletedFeatures, insertedModels, isPlacingModel, etc.

this pattern repeats for about a dozen state variables. it's not elegant, but it's necessary for mapbox's callback model.

each generated model costs approximately:

the range depends on whether landmark enhancement triggers (extra gpt-4o vision calls + image search).

arcki was a massive learning experience. the technical challenges were genuinely hard. combining mapbox's terrain rendering with react three fiber, engineering prompts that produce reconstruction friendly images, building a multi stage async pipeline with job tracking, and persisting model data across browser sessions all took real work.

the prompt engineering alone took ample iteration. small changes to the dall-e prompt, like adding "no pedestal" or "floating in white void", dramatically improved the output quality. the landmark enhancement pipeline was a late addition that made a huge difference.

the 3d model search was equally irritating. going from hugging face models on modal to fal.ai's trellis, then realizing multi image mode caused issues with dall-e's inconsistent shadows all show constraints of image to 3d reconstruction.

thanks to austin for doing most of the mapbox work, and faiz and maaz for helping with the image gen pipeline!

check out the github repo for the full source (give us a star ⭐️)!