Multimodal Image Geolocation (Geo-HiPO)
Trained Vision-Language Models to predict where photos were taken by teaching models physical distance instead of classification, running experiments across multi-node H100 and V100 GPU clusters.
The Problem
Standard image geolocation treats location prediction like multiple-choice classification. Under that setup, guessing a neighboring town is scored as completely wrong, the same as guessing an opposite continent.
During my research appointment at the University of Central Florida Center for Research in Computer Vision (May 2025 - Aug 2025), I worked on Geo-HiPO. The goal was to train vision-language models to output real-world geographic coordinates using continuous physical distance as the supervisory signal.
How It Works
We formulated geolocalization as preference optimization (DPO) based on physical distance:
- Candidate generation: The vision-language model outputs multiple possible location guesses for a given image.
- Geocoding & distance calculation: Predicted locations are resolved to GPS coordinates and scored using Haversine great-circle distance against ground truth.
- Preference formulation: The prediction closer to ground truth becomes the chosen sample and the farther guess becomes the rejected sample for DPO.
- Curriculum scaling: The training pipeline scales hierarchically, learning continents and countries first before fine-tuning on regional cities.
Research Poster Presentation
Published & Presented Research Poster
Cluster Engineering & Training Stack
- 30M+ image pipeline: Built multiprocessing data loaders in Python to clean and curate 30M+ images from MP-16 and IM2GPS3K, eliminating over 100 hours of pipeline overhead.
- Geocoding verification: Filtered out invalid coordinates, ambiguous locations, and model hallucinations using GeoNames and OpenStreetMap datasets before training.
- High-throughput inference: Integrated vLLM to run batched candidate generation on Qwen2.5-VL and Kimi-VL checkpoints.
- Distributed optimization: Configured FlashAttention-2, PyTorch automatic mixed precision (AMP), and DeepSpeed to fit multimodal models across NVIDIA H100 and V100 clusters.
- Systems debugging: Solved GCC and CUDA version conflicts on cluster nodes and resolved token padding bugs between image patch embeddings and text sequences.
Results
Benchmarking on IM2GPS3K showed clear advantages over standard training:
- Qwen2.5-VL-7B with HiPO reached 35.87% accuracy within 25 km and 54.59% within 250 km, improving over the base model (34.13% and 52.22%).
- Standard supervised fine-tuning (SFT) degraded accuracy to 33.97%, proving that distance-based preference ranking produces a significantly stronger supervisory signal than direct token imitation.
Technologies Used
- Vision-Language Models: Qwen2.5-VL (3B, 7B, 32B, 72B), Kimi-VL
- Deep Learning Frameworks: PyTorch, Hugging Face Transformers, Hugging Face Datasets, Hugging Face Accelerate, TRL, PEFT, LoRA
- Optimization & Alignment: Direct Preference Optimization (DPO), SimPO, Supervised Fine-Tuning (SFT), Curriculum Learning
- ML Acceleration & Runtimes: vLLM, DeepSpeed, FlashAttention-2, PyTorch AMP, bitsandbytes, FP8, NF4, INT8 Quantization
- HPC & Cluster Infrastructure: SLURM Workload Manager, NVIDIA H100 GPUs, NVIDIA V100 GPUs, Multi-Node Clusters, CUDA, GCC
- Computer Vision & Image Processing: OpenCV, Pillow (PIL), Albumentations, Scene Understanding, Visual Reasoning
- Data Engineering & Geocoding: Python Multiprocessing, Custom Data Loaders, Haversine Distance, GeoNames, OpenStreetMap (OSM), NumPy, pandas
- Systems & Tooling: Linux (Ubuntu, RHEL), Docker, Conda, Git, Bash