Aug 2024 - May 2025
Custom LLM & RAG Architecture
Built and trained custom Transformer language models in PyTorch, developed a vector search RAG pipeline for verified documents, and presented attention mechanism research to university leadership.
The Goal
To understand language model internals from first principles rather than consuming hosted API endpoints, I implemented autoregressive Transformer models from scratch in PyTorch. The project focused on training compact models on local GPUs and connecting them to a semantic search pipeline (RAG) to ground responses in verified reference manuals.
Transformer Architecture from Scratch
- Self-attention mechanics: Coded the core Transformer architecture in PyTorch, implementing multi-head self-attention, scaled dot-product calculations, and rotary positional encodings.
- Tokenization & data packing: Built Byte-Pair Encoding (BPE) tokenizers and custom token-packing data loaders to maximize GPU training throughput and eliminate padding waste.
- Memory-efficient fine-tuning: Tested LoRA and QLoRA adapters with 4-bit and 8-bit quantization (bitsandbytes) to evaluate parameter-efficient training quality on constrained local GPUs.
Semantic Search & Document Grounding
- Document ingestion: Built an automated document processing pipeline that parses, chunks, and embeds technical manuals and academic papers into dense vector representations.
- Similarity search: Stored embeddings in a vector database and implemented similarity search using cosine distance to pull relevant context passages for user queries.
- Answer verification: Added a semantic reranking step to prioritize high-confidence context snippets, ensuring generated responses cite verified reference passages.
Faculty Presentation & Research
- Attention weight analysis: Analyzed attention weight distributions across network layers to evaluate how self-attention heads learn syntactic and semantic token relationships.
- Formal demonstration: Authored a formal research poster and delivered live technical demonstrations to university faculty, department chairs, and the University President.
Educational AI Enablement & LLM Architecture Research Poster
Presented to LSSU leadership and faculty explaining Transformer architecture and retrieval pipelines
Research Poster View
Technologies Used
- Deep Learning & NLP: PyTorch, Hugging Face Transformers, Hugging Face Datasets, Hugging Face Accelerate, Attention Mechanisms (Multi-Head Self-Attention, Scaled Dot-Product), Positional Encodings
- Tokenization & Data Pipelines: Byte-Pair Encoding (BPE), Token Packing, Custom Data Loaders, Multiprocessing, NumPy
- Parameter-Efficient Training: PEFT, LoRA, QLoRA, 8-Bit Quantization, 4-Bit Quantization, bitsandbytes
- Retrieval & Semantic Search: Vector Databases, Sentence Embeddings, Cosine Similarity Retrieval, Semantic Reranking, Document Chunking & Parsing, RAG
- Languages & Systems: Python, CUDA, C++, Bash, Linux, Conda, Git