Publications

2025

1. Arxiv

   

   Mortgage Language Model: Domain-Adaptive Pretraining with Residual Instruction, Alignment Tuning, and Task-Specific Routing

   Manish Jain, Satheesh Kumar Ponnambalam, Salman Faroz, Chandrakanth Lns, and 1 more author

   In Mortgage Language Model: Domain-Adaptive Pretraining with Residual Instruction, Alignment Tuning, and Task-Specific Routing, Nov 2025

   Abs arXiv PDF

   Large Language Models (LLMs) demonstrate exceptional capabilities across general domains, yet their application to specialized sectors such as mortgage finance requires domain-specific knowledge augmentation while preserving instruction-following fidelity. We present MortgageLLM, a novel domain-specific large language model that addresses this dual challenge. It is developed using a dual-track specialization framework from a single base model (LLaMA-3.1-8B). We opted for this dual-expert approach as a single multi-task model suffers from performance trade-offs, where optimizing for structured tasks (via SFT) degrades conversational fidelity (via DPO). Our dual-track method solves this by creating two specialists, allowing each to be optimally trained for its distinct capability. Our approach applies the instruction residual technique to restore instruction-following capabilities post-domain adaptation without supervised fine-tuning. We contribute: (1) application of this residual technique to the highly specialized mortgage finance domain; (2) a dual-expert architecture combining a conversational Q&A model and a structured task model for classification and summarization; and (3) an intelligent task routing mechanism using few-shot classification performed by one of the expert models itself. We validate our approach on domain-specific benchmarks, where our final model (MLM v2) significantly outperforms the base LLaMA-3.1-8B-Instruct, achieving an LLM-as-a-Judge summarization score of 4.58 (vs. 3.99), a Q&A score of 4.09 (vs. 4.0), and a classification score of 2.6 (vs. 1.2). On semantic similarity, our model achieved a BERTScore of 0.77 for summarization (vs. 0.74), 0.68 for Q&A (vs. 0.58), and 0.75 for classification (vs. 0.73), substantially outperforming baseline approaches.

2024

1. Arxiv

   

   DisEmbed: Transforming Disease Understanding through Embeddings

   Salman Faroz

   In DisEmbed: Transforming Disease Understanding through Embeddings, Dec 2024

   Abs arXiv PDF

   The medical domain is vast and diverse, with many existing embedding models focused on general healthcare applications. However, these models often struggle to capture a deep understanding of diseases due to their broad generalization across the entire medical field. To address this gap, I present DisEmbed, a disease-focused embedding model. DisEmbed is trained on a synthetic dataset specifically curated to include disease descriptions, symptoms, and disease-related Q&A pairs, making it uniquely suited for disease-related tasks. For evaluation, I benchmarked DisEmbed against existing medical models using disease-specific datasets and the triplet evaluation method. My results demonstrate that DisEmbed outperforms other models, particularly in identifying disease-related contexts and distinguishing between similar diseases. This makes DisEmbed highly valuable for disease-specific use cases, including retrieval-augmented generation (RAG) tasks, where its performance is particularly robust.

2. Arxiv

   

   Sensitivity Analysis On Loss Landscape

   Salman Faroz

   In this paper. Using gradients for sensitivity analysis, exploring the loss landscape through first, second, and third derivatives, Mar 2024

   Abs arXiv Bib PDF

   Gradients can be employed for sensitivity analysis. Here, we leverage the advantages of the Loss Landscape to comprehend which independent variables impact the dependent variable. We seek to grasp the loss landscape by utilizing first, second, and third derivatives through automatic differentiation. we know that Spearman’s rank correlation coefficient can detect the monotonic relationship between two variables. However, I have found that second-order gradients, with certain configurations and parameters, provide information that can be visualized similarly to Spearman results, In this approach, we incorporate a loss function with an activation function, resulting in a non-linear pattern. Each exploration of the loss landscape through retraining yields new valuable information. Furthermore, the first and third derivatives are also beneficial, as they indicate the extent to which independent variables influence the dependent variable.

   @inproceedings{10.48550/arXiv.2403.01128,
     author = {Faroz, Salman},
     booktitle = {this paper. Using gradients for sensitivity analysis, exploring the loss landscape through first, second, and third derivatives},
     month = mar,
     publisher = {Arxiv},
     title = {Sensitivity Analysis On Loss Landscape},
     year = {2024}
   }

2023

1. IJSREM

   

   Health Care Prediction System Using Deep Learning

   Salman Faroz

   In this research, deep learning-based system is proposed for early detection of diseases like pneumonia and diabetic retinopathy, enabling patients and hospitals to predict disease severity and improve diagnosis accuracy. This system builds on successful existing models, promising advancements in disease diagnosis and patient outcomes., Aug 2023

   Abs Bib PDF

   Annually, approximately 12 million people worldwide suffer due to the misdiagnosis of diseases. To tackle this critical issue, a novel approach for early disease detection is proposed, encompassing conditions such as pneumonia, diabetic retinopathy, and more. Patients can assess disease severity based on the provided medical reports, enabling predictions of mild, moderate, severe, or proliferative stages. Leveraging deep learning models, this system empowers individuals to accurately predict the likelihood of these diseases. The primary targets of this model are patients and hospitals, benefitting from unique image processing techniques that ensure higher accuracy. Building upon the success of existing deep learning models for diabetic retinopathy and pneumonia, the proposed system holds promising potential for advancing disease diagnosis and improving patient outcomes.

   @inproceedings{10.55041/IJSREM17431,
     author = {Faroz, Salman},
     booktitle = {this research, deep learning-based system is proposed for early detection of diseases like pneumonia and diabetic retinopathy, enabling patients and hospitals to predict disease severity and improve diagnosis accuracy. This system builds on successful existing models, promising advancements in disease diagnosis and patient outcomes.},
     doi = {https://www.doi.org/10.55041/IJSREM17431},
     month = aug,
     publisher = {International Journal of Scientific Research in Engineering and Management (IJSREM)},
     title = {Health Care Prediction System Using  Deep Learning},
     year = {2023}
   }