A state-of-the-art spam message classifier built with RoBERTa transformer model, fine-tuned on multiple SMS spam datasets. This model achieves exceptional performance with 0.9782 F1-score for spam detection and 99.41% overall accuracy, making it ideal for real-world deployment in messaging platforms and content moderation systems.
This project develops an intelligent spam detection system using advanced natural language processing techniques. The classifier is designed to accurately distinguish between legitimate messages (ham) and spam content, with a focus on minimizing both false positives and false negatives.
- 🤖 Transformer-based Architecture: Built on RoBERTa-base for superior text understanding
- ⚡ High Performance: 0.9782 F1-score for spam detection, 99.41% overall accuracy
- 🔧 Hyperparameter Optimization: Automated tuning using Optuna framework (25 trials)
- ⚖️ Class Imbalance Handling: Weighted loss function for optimal training
- 🔗 URL Bias Mitigation: Enhanced with real-world ham messages containing links
- 📊 Comprehensive Evaluation: Multiple metrics including precision, recall, and confusion matrix
- 🚀 Production-Ready: Saved in HuggingFace format for easy deployment
- Overall Accuracy: 99.41%
- Weighted F1-Score: 0.9941
- Spam F1-Score: 0.9782
- Spam Precision: 96.55%
- Spam Recall: 99.12%
- Ham Precision: 99.86%
- Ham Recall: 99.45%
✅ Model Accepted: The F1-score for spam class (0.9782) significantly exceeds our predefined acceptance threshold of 0.95, indicating exceptional performance for real-world deployment.
📊 Strong Generalization: All performance metrics are evaluated on a completely unseen test set (15% of data) that was never used during training or hyperparameter tuning, ensuring robust real-world performance and preventing overfitting.
During initial training, the model became overconfident and labeled almost all messages containing <URL> as spam, even if some were legitimate ham. To mitigate this bias, I went through Discord servers and collected additional real ham messages containing links.
This helps the model understand that URLs can appear in non-spam messages and improves generalization for real-world inference, particularly important for Discord bot deployment where legitimate messages often contain links.
- Base Model: FacebookAI/roberta-base
- Task: Binary sequence classification (ham vs spam)
- Fine-tuning: Custom classification head with 2 output labels
- Tokenization: RoBERTa tokenizer with optimal sequence length
- Data Preprocessing: SMS text cleaning and label encoding
- Tokenization: Dynamic padding with maximum sequence length of 128 tokens
- Class Balancing: Weighted loss function to handle imbalanced dataset
- Hyperparameter Optimization: Optuna-based automated tuning
- Evaluation: Comprehensive metrics on held-out test set
Used Optuna framework to optimize (25 trials):
- Dropout rates: Hidden dropout (0.1-0.3), Attention dropout (0.1-0.2)
- Learning rate: 1e-5 to 5e-5 range
- Weight decay: 0.0 to 0.1 regularization
- Batch size: 8, 16, or 32 samples
- Gradient accumulation steps: 1 to 4
- Training epochs: 2 to 5 epochs
- Warmup ratio: 0.05 to 0.1 for learning rate scheduling
Best Parameters Found (Trial 6/25):
- Hidden dropout: 0.10069482002001506
- Attention dropout: 0.12460257350587067
- Learning rate: 4.976184540342024e-05
- Weight decay: 0.04490021845024478
- Batch size: 16
- Gradient accumulation steps: 4
- Epochs: 4
- Warmup ratio: 0.07622459860163384
spam-message-classifier/
├── data/
│ └── sms_spam_uci.csv
│ └── url_dataset.csv
├── notebooks/
│ └── spam-message-classifier.ipynb # Complete development notebook
├── README.md # This file
└── .gitignore # Git ignore rules
Note: The trained model and tokenizer can be found on HuggingFace at roshana1s/spam-message-classifier
Install required packages:
pip install transformers torchLoad and use the trained model from HuggingFace:
from transformers import RobertaTokenizer, RobertaForSequenceClassification
import torch
# Load the trained model and tokenizer from HuggingFace
model = RobertaForSequenceClassification.from_pretrained("roshana1s/spam-message-classifier")
tokenizer = RobertaTokenizer.from_pretrained("roshana1s/spam-message-classifier")Sources:
- SMS Spam Collection Dataset from UCI Machine Learning Repository
- Discord Text Messages — a manually collected dataset of real Discord messages containing both ham and spam samples. (This dataset was created to mitigate
<URL>bias.)
Preprocessing Steps:
- Label encoding (ham → 0, spam → 1)
- Text cleaning and normalization with Discord-specific preprocessing
- Train/validation/test split (70/15/15)
- Tokenization with RoBERTa tokenizer
- Dynamic padding and truncation
- 🤗 Transformers: HuggingFace transformers library
- 🔥 PyTorch: Deep learning framework
- 📊 Scikit-learn: Evaluation metrics and preprocessing
- 🎯 Optuna: Hyperparameter optimization
- 📈 Matplotlib/Seaborn: Data visualization
- 🐼 Pandas: Data manipulation
- Weighted Loss Function: Handles class imbalance effectively
- Label Smoothing: 0.1 to prevent overconfidence
- Custom Metrics: Specialized spam detection metrics
- Confusion Matrix Analysis: Detailed error analysis
- Class-specific Performance: Separate metrics for ham and spam
| Predicted Ham | Predicted Spam | |
|---|---|---|
| Actual Ham | 725 | 4 |
| Actual Spam | 1 | 112 |
- True Positives (Spam correctly identified): 112
- True Negatives (Ham correctly identified): 725
- False Positives (Ham incorrectly flagged): 4
- False Negatives (Spam missed): 1
This spam classifier is ideal for:
- Discord bot moderation (Primary use case)
- SMS filtering systems
- Chat application content filtering
- Social media platforms
- Comment section filtering
- User-generated content screening
The trained model is available on HuggingFace: 👉 roshana1s/spam-message-classifier
This model serves as the core spam detection component for Amy, an intelligent Discord moderation bot that:
- Detects spam messages in real-time
- Provides automated content moderation
- Maintains server quality and user experience
⭐ If you found this project helpful, please consider giving it a star! ⭐