Author: Grace Esteban MA Ed. Email: mesteban@ccsf.edu Date: January 2025
This study applies machine learning classification techniques to investigate whether student support service utilization can predict course success rates at California community colleges. Utilizing data from the California Community Colleges Chancellor's Office DataMart, we analyzed 75 term-level observations from the San Francisco Community College District (SFCCD) and Allan Hancock College, spanning Fall 2014 through Fall 2024.
These institutions were selected specifically for the completeness of their longitudinal data; attempts to include additional districts such as Solano, West Valley, Foothill, and Cabrillo—were precluded by significant reporting gaps. Consequently, the study focuses on predictive modeling rather than a direct comparison between the two colleges. The primary objective was to construct interpretable models capable of classifying academic terms as 'high success' or 'low success' based on the utilization of five key support services: counseling, education planning, assessment, orientation, and academic probation services.
-
Counseling is the strongest predictor of term-level success rates across all three models tested, suggesting a meaningful relationship between advising access and student outcomes.
-
Assessment and orientation services also demonstrated significant predictive value, indicating that early intervention services contribute to success indicators.
-
Institutional differences matter less than service patterns—the college variable contributed minimally to predictions, suggesting these findings may generalize across similar institutions.
-
The Decision Tree model achieved the best performance with 78.95% accuracy and an F1 score of 0.80, providing interpretable decision rules for practitioners.
| Model | Accuracy | Precision | Recall | F1 Score |
|---|---|---|---|---|
| Logistic Regression | 68.42% | 70.00% | 70.00% | 0.70 |
| Decision Tree | 78.95% | 80.00% | 80.00% | 0.80 |
| Random Forest | 73.68% | 72.73% | 80.00% | 0.76 |
The Decision Tree classifier was selected as the best model based on its highest F1 score and balanced precision/recall. The model correctly classifies approximately 4 out of 5 academic terms into the appropriate success category.
- Lagged analysis: Examine whether services in one term predict success in subsequent terms
- Student-level modeling: Analyze individual student outcomes rather than aggregate data
- Expanded sample: Include additional colleges to improve generalizability
- Causal inference: Apply propensity score matching to address selection bias
- For Institutional Researchers: Use these models as a starting point for local analysis with institution-specific data
- For Administrators: Consider the strong relationship between counseling utilization and success when allocating resources
- For Policy Makers: Investigate whether increasing access to high-impact services improves outcomes
California Community Colleges serve about 2.2 million students annually across 116 colleges, representing the largest system of higher education in the United States. Despite significant investments in student support services mandated through initiatives like the Student Success and Support Program (SSSP) the relationship between service utilization and student outcomes remains difficult to quantify at scale.
Understanding which services most strongly correlate with success can help institutions:
- Allocate limited resources more effectively
- Identify at-risk terms or populations proactively
- Design targeted interventions based on evidence
- Demonstrate accountability to stakeholders and funders
This project provides a data-driven framework for examining these relationships using publicly available data and reproducible machine learning methods.
Does greater participation in student support services at California Community Colleges correspond to higher course success rates?
Specifically, we investigate:
- Can term-level service counts predict whether a term will have above-median or below-median success rates?
- Which services are most predictive of success outcomes?
- Do these patterns hold across different institutions?
All data was obtained from the California Community Colleges Chancellor's Office DataMart, a public repository of institutional data.
| Dataset | Description | Records |
|---|---|---|
| Student Success Services Summary | Counts of students receiving each support service by term | SFCCD: 32 terms, Allan Hancock: 43 terms |
| Course Retention and Success Summary | Course success rates by term and course category | Credit course success rates extracted |
| Feature | Description |
|---|---|
counseling |
Count of students receiving counseling/advisement services |
education_plan |
Count of students with education plan services |
assessment |
Count of students receiving initial assessment/placement |
orientation |
Count of students completing orientation |
probation |
Count of students receiving academic probation services |
college_encoded |
Binary indicator (SFCCD=0, Allan Hancock=1) |
- Credit Course Success Rate: Percentage of credit course enrollments resulting in a grade of A, B, C, or Pass
- Binary Classification: Terms with success rate >= 78.56% (median) classified as "High Success" (1), otherwise "Low Success" (0)
- 75 observations (term-college combinations)
- 6 features + 1 target variable
- Class distribution: 38 High Success, 37 Low Success (ratio: 0.97)
- Time span: Fall 2014 – Fall 2024
- Data Extraction: Parsed multi-row header structure from DataMart CSV exports
- Service Aggregation: Combined "Directed" and "Exempted" student counts for each service
- Feature Selection: Excluded "Other Services" due to inconsistent reporting
- Merging: Inner join of services and success data on term and college
- Target Creation: Binary classification based on median success rate threshold
- Train/Test Split: 75/25 stratified split (56 training, 19 testing)
- Hyperparameter Tuning: 5-fold cross-validation with GridSearchCV
- Feature Scaling: StandardScaler applied for Logistic Regression
| Model | Purpose | Key Hyperparameters |
|---|---|---|
| Logistic Regression | Linear baseline | C=0.1, L2 penalty |
| Decision Tree | Interpretable rules | max_depth=4, gini criterion |
| Random Forest | Ensemble robustness | n_estimators=200, max_depth=3 |
- Accuracy: Overall correct classification rate
- Precision: Proportion of predicted high-success terms that were actually high-success
- Recall: Proportion of actual high-success terms correctly identified
- F1 Score: Harmonic mean of precision and recall (primary selection metric)
| Model | Best CV Accuracy |
|---|---|
| Logistic Regression | 76.36% |
| Decision Tree | 75.15% |
| Random Forest | 80.00% |
| Model | Accuracy | Precision | Recall | F1 Score |
|---|---|---|---|---|
| Logistic Regression | 0.6842 | 0.7000 | 0.7000 | 0.7000 |
| Decision Tree | 0.7895 | 0.8000 | 0.8000 | 0.8000 |
| Random Forest | 0.7368 | 0.7273 | 0.8000 | 0.7619 |
| Feature | Random Forest Importance | Decision Tree Importance |
|---|---|---|
| counseling | 0.324 | 0.784 |
| assessment | 0.244 | 0.124 |
| orientation | 0.239 | 0.092 |
| probation | 0.101 | 0.000 |
| education_plan | 0.078 | 0.000 |
| college_encoded | 0.014 | 0.000 |
Decision Tree was selected based on:
- Highest F1 Score (0.80) indicating balanced precision and recall
- Interpretable decision rules suitable for practitioner use
- Strong test accuracy (78.95%)
- Small sample size (75 observations) limits generalizability
- Same-term analysis cannot establish causality
- Aggregated data masks individual-level variation
- Two colleges only may not represent all CA community colleges
- COVID-era grading policies (2020-2021) elevated success rates artificially
final_capstone_project/
├── README.md # Project documentation (this file)
├── requirements.txt # Python dependencies
│
├── raw_data/ # Original data files
│ ├── SFCCD_StudentSuccessServiceSumm.csv
│ ├── SFCCD_CourseRetSuccessSumm.csv
│ ├── Allan_Hancock_StudentSuccessServiceSumm.csv
│ └── Allan_Hancock_CourseRetSuccessSumm.csv
│
├── data/
│ └── processed/
│ └── master_dataset.csv # Clean, merged dataset
│
├── notebooks/
│ └── final_capstone.ipynb # Main analysis notebook
│
└── images/
├── confusion_matrices.png # Model confusion matrices
├── feature_importance.png # Feature importance charts
└── eda_visualizations.png # Exploratory data analysis plots
| File | Description |
|---|---|
| final_capstone.ipynb | Complete analysis notebook with code and results |
| master_dataset.csv | Cleaned and merged dataset ready for analysis |
| requirements.txt | Python package dependencies |
- Python 3.9+
- Jupyter Notebook or JupyterLab
# Clone repository
git clone https://github.com/yourusername/final_capstone_project.git
cd final_capstone_project
# Create virtual environment (optional but recommended)
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Launch Jupyter
jupyter notebook notebooks/final_capstone.ipynb- Open
notebooks/final_capstone.ipynbin Jupyter - Run all cells sequentially (Cell > Run All)
- Results and visualizations will be generated automatically
- Processed data saved to
data/processed/master_dataset.csv - Images saved to
images/directory
Grace Esteban MA Ed. City College of San Francisco
- Email: mesteban@ccsf.edu
- LinkedIn: linkedin.com/in/graceesteban
- California Community Colleges Chancellor's Office for providing public data access
- UC Berkeley Extension Professional Program in Machine Learning and Artificial Intelligence
This project is for educational purposes. Data is publicly available from the California Community Colleges Chancellor's Office DataMart.