85.regression-models.md

Regression Models

Regression models are basic models in machine learning and statistics. They are used to predict continuous output variables. To put it simply, given a group of input variables and the corresponding output variable, a regression model tries to find the mapping relationship between the input variables and the output variable. The form of a regression model may be simple, but it contains one of the most important modeling ideas in machine learning. Usually, we build regression models mainly for two goals:

1. Describe the relationship between data. We said before that the key of machine learning is to learn how to map from features to target values through historical data. In this process, we do not need to set rules in advance. Instead, the machine learns from historical data by itself. Regression models can help us express the relationship between input and output through a model.
2. Make predictions for unknown data. After learning the mapping relationship, the model can make predictions for new input data.

Regression models are used very widely. Here are some specific examples:

1. Retail industry. Amazon, the world's largest e-commerce platform, can use features such as historical sales, product attributes like price, discount, brand, and category, time features such as season, working day, and holiday, and outside factors such as weather and social media to build regression models and predict the demand of different products in a future period of time. During promotion events, it can also use multiple regression with interaction terms to analyze the influence of promotions on sales.
2. Automobile industry. To optimize battery charging strategy, extend battery life, and give electric-car users more accurate battery warnings, Tesla can use regression models. It can build regression models with the number of battery charge and discharge cycles, environment temperature, depth of discharge, battery physical parameters, and other features to predict the remaining life of batteries.
3. Real estate industry. Zillow, the largest online real-estate platform in the United States, once used regression models to help users estimate house value. It predicted the market price of a house according to house area, house age, location, house type, neighborhood safety level, school rating, and other factors.

Categories of Regression Models

According to the complexity of the model and its assumptions, regression models can be divided into the following types:

1. Linear Regression: it assumes there is a linear relationship between the input variables and the output variable.

Simple linear regression builds a model for the linear relationship between one dependent variable and one independent variable.

$$ y = \beta_0 + \beta_1 x + \varepsilon $$

Here, $\small{y}$ is the target variable, $\small{x}$ is the input variable, $\small{\beta_0}$ is the intercept, which means the predicted value when $\small{x=0}$, $\small{\beta_1}$ is the regression coefficient or slope, which shows how much the input variable affects the output variable, and $\small{\varepsilon}$ is the error term, used to represent random noise or the part that cannot be explained.

Multiple linear regression builds a model for the linear relationship between one dependent variable and several independent variables.

$$ y = \beta_{0} + \beta_{1} x_{1} + \beta_{2} x_{2} + \cdots + \beta_{n} x_{n} + \varepsilon $$

The formula above can also be simplified in vector form:

$$ y = \mathbf{x}^{T} \mathbf{\beta} + \varepsilon $$

Here, $\small{\mathbf{x} = [1, x_{1}, x_{2}, \dots, x_{n}]^{T}}$ is the input vector with intercept, $\small{\mathbf{\beta} = [\beta_{0}, \beta_{1}, \beta_{2}, \dots, \beta_{n}]^{T}}$ is the vector of model parameters, and $\small{\varepsilon}$ is the error term.

2. Polynomial Regression: it introduces higher-order features, so the model can fit more complex nonlinear relationships. It is still an extension of linear models, because the solution for the parameters $\small{\beta}$ is still linear in form. For example, a quadratic relationship can be written as:

$$ y = \beta_{0} + \beta_{1} x + \beta_{2} x^{2} + \varepsilon $$

3. Nonlinear Regression: nonlinear regression completely gives up the linear assumption, and the model form can be any nonlinear function.

4. Ridge Regression, Lasso Regression, and Elastic Net Regression: these add regularization terms on top of linear regression, and are used to deal with overfitting, multicollinearity, and feature selection.

5. Logistic Regression: although the name has "regression", logistic regression is actually a model for classification problems. It uses the Sigmoid function to map the linear combination of input values to the interval $\small{(0, 1)}$, which represents classification probability. It is suitable for binary classification problems, and it can also be extended to Softmax regression for multi-classification problems.

$$ P(y=1 \vert x) = \frac{1}{1 + e^{-(\beta_{0} + \beta_{1} x_{1} + \cdots + \beta_{n} x_{n})}} $$

Calculating Regression Coefficients

The key to building a regression model is finding the best regression coefficients $\small{\mathbf{\beta}}$. The so-called best regression coefficients mean the model parameters that make the model fit the data best, that is, the parameters that minimize the difference between the model prediction $\small{\hat{y_i}}$ and the actual observed value $\small{y_i}$. For this, we first define the following loss function:

$$ L(\mathbf{\beta}) = \sum_{i=1}^{m}(y_{i} - \hat{y_{i}})^{2} $$

Here, $\small{m}$ is the sample size. If we put the regression model into the formula, we get:

$$ L(\mathbf{\beta}) = \sum_{i=1}^{m}(y_{i} - \mathbf{x}_{i}^{T}\mathbf{\beta})^{2} $$

If we write it in matrix form, we get:

$$ L(\mathbf{\beta}) = (\mathbf{y} - \mathbf{X\beta})^{T}(\mathbf{y} - \mathbf{X\beta}) $$

Here, $\mathbf{y}$ is the vector of target values, $\mathbf{X}$ is the feature matrix, and $\small{\mathbf{\beta}}$ is the vector of regression coefficients.

By minimizing the loss function $\small{L(\mathbf{\beta})}$, we can get the analytic solution of the linear regression model. Taking the derivative of $\small{L(\mathbf{\beta})}$ and setting it to 0 gives:

$$ \frac{\partial{L(\mathbf{\beta})}}{\partial{\mathbf{\beta}}} = -2\mathbf{X}^{T}(\mathbf{y} - \mathbf{X\beta}) = 0 $$

After rearranging, we get:

$$ \mathbf{\beta} = (\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbf{y} $$

When the matrix $\small{\mathbf{X}^{T}\mathbf{X}}$ is not full rank, we can add a regularization term to make the matrix invertible, as shown below:

$$ \mathbf{\beta} = (\mathbf{X}^{T}\mathbf{X} + \mathbf{\lambda \mit{I}})^{-1}\mathbf{X}^{T}\mathbf{y} $$

Note: If you do not understand the regularization mentioned here, you can leave it for now. We will talk about it later.

The method above is suitable for small datasets. When the amount of data is not large, the computing efficiency is fine. For large datasets or more complex optimization problems, we can use gradient descent. It updates parameters step by step by iteration to gradually approach the best solution. The goal of gradient descent is also to minimize the loss function. For the loss function $\small{L(\mathbf{\beta})}$ above, the gradient can be written as:

$$ \nabla L(\mathbf{\beta}) = \left[ \frac{\partial{L}}{\partial{\beta_{1}}}, \frac{\partial{L}}{\partial{\beta_{2}}}, \cdots, \frac{\partial{L}}{\partial{\beta_{n}}} \right] $$

Gradient descent updates the parameters $\small{\mathbf{\beta}}$ by the following rule:

$$ \mathbf{\beta}^{\prime} = \mathbf{\beta} - \alpha \nabla L(\mathbf{\beta}) \\\ \mathbf{\beta} = \mathbf{\beta^{\prime}} $$

Here, $\small{\alpha}$ is the learning rate. It is usually a small positive number used to control the update size each time. If the learning rate is chosen well, gradient descent will converge to a local minimum of the target function. If the learning rate is too large, it may shake and fail to converge. If it is too small, convergence will be slow and need more iterations.

New Dataset Introduction

The iris dataset we used before is not suitable for explaining regression models. So here we introduce another classic dataset, the Auto MPG dataset. The Auto MPG dataset was originally provided by the American Automobile Association. We can use this dataset to predict vehicle fuel efficiency, that is, Miles Per Gallon, or MPG. Note that scikit-learn does not have this dataset built in. We can download it from the UCI Machine Learning Repository, or load it online with the code below.

import ssl
import pandas as pd

ssl._create_default_https_context = ssl._create_unverified_context
df = pd.read_csv('https://archive.ics.uci.edu/static/public/9/data.csv')
df.info()

Output:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 398 entries, 0 to 397
Data columns (total 9 columns):
 #   Column        Non-Null Count  Dtype  
---  ------        --------------  -----  
 0   car_name      398 non-null    object 
 1   cylinders     398 non-null    int64  
 2   displacement  398 non-null    float64
 3   horsepower    392 non-null    float64
 4   weight        398 non-null    int64  
 5   acceleration  398 non-null    float64
 6   model_year    398 non-null    int64  
 7   origin        398 non-null    int64  
 8   mpg           398 non-null    float64
dtypes: float64(4), int64(4), object(1)
memory usage: 28.1+ KB

From the output above, we can briefly introduce the 9 fields in the dataset. The first 8 are input variables, with the first one not used for now, and the last one is the output variable.

Field name	Description
car_name	car name, string, not useful for modeling for now
cylinders	number of cylinders, integer
displacement	engine displacement in cubic inches, float
horsepower	horsepower, float, has missing values that need to be handled first
weight	car weight in pounds, integer
acceleration	acceleration, time needed for 0 to 60 mph, float
model_year	model year from 1970 to 1982, here a two-digit year is used
origin	car origin, where 1, 2, and 3 should be treated as three categories, not integers
mpg	vehicle fuel efficiency, the target variable

First, we delete the car_name field, which is not useful for now. Then we use the corr method of DataFrame to check whether there is correlation between the input variables, that is, features, and the output variable, that is, the target value. Through correlation analysis, we can choose features with strong correlation and remove features with weak correlation to the target value. This helps reduce model complexity and the risk of overfitting. In multiple regression, multicollinearity, which means strong correlation between input variables, may affect the estimation of regression coefficients and make the model unstable. We can detect collinearity problems by calculating correlation between features and variance inflation factor, or VIF.

# Delete the specified column.
df.drop(columns=['car_name'], inplace=True)
# Compute the correlation coefficient matrix.
df.corr()

Note: The corr method of DataFrame computes Pearson correlation by default. Pearson correlation is suitable for continuous values from a normal population. For ranked data, we can set the method parameter to spearman or kendall to calculate Spearman rank correlation or Kendall coefficient. Of course, continuous values can also be turned into ranked data by binning first, and then we can judge the correlation.

Before using this dataset for modeling, we need to do some preparation work. First, handle the missing values in the horsepower field, and then turn the origin field into one-hot encoding. One-hot encoding is a common encoding way for handling categorical variables. Usually, categorical data such as gender, color, and season cannot be directly given to a machine learning model for training, because most algorithms can only handle numeric data. One-hot encoding turns each categorical variable into several new binary features, with values 0 or 1, so these variables can be given to the machine learning model.

Suppose we have a feature column called "color", and its possible values are red, green, and blue. We can turn it into three binary features through one-hot encoding, as shown below.

red	green	blue
1	0	0
0	1	0
0	0	1
0	1	0
1	0	0

Note: We can also keep only the green and blue columns. If both columns have value 0, that means our color is red.

One-hot encoding is simple, direct, easy to understand, and easy to implement. It is very effective for unordered categories because it does not introduce any fake order relationship, and the processed data type is numeric, so many machine learning algorithms can handle it well. Of course, if a categorical feature has many different category values, one-hot encoding will create many new features. This may greatly increase the data dimension and affect computing performance and storage efficiency, especially when the data has many sparse categories.

The code below does the preprocessing:

# Delete samples with missing values.
df.dropna(inplace=True)
# Turn the origin field into categorical type.
df['origin'] = df['origin'].astype('category') 
# Convert the origin field to one-hot encoding.
df = pd.get_dummies(df, columns=['origin'], drop_first=True)
df

Output:

     cylinders  displacement  horsepower  weight  ...  model_year   mpg  origin_2  origin_3
0            8         307.0       130.0    3504  ...          70  18.0     False     False
1            8         350.0       165.0    3693  ...          70  15.0     False     False
2            8         318.0       150.0    3436  ...          70  18.0     False     False
3            8         304.0       150.0    3433  ...          70  16.0     False     False
4            8         302.0       140.0    3449  ...          70  17.0     False     False
..         ...           ...         ...     ...  ...         ...   ...       ...       ...
393          4         140.0        86.0    2790  ...          82  27.0     False     False
394          4          97.0        52.0    2130  ...          82  44.0      True     False
395          4         135.0        84.0    2295  ...          82  32.0     False     False
396          4         120.0        79.0    2625  ...          82  28.0     False     False
397          4         119.0        82.0    2720  ...          82  31.0     False     False

[392 rows x 9 columns]

Note: In the code above, the get_dummies function in pandas turns the origin column into one-hot encoding. Because drop_first=True, the original values 1, 2, and 3 become only two columns, origin_2 and origin_3. The OneHotEncoder in the preprocessing module of scikit-learn also supports turning categorical features into one-hot encoding.

Next, we still split the dataset into a training set and a test set.

from sklearn.model_selection import train_test_split

X, y = df.drop(columns='mpg').values, df['mpg'].values
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, random_state=3)

Linear Regression Code Implementation

First, we use LinearRegression in the linear_model module of scikit-learn to create a linear regression model. LinearRegression uses the least-squares method to compute the parameters of the regression model.

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

If we want to see the parameters of the linear regression model, that is, the regression coefficients and the intercept, we can use the following code.

print('Regression coefficients:', model.coef_)
print('Intercept:', model.intercept_)

Output:

Regression coefficients: [-0.70865621  0.03138774 -0.03034065 -0.0064137   0.06224274  0.82866534
  3.20888265  3.68252848]
Intercept: -21.685482718950933

Evaluation of Regression Models

We can evaluate the prediction effect of a regression model by the following metrics:

1. Mean Squared Error (MSE). MSE is one of the most commonly used evaluation metrics for regression models. It is defined as the average of the squared errors between predicted values and true values.

$$ \text{MSE} = \frac{1}{m} \sum_{i=1}^{m}(y_{i} - \hat{y_{i}})^{2} $$

2. Root Mean Squared Error (RMSE). RMSE is the square-root form of MSE. It is used to measure the real scale of the error more directly.

$$ \text{RMSE} = \sqrt{\text{MSE}} = \sqrt{\frac{1}{m} \sum_{i=1}^{m}(y_{i} - \hat{y_{i}})^{2}} $$

3. Mean Absolute Error (MAE). MAE is another commonly used error metric. It is defined as the average of the absolute errors between predicted values and true values.

$$ \text{MAE} = \frac{1}{m} \sum_{i=1}^{m} \lvert y_{i} - \hat{y_{i}} \rvert $$

4. Coefficient of determination (R-Squared, $\small{R^{2}}$). $\small{R^{2}}$ is a relative metric used to measure how well the model fits the data. The closer its value is to 1, the better.

$$ R^{2} = 1 - \frac{SS_{res}}{{SS}_{tot}} $$

Here, $\small{SS_{res}}$ is the residual sum of squares, and $\small{SS_{tot}}$ is the total sum of squares, as shown below.

The sum of the red square areas on the left side of the figure above is the total sum of squares, and the sum of the blue square areas on the right side is the residual sum of squares. Obviously, the better the model fits, the closer the value of residual sum of squares divided by total sum of squares is to 0, and the closer the value of $\small{R^{2}}$ is to 1.

We can use the functions already provided in scikit-learn to calculate MSE, MAE, and $\small{R^{2}}$.

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

mse = mean_squared_error(y_test, y_pred)
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f'Mean squared error: {mse:.4f}')
print(f'Mean absolute error: {mae:.4f}')
print(f'Coefficient of determination: {r2:.4f}')

Output:

Mean squared error: 13.1215
Mean absolute error: 2.8571
Coefficient of determination: 0.7848

Adding Regularization Terms

Ridge regression adds an $\small{L2}$ regularization term on top of linear regression. Its purpose is to prevent overfitting, especially when there are many features or there is collinearity between features. The loss function of ridge regression is:

$$ L(\beta) = \sum_{i=1}^{m}{(y_{i} - \hat{y_{i}})^{2}} + \lambda \cdot \sum_{j=1}^{n}{\beta_{j}^{2}} $$

The $\small{L2}$ regularization term punishes large regression coefficients. It is like shrinking the size of the coefficients, but it will not make the coefficients become 0. We can use the Ridge class in scikit-learn to implement ridge regression.

from sklearn.linear_model import Ridge

model = Ridge()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print('Regression coefficients:', model.coef_)
print('Intercept:', model.intercept_)
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f'Mean squared error: {mse:.4f}')
print(f'Coefficient of determination: {r2:.4f}')

Output:

Regression coefficients: [-0.68868217  0.03023126 -0.0291811  -0.00642523  0.06312298  0.82583962
  3.04105754  3.49988826]
Intercept: -21.390402697674855
Mean squared error: 12.9604
Coefficient of determination: 0.7874

Lasso regression adds an $\small{L1}$ regularization term. It not only prevents overfitting, but also has the function of feature selection, especially for high-dimensional data. The loss function of lasso regression is:

$$ L(\mathbf{\beta}) = \sum_{i=1}^{m}{(y_{i} - \hat{y_i})^{2}} + \lambda \cdot \sum_{j=1}^{n}{\lvert \beta_{j} \rvert} $$

The $\small{L1}$ regularization term will shrink some unimportant regression coefficients to 0, so it can do feature selection. We can use the Lasso class in scikit-learn to implement lasso regression.

from sklearn.linear_model import Lasso

model = Lasso()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print('Regression coefficients:', model.coef_)
print('Intercept:', model.intercept_)
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f'Mean squared error: {mse:.4f}')
print(f'Coefficient of determination: {r2:.4f}')

Output:

Regression coefficients: [-0.00000000e+00  4.46821248e-04 -1.22830326e-02 -6.29725191e-03
  0.00000000e+00  6.91590631e-01  0.00000000e+00  0.00000000e+00]
Intercept: -9.109888229245005
Mean squared error: 11.1035
Coefficient of determination: 0.8179

Note: In the result of the code above, the regression coefficients of four features are set to 0. This is equal to choosing 4 important features from 8 features. From MSE and $\small{R^{2}}$, we can see that this model fits better than the previous regression models.

Elastic Net regression combines the advantages of ridge regression and lasso regression. It introduces both $\small{L1}$ and $\small{L2}$ regularization terms at the same time. It is suitable for high-dimensional data and for cases where features are correlated. Its loss function is:

$$ L(\mathbf{\beta}) = \sum_{i=1}^{m}{(y_{i} - \hat{y_i})^{2}} + \alpha \cdot \lambda \sum_{j=1}^{n}{\lvert \beta_{j} \rvert} + (1 - \alpha) \cdot \lambda \cdot \sum_{j=1}^{n}{\beta_{j}^{2}} $$

Here, $\small{\alpha}$ controls the weight ratio between $\small{L1}$ and $\small{L2}$ regularization.

Another Linear Regression Implementation

We mentioned above that besides the least-squares method, we can also use gradient descent to solve the parameters of regression models. SGDRegressor in the linear_model module of scikit-learn uses this method. SGD is short for Stochastic Gradient Descent. In each iteration, stochastic gradient descent uses only one random sample to compute the gradient. It is fast and suitable for large-scale datasets. Note that it is very sensitive to the learning rate, and it is also sensitive to feature scale, so usually we need to standardize the features before training. The full code is shown below.

from sklearn.linear_model import SGDRegressor
from sklearn.preprocessing import StandardScaler

# Select features and standardize them.
scaler = StandardScaler()
scaled_X = scaler.fit_transform(X[:, [1, 2, 3, 5]])
# Split the training set and test set again.
X_train, X_test, y_train, y_test = train_test_split(scaled_X, y, train_size=0.8, random_state=3)

# Create, train, and predict.
model = SGDRegressor()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print('Regression coefficients:', model.coef_)
print('Intercept:', model.intercept_)

# Model evaluation.
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f'Mean squared error: {mse:.4f}')
print(f'Coefficient of determination: {r2:.4f}')

Output:

Regression coefficients: [-0.25027084 -0.41349219 -4.9559786   2.83009217]
Intercept: [23.48707219]
Mean squared error: 11.3853
Coefficient of determination: 0.8133

Here, we also need to emphasize some important parameters of the SGDRegressor constructor. These are also important hyperparameters of regression models:

1. loss: specifies the optimization target, that is, the loss function.
2. penalty: specifies the regularization method used to prevent overfitting.
3. alpha: the coefficient of regularization strength.
4. l1_ratio: when penalty='elasticnet', it controls the weight between L1 and L2 regularization.
5. tol: the tolerance of the optimization algorithm, that is, the threshold for judging convergence.
6. learning_rate: specifies the adjustment strategy of the learning rate.
7. eta0: the initial learning rate.
8. power_t: when learning_rate='invscaling', it controls the decay speed of the learning rate.
9. early_stopping: whether to enable early stopping.
10. validation_fraction: the proportion of training data used as validation set.
11. max_iter: the maximum number of training iterations.
12. shuffle: whether to shuffle training data at the beginning of each epoch.
13. warm_start: whether to continue training with the parameters from the last training.
14. verbose: controls the log output during training.

Polynomial Regression

Sometimes, the relationship between the independent variables and the dependent variable is not a simple linear relationship. Also, if there are clear turning points in the data, or we want to use a simple formula to approximate some complex phenomenon, a linear regression model may not be enough. At this time, we need to build a polynomial regression model.

Below we use a simple example to explain polynomial regression. First generate a group of data points and draw the scatter plot.

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 6, 150)
y = x ** 2 - 4 * x + 3 + np.random.normal(1, 1, 150)
plt.scatter(x, y)
plt.show()

Output:

Obviously, it is hard to fit such a group of data points with a linear model. The code below can prove that.

x_ = x.reshape(-1, 1)

model = LinearRegression()
model.fit(x_, y)
a, b = model.coef_[0], model.intercept_
y_pred = a * x + b
plt.scatter(x, y)
plt.plot(x, y_pred, color='r')
plt.show()

Output:

Obviously, this is an underfitting result. Let us look at the value of $\small{R^{2}}$.

r2 = r2_score(y, y_pred)
print(f'Coefficient of determination: {r2:.4f}')

Output:

Coefficient of determination: 0.5933

The PolynomialFeatures class in the preprocessing module of scikit-learn can expand original features into polynomial features, and in this way turn a linear model into a model with higher-order terms. Below we show how to use PolynomialFeatures for feature preprocessing to implement polynomial regression.

from sklearn.preprocessing import PolynomialFeatures

poly = PolynomialFeatures(degree=2)
x_ = poly.fit_transform(x_)

model = LinearRegression()
model.fit(x_, y)
y_pred = model.predict(x_)
r2 = r2_score(y, y_pred)
print(f'Coefficient of determination: {r2:.4f}')

Output:

Coefficient of determination: 0.9497

After introducing higher-order terms through feature preprocessing, the fitting effect of the model is clearly improved. But note that after adding higher-order terms, the risk of overfitting also becomes much larger, especially when the amount of data is small.

Logistic Regression

Although logistic regression has "regression" in its name, it is actually a classification algorithm used for binary classification problems, such as whether an email is spam, whether a user will click an ad, or whether a credit-card customer has default risk. The core idea of logistic regression is to use the Sigmoid function to map the output of linear regression into the interval $\small{(0, 1)}$ as the predicted classification probability. The curve of the Sigmoid function is shown below.

Below, we use the make_classification function in the datasets module of scikit-learn to generate a group of simulated data, and then use logistic regression to build a classification prediction model.

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

# Generate 1000 sample records, each with 6 features.
X, y = make_classification(n_samples=1000, n_features=6, random_state=3)
# Split the 1000 samples into training set and test set.
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, random_state=3)

# Create and train logistic regression model.
model = LogisticRegression()
model.fit(X_train, y_train)

# Predict and evaluate on the test set.
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred))

Output:

              precision    recall  f1-score   support

           0       0.95      0.93      0.94       104
           1       0.93      0.95      0.94        96

    accuracy                           0.94       200
   macro avg       0.94      0.94      0.94       200
weighted avg       0.94      0.94      0.94       200

Here, let us again emphasize some important parameters of the LogisticRegression constructor. These are also important hyperparameters of logistic regression models:

1. penalty: specifies the regularization type.
2. C: the inverse of regularization strength.
3. solver: specifies the optimization algorithm.
4. multi_class: specifies how to handle multi-class problems.
5. fit_intercept: whether to calculate the intercept.
6. class_weight: class weights, used to handle imbalanced classes.

Note: Some hyperparameters of logistic regression are similar to the SGDRegressor we talked about before, so we will not repeat them here.

Summary

Regression models are a statistical analysis method used to build the relationship between independent variables and dependent variables. By fitting data, they can predict the value of target variables. They are widely used in economics, engineering, medicine, and many other fields, and can help decision-makers do data-driven prediction and analysis.