What is the Least Absolute Shrinkage and Selection Operator (LASSO)?

🛠 Tools and Programming
,
1

Overview

Set the coefficients of unimportant image features to zero, excluding them from subsequent analysis.

Detailed Explanation

Now that you understand the “paying chefs” logic, let’s introduce the mathematical formula. In fact, the core idea of LASSO is to add a “backpack full of weight” to an ordinary regression equation.

The standard linear regression (OLS) formula looks like this:

Y = \beta_1X_1 + \beta_2X_2 + \dots + \beta_nX_n + \epsilon

In contrast, LASSO aims to minimize the following total cost function:

Loss = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^{p}|\beta_j|

We can break this formula into two parts for interpretation:

Part 1: \sum_{i=1}^{n}(y_i - \hat{y}_i)^2 — “Accuracy Penalty”

  • Name: Residual Sum of Squares (RSS).
  • Plain-language interpretation: This measures how accurately your model predicts outcomes. The farther the predicted value \hat{y}_i is from the true value y_i, the larger this term becomes—and thus, the heavier the penalty.
  • Goal: Forces the model to assign weights (coefficients \beta) to features in order to improve prediction accuracy.

Part 2: \lambda \sum_{j=1}^{p}|\beta_j| — “Power Penalty” (L1 Regularization Term)

This is the heart of LASSO.

  • |\beta_j|: The absolute value of the “weight” (coefficient) assigned to each feature.
  • \lambda (Lambda): The penalty strength coefficient, analogous to a “tax rate.”
  • Plain-language interpretation: Every time the model wants to use a feature (i.e., assign it a non-zero weight \beta), it must pay a “tax.” The larger the weight, the higher the tax.

Why Does This Formula Force Features to Become Zero?

This is the most crucial point: Why does adding the sum of absolute values (the L1 norm) cause irrelevant features’ coefficients to shrink exactly to zero—while ordinary regression does not?

We’ll explain using the concept of a “budget constraint”:

  1. Diamond-shaped boundary: In mathematical space, the L1 regularization constraint on feature weights defines a diamond-shaped region (in two dimensions).
  2. Hitting vertices: When searching for the optimal balance between prediction accuracy and low tax burden, the optimization path is highly likely to hit one of the diamond’s vertices.
  3. Zeros on coordinate axes: In a coordinate system, the diamond’s vertices lie precisely on the axes. Hitting an axis means the weight \beta of one feature becomes exactly zero.

“Formula-Based Summary” for Absolute Beginners

  • If \lambda = 0: No tax. To maximize accuracy, the model uses all features aggressively—leading to overfitting.
  • If \lambda is very large: Tax burden is extremely high. To save costs, the model sets nearly all feature weights to zero, retaining only the single most useful one—resulting in underfitting.
  • If \lambda is just right: This is what you do in your project. Using cross-validation, you find the perfect “tax rate,” driving irrelevant or redundant features into financial “bankruptcy” (i.e., their weights become zero), leaving only your 54 most predictive features.

Simulation Code

import numpy as np  
import matplotlib.pyplot as plt  
from sklearn.linear_model import Lasso, LassoCV  
from sklearn.model_selection import train_test_split  
from sklearn.preprocessing import StandardScaler  

# 1. Simulate data generation (mimicking radiomics scenario)  
# Assume 100 samples, each with 100 extracted features  
np.random.seed(42)  
n_samples, n_features = 100, 100  
X = np.random.randn(n_samples, n_features)  

# Simulate reality: Among these 100 features, only 3 are truly associated with the outcome  
# Here, 3, -4, and 2 represent the “true contributions” of those top chefs  
true_weights = np.zeros(n_features)  
true_weights[0] = 3  
true_weights[1] = -4  
true_weights[2] = 2  

# Generate labels: y = Xw + noise  
y = np.dot(X, true_weights) + np.random.normal(0, 1, size=n_samples)  

# 2. Data preprocessing  
# LASSO is highly sensitive to feature scales; standardization ensures all features start on equal footing for taxation  
scaler = StandardScaler()  
X_scaled = scaler.fit_transform(X)  

# Split into training and test sets  
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)  

# 3. Use LassoCV to automatically search for the optimal penalty coefficient Lambda (called 'alpha' in scikit-learn)  
# LassoCV employs cross-validation to automatically identify the ideal tax rate  
lasso_model = LassoCV(cv=5, random_state=42).fit(X_train, y_train)  

print(f"Optimal tax rate (Alpha/Lambda): {lasso_model.alpha_:.4f}")  

# 4. Examine feature selection results  
# Retrieve all feature coefficients (betas)  
coefficients = lasso_model.coef_  
# Count how many coefficients are non-zero  
selected_features = np.sum(coefficients != 0)  

print(f"Total original features: {n_features}")  
print(f"Number of effective features after LASSO selection: {selected_features}")  

# 5. Visualization: See which features were “cut”  
plt.figure(figsize=(10, 6))  
plt.stem(np.where(coefficients != 0)[0], coefficients[coefficients != 0], markerfmt='go', label='Selected')  
plt.stem(np.where(coefficients == 0)[0], coefficients[coefficients == 0], markerfmt='rx', label='Eliminated')  
plt.title("LASSO Feature Selection Results")  
plt.xlabel("Feature Index")  
plt.ylabel("Coefficient Value (Weight)")  
plt.legend()  
plt.show()  

# 6. Model evaluation  
train_score = lasso_model.score(X_train, y_train)  
test_score = lasso_model.score(X_test, y_test)  
print(f"Training set R^2 score: {train_score:.4f}")  
print(f"Test set R^2 score: {test_score:.4f}")

References