Logistic Regression: The Multi-Feature Foundation of Modern Recommender Systems

Hi, today I will talk about a pivotal model that fundamentally changed the direction of recommender systems — Logistic Regression (LR). Unlike Collaborative Filtering, which only leverages user-item interaction data, Logistic Regression introduced the powerful paradigm of multi-feature fusion, integrating user attributes, item properties, and contextual information to generate more comprehensive recommendations. ¹

Logistic Regression can be viewed as a single-layer neural network with a sigmoid activation function.

Logistic Regression also laid the groundwork for deep learning in recommender systems. In this post, we will focus entirely on the Logistic Regression model itself: its recommendation workflow, mathematical formulation, training process, advantages, and limitations.

Logistic Regression: A Paradigm Shift

Unlike traditional collaborative filtering that predicts ratings or similarities solely from interaction data, Logistic Regression formulates recommendation as a supervised binary classification problem over rich feature representations.

By predicting the probability of positive user interactions such as clicks, watches, and purchases, we can rank candidate items by their predicted probabilities. Logistic Regression established the modern supervised-learning paradigm for Click-Through Rate (CTR) prediction.

graph TB
    subgraph "-- Paradigm Shift -->"
        CF["Collaborative Filtering"]
        LR["Logistic Regression"]
    end

    CF_Input["Input:<br/> User × Item<br/>Interaction Matrix"] --> CF
    CF --> CF_Output["Output:<br/> Rating or Similarity Score"]

    LR_Input["Input:<br/> User + Item + Context<br/>Multi-Feature Vector"] --> LR
    LR --> LR_Output["Output:<br/> Probability of Positive Feedback"]

    classDef new fill:#d5e8d4,stroke:#82b366,stroke-width:2px;
    class LR,LR_Input,LR_Output new;

This shift meant recommender systems could now leverage far richer signals beyond just which user interacted with what item.

Recommendation Workflow with Logistic Regression

The recommendation process using Logistic Regression can be decomposed into four key steps:

graph LR
    A["Feature<br/>Engineering"] --> B["Model<br/>Training"]
    B --> C["Inference"]
    C --> D["Recommendation List<br/>(Top-N most likely clicked)"]

    classDef step fill:#dae8fc,stroke:#6c8ebf,stroke-width:2px;
    class A,B,C,D step;

Step 1: Feature Engineering — Convert all available information into a numerical feature vector:

• User features: age, gender, historical CTR, purchase categories, browsing history
• Item features: category, price, brand, description, release date
• Context features: time of day, location, device type, page position, season

Step 2: Model Training — Define the optimization objective (e.g., maximize click probability) and train the model using labeled historical data. The goal is to determine the optimal weight for each feature.

Step 3: Online Inference — At serving time, construct the feature vector for the current (user, item, context) tuple, feed it into the trained model, and output the probability that the user will engage with the item.

Step 4: Generate Recommendation List — Sort all candidate items by their predicted probabilities and return the top-N items.

Mathematical Formulation of Logistic Regression

The inference process of Logistic Regression can be formally broken down into three stages.

Stage 1: Feature Vector as Input

Given $n$ features, the model receives a feature vector:

Stage 2: Weighted Sum of Features

Each feature is assigned a weight that represents its importance. Weights are learned during training:

Where $b$ is the bias term. The weighted sum is:

Logistic Regression as a Single Neuron

Logistic Regression is mathematically equivalent to a single neuron with a sigmoid activation function, which is why it is often regarded as one of the fundamental building blocks of modern neural networks.

graph LR
    subgraph "Logistic Regression Example"
        X1["x₁<br/>age"] --> W1["× w₁"]
        X2["x₂<br/>gender"] --> W2["× w₂"]
        X3["x₃<br/>category"] --> W3["× w₃"]
        B["1<br/>(bias)"] --> WB["× b"]

        W1 --> SUM["Σ<br/>weighted sum"]
        W2 --> SUM
        W3 --> SUM
        WB --> SUM

        SUM --> SIG["σ(z)<br/>Sigmoid"]
        SIG --> Y["ŷ ∈ [0, 1]<br/>Click Probability"]
    end

    classDef bias fill:#dae8fc,stroke:#6c8ebf,stroke-width:2px;
    classDef sigmoid fill:#fff2cc,stroke:#d6b656,stroke-width:2px;
    class B,WB bias;
    class SIG,Y sigmoid;

Notice how the constant 1 (bias term) is treated just like any other feature — it has its own weight $b$ and participates in the weighted sum exactly like $x_1, x_2, x_3$.

Stage 3: Sigmoid Function for Probability Mapping

The weighted sum $z$ can take any real value $(-\infty, +\infty)$, but we need a probability in $[0, 1]$. The Sigmoid function performs this mapping:

Putting it all together, the complete Logistic Regression model is:

Properties of the Sigmoid Function

One elegant property of Sigmoid is its derivative:

This self-referential form simplifies gradient computation during training.

Training Logistic Regression via Gradient Descent

Now we come to the crucial question: how do we find the optimal weights $w$? The most widely used method is Gradient Descent.

Step 1: Define the Probability Model

For a given sample $x$, the probability of being a positive sample ($y=1$, clicked) or negative sample ($y=0$, not clicked) is:

We can combine both cases into a single expression:

Step 2: Maximum Likelihood Estimation

For $m$ independent training samples, the likelihood of observing the entire dataset is the product of individual probabilities:

Products are numerically unstable and difficult to differentiate. We take the logarithm to convert the product into a sum:

Step 3: Cross-Entropy Loss Function

Since Gradient Descent minimizes rather than maximizes, we introduce a negative sign and normalize by the number of samples. This gives us the Cross-Entropy Loss:

NOTE: Why NOT Mean Squared Error (MSE)? $L = \frac{1}{m} \sum (y_i - \hat{y}_i)^2$

Using MSE together with a sigmoid output layer for classification has two critical flaws:

1. Non-convex optimization landscape with multiple local minima
2. Vanishing gradients when wrong: If $y_i=1$ but $\hat{y}_i\approx0$, gradient vanishes and learning stops

Cross-Entropy Loss alleviates these optimization problems.

Step 4: Gradient Derivation

To apply Gradient Descent, we need the gradient of the loss with respect to each weight $w_j$. Through the chain rule:

The final result (derivation omitted here) is that the gradient simplifies to:

Where $x_{ij}$ is the $j$-th feature value of the $i$-th training sample.

And the gradient is the prediction error multiplied by the feature value.

Step 5: Parameter Update Rule

Finally, we update each weight by moving it in the direction opposite to the gradient:

Where $\gamma$ is the learning rate — a hyperparameter that controls the step size of each update.

We iterate until the loss converges or we reach a maximum number of epochs.

Why Logistic Regression Dominated the Industry

Before deep learning, Logistic Regression was the undisputed choice in recommender systems and computational advertising for three compelling reasons.

1. Solid Mathematical Foundation

Logistic Regression is a member of the Generalized Linear Model (GLM) family, where the response variable is assumed to follow a Bernoulli distribution.

A click event is essentially a biased coin flip — it either happens or it doesn’t. The Bernoulli distribution perfectly models this physical reality. In contrast, Linear Regression assumes the conditional response follows a Gaussian distribution, which is fundamentally unsuitable for binary outcomes.

This mathematical alignment gave practitioners confidence that LR was modeling the right phenomenon.

2. Strong Interpretability

Logistic Regression’s simple weighted-sum-plus-sigmoid form is highly intuitive:

• After appropriate feature normalization, larger absolute weights generally indicate stronger influence (assuming other features remain fixed)
• A large positive weight means this feature strongly predicts clicks
• A large negative weight means this feature discourages clicks
• Weights near zero indicate irrelevant features

Interpretability is invaluable in industry:

• Engineers can quickly debug when CTR predictions are off-target
• Business teams receive concrete explanations (“This campaign performed well because it targeted young male users in the US — weight = +2.3”)
• It drastically reduces communication costs between technical and non-technical teams

3. Engineering Efficiency

With internet companies generating terabytes of data daily, training and inference efficiency are critical:

Before GPUs became widely available (pre-2012), these engineering advantages made Logistic Regression the dominant industrial solution.

Limitations of Logistic Regression

Despite its strengths, Logistic Regression has a fundamental limitation that sparked the next generation of recommendation models: it cannot model feature interactions automatically.

Consider this example:

• “Young users” (feature A) has weight = +1.5 → higher CTR
• “Games category” (feature B) has weight = +1.2 → higher CTR

LR independently learns these weights. But what if young users interested in games have an especially high CTR — higher than the sum of individual effects?

Logistic Regression cannot capture this "$1 + 1 > 2$" synergy unless engineers manually introduce the new cross feature (in this case, “Young × Games”) — this manual process is known as feature crossing.

graph LR
    LR["Logistic Regression"] --> Lim["Limitation: No Automatic Feature Crossing"]
    Lim --> Solution["Next Generation Models:<br/>POLY2 → FM → FFM → Deep Learning"]

    classDef limit fill:#f8cecc,stroke:#b85450,stroke-width:2px;
    class Lim,limit limit;

This limitation drove the development of models like POLY2 (brute-force crossing), Factorization Machines (FM) (latent vector crossing), Field-aware Factorization Machines (FFM) (field-aware crossing), and eventually deep learning models with their nearly unlimited feature combination capability. But those are stories for another post.

Summary

Logistic Regression occupies a special place in the history of recommender systems. It represented far more than just another algorithm — it was a paradigm shift from pure collaborative filtering to multi-feature fusion.

Key takeaways:

1. CTR Prediction Paradigm: LR popularized the supervised classification paradigm for CTR prediction, enabling the use of rich side information.

2. Mathematical Elegance: Sigmoid + Cross-Entropy + Gradient Descent forms a complete, well-understood training framework.

3. Industry Advantages: Mathematical soundness, strong interpretability, and engineering efficiency made it the practical choice for over a decade.

4. Legacy: Its limitation in feature crossing directly inspired subsequent innovations — from Factorization Machines through to modern deep learning architectures.

While today’s state-of-the-art recommender systems use deep neural networks, Logistic Regression remains the essential foundation. Understanding LR thoroughly is not just a historical exercise — it provides the conceptual framework for understanding all modern CTR prediction models.