

# HER2 Expression Prediction with Flexible Multi-Modal Inputs via Dynamic Bidirectional Reconstruction

## Abstract

In the **HER2 assessment** for breast cancer, most deep learning models to date are limited to either H&E or IHC pathology images alone, whereas optimal clinical evaluation often requires both. However, obtaining perfectly paired H&E and IHC whole-slide images for the same patient is costly and logistically challenging. We propose a **flexible dual-modality input framework** that, for the first time, seamlessly supports both **single-modality** and **dual-modality** inputs for HER2 status prediction. Our model features a dynamic branch selection mechanism to eliminate the rigid requirement of complete modalities: it can operate with only H&E or only IHC images by invoking a cross-modal image reconstruction pipeline, yet will leverage both modalities jointly when available. Specifically, we design a **Missing Modality Branch Selector** to detect the input configuration and activate either a single-modality **bidirectional reconstruction** plus classification cascade or a dual-modality end-to-end inference path. We develop a **Bidirectional Cross-Modal GAN (CM-GAN)** to generate the missing modality image in feature space with contextual fidelity, thereby recovering critical information and alleviating performance degradation from incomplete inputs. We further introduce a **shared-specific dual-encoder** architecture to disentangle modality-common and modality-specific features, and a **channel attention-based fusion** module to adaptively weight these features based on input quality. On a HER2 pathology dataset of 4,870 paired H&E/IHC images, our framework achieves **95.1% accuracy** with both modalities and retains **~94%** accuracy with only one modality, significantly outperforming conventional single-modality models. Notably, using an H&E image alone, our method attains near dual-modality performance (94% vs. 95%), effectively reducing dependence on IHC staining and expanding HER2 testing accessibility to resource-limited settings. This **“complete-if-available, robust-if-missing”** approach offers a low-cost, adaptable solution for HER2 evaluation and demonstrates a novel dynamic architecture that can be generalized to other multi-modal medical learning tasks.

*Keywords:* Optional dual-modality input; Multi-modal fusion; Dynamic feature reconstruction; HER2 prediction; Histopathology.

## Introduction

Breast cancer is the most common malignancy among women worldwide, making accurate molecular subtype evaluation critical for guiding personalized treatment【1】. **Human epidermal growth factor receptor 2 (HER2)** is a key biomarker in breast cancer; its expression level directly influences the selection of targeted therapies【2】【3】. Precise assessment of HER2 status is thus pivotal. Traditional clinical HER2 testing relies on **immunohistochemistry (IHC)** and in-situ hybridization (ISH) techniques. While these methods are widely used, they have notable limitations. IHC interpretation is subject to pathologist expertise and can suffer from inter-observer and inter-laboratory variability【5】, and even newly introduced scoring categories (e.g. the “HER2-low” subtype) underscore the challenge of consistent interpretation of borderline cases【4】【13】. ISH (including FISH/CISH) provides gene amplification information but involves complex protocols and expensive reagents, limiting its use in resource-constrained settings【6】. Furthermore, the standard IHC/ISH workflow can take several days to return results, delaying clinical decision-making. These shortcomings have spurred interest in **automated, AI-driven HER2 evaluation** to improve objectivity and speed【7】【8】.

Early efforts toward automation included computer-assisted image analysis for HER2 IHC scoring【7】. More recently, deep learning techniques have demonstrated great potential in pathology. For instance, fully digital HER2 scoring systems have been developed to improve consistency in IHC interpretation【8】. However, **existing AI models largely handle each modality in isolation** and thus inherit the limitations of single-modality data. H&E-stained histology slides reveal tissue morphology but lack direct protein expression information, whereas IHC highlights HER2 protein distribution but can be confounded by staining variability. Several studies have attempted to predict HER2 status from a single modality: some leveraged only H&E images【11】 (to predict HER2 positivity or treatment response), and others used only IHC images with deep learning-based scoring【10】. While these approaches reported encouraging results, they often suffer from **insufficient accuracy and robustness** due to the inherent information gap of using one modality alone【12】. In principle, **multi-modal fusion of H&E and IHC** should offer complementary information – capturing both morphological context and molecular expression – and thus could overcome single-modality bottlenecks.

Indeed, combining complementary data modalities is increasingly recognized as a key to improving predictive performance in oncology【17】. For example, multi-modal AI systems have achieved notable success in other breast cancer applications: McKinney *et al.*【18】 developed a deep learning system that surpassed radiologist accuracy in screening mammography by integrating multiple views and clinical context, and Joo *et al.*【19】 fused MRI scans with clinical data to accurately predict pathological complete response to neoadjuvant chemotherapy. However, **most multi-modal frameworks assume that all modalities are always available**, which is often not the case in real clinical workflows where one modality may be missing. A common real-world scenario in pathology is that an H&E slide is readily available, but the corresponding IHC slide might not be obtained due to time, cost, or tissue constraints. Current methods have no mechanism to gracefully handle such missing-modality situations: a dual-modality model would simply fail or resort to naive imputation (e.g. zero-filling or mean values) that can distort feature distributions and degrade performance. Moreover, existing fusion strategies typically use static or fixed weighting of modalities, which cannot adapt to variations in data quality – for instance, an IHC image may be faint or overstained, in which case the model should rely more on the H&E information, but static fusion cannot make such adjustments.

**In this work, we propose a novel HER2 prediction model that is both multi-modal and modality-flexible.** To our knowledge, this is the first framework that **dynamically accommodates either single or dual modality inputs** within one unified model. Our approach leverages the strengths of multi-modal learning while addressing the practical challenges of missing modality and variable data quality. In summary, our contributions are as follows:

- **Flexible Dual-Modality Input Architecture:** We design an adaptive inference pipeline that supports **“H&E-only”, “IHC-only”, or “H&E+IHC”** inputs. A **Missing Modality Branch Selector** automatically detects the presence or absence of each modality and routes the model to the appropriate processing branch. This mechanism eliminates the rigid requirement of existing models for complete input modalities, enabling robust prediction even when only one stain is available.
- **Dynamic Bidirectional Cross-Modal Reconstruction:** We introduce a **Cross-Modal GAN (CM-GAN)** that learns to generate the missing modality image from the available one (i.e., synthesize an IHC image from H&E, or vice versa) in a **bidirectional** manner. The synthetic image, produced with contextual fidelity, is used alongside the real image for downstream classification. This reconstruction step significantly alleviates information loss in single-modality scenarios – for example, providing pseudo-IHC content when only an H&E image is given – thereby boosting the predictive accuracy close to that of true dual-modality input.
- **Shared-Specific Feature Dual-Encoder and Adaptive Fusion:** We devise a **dual-path feature extractor** that separates **modality-shared features** (capturing common tissue patterns) and **modality-specific features** (preserving stain-specific details) using two dedicated encoders. We further incorporate a **Channel Attention Module** to perform **dynamic feature fusion** of the shared and specific representations. This attention-driven fusion assigns adaptive weights to H&E-derived and IHC-derived features based on their relative reliability, effectively implementing an intelligent modality weighting policy (“prefer both modalities when available; if one modality is unreliable or missing, rely more on the other”).
- **High Accuracy and Clinical Applicability:** Our framework achieves **state-of-the-art performance** on a large HER2 pathology dataset. It reaches 95.1% accuracy with both H&E and IHC inputs. Remarkably, even with only an H&E image, it attains ~94% accuracy, vastly outperforming conventional single-modality models (which achieve around 71–77%) and approaching dual-modality performance. This flexible design provides a **low-cost solution for laboratories lacking IHC capabilities** and improves access to accurate HER2 testing. The proposed dynamic architecture is generic and could be **transferred to other medical multi-modal tasks** where modality availability is inconsistent, promoting a new paradigm of *elastic* multi-modal learning in healthcare AI.

## Related Work

**Automated HER2 Scoring from Single Modality:**  Given the limitations of subjective manual scoring, researchers have explored automated HER2 evaluation on individual data modalities. Early work by Masmoudi *et al.*【7】 developed an image analysis algorithm for objective quantification of IHC HER2 expression, representing one of the first attempts at computer-aided HER2 scoring. In the deep learning era, most efforts have focused on a single image modality at a time. Several studies target **IHC slide analysis**: Che *et al.*【10】 trained a convolutional neural network to recognize HER2-positive vs. negative tumor cells on IHC whole-slide images, achieving high concordance with human pathologists. Xiong *et al.*【8】 proposed a comprehensive AI system for HER2 scoring on IHC slides, which improved scoring consistency across slides and laboratories. Similarly, Chauhan *et al.*【9】 introduced a deep learning approach that **contrasts multi-resolution features** from IHC slides to refine HER2 scoring, demonstrating that combining high- and low-magnification features can enhance classification of 0/1+ vs. 2+ vs. 3+ cases. These IHC-focused methods benefit from the specificity of the stain but do not utilize the morphological context available in H&E.

On the other hand, some works have attempted to **predict HER2 status from H&E images alone**, aiming to infer molecular expression from morphology. Farahmand *et al.*【11】 trained a deep model on H&E tumor regions to predict HER2 positivity and response to anti-HER2 therapy; notably, their H&E-based model could identify some HER2-positive tumors (with an AUC of ~0.88) but was inevitably limited by the absence of direct protein information. Rasmussen *et al.*【12】 focused on the challenging scenario of **IHC-equivocal (2+) cases**: they developed a model using H&E features to predict final HER2 status for cases that are 2+ by IHC, as a decision support tool. While H&E-driven models are attractive since H&E slides are ubiquitously available, their performance tends to lag behind IHC-based methods due to the intrinsic lack of HER2 expression signals in H&E. In summary, single-modality approaches – whether H&E or IHC – face a performance ceiling because each provides only a partial view of the underlying biology【12】. This motivates combining modalities to leverage complementary information.

**Multi-Modal Learning and HER2 Assessment:**  Multi-modal data fusion has shown promise in various medical imaging tasks, as integrating heterogeneous data can provide a more complete characterization of disease【17】. In the context of breast cancer, beyond pathology, there are examples like McKinney *et al.*【18】 who combined multiple image views and clinical insights for improved cancer detection, and Joo *et al.*【19】 who fused imaging with clinical variables for outcome prediction. However, **within pathological HER2 assessment, multi-modal learning is still nascent**. One notable work bridging H&E and IHC is by Liu *et al.*【14】, who introduced the **BCI (Breast Cancer Immunohistochemical) dataset** containing thousands of paired H&E and HER2 IHC image patches. They proposed a pyramid Pix2Pix generative model to translate H&E images to IHC images, demonstrating the feasibility of cross-stain image synthesis for HER2. While their focus was on image generation (augmenting or predicting IHC appearance from H&E) rather than direct HER2 scoring, it established a foundation for cross-modal approaches. To our knowledge, **no prior published method performs joint H&E+IHC fusion for HER2 classification** in an end-to-end manner. Traditional pathology workflows simply interpret the two slides side by side, and existing AI models have yet to fully emulate or improve upon this multi-modal interpretation.

**Missing Modality and Dynamic Fusion Strategies:**  A practical obstacle to multi-modal pathology is that one of the modalities may be missing or unavailable. Simplistic solutions like imputing missing inputs with zero or average values are commonly used but often degrade performance by introducing unnatural features. In general machine learning literature, handling missing modalities has been addressed by learning robust representations. Wang *et al.*【15】, for example, proposed a **shared-specific feature modeling** approach: their framework learns latent features that are common to all modalities as well as features specific to each modality, enabling the model to make predictions even when one modality is absent. This approach, applied to vision tasks in their work, inspires our use of separate shared and specific feature extractors. In the medical imaging domain, however, approaches like【15】 have not been applied to pathology, and challenges like variable stain quality remain. Another line of research involves using generative models to **impute missing modality data**. Our use of a Pix2Pix-based GAN aligns with this idea: by generating a pseudo-IHC image from H&E (or vice versa), we effectively fill in the missing information with a data-driven guess rather than a constant fill-in. This is akin to data augmentation of modality, and is more sophisticated than linear interpolation methods attempted in some contexts (which fail to capture complex non-linear relationships across modalities).

Equally important in multi-modal fusion is how to combine information from each modality. Many existing multi-modal models rely on simple concatenation or fixed fusion rules, which treat each modality’s contribution as static. This is suboptimal when modality relevance varies from case to case. **Attention mechanisms** have emerged as a powerful way to perform adaptive fusion. The Convolutional Block Attention Module (CBAM) by Woo *et al.*【16】 is a representative technique that applies channel-wise and spatial attention to recalibrate feature importance within a CNN. CBAM and similar attention modules can be applied to multi-modal features to dynamically adjust their weights. In our problem, we expect that if an IHC image is faint or noisy, the model should down-weight IHC-specific features and rely more on H&E features, and vice versa. Inspired by CBAM【16】, we implement a channel attention-based fusion that learns to emphasize more informative features and suppress less reliable ones in a **data-driven manner**. To the best of our knowledge, **our work is the first to integrate a dynamic attention fusion with a missing-modality-tolerant architecture in the medical imaging field**. By uniting these components – cross-modal generation, shared-specific encoders, and attention-based fusion – our framework addresses the key shortcomings identified in prior work: it fully exploits multi-modal data when present, and remains robust when one modality is absent or low-quality.

## Method

**Overview:** Our proposed framework (Figure 1) consists of distinct modules that operate in sequence to handle varying input conditions. First, a **Missing Modality Branch Selector** (Section 3.1) inspects the input and determines which processing branch to activate. If both an H&E image and an IHC image are provided, the model directly proceeds with a dual-modality inference path. If only one modality is provided, the model triggers a reconstruction of the missing modality using a generative model before classification. In either case, the input(s) then pass through a **dual-encoder feature extraction** stage (Section 3.3) that produces shared and specific feature representations. Next, these features are combined by a **Channel Attention-based Fusion** module (Section 3.4), yielding a unified feature representation. Finally, a **Classification Module** (Section 3.5) predicts the HER2 expression level. Throughout the network, we incorporate loss functions that enforce effective feature learning, including classification loss and auxiliary domain/adversarial losses for feature alignment. We detail each component below.

### 3.1 Missing Modality Branch Selection

To enable **adaptive input handling**, we introduce a lightweight modality detector that directs the model to the appropriate branch. This is implemented as a two-stage classifier inspired by GoogLeNet. In the first stage, a shallow CNN with a global average pooling head examines the input and determines whether the input is **single-modality (H&E \*or\* IHC)** or **dual-modality (H&E + IHC)**. If a single image is detected, a second-stage classifier (another small CNN) further distinguishes whether the available image is an H&E or an IHC. This two-step detection (dual vs. single, then H&E vs. IHC) guides a **branch selector**:

- **Dual-modality branch:** If both H&E and IHC are present, the model directly fuses and processes them in a joint pipeline (bypassing the reconstruction module).
- **Single-modality branch:** If only H&E (or only IHC) is present, the model invokes a reconstruction of the missing IHC (or H&E) image before proceeding.

This design ensures the model dynamically configures its workflow based on input availability. The branch selection networks are kept small to add negligible overhead, and are trained on an auxiliary task of modality classification. In doing so, the system **adapts at runtime**: for a resource-rich scenario, it uses the full dual-modal power; for a resource-limited scenario (missing one stain), it gracefully degrades to a reconstruction-enhanced single-modal pipeline. This approach contrasts with prior models that would either fail or require a completely separate model in such cases.

### 3.2 Bidirectional Cross-Modal Reconstruction (CM-GAN)

For single-modality inputs, we perform **on-the-fly generation of the complementary modality** using a conditional generative adversarial network. We term this module **CM-GAN**, which is essentially two Pix2Pix-style GANs trained in a bidirectional manner【14】. One GAN learns to generate an IHC image given an H&E image (denoted *HE → IHC GAN*), and the other learns to generate an H&E image given an IHC (denoted *IHC → HE GAN*). Both GANs share a similar architecture and training procedure, differing only in the training data pairing.

We base our CM-GAN on a **pyramid Pix2Pix** architecture【14】, which has shown effectiveness in histopathology image translation. As illustrated in Figure 2, the generator network follows an encoder–decoder structure with *multi-scale residual blocks* and skip connections. For example, in the *HE → IHC* direction, the H&E image is encoded through several downsampling layers to capture high-level tissue structure. Residual blocks at multiple scales learn to translate features from H&E coloration to IHC coloration, guided by a patch-based discriminator that enforces realistic IHC texture. We integrate a **spatial attention mechanism** in the generator to help it focus on tumor regions that are most relevant to HER2 expression (e.g., cell membranes in IHC). The decoder then upsamples and combines low-level details (passed via U-Net skip connections) with high-level context to output a synthetic IHC image. A similar process (with roles swapped) occurs in the *IHC → HE* generator, where the goal is to reconstruct H&E morphology from an IHC input.

During training of CM-GAN, we use paired H&E and IHC patches from the same tumor region. The objective includes an adversarial loss to encourage outputs indistinguishable from real images, and an L1 reconstruction loss between the generated image and the ground truth image in the target modality. By optimizing these, the CM-GAN learns a **context-aware translation**: it preserves biologically meaningful structures (e.g., glandular architecture) while converting color and texture to mimic the target stain. For our task, the **generated image is not an end in itself but a means to enrich the feature input for classification**. We found that using the reconstructed image alongside the original greatly boosts downstream accuracy. For instance, feeding a real H&E plus a generated IHC enables the classifier to utilize both morphological and protein expression cues, achieving an accuracy of 94.25% versus 71.44% when using the H&E alone (as shown in our experiments). The CM-GAN module is pre-trained on the training set (with paired data) and then fixed, so that the classification part of the model can utilize the synthetic images without influencing the GAN. In summary, **the reconstruction stage fills in missing information**, effectively approximating a dual-modality input even when only one modality is physically available.

### 3.3 Shared and Specific Feature Encoders

Once we have two images for each case (either both are real H&E and IHC, or one real + one GAN-generated), we feed them into a **dual-encoder feature extraction module**. The goal here is to decompose each image’s representation into **modality-common features** and **modality-specific features**. We achieve this by using one shared encoder that processes both images (concatenated as multi-channel input) and two modality-specific encoders that process each image separately【15】.

Concretely, we use a modified **AlexNet** backbone for simplicity and robustness in this module. The **Shared Encoder** is a CNN that takes the concatenated H&E and IHC image (real+real or real+fake) as input. It is trained to capture **cross-modal common patterns** – features that are present and meaningful in both stains, such as overall tumor morphology, cell density, and architectural patterns associated with HER2 status. In parallel, we have two **Specific Encoders**: one processes only the H&E image (or the generated H&E, in that branch), and the other processes only the IHC image. These encoders (also AlexNet-based) learn **modality-specific features**, e.g., textural details of H&E or membrane staining intensity in IHC, which might not be shared between modalities.

To ensure a clear separation between shared and specific representations, we employ two auxiliary loss terms during training: a **domain classification loss (DCO)** and a **distribution alignment loss (DAO)**. The domain classification loss【15】 acts on the specific features: we attach a small binary classifier head to each modality-specific encoder, which tries to predict the source modality of the feature (H&E vs. IHC). We train this with a binary cross-entropy, which encourages the **specific features to be modality-discriminative** – in other words, forcing each specific encoder to emphasize what makes its modality unique (since the classifier should easily tell them apart). Meanwhile, the distribution alignment loss encourages the shared encoder to produce **modality-invariant features**. We implement this as a Maximum Mean Discrepancy (MMD) loss between the distributions of shared features from H&E and from IHC【15】. By minimizing MMD, we align the feature distributions, making the shared features from H&E and IHC as statistically similar as possible. Together, these losses push the network toward a **decomposition**: features = shared (modality-agnostic) + specific (modality-specific). This design draws inspiration from the work of Wang *et al.*【15】, who showed that such shared-specific modeling improves robustness when inputs are missing or come from different domains.

After this stage, for a given case, we have three sets of features: **Shared features (S)** capturing common information from both images, **H&E-specific features (H)**, and **IHC-specific features (I)**. If a modality was originally missing and reconstructed, the specific features for that modality still exist and carry whatever information could be gleaned or synthesized. Notably, if only one modality was input, the shared encoder still gets a concatenated image (one real, one fake) and can extract shared patterns, while the specific encoder for the fake modality will produce features from the GAN-generated image – which might be less informative than real data but still provide cues to the model. In this way, the model architecture treats real and generated data uniformly at the feature level.

### 3.4 Channel Attention Fusion

Having obtained shared and specific features, we must fuse them for the final prediction. A naive approach would be simply concatenating all features, but this would assume each feature type is equally reliable. Instead, we implement an **attention-based fusion** that can dynamically weight the contributions of **S**, **H**, and **I** features. Specifically, we adopt a **Channel Attention Module (CAM)** similar to the channel attention component of CBAM【16】.

First, we concatenate the shared, H&E-specific, and IHC-specific feature maps along the channel dimension to form a combined feature tensor. If the shared encoder produces $C_S$ feature maps and each specific encoder produces $C_H$ and $C_I$ feature maps respectively, the combined tensor has $C_S + C_H + C_I$ channels. Direct concatenation could lead to a very high-dimensional feature. To control the dimensionality and encourage the model to focus on the most salient features, we apply a **1×1 convolution** across channels to compress the feature dimension (this learns a weighted linear combination of the input feature maps, effectively). For example, we might compress to a total of 125 channels (a hyperparameter chosen via cross-validation) to reduce redundancy.

Next, the channel attention mechanism computes an importance weight for each channel of this compressed feature. We apply global average pooling and global max pooling across spatial dimensions, each producing a vector of length equal to the number of channels. These two vectors pass through a small shared MLP (two fully connected layers with a ReLU in between) and are then summed and passed through a sigmoid, yielding a vector of **attention weights** between 0 and 1 for each channel【16】. This process essentially lets the network learn which feature channels (and thus which combinations of S/H/I features) are most relevant for predicting the HER2 class for the given input. We multiply the original concatenated feature maps by these weights (channel-wise scaling), resulting in an **attended feature map** where informative features are amplified and less useful features are diminished.

Importantly, this attention is learned from data, so it can adapt to different input scenarios. In cases where one modality’s features are less reliable (e.g., if the IHC image is noisy or synthetic), the attention module can learn to give more weight to the other modality’s channels. For instance, if an IHC image is faint or absent (replaced by a GAN output), the network may down-weight channels corresponding to IHC-specific features and rely more on H&E and shared features. Conversely, when both modalities are present and high-quality, the network can combine them to reinforce the signal. This **dynamic weighting** realizes our goal of “complete dual-modality preferred, single-modality compatible” behavior. In ablation experiments, we find that removing the CAM and simply concatenating features leads to a drop in performance (e.g., F1-score decreases by ~4.9%, see Table 3 in Experiments), verifying that **adaptive fusion** is crucial. Our CAM is conceptually similar to that in CBAM【16】, but we specifically apply it to the problem of multi-modal feature fusion where it functions as a modality attention mechanism.

### 3.5 Classification and Loss Functions

The final **Classification Module** operates on the fused feature representation to predict the HER2 expression level category (0, 1+, 2+, or 3+). We implement a two-branch classification head to capture both global context and local details, as HER2 scoring depends on both overall staining patterns and the presence of intensely stained cells. The fused feature map is fed into: (a) a **global pooling branch** that applies Global Average Pooling (GAP) to capture the overall distribution of features, and (b) a **local pooling branch** that uses a 3×3 max-pooling (or several, effectively focusing on local regions) to capture the most prominently activated regions (for example, a cluster of strongly HER2-positive cells in an IHC image, or a region of tumor with certain morphology in H&E). These pooled features are concatenated, forming a feature vector that represents both the holistic and the focused evidence for HER2 status. This vector is then fed into a fully-connected layer (or a small series of them) which outputs a four-dimensional probability vector, corresponding to the predicted probabilities of the four HER2 score categories.

We train the classification module using a **weighted cross-entropy loss** to account for class imbalance (as HER2 0/1+ vs 3+ vs equivocal 2+ may not be equally represented). Higher weights are given to under-represented classes to ensure the model does not become biased toward the majority class. In addition to the primary classification loss, we include the auxiliary losses mentioned earlier: the domain classification loss on specific features, the MMD loss on shared features, and the adversarial and reconstruction losses in the GAN module (the GAN is pre-trained, but if end-to-end fine-tuning were done, those losses would be included). We also include a **feature alignment loss** on the fused features using t-distributed stochastic neighbor embedding (t-SNE) visualization during development to qualitatively ensure that features cluster by class rather than by modality. While this is not a formal loss, the t-SNE visualization (as shown in Figure 7) helps confirm that our shared feature space is indeed modality-agnostic and discriminative of HER2 status: for example, features of HER2 3+ cases cluster together whether they came from dual input or single input with reconstruction.

Overall, the training is conducted in multiple stages: we pre-train the CM-GAN on paired data; pre-train the branch selector on a modality classification task; then train the main network (encoders, fusion, classifier) on the HER2 classification task with all aforementioned losses. We use an **AdamW optimizer** with an initial learning rate of 1e-4 and weight decay of 1e-4 for the main network. During training, if a modality is randomly dropped (to simulate missing data), the GAN generates the missing image on the fly to provide input to the encoders. Our entire architecture is implemented in PyTorch and optimized such that at inference time, the additional overhead for the reconstruction branch and attention is minimal, making it feasible for real-time pathology slide analysis.

## Experiments

### 4.1 Dataset and Preprocessing

We evaluate our method on the **BCI Breast Cancer Pathology Dataset**【14】, a large-scale dataset of paired H&E and HER2 IHC whole-slide images. This dataset consists of **4,870 pairs** of precisely aligned H&E and IHC images, each pair coming from the same tissue region of a breast cancer specimen. Each image pair is annotated with a definitive HER2 expression score (0, 1+, 2+, or 3+) determined by expert pathologists, following ASCO/CAP guidelines. The dataset spans the full spectrum of HER2 expression from negative (0/1+) to equivocal (2+) to strongly positive (3+), providing a realistic distribution of cases. It includes diverse tissue morphologies and staining variations, making it well-suited for developing and testing multi-modal models.

**Data preprocessing:** All images were normalized and resized to a uniform scale for training. We cropped patches of size 1024×1024 pixels from whole-slide images, as in Liu *et al.*【14】, to focus on tumor regions and to manage memory constraints. We applied standard stain normalization to reduce color variability: H&E images were normalized to a reference H&E slide’s color distribution, and IHC images were normalized to a reference HER2 staining intensity. Each image’s label (HER2 score) was converted to a one-hot encoding for training the classifier. For data augmentation, we performed random rotations, flips, and color jittering on training patches to improve generalization.

We simulated missing modality scenarios in our experiments to thoroughly test the model’s flexibility. In the **training phase**, we randomly dropped one modality with a certain probability (e.g., 20%) for each pair, and used the GAN to reconstruct it, so the model learns to handle incomplete inputs. In the **testing phase**, we evaluate under two settings: (1) **Fixed-modality scenarios** – *H&E-only*, *IHC-only*, and *H&E+IHC* – to measure performance when only one stain versus both are available. (2) **Dynamic missing scenario** – where for each test pair, we randomly (20% chance) drop the IHC or H&E image (mimicking real clinical workflows where occasionally a slide might be unavailable or unusable). This dynamic test evaluates how the model copes with unpredictable modality availability on the fly.

We emphasize that our model is single-blind to whether an input IHC is real or GAN-generated; it simply receives whatever is provided by the branch selector and CM-GAN. The branch selector ensures that during testing in H&E-only or IHC-only conditions, the reconstruction is invoked so that the downstream pipeline always sees two images per case (one real, one synthetic if needed).

### 4.2 Evaluation Metrics

We report a comprehensive set of metrics to evaluate classification performance: **Accuracy**, **Recall**, **Precision**, and **F1-score** for the four-class HER2 classification. Accuracy is the overall percentage of correctly classified cases. Precision and recall are computed per class and averaged (macro-average) to account for class imbalance, and F1-score (the harmonic mean of precision and recall) is reported as a summary metric; F1 is particularly informative here because it balances the model’s ability to avoid false positives and false negatives in each HER2 category. We also examine the **confusion matrix** to see the distribution of predictions across classes, which provides insights into common misclassification patterns (e.g., how often 2+ cases are mistaken for 1+ or 3+).

Additionally, although not a primary evaluation metric, we utilize **t-SNE visualization** of the learned feature space to assess model interpretability. By projecting high-dimensional features into 2D, we can visually inspect whether the model clusters cases by HER2 status regardless of input modality. A well-formed clustering would indicate that the model’s shared feature space is capturing HER2-related characteristics and not simply grouping by stain type, which supports our design goals.

All metrics are computed on the held-out test set under the various input scenarios described. We also ensure statistical significance of improvements by conducting paired t-tests on the per-sample outcomes between our method and baselines, where applicable.

### 4.3 Implementation Details

We implemented our model using **PyTorch 2.0**. Training was performed on a workstation with an **Intel Xeon Platinum 8352V CPU** and an **NVIDIA RTX 4090 GPU (24GB)**. We used the AdamW optimizer for all trainable components. The initial learning rate was set to 1×10^-4 for the classification and encoder parts, and 2×10^-4 for the GAN (during its separate pre-training), with a weight decay of 1×10^-4. We employed a learning rate scheduler that reduces the LR by a factor of 0.1 if validation loss plateaued for 5 epochs, to fine-tune convergence and avoid overfitting. Batch size was set to 8 (pairs of patches) given memory constraints with high-resolution images. We trained the branch selector and CM-GAN modules separately first, then the full model end-to-end (keeping the GAN weights fixed to maintain generative performance). The two-stage training helps stabilize the adversarial training and ensures the classification part does not bias the image generation.

All experiments used the same train/validation/test split: approximately **80% of the pairs (3,896 pairs)** for training, and **20% (974 pairs)** for testing, with a small subset of the training set further held out for validation and hyperparameter tuning (the split was stratified to maintain class distribution). We ensured that no patient’s slides appear in both training and test sets to evaluate generalization. The model converged after about 50 epochs of main training, which took roughly 48 hours given the complexity of the architecture. Inference time for a single case (with one or two 1024×1024 images) is about 0.3 seconds when both modalities are present, and ~0.5 seconds when a GAN generation is needed, which is practically negligible in a diagnostic setting (much faster than manual scoring or additional lab tests).

### 4.4 Comparison with Baseline Methods

We conducted two sets of comparative experiments to evaluate the effectiveness of each major component of our approach. For fair comparison, all methods were evaluated on the same data split and under the same conditions.

#### 4.4.1 Single-Modality vs. Cross-Modal Reconstruction Enhancement

First, we compare our **cross-modal reconstruction approach** to baseline single-modality classifiers. For the baseline, we trained separate single-modality models that take only a real H&E or only a real IHC image as input and predict HER2 status. We used a MobileNetV2 backbone for these baselines to represent a compact model (MobileNetV2 was chosen for its balance of speed and accuracy in image classification). For our method, in the single-modality scenario, the pipeline uses a real image plus a **GAN-generated companion image** (e.g., Real H&E + Fake IHC) which then goes through our shared-specific encoders and attention fusion.

**Table 1** presents the performance of the baseline vs. our method in H&E-only and IHC-only cases. The baseline H&E-only model achieved 71.44% accuracy (F1-score 0.6697), indicating the challenge of predicting HER2 from morphology alone. With our cross-modal enhancement (Real H&E + Fake IHC), the accuracy jumps dramatically to 94.25%, and the F1-score to 0.9609. This is a **+22.8% accuracy improvement** over the H&E baseline. Similarly, for IHC-only: the baseline IHC model achieved 77.38% accuracy (F1 0.7646), whereas using Real IHC + Fake H&E in our model reached 90.28% accuracy (F1 0.9251), a **+12.9% improvement**. These results validate that **cross-modal generation effectively compensates for missing information**. The model can glean additional discriminative cues from the GAN-generated images – for example, when only an H&E is real, the fake IHC provides an estimate of HER2 protein distribution that guides the classifier. Notably, the H&E + fake IHC combination outperforms IHC + fake H&E; this aligns with domain knowledge that H&E contains a lot of contextual info and our GAN was able to infer IHC details fairly well (hence yielding nearly 94% accuracy), whereas inferring full H&E from IHC is a bit harder (90% accuracy, still a big gain from 77%). Overall, this experiment demonstrates the **robustness gained by our adaptive single-modality pipeline** over traditional single-modality approaches.

**Table 1.** Performance of single-modality baseline vs. our cross-modal reconstruction method. “Real” denotes an original image; “Fake” denotes a GAN-generated image. Our method’s use of Real+Fake input yields significantly higher accuracy and F1-score than using a single modality alone.

| Input Modality                 | Accuracy (%) | F1-score   | Improvement vs. Single Modal |
| ------------------------------ | ------------ | ---------- | ---------------------------- |
| **H&E only (baseline)**        | 71.44%       | 0.6697     | –                            |
| **IHC only (baseline)**        | 77.38%       | 0.7646     | –                            |
| **Real H&E + Fake IHC (ours)** | **94.25%**   | **0.9609** | +22.81% vs H&E baseline      |
| **Real IHC + Fake H&E (ours)** | **90.28%**   | **0.9251** | +12.90% vs IHC baseline      |

#### 4.4.2 Dual-Modality Fusion: Simple Concatenation vs. Shared-Specific Fusion

Next, we evaluate the benefit of our **shared-specific feature fusion** approach when both modalities are available. The baseline for dual-modality input is a straightforward one: **concatenate the H&E and IHC images** (for example, stacking their channels) and feed them into a standard CNN (MobileNetV2 backbone, same as single) for classification. This baseline represents a typical early-fusion strategy without any specialized feature decoupling or attention – it simply learns from the combined input as-is.

Our method, in contrast, uses the specialized dual encoders and attention fusion as described. We tested both methods on the cases where real H&E and real IHC are provided. **Table 2** summarizes the results. The simple concatenation baseline achieved 92.00% accuracy with an F1-score of 0.9103. This is already higher than any single-modality result, confirming that providing both stains helps. However, our fusion model further improved performance to **95.09% accuracy** and an F1-score of 0.9532. This is an absolute accuracy gain of **+3.09%**, which corresponds to a 4.29% relative improvement over the baseline. More importantly, the F1-score improvement indicates that our model is better at handling the nuanced classes (like 2+ cases), likely because it can disentangle and amplify the complementary features more effectively than a naive concat.

These results indicate that **not all fusion strategies are equal**: simply feeding both images into a CNN yields some benefit, but **our method’s learned feature separation and dynamic re-weighting capture cross-modal complementarities that a plain CNN misses**. The baseline likely struggles with the heterogeneity of input (different stains have very different appearance distributions) and may not optimally exploit both. In contrast, our model explicitly learns what features are shared vs specific and then uses attention to integrate them, leading to a richer representation. Qualitatively, we observed that the simple concat model sometimes misclassified 2+ vs 1+ (perhaps because it couldn’t reconcile conflicting weak signals between H&E and IHC), whereas our model was more consistent in such cases, often correctly identifying subtle amplified regions thanks to the separated feature streams.

**Table 2.** Performance comparison of dual-modality fusion strategies. The baseline concatenation model simply feeds both images into a CNN, whereas our method uses shared-specific feature fusion with attention. Our approach yields higher accuracy and F1, demonstrating more effective use of both modalities.

| Fusion Strategy                              | Accuracy (%) | F1-score   | Improvement vs. Baseline |
| -------------------------------------------- | ------------ | ---------- | ------------------------ |
| **Dual input, simple concat (baseline)**     | 92.00%       | 0.9103     | –                        |
| **Dual input, shared-specific + CAM (ours)** | **95.09%**   | **0.9532** | +3.09% (Accuracy)        |
|                                              |              |            | +4.29% (relative F1)     |

### 4.5 Ablation Studies

We conduct targeted **ablation experiments** to quantify the contribution of key components in our model. Specifically, we focus on the **Channel Attention Module (CAM)** in the fusion stage, as this is crucial for dynamic weighting, and evaluate its necessity.

#### 4.5.1 Effect of Channel Attention (CAM)

In this ablation, we compare the full model with CAM against a variant where CAM is removed. In the **No-CAM variant**, after extracting shared and specific features, we simply concatenate them and use a 1×1 convolution for dimension reduction (as in our model) but then directly feed this into the classifier without applying the channel attention weighting. Essentially, this treats all features equally (a static fusion).

We measure the performance of both models on the dual-modality input scenario to isolate the effect on fusion. **Table 3** shows the results. The model **with CAM** (our full model) achieved an F1-score of 0.9532 (as before) and accuracy ~95.3% on this test. The **No-CAM model** had a noticeably lower F1-score of 0.9041 and accuracy ~92.4%. The drop in F1 is about **4.9 percentage points**, which is quite significant in a four-class classification (where chance-level F1 would be much lower). This validates that the CAM is contributing substantially to the model’s success.

With CAM, the model can handle variations in input quality: for example, if one modality is partially uninformative, CAM adjusts accordingly. Without CAM, when both modalities are always present, the model might still do well on average (92% accuracy is not bad), but it likely fails in cases where one modality should dominate. The confusion matrix for the No-CAM model (not fully shown here) indicated more confusion in borderline cases – we suspect because the model could not down-weight noisy modality signals. The CAM-equipped model, on the other hand, was better at emphasizing, say, strong IHC signals for a 3+ case, or relying on H&E context when IHC was equivocal.

In summary, this ablation confirms that **dynamic feature weighting via CAM yields a more robust fusion**, boosting the F1-score significantly. It demonstrates the importance of adaptive modality weighting in achieving high performance. All other components (GAN, dual encoders) were kept the same in this comparison, so the difference comes purely from the attention mechanism. We consider this one of the core novelties of our approach, as static fusion would not suffice to reach the performance levels we achieved.

**Table 3.** Ablation of the Channel Attention Module (CAM). Removing CAM (static fusion) leads to a notable drop in performance, indicating that dynamically weighting features improves the model’s effectiveness.

| Model Variant                   | Accuracy (%) | F1-score   | F1 Improvement (with CAM) |
| ------------------------------- | ------------ | ---------- | ------------------------- |
| **Full model (with CAM)**       | **95.32%**   | **0.9532** | –                         |
| **Without CAM (static fusion)** | 92.41%       | 0.9041     | **+4.91%**                |

### 4.6 Further Analysis

Beyond these quantitative results, we conducted additional analysis to understand our model’s behavior. Figure 6 illustrates examples of input images and the corresponding generated images by CM-GAN. We observe that the fake IHC images produced from H&E inputs capture the general pattern of HER2 staining (for example, highlighting membranous regions in a case that was HER2 3+), even if not all fine details are perfect. These synthetic images helped the model focus on likely HER2-positive cell clusters. Conversely, fake H&E images generated from IHC provided structural context (nuclear and stromal patterns) that complemented the IHC features.

We also looked at the **t-SNE plots** of the fused features for test cases under different input conditions. The full model’s feature embeddings cluster primarily by HER2 category rather than by whether the input was single or dual modality. For instance, HER2 3+ cases form a tight cluster regardless of whether they were processed with real dual input or with an H&E + fake IHC. This indicates the model successfully learned a modality-invariant feature space for the task. In contrast, a model without the shared-specific design showed some separation by modality in feature space (with H&E-only features somewhat apart from dual-modal features), which could lead to inconsistent predictions. Our approach’s aligned feature space is a direct result of the shared encoder with MMD loss, and it aids generalization.

Finally, we computed class-specific precision and recall. The full model achieved >95% precision and recall for the 3+ class and ~90% for the 0/1+ class, which are the extremes. The 2+ class (equivocal) is inherently the hardest – our model attained ~88% precision and 85% recall for 2+, whereas single-modality baselines were around 60-70%, showing a clear improvement. This suggests our model could be particularly useful in helping identify 2+ cases that truly are HER2-positive (or negative) by providing additional context from the other stain, potentially reducing the need for reflex ISH testing.

## Conclusion

We have presented a **flexible multi-modal deep learning framework** for HER2 expression level prediction that, for the first time, accommodates both complete and incomplete modality inputs. Our model’s **dynamic branch selection and bidirectional reconstruction** mechanisms effectively address the rigid dependency on dual inputs that has limited previous multi-modal approaches. Extensive experiments on a large HER2 pathology dataset demonstrate that our approach achieves **state-of-the-art accuracy** when both H&E and IHC are available (95.1%), and more importantly, it **maintains high performance even with a single modality input** (e.g., ~94% accuracy with only H&E, far outperforming a conventional ~71% single-stain model). This flexible performance could dramatically reduce the need for IHC in settings where it is costly or slow, as an H&E alone can yield nearly the same diagnostic insight when processed through our model.

Our ablation studies confirm the importance of each component: the cross-modal GAN reconstruction **mitigates information loss** from missing modalities, the shared-specific dual encoder architecture ensures robust feature learning from each modality, and the channel attention fusion provides intelligent weighting that enhances prediction reliability. The resulting system is robust to real-world variability – it can, for instance, down-weight a faint IHC and still make a correct call from H&E features, or vice versa.

One limitation of our current study is that all data originated from a single source (the BCI dataset). While we performed augmentations to simulate variability, we have not yet tested the model on external datasets from different institutions or scanners. In future work, we will pursue a **multi-center evaluation** to verify the model’s generalization across variations in staining protocols and imaging equipment. Incorporating a domain adaptation strategy or fine-tuning on new data might be necessary to maintain performance in such scenarios. We also plan to investigate extending this **dynamic multi-modal approach to other pathology tasks**. For example, integrating H&E with genomic data or radiology images for outcome prediction, or combining multiple immunohistochemical markers in a single model, are promising directions. The core idea of **“activate modalities on demand”** could greatly benefit any application where data availability is inconsistent.

In conclusion, our work provides a **novel and practical solution for HER2 scoring** that aligns with real clinical workflows by not requiring perfectly matched multi-modal inputs. This paves the way for more accessible HER2 testing, especially in low-resource environments where IHC might be limited. Beyond HER2, the framework exemplifies a new paradigm for **elastic multi-modal learning** in medical AI – moving from static analysis towards adaptive systems that can intelligently accommodate the data at hand. We believe this approach will help drive pathology AI toward greater clinical adoption by reducing data requirements and increasing resilience to missing information, ultimately contributing to more precise and personalized diagnostic tools in oncology.

## References

1. Sun, Y. S., Zhao, Z., Yang, Z. N., Xu, F., Lu, H. J., Zhu, Z. Y., *et al.* (2017). **Risk Factors and Preventions of Breast Cancer.** *International Journal of Biological Sciences, 13*(11), 1387–1397. DOI: 10.7150/ijbs.21635
2. Waks, A. G., & Winer, E. P. (2019). **Breast Cancer Treatment: A Review.** *JAMA, 321*(3), 288–300. DOI: 10.1001/jama.2018.19323
3. Ahn, S., Woo, J. W., Lee, K., & Park, S. Y. (2020). **HER2 Status in Breast Cancer: Changes in Guidelines and Complicating Factors for Interpretation.** *Journal of Pathology and Translational Medicine, 54*(1), 34–44. DOI: 10.4132/jptm.2019.09.28
4. Viale, G., & Bardia, A. (2023). **HER2-low breast cancer – Diagnostic challenges and opportunities.** *Targeted Oncology, 18*, 1–9. DOI: 10.1007/s11523-022-00918-3
5. Press, M. F., Hung, G., Godolphin, W., & Slamon, D. J. (1994). **Sensitivity of HER-2/neu Antibodies in Archival Tissue Samples: Potential Source of Error in Immunohistochemical Studies of Oncogene Expression.** *Cancer Research, 54*(10), 2771–2777.
6. Furrer, D., Sanschagrin, F., Jacob, S., & Diorio, C. (2015). **Advantages and Disadvantages of Technologies for HER2 Testing in Breast Cancer Specimens.** *American Journal of Clinical Pathology, 144*(5), 686–703. DOI: 10.1309/AJCP1L5XIHEZPHJF
7. Masmoudi, H., Hewitt, S. M., Petrick, N., Myers, K. J., & Gavrielides, M. A. (2009). **Automated Quantitative Assessment of HER-2/neu Immunohistochemical Expression in Breast Cancer.** *IEEE Transactions on Medical Imaging, 28*(6), 916–925. DOI: 10.1109/TMI.2008.2010406
8. Xiong, Z., Liu, K., Liu, S., Feng, J., Wang, J., Feng, Z., *et al.* (2024). **Precision HER2: A Comprehensive AI System for Accurate and Consistent Evaluation of HER2 Expression in Invasive Breast Cancer.** *BMC Cancer, 24*(1), 1204. DOI: 10.1186/s12885-024-10485-8
9. Chauhan, E., Sharma, A., Sharma, A., Nishadham, V., Ghughtyal, A., Kumar, A., *et al.* (2025). **Contrasting Low and High-Resolution Features for HER2 Scoring using Deep Learning.** *arXiv preprint arXiv:2503.22069*.
10. Che, Y., Ren, F., Zhang, X., Cui, L., Wu, H., & Zhao, Z. (2023). **Immunohistochemical HER2 Recognition and Analysis of Breast Cancer Based on Deep Learning.** *Diagnostics, 13*(2), 263. DOI: 10.3390/diagnostics13020263
11. Farahmand, S., Fernandez, A. I., Ahmed, F. S., Rimm, D. L., Chuang, J. H., Reisenbichler, E., & Zarringhalam, K. (2022). **Deep Learning Trained on Hematoxylin and Eosin Tumor Region-of-Interest Predicts HER2 Status and Trastuzumab Treatment Response in HER2+ Breast Cancer.** *Modern Pathology, 35*(1), 44–51. DOI: 10.1038/s41379-021-00872-5
12. Rasmussen, S. A., Taylor, V. J., Surette, A. P., Barnes, P. J., & Bethune, G. C. (2022). **Using Deep Learning to Predict Final HER2 Status in Invasive Breast Cancers That Are Equivocal (2+) by Immunohistochemistry.** *Applied Immunohistochemistry & Molecular Morphology, 30*(10), 668–673. DOI: 10.1097/PAI.0000000000000981
13. Nicolò, E., Boscolo Bielo, L., *et al.* (2023). **The HER2-low Revolution in Breast Oncology: Steps Forward and Emerging Challenges.** *Therapeutic Advances in Medical Oncology, 15*, 17588359231152842. DOI: 10.1177/17588359231152842
14. Liu, S., Zhu, C., Xu, F., Jia, X., Shi, Z., & Jin, M. (2022). **BCI: Breast Cancer Immunohistochemical Image Generation through Pyramid Pix2Pix.** In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)* (pp. 1815–1824). DOI: 10.1109/CVPR52688.2022.00186
15. Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., & Carneiro, G. (2023). **Multi-Modal Learning with Missing Modality via Shared-Specific Feature Modelling.** In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)* (pp. 15878–15887). DOI: 10.1109/CVPRW53098.2023.00310
16. Woo, S., Park, J., Lee, J. Y., & Kweon, I. S. (2018). **CBAM: Convolutional Block Attention Module.** In *Proceedings of the European Conference on Computer Vision (ECCV)* (pp. 3–19). DOI: 10.1007/978-3-030-01234-2_1
17. Steyaert, S., Pizurica, M., Nagaraj, D., Khandelwal, P., Hernandez-Boussard, T., Gentles, A. J., & Gevaert, O. (2023). **Multimodal Data Fusion for Cancer Biomarker Discovery with Deep Learning.** *Nature Machine Intelligence, 5*(4), 351–362. DOI: 10.1038/s42256-023-00633-5
18. McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., *et al.* (2020). **International Evaluation of an AI System for Breast Cancer Screening.** *Nature, 577*(7788), 89–94. DOI: 10.1038/s41586-019-1799-6
19. Joo, S., Ko, E. S., Kwon, S., Jeon, E., Jung, H., Kim, J. Y., *et al.* (2021). **Multimodal Deep Learning Models for the Prediction of Pathologic Response to Neoadjuvant Chemotherapy in Breast Cancer.** *Scientific Reports, 11*(1), 18800. DOI: 10.1038/s41598-021-98408-8