sform(X_compressed)

def get_variance_report(self) -> dict:
    return {
        "components_selected": self.reducer.n_components_,
        "cumulative_variance": float(np.sum(self.reducer.explained_variance_ratio_)),
        "individual_ratios": self.reducer.explained_variance_ratio_.tolist()
    }

--- Execution & Validation ---

if name == "main": dataset = load_digits() X_raw, y_labels = dataset.data, dataset.target

compressor = OrthogonalCompressor(variance_threshold=0.95)
X_compressed = compressor.fit_transform(X_raw)

report = compressor.get_variance_report()
print(f"Selected components: {report['components_selected']}")
print(f"Total variance retained: {report['cumulative_variance']:.2%}")

# Reconstruction fidelity check
X_reconstructed = compressor.reconstruct(X_compressed)
reconstruction_mse = np.mean((X_raw - X_reconstructed) ** 2)
print(f"Reconstruction MSE: {reconstruction_mse:.4f}")

# Downstream model comparison
baseline_pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000, random_state=42))
])

compressed_pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("compressor", PCA(n_components=0.95, random_state=42)),
    ("classifier", LogisticRegression(max_iter=1000, random_state=42))
])

baseline_scores = cross_val_score(baseline_pipe, X_raw, y_labels, cv=5)
compressed_scores = cross_val_score(compressed_pipe, X_raw, y_labels, cv=5)

print(f"Baseline accuracy: {baseline_scores.mean():.3f} ± {baseline_scores.std():.3f}")
print(f"Compressed accuracy: {compressed_scores.mean():.3f} ± {compressed_scores.std():.3f}")

### Architecture Decisions & Rationale

1. **Pipeline Encapsulation**: Wrapping `StandardScaler` and `PCA` inside a single `Pipeline` guarantees that scaling parameters are learned exclusively on training folds during cross-validation. This eliminates a common source of optimistic bias in production metrics.
2. **Float-Based `n_components`**: Passing `0.95` instead of an integer allows the compressor to adapt to dataset drift. If new features introduce additional variance, the algorithm automatically allocates more axes without manual reconfiguration.
3. **Explicit Reconstruction Validation**: Computing MSE between original and inverse-transformed data provides a deterministic quality gate. In CI pipelines, you can fail builds if reconstruction error exceeds a predefined threshold, catching data schema changes early.
4. **Deterministic Random State**: Setting `random_state` ensures reproducible component ordering across environments. While PCA is deterministic via SVD, downstream sampling or parallelized variants benefit from explicit seeding.

## Pitfall Guide

### 1. Skipping Feature Standardization
**Explanation**: PCA maximizes variance. Features with larger numerical ranges dominate the covariance matrix, forcing principal components to align with scale rather than signal.
**Fix**: Always apply `StandardScaler` or `MinMaxScaler` before fitting. Verify that all columns share a comparable magnitude distribution.

### 2. Assuming Variance Equals Predictive Power
**Explanation**: PCA is unsupervised. It identifies directions of maximum spread, not directions that separate classes. The highest-variance components may contain background noise or irrelevant trends.
**Fix**: Validate downstream model performance after compression. If accuracy drops significantly, consider supervised alternatives like Linear Discriminant Analysis (LDA) or feature importance filtering.

### 3. Data Leakage Across Validation Splits
**Explanation**: Fitting PCA on the entire dataset before cross-validation leaks global statistics into training folds. This inflates validation scores and fails to simulate production inference.
**Fix**: Embed PCA inside a `Pipeline` or manually fit the transformer only on training indices during each CV iteration.

### 4. Ignoring the Linearity Assumption
**Explanation**: PCA constructs linear combinations of features. It cannot untangle curved manifolds, spirals, or hierarchical clusters. Applying it to non-linear data flattens structure and destroys local neighborhoods.
**Fix**: Use UMAP or t-SNE for visualization of non-linear data. For compression, explore Kernel PCA or autoencoders when linear projections fail to preserve topology.

### 5. Over-Compression Thresholds
**Explanation**: Setting `n_components=0.99` retains nearly all variance but negates the computational benefits of dimensionality reduction. The pipeline pays the transformation cost without meaningful latency gains.
**Fix**: Profile downstream model inference time against compression ratios. In production, 0.90–0.95 typically delivers the best latency-accuracy Pareto frontier.

### 6. Treating Components as Interpretable Features
**Explanation**: Principal components are weighted sums of original attributes. A single PC rarely maps to a business concept, making debugging and feature attribution difficult.
**Fix**: Extract component loadings (`pca.components_`) to document attribute contributions. Use SHAP or permutation importance on the compressed space if explainability is required.

### 7. Applying PCA to Sparse or Categorical Data
**Explanation**: PCA relies on dense covariance calculations. Sparse matrices (e.g., TF-IDF, one-hot encodings) produce unstable components, and categorical variables lack meaningful variance metrics.
**Fix**: Use TruncatedSVD for sparse text data. Encode categoricals with target encoding or embeddings before applying linear compression, or switch to non-linear manifold learners.

## Production Bundle

### Action Checklist
- [ ] Standardize all numerical features before fitting PCA
- [ ] Wrap scaler and compressor in a single Pipeline to prevent leakage
- [ ] Set variance threshold between 0.90 and 0.95 based on latency profiling
- [ ] Validate reconstruction MSE to quantify information loss
- [ ] Compare downstream model accuracy with and without compression
- [ ] Document component loadings for auditability and debugging
- [ ] Monitor feature distribution drift; re-fit compressor if covariance shifts significantly

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| High-dimensional numerical data with linear correlations | PCA with 0.95 variance threshold | Deterministic compression, fast inference, reduces noise | Low compute, moderate memory savings |
| Sparse text or one-hot encoded features | TruncatedSVD or HashingVectorizer | PCA fails on sparse covariance; SVD handles sparsity efficiently | Low compute, high memory efficiency |
| Non-linear manifolds or complex clustering | UMAP / t-SNE for visualization, Autoencoder for compression | Linear projection destroys local topology; non-linear methods preserve structure | High compute, moderate memory |
| Strict latency SLA with marginal accuracy tolerance | Fixed component count (e.g., 10–20) | Predictable inference time, avoids dynamic threshold overhead | Lowest compute, highest speedup |
| Regulatory/audit requirements for feature traceability | PCA with loading extraction + SHAP analysis | Components are opaque; loadings map back to original attributes | Moderate compute, high compliance value |

### Configuration Template

```python
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier

def build_production_pipeline(variance_target: float = 0.92):
    """
    Returns a production-ready pipeline with standardized scaling,
    adaptive PCA compression, and a robust downstream estimator.
    """
    return Pipeline([
        ("feature_standardization", StandardScaler()),
        ("dimensionality_reduction", PCA(
            n_components=variance_target,
            random_state=42,
            svd_solver="auto"
        )),
        ("predictor", RandomForestClassifier(
            n_estimators=200,
            max_depth=12,
            random_state=42,
            n_jobs=-1
        ))
    ])

# Usage
# pipeline = build_production_pipeline(variance_target=0.90)
# pipeline.fit(X_train, y_train)
# predictions = pipeline.predict(X_test)

Quick Start Guide

1. Install dependencies: pip install scikit-learn numpy
2. Prepare your dataset: Ensure all features are numerical. Handle missing values and encode categoricals before proceeding.
3. Instantiate the compressor: Create a Pipeline containing StandardScaler and PCA(n_components=0.95).
4. Fit and transform: Call .fit_transform(X_train) on training data. Apply .transform(X_test) to validation/test sets.
5. Validate downstream performance: Train your target model on the compressed data. Compare latency and accuracy against the raw feature baseline. Adjust the variance threshold if inference speed or predictive power falls outside SLAs.