Dynamic Malware Analysis: A Complete Guide to Behavioral Monitoring (Part 2)

February 08, 2026
Dynamic Malware Analysis: A Complete Guide to Behavioral Monitoring (Part 2)

Dynamic Malware Analysis: A Complete Guide to Behavioral Monitoring (Part 2)

Updated on February 08, 2026

This is Part 2 of our malware analysis series. If you missed the fundamentals of examining files without execution, read Part 1: Analyze Malicious Office Documents and Static Analysis first.

In Part 1, we covered static malware analysis where we examined files without executing them. Now, dynamic analysis is where the real action happens. You're going to execute the malware in a controlled environment and watch it do its thing in real-time. This is where you catch behaviors that static analysis misses - like command-and-control callbacks, process injection, persistence mechanisms, and data exfiltration attempts.

Dynamic analysis reveals the true intent of malware by observing its runtime behavior, network communications, registry modifications, and file system changes. While static analysis gives you the blueprint, dynamic analysis shows you the building being constructed. This approach is critical for understanding zero-day threats, polymorphic malware, and heavily obfuscated samples that change their code signatures constantly.

This guide walks through the complete dynamic analysis workflow using both local sandboxes and cloud-based platforms. You'll learn how to set up isolated environments, monitor system activity, capture network traffic, and extract actionable indicators of compromise.

Note: Before analyzing any malware sample, ensure you have proper authorization and are working in an isolated environment. Never execute malware on production systems or networks.

Dynamic Malware Analysis: A Complete Guide to Behavioral Monitoring

Prerequisites

Environment Requirements:

  • Isolated analysis machine (VM recommended)
  • Snapshot capability for quick restoration
  • Network isolation or simulated network environment
  • Monitoring tools installed and configured

Access Level:

  • Local administrator access on analysis VM
  • Ability to modify firewall and network settings
  • Permissions to install monitoring tools

Base Tools Installation:

# Windows Sysinternals Suite
Invoke-WebRequest -Uri "https://download.sysinternals.com/files/SysinternalsSuite.zip" -OutFile "SysinternalsSuite.zip"

# Python for FakeNet-NG
choco install python -y

# FakeNet-NG (Network Simulator)
pip install https://github.com/mandiant/flare-fakenet-ng/zipball/master

# Wireshark
choco install wireshark -y

Setting Up the Analysis Environment

Before executing any malware, you need a proper sandbox environment that isolates the threat while providing full visibility into its actions.

Local Sandbox Setup

Virtual Machine Configuration:

Create a disposable Windows VM with the following specs:

# Recommended VM settings
- OS: Windows 10/11 (match your target environment)
- RAM: 4GB minimum
- Disk: 50GB
- Network: Host-only or NAT (isolated)
- Snapshots: Clean baseline snapshot

Network Isolation:

# Disable internet access while maintaining local network simulation
netsh advfirewall set allprofiles state on
netsh advfirewall firewall add rule name="Block All Outbound" dir=out action=block

# Or use VMware/VirtualBox network settings
# Set adapter to "Host-only" or "Internal Network"

Install Monitoring Tools:

# Process Monitor from Sysinternals
.\Procmon.exe /AcceptEula /Minimized /BackingFile procmon.pml

# Regshot for registry comparison
# Download from https://sourceforge.net/projects/regshot/
# Run 1st shot before malware execution

# System Informer (formerly Process Hacker)
# Download from https://systeminformer.com/

Network Simulation Tools

Malware often requires network connectivity to reach command-and-control servers. Network simulation tools fake these connections locally.

FakeNet-NG Setup:

# Install FakeNet-NG
pip install https://github.com/mandiant/flare-fakenet-ng/zipball/master

# Start FakeNet-NG with default config
fakenet.exe

# Custom configuration
fakenet.exe -c custom_config.ini

# Log all network activity
fakenet.exe -l fakenet.log

INetSim Alternative:

# On Linux host for more control
apt-get install inetsim -y

# Start INetSim
inetsim --config /etc/inetsim/inetsim.conf

# Configure VM to use host IP as DNS/gateway

Local Dynamic Analysis Workflow

Step 1: Baseline System State

Before executing malware, capture the clean state of your system.

Registry Baseline:

# Launch Regshot
regshot.exe

# Click "1st shot" and wait for completion
# This creates a baseline of all registry keys

Process Baseline:

# List running processes
Get-Process | Export-Csv -Path baseline_processes.csv

# Network connections baseline
netstat -ano > baseline_netstat.txt

File System Baseline:

# Quick directory listing of common locations
dir C:\Users\* /s > baseline_users.txt
dir C:\Windows\Temp /s > baseline_temp.txt
dir C:\ProgramData /s > baseline_programdata.txt

Step 2: Start Monitoring Tools

Launch all monitoring tools before executing the malware sample.

Process Monitor Configuration:

# Start Process Monitor with filters
.\Procmon.exe /AcceptEula /Minimized /BackingFile procmon.pml

# Set filters to reduce noise:
# Include: Process Name -> contains -> malware.exe
# Include: Operation -> begins with -> Reg
# Include: Operation -> begins with -> File

System Informer Monitoring:

# Launch System Informer
.\SystemInformer.exe

# Enable these views:
# - Processes (with full command lines)
# - Network connections
# - Services

Network Capture:

# Start Wireshark capture
"C:\Program Files\Wireshark\Wireshark.exe" -i 1 -k -w malware_traffic.pcap

# Or use tcpdump on Linux host
tcpdump -i eth0 -w malware_traffic.pcap

FakeNet-NG for Network Simulation:

# Start FakeNet in new terminal
cd C:\Tools\FakeNet-NG
python fakenet.py

# FakeNet will simulate responses from C2 servers

Step 3: Execute Malware Sample

Now execute the malware while all monitoring tools are running.

Basic Execution:

# Direct execution
.\malware.exe

# With arguments (if needed)
.\malware.exe --install

# Via rundll32 for DLL analysis
rundll32.exe malware.dll,EntryPoint

Execution with Interaction:

Some malware requires user interaction to trigger behaviors.

# Open document-based malware
start malicious.docm

# Execute installer
.\setup.exe /S

# Trigger via script
powershell.exe -ExecutionPolicy Bypass -File trigger.ps1

Monitor Immediate Behavior:

Watch for:

  • New processes spawned
  • Network connections established
  • Files created/modified
  • Registry keys added

Step 4: Behavioral Observation

Let the malware run for 5-10 minutes while observing its behavior patterns.

Process Activity:

# Check process tree in System Informer
# Look for:
# - Child processes
# - Unusual parent-child relationships
# - Process injection (processes with mismatched paths)
# - Hollow processes

Network Activity:

# Monitor active connections
netstat -ano | findstr ESTABLISHED

# Check DNS queries
ipconfig /displaydns

# Review FakeNet logs
type C:\Tools\FakeNet-NG\fakenet.log

File System Changes:

# New files in common locations
dir C:\Users\%USERNAME%\AppData\Roaming /s /od
dir C:\Users\%USERNAME%\AppData\Local\Temp /s /od
dir C:\Windows\Temp /s /od
dir C:\ProgramData /s /od

Registry Modifications:

# Common persistence locations
reg query HKCU\Software\Microsoft\Windows\CurrentVersion\Run
reg query HKLM\Software\Microsoft\Windows\CurrentVersion\Run
reg query HKCU\Software\Microsoft\Windows\CurrentVersion\RunOnce
reg query "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon"

# Scheduled tasks
schtasks /query /fo LIST /v

Step 5: Post-Execution Analysis

After malware execution, capture the final state and compare with baseline.

Registry Comparison:

# Take 2nd shot in Regshot
# Click "2nd shot" and wait
# Click "Compare" to see all registry changes
# Export results for documentation

Process Dump:

# Dump suspicious process memory with System Informer
# Right-click process -> Create dump file -> Full dump

# Or use ProcDump
procdump.exe -ma  process_dump.dmp

Network Traffic Analysis:

# Stop Wireshark capture
# Analyze PCAP for:
# - DNS queries to suspicious domains
# - HTTP/HTTPS connections
# - Unusual protocols (IRC, custom protocols)
# - Data exfiltration patterns

Timeline Reconstruction:

# Export Process Monitor logs
# File -> Save -> All events -> CSV

# Review timeline of events:
# - File operations
# - Registry operations
# - Network operations
# - Process/thread creation

Cloud-Based Sandbox Analysis

Cloud sandboxes provide instant analysis without local setup. They offer advanced features like automated IOC extraction and threat intelligence integration.

ANY.RUN Interactive Sandbox

ANY.RUN provides real-time interaction with malware during execution.

Basic Usage:

# 1. Visit https://any.run
# 2. Click "New task"
# 3. Upload malware sample
# 4. Select OS version (Windows 7/10/11)
# 5. Enable network simulation
# 6. Start analysis

Interactive Analysis Features:

During execution, you can:

  • Interact with malware UI
  • Open files to trigger document-based malware
  • Browse to specific URLs
  • Simulate user actions
  • Download additional payloads

Monitoring Views:

# Process Tree View
# Shows parent-child relationships
# Color-coded threat indicators

# Network Activity
# HTTP requests with full URLs
# DNS queries and resolutions
# TCP/UDP connections
# Detected threats via Suricata rules

# File System Changes
# Created files
# Modified files
# Deleted files

# Registry Changes
# Added keys
# Modified values
# Persistence mechanisms

IOC Extraction:

# ANY.RUN automatically extracts:
# - File hashes (MD5, SHA1, SHA256)
# - IP addresses contacted
# - Domain names
# - URLs accessed
# - Mutex names
# - Registry keys
# - File paths

Cuckoo Sandbox (Self-Hosted)

Cuckoo is the most popular open-source automated malware analysis system. For those interested in deeper system manipulation, understanding Windows shell techniques is often a prerequisite for advanced configuration.

Installation:

# Install dependencies (Ubuntu/Debian)
apt-get install python python-pip python-dev libffi-dev libssl-dev -y
apt-get install python-virtualenv python-setuptools -y
apt-get install libjpeg-dev zlib1g-dev swig -y

# Install MongoDB
apt-get install mongodb -y

# Install PostgreSQL
apt-get install postgresql libpq-dev -y

# Install Cuckoo
pip install -U pip setuptools
pip install -U cuckoo

Configuration:

# Initialize Cuckoo working directory
cuckoo init
cuckoo community

# Configure virtual machine
# Edit conf/virtualbox.conf or conf/vmware.conf
cuckoo machine --add vm1 --platform windows --ip 192.168.56.101

Submit Sample:

# Submit file for analysis
cuckoo submit malware.exe

# Submit with options
cuckoo submit malware.exe --package exe --timeout 300

# Submit URL
cuckoo submit --url http://malicious-site.com/payload.exe

# Submit with custom arguments
cuckoo submit malware.exe --options arguments="--install"

Analysis Retrieval:

# Start Cuckoo web interface
cuckoo web runserver 0.0.0.0:8000

# View results
# Navigate to http://localhost:8000

# Export JSON report
cuckoo report  --format json > analysis.json

Joe Sandbox Cloud

Joe Sandbox provides detailed behavior analysis with MITRE ATT&CK mapping.

Submission Methods:

# Web interface
# 1. Visit https://www.joesandbox.com
# 2. Upload sample (up to 40MB free tier)
# 3. Select analysis options
# 4. Submit

# API submission
curl -X POST "https://jbxcloud.joesecurity.org/api/v2/submission/new" \
  -F "sample=@malware.exe" \
  -F "apikey=YOUR_API_KEY"

Analysis Features:

  • Behavior graphs showing process interactions
  • MITRE ATT&CK technique mapping
  • Yara rule matching
  • Signature-based detection
  • Code execution tracing

Hybrid Analysis (CrowdStrike)

Hybrid Analysis combines static and dynamic analysis with threat intelligence.

Usage:

# 1. Visit https://hybrid-analysis.com
# 2. Submit file or hash
# 3. Select environment (Windows 7/10, Linux)
# 4. Choose analysis type (quick/full)
# 5. Review automated analysis

Key Features:

  • Automated IOC extraction
  • MITRE ATT&CK mapping
  • VirusTotal integration
  • Community threat intelligence
  • Historical analysis comparison

Triage by Hatching

Triage offers high-speed automated analysis with detailed behavioral reports.

Submission:

# Web interface at https://tria.ge

# API submission
curl -X POST "https://api.tria.ge/v0/samples" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -F "file=@malware.exe"

Analysis Output:

  • Complete behavior timeline
  • Network traffic analysis
  • Process tree with injections
  • Configuration extraction (for known families)
  • Automated family detection

Advanced Dynamic Analysis Techniques

API Call Monitoring

Capture Windows API calls to understand malware functionality.

API Monitor:

# Download API Monitor from rohitab.com
# Launch apimonitor-x64.exe

# Configuration:
# 1. Select processes to monitor
# 2. Choose API categories (File, Registry, Network, Process)
# 3. Start monitoring before executing malware
# 4. Observe real-time API calls

Frida for Dynamic Instrumentation:

# Install Frida
pip install frida-tools

# List processes
frida-ps

# Hook specific API calls
frida -p  -l hook_script.js

# Example hook script for CreateFileW
var CreateFileW = Module.findExportByName('kernel32.dll', 'CreateFileW');
Interceptor.attach(CreateFileW, {
  onEnter: function(args) {
    console.log('CreateFileW called: ' + Memory.readUtf16String(args[^0]));
  }
});

Memory Analysis

Extract running malware from memory for further analysis.

Process Memory Dump:

# Using Process Hacker/System Informer
# Right-click process -> Create dump file

# Using ProcDump
procdump.exe -ma  memory.dmp

# Dump all processes
procdump.exe -ma -w malware.exe output.dmp

Memory String Extraction:

# Extract strings from memory dump
strings -n 10 memory.dmp > memory_strings.txt

# Search for specific patterns
strings memory.dmp | findstr /i "http://"
strings memory.dmp | findstr /i "C:\\"
strings memory.dmp | findstr /i "cmd.exe"

Volatility Analysis:

# Install Volatility
pip install volatility3

# Analyze memory dump
vol.py -f memory.dmp windows.info
vol.py -f memory.dmp windows.pslist
vol.py -f memory.dmp windows.netscan
vol.py -f memory.dmp windows.malfind

Debugger-Based Analysis

Use debuggers to step through malware execution.

x64dbg/x32dbg:

# Launch debugger
x64dbg.exe malware.exe

# Set breakpoints on key APIs
bp CreateFileW
bp RegSetValueExW
bp InternetOpenW
bp CreateProcessW

# Run until breakpoint
# Examine registers and stack
# Step through instructions

WinDbg:

# Attach to running process
windbg.exe -p 

# Set breakpoints
bp kernel32!CreateFileW
bp ntdll!NtWriteFile

# Display call stack
k

# Examine memory
db

Behavior-Based Analysis

Focus on specific malicious behaviors during execution.

Persistence Mechanism Detection:

# Monitor registry Run keys
reg query HKCU\Software\Microsoft\Windows\CurrentVersion\Run /s
reg query HKLM\Software\Microsoft\Windows\CurrentVersion\Run /s

# Check scheduled tasks
schtasks /query /fo LIST /v | findstr /i "malware"

# Service creation
sc query | findstr /i "running"

# WMI event subscriptions
Get-WMIObject -Namespace root\Subscription -Class __EventFilter
Get-WMIObject -Namespace root\Subscription -Class __EventConsumer

Privilege Escalation Detection:

# Monitor token manipulation
# Use Process Hacker to view process token privileges

# Check for UAC bypass attempts
reg query HKCU\Software\Classes\ms-settings /s

# Monitor for exploit attempts
# Watch for unusual child processes from vulnerable services

Data Exfiltration Monitoring:

# Monitor outbound connections
netstat -ano 5 | findstr ESTABLISHED

# Check for large data transfers
# Review Wireshark for POST requests with large payloads
# Look for base64 encoded data in HTTP traffic

# DNS tunneling detection
# Analyze DNS queries for unusually long subdomains
# Monitor query frequency

Unpacking and Deobfuscation

Many malware samples are packed or obfuscated. Dynamic analysis helps reveal the true payload.

Automated Unpacking:

# Let malware unpack itself in memory
# Set breakpoint on common unpack points
bp VirtualAlloc
bp VirtualProtect
bp WriteProcessMemory

# Dump unpacked code from memory
# Use Process Hacker to dump modified memory regions

PE-sieve for Process Scanning:

# Download PE-sieve from hasherezade/pe-sieve
pe-sieve.exe /pid 

# Dump suspicious modules
pe-sieve.exe /pid  /dmp 1

IOC Extraction and Documentation

After completing dynamic analysis, extract actionable indicators of compromise.

Network IOCs:

# Extract from Wireshark/FakeNet logs
# - C2 server IPs
# - C2 domains
# - URL patterns
# - User-agents
# - HTTP headers

File System IOCs:

# Files created/modified
# - Full file paths
# - File hashes (MD5, SHA256)
# - File sizes
# - Creation timestamps

Registry IOCs:

# From Regshot comparison
# - Persistence keys
# - Configuration values
# - Mutex names

Behavioral IOCs:

# Process behaviors
# - Command line arguments
# - Process injection techniques
# - API call sequences
# - Network connection patterns

Documentation Format:

# Create structured IOC report
{
  "sample_hash": "abc123...",
  "execution_date": "2026-02-08",
  "family": "Detected malware family",
  "network_iocs": [
    {"type": "ip", "value": "192.168.1.1", "protocol": "HTTP"},
    {"type": "domain", "value": "malicious.com", "first_seen": "14:30:22"}
  ],
  "file_iocs": [
    {"path": "C:\\Users\\Public\\malware.exe", "hash": "def456..."}
  ],
  "registry_iocs": [
    {"key": "HKCU\\Software\\Microsoft\\Windows\\CurrentVersion\\Run", "value": "Malware"}
  ],
  "mitre_attack": ["T1055", "T1071", "T1547.001"]
}

Evasion Techniques and Counter-Measures

Advanced malware detects analysis environments and alters behavior. Understanding these techniques helps improve your sandbox.

VM Detection Bypass:

# Malware checks for:
# - VM artifacts (VMware tools, VirtualBox additions)
# - Registry keys indicating virtualization
# - Specific hardware identifiers

# Counter-measures:
# Hide VM artifacts
sc stop VBoxService
sc delete VBoxService

# Modify hardware identifiers
# Use VMCloak or similar tools to create realistic VMs

Sandbox Detection Bypass:

# Malware checks for:
# - Known sandbox usernames (e.g., "sandbox", "malware")
# - Limited execution time
# - Monitoring tools running

# Counter-measures:
# Use realistic system configuration
# Increase execution timeout
# Hide monitoring tools (use kernel drivers)

Timing-Based Evasion:

# Malware delays execution
# - Sleep for extended periods
# - Check system uptime
# - Wait for user interaction

# Counter-measures:
# Accelerate time in VM
# Simulate user activity
# Use extended analysis timeouts

Troubleshooting Common Issues

Malware Won't Execute:

# Check dependencies
# - Missing DLLs
# - .NET Framework version
# - Java runtime

# Try different execution methods
rundll32.exe malware.dll,EntryPoint
regsvr32.exe malware.dll

No Network Activity:

# Verify FakeNet is running
# Check firewall rules aren't blocking
# Ensure VM has network adapter enabled
# Try changing DNS settings to point to FakeNet host

Monitoring Tools Crash:

# Process Monitor buffer full
# Increase backing file size or apply stricter filters

# System Informer requires admin rights
# Run as administrator

# Insufficient disk space
# Clear temp files and increase VM disk

Detection and Mitigation

Understanding malware behavior helps implement effective defenses.

Endpoint Detection:

# Deploy EDR solutions that monitor:
# - Process creation chains
# - Unusual API call patterns
# - Memory manipulation attempts
# - Network anomalies

# Enable Windows Defender ATP
Set-MpPreference -DisableRealtimeMonitoring $false

Network Detection:

# Deploy IDS/IPS rules for:
# - Known C2 communication patterns
# - Data exfiltration attempts
# - Suspicious DNS queries

# Suricata rule example
alert http any any -> any any (msg:"Malware C2"; content:"GET"; http_method; content:"/gate.php"; http_uri; sid:1000001;)

Prevention Strategies:

# Application whitelisting
# Use AppLocker or Windows Defender Application Control

# Restrict script execution
Set-ExecutionPolicy Restricted

# Disable unnecessary protocols
# Block SMBv1, LLMNR, NetBIOS

Now you've got the complete dynamic analysis toolkit. Remember, combining dynamic analysis with the static analysis techniques from Part 1 gives you the full picture of malware behavior. Static tells you what it could do, dynamic shows you what it actually does. When analyzing sophisticated threats, you'll often iterate between both approaches - using static analysis to understand code sections that dynamic analysis revealed as suspicious, and using dynamic analysis to validate hypotheses from static analysis.

Enjoyed this guide? Share your thoughts below and tell us how you leverage Dynamic Malware Analysis in your projects!

Dynamic Malware Analysis, Malware Sandbox, Network Simulation, Behavioral Monitoring, Reverse Engineering, Cybersecurity Tools, Threat Intelligence

By:
Bhanu Namikaze

Bhanu Namikaze is an Penetration Tester, Red Teamer, Ethical Hacker, Blogger, Web Developer and a Mechanical Engineer. He Enjoys writing articles, Blogging, Debugging Errors and CTFs. Enjoy Learning; There is Nothing Like Absolute Defeat - Try and try until you Succeed.

No comments:

Post a Comment