Case file · Sysmon-ML-2026Interactive investigationEsmaeili · Ghafarian — Univ. of North Georgia
$ load sysmon_dataset.csv ✓ 6,258 events · 3,236 benign · 3,022 burst $ train RandomForestClassifier() fitting done
Accuracy on the test set
0%
Misleading
Precision 100%Recall 100%F1 100%Errors 0

Why did this happen?

A ransomware detector trained on Windows Sysmon events just scored a perfect 100% on every metric — and the paper behind this experience proves it learned almost nothing. You have six questions between you and the answer.

Q1

Why was it 100%?

First move in any investigation: ask the suspect what it was paying attention to. Feature-importance analysis interrogates the Random Forest directly — which columns actually drove its predictions?

What the model relied on

ThreadId and ProcessId carried nearly all the predictive weight. Those are identifiers — labels the operating system assigns — not behaviour. The model matched IDs to classes the way you'd memorize answers, not learn material.

The “perfect” confusion matrix

Zero errors in both classes. In event-level Sysmon data, a flawless matrix is not a triumph — it's the first clue.

View as
Figure 14 from the paper: feature importance bar chart showing ThreadId and ProcessId dominating, EventId minimal.
Figure 14 (as published). Feature importance for the balanced event-level dataset — dominant reliance on identifier-based features (ThreadId, ProcessId), minimal contribution from EventId.
Figure 13 from the paper: confusion matrix and classification report showing 100% across all metrics.
Figure 13 (as published). Confusion matrix and classification report for the balanced event-level dataset — 100% accuracy, precision, recall and F1.
Exhibit A · defined in this work

Event-level leakage

The reliance of machine learning models on identifier-based or repetitive event features (e.g., ProcessId, ThreadId) that correlate with class labels but do not represent meaningful behavioural patterns.

So the “perfect” score rests on two ID columns. What happens if we take them away?
Q2

Remove the leak. Then what?

The experiment: delete ProcessId and ThreadId, retrain on the same balanced event-level data. Before you run it — commit to a prediction. That's how science works.

🧪
Your hypothesis · Experiment 1 of 3
With identifiers removed, Random Forest accuracy will…
Pick a hypothesis to unlock the run.

Accuracy without identifiers

100%Shortcut removed

Where the importance went

Confusion matrix · Random Forest

View as
Figure 15 from the paper: confusion matrix, classification report and feature importance after removing identifier features, showing large performance drop.
Figure 15 (as published). After removing ProcessId and ThreadId: significant degradation, 500+ benign instances misclassified, and importance shifting entirely to EventId (1.0).
What the data showed

Accuracy crashed to ~61% (RF), ~55% (LR), ~60% (SVM, XGBoost). And the model didn't start learning behaviour — it just grabbed the next shortcut, putting all of its weight on EventId (1.0). The perfection was never real.

The models can't tell ransomware from routine work without ID crutches. Why not? What's wrong with the events themselves?
Q3

Why can't events carry meaning?

Look at the crime scene itself. The dataset's counts are balanced — 3,236 benign vs 3,022 burst. But behaviour lives in time, and time tells a completely different story: one ransomware-like action exploded into thousands of near-identical file-creation events inside roughly 3 seconds.

Evidence · the full timeline, one dot per Sysmon event
Benign · spread across the captureBurst · ~3 seconds 0 / 6,258 events
View as
Figure 17 from the paper: temporal distribution of FileCreate events showing extreme concentration in a short interval.
Figure 17 (as published). Temporal distribution of FileCreate (Event ID 11) events — extreme concentration within a short interval.
The insight

Event frequency is not behavioural diversity. Treated as independent samples, thousands of copies of one action over-represent that pattern and hand the model repetition to memorize. Events are components of behaviours — so the representation should describe behaviours, not events.

The paper's fix: group events into fixed time windows and summarize each window. But what happens to 6,258 events when you do that?
Q4

Aggregate behaviour. What survives?

Behavioural aggregation groups events into 10-second windows, each described by summary features — total event count, number of unique event types. Predict the outcome before you watch it.

🧪
Your hypothesis · Experiment 2 of 3
After 10-second aggregation, the class split across windows will be…
Pick a hypothesis to unlock the run.
Aggregation window
Events in
6,258
Windows out
130
Benign windows
127
Malicious windows
3
Class balance after aggregation

View as
Figure 10 from the paper: aggregated dataset inspection output showing 130 windows with 127 benign and 3 malicious.
Figure 10 (as published). Aggregated dataset inspection: total behavioural windows and the resulting class distribution — the imbalance introduced by temporal aggregation.
Now the representation is honest — but the classes are 127 vs 3. Train on that. What's the accuracy?
Q5

Train on the collapsed data.

Random Forest, aggregated dataset, natural class distribution: 127 benign windows, 3 malicious. Last prediction before the twist.

🧪
Your hypothesis · Experiment 3 of 3
On the unbalanced aggregated dataset, accuracy will be…
Pick a hypothesis to unlock the run.
Misleading

The result

100%Majority-class trap
100%
Accuracy
0
Malicious found

The model predicted every single window as benign. With 127 of 130 windows in one class, that alone buys ~perfect accuracy. Evaluation even warned that only a single label appeared in the predictions.

Confusion matrix · all predictions benign

The 100% from the cold open and this 100% are two different failures — leakage there, imbalance here — with the same root cause: representation.

View as
Figure 16 from the paper: confusion matrix for the unbalanced aggregated dataset showing majority-class-only predictions.
Figure 16 (as published). Unbalanced aggregated dataset — the model predicts only the majority class: misleading 100% accuracy, zero malicious detection.
Two perfect scores, zero real learning. One question left: how do we get an honest number?
Q6

Balance it — and meet the truth.

Random up-sampling duplicates the minority class until both sides hold 127 windows. No new information is invented — it simply stops the majority class from drowning the signal. Press the button.

The final experiment.

Random Forest · balanced aggregated (10 s)

82%
Accuracy
1.00
Precision (mal.)
0.68
Recall (mal.)

82% — and it's the best number in this entire investigation. Zero false positives. Nine malicious windows genuinely missed. Real precision–recall trade-offs are what learning actually looks like. Window size barely matters: ~80% at 5 s, ~75% at 30 s.

An honest confusion matrix, at last

23 TN · 19 TP · 9 FN · 0 FP. Imperfect, informative, trustworthy.

View as
Figure 18 from the paper: confusion matrix and classification report for the balanced aggregated dataset showing 82% accuracy.
Figure 18 (as published). Balanced aggregated dataset — 82% accuracy, perfect precision for the malicious class, recall reduced by missed detections.
The verdict — yours before it was ours

You just concluded what the paper concludes: representation governs performance. Not the algorithm.

You watched a perfect score collapse when two ID columns vanished, watched 3,022 events compress into 3 windows, and watched imbalance manufacture a second fake 100%. Nobody told you the conclusion — the evidence did. The paper's cross-model experiments confirm it below.

–/3
Your hypotheses versus the data. Run the three experiments above to complete your case file.

Corroboration · five models, four representations — tap any cell

Tip: select a cell to see what was really behind that number.
Inflated / misleading Realistic difficulty Weak detection
Bonus reel

Replay the research.

The investigation you just ran took the authors months. Watch the journey unfold the way it actually happened — including the celebration that turned out to be premature.

Step 1 of 12
Epilogue · what this changes

Three findings worth carrying out of the case file.

Finding 1

Event-level modelling misleads

Feature leakage and structural bias let models bind identifiers and repetition to labels — near-perfect metrics with no real detection capability.

Finding 2

Aggregation is honest — but collapses

Behavioural windows force learning over time, yet compress high-frequency attacks into a handful of samples: temporal collapse and severe imbalance.

Finding 3

Consistent across all five algorithms

Random Forest, Logistic Regression, SVM, XGBoost, Isolation Forest — same pattern everywhere. Data representation, not model selection, is the dominant factor.

Open leads for future investigators

Adaptive or variable-length windows to preserve temporal structure without crushing bursts; sequence models (RNNs, transformers) and graph-based approaches linking processes, files and network events; larger real-world attack datasets; and continued work on leakage reduction and interpretability.