# Network-flow anomaly triage

This Python project uses Scikit-learn's Isolation Forest to rank five-minute host windows for review. The public example runs on generated traffic records. It does not connect to a network or collect packet contents.

## Run the example

Use Python 3.12. The pinned dependencies match the environment used to generate the included output.

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python generate_fixture.py
python ids.py train
python ids.py evaluate
python -m unittest discover -s tests -v
```

On Windows, activate with `.venv\Scripts\activate`.

The train command writes a local `output/model.joblib` and its metadata. Evaluation writes `output/demo.json` for the portfolio interface and `output/scored-windows.csv` for inspection. A previously generated copy of those text outputs is included in `example-output/`. The model binary is intentionally generated locally. Only load a model file you created or trust, because joblib uses pickle deserialization.

## What enters the model

The CSV holds one non-overlapping five-minute window per row. `source_ip` is a documentation address, not a model feature.

| Feature | Definition |
| --- | --- |
| `connection_count` | Number of connections opened by the source during the window |
| `destination_port_count` | Distinct destination ports across those connections |
| `destination_host_count` | Distinct destination hosts across those connections |
| `mean_bytes_out` | Mean source-to-destination bytes per connection |
| `failed_auth_count` | Failed login events correlated to the source during the same window |

Failed login counts would require an authentication log join. NetFlow records alone cannot supply that feature. This example starts with the joined, aggregated CSV, rather than pretending to implement an exporter or authentication parser.

## Training and evaluation

The generator uses seed 41 to create 1,000 windows. The first 600 form the benign training baseline. The next 200 benign windows set the alert threshold. The final 200 are held out until evaluation.

Isolation Forest fits only the training rows. A fixed `log1p` transform reduces the spread of the count features without estimating statistics on the holdout. Scenario labels never enter the feature matrix. The cutoff is the 98th percentile of calibration scores, using NumPy's `higher` quantile method. It is chosen before scoring the test partition.

Scikit-learn returns lower `score_samples` values for more unusual observations. This project negates those values so that higher scores mean more unusual windows. A window is flagged only when its score is strictly greater than the cutoff. The score is not an attack probability.

The holdout includes synthetic port probes and authentication bursts. Low-rate probes overlap routine traffic, while a benign backup introduces a pattern absent from the training baseline. A useful review should surface both the missed probes and the backup's false alarms.

| Actual output on the included synthetic holdout | Windows |
| --- | ---: |
| Flagged synthetic attack scenarios | 40 |
| Benign windows flagged | 18 |
| Benign windows left unflagged | 122 |
| Synthetic attack scenarios missed | 20 |

These counts describe the generated fixture, not a deployed IDS. The model does not classify attack families, establish compromise, or demonstrate zero-day detection. A deployment would need a representative local baseline and a threshold calibrated to its review capacity.

## Inspect or adapt

`generate_fixture.py` defines the scenarios and writes the reproducible CSV. `ids.py` contains the input checks and the train/evaluate commands. `tests/test_ids.py` checks the temporal split, label exclusion and score orientation.

To try another cutoff, rerun training with `--quantile 0.95`, then evaluate. Comparing cutoffs on the same holdout is exploratory analysis. Keep a new untouched test period for an unbiased final estimate.

The input validator rejects duplicate IDs and overlapping or out-of-order windows. The saved model records the fixture SHA-256 so that the example's partitions cannot be silently swapped between training and evaluation. This command-line example deliberately does not accept uploaded models or perform online inference.

## References

- [Scikit-learn 1.8 IsolationForest API](https://scikit-learn.org/1.8/modules/generated/sklearn.ensemble.IsolationForest.html)
- [Scikit-learn model persistence](https://scikit-learn.org/stable/model_persistence.html)

## Data and provenance

All addresses and activity in `fixtures/flows.csv` are synthetic. Dates are fixture timestamps. `example-output/demo.json` records the dependency versions and the exact fixture hash used for its scores. No school, student, colleague or private network records are included.