Introducing sevif

Our first user is an investigation engine that reads Windows event logs and checks claims against them. A claim like "this process was launched by that account" needs one of three answers: supported, contradicted, or not enough evidence. The next step of the workflow depends on which one it is. A supported claim moves on, a contradicted one gets flagged, and "not enough evidence" sends the engine to fetch another log.

A model's written assessment doesn't fit that. Somebody has to parse the text into one of the three answers, and the parsed answer carries no sense of how sure the model was. sevif returns the answer as a probability for each option.

You send some state and a few closed questions. Each question lists its possible answers up front. You get back a probability for every one of them. The name is short for "severity, if", because "how severe is this if it's true" is a question we ask constantly.

How it works

We show the model each question as a multiple choice prompt, with the options under the letters A, B, C, and read the probabilities it assigns to those letters. Nothing is generated, so the answer can't fall outside the options you declared.

Models favor some answer positions over others, so we ask each question several times with the options in rotated order and average the log-probabilities. Every answer records the model, the commit of its weights, the prompt version and the number of rotations used. A prompt that would exceed the input limit is refused, and never cut short.

The method comes from SemIf, Theo Lee's research project. We wrote our own implementation, we credit theirs, and we run their published benchmark rows through ours so the two can be compared on the same cases. Our request and response shapes match TypeSafe's, so a client written for their API works with ours.

Results

Reading one token instead of generating text is cheap, which is why we wanted a small model to do this. Whether a 4B model is accurate enough at security telemetry is a separate question, so we tested it.

The test is 1,306 closed questions over real Sysmon and Windows Security logs from the public attack_data collection. Each label is computed from the events shown in the question, and a second pass re-derives every label from the rendered text alone. If one disagrees, the set is rejected. Cases are split by ATT&CK technique, so the test asks about attacks that no training example came from.

We fine-tuned a model on 3,851 cases from the training split, which took about two hours. The chart compares the stock model and the tuned one, both scored in the same job. Accuracy is the share of questions where the top-probability answer matches the computed label. Calibration error is the average gap between the probability the model states and how often it is right, taken over ten confidence bins and weighted by how many questions fall in each.

On the DFIR test, calibration error went from 0.130 to 0.004.

On a paired bootstrap the accuracy gain is separated from zero on all three sets, and the tuned model is not significantly worse on any. A calibration error of 0.004 means that on this test its stated confidence is off from its observed accuracy by less than half a point on average. It says nothing about your alerts. Our test is templated questions over real logs, and authored144 is only 144 rows in 36 groups, so the intervals there are wide.

The improvement that matters most to us is "insufficient". The stock model rarely says the evidence doesn't settle a question. When the event that decides it is missing, it picks an answer anyway. Here is one such case from the saved predictions (id 4b0bca8bc8075b976b56, an evidence claim where the launch record that would settle it was withheld):

The correct answer is "insufficient". The stock model puts 88 percent on "contradicted". An application reading that would flag a claim as false that it had no evidence about. On the test set the tuned model gets every "insufficient" case right, on both evidence claims and policy questions.

The training run that failed

Our first attempt made the model worse. We fine-tuned on a large corpus of soft targets, meaning probability distributions in place of single labels. With 1,000 examples, accuracy on authored144 dropped from 0.868 to 0.840 and calibration error nearly doubled, from 0.080 to 0.153. With 2,000 it was 0.199. The run cost about $1.70, so we stopped there and did not launch the full 20,000-example run.

The cause turned out to be simple. The average top probability in the corpus was 0.664, and the 2,000-example model's average top probability on the test set was 0.660. It had learned to hedge exactly as much as it was taught to. It also over-predicted "insufficient", 45 times against 36 true cases.

We changed the targets, not the loss. The version that worked trains on one-hot labels that we can verify, drawn from the logs. Our first model on that data scored 0.988 on the test set, which was suspiciously high, so we wrote probe cases in shapes the generator never produces. It failed them. It had learned that a missing launch record means "insufficient" and was not reading the claim. We rebuilt the training mix to include those shapes and kept the probe cases as a separate test.

Every figure in this post comes from a saved results file for a model pinned to an exact commit, and we're happy to share the files on request.

Generated text versus read probabilities

Try it

State can be a string, an object or an array of events. Questions are yes or no, one label from a set, or one label from an ordered scale.

{
  "state": {
    "alert": "Encoded PowerShell launched by a service account on a domain controller",
    "actor": "svc-backup"
  },
  "questions": {
    "verdict": {
      "type": "choice",
      "instructions": "Which reading of the state is best supported?",
      "options": {
        "benign": "Routine administrative activity",
        "suspicious": "Unusual and worth an analyst's look",
        "insufficient": "The state does not settle it"
      }
    },
    "severity": {
      "type": "score",
      "instructions": "How severe is the impact if the alert is true?",
      "levels": {
        "low": "Contained to one host",
        "med": "Lateral movement possible",
        "high": "Domain-wide compromise possible"
      }
    }
  }
}

You get a probability for each label, the top label, and the provenance record for each question. A typical application routes on those numbers. For example, it can act automatically above a probability threshold that you set from your own labelled cases, treat "insufficient" as a request for more data, and send everything else to a person. There is a Python client and a browser console with a playground.

Limits

sevif answers closed questions and cannot write a summary. A forced choice can still be wrong if none of your options is right. The probabilities are the model's own until you calibrate them, so fit a temperature on your own labelled cases and re-fit when the model, prompt version or rotation count changes. A temperature we fitted on the 144 authored rows did not generalize out of fold, so we don't ship one.

The tuned model is strongest on process launches, because that is what it was trained on.

What's next

We have built a training and test set for eight more kinds of evidence: network connections, DNS answers, file and registry writes, process handles, module loads and logon sessions. After that we want labels from real analyst adjudication, on questions their investigations actually asked.

If you're building security alert triage and want to try the tuned model, email hello@pentrasecurity.com and tell us which decisions your workflow needs to make.

FAQ