Anomaly Detection Without Labels

There are no labels. Nobody wrote down when the line was misbehaving, which is why they are asking for a detector in the first place, and it means there is no ROC curve to draw, no threshold to tune, and no way to score one method against another on accuracy.

The obvious first system is a limit on each sensor: alert when anything moves more than three standard deviations from its mean.

The reason is in the brief. A worn bearing raises vibration and motor temperature and current together, each by an amount that is unremarkable on its own. Nothing goes out of range. A filler valve drifts while the control loop compensates, so fill volume rises as valve position falls and both stay inside their limits. The information is entirely in the combination, which is what a multivariate detector is for.

Three standard methods with three different ideas about what "unusual" means. Isolation forest scores a point by how few random splits isolate it. Local outlier factor compares a point's local density with its neighbors', so it is relative. Robust Mahalanobis distance fits a contamination-resistant covariance and measures distance in units of it, which is the one that directly notices when correlated sensors stop agreeing.

Give each of them exactly 120 alerts, which is the technician's month.

Two machine faults and a hundred and two data problems. Ninety of the isolation forest's alerts are the air pressure transmitter reporting in psi after its firmware update, so the technician spends the month confirming, ninety separate times, that a firmware update happened on day 12. The robust Mahalanobis distance does worse still: zero faults out of 120.

Three well-regarded detectors, the same data, the same budget, agreeing on four readings out of a hundred and twenty.

With labels this would be easy to adjudicate. Without them there is no basis for preferring one list to another, which is uncomfortable and is the honest position. It also suggests a use for the disagreement: if they overlap this little, the readings all of them dislike are unusual in more than one sense, and section 6 comes back to that.

The engineering log lists four things that happened to the plant during the window and none of them were faults. Every one is visible in the data.

Refit the isolation forest on the repaired data and it finds five faults instead of two. Then look at what it is flagging instead.

The detector is not wrong. A stopped line genuinely is unusual: speed near zero, the motor cooling toward ambient, vibration down to a background hum. It is a completely different operating mode from a line running at 448 bottles a minute, and it is not a fault, and nobody needs to be told about it.

Data repaired and restricted to the 7,529 readings with the line running, of which 907 are during a fault.

From two useful alerts in a month to a hundred and two. The ordering of the three has also reversed: the robust Mahalanobis distance was the worst on the raw export and is by a distance the best here. That is not luck. The faults in this dataset are correlated sensors ceasing to agree, and a Mahalanobis distance measures exactly that. Once the two things breaking the covariance estimate are gone, it has the right tool for the job.

There is no validation set, so the threshold cannot be optimized. It can be chosen, and the thing to choose it against is the technician's capacity. First the metric has to change: maintenance does not care about individual five-minute readings, it cares about being told once that an episode is under way. There are 23 distinct fault episodes in the running data.

That table is the conversation to have with the maintenance manager, and it requires nobody to understand the algorithm. One alert a day is right every time and misses seven episodes in a month. Eight a day catches all twenty-three and is wrong nearly a quarter of the time. Four a day, which is what they said they could handle, gets twenty of twenty-three at 85 percent.

The threshold is an operations decision presented as a table, not a statistical one presented as a number.

Section 3 found the three detectors overlapping on four alerts out of 120. Rather than picking a winner with no evidence, require agreement.

Two of three is the policy to ship. Ninety alerts over the month, three a day, comfortably inside the budget, 72 percent of them real, and it catches twenty-two of the twenty-three episodes. Requiring all three to agree is more accurate per alert and misses a third of the episodes, because the detectors are looking for different things and only the most blatant faults look wrong in all three ways at once.

It is also worth knowing which failure mode the system is weak on before it is switched on. At four alerts a day the best single detector catches 9 of 9 bearing-wear episodes and 8 of 8 valve drifts, and only 3 of 6 air leaks.

Every score so far used the answer key. In practice there is not one, and the question "how bad does a fault have to get before this notices" still needs an answer.

Injection answers it. Take the clean data, add a fault of known size and shape, and see whether the detector catches it. This uses no labels, it runs against production data, and it is the only sensitivity figure actually available.

That is a real answer to "how much warning can we expect", obtained without a single labeled fault. Run it monthly against live data and it doubles as a drift check: if the injected-fault detection rate falls, something about the line or the sensors has changed and the detector needs refitting.

• ✓Read the engineering log before the data. Every alert in the first month traced to something somebody already knew about, and it was written down.
• ✓Expect the first run to find your data problems. That is not a failed project, it is the first useful output, and it should be handed to whoever owns the historian.
• ✓Define normal operation explicitly. A stopped line is genuinely anomalous and genuinely uninteresting, and no algorithm can tell the difference for you.
• ✓Do not expect detectors to agree. Ten alerts in common out of 120, and with no labels there is no basis for preferring one list. Use the agreement instead.
• ✓Score episodes, not readings. Nobody wants to be told forty times about the same drifting valve.
• ✓Set the threshold from capacity and present it as a table. It has no statistical justification and does not need one.
• ✓Inject known faults to measure sensitivity, then keep doing it on a schedule as a drift monitor.
• ✓Write the false alarms back into the log. When an alert turns out to be a data problem, record it, so the next model does not have to rediscover it.