In September 2022, the Department of Labor’s (DOL) Inspector General announced new estimates for the amount of potentially fraudulent payments made for pandemic-era unemployment benefits: $45.6 billion. The bad actors are believed to be individuals who illegally claimed benefits in more than one state, successfully hid their identities, and used ineligible SSNs, such as those belonging to deceased individuals. 

This report illuminates the need for greater fraud detection in the public sector, particularly as crime rates continue to increase. PwC’s 2022 Global Economic Crime and Fraud Survey found that 46 percent of surveyed organizations have experienced fraud, corruption or other economic crimes in the last 24 months.

So, how can public sector organizations defend themselves against advancing fraud techniques that can cost governments and citizens millions of dollars? One way is through deploying machine learning (ML) models capable of identifying patterns, such as signs of fraudulent transactions or abnormalities, and immediately flagging the suspicious activity for analyst review. But to ensure the strongest, most accurate models, organizations need model monitoring. 

This article will explain how ML model monitoring could have helped the DOL–and can help other government organizations–gain visibility into their decisioning models to identify model drift and better prevent fraudulent activity from happening. 

What are ML Models?

AI technology use is growing across every industry, with data scientists and ML engineers creating algorithms to supplement and enhance human judgment in workplaces. This technology can not only lead to greater efficiency and lower operational costs, but it can also help prevent fraudulent activity.

An ML model is a type of algorithm that combs through massive volumes of data to find patterns or make predictions. For example, companies rely on ML models to analyze banking transactions to uncover anomalies and understand customer behaviors. Online retailers use it to train recommendation engines that display relevant product suggestions to new and returning shoppers. Customer service teams deploy chat bots that use a combination of natural language processing (NLP) and ML to interact intelligently with humans. ML models have tremendous potential for countless other applications that can drive substantial value to businesses. 

When it comes to predicting fraudulent activity, teams can deploy ML models in various ways. There are models that can detect when email accounts are created and flag them for potentially suspicious or fraudulent behavior. Organizations can also build models that detect the location of a user’s transaction to identify potential foul play if it occurs thousands of miles away from a home address. 

The Importance of Model Monitoring

Government organizations need to monitor the performance of these ML models because they inevitably decay over time as real-world input changes data. This is commonly known as data drift, which occurs due to changes in the environment. It is critical to ensure that the underlying ML models are making accurate predictions, are robust to shifts in the data, are not relying on false features and are not discriminating against minority groups. Only with constant model monitoring can government leaders feel confident in their ML models decisions.

In addition to monitoring for data drift, organizations that use ML models to predict or detect instances of fraud need to ensure that models understand and consider class imbalance. Class imbalance refers to a classification data set with skewed class proportions. In other words, class imbalance occurs when training data contains information that is disproportionately represented. Fraud detection models need to uncover drift in minority classes – information that makes up a smaller proportion of the dataset – since fraudulent events happen sporadically. The inability to identify unusual drift in minority classes can cost organizations enormous sums of money. Consider the DOL’s recent findings: tens of billions of dollars of potentially fraudulent payments in a few short years. With AI and model monitoring in place, the DOL would’ve been able to better detect unemployment payment errors and reduce significant costs.

Monitoring models with imbalanced datasets to detect even the slightest data change helps government entities avoid malicious intent before it impacts organizations and taxpayers. Knowing how to work with class imbalance and monitor drift across unbalanced, ever-changing data is the key to maintaining operational integrity in the big data era. 

Fraudulent behavior is a major concern for government agencies, not just corporate enterprises. Attacks on the public sector are becoming more sophisticated and commonplace all the time. Fortunately, with advancements in AI technology, organizations can now create ML models to detect potentially fraudulent activity across wide-scale government operations. But for these models to be successful, they must be monitored continuously. Monitoring ML models can help identify data drift, no matter how big or small, allowing organizations to avoid malicious activity and save billions of dollars. For government leaders, developing such ML capabilities is not only valuable, but paramount.

The author, Krishna Gade, is CEO and co-founder, Fiddler AI.