Fraud System: Update 2

Progress has slowed a bit on the fraud management system… other priorities have come up over the last month or so but here’s a new walk though.  The backend hasn’t changed much.  What has changed is the hardware requirements.  When I started this project I had no idea how much processing power or space or IO was going to be required so I built the system in such a way that it could be easily scaled at several points.  As it turns out.. for this size of a network (about 250k customers) that was unnecessary.

What I Do: Fraud Detection System Initial Overview

I’ve been working on a big new project since just before the new year and it’s starting to take shape and generate useful results.   I can’t give away too many details on how exactly it works but I wanted to share this with some of you who are also working in telecom.  I was asked to develop a real-time system to identify toll fraud that would work for our entire voip carrier network that currently originates calls from 19 different countries for both residential, SMB, and wireless.  For those who don’t know.. I spent a year working for another telecom software company helping to run and debug a call mediation and rating platform for a tier2 carrier.  This experience was useful in that I was able to quickly develop a scalable, distributed processing framework while avoiding the cumbersome overhead I’ve observed in other systems.  Continue after the jump for more details…

What I do: Dynamic Daily Table Partitions With Postgres

As part of a new and fairly large project I have a need to partition a few postgres tables and have a rolling daily window.  That is.. I want to organize data by a timestamp storing each day in its own partition and maintain 90 days of historical data.  Doing this is possible in Postgresql but it’s not pretty or very clean to set it up.  To simplify the process I wrote this perl script that (when run daily) will pre-create a certain number of empty partitions into the future and remove the oldest partitions from your window.