Data Quality management is an integral part of RODIN’s ETL capabilities.

Whether you believe it or not, your data does contain errors -or at least you must assume that it might. Just because you don’t find errors today, doesn’t mean they won’t occur at all. Program changes can introduce bugs, data entry errors can occur and program crashes can corrupt data.

Poor data quality is one of the major causes of failure of business intelligence projects. As soon as your users find errors in the data, they start to distrust the data warehouse. This leads to reduced usage, and eventual failure.

In many cases RODIN provides automatic error management. If a join fails, by default the data is rejected and directed to a staging table instead. RODIN then writes an error log record indicating not only the type of error, but also the key values used for the failed join! 

Many other types of error (such as trying to move character data to a numeric column) are automatically handled too. You just need to build business rules to apply you own data quality rules. For example:

          Reject If  COST = Zero

          Reject If  TRANTYPE = ‘C’   and   SHIPQTY > ORDERQTY

RODIN will take care of the rest.  An error report at the end of the job will detail all of the errors. You can then take action to correct and re-process them. If the data still fails validation, you can try again!   Even when you’ve corrected the data and loaded it, the error logs remain in the metadata until you purge them. So if there’s ever a need, you can go back and review errors that occurred a year ago.