Integration: Data Transformations

Introduction

One of the key design questions when working with any database is whether the data is usable in the raw format, or if it needs to be scrubbed or transformed in any way.

Data scrubbing

Data may need to be scrubbed if your system does not ensure clean, consistent data. A classic example of this is a name field with half a dozen variations on a single name. Thomas Stearns Eliot might show up in a database as:

  • T.S. Eliot
  • TS Eliot
  • Tom Eliot
  • T.S. Elliott
  • Eliot, T.S.
  • Eliot, TS

Data transformation

Data transformation is the process of converting data to more useful information. Some examples include:

  • Calculating a cycle time based on the two time stamps in the data.
  • Converting Kilograms per day into pounds per hour.