Subscribe to mailing list

How to: Billing system migration and ETL automation

Pavel Grachev, 08 August 2016

Service providers who, for whatever reason, decide to change their current billing solution, are to make a choice between migrating on their own or seeking help from third-party developers. The 1st case always supposes migration to be implemented in a half-manual mode – it is a single operation and there’s no point in automating it. As a result, errors tend to occur, deadlines shift, while resource costs are constantly increasing.

In fact billing developers implement dozens of such projects annually. This means that professionals, who are constantly engaged in migration, can automate even most complicated bottlenecks of this process.

One of such bottlenecks is the data migration from the previous billing to the new one. Today we will explain a couple of ways this challenging process can be automated.

The complexities of migration

When management of a CSP authorizes a new billing system, it expects the acquired solution not only to reduce future expenses, but to get rid of the issues on the spot as well. Moreover, nobody wants to deal with complications during the process of implementation — ideally, end users should receive no alters. This means that all subscribers should be provided with the same services after the end of the implementation process, while balance of the account must remain untouched, network addresses and contact details must be up-to-date.

Although it is easier said than done, because issues already pop up at the stage of data export from the former system. Even in case of billing systems with hundreds of installations it may turn out to be a real challenge. Normally only a few employees are aware of the implementation details, and if something occurs to these people (for instance, illness or they quit), the reconstruction of the entire picture will take a lot of time and efforts.

Eventually, companies have to choose: either to try to export the data from the previous billing somehow and introduce it into the new one manually, or undertake the development of solution for automation. Small CSPs can afford to embark on the 1st path – to allocate several low-paid employees who will manually transfer data from the previous billing to the new one via the interface.

But for larger corporations, this approach is unacceptable – it is too time-consuming and labor-intensive. In addition, billing information changes on a real-time basis, and these alterations must be taken into consideration too.

It is often very challenging for business to develop own data migration tool from scratch, so we have undertaken this mission ourselves.

Automation of data migration to Hydra Billing

Billing is a large-scale product which requires resorting to such approaches and decisions that are not always obvious. Therefore, we decided to abandon attempts to forge a tool which would migrate data from one alike system to another one in a single step. It is a lot more rational to divide this process in the stages.

We have developed a so-called “intermediate scheme” — a simplified format (set of tables) in which you need to export the data from the previous billing. Consequently, this data end up in Hydra. This approach greatly facilitates the export. Moreover, we promote automation of this process.

ETL tools (Extract, Transform, Load) — for example, Pentaho Data Integration or Talend Open Studio – can be used to automate the data export and migration tasks. We use Kettle (one of the modules of Pentaho Data Integration Package) to create and configure the ETL tasks and transformations. Then export is performed in a single mouse click, this significantly speeds up the migration.

ETL tools enable solving problems of data quality as well. Sometimes the previous system is not able to ensure conformity of data with standards (this is particularly important, for example, for addresses and telephone numbers) or forces to apply controversial decisions (to store IP addresses alongside in a taxpayer ID number field because there is no field for IP address in the source system). If the new billing data is stored in a structured way, it makes sense to bring the data into conformity with the standards or to the new storage scheme instead of “entailing” poor decisions.

For example, we have automated the conversion of manually entered addresses to the proper and consistent form using specialized cloud service. “Proper and consistent” is not always a piece of cake. For example this is how adresses in Nicaragua (we have customers there) are formed:

Instead of street names or numbers Nicaraguans use reference points from where they start describing a certain address. There are many known buildings, companies, churches, or other reference points, which are used to get a general idea where something is located. The next step is to state how many blocks away the particular address lies from the reference point.

An example could be From the Calvario Church, 1 block south, half a block east

[…]

There are instances, however, in which the reference points do not exist anymore! Before, they were well known points and were therefore used as reference points. Somewhere in the past the building or object was removed or destroyed. The addresses using this reference point now changed slightly but they kept the original object:

Donde fue Lacmiel, 2 cuadras arriba, 1/2 cuadra al sur (translated: where previously was located Lacmiel 2 blocks east, 1/2 block south).

As you can probably understand, finding out where some building was located can be tough!

pr_15

How it works

It is easier to illustrate the information above by an example. The following is a part of the transformation of the data from another billing system to the simplified intermediate Hydra scheme. Here we generate data to grant the subscribers access to network and personal account.

migration1
This is how the transformation of phone numbers to E.164 format looks like:

mig2

You can customize each step of the process in a flexible manner, for example as follows:

mig3

To sum up

The service of automated migration to Hydra resolves many problems. Service providers do not need to involve their developers since working with the ETL configuration is not about programming, and developers’ support is not required, while ease and speed of export makes it possible to detect data errors quickly and fix them, improving billing data quality.

With commercial billing solution everything is even better — once the project for automated migration is implemented, we can adjust this solution for new customers in the future. This allows us to offer simple and quick migration service.

Commented: 0
COMMENTS

LOG IN AND LEAVE A COMMENT

OR

↑ TO THE TOP