Subscribe to mailing list

3x Faster: What We Learned from Speeding Up Hydra Launch in a 300,000-Subscriber Network

Pavel Vinogradov, 29 September 2016

We’ve explained why implementing complex products like billing systems on your own is not a good idea. It’s often easier to avoid mistakes by letting the professionals who developed the system do it.

During the years working on the Hydra billing system for service providers, we’ve done over 100 launches and were convinced we had optimized everything — but there’s always room for improvement.

Reality has taught us to be faster, more reactive and better at improving. This is what we learned when we had to complete a project for a relatively big customer in record time. Today, we’ll share this insight with you.

The Redenomination Factor

We’d already launched Hydra in bigger companies before, so we didn’t panic when Cosmos TV in Belarus (subsidiary of Russian AKADO Telecom) knocked at our door on New Year’s Eve. Cosmos TV is an Internet and TV services provider for 300,000 clients in the capital, Minsk, and its suburbs. On that fateful December day, the client told us they wanted to transition to Hydra by May 2016.

For us, this equated to about 3 months of working days to settle every legal and organizational issue when, from experience, this scale of a project would usually take up to 1.5 years. They were giving us just less than 100 working days to:

After long negotiation, the client agreed to move the deadline to July — but no further. They argued that on July 1, the Belarusian national currency would be redenominated. We thought this reason a bit, well, strange until we understood that it would be very hard for Cosmos to transfer its billing architecture into a new currency.

“Redenomination was a critical factor pushing us to install a new billing system ASAP. As you might guess, it’s easier to carry out redenomination on a new system, than to do it on the old one and then transfer the data. That’s why not completing the project on time was not an option — frankly, we didn’t even have a plan B.”

— Andrei Zhelyabovsky, IT department director at Cosmos TV

Clearly, the company just didn’t have a choice — so we got to work. With our newly extended deadline, we took on the job, even though we knew it would be a project of epic proportions.  

Need for Speed: Problems and Fixes

We now knew the rules of the game, but it was obvious that everything wouldn’t be completely finished before the deadline.

That’s why we optimized the work plan, highlighting critical tasks, including data migration, setting up the Payment Gateway, Provisioning, RADIUS server and subscriber accounts, and those we could solve after the launch.

It became clear right away that we’d have to be more flexible than usual. Generally, we’d work on projects using our own project management system. That made clients more disciplined and allowed us to see where things weren’t working out as planned.

It also trimmed down the number of “email me a document please” requests to a minimum, limiting the communication tools we used. This mode of communication allows our engineers to work more efficiently, but it’s not always convenient for individual clients.   

We had to start using Skype to speed things up, which inevitably led to some trivial topic discussions during conference calls. The implementation process was a source of many complicated questions, each one of which required lengthy discussion. A project management system or emails would have made it impossible to meet our deadline.

Shortly after we started, there were surprises that forced us to think and act outside of the box.

For instance, we usually start by getting the client’s’ infrastructure requirements to make them work within the billing system. When the hardware is available, we start active work. In this case, the servers ordered for Hydra would only be delivered in a couple of months — a serious delay we couldn’t afford. So we swiftly deployed virtual machines to create the implementation environment and transfer the config to the client’s active server.   

Also, we had to change how we run projects: our engineers are usually working on several projects at one time, but not with Cosmos TV. Instead, we selected a handful of engineers that would just focus on installing Hydra for our client. As we progressed, we learned that our engineers and our client’s IT team didn’t always speak the same language. We mean, literally: they often understood the same concepts differently, leading to confusion and delays. When it became clear that it wasn’t working, we agreed that the client’s engineers would join us in the office for a week so they could work closely with our team.

“Hydra’s philosophy was completely different from what we’d faced before. It was hard for us to understand how data from our old billing system could be represented in the new one. It was mind-bending: concepts that used to mean one thing now had another name and were executed in a different manner. Skype and email communication couldn’t fix this problem. Latera staff couldn’t come to us, so we came to them.”

— Andrei Zhelyabovsky, IT department director at Cosmos TV

This turned out to be the single most important decision that shaped the success of the project. We completed an amount of work that could take 2–3 months in just a couple of days.

By July, we’d completed about 80% of the project. This was enough to launch Hydra in its basic mode, without automating the business processes in Hydra OMS, managing job orders in Planado and other useful stuff. From July 1–4, our engineers worked round the clock, and we managed to launch the main billing processes of Hydra.   

“We also benefited from the fact that, in early July, banks all over the country were closed. We warned our subscribers that we couldn’t accept payments on July 1–4 and asked them to pay in advance. This gave us extra time to transfer all the processes from the old billing software to the new one.”

— Andrei Zhelyabovsky, IT department director at Cosmos TV

Technical Details

With the organizational issues solved, let’s have a look at the technical details. Here’s the outline of the overall operation scheme of the CSP:

Subscribers connect to the network via DOCSIS 2.0/3.0 technology or Metro Ethernet. They used MiktoTik and Cisco equipment as BRAS. The implementation didn’t have any unique features except that Hydra had to return the file address with DHCP configuration to DOCSIS modems and reboot if the IP address changed. In this case, customer equipment was authorized on Cisco BRAS, using IP address, or on MikroTik, using MAC addresses. To control DOCSIS equipment, we set up events for disconnecting PPP sessions and rebooting CPE in Hydra’s provisioning.

To manage Irdeto CAS, the client’s engineers provided us with a special script containing two input parameters: card ID and a list of TV channel bundles clients were subscribed to. If the second parameter was empty, all bundles would be disconnected.  

Another script was responsible for interaction with external online payment gateways. It executed payments directly through Hydra API skipping Hydra’s own payment gateway. The only thing we had to do was create a special module that accepted parameters in the required format and then called the core procedures.

Migration also brought about some difficulties. According to our scheme, the equipment was represented by a modem and a component on the equipment connected to the modem port. The problem was that the instrument we’d developed to automate migration (as seen above) could only migrate one component of the equipment, so we had to create extra scripts expanding its functionality.

We were faced with insufficient performance of the migration system for Cosmos’ specific data scheme. That’s why we optimized it using SQL optimizer hints, a couple of new indexes and some swear words. Thanks to that, we successfully completed more migration rounds and focused on the quality of data loaded to Hydra.    

A huge quantity of services provision schemes for the client’s customers made the process more complicated. Plenty of price plans, price specification documents and sample agreements resulted in rating and charging errors.

It was also hard to fix the migration of discounts given to subscribers as part of various loyalty programs. We had to migrate a discount from the old billing system to Hydra and include its validity period. Our instruments didn’t have a standard scheme for migrating discounts then, so we had to update them and introduce extra discount specification documents to Hydra.  

In Brief: What We Learned

The client told us that their previous billing system took just about as much time to install. But the contractor’s engineers had moved to Minsk and worked in Cosmos TV’s office from dawn to dusk for several months. And even this didn’t avoid problems: serious errors with erroneous charges for services kept popping up over the next few months.  

We had to carry out an exceptionally fast implementation without any room for error. Never had we been in a situation when we had to complete a project three times faster than usual. It was hard and tough, but it taught us to reconsider certain approaches regarding the implementation of projects. Here are our conclusions:

That’s it for today, thank you for your attention. Read our blog and follow us on Facebook!

Commented: 0