We at Hydra develop billing systems for telecoms (Internet, TV and telephony service providers). We ship complex products and the process of building them has some specific aspects.
Highly specialized enterprise level instruments have to work 24/7/365 and billing is a critical one. Here’s our take at developing a complex billing system.
Don’t expect customers to have free time for providing you with detailed instructions. Businesses have tasks that need to be solved and problems that require fixing. Customers are not supposed to take into account all the planning difficulties, development issues and implementation outcomes. It’s our job to understand the situation at hand.
An example can be found in the world of telecom we’re engaged in. Different operators want different billing features. Every region has its own specifics. For instance, cities located in the tundra regions of Northern Siberia, don’t have wired communications and are forced to use satellites. This is why going online is so expensive there, and data price plans with time-of-day-dependent caps are still in place.
The lack of fiber links is common for some Central Asian countries such as Afghanistan and in Africa. The size of a subscriber base could also be important. Solutions for local providers differ from product for large ones.
It’s impossible to create a universal system but it’s possible to highlight directions where future users would want to see improvements. You’ll avoid many problems if you do that at the stage of planning. It’s important to make it easy to introduce the changes without too much damage or workarounds. This requires deep analysis prior to actual implementation.
In case of some serious features, we tend do spend up to 20 percent of our time communicating with clients, understanding their business task, studying common practice and developing functionality demands. If you limit the development only to things a certain client needs, it will be useless for others. And this inevitably brings losses to your company.
Forecasting future use cases of a product seems like a good thing to do, but there’re some risks involved. Extrapolation may result in bulky code and extra maintenance costs.
Clients have already got used to their in-house systems, which a typically far from perfect, and to workarounds to their common tasks that they may expect in a new product. Building on this (wrong) view of things undermines system’s architecture, makes it less stable and complicates support.
For example, several clients ask us to automate the process of connecting a new subscriber. They were used to how things went for years, so we’ve just played the ball. Almost a year later finally we’ve realized that this was a mistake: in such configuration the functionality turns out to be limited and the billing process too complicated. So we recover it by passing it on to a separate product and making it open-source.
Listening to your clients is necessary and important. However, you should approach their ideas critically and not be afraid to add some features to your no-no list. And then, turn down the clients who ask you to do whatever is in the list. Figuring out their true problems that always have roots in business domain, studying the experience of competitors and doing the right thing from the start will be much more helpful.
High-quality architecture decisions save time and resources for support but the truth is that it’s almost impossible to implement it without extra tuning. It could be far from easy to realize what exactly you need to fix, though.
An example of such a situation is working on the functionality of events in Hydra’s provisioning module. it could be a change of important parameters that demands certain actions from the system. For instance, you should disable access for a subscriber once his account balance hits zero. Obviously, in this case an event has to carry some subscriber data, like his ID and MAC address.
All of this seems right and logical during the planning, but then problems start to occur. An example of that is when under certain circumstances the system works not in a way anyone expected, as someone had changed the code of a service that had been used in a command enabling access to it. Should the events be re-created for all subscribers?
It turned out that we had set a time bomb in implementing the events. This module’s code has grown very complex, and provisioning problem was a frequent thing. Interestingly, it wasn’t becoming obvious right away – the events weren’t re-created until something happened to this very subscriber. And when it happened, we had to call the client to figure out the reasons. The developers were afraid to even touch that code in order not to break something else while fixing other things.
The idea that seemed bright in the beginning, demanded a total change of concept. And by change of concept we mean fully re-developing provisioning module from scratch.
We can’t say it was easy and pleasant, but we didn’t have other options. You have to be ready for this kind of situations at any time. Rewriting makes it possible to do everything right, but it also requires prep work and deep analysis of possible drawbacks.
It’s impossible to do everything right once and for all if you deal with complex systems such as billing. However, sticking to some simple rules reduces the number of possible problems and makes future work with the product easier.
This check-list will help you develop more sustainable complex products:
