In this story, Ravid Brown, ALM manager at ECI Telecom, discusses the company's experiences in creating an innovative range of products in a highly complex development environment.
ECI Telecom may not be a 'household name', but its voice and data networking equipment and services support millions of voice calls and data communications daily across mission-critical networks worldwide. Customers include British Telecom, Bharti Airtel, China Power, Deutsche Telekom, France Telecom, Tata, Vimpelcom, and Vodafone, plus energy, government, defence, and transportation companies. Headquartered in Israel, ECI Telecom has offices throughout Europe, India, Africa, North America, Latin America, and Asia Pacific.
With an emphasis on R&D, ECI Telecom positions itself as a pioneer of optical packet networks and next-generation telecommunications. Its products and services cover network infrastructure solutions; smart grid, optical transport, and packet transport networks; multi-service provisioning platforms; next-generation access; and network management. The company also covers end-to-end program solutions, maintenance service packages, and professional services.
Ravid Brown, ALM manager, takes up the story:
With such a wide range of products and continued innovation, our development teams play a vital role within the organisation. Our 300 developers are divided into three main teams, but within those are smaller teams, each working on specific projects. Each product probably has a couple of main releases a year, so we are a fairly typical Waterfall style of development environment, although small fixes happen in between. Most of the developers are in one of several offices in Israel, but we also have contributors in China and India. It's important that we have good control and visibility wherever people are working.
We have a diverse environment with different IDEs and tools, across Java, .Net, C, C++, and some Web technologies. Although developers choose their own development tools, our role in the application lifecycle management (ALM) team is to support them with a common set of tools. These tools ensure consistency, quality, and efficient release of products while making it easier for developers to do their jobs.
The move towards off-the-shelf and automated tools
It's a constantly evolving and challenging environment, one that I and my team are dedicated to ensuring always has the best possible tools for the job in hand. For instance, we have built many of the ALM support systems and tools in-house, including creating our own build system. However, we are reviewing all our existing ALM products - particularly those that have not been updated recently - and wherever possible, we are researching and switching to 'off the shelf' tools. There are a couple of reasons for this step.
First, while we love writing code, maintaining systems built in-house takes up a lot of time and effort. That has been fine in the past, but as the company continues to grow, so will demands on the ALM team; using third-party tools helps us to manage our workload and enables us to concentrate on other tasks.
Second, third-party tools have evolved a great deal in the past few years and have functionality that will really benefit ECI Telecom. In particular, we are looking for ways to automate processes as much as possible, not only to help reduce the time it takes to complete tasks but also to ensure consistency and reduce manual intervention. While we want developers to have a working environment that they enjoy and give them the flexibility to be creative, we also want to set rules and standards that ensure the best quality code and avoid unnecessary traffic on the network.
We are currently moving to Jira for bug-tracking and Jenkins for our build system, but the first major transition in our upgrade programme has been the move to Perforce for our version control. We approached that in a very methodical way, looking at how we can achieve those goals of automation and control while also creating a version management environment that developers are happy to use.
Version control
As is the case in many companies, version control is an integral part of our development environment, ensuring that we have a 'single source of truth' for all code. If we need to check something, we can roll back to a particular instance in the development process to see what happened. Fast and reliable branching means that developers can explore ideas without affecting the code mainline.
We have many millions of source code files, equating to over 100 Gb on the version control server. We had outgrown our previous version management system. It had performed well enough over the years, but it was taking 17 minutes to synchronise files; that was far too long, especially when you consider the amount of times each day that a developer might be checking files in and out. That same process now takes two or three minutes, which means our developers have fewer reasons to take a coffee break! The entire repository is sitting on one server, and we anticipate 100 per cent return on investment on Perforce within two years of implementation.
Besides this improvement in productivity, there have been other benefits, particularly in automation. We previously had to create a large amount of custom code to automate simple tasks. We had to write a lot of proprietary queries and code but Perforce supports native SQL, so we've eliminated that extra step. In the old system, all the SQL queries went to the same server that a developer was working on for checking work in and out. When we tried to run reports, it affected the daily work of developers, so we only ran reports at night. Now, we can replicate all the data on another server, meaning we can run massive reports at any time, without disrupting the developers.
Another feature that has helped us is the trigger mechanism. Triggers allow extensions in the version control server, which enables our developers to add specific functionality or checking our developers need. In the old system, we had to include lots of guidelines for developers and it was hard to enforce those across 300 developers. Now, with triggers, we've been able to customise messages so that when a developer tries to do something, the code is automatically validated. Those polices can be very specific, right down to individual code branches.
Customisation
While having tools that include rich functionality and automate so many processes is great, we still wanted the ability to customise them to our own environment. Besides triggers, we have also added more merge options for users.
We also created what we call 'wrappers' - easy ways for all team members, not just developers, to access the version control system. We wanted everyone to be able to access and understand the information in Perforce and that includes project managers, testers, and support teams in the field. Unlike developers, these staff members are not familiar with computer code terms, so we have created new message windows that explain things. For example, consider the difference between two variations on the Website that a project manager is working on. The project manager does not know the code or branch reference, but if the build number is given, the wrapper tool makes it easy for us to look behind the scenes. We can then compare the changes and all the jobs associated with that project, make necessary changes to that file, and send it back to the project manager via email.
Next steps
We are still using our legacy bug-tracking system, but we are planning to move to Jira because it integrates with Perforce. We use Coverity for static source code analysis. It integrates with our build system and with Perforce, as does Jenkins, which we will use to automate our build systems. We are gradually moving towards an environment where we have a set of ALM tools that all work together.
Integration in itself is an advantage, but equally important are the efficiency benefits that the latest-generation tools give us: better use of everyone's time and reduced impact on the network while maintaining consistent quality across all development projects. For myself and the ALM team, this evolution has been an exciting time, helping us to move the development environment up a gear to support the company's continued growth.
