DZ BANK AG is the head institute of one of the largest banks in Germany and they had a big problem. They needed to deploy and maintain X.509 certificates to thousands of Linux servers to meet regulatory compliance requirements and to improve security in an effective manner. They had an existing Microsoft Public Key Infrastructure (PKI) implementation of Active Directory Certificate Services (ADCS) which worked fine for Windows based systems, but lacked integration with other platforms like Linux. Revocent’s CertAccord Enterprise solution solved this problem resulting in meeting regulatory compliance requirements, improved security posture, lower costs, and freeing IT staff to invest in other value added services.
"Due to compliance requirements nearly all communication between IT assets have to be encrypted. The related X509 certificates have to be signed by a trusted issuing CA."
Dr Mario Lischka, Cyber Security Specialist, DZ BANK AG
The Challenge of Secure Scaling
DZ BANK AG, based in Frankfurt Germany, is the head institute of the DZ BANK Group with revenue of $5.1B (2021) and more than 5000 employees. They have a huge investment in thousands of servers, desktops, laptops, and network devices that easily exceed the number of employees. Based on their experience they knew that using manual processes to create, deploy, and integrate certificates on thousands of Linux servers would quickly spiral costs out of control, lead to security vulnerabilities, and cause service outages.
Like many enterprises they already had a major investment in Microsoft ADCS as their PKI.
Most solutions they looked at required throwing out that investment and starting from scratch with a new PKI platform. They selected Revocent’s CertAccord Enterprise because it allowed them to quickly add it to their existing Microsoft PKI. This saved huge amounts of time and money that would otherwise have to be spent to re-implement a new PKI platform.
Why Automate?
ADCS provides a number of important benefits, particularly around certification administration, including automated certificate provisioning and lifecycle management. Without automation, certificate management at scale can become a nightmare. But with ADCS, certificates can be set to automatically renew, allowing the use of short-lived certificates without concerns about unexpected expiration and gaps in coverage.
This is all well and good for companies that have a homogeneous Windows-only landscape. But the reality for most companies is that they have a much more diverse IT infrastructure that includes a cross-section of Linux distributions such as Red Hat Enterprise Linux, CentOS, Ubuntu, macOS, and Unix based systems. Such was the case at DZ Bank. As the number of non-Microsoft endpoints proliferated into the thousands, so too did the problems associated with managing and updating digital certificates. There was simply no easy method to manage the entire network using ADCS alone.
"During the planning phase we opted against a large key/identity management solution and were looking towards solutions which focus on certificate requests and automatic renewal primarily for our Unix machines"
Dr Mario Lischka, Cyber Security Specialist, DZ BANK AG
The Results Paid Off
DZ BANK’s implementation of CertAccord Enterprise was able to quickly and securely meet regulatory requirements of encrypting all IT communications. Their primary use of X.509 certificates was for web server HTTPS and application servers which use SSL/TLS to secure communications. CertAccord Enterprise was able to automate not only the initial creation of certificates but also the renewal of certificates prior to expiration.
"Faster and lean deployment of default certificates is now also possible for the Linux environment. Service outages due to outdated certificates are expected to further decrease."
Dr Mario Lischka, Cyber Security Specialist, DZ BANK AG
Time Savings = Real Money Saved
The time savings has been significant and is calculated by DZ BANK as saving 40 minutes per certificate vs the previous manual process. Calculating the savings is very straight-forward using a 2000 system environment, 1 certificate per system, and renewing certificates every 1 year. Here’s the breakdown:
Year 1 Saved 2,666 Hours
2000 systems * (1 new cert + 1 renewed cert ) * 40 minutes = 160,000 minutes (2,666 hours)
Year 2+ Saved 1,333 Hours
2000 systems * 1 renewed cert * 40 minutes = 80,000 minutes (1,333 hours)
Estimated Cost Savings in Germany
DZ BANK doesn’t disclose its staffing costs, but according to Salary Explorer the average salary in Germany for IT professionals is 6,240€ per month or about 39€ per hour. Using that figure we can calculate the estimated savings:
Year 1 Saved €103,974
2,666 hours * €39 /hour = €103,974 cost savings
Year 2+ Saved €51,987
1,333 hours * €39 /hour = €51,987 cost savings
Estimated Staff Cost Savings in US
We can use this model to calculate the savings if this were a US based company. According to Salary Explorer the average cost of an IT professional in the US is $12,917 per month or $81 per hour.
Year 1 Saved $215,945
2,666 hours * $81 /hour = $215,946 cost savings
Year 2+ Saved $107,973
1,333 hours * $81 /hour = $107,973 cost savings
Cost Savings Achieved
These basic administration savings are purely based on the time to create and renew certificates. DZ BANK is also saving on avoided service outages due to expired certificates. Such outages can easily dwarf the administrative savings outlined above.
Let’s also not forget the incalculable cost of a security intrusion and what that might cost. A malicious intrusion might take the form of financial theft, intellectual property (IP) theft, or a ransomware attack. All of these can lead to losses in the millions or even billions of dollars/euros. Intrusions are often based on the ability to read and intercept communications between users and machines. A strong PKI can mitigate this risk.
DZ BANK is now Crypto Agile. They can choose to reduce certificate validity from 12 months down to 3 months (or less) which would significantly improve their security posture with virtually no cost increase. They also are now positioned to quickly deploy new cryptography such as quantum resistant certificates when they become available. Such certificates can be automatically pushed to their endpoints and integrated with applications without any time consuming and expensive manual processes.
How Did DZ BANK Become Crypto Agile?
To implement CertAccord Enterprise DZ BANK used a staged approach common when deploying enterprise software. The process is typical of a highly available, production infrastructure with little tolerance for service outages. The IT team had to work with restraints such as long “quiet” periods during which changes to infrastructure and applications are not permitted due to critical and high volume processing times.
Other restraints included vetting by security experts to evaluate CertAccord Enterprise for any potential security risks and testing integrations with applications to prove proper operations.
Stage 1 Proof of Concept
During this stage CertAccord Enterprise was installed and configured in a test environment complete with its own Microsoft ADCS PKI which is separate from the production environment. This allowed for isolated testing of CertAccord Enterprise without fear of interfering with production systems.
The POC environment included a VM running Windows Server 2019 configured with CertAccord Enterprise server. This server was configured to use a Microsoft SQL Server on a separate VM in the test environment. This SQL Server was a multi-purpose DB server used for different applications and was not dedicated to CertAccord Enterprise.
The CertAccord Enterprise agent was installed on numerous end-points running Red Hat Enterprise Linux in the test environment and was used to test application integration.
During this stage Revocent worked closely with DZ BANK IT staff to quickly get CertAccord Enterprise operational and assist with common integration and deployment questions.
"The latest version of CertAccord Enterprise is fulfilling all major requirements. Thanks to the support by Revocent, the identified obstacles could be removed and missing features have been added."
Dr Mario Lischka, Cyber Security Specialist, DZ BANK AG
Stage 2 Initial Production Deployment
DZ BANK IT staff next setup CertAccord Enterprise in their production environment. A new VM was created and the CertAccord Enterprise server was installed and configured using the POC server as a template. A multi-use Microsoft SQL Server system was used to store the CertAccord Enterprise database.
The CertAccord Enterprise agent was installed on a small number of endpoints and registered with the production CertAccord Enterprise server. These endpoints were used to further test functionality, packaging to automatically distribute CertAccord Enterprise Agent, and application integrations.
Stage 3 Broad Endpoint Rollout
The CertAccord Enterprise agent was further deployed to a wide range of Linux systems. Typically this was done by application groups. Each type of application is often deployed to similarly configured Linux systems. Once an application was identified for migration to CertAccord, the IT staff would create a CertAccord Enterprise configuration for that application and deploy it to the proper endpoints which provided the application.
Over time CertAccord Enterprise was integrated into multiple applications and processes. Most of these previously used manual processes for certificate management while others didn’t require certificates under previous regulatory requirements.
Conclusions
"Faster and lean deployment of default certificates is now also possible for the Linux environment. Service outages due to outdated certificates are expected to further decrease."
Dr Mario Lischka, Cyber Security Specialist, DZ BANK AG
By implementing CertAccord Enterprise, DZ BANK AG has saved more than 2,666 hours of IT staffing time just to begin with and will see significant time savings over time that can now be allocated to other value added projects and services.
DZ BANK is now positioned to quickly add large numbers of Linux systems and applications requiring X.509 certificates for HTTPS, SSL/TLS, and many other certificate based systems. By automating Certificate Lifecycle Management (CLM) with CertAccord Enterprise they have positioned themselves to be crypto agile and ready to quickly respond to future challenges such as post-quantum cryptography and other compromises in cryptography.