Building Uber’s Multi-Cloud Secrets Management Platform to Enhance Security

Sharing a fantastic article by the Uber team on how they built a Multi-Cloud Secrets Management Platform - amazing job by the Uber team including Hasibul Haque, Arun Kumar R, Sung Hon Wu, and Gaurav Bansal.

Read the full article here.

Uber runs over 5,000 microservices, 5,000 databases, and over 500,000 analytical jobs per day to support millions of people worldwide using our apps. Over 150,000 secrets facilitate authentication among these large, distributed ecosystems with multiple stakeholders. This also includes over 400 third-party vendor integrations and 400 SaaS applications.

Uber developed the Multi-Cloud Secrets Management Platform to enhance security across its vast infrastructure, which includes thousands of microservices, databases, and third-party integrations. This platform was designed to address challenges such as secrets sprawl, inconsistent vault management, insecure sharing practices, and the complexities of rotating credentials.

Challenges Faced

1. Secrets Sprawl: Sensitive credentials were scattered across various codebases, configurations, and storage locations, increasing the risk of exposure.
2. Shadow IT Vaults: Different teams maintained their own secret storage solutions, leading to inefficiencies and security gaps.
3. Insecure Secret Sharing: Credentials were often shared manually or stored in unsecured locations, raising the likelihood of breaches.
4. Secret Rotation Complexity: Managing and rotating secrets across thousands of services and databases required automation to prevent security lapses.
5. Third-Party Integrations: Exchanging credentials securely with external vendors and SaaS applications posed additional security risks.

Approach Taken

Uber implemented a multi-layered strategy to strengthen security and streamline secrets management:

• Preventive Measures: A command-line tool blocks commits containing secrets before they reach repositories, reducing credential exposure.
• Remediation Strategies: Real-time scanning monitors code changes and conversations for exposed secrets, while scheduled scans audit repositories, filesystems, and containers.
• Automated Governance: The consolidation of multiple vaults into a unified governance layer improves security oversight.
• Secret Rotation: Credentials are automatically refreshed to maintain security integrity and minimize the risks associated with static secrets.
• Secretless Authentication: Uber is reducing reliance on traditional secrets by developing alternative authentication methods.

By implementing these security strategies, Uber enhances its ability to mitigate cyber threats, reduce human error, and automate credential management across its expansive infrastructure.