Hiren Patel

▪ Started as an Intern on the Release team at Tesla in 2020

▪ We had over 1000s of jobs on our Jenkins helping to build, deploy, and automate different functionalities for over 1000s of repositories

▪ Each job and repository had their own uniquely configured CICD pipeline with hard-coded values in the Jenkins job configuration itself or the Jenkinsfile in the repo

▪ The Jenkinsfile had all the information for building and deploying the repo including what servers were used and the commands ran

▪ When there were failures, developers would reach out to our team for help

▪ So, all our engineers were swamped with debugging build and deployment failures

▪ On-boarding new engineers was very difficult due to the variety of CICD processes we had

▪ Scaling was impossible since we had so many different CICD processes

▪ We had exposed server and command information in the Jenkinsfile to the developers

▪ On-boarding new applications took a lot of time and repeated code

▪ We had no standard practices when it came to our CICD

▪ We rebuilt for every environment deployment leading to repeated builds of the same code

▪ Resources kept being overused and causing random failures

▪ Network commands were a single try attempt and would fail for unknown reasons on occasions

▪ Create a centralized library in charge of all builds and deployments at Tesla

▪ Use best design patterns like functional and dynamic programming to create this library

▪ Limit redundant and repeated code

▪ Centralize all unique information such as framework and deployment servers needed to build and deploy code in a central database

▪ Make the pipeline fault-tolerant with network and failure retry logic

▪ Make builds and deployment platform agnostic

▪ Reduce the time and memory needed to build and deploy

▪ Containerize everything for consistent results

▪ Decouple builds from deployment

▪ Going from ideation to an incredible end product takes time and iterative improvements

▪ Getting end-user feedback can help save a lot of ideation and implementation time and energy because you know what is wanted and to what extent

▪ It is important to have a clear understanding of how much work is left so you can plan how much work to space out

▪ It is important to delegate tasks instead of worrying about doing it yourself because you want things to be perfect

▪ Don't repeat code. If those red flags go off in your head when you're writing code that you seem to be copy-pasting a lot, find a better long-term solution where you aren't repeating code

▪ Things will never be perfect. Requirements will always change and new requirements will always be added. Don't focus on making it perfect, focus on making it easy to add/remove/edit features later. Functionalize everything.

▪ Be patient. Working with multiple developers can take time.

▪ Groovy as the function programing language which performed functions like

  ▪ API calls

  ▪ Stored all the commands in classes

  ▪ Performed Batch and Powershell commands on Windows

  ▪ Performed Shell and Kubectl commands on Linux

  ▪ Worked with Jenkins to trigger downstream pipelines

  ▪ Notify the correct team depending on the failure

  ▪ Validate all the expected files exist

  ▪ Create and modify services as needed

▪ Developed a new CICD pipeline that completely automated Tesla's build and deployment process, reducing overall deployment time by 67%

▪ Scaled Tesla into 2 new countries and 2 new regions within the US using the new CICD pipeline

▪ Automated the deployment of 85% of the Linux and Windows hosted applications at Tesla using Jenkins, Groovy, Powershell, and Shell

▪ Enabled developers to execute self-service deployments with the new CICD pipeline which reduced manual releases by more than 50%

▪ Add other organization's repositories in Tesla to this pipeline

▪ Continue adding new functionality requested by developers

▪ Add long-term solutions to recurring problems