Battle Tested: How We Built and Master Our CI/CD Pipeline
In this article, we’re going to share our own experience of how we built and run our CI/CD pipeline, as well as some basic CI/CD (continuous integration/continuous delivery) concepts.
As a Hong Kong leading software company which needs to continue delivering tech solutions to our clients worldwide, we, Oursky, embraced DevOps practices to maximize the efficiency of software engineering for quite a long time.
This article aims to help DevOps or CI/CD newcomers to understand the basics, and share our practical experience about how we implemented our CI/CD pipeline that you can take reference. By the end of this article, we’ll walk through:
- Basics: What is CI/CD and how it is related to DevOps (you may skip this section if you already have a basic understanding of CI/CD)
- How CI/CD benefit to our development and business
- Our own example of CI/CD pipeline
- Suggestions about how to get started with CI/CD
- 7 best practices we adopted to optimize our CI/CD
Let’s begin.
Introduction
In the age of digital transformation, optimizing software development and delivery creates a path to success. The rise of DevOps shed light on how to automate things in software engineering. More and more software companies plan to embrace DevOps and implement CI/CD pipeline in order to accelerate product delivery. If you are undergoing digital transformation and are new to the DevOps or CI/CD concept, you may probably feel confused about these terms.
What is CI/CD in brief?
CI refers to continuous integration. It aims to reduce integration overhead by continuously integrating all works within a team to one shared repository. It is usually worked out by automated building processes.
CD refers to continuous delivery. Teams adopted CD will produce and release software in shorter cycles, and tends to automate and carry out testing and monitoring more frequently. CD aims to get the software ready for release at any time. When the code is proven to be deployable all the time, continuous deployment can be considered as the next step to automate the deployment process.
CI/CD are the fundamentals of DevOps culture. By building a successful pipeline with automation, a DevOps team is able to deliver product to end users rapidly at a fast pace.
What is the Difference Between CI/CD and DevOps?
You probably heard often that people mention CI/CD and DevOps together.
While DevOps (combination of “Development” and “Operations”) promotes a culture/philosophy of team collaboration and work transparency, CI/CD are the pillars that enable DevOps — CI/CD focuses on automation of building, testing and deployment by properly setting up a CI/CD pipeline with appropriate CI/CD tools.
Automation sounds good and efficient. However, comparing to the traditional process of software development, what actual benefits can CI/CD bring to your development and business?
How can CI/CD Benefit to Development and Business?
CI/CD is essential for the software development process. A successful CI/CD pipeline will bring you compelling benefits. To name but a few, CI/CD has helped us to make engineering easier in the following ways:
Benefits of CI/CD for Development
Fewer bugs
Continuous integration enforces merging codes into one mainline constantly, which enables testing to be done more frequently. Moreover, as developers are submitting codes in small batches, each batch of code contain smaller and atomic changes, it is easier for developers to review the code and spot problems early, resulting in fewer hidden bugs.
Make testing efficient
Continuous integration makes us able to conduct test more frequently and run specific tests for different regression changes. More testing also encourage a much bigger test coverage.
Reduce manual work
When building, testing and deployment processes are automated, we can spend less human effort on the tasks.
Benefits of CI/CD for Business
What makes CI/CD more attractive is, a well established CI/CD pipeline not only favors development, but also maximizes business potential:
Better product quality
Automated testing enables early bug/failure detection. It is easier to make quality control and ensure our product quality after the release.
Speed up time-to-market
When the whole development process is accelerated with CI/CD, more softwares can be released to production. Innovations can be delivered to the market and yield profit earlier.
Faster response to the market
With shorter time-to-market and frequent software updates, we can collect end users’ feedback earlier. By gathering more updated user feedback, our clients are able to iterate their products and make instant improvement to suit the market needs.
Before reaping these benefits, a well established CI/CD pipeline is required. You may have read enough technical articles about how to setup a CI/CD pipeline already, so we’re not going to dig into technical details here.
Instead, we would like to share our experience with you: how a software company runs CI/CD in the real world?
How Do We Run a CI/CD Pipeline in Daily Operations?
In a typical DevOps cycle, CI is usually integrated into coding, building and testing stage; while CD would be applied at deployment stage. Here’s how we run our CI/CD processes roughly:
Continuous Integration
Coding Stage
- We know many DevOps teams using git pre-push hook to automate push commits, but we don’t. We allow our developers enjoy freedom in his own workspace for PoC or WIP.
- All CI/CD commands are designed to be able to run locally, so that developers can check the code locally in their developer environments when necessary.
Building Stage
- When commits are pushed to the repository, it will trigger a build. For app development, we use App Center to build native apps. Other codes are built on Travis.
Testing Stage
- In some projects, we integrated with our DevSecOps security checker to enhance security of our software. By simply invoking the checker toolset in the CI pipeline, it will perform all necessary checks with different lint and static code analysis tools.
- We have extra scripts to run test cases. In addition to unit tests, we also have static code analytic tools to check the code health. It is an important step because codes with bad code health may not be maintainable in long term. For example, we check train wreck, cyclomatic complexity and bandit etc.
- Dependency vulnerabilities check is also conducted. (Yes we have this even before GitHub!)
- We adopt TSLint to check typescript code health and use ESLint to check general code health.
- For language-specific testing, we use tools like Golint, Pylint to conduct code analysis.
- We host our code on GitHub and connect with Travis CI. Travis CI will be triggered for code testing whenever there is a new pull request or change of master.
- After passing all tests, the CI server will notify about the integration results. We have integrated CI into Slack so we can receive notifications on Slack directly. Afterwards, either developer can fix bugs (if the test failed), or the code will be merged to the main repository (if it passes all tests).
Continuous Delivery
Monitoring Stage
- For operation and monitoring, we will utilize error-tracking tools (e.g. Sentry or Crashlytics) to provide bugs/crashes alerts.
- We use the portal of our own product, the serverless Skygear, to access server logs.
Deployment Stage
- We trigger deployment to development environment per push.
- For staging deployment, we enabled a chatbot that PM can decide when to deploy with a simple command.
Note: in case you’re interested in our deployment process, we also use Skygear to deploy. Uploading server code and deploying Lambda functions could be completed with just one simple command. Just type “skycli” and it’s done!
You may have noticed that lots of tools we are using are open source projects. What does it imply?
An efficient CI/CD pipeline is not necessarily costly.
We didn’t spend a fortune to set up our CI/CD pipeline, you don’t need to either. Don’t let cost stop you if you want to start adopting DevOps and CI/CD in your company. In fact, in our journey to CI/CD, planning and implementation are much more important.
How to Plan for Implementation of CI/CD?
Many newcomers to DevOps or CI/CD concept may ask:
“Do I need to hire a DevOps engineer if I want to adopt DevOps or CI/CD?”
Typically, a DevOps engineer should be able to master a wide scope of technologies and automation tools which help to maximize the positive outcomes of DevOps, also he/she should be good at collaborating different functional teams and facilitate communication between these teams.
It’s good to leverage the expertise of a DevOps engineer when adopting a new culture or workflow, especially if you are new to DevOps and not sure about where to start. However, in our experience, hiring a DevOps engineer is optional instead of a must.
The collaborative spirit in DevOps is to empower engineers to control the operations. Thanks to this spirit, it can close the gap between development and operations. In Oursky, we have our CTO and a senior engineer setting up the basic infrastructure of CI/CD, at the same time, we educate all developers to run and integrate DevOps and CI/CD in their work process themselves.
How We Adopted DevOps & CI/CD Without Hiring a DevOps Engineer
So instead of hiring a dedicated DevOps engineer, here is how we built the culture of DevOps in Oursky:
- Set up a good authoritative example to follow.
- Team members would know whom to ask at roadblock.
- Evangelist to promote the tools and software in Company weekly update.
When we started to implement CI/CD, the first thing to do was to collaborate different functional teams. If you finally decided to go for CI/CD, get your teams know the common goal and consolidate their opinions. Here are the steps we taken:
- Use a version control system if you don’t have one yet. For example, we use Git for version control and GitHub/ GitLab as codebase server.
- Have at least two configurations, one for development and one for production.
- Pick the appropriate CI/CD tools. Popular CI/CD tools include Jenkins, Bamboo, Travis CI, GitLab etc. You can pick those which fit your current development tools.
- Implement CI first. When CI is proven to be reliable, you’ll have a solid foundation for implementing CD at next step.
- Compare the velocity of a team before and after implementation of CI/CD. Check if CI/CD does improve your team efficiency. Besides, check if code quality improves too with your own metrics or indexes. All these help to indicate if your CI/CD pipeline is successful.
In fact, setting up a CI/CD pipeline is not technically difficult. The most difficult part is to keep it fast and reliable in long term. Below are some best practices we adopted to optimise our CI/CD:
7 Best Practices We Adopted to Optimise CI/CD
1. Build only once
The binary artifacts should be built once and only once. It is to ensure a clean environment in CI so that you can trust the build. If you try to rebuild binaries, it’s hard to make sure that the one you built and the one passing CI/in production are the same one.
2. Small and incremental commit to reduce branching
To keep CI efficient at testing and monitoring, all changes should be merged to the main repository and minimize branching. To do this, encourage developers to commit atomically and implement features incrementally. Educate them about how to make pull request that is easy for code review, and enforce them to merge as soon as possible.
3. Plan for test priority
Another way to empower CI testing is to better arrange the tests. Prioritize which test should go first. For example, you can arrange some smaller and quicker tests (e.g. unit test) at early stage.
In this way, you can detect and fix bugs earlier when your code fails before going through time-consuming tests. It also helps to allocate right resources for different tests. Hence, accelerate the whole testing process.
4. Version control everything including documentation, deployment and configurations
Version control (or source control) is one of the must-have technique in CI/CD. Developers should separate build artifacts by versioning in order to differentiate different versions as well as to track changes. Otherwise it would be painful if you mistakenly overwrite the document and cannot go back again.
5. Isolate your CI/CD pipeline for security
You definitely don’t want your codebase and internal data being exposed to the public. Since CI/CD has full access to your codebase and credentials, it is recommended to isolate your CI/CD pipeline and limit to private access in secure networks.
6. Only use CI/CD pipeline to deploy
CI/CD already serves as the best practices of testing and deployment. However, it could not safeguard your production if you keep using other channels for deployment and bypassing CI/CD.
For quality assurance, make CI/CD the single channel to deploy your code. Ensure everything is going through the CI/CD pipeline, hence you can make sure the codes in testing and production are exactly the same. You can also reduce human error and avoid deploying other versions not being tested by mistake.
In Oursky, though we enable developers to override the CI/CD pipeline, it’s limited to emergency cases only. Let’s say, Travis CI is crashed and the whole CI/CD pipeline is down. Except these special cases, CI/CD pipeline should be the only channel for deployment.
7. Consider using container technology for testing
If you are experienced in CI/CD, you may start to consider using containers like Docker for testing. Container technology is an amazing tool that can help you set up a testing environment quickly, at the same time, ensure the cleanliness of the environments.
In our CI/CD pipeline, we use Docker for automated tests to improve QA. You can take a look at our open-source Github repo with 100,000+ Docker image pulls if you’re interested.
One Step Closer to DevOps
CI/CD is a definitely one of the core capabilities of DevOps culture. It helps us and many software teams out there to improve efficiency and maximize productivity. Though CI/CD emphasizes on tooling mostly, it relies on a cultural mindset among the company to make it truly works at the end of the day.
CI/CD, or the whole DevOps concept are only theories until every team member implements the idea in every step during their development. — Rick Mak, CTO of Oursky
Welcome to leave your comments below on this topic! If you are planning to embrace DevOps or implement CI/CD for your company, talk to our consultant and we’d love to help by sharing our expertise with you!
Originally published at https://blog.oursky.com on August 19, 2019.
