From Development to Production Faster in a Remote First Team

We want to make remote working easier faster and more efficient than ever before. Giving a developer all of the benefits that a cloud based work environment should offer. If you answer yes to any of the following questions then you should seriously consider a cloud based work environment.

1. Do you want to work from anywhere?
2. Do you need to reduce the amount of effort it takes to get a change to production?
3. Do you want to have more confidence in your changes and deployments?
4. Do you want to make your workspace and dependency management consistent and visible so it can be source controlled?
5. Do you want to reduce your build and test times across all your developers and your continuous integration pipeline?
6. Do you want your ci/cd pipelines to work with you and make your developers lives simpler not more complex?

With Cosmo development tools you can solve all of these problems simply and effectively across your whole team or organisations. Taking a load off your engineers and making them more efficient at the same time.

The way this happens is everyone gets a workspace in the cloud that can be customised or consistent across your whole team or organisation. Making it easy for any new or experienced member of your team to sit down and be productive no matter how the dependencies or environment has changed since they last contributed. This workspace is available through a browser allowing your team to work from anywhere. The Cosmo dev solution helps make your environment consistent and allows you to work from anywhere.

But how do you make your build faster in the cloud if you don’t want to pay for each developer to have a hugely expensive virtual machine that they don’t use most of the time?

A remote execution system should be used to accomplish this. Examples of these systems include bazel, buildstream, goma server, recc and more. When you use a remote execution system to compile your code the actual compilation is remove from the developers workstation and spread across hundreds or thousands of cores in the cloud. This means you can get all the resources exactly when you want them and only pay for what you use. This is much better than say paying for a 8 core machine all the time and the actual compilation still being less than optimal. Remote execution allows you to remove that compromise. Your developer workstation only needs to have 2 cores but when they want to compile they can execute their builds on hundreds or thousands of cores making it seem like they are using a super computer. Meanwhile all you have to pay for is the exact amount of execution time you need. This can take builds down from minutes to seconds or hours to minutes. These resources can be shared across your whole dev team and your ci pipeline. Making every piece of your development workflow faster.

Another common piece of a remote execution system is the ability to cache results from these compilations as well. That way you only have to rebuild the pieces that have changed. This means there can be a shared cache between all your developers and ci for all your build and testing. Meaning you don’t even have to pay for the compute costs when some changes are made. Most of the unchanged code is simply retrieved from the cache making it faster and more cost efficient.

For example with all of this working together a new developer on say the bazel project could open a new browser window and connect to their workstation that is fully configured to build the project. Then building for the first time with that workstation the build does not execute anything and only retrieves artifacts from the cache as this version has already been built. Meaning all it’s artifacts are already stored, so no new compilation required. When the developer makes a small change to the project only the pieces of the project effected by the change are taken and built/tested across hundreds of cores providing feedback in a matter of seconds sometimes. If everything passes properly the small change is submitted and the same build and tests are verified in the pre-commit stage which can share a cache with the developers if needed. Then a post commit build and test can be done possibly using a different build cluster and cache so no pre-commit or developer changes could accidentally poison the cache and end up in production. This example can take your developer workflow to production down from ~50 min for a fresh build at each stage to ~5 min or less depending on your changes. This can allow your developers to land more changes in production and decrease the amount of time it takes takes developer to fix bugs or problems while still running your full build and test suite.

Live demo of bazel being built remotely from a small container in under 2 min with results streamed to a UI

With continuous integration tools like prow a team can set requirements for commits and when they pass the changes can automatically be merged removing another friction point from your developers workflow. Using tekton with prow also makes it easy to create and share build pipelines for each of these stages. That way build and test pipelines can be standardised or customised depending on the teams resources and requirements. This workflow makes asynchronous work for teams much more consistent as you won’t have to wait for the original author to come back online to get their commit merged if everything looks good.

Combining all of these pieces, a team could significantly increase their velocity and make their builds more consistent. Making the cosmo dev package of tools the best way to do that.

If you would like information about how all of this can be setup and managed for you email info@cosmodev.tech