How to Build Your Development Stack in the Cloud from Scratch

Over the past couple of weeks, a number of friends approached me to ask how I had set up my website. I told them I’d created the site using Jekyll and hosted it on Github Pages, but I had to host my web apps on Heroku because Github Pages only serves static content, and I also needed a database so I had to connect the web apps to Firebase…

And it struck me how inefficient my development stack was. I started wondering if I could do any better, and so this post is the result of completely recreating this stack from scratch in the cloud. There are a lot of great tutorials out there for each of the topics I’ll be discussing, so this post does not intend to replicate any of those but instead provides an insightful collection of resources to hopefully give an idea of what a cloud computing setup entails.

Spoilt for Choice

In today’s open-source landscape, there are an unimaginable number of libraries and frameworks for each task. Want to create a website with minimal effort? Use a drag-and-drop site creator like Wix, Weebly, or Wordpress, or maybe purchase some hosting on GoDaddy or Namecheap and let them do all the work. What about a web application? Use one of the million frameworks like Meteor, Flask or Django. Then go ahead and host it on something like Heroku or PythonAnywhere. Need a backend for the app? Connect it to Firebase.

Building and hosting something simple on the web today has become at once simple and complex. Not only is it time consuming to explore all of these different services - creating accounts, setting them up, maintaing the services, it is also expensive. Hosting on Namecheap can cost nearly $40 a year, a backend on Firebase $25 a month, and application hosting on Heroku anything from $25 to $500 a month per instance. If you’re a hobbyist developer or a student trying to host your work online like me, it just doesn’t make any sense to pay that much.

Built for the Cloud

A wise programmer once said, “There is no cloud; it’s just someone else’s computer.”

We live in the age of cloud computing - every like, every tweet, every click. So why shouldn’t the applications we build? Cloud computing is driving down development costs through economies of scale. In fact, major providers of cloud services such as Amazon AWS and Google Cloud Platform are getting into bidding wars to remain competitive, which means lower prices for you and me. And the best part? The flexibility. Most cloud providers employ a pay-what-you-use business model, meaning that your business drives your cost.

Cloud computing is also instant. With a click of a button, you can scale your application across the world, adding cloud instances in New York, Bangalore or Singapore. The modular nature of cloud computing has encouraged businesses, startups, and even hobbyist developers, to make the switch. Need more processing power? Upgrade your cloud instance to one with a higher performance. Need more space? Connect a storage volume to your instance. And if you don’t need it anymore, simply turn it off.

I think that by now, I’ve convinced you why it makes so much sense to both move your development to the cloud, and also streamline it onto a single platform. For the remainder of this post, I hope to briefly describe the approach that I took to do this. This approach is by no means the best, and as mentioned, there are plenty of different tools and services to do the same thing - but I’ll try to make the case for the stack I used and why I thought it worked the best for my use case.

The Development Stack

1. Infrastructure as a Service (IaaS)

This is the very core of your stack, the single platform on which everything is built. It’s the cloud, i.e a server halfway across the world with an operating system installed on it (typically a Linux distribution like Ubuntu). There are plenty of different options available, like Amazon AWS, Google Cloud Platform (GCP) and DigitalOcean. AWS and GCP are extremely powerful, but only really necessary if you require any of their plethora of additional services like Google Cloud Dataproc, which allows you to set up Spark and Hadoop for data processing, or Amazon S3, which offers scalable storage in the cloud. DigitalOcean is one of the cheapest options, and has everything I need (along with a really nice user-friendly interface).

My Pick: DigitalOcean

• Cheap
  I am paying $5 a month for an instance with 512MB memory, 1 core processor, 20GB SSD disk and 1TB transfer.

• Quick
  DigitalOcean claims that you can ‘spin up a droplet and get root access to a compute instance in only 55 seconds’ (it’s true).

• User Friendly
  Their platform and UI is extremely user friendly, and they have great tutorials and documentation. You can pretty much find a tutorial for anything that you’re stuck on.

Get Started: Initial Server Setup with Ubuntu 16.04

2. Domain Name Registrar

This is where you register your domain name - in my case, sudeep.co. Again, there are plenty of options but it really doesn’t matter where you get your domain from. Just compare prices across registrars. I got mine with Namecheap simply because I had registered another domain with them before, so I was familiar with the interface. But using Namecheap made me realize how much I appreciated their support.

My Pick: NameCheap

• Excellent Support
  One of the best things about Namecheap is their live chat. If I’m ever stuck on a hosting or DNS issue, I can instantly chat with one of their representatives to get live support instead of searching on StackOverflow for hours.

Get Started: Register a New Domain

3. Web Server

Once you have a cloud instance and domain, you need a web server to load your content. Web servers control how users access the hosted files using HTTP, which is the protocol that your browser uses to view web pages. With a web server, you can point different web applications to different ports. This makes your cloud instance very powerful - the same web server can serve multiple domains, subdomains and various web applications, allowing you to consolidate all your development and hosting.

Two of the most common open source web servers are Apache and Nginx, and both are solid options. Apache has been around for a really long time, and has great documentation and support. However, it follows a one-server-does-all model, which can sometimes struggle with richer web pages, causing it to be bloated. Nginx, which follows an event-driven architecture, was created to solve many of these problems. Naturally, I decided to go with Nginx.

My Pick: Nginx

• Event-Driven Architecture
  Nginx was designed to solve many of the problems that Apache web servers face. For a more technical discussion, check out Apache vs Nginx: Practical Considerations.

• Good Documentation for DigitalOcean
  The support on the DigitalOcean community seems to be better for Nginx.

Get Started: How To Install Nginx on Ubuntu 16.04

4. Static Site Generator

If you need a simple web page for a portfolio or blog (similar to what you’re looking at right now), a static site is the way to go. Static sites have been extremely popular recently, and for good reason. They’re extremely lightweight and fast, and web servers are really good at delivering static sites quickly. Many static site generators use templating engines like Jinja to easily generate dynamic markup and source code. While there are various options for static site generators, Jekyll is by far the most popular.

My Pick: Jekyll

• Great Community
  There are plenty of example projects, tutorials and themes available for you to explore. It’s very easy to find answers to any question on StackOverflow as well.

• Quick and Easy
  You can create a new site in 3 lines of code. Yup, that’s not a typo.

Get Started: How to Set Up a Jekyll Development Site on Ubuntu 16.04

5. Web Framework

Awesome, so you’ve got a website. But what about your web applications? Great news - you can host all of them right here on your cloud instance. There are many different frameworks depending on which language you’re comfortable with, such as Meteor if you prefer JavaScript, and Flask if you like Python. I am more comfortable with coding in Python, so I use Flask for all my web applications.

My Pick: Flask

• Python
  It’s in Python, which means you get access to all of Python’s amazing libraries from numpy for matrix manipulation, to pandas and scikit-learn for data analysis.

• Easy to Setup
  You can get a ‘Hello World’ application running in 7 lines of code, and it is very easy to add functionality. Flask is also one of the youngest Python web frameworks, which means that it has learnt from its predescessors, resulting in a very user-friendly experience.

Get Started: The Flask Mega-Tutorial

6. WSGI Server

Just as we need web servers to serve our web pages, we need also need web servers to serve our web applications. A Web Server Gateway Interface (WSGI) is essentially a container in a seperate process that is used to persist Python web applications on a different port than the web server used for your web page. Typically, uWSGI or gunicorn is used to set this up. After trying both, gunicorn seemed easier to set up, but that could just be my personal experience.

My Pick: gunicorn

• Easier to Set Up
  This could just be my personal experience. I would recommend starting with gunicorn, and jumping to uWSGI only if that fails.

Get Started: How To Serve Flask Applications with Gunicorn and Nginx on Ubuntu 16.04

7. Remote Development

Another thing that makes this entire set up so powerful is that you essentially have an entire machine at your fingertips, so you can continue to develop your applications on your cloud instance as you would on your own computer. Jupyter Notebook is a really powerful web app that allows you create documents with live code. It’s hard to explain with words, but I assure you that if you try it, you won’t go back to an IDE. The best part is that you can use it remotely so you can edit code for an application that lives on the cloud, in the browser on your computer.

My Pick: Jupyter Notebook

• Interactive Coding
  The notebook offers a choice of 40 different programming languages, and a bunch of different widgets to improve your coding experience. You can even add your documentation or notes along the way in Markdown.

• Notebook Server
  Run the notebook on your cloud instance, and you can pull it up in your own browser.

Get Started: Running a Notebook Server

8. Code Repository

This is pretty self explanatory. GitHub seems to be the obvious choice for code management and maintainence, but I added this section in to discuss the possibility of local code maintainence. If you wanted, you could completely remove your dependence on GitHub and manage your gits in the cloud with GitLab. I’ve not had a chance to look into this yet, but it sounds like this prove to be especially useful if you are collaborating with others on your projects, and would like the code to live locally on your server.

Get Started: How To Install and Configure GitLab on Ubuntu 16.04

A Learning Experience

This entire exercise has been fundamental in building my understanding of web development, as well as of other concepts such as DNS, proxies, ports, server configurations, and even UNIX commands. It is by no means an easy exercise - it can be frustrating at times when you have no idea what you’re doing. But it is ultimately really fulfilling to know that you’ve essentially built your own web hosting at a fraction of the cost (and learnt so much along the way).

The possibilities beyond this are endless. Some of the next steps include:

• Implementing a database that I can connect to my web applications
• Serving multiple websites from the same cloud instance
• GitLab integration to localize my development in the cloud
• Continue adding more projects (web applications, data visualizations, etc)