The Era of Automation and the Lone Orchestrator Has Begun
How to run websites, for mortals who value their time and sanity
Let me tell you how I did this website thing in the old days—that’s the mid-1990s, kids.
I cooked a bunch of HTML files, with plenty of hard-coded URLs sprinkled in, and seasoned the mix with a bloated style sheet; launched an FTP client, preferably one with buttons to click on; and then uploaded my beautiful goop to a directory on a web server—that was all I needed and could afford.
Then the web hosting company performed some black magic rituals and the universe—OK, just me—marveled at my creation.
You can still play that game, that is, if your name is Timmy and the website belongs to your cat, Lassie, but if you intend to run a website where other adults do, you know, important stuff, as loyal Tank would say: “Timmy, that is loco.”
Everybody loves to fill their days with error messages, right?
These days, real-world web applications must be resilient and perform well; may involve multiple web servers, efficient caching mechanisms, and a host of other services; and have to query classic relational databases, NoSQL storage, or both.
For more involved configurations, you’ll need tasks queues, extra development libraries—no, not that version, the other one—and access to a myriad of software components and web services.
Oh, and did I mention that everything has to keep running if any part breaks? I’m sorry, Timmy, there’s no room for you and your cat in this, but we’ll keep you in our thoughts (no, we won’t).
If you are down with setting up an architecture like this by hand, expect to burn countless hours typing tedious, prone-to-error commands and watching messages scrolling down all night, like a disappointed Cypher. I guarantee you plenty of silly mistakes and constant suffering. What? You like it that way? Stop reading and go back to your basement.
Waits for masochist visitor to close the browser.
Still here? Cheer up, there’s a better way to spend your crammed days: You’ve got to automate and orchestrate.
Don’t repeat yourself, I repeat: don’t repeat yourself
Automation means describing a task, usually a single one, to run without human intervention. Any task that makes you repeat the same steps every time is a candidate for automation: launching a web server, refreshing configuration files, adding a back end to a load balancer.
Orchestration, on the other hand, refers to the arranging and coordination of all the automated tasks. It’s a workflow, such as running a test suite after a commit, deploying the latest code to production, or syncing data with staging. In human speak: Orchestration is the conductor waving the baton to get the best out of the musicians.
There are many tools to help you with this, and the ones I like are mostly open source. I used to set up my local development environments with Vagrant and its provisioning system but then I was lured to the containers world by Docker and now I see whales everywhere.
I also keep a couple of configuration management tools in my arsenal: SaltStack and Ansible, both written in Python. The original version of Zinibu used a set of Salt formulas I wrote to run my Django stack, but I’m moving to Ansible soon; I like its simplicity and agentless approach.
Pay attention to the tasks that you find yourself repeating every day, especially those where you always make mistakes as they are the perfect opportunities to introduce automation and orchestration. Yes, you will spend extra time at the beginning but will save effort and headaches in the future.
Ideally, you shouldn’t communicate directly with each server or service to perform repetitive tasks, and just one command or—even better—a triggered event should suffice. If you get there, you may want to take the extra step to continuous integration but that’s a story for another day.
To infinity and beyond?
You can go as far as you want building your architecture but don’t get carried away trying to make it perfect; remember you still have a website to run. I like to automate and orchestrate just enough to launch, deploy code, and achieve high availability, and then I continue working on other parts of the project knowing that I can return later to teach new tricks to little Skynet.
Don’t fret about leaving a few things to the experts, such as scaling Elasticsearch or maintaining a Redis cluster; instead, focus your time and energy where they are the most effective—let’s say, writing software or content. This is critical if you are running a one-person operation.
In sum, you are in charge: make your software work for you.
What’s that, Timmy, you just started writing your own Ansible playbooks? Beautiful. Now get rid of that silly cat and join us in Zion.