As I sit on a plane going to my next DevOps adventure, I write this little nugget about config management platforms. Why? One, I have time. Two, a little Ruby/Puppet DSL (Domain Specific Language) deep-dive I had some weeks ago made me think about this.

Recently, I had to explain to a good bunch of folks why Puppet uses a DSL. Not only Puppet, but any other fine config management tool like it. Explaining a DSL was a bit harder than I thought so I decided to do a “Why Ruby?” presentation. During the presentation, I did the unthinkable, I wrote Ruby! It was a simple exercise: how many lines of Ruby vs. a DSL takes you to perform a specific task. The task was to install apache server. I won’t go into detail about it…but it was roughly 3 lines on DSL vs. 74+ on a given programming language without a lot of the dependencies.

However, that little exercise made me think: “Hey, let me build a small tool in Ruby to show how installing a package works and have the plumbing to operate like a Puppet, Chef, or Salt platform”. Hence, Sysville, as I call it, will be explained on this post.

Sysville is a simple concept. It is a small “neighborhood” of systems that get “mail” delivered and/or returned to a neighborhood post office to act upon. A main hub for a neighborhood of servers.

For this concept and exercise, I decided to setup a simple RabbitMQ message queue and a “post office” server. If I expand on it, there will be future posts. As usual on my blog, let’s begin.

The Architecture

Below is the basic logical architecture of our application:

untitled-diagram

Yes!, we have a MongoDB piece!

The Setup

Build at least 2 CentOS VMs, 1 will be the post office, the other a node or client.

The RabbitMQ piece

Install and configure RabbitMQ per this guide on the main server that will be the post office.

The MongoDB piece

Install and configure MongoDB on the same server that acts as the post office per this guide.

Now, you should have a running RabbitMQ installation and a MongoDB to store our stuff, whatever that stuff happens to be. The intent is to let you be creative and come up with cool things to do. For now, we will capture install records that serve as an audit trail. This is one of the main reasons companies look for platforms like Puppet.

The Post Office

Our post office is a small 100% Ruby middleware that posts messages to RabbitMQ to be retrieved by a node.

The code

First, let’s create a configuration YAML file to store our values. This is very similar how Puppet abstracts common values from the actual infrastructure code.

Create a folder called sysville and inside, a folder called config:

mkdir -p sysville/config

Inside the config folder, create the file rabbit_config.yaml and populate replacing your values:

---
 # RabbitMQ values
 mq_user: admin
 mq_pass: admin
 mq_server: ls0

 info_channel: info
 notification_channel: notify
 provision_channel: provision
 init_channel: init
 enforce_channel: enforce
 status_channel: status

# Mongo values
 mongo_host: ls0
 mongo_port: 27017
 db: sysville

In my setup, ls0 is my main server. Replace with your targeted host server.

For both servers, install the Bunny gem:

gem install bunny

Now, at the root of the sysville directory, we create post_office.rb. Read the comments on the code to understand what all the pieces do:

#!/usr/bin/env ruby
# encoding: utf-8

# Libraries required. Yaml is included. You will need to install bunny
# and mongo ( gem install bunny && gem install mongo )
require 'bunny'
require 'yaml'
require 'date'
require 'mongo'

class Postoffice

# load configs to use across the methods

fn = File.dirname(File.expand_path(__FILE__)) + '/config/rabbit_config.yaml'
 config = YAML.load_file(fn)

# export common variables

@@datetime = DateTime.now()

# export the connection variables
 @@host = config['mq_server']
 @@mq_user = config['mq_user']
 @@mq_pass = config['mq_pass']

# export the channels to be created/used
 @@info = config['info_channel']
 @@notif = config['notification_channel']
 @@provi = config['provision_channel']
 @@init = config['init_channel']
 @@enfo = config['enforce_channel']
 @@stat = config['status_channel']

# mongo database values
 @@db = config['db']
 @@mongo_host = config['mongo_host']

# export connection to RabbitMQ
 @@conn = Bunny.new(:hostname => @@host,
 :user => @@mq_user,
 :password => @@mq_pass)

# export connection to MongoDB
 @@db_conn = Mongo::Client.new([ "#{@@mongo_host}:27017" ], :database => "#{@@db}")

def initialize()
 end

# define methods to use by server and clients
 # REQUEST STATUS OF NODE, BASICALLY A PING  TODO

def request_status(hostname)
# open connection to MQ
@@conn.start

# generate a random message ID 
id = rand(0...1000000)
 type = "PING_REQUEST"
 message = type + "," + hostname + "," + String(id) + "," + String(@@datetime)

 # create channel to post messages
 ch = @@conn.create_channel
 q = ch.queue(@@stat)
 ch.default_exchange.publish(message, :routing_key => q.name)

puts " [x] Sent Status Request to " + hostname

@@conn.close

# place record on database

collection = @@db_conn[:status]
 doc = { type: type, client: hostname, msg_id: id, time: @@datetime }
 result = collection.insert_one(doc)
 puts result.n


 end

# REQUEST A NODE TO INSTALL A PACKAGE
 def install_parcels(parcel, hostname)

id = rand(0...1000000)
 type = "INSTALL_REQUEST"
 message = type + "," + parcel + "," + String(id) + "," + String(@@datetime)

@@conn.start
 ch = @@conn.create_channel
 q = ch.queue(@@provi)
 ch.default_exchange.publish(message, :routing_key => q.name)

puts " [x] Sent Installation Request for " + parcel + " to " + hostname

@@conn.close

collection = @@db_conn[:provision]
 doc = { type: type, package: parcel, client: hostname, msg_id: id, time: @@datetime }
 result = collection.insert_one(doc)
 puts result.n

end

# REMOVE A PARCEL, PROCESS A RETURN
 def remove_parcels(parcel, hostname)

id = rand(0...1000000)
 type = "UNINSTALL_REQUEST"
 message = type + "," + parcel + "," + String(id) + "," + String(@@datetime)

@@conn.start
 ch = @@conn.create_channel
 q = ch.queue(@@provi)
 ch.default_exchange.publish(message, :routing_key => q.name)

puts " [x] Sent Installation Request for " + parcel + " to " + hostname

@@conn.close

# CREATE A RECORD ON THE MONGO DATABASE
collection = @@db_conn[:provision]
 doc = { type: type, package: parcel, client: hostname, msg_id: id, time: @@datetime }
 result = collection.insert_one(doc)
 puts result.n

end
end

On the root of sysville, create a small script to send messages. Let’s call it try.rb since I don’t have an original name for it:

require "./post_office"

d = Postoffice.new()
d.provision("httpd","ls1")

The file above imports our Postoffice class and passes two values to the provision method: httpd, ls1. This tells the post office to please send a message to host ls1 that it needs to install httpd. Ls1 is your client or node.

The Domicile

Following the neighborhood’s post office analogy, we now treat our client node as a home or a domicile which receives mail, messages.

The code

Make a directory called house and inside create the file domicile.rb with this content:

#!/usr/bin/env ruby
# encoding: utf-8

require "bunny"

# CREATE CONNECTION TO RABBIT MQ
conn = Bunny.new(:hostname => "ls0", :user => "admin", :password => "admin")
conn.start

# STATE WHICH QUEUE WE WILL BE LISTENING ON
ch = conn.create_channel
q = ch.queue("provision")

puts " [*] Waiting for messages in #{q.name}. To exit press CTRL+C"
q.subscribe(:block => true) do |delivery_info, properties, body|
 # GRAB THE MESSAGE AND SEGMENT IT TO GRAB VALUES
 res = body.split(',')
 req = res[0]
 bin = res[1]

puts " [x] Received #{body}"

# EVALUATE THE FIRST FIELD TO DETERMINE IF I AM TO INSTALL OR REMOVE
# I AM FORKING THE INSTALL PROCESS TO KEEP THE LISTENER OPEN WAITING FOR MORE
# REQUESTS
if req == "INSTALL_REQUEST"
 install_job = fork do
 puts "I am an install request"
 exec "yum install #{bin} -y"
 end
 Process.detach
end

# FEEL FREE TO CHANGE THE POST OFFICE INSTALL TO UNINSTALL TO TRY
if req == "UNINSTALL_REQUEST"
 install_job = fork do
 puts "I am an install request"
 exec "yum erase #{bin} -y"
 end
 Process.detach
end
end

You should now be ready to test.

Try it!

Go to your Post Office instance, ls0, navigate to our sysville folder and run the following:

ruby try.rb

The output should be like this:

[root@ls0 sysville]# ruby try.rb
D, [2016-10-17T13:38:24.600211 #2304] DEBUG -- : MONGODB | Adding ls0:27017 to the cluster.
 [x] Sent Installation Request for httpd to ls1
D, [2016-10-17T13:38:24.619113 #2304] DEBUG -- : MONGODB | ls0:27017 | sysville.insert | STARTED | {"insert"=>"provision", "documents"=>[{:type=>"INSTALL_REQUEST", :package=>"httpd", :client=>"ls1", :msg_id=>316469, :time=>#<DateTime: 2016-10-17T13:38:24+00:00 ((2457679j,49104s,599731872n),+0s,2299161j)>, :_id=>BSON::ObjectId('5804d450ec0c7b090000d...
D, [2016-10-17T13:38:24.620682 #2304] DEBUG -- : MONGODB | ls0:27017 | sysville.insert | SUCCEEDED | 0.001470799s
1

Now the request to install has been posted to RabbitMQ waiting to be picked up. We also have created an audit trail by inserting a record on the database for this install.

Now, go to our client node, ls1:

ruby domicile.rb

Output:

[root@ls1 sysville]# ruby domicile.rb
 [*] Waiting for messages in provision. To exit press CTRL+C
 [x] Received INSTALL_REQUEST,httpd,316469,2016-10-17T13:38:24+00:00
INSTALL_REQUEST
httpd
I am an install request
# HERE BEGINS THE YUM INSTALL PROCESS
Loaded plugins: fastestmirror
Loaded plugins: fastestmirror

Now, your node has httpd since it was told to install it.

So…

Platforms like Puppet and others use a DSL to make the process above a lot slimmer and simpler, among a lot of other things. In this fashion, you don’t have to write all this code to do this simple installation. Also, applications like this come with everything integrated on it, such as message queues so you don’t have to spend time figuring out how to integrate it.

Feel free to download the repo for this exercise and play with it. Also, add stuff to it an have fun!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s