I totally fucked up the other week keeping a calibrated way to clean up nodes in Chef at Artifact Uprising. Since now I migrated from managed Chef to my own Chef OSS implementation, I know have to begin doing some upkeep and cleanup work.

Some Lambda functions that rely on Cloudwatch need to be cleaned up but I write this sitting down after an outpatient procedure I had yesterday, so I made a totally fucking hack-script to do the following:

  • Find AWS EC2 instances in a “Terminated” state
  • Have knife retrieve that array and search for them
  • Have knife clean up the node and client

Why do this? Because EC2 instances can go away at any time and Auto-scaling is setup to re-provision them automatically and bind them to Chef (another future post here). In other words, Chef needs to remain current within reasonable time.

Frankly, right now I don’t feel like messing around with Golang or Ruby so Bash is my way to go. If you don’t use Lambda, Cloudwatch or any fancy tooling, cron this bash somewhere:

# Scan for terminated nodes and remove from Chef

getTerminatedNodes() {

# get nodes that are terminated via aws-cli
nodes=$(aws ec2 describe-instance-status --filters Name=instance-state-name,Values=terminated | grep "InstanceId" | awk -F ":" '{print $2}' | awk -F '"' '{print $2}')
 echo $nodes

# loop and have knife find them, then delete client and node
for node in $nodes;
 found=`knife search node "*${node}*" | grep "Node Name" | awk -F ":" '{print $2}'`
 if [ ! -z "${found}" ];
 knife client delete ${found} -y && knife node delete ${found} -y 



That’s it! Cron it at a desired interval and Chef will stay relatively updated. Next will be a writeup on Lambda and Cloudwatch.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s