Education and the Cloud

January 1, 2015

Using Rundeck on Ubuntu to automate server deployments into LXC (local or remote) containers

Filed under: Cloud Management, LXC, ubuntu, Virtualization Tools — Tags: , , , , — bmullan @ 11:59 am

Continuing my last posts regarding LXC (linux containers) I realize that managing them from the command line might be a bit tedious when there can be hundreds or thousands of containers between your local PC/laptop and any “remote” (ie cloud) servers & LXC containers you utilize/manage.

I just recently found out about Rundeck while searching for orchestration/mgmt tools.

My use-case was that I was looking for something that could help in managing LXC (linux containers) whether remote or local.

Note:  many people confuse LXC and other container technologies like Docker, LMCTFY, etc.   They are all different solutions that underneath utilize Linux Namespaces.   Here is a good multi-part series describing Linux Namespaces.

LXC (https://linuxcontainers.org/) is an incredible technology.

With the release of 1.x this past year it now supports nested containers, unprivileged containers and much more.

Anyway, I decided to see if I could get Rundeck to work in an LXC container and also be able to create workflows/jobs etc to work with LXC containers.

LXC has a rich set of CLI commands:

  • lxc-create
  • lxc-start
  • lxc-attach
  • lxc-stop
  • lxc-clone
  • lxc-destroy
  • etc

There is also an API that supports Python, Go, Ruby etc.

Stephane Graber (one of the LXC core developers) has a great 10 part Blog series that tells you all about LXC.

For me,  I just wanted to get Rundeck to issue the above lxc-xxxxx commands.

Turns out it only took a couple configuration changes so I thought I’d share my notes here.

Note: all of this was done on Ubuntu 14.04

Steps I took to install Rundeck in an LXC container.

create a new container on the Host.   I called mine “rundeck”

$ sudo lxc-create -t download -n rundeck

start the container which will run detached from the terminal you started it on.

$ sudo lxc-start -n rundeck

attach (re get a console into the container)

$ sudo lxc-attach -n rundeck

Note: at this point your console prompt should change to show you are logged in as Root in the Container whose hostname is “rundeck”.

At this point you can do whatever you would do with any ubuntu server but here were my steps

root@rundeck#  apt-get update && apt-get upgrade -y
root@rundeck# apt-get install wget nano default-jre

then I used wget to download the latest Rundeck .deb file:

root@rundeck# wget http://dl.bintray.com/rundeck/rundeck-deb/rundeck-2.4.0-1-GA.deb

Note:  check on their website for the rundeck version number as it may change often

install the rundeck .deb it

root@rundeck# gdebi ./rundeck-2.4.0-1-GA.deb

When the Rundeck installation is done I needed to do a couple of things.

LXC containers in Ubuntu by default are started in their own 10.0.3.x network.   By default applications in the container have internet access and as I’d mentioned before are like being logged into any other ubuntu server in regards to what you can do.

Because its possible that each time you stop/restart an LXC container it may get a different 10.0.3.x address I wanted to create a solution to where the Rundeck webapp would acquire the “current” IP address of the container Rundeck is running inside of, each time that container starts & rundeck starts.

My script looks like this and I saved it into the containers /usr/bin directory after setting it as executable (chmod +x sethost.sh) …  I called mine “sethost.sh” or whatever you called it.


#!/bin/bash
#
# sethost.sh
# purpose:
#            get ip address while running inside a container
#            Stream Edit (SED) and set the /etc/rundeck/rundesk-config.
properties file so that it substitutes
#            “localhost” with that IP
#
#            $ sudo service rundeskd start
#
# assumptions:   container is using its own eth0 for network connection and its primary address
#
# This script is called from /etc/rc.local during system boot but after network IP is set and rundeckd is started
#
# first set rundeck-config.properties back to original
cp /etc/rundeck/rundeck-config.properties-save /etc/rundeck/rundeck-config.properties

# get eth0 IP address (we assume that’s what the container is using

my_ip=$(ifconfig eth0 | grep “inet addr” | awk -F: ‘{print $2}’ | awk ‘{print $1}’)

# swap the term “localhost” with the real IP of the container in the file rundeck-config.properties
FILE=/etc/rundeck/rundeck-config.properties
sed -i -e “s|localhost|$my_ip|” $FILE
# restart the rundeckd service with the new & now actual IP address
sudo /etc/init.d/rundeckd restart

exit

Run this script by adding it into the rc.local file  inside of the lxc container in which you installed rundeck (my container is called rundeck)

in /etc/rc.local just add the following line at the end of whats already there

/usr/bin/sethost.sh

Next, as you may note above I am simply searching rundeck-config.properties for the word “localhost” and substituting the current IP address of the eth0 of the LXC container Rundeck is running in as again “by default” an LXC container will use that IP address and I am assuming defaults here.

Secondly, to keep this simple, before I ever restart the system for the first time I copied the rundeck-config.properties to rundeck-config.properties-save so I had a virgin copy of the original file with “localhost” still in it.

The first step of the sethost.sh script is to restore the original file, then do the SED substitution that way I could always find/substitute the actual IP of the container.

root@rundeck# cp /etc/rundeck/rundeck-config.properites  /etc/rundeck/rundeck-config.properties-save

My next step was to enable use of SUDO in job command so I could have Rundeck work with Privileged LXC containers

Remember to create/start/stop etc those you have to have SUDO privileges on the Host.

I searched the Rundeck forum and found others were grappling with this problem too.

For me my solution (whether its best or not) worked.

I used visudo to edit the SUDOERS file and set the user “rundeck” so “rundeck” user does NOT require a password to execute a sudo command.

Note:  Again, you are doing this WHILE LOGGED INTO the “rundeck” container – NOT – the Host !

This will enable the rundeck web app to execute commands that require “sudo” in them.

in the rundeck container…

$ sudo visudo

Add the following at the end of the sudoers file

rundeck ALL=(ALL) NOPASSWD: ALL

Ctrl-X to leave, save your changes, and you’re done!

Now while logged INTO the rundeck container reboot it.

root@rundeck# shutdown -r now

Note that this will log you out of the container and return you to the original terminal prompt on your Host OS.

If you want to log back into the container “rundeck” you should be able to almost immediately log back in using the lxc-attach command again

$ sudo lxc-attach -n rundeck

But at this point you should be able to log into Rundeck which is running in the separate and isolated LXC container we also called rundeck by pointing your browser to the IP address of the container.

You can find out the containers IP address using the following LXC command while in a terminal on the Host OS:

$ sudo lxc-ls -f

NAME     STATE    IPV4                            IPV6  GROUPS  AUTOSTART 
———————————————————————————————————
base_cn  STOPPED  –                                  –          –                 NO        
rundeck  RUNNING  10.0.3.136, 10.0.4.1    –          –                 NO        
wings    STOPPED  –                                    –          –                 NO  

so in the above case I point my browser to:     http://10.0.3.136:4440

and log into Rundeck as normal (admin/admin  -or- user/user)

However, now when I create a “job” for the localhost… that job executes inside of the LXC container “rundeck” and NOT on the Host OS …!

If you read the linuxcontainers.org website you will also have noticed a new capability/extension to LXC that is now available called LXD (lex-dee).

LXD is introducing a whole new exciting  capability to LXC that includes the ability to easily create/run/manage LXC containers anywhere on any LXC capable host (LXC is part of the linux kernel) whether that host is remote (re Cloud) or local.

This means that even on your laptop you can have dozens or many dozens (depending on memory, applications, etc) of containers all isolated as much/little as you want from each other, from the Host or from the internet.

So now you can use Rundeck to manage/orchestrate all your local PC LXC containers BUT… you should also be able to use LXC & LXD to do the same with remote (re Cloud) servers/LXC containers.

As I am no expert in Rundeck, LXC or Linux feel free to suggest improvements, changes etc where you think this post requires it as I am sure I probably have made some incorrect assumptions w/Rundeck and/or LXC here.

thanks.

Brian

August 3, 2009

Part 2 – Using Cloud & Virtualization Technologies for Education -or- how Education and the Cloud met, married and had smarter kids!

Here I continue my last discussion about K-20 education and how to use cloud technology to possibly do things.

Lately, I’ve been following this thread… and would like to share some ideas and thoughts with you all…

===============================================================================================================================================


Message: 1
Date: Thu, 30 Jul 2009 15:32:18 -0600
From: xxxxxxxxxxxxxx
Subject: Re: [Ltsp-discuss] Recommend Server for 25 clients
To: ltsp-discuss@lists.sourceforge.net
Message-ID:
Content-Type: text/plain; charset=UTF-8

On Thu, Jul 30, 2009 at 1:40 PM, xxxxx xxxxxxxxx<xxxxxxxxxx> wrote:
> xxxxxx xxxxxxxxxx :
>
>> How powerful server would you recommend for 25 users ?
>
> “Server sizing in an LTSP network is more art than science. Ask any LTSP
> administrator how big a server you need to use, and you’ll likely be
> told “It depends”.”
>
> http://www.ltsp.org/~sbalneav/LTSPManual.html#id2697011

===============================================================================================================================================

So I replied to that thread with the following response with I’ll share here on my blog…

I’ve been using Amazon Web Services (AWS) ie Amazon’s cloud for K-20 proof-of-concept work. So bear with me while I describe some things…

  1. Amazon’s Elastic Compute Cloud (EC2) service is very inexpensive and easy to use and provides 5-6 different choices for “compute resources” (ie servers).
  2. Amazon uses a “Utility” based pricing model (you pay only for how much of something you use like water or electricity) and only when you are using it.

ie.  need a bigger server… just pick one and start it up (ie Launch it in AWS terminology) migrate your apps (won’t go into that here)

Need 10 or 100 servers… easy… pick the server model (linux/windows, 32/64 bit etc) — this is called an AMI – Amazon Machine Instance — and when you LAUNCH the AMI just put the # of servers you need into the “Number of Instances” box that pops up when you select to LAUNCH the AMI you picked.

5 minutes later… they will all be running.

You manage all the startup/shutdown, IP address’s, Security Firewall/Access lists etc using Amazon’s web based AWS Management Console.

Now I’ve always wanted say this … But WAIT there’s MORE… it gets better yet <g> !!

You can take ADVANTAGE of Amazon’s Auto-Scaling and Auto-Load-Balancing features.

Since AWS costs are based like a Utility …  you can start off with just 1 server at 5am and if you set it up for auto-scaling …

As students/teachers (ie Load) starts to build say around 9am… the server “can” Auto-Scale UP by cloning itself and at the end of the day the servers will Auto-Scale DOWN by terminating
themselves when no longer needed (ie you don’t pay for them when they aren’t running).   You are the one to configure the parameters for the UP/DOWN auto-scaling.

try doing that in your school or data center where 1st you have to buy the servers, rack/stack/cable/ pay for HVAC, maintenance contracts, insurance, replace parts, etc.

I like letting Amazon worry about that stuff!

I will copy some information from the AWS web site.

You can sign up for an AWS account free (again you only get billed if you start using something).

As you can see below a “small” server costs just 10 cents/hr while the largest (8 or 20 core) just 80 cents/hr.

I learned about AWS by starting a “small” Ubuntu server, installing my applications, testing etc. then blowing it away when I was done.   I spent 4-5 hours a day ($0.50/day) to do this.
It was very easy to learn !

===============================================================================================================================================

Instance Types

Standard Instances

Instances of this family are well suited for most applications.

  • Small Instance (Default) (ie virtual server)
    • 1.7 GB of memory
    • 1 virtual core
    • 160 GB of instance storage
    • 32-bit platform
  • Large Instance 7.5 GB of memory, 4 core, 850 GB of instance storage, 64-bit platform
  • Extra Large Instance (ie virtual server)
    • 15 GB of memory
    • 8 core
    • 1.7 TB of instance storage
    • 64-bit platform

High-CPU Instances

Instances of this family have proportionally more CPU resources than memory (RAM) and are well suited for compute-intensive applications.

  • High-CPU Medium Instance 1.7 GB of memory, 5 core, 350 GB of instance storage, 32-bit platform
  • High-CPU Extra Large Instance
    • 7 GB of memory,
    • 20 core
    • 1.7 TB of instance storage,
    • 64-bit platform


===============================================================================================================================================

Pricing

NOTE:   as of 9/2010 AWS has introduced an approximately 18% price decrease for most of the AWS EC2 compute instance sizes.    The pricing below does NOT reflect this change.

AWS has also introduced a new “micro” instance which provides 640Meg of RAM,  1/2 a cpu for only  $0.02 cents per hour —  48 cents per day ??

Pay only for what you use. There is no minimum fee. Estimate your monthly bill using AWS Simple Monthly Calculator.

On-Demand Instances

On-Demand Instances let you pay for compute capacity by the hour with no long-term commitments.

This frees you from the costs and complexities of planning, purchasing, and maintaining hardware and transforms what are commonly large fixed costs into much smaller variable costs.

The pricing below includes the cost to run private and public AMIs on the specified operating system.

Amazon also provides you with additional instances with other option for Amazon EC2 running Microsoft and Amazon EC2 running IBM that are priced differently.

United States

Europe
Standard On-Demand Instances Linux/UNIX Usage Windows Usage
Small (Default) $0.10 per hour $0.125 per hour
Large $0.40 per hour $0.50 per hour
Extra Large $0.80 per hour $1.00 per hour
High CPU On-Demand Instances Linux/UNIX Usage Windows Usage
Medium $0.20 per hour $0.30 per hour
Extra Large $0.80 per hour $1.20 per hour

United States
Europe
Standard On-Demand Instances Linux/UNIX Usage Windows Usage
Small (Default) $0.11 per hour $0.135 per hour
Large $0.44 per hour $0.54 per hour
Extra Large $0.88 per hour $1.08 per hour
High CPU On-Demand Instances Linux/UNIX Usage Windows Usage
Medium $0.22 per hour $0.32 per hour
Extra Large $0.88 per hour $1.28 per hour

Pricing is per instance-hour consumed for each instance type, from the time an instance is launched until it is terminated. Each partial instance-hour consumed will be billed as a full hour.

Reserved Instances

Reserved Instances give you the option to make a low, one-time payment for each instance you want to reserve and in turn receive a significant discount on the hourly usage charge for that instance.

After the one-time payment for an instance, that instance is reserved for you, and you have no further obligation.

You may choose to run that instance for the discounted usage rate for the duration of your term, or when you do not use the instance, you will not pay usage charges on it.

United States

Europe
Linux/UNIX One-time Fee
Standard Reserved Instances 1 yr Term 3 yr Term Usage
Small (Default) $325 $500 $0.03 per hour
Large $1300 $2000 $0.12 per hour
Extra Large $2600 $4000 $0.24 per hour
High CPU Reserved Instances 1 yr Term 3 yr Term Usage
Medium $650 $1000 $0.06 per hour
Extra Large $2600 $4000 $0.24 per hour

United States
Europe
Linux/UNIX One-time Fee
Standard Reserved Instances 1 yr Term 3 yr Term Usage
Small (Default) $325 $500 $0.04 per hour
Large $1300 $2000 $0.16 per hour
Extra Large $2600 $4000 $0.32 per hour
High CPU Reserved Instances 1 yr Term 3 yr Term Usage
Medium $650 $1000 $0.08 per hour
Extra Large $2600 $4000 $0.32 per hour

Reserved Instances can be purchased for 1 or 3 year terms, and the one-time fee per instance is non-refundable.

Usage pricing is per instance-hour consumed.

Instance-hours are billed for the time that instances are in a running state; if you do not run the instance in an hour, there is zero usage charge. Partial instance-hours consumed are billed as full hours.
===============================================================================================================================================

Here’s how I make use of this.

On AWS you can pick from hundreds of pre-built “public” servers types (different flavors of Linux – Fedora, Ubuntu, Centos etc etc), 32 bit or 64 bit.

Some are “server” linux some are desktop linux.

Some have been built with apps already installed (Apache, MySQL, etc etc)

You get the idea.

So what have I been doing for kids/education… ?

Server Side:

I’m using AWS Desktop images where I’ve installed the x2go one-server.

x2go utilizes the NoMachine NX transport protocol libraries that are Open Source but x2go implements its own server-side and client modules.   The server side comes in a single user home version and also a x2go server implementation that is clustered and load balanced.

Unlike NoMachine’s current NX server/client …. where audio is a big problem.   x2go supports audio extremely well from server to client.    Local printing and sharing of folders between server and client is also supported.

Client Side:

Client side boots off of a Ubuntu USB thumb drive – preloaded with the x2go Open Source Windows, Mac or Linux clients.

x2go also has introduced a Web Portal capability for accessing the remote desktop.    Any user with a Browser that supports java can now access the Remote Desktop without installing any other client software on their local PC.

Each kid can have one and that way they can use it at school or — at home (same desktop, same cloud servers as at school).

Since the “real work” in terms of CPU and Storage is out on the AWS “cloud” it does NOT even matter what type PC they use…. all you use the local machine for is basically to boot off of
the USB and the local keyboard, mouse, screen and network connection (everything becomes a thin-client)

  • old pc, new pc
  • old laptop, new laptop
  • netbook
  • thin client

Since the “Desktop” that the students see is exported over NX from the AWS Desktop server where I can have from 1 – 20 CPU and I can have as many servers as I want… or can pay for <g>?

— and —

because storage using AWS’s S3 – Simple Storage Service and EBS – Elastic Block Storage is more or less infinite (at least as far as I’m concerned)

Now how’s performance.

Well you have to have a working and stable local network first of all but that’s true even if using a client/server model or a Thin Client model LTSP or Citrix etc.

The NX protocol is terrific and you can read about just how good it is here.

Here’s my basic process to create a server IF I start by using one of AWS’s Public Amazon Machine Image (AMI) that are  available.

  1. Launch the AMI instance I want
  2. Modify it by adding all the applications I need and configuring everything.
  3. Save the running “instance” using the free AWS EC2 AMI tools to what is called an S3 storage “bucket”.
  4. Re-register my now saved AMI “image” as a NEW Amazon AMI (once registered w/AWS I’ll be able to LAUNCH it from the AWS Management Console like any other AWS AMI.
  5. I then LAUNCH my new image like any other AWS AMI
    1. tell AWS how many “instance” … ie # virtual machines
    2. tell AWS what size server (32/64 bit small … up to Extra Large)
    3. Assign my firewall/access lists to the new instance
    4. Create and Assign an AWS Elastic IP address to MY “instance” (simple – takes 2 seconds)
  6. Once it’s in a “running” state.. just use the AWS cloud based server

Elastic IP Addresses – Elastic IP addresses are static IP addresses designed for dynamic cloud computing.
An Elastic IP address is associated with your account not a particular instance, and you control that address until you choose to explicitly release it.
Unlike traditional static IP addresses, however, Elastic IP addresses allow you to mask instance or Availability Zone failures by programmatically
remapping your public IP addresses to any instance in your account. Rather than waiting on a data technician to reconfigure or replace your host,
or waiting for DNS to propagate to all of your customers, Amazon EC2 enables you to engineer around problems with your instance or software by
quickly remapping your Elastic IP address to a replacement instance.

By the way, in case this isn’t obvious… got a new school that needs to be setup?

Other than the USBs for the kids and some kind of computer for them to use … the server can take only minutes to setup and there’s no physical installation involved !!!

Finally, I use my local machine with NX Client software to log in and I get a Desktop… and it’s all PFM …  magic !

Today (right now) I’m writing this while I have 4 AWS servers running that I am testing.

On my desk is a Lenovo T61p laptop

  • Dual Core
  • 4 Gig RAM

next to it I have an ASUS 1000HE Netbook

  • Atom processor
  • 1 G RAM

Both machines booted off of a USB.

I next used the  NX Client software to log into one of my AWS Desktop servers on each one and started working.

Performance is exactly the same on both clients (well  they ASUS display can only go 1400×600)

I wrote this on my AWS desktop server session using the ASUS while several of the  sessions on the Lenovo were doing some other things for me

I’d really like to get more in the Linux K-12 and K-20 community trying this so we can all share more of what we are doing for education of our kids.

Let me know if any of you would like some more pointers or information as I said I’d like some folks to work with on all of this.

I’ve also got some pretty cool AWS based solutions for the “Windows” in your life…

Hope you found this interesting!

Brian Mullan

June 18, 2009

Part 1 – Using Cloud & Virtualization Technologies for Education -or- how Education and the Cloud met, married and had smarter kids!

U.S. Education Secretary Arne Duncan wants to use some seed money in a Race to the Top to see what innovative States can come up with in regards to best ideas, concepts, implementations and results.   Good idea… kind of like prototyping and trialing then picking the best.

From my view there are many things that can be addressed in education.   Technology being just one of several approaches to the overall issues related to improving K-12 education.

I recently heard a short comment that made an impression.

In 1909 if you had gone into a classroom in a large city school you would have seen kids seated at desks with pencils and paper.

At the front of the classroom would be a teacher sitting facing the children with the teacher’s pencil and paper on her desk.

Of course books would be on the desks and a blackboard with chalk on the front wall.

Fast forward 100 years to 2009.

How much has that picture really changed ?

Ok… there may be some classrooms at some schools that have some “newer” technologies

  1. a projector ? some
  2. <lets skip a few era’s of technology here>?
  3. computer on every desk ? more rare than common
  4. networked servers/computers — rarer than #3
  5. maintained network computers – rarer than #4 and #5
  6. #5 & #6 maintained by someone other than the Librarian and Librarian assistant ???

Well you get the idea and if you work at or for a school you know the picture.

Click here to see some “Race to the Top” Slides

Geez where to start?

I am fairly certain that Cloud and Virtualization technologies are going to play major roles in some of the successes.

But what kind of Cloud ?   Private, Public .. hybrid and whats the Total Cost of Ownership (TCO) for each of those paths.

Private

  • the State or the LEA owns/manages/pays for a Data Center and support staff, electricity equipment, heat/air, safety, insurance

Public

  • Amazon Web Services (AWS), Rackspace, Google owns the infrastructure, etc but you may still be the “operator”

Hybrid

  • Private Data Center augmented by compute or storage resource provided by a Public cloud provider

Well lets make it more muddled?

Should you go with an Infrastructure-as-a-Service (IaaS) Cloud provider like Amazon.

or Amazon as a Software-as-a-Service (SaaS) yes… it does exist via 3rd party developers that are offering many services ranging from Db2, Oracle, Mail, WebServers, Video servers etc.

What about using Google as a Platform-as-a-Service (PaaS) where you write or rewrite you own applications using Java/PHP and then host them on Google.

or possibly Google as a Software-as-a-Service (SaaS) cloud provider (think gMail, Google Docs).

I don’t think there necessarily has to be one choice… or one Cloud Service…  after all it is the Internet.

To get started I think one of the first things that should be done is getting all the schools in all the LEAs on a level starting platform.   Why?

Some schools have

  • old Desktops
  • new Desktops
  • old laptops
  • new laptops
  • thin client (re using something like citrix)
  • maybe netbooks

The above computers may vary

  • CPU’s ranging from Pentium to Dual Core Intel to AMD to Atom processors
  • Memory ranges from 512Meg to 4 Meg
  • Hard disks (if they have them) 40G – 100 G

Network connectivity ability from

  • 10Mbps to 100Mbps ethernet
  • Wireless B, G or maybe N

For the most part those computers run Windows -but- that can mean anything from Windows 95 to  Windows 98, Windows 2000, XP  or Vista

Sorry Mac and Linux users … gotta focus here to make a point.  We’ll get to you later.

To level the starting platform you can’t just tell people to junk everything… and for the most part there isn’t a reason to if you think of clever solutions.

That’s enough to start the conversation… I’ll add more later but wanted to get my ramblings on this topic started.

Brian Mullan

Create a free website or blog at WordPress.com.