Customize Packages for Atomic Host : Ansible Automation

My earlier post automate-building-your-own-atomic-host describes how to Automate building Atomic Host with Ansible. But it is not capable of customizing packages for OSTree and build Atomic host based on it.

This post includes Ansible Automation for customizing packages of OSTree and build Atomic Host. Below are the improvements on this updated post :

  • Add packages to OSTree (It is suggested not to delete any pre-existing packages from the OSTree, that might break your Atomic host).
  • No need to create VM from the QCOW2 image will shell script anymore, It will be now all-in-one playbook.

If you do not know about Atomic host, please refer to http://www.projectatomic.io.

Requirements:

The requirements remain the same just like the earlier post. I will use Fedora distribution for the demo, but the same will be applicable for CentOS as well.

  • Make sure Ansible is installed on your system. If you are using Fedora workstation/any other platform, do not forget to install python2-dnf as well.
  • Download Atomic QCOW2 image: Fedora-Atomic.

Procedure:

Clone:

$ git clone https://github.com/trishnaguha/build-atomic-host.git
$ cd build-atomic-host/

Environment Setup:

The following will install requirements on your system, initializes OSTree and starts HTTP server as daemon service. The OSTree is made available via web server at TCP port 35000. After running the playbook you may use ip addr to check the IP Address of your HTTP server.

$ ansible-playbook setup.yml --ask-sudo-pass

Variables:

I am going to use variables applicable for Fedora. If you are using CentOS please modify the variables based on it.

  • Primary Variables.
    vars/atomic.yml
---
# Variables for Atomic host
atomicname: my-atomic                          # OSTree name
basehost: fedora-atomic/25/x86_64/docker-host  # Basehost
httpserver: 192.168.122.1                      # IP Address of HTTP Server
  • Additional packages you would like to have in your OSTree.
    vars/buildrepo.yml
# Variables for OSTree packages
repo: https://pagure.io/fedora-atomic.git
branch: f25
repodir: fedora-atomic
abs_path: /workspace                                # The absolute path to the git repo.
custommanifest: customized-atomic-docker-host.json  # The manifest that goes into the custom host(ostree) content that we are going to build.
sourcemanifest: fedora-atomic-docker-host.json      # The manifest that goes into the actual Base Fedora host(ostree) content.
packages: '"vim-enhanced", "git"'                   # Packages you want to have in your Atomic host.

Add packages like above separate by comma.

  • Variables for the VM
    vars/guests.yml
# Variables for Creating VM
domain: atomic-node                        # Domain name
image: Fedora-Atomic-25-20170228.0.x86_64  # Image name
cpu: 1
mem: 1536
os:
  variant: fedora23
path: /tmp                                 # Path to QCOW2 Image

Main Playbook:

Run the main Playbook which will create VM from QCOW2 image, compose OSTree and perform SSH-Setup and Rebase on OSTree:

$ ansible-playbook main.yml --ask-sudo-pass

Note: user-name: atomic-user, password: atomic are the credentials for the instance.
If you wish to change it, modify the cloud-init user-data.
We will have the credentials as variables/vault later.

To Check the IP Address of the VM running:

$ sudo virsh domifaddr atomic-node

Now SSH to the Atomic host and perform Reboot so that it will reboot in to custom OSTree.

$ ssh atomic-user@IP-ADDRESS-OF-VM
$ sudo systemctl reboot

SSH again and You will have your own OSTRee.

[atomic-user@atomic-node ~]$ sudo rpm-ostree status
State: idle
Deployments:
● my-atomic:fedora-atomic/25/x86_64/docker-host
       Version: 25.1 (2017-02-07 05:34:46)
        Commit: 15b70198b8ec7fd54271f9672578544ff03d1f61df8d7f0fa262ff7519438eb6
        OSName: fedora-atomic

  fedora-atomic:fedora-atomic/25/x86_64/docker-host
       Version: 25.51 (2017-01-30 20:09:59)
        Commit: f294635a1dc62d9ae52151a5fa897085cac8eaa601c52e9a4bc376e9ecee11dd
        OSName: fedora-atomic
[atomic-user@atomic-node ~]$ rpm -q git vim-enhanced
git-2.9.3-2.fc25.x86_64
vim-enhanced-8.0.386-1.fc25.x86_64

If you find any bug/idea please open up issues here. Thanks.

Atomic-Host

Contribute to Atomic Host Documentation

The post describes how you can contribute to Atomic Host Documentation. If you do not know what Atomic Host is, please refer to: http://www.projectatomic.io.

Atomic Host Documentation focuses on covering Docs required for Atomic Host Introduction, Installation, Cluster set up with Kuberenetes/Openshift etc to how to compose, manage and deploy Atomic Host and its various application.

Atomic Host Documentation will also support Fedora Atomic Documentation and CentOS Atomic Documentation within the same repository in near future as AsciiBinder has ability to build Docs with multiple distros and versions.

Requirements:

  • AsciiDoc markup language to write Docs.
  • Asciidoctor that acts as text processor to convert AsciiDoc content to HTML5, DocBook and others.
  • AsciiBinder that helps to build, maintain documentation in easier way.

Set up Development Environment:

The following creates development environment and installs required libraries/packages on your local system which are required in order to write/build Doc for Atomic Host.

$ sudo dnf install ansible
$ git clone https://github.com/projectatomic/atomic-host-docs.git
$ cd atomic-host-docs/
$ git checkout -b branchname
$ ansible-playbook setup.yml --ask-sudo-pass

How to Write Doc:

Atomic Host Documentation uses AsciiDoc markup language. You can have a look at the Reference for AsciiDoc Syntax.

The following procedure demonstrates how you can write and build Doc.

$ mkdir container
$ touch container/overview.adoc

 

container/overview.adoc

[[container-overview]]
= Container Overview
{product-author}
{product-version}
:data-uri:
:icons:

Container contains applications in a way to keep itself isolated from the host system that it runs on and
container allows developer to package an application with all of it parts, such as libraries and other packages
it needs to run and ship it all as one package.
I love Containers!!!

 

After the Doc is ready, we need to make entry in topic_map.yml file. This file tells AsciiBinder which topic groups and topics to generate.

_topic_map.yml

---
Name: Tools
Dir: container
Topics:
  - Name: Overview
    File: overview

 

Now go to the root directory of the repo. The following command will build the Documentation.

$ asciibinder

Verify:

A new directory will be created named _preview. You will be able to browse Documentation from there that you just build.

This is how it will look like after the Doc is build:

contribution-guide-demo

Join and Help:

If you are looking forward to contribute to Atomic Host Docs, this URL contains the issues based on the topics that is required to be documented: https://github.com/projectatomic/atomic-host-docs/issues.

Mailing list:

IRC: #atomic on Freenode server.

Automate Building your Own Atomic Host

Project Atomic hosts are built from standard RPM packages which have been composed into filesystem trees using rpm-ostree. This post provides method for automation of Building Atomic host (Creating new trees).

Requirements

Process

Clone the Git repo on your working machine Build-Atomic-Host.

$ git clone https://github.com/trishnaguha/build-atomic-host.git
$ cd build-atomic-host

Create VM from the QCOW2 Image

The following creates VM from QCOW2 Image where username is atomic-user and password is atomic. Here atomic-nodein the instance name.

$ sudo sh create-vm.sh atomic-node /path/to/fedora-atomic25.qcow2
# For example: /var/lib/libvirt/images/Fedora-Atomic-25-20170131.0.x86_64.qcow2

Start HTTP Server

The tree is made available via web server. The following playbook creates directory structure, initializes OSTree repository and starts the HTTP server.

$ ansible-playbook httpserver.yml --ask-sudo-pass

Use ip addr to check IP Address of the HTTP server.

Give OSTree a name and add HTTP Server IP Address

Replace the variables given in vars/atomic.yml with OSTree name and HTTP Server IP Address.

For Instance:

# Variables for Atomic host
atomicname: my-atomic
httpserver: 192.168.122.1

Here my-atomic is OSTree name and 192.168.122.1 is HTTP Server IP Address.

Run Main Playbook

The following playbook installs requirements, starts HTTP Server, composes OSTree, performs SSH-setup and rebases on created Tree.

$ ansible-playbook main.yml --ask-sudo-pass

Check IP Address of the Atomic instance

The following command returns the IP Address of the running Atomic instance

$ sudo virsh domifaddr atomic-node

Reboot

Now SSH to the Atomic Host and reboot it so that it can reboot in to the created OSTree:

$ ssh atomic-user@<atomic-hostIP>
$ sudo systemctl reboot

Verify: SSH to the Atomic Host

Wait for 10 minutes, You may want to go for a Coffee now.

$ ssh atomic-user@192.168.122.221
[atomic-user@atomic-node ~]$ sudo rpm-ostree status
State: idle
Deployments:
● my-atomic:fedora-atomic/25/x86_64/docker-host
       Version: 25.1 (2017-02-07 05:34:46)
        Commit: 15b70198b8ec7fd54271f9672578544ff03d1f61df8d7f0fa262ff7519438eb6
        OSName: fedora-atomic

  fedora-atomic:fedora-atomic/25/x86_64/docker-host
       Version: 25.51 (2017-01-30 20:09:59)
        Commit: f294635a1dc62d9ae52151a5fa897085cac8eaa601c52e9a4bc376e9ecee11dd
        OSName: fedora-atomic

Now you have the Updated Tree.

Shout-Out for the following folks:

My future post will have customizing packages (includes addition/deletion) for OSTree.

PyCon India 2016

Heya! First of all I’m really sorry for such a delay with PyCon India 2016 blog post.

It was my first PyCon India. I have always wanted to attend such nice conference. But somehow I was probably going to miss it because of fund. It was only DGPLUG for which I could attend PyCon India 2016. They made sure about my travelling from Pune to Delhi and my accommodation. Have a look at 🙂 https://kushaldas.in/posts/dgplug-contributor-grant-recipient-trishna-guha.html.

DAY 1 started with Workshop and Openspace. I stayed in the Openspace since I didn’t buy ticket for workshop. I came to know about an useful project Ansible-Container from Shubham Miglani. We started hacking on the project. I also created an issue for the project but couldn’t work on the patch since the patch was already fixed and out with next release.

I met many of the faces whom I used to know on IRC/Twitter only. It was really exciting and my first day of the conference was over.

The main conference started from Day 2. It started with Keynote by Baishampayan Ghose. He gave a nice keynote on building bridges, distributed architecture and functional testing. Then the other talks carried on.  I was at the Red Hat and PyLadies booth most of the time.

We had a keynote by VanL as well about software design and failure which was great.

Many people came down with the interest about internship at Red Hat and joining PyLadies Community. We had DGPLUG + PyCon India Dinner at BBQ- Delhi later at night.

Day 3 started with Keynote by Andreas Muller on Machine Learning. Then multiple tracks talks carried on. I really enjoyed microservices talk by Ratnadeep Debnath. We had DGPLUG staircase meeting. There after we had open discussion on PyLadies – Diversity and FOSS community with Paul Everitt, Dmitry Filippov, VanL.

There was Red Hat sponsored talk by Kushal Das.

Oh yes I gave a short lightning talk on Project Atomic and Fedora Infrastructure Application Bodhi as well. The day ended with DGPLUG photo shoot.

We had dinner outside and headed back to Pune that night.

Below are the few photos I have :-).

ctqgad2vmaag13g

29313770323_05feccdc10_z

For more photos visit: https://www.flickr.com/photos/sayanchowdhury/albums/72157674406421245

We are going to have another conference on Python really soon Pycon Pune 2017 :-).

Talking to Docker daemon of Fedora Atomic Host

This post is now deprecated, Please follow the more enhanced one: http://www.projectatomic.io/blog/2017/01/remote-access-docker-daemon

This post will describe how to use Docker daemon of Fedora Atomic host remotely.  Note that we are also going to secure the Docker daemon since we are connecting via Network which we will be doing with TLS.

TLS (Transport Layer Security) provides communication security over computer network. We will create client cert and server cert to secure our Docker daemon. OpenSSL will be used to to create the cert keys for establishing TLS connection.

I am using Fedora Atomic host as remote and workstation as my present host.

Thanks to Chris Houseknecht for writing an Ansible role which creates all the certs required automatically, so that there is no need to issue openssl commands manually. Here is the Ansible role repository: https://github.com/ansible/role-secure-docker-daemon. Clone it to your present working host.

$ mkdir secure-docker-daemon
$ cd secure-docker-daemon
$ git clone https://github.com/ansible/role-secure-docker-daemon.git
$ touch ansible.cfg inventory secure-docker-daemon.yml
$ ls 
ansible.cfg  inventory  role-secure-docker-daemon  secure-docker-daemon.yml

$ vim ansible.cfg
[defaults]
inventory=inventory
remote_user='USER_OF_ATOMIC_HOST'

$ vim inventory 
[serveratomic]
'IP_OF_ATOMIC_HOST' ansible_ssh_private_key_file='PRIVATE_KEY_FILE'

$ vim secure-docker-daemon.yml
---
- name: Secure Docker daemon for Atomic host
  hosts: serveratomic
  gather_facts: no
  become: yes
  roles:
    - role: role-secure-docker-daemon
      dds_host: 'IP_OF_ATOMIC_HOST'
      dds_server_cert_path: /etc/docker
      dds_restart_docker: no

Replace ‘USER_OF_ATOMIC_HOST’ with the user of your Atomic host, ‘IP_OF_ATOMIC_HOST’ with the IP of your Atomic host, ‘PRIVATE_KEY_FILE’ with the ssh private key file of your workstation.

Now we will run the ansible playbook. This will create client and server certs on the Atomic host.

$ ansible-playbook secure-docker-daemon.yml

Now ssh to your Atomic host.

We will copy the client certs created on the Atomic host to the workstation. You will find the client certs file in ~/.docker directory as root user. Now create ~/.docker directory on your workstation for regular user and copy the client certs there. You can use scp to copy the cert files from Atomic host to Workstation or do it manually ;-).

We are going to append some Environment variables in the ~/.bashrc file of the workstation for regular user.

$ vim ~/.bashrc
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=~/.docker/
export DOCKER_HOST=tcp://IP_OF_ATOMIC_HOST:2376

Docker’s port is 2376 for TLS (secured port).

Now go your Atomic host. We will add tls options to docker command on atomic host.

Add –tlsverify –tlscacert=/etc/docker/ca.pem –tlscert=/etc/docker/server-cert.pem –tlskey=/etc/docker/server-key.pem -H=0.0.0.0:2376 -H=unix:///var/run/docker.sock in the /etc/sysconfig/docker file.

$ vi /etc/sysconfig/docker
OPTIONS='--selinux-enabled --log-driver=journald --tlsverify --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server-cert.pem --tlskey=/etc/docker/server-key.pem -H=0.0.0.0:2376 -H=unix:///var/run/docker.sock'

We will need to reload and restart the docker daemon.

$ sudo systemctl docker-reload
$ sudo systemctl restart docker.service

Reboot both of your Atomic host and Workstation.

So now if you try running any docker command as regular user on your workstation it will talk to the docker daemon of the Atomic host and execute the command there. You do not need to manually ssh and issue docker command on your Atomic host :-).

Here are some screenshots for demonstration:

Atomic Host:

screenshot-from-2016-12-09-10-27-47

screenshot-from-2016-12-09-10-29-46

screenshot-from-2016-12-09-10-26-31

Workstation:

fotoflexer_photo

screenshot-from-2016-12-09-10-26-35

 

Containerization and Deployment of Application on Atomic Host using Ansible Playbook

This article describes how to build Docker image and deploy containerized application on Atomic host (any Remote host) using Ansible Playbook.

Building Docker image for an application and run container/cluster of containers is nothing new. But the idea is to automate the whole process and this is where Ansible playbooks come in to play.

Note that you can use Cloud/Workstation based Image to execute the following task. Here I am issuing the commands on Fedora Workstation.

Let’s see How to automate the containerization and deployment process for a simple Flask application:

We are going to deploy container on Fedora Atomic host.

First, Let’s Create a simple Flask Hello-World Application.

This is the Directory structure of the entire Application:

flask-helloworld/
├── ansible
│   ├── ansible.cfg
│   ├── inventory
│   └── main.yml
├── Dockerfile
└── flask-helloworld
    ├── hello_world.py
    ├── static
    │   └── style.css
    └── templates
        ├── index.html
        └── master.html

hello_world.py

from flask import Flask, render_template

APP = Flask(__name__)

@APP.route('/')
def index():
    return render_template('index.html')

if __name__ == '__main__':
    APP.run(debug=True, host='0.0.0.0')

static/style.css

body {
  background: #F8A434;
  font-family: 'Lato', sans-serif;
  color: #FDFCFB;
  text-align: center;
  position: relative;
  bottom: 35px;
  top: 65px;
}
.description {
  position: relative;
  top: 55px;
  font-size: 50px;
  letter-spacing: 1.5px;
  line-height: 1.3em;
  margin: -2px 0 45px;
}

templates/master.html

<!doctype html>
<html>
<head>
    {% block head %}
    <title>{% block title %}{% endblock %}</title>
    {% endblock %}
    												<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" crossorigin="anonymous">
    												<link href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.3/css/font-awesome.min.css" rel="stylesheet" integrity="sha384-T8Gy5hrqNKT+hzMclPo118YTQO6cYprQmhrYwIiQ/3axmI1hQomh7Ud2hPOy8SP1" crossorigin="anonymous">
    												<link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">
    												<link href='http://fonts.googleapis.com/css?family=Lato:400,700' rel='stylesheet' type='text/css'>

</head>
<body>
<div id="container">
    {% block content %}
    {% endblock %}</div>
</body>
</html>

templates/index.html

{% extends "master.html" %}

{% block title %}Welcome to Flask App{% endblock %}

{% block content %}
<div class="description">

Hello World</div>
{% endblock %}

Let’s write the Dockerfile.

FROM fedora
MAINTAINER Trishna Guha<tguha@redhat.com>

RUN dnf -y update && dnf -y install python-flask python-jinja2 && dnf clean all
RUN mkdir -p /app

COPY files/ /app/
WORKDIR /app

ENTRYPOINT ["python"]
CMD ["hello_world.py"]

Now we will work on Ansible playbook for our application that deals with the automation part:

Create inventory file:

[atomic]
IP_ADDRESS_OF_HOST ansible_ssh_private_key_file=<'PRIVATE_KEY_FILE'>

Replace IP_ADDRESS_OF_HOST with the IP address of the atomic/remote host and ‘PRIVATE_KEY_FILE’ with your private key file.

Create ansible.cfg file:

[defaults]
inventory=inventory
remote_user=USER

[privilege_escalation]
become_method=sudo
become_user=root

Replace USER with the user of your remote host.

Create main.yml file:

---
- name: Deploy Flask App
  hosts: atomic
  become: yes

  vars:
    src_dir: [Source Directory]
    dest_dir: [Destination Directory]

  tasks:
    - name: Create Destination Directory
      file:
       path: "{{ dest_dir }}/files"
       state: directory
       recurse: yes

    - name: Copy Dockerfile to host
      copy:
       src: "{{ src_dir }}/Dockerfile"
       dest: "{{ dest_dir }}"

    - name: Copy Application to host
      copy:
       src: "{{ src_dir }}/flask-helloworld/"
       dest: "{{ dest_dir }}/files/"

    - name: Make sure that the current directory is {{ dest_dir }}
      command: cd {{ dest_dir }}

    - name: Build Docker Image
      command: docker build --rm -t fedora/flask-app:test -f "{{ dest_dir }}/Dockerfile" "{{ dest_dir }}"

    - name: Run Docker Container
      command: docker run -d --name helloworld -p 5000:5000 fedora/flask-app:test
...

Replace [Source Directory] in src_dir field in main.yml with your /path/to/src_dir of your current host.

Replace [Destination Directory] in dest_dir field in main.yml with your /path/to/dest_dir of your remote atomic host.

Now simply run $ ansible-playbook main.yml :).  To verify if the application is running issue this command $ curl http://localhost:5000 on your atomic/remote host.

You can also manage your containers running on remote host using Cockpit. Check this article to know how to use Cockpit to manage your containers: https://fedoramagazine.org/deploy-containers-atomic-host-ansible-cockpit

fotoflexer_photo

screenshot-from-2016-10-21-18-52-45

Here is the repository of the above example:  https://github.com/trishnaguha/fedora-cloud-ansible/tree/master/examples/flask-helloworld

My future post will be related to ansible-container where I will describe how we can build Docker image and orchestrate container without writing any Dockerfile :).

What is if __name__ == ‘__main__’ ?

 

Module is simply Python file that has .py extension. Module can contain variables, functions, classes that can be reused.

In order to use module we need to import the module using import command. Check the full list of built-in modules in Python here https://docs.python.org/3.6/library.

The first time a module is loaded in to running Python script, it is initialized by executing the code in the module once. To know various ways of importing modules visit here: https://docs.python.org/3.6/tutorial/modules.html

if __name__ == ‘__main__’:

We see if __name__ == ‘__main__’: quite often. Let’s see what this actually is.

__name__ is global variable in Python that exists in all namespaces. It is attribute of module. It is basically the name of the module as str (string) type.

Show Me Code:

Create a file named ‘mymath.py’ and type the following code and save it. We have defined a simple mathematical square method here.

screenshot-from-2016-09-30-12-51-33

Now create another file named ‘result.py’ in the same directory and type the following code and save it.

screenshot-from-2016-09-30-12-57-10

Now on terminal run the program with ‘python3 result.py’
fotoflexer_photo

Here we have defined a method in a module and using it in another file.

Now let’s look into if __name__ == ‘__main__’:

Open the ‘mymath.py’ file and edit it as given in following:

screenshot-from-2016-09-30-13-56-50

Leave ‘result.py’ unchanged.

Now on your terminal run ‘result.py’. 

fotoflexer_photo1

Here we have imported the module mymath. The variable __name__ is set to the name of the module that is imported.

Now on terminal run ‘mymath.py’

fotoflexer_photo3

We have run the file mymath.py as program itself. And you can see here the variable __name__ is set to the string “__main__”.
And we have checked if __name__ == “__main__” is True execute the following instructions which means if the file is run as standalone program itself execute the following instructions.

If you do  print(type(__name__)) in the program, you will see it returns ‘str’ (string) type.

Happy Coding!