Ansible/Full Stack Playbook: Difference between revisions

From charlesreid1
 
(16 intermediate revisions by the same user not shown)
Line 10: Line 10:
=Setup=
=Setup=


==Vagrant multi-machine setup==
See [[Ansible/Full Stack Playbook/Vagrant Setup]] for details of setting up Vagrant to test out this configuration


Here we walk through how to get set up with [[Vagrant]] before writing and testing the playbook.
=Full Stack Playbook=


Note: before running any vagrant boxes, destroy and clean up prior boxes via
==Overview of Full Stack==


<pre>
===Services===
vagrant destroy ---force
 
</pre>
For this particular full stack deployment, we have the following services:
 
* Django app + http server
* Nginx web server
* Celery task queue
* RabbitMQ (Celery backend)
* Postgres (persistent store)
 
===Environments===


===Vagrantfile===
We have three environments to deploy to:


Create a Vagrantfile with 3 hosts:
* vagrant (local testing)
* staging (testing)
* production


'''<code>Vagrantfile</code>'''
===Architecture===


<pre>
Here is the architecture used for the above apps:
VAGRANTFILE_API_VERSION = "2"


Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
* Web application is run on multiple hosts for better performance, with a load balancer in front
* Task queue servers are run on multiple hosts for better performance
* Celery, RabbitMQ, Postgres all on separate servers
* 2 Postgres hosts - primary and replica


  # Use the same key for each machine
That is a total of 10 hosts (1 load balancer, 3 web servers, 3 task queues, 1 mq server, 2 database servers).
  config.ssh.insert_key = false


  config.vm.define "vagrant1" do |vagrant1|
In the production environment: 10 separate hosts
    vagrant1.vm.box = "ubuntu/xenial64"
    vagrant1.vm.network "forwarded_port", guest: 80, host: 8080
    vagrant1.vm.network "forwarded_port", guest: 443, host: 8443
  end
  config.vm.define "vagrant2" do |vagrant2|
    vagrant2.vm.box = "ubuntu/xenial64"
    vagrant2.vm.network "forwarded_port", guest: 80, host: 8081
    vagrant2.vm.network "forwarded_port", guest: 443, host: 8444
  end
  config.vm.define "vagrant3" do |vagrant3|
    vagrant3.vm.box = "ubuntu/xenial64"
    vagrant3.vm.network "forwarded_port", guest: 80, host: 8082
    vagrant3.vm.network "forwarded_port", guest: 443, host: 8445
  end
end
</pre>


Note that without <code>config.ssh.insert_key=false</code> each machine would use its own SSH key, which would be a bit of a headache. With this directive, we can define a single SSH key in our ansible config file.
In the staging environment: use fewer hosts (only 2 hosts: web server and task queue on one host, rabbitmq and postgres on another)


===Ansible config file===
In the vagrant environment: use 3 servers (1 for web app, 1 for task queue, and 1 for postgres)


Now the <code>ansible.cfg</code> file should be modified to configure Ansible. Most important is the location of the private key:
==Ansible hosts file==


<pre>
{{Main|Ansible/Hosts}}
[defaults]
inventory = inventory
remote_user = vagrant
private_key_file = ~/.vagrant.d/insecure_private_key
host_key_checking = False
</pre>


===Run vagrant===
Note that this initial hosts file (below) can be streamlined using suggestions from [[Ansible/Groups]], [[Ansible/Group Variables]], and [[Ansible/Host Naming]], so consider it a first pass.


Run the vagrant machines with
Using the information given above about the architecture, we have the following hosts file for our Django app:


<pre>
<pre>
vagrant up
[production]
</pre>
delaware.example.com
georgia.example.com
maryland.example.com
newhampshire.example.com
newjersey.example.com
newyork.example.com
northcarolina.example.com
pennsylvania.example.com
rhodeisland.example.com
virginia.example.com
 
[staging]
redblue.example.com
orangered.example.com


See details about SSH ports using
[vagrant]
vagrant1 ansible_host=127.0.0.1 ansible_port=2222
vagrant2 ansible_host=127.0.0.1 ansible_port=2200
vagrant3 ansible_host=127.0.0.1 ansible_port=2201


<pre>
[lb]
vagrant ssh-config
delaware.example.com
</pre>


which will output something like this
[web]
georgia.example.com
newhampshire.example.com
newjersey.example.com
redblue.example.com
vagrant1


<pre>
[task]
Host vagrant1
newyork.example.com
  HostName 127.0.0.1
northcarolina.example.com
  User vagrant
maryland.example.com
  Port 2222
redblue.example.com
  UserKnownHostsFile /dev/null
vagrant2
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/lorin/.vagrant.d/insecure_private_key
  IdentitiesOnly yes
  LogLevel FATAL


Host vagrant2
[rabbitmq]
  HostName 127.0.0.1
pennsylvania.example.com
  User vagrant
orangered.example.com
  Port 2200
vagrant3
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/lorin/.vagrant.d/insecure_private_key
  IdentitiesOnly yes
  LogLevel FATAL


Host vagrant3
[db]
  HostName 127.0.0.1
rhodeisland.example.com
  User vagrant
virginia.example.com
  Port 2201
orangered.example.com
  UserKnownHostsFile /dev/null
vagrant3
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/lorin/.vagrant.d/insecure_private_key
  IdentitiesOnly yes
  LogLevel FATAL
</pre>
</pre>


===Create Ansible inventory file===
Note that we make the vagrant machines part of the vagrant group when their behavioral inventory parameters are specified, but we can later add them to other groups by adding their short name (only) to the group.


Once we know the SSH port for each machine, we can create an inventory file.
In this way, we've defined which machines are in each of the 3 environments (production, staging, vagrant) and which one runs which service.


A basic <code>playbook/hosts</code> file would contain:
===Passing credentials via the host file===
 
<pre>
vagrant1 ansible_host=127.0.0.1 ansible_port=2222
vagrant2 ansible_host=127.0.0.1 ansible_port=2200
vagrant3 ansible_host=127.0.0.1 ansible_port=2201
</pre>


This, together with the playbook, tells Ansible how to reach and how to connect to each of the remote hosts. Now we can run a test command on the machines with Ansible:
Now that we're deploying these services, we have more details to take care of:


<pre>
* Web servers must have hostname, port, username, password of Postgres, and name of database
ansible vagrant2 -a "ip addr show dev eth0"
* Task queues must have hostname, port, username, password of Postgres, and name of database
</pre>
* Web servers must have hostname, port of RabbitMQ server
* task queues must have hostname, port of RabbitMQ server
* Postgres primary must have hostname, port, username, password of replica Postgres server (production only)


===Group the Ansible hosts file===
This information can be properly distributed by using groups, and assigning different variable values for each different group.


As the hosts file gets more complicated, we may want to perform actions only on a group of machines. Toward this purpose we can group the machines under <code>[headings]</code> that indicate they are part of a group.
Specifically, things break down best by environment (vagrant, staging, and production).  


If we added many more hosts to the hosts file, we would want to group the Vagrant machines, so the host file would look like this:
Here's what the Vagrant hosts file looks like, grouping by environment:


<pre>
<pre>
ontario.example.com
[all:vars]
newhampshire.example.com
ntp_server=ntp.ubuntu.com
maryland.example.com
virginia.example.com
newyork.example.com
quebec.example.com
rhodeisland.example.com
 
[vagrant]
vagrant1 ansible_host=127.0.0.1 ansible_port=2222
vagrant2 ansible_host=127.0.0.1 ansible_port=2200
vagrant3 ansible_host=127.0.0.1 ansible_port=2201
</pre>


===Note on inventory files===
[production:vars]
db_primary_host=rhodeisland.example.com
db_primary_port=5432
db_replica_host=virginia.example.com
db_name=widget_production
db_user=widgetuser
db_password=pFmMxcyD;Fc6)6
rabbitmq_host=pennsylvania.example.com
rabbitmq_port=5672


Inventory files can define a number of parameters for each host. Above, we define the host and port, but other variables are also available:
[staging:vars]
db_primary_host=orangered.example.com
db_primary_port=5432
db_name=widget_staging
db_user=widgetuser
db_password=L@4Ryz8cRUXedj
rabbitmq_host=redblue.example.com
rabbitmq_port=5672


* ansible_host
[vagrant:vars]
* ansible_port
db_primary_host=vagrant3
* ansible_user
db_primary_port=5432
* ansible_password
db_name=widget_vagrant
* ansible_private-key_file
db_user=widgetuser
* ansible_shell_type
db_password=password
* ansible_python_interpreter
rabbitmq_host=vagrant3
rabbitmq_port=5672
</pre>


(But apparently only ansible_port, ansible_user, ansible_private_key_file,and ansible_shell_type can be changed in Ansible config file???)
=Flags=


===Next steps===
{{AnsibleFlag}}


We will modify this inventory file as needed for the actual full stack application, but this gets you going with a basic multi-machine Vagrantfile.
[[Category:HTTPS]]
[[Category:SSL]]


=Full Stack Overview=
[[Category:Web Server]]
[[Category:Nginx]]
[[Category:Django]]
[[Category:Celery]]
[[Category:RabbitMQ]]
[[Category:Postgres]]

Latest revision as of 19:14, 7 December 2018

Main article: Ansible/Playbooks

This page covers an Ansible playbook for a full stack example. This full stack example will run the following services:

  • Django web server
  • Celery task queue
  • RabbitMQ message queue
  • Postgresql for data storage

Setup

See Ansible/Full Stack Playbook/Vagrant Setup for details of setting up Vagrant to test out this configuration

Full Stack Playbook

Overview of Full Stack

Services

For this particular full stack deployment, we have the following services:

  • Django app + http server
  • Nginx web server
  • Celery task queue
  • RabbitMQ (Celery backend)
  • Postgres (persistent store)

Environments

We have three environments to deploy to:

  • vagrant (local testing)
  • staging (testing)
  • production

Architecture

Here is the architecture used for the above apps:

  • Web application is run on multiple hosts for better performance, with a load balancer in front
  • Task queue servers are run on multiple hosts for better performance
  • Celery, RabbitMQ, Postgres all on separate servers
  • 2 Postgres hosts - primary and replica

That is a total of 10 hosts (1 load balancer, 3 web servers, 3 task queues, 1 mq server, 2 database servers).

In the production environment: 10 separate hosts

In the staging environment: use fewer hosts (only 2 hosts: web server and task queue on one host, rabbitmq and postgres on another)

In the vagrant environment: use 3 servers (1 for web app, 1 for task queue, and 1 for postgres)

Ansible hosts file

Main article: Ansible/Hosts

Note that this initial hosts file (below) can be streamlined using suggestions from Ansible/Groups, Ansible/Group Variables, and Ansible/Host Naming, so consider it a first pass.

Using the information given above about the architecture, we have the following hosts file for our Django app: 

[production]
delaware.example.com
georgia.example.com
maryland.example.com
newhampshire.example.com
newjersey.example.com
newyork.example.com
northcarolina.example.com
pennsylvania.example.com
rhodeisland.example.com
virginia.example.com

[staging]
redblue.example.com
orangered.example.com

[vagrant]
vagrant1 ansible_host=127.0.0.1 ansible_port=2222
vagrant2 ansible_host=127.0.0.1 ansible_port=2200
vagrant3 ansible_host=127.0.0.1 ansible_port=2201

[lb]
delaware.example.com

[web]
georgia.example.com
newhampshire.example.com
newjersey.example.com
redblue.example.com
vagrant1

[task]
newyork.example.com
northcarolina.example.com
maryland.example.com
redblue.example.com
vagrant2

[rabbitmq]
pennsylvania.example.com
orangered.example.com
vagrant3

[db]
rhodeisland.example.com
virginia.example.com
orangered.example.com
vagrant3

Note that we make the vagrant machines part of the vagrant group when their behavioral inventory parameters are specified, but we can later add them to other groups by adding their short name (only) to the group.

In this way, we've defined which machines are in each of the 3 environments (production, staging, vagrant) and which one runs which service.

Passing credentials via the host file

Now that we're deploying these services, we have more details to take care of:

  • Web servers must have hostname, port, username, password of Postgres, and name of database
  • Task queues must have hostname, port, username, password of Postgres, and name of database
  • Web servers must have hostname, port of RabbitMQ server
  • task queues must have hostname, port of RabbitMQ server
  • Postgres primary must have hostname, port, username, password of replica Postgres server (production only)

This information can be properly distributed by using groups, and assigning different variable values for each different group.

Specifically, things break down best by environment (vagrant, staging, and production).

Here's what the Vagrant hosts file looks like, grouping by environment: 

[all:vars]
ntp_server=ntp.ubuntu.com

[production:vars]
db_primary_host=rhodeisland.example.com
db_primary_port=5432
db_replica_host=virginia.example.com
db_name=widget_production
db_user=widgetuser
db_password=pFmMxcyD;Fc6)6
rabbitmq_host=pennsylvania.example.com
rabbitmq_port=5672

[staging:vars]
db_primary_host=orangered.example.com
db_primary_port=5432
db_name=widget_staging
db_user=widgetuser
db_password=L@4Ryz8cRUXedj
rabbitmq_host=redblue.example.com
rabbitmq_port=5672

[vagrant:vars]
db_primary_host=vagrant3
db_primary_port=5432
db_name=widget_vagrant
db_user=widgetuser
db_password=password
rabbitmq_host=vagrant3
rabbitmq_port=5672

Flags


Defcon.png
Ansible
Ansible is a Python library for managing IT and compute infrastructure.

Ansible  · Category:Ansible


Ansible Configuration

Config file: Ansible/Configuration

Secrets and Sensitive Info: Ansible/Security  · Ansible/Vaults


Ansible Playbooks

Basics: Ansible/Playbooks

Examples: Ansible/Nginx Playbook (simple)  · Ansible/Full Stack Playbook (advanced)


Ansible Hosts and Inventory Control

All topics in Ansible Hosts: Ansible/Hosts

Splitting large host files into subfiles: Ansible/Splitting Hosts File

Automatically naming hosts: Ansible/Host Naming

Grouping Hosts: Ansible/Groups

Assigning Variables to Groups: Ansible/Group Variables

Roles: Ansible/Roles


Ansible and Vagrant (Local)

Ansible/Vagrant  · Ansible/Vagrant/Static Inventory  · Ansible/Vagrant/Dynamic Inventory


Ansible and EC2 (Cloud)

Ansible/EC2  · Ansible/EC2/Static Inventory  · Ansible/EC2/Dynamic Inventory


Best Practices

Ansible/Directory Layout  · Ansible/Directory Layout/Details

Ansible/Separate Playbooks by Role

Ansible/Tag Inventory with Environment

Ansible/Group by Distribution

Ansible/Variables and Vaults


Ansible Galaxy

Ansible/Galaxy


Flags  · Template:AnsibleFlag  · e



Retrieved from "https://charlesreid1.com/w/index.php?title=Ansible/Full_Stack_Playbook&oldid=25626"