initial commit
This commit is contained in:
Chris Cowley
commit
ca6a4d45d1
113 changed files with 10501 additions and 0 deletions
182
content/blog/automated-glusterfs/index.md
Normal file
182
content/blog/automated-glusterfs/index.md
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
---
|
||||
date: 2013-06-23
|
||||
title: Automated GlusterFS
|
||||
category: devops
|
||||
---
|
||||
|
||||
{% img right
|
||||
[https://www.hastexo.com/system/files/imagecache/sidebar/20120221105324808-f2df3ea3e3aeab8\\\_250\\\_0.png](https://www.hastexo.com/system/files/imagecache/sidebar/20120221105324808-f2df3ea3e3aeab8\_250\_0.png)
|
||||
%} As I promised on Twitter, this is how I automate a GlusterFS
|
||||
deployment. I\'m making a few assumptions here:
|
||||
|
||||
- I am using CentOS 6, so should work on RHEL 6 and Scientific Linux 6
|
||||
too. Others may work, but YMMV.
|
||||
- As I use XFS, RHEL users will need the *Scalable Storage* option.
|
||||
Ext4 will work, but XFS is recommended.
|
||||
- That you have a way of automating your base OS installation. My
|
||||
personal preference is to use
|
||||
[Razor](https://github.com/puppetlabs/Razor).
|
||||
- You have a system with at least a complete spare disk dedicated to a
|
||||
GlusterFS brick. That is the best way to run GlusterFS anyway.
|
||||
- You have 2 nodes and want to replicate the data
|
||||
- You have a simple setup with only a single network, because I am
|
||||
being lazy. As a proof-of concept this is fine. Modifying this for
|
||||
second network is quite easy, just change the IP address in you use.
|
||||
|
||||
{% img
|
||||
<https://docs.google.com/drawings/d/1XA7GH3a4BL1uszFXrSsZjysi59Iinh-0RmhqdDbt7QQ/pub?w=673&h=315>
|
||||
\'simple gluster architecture\' %}
|
||||
|
||||
The diagram above shows the basic layout of what to start from in terms
|
||||
of hardware. In terms of software, you just need a basic CentOS 6
|
||||
install and to have Puppet working.
|
||||
|
||||
I use a pair of Puppet modules (both in the Forge):
|
||||
[thias/glusterfs](https://forge.puppetlabs.com/thias/glusterfs) and
|
||||
[puppetlabs/lvm](https://forge.puppetlabs.com/puppetlabs/lvm). The
|
||||
GlusterFS module CAN do the LVM config, but that strikes me as not the
|
||||
best idea. The UNIX philosophy of \"do one job well\" holds up for
|
||||
Puppet modules as well. You will also need my
|
||||
[yumrepos](https://github.com/chriscowley/puppet-yumrepos) module.
|
||||
|
||||
Clone those 3 modules into your modules directory:
|
||||
|
||||
cd /etc/puppet/
|
||||
git clone git://github.com/chriscowley/puppet-yumrepos.git modules/yumrepos
|
||||
puppet module install puppetlabs/lvm --version 0.1.2
|
||||
puppet module install thias/glusterfs --version 0.0.3
|
||||
|
||||
I have specified the versions as that is what was the latest at the time
|
||||
of writing. You should be able to take the latest as well, but comment
|
||||
with any differences if any. That gives the core of what you need so you
|
||||
can now move on to you `nodes.pp`.
|
||||
|
||||
class basenode {
|
||||
class { 'yumrepos': }
|
||||
class { 'yumrepos::epel': }
|
||||
}
|
||||
|
||||
class glusternode {
|
||||
class { 'basenode': }
|
||||
class { 'yumrepos::gluster': }
|
||||
|
||||
volume_group { "vg0":
|
||||
ensure => present,
|
||||
physical_volumes => "/dev/sdb",
|
||||
require => Physical_volume["/dev/sdb"]
|
||||
}
|
||||
physical_volume { "/dev/sdb":
|
||||
ensure => present
|
||||
}
|
||||
logical_volume { "gv0":
|
||||
ensure => present,
|
||||
require => Volume_group['vg0'],
|
||||
volume_group => "vg0",
|
||||
size => "7G",
|
||||
}
|
||||
file { [ '/export', '/export/gv0']:
|
||||
seltype => 'usr_t',
|
||||
ensure => directory,
|
||||
}
|
||||
package { 'xfsprogs': ensure => installed
|
||||
}
|
||||
filesystem { "/dev/vg0/gv0":
|
||||
ensure => present,
|
||||
fs_type => "xfs",
|
||||
options => "-i size=512",
|
||||
require => [Package['xfsprogs'], Logical_volume['gv0'] ],
|
||||
}
|
||||
|
||||
mount { '/export/gv0':
|
||||
device => '/dev/vg0/gv0',
|
||||
fstype => 'xfs',
|
||||
options => 'defaults',
|
||||
ensure => mounted,
|
||||
require => [ Filesystem['/dev/vg0/gv0'], File['/export/gv0'] ],
|
||||
}
|
||||
class { 'glusterfs::server':
|
||||
peers => $::hostname ? {
|
||||
'gluster1' => '192.168.1.38', # Note these are the IPs of the other nodes
|
||||
'gluster2' => '192.168.1.84',
|
||||
},
|
||||
}
|
||||
glusterfs::volume { 'gv0':
|
||||
create_options => 'replica 2 192.168.1.38:/export/gv0 192.168.1.84:/export/gv0',
|
||||
require => Mount['/export/gv0'],
|
||||
}
|
||||
}
|
||||
|
||||
node 'gluster1' {
|
||||
include glusternode
|
||||
file { '/var/www': ensure => directory }
|
||||
glusterfs::mount { '/var/www':
|
||||
device => $::hostname ? {
|
||||
'gluster1' => '192.168.1.84:/gv0',
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
node 'gluster2' {
|
||||
include glusternode
|
||||
file { '/var/www': ensure => directory }
|
||||
glusterfs::mount { '/var/www':
|
||||
device => $::hostname ? {
|
||||
'gluster2' => '192.168.1.38:/gv0',
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
What does all that do? Starting from the top:
|
||||
|
||||
- The `basenode` class does all your basic configuration across all
|
||||
your hosts. Mine actually does a lot more, but these are the
|
||||
relevant parts.
|
||||
- The `glusternode` class is shared between all your GlusterFS nodes.
|
||||
This is where all your Server configuration is.
|
||||
- Configures LVM
|
||||
- Defines the Volume Group \"vg0\" with the Physical Volume
|
||||
`/dev/sdb`
|
||||
- Creates a Logical Volume \"gv0\" for GlusterFS use and make it
|
||||
7GB
|
||||
- Configures the file system
|
||||
- Creates the directory `/export/gv0`
|
||||
- Formats the LV created previously with XFS (installs the package
|
||||
if necessary)
|
||||
- Mounts the LV at `/export/gv0`
|
||||
|
||||
This is now all ready for the GlusterFS module to do its stuff. All this
|
||||
happens in those last two sections.
|
||||
|
||||
- The class `glusterfs::Server` sets up the peering between the two
|
||||
hosts. This will actually generate a errors, but do not worry. This
|
||||
because gluster1 successfully peers with gluster2. As a result
|
||||
gluster2 fails to peer with gluster1 as they are already peered.
|
||||
- Now `glusterfs::volume` creates a replicated volume, having first
|
||||
ensured that the LV is mounted correctly.
|
||||
- All this is then included in the node declarations for `gluster1`
|
||||
and `gluster2`.
|
||||
|
||||
All that creates the server very nicely. It will need a few passes to
|
||||
get everything in place, while giving a few red herring errors. It
|
||||
should would however, all the errors are there in the README for the
|
||||
GlusterFS module in PuppetForge, so do not panic.
|
||||
|
||||
A multi-petabyte scale-out storage system is pretty useless if the data
|
||||
cannot be read by anything. So lets use those nodes and mount the
|
||||
volume. This could also be a separate node (but once again I am being
|
||||
lazy) the process will be exactly the same.
|
||||
|
||||
- Create a mount point for it ( \`file {\'/var/www\': ensure =\>
|
||||
directory }
|
||||
- Define your `glusterfs::mount` using any of the hosts in the
|
||||
cluster.
|
||||
|
||||
Voila, that should all pull together and give you a fully automated
|
||||
GlusterFS set up. The sort of scale that GlusterFS can reach makes this
|
||||
sort of automation absolutely essential in my opinion. This should be
|
||||
relatively easy to convert to Chef or Ansible, whatever takes your
|
||||
fancy. I have just used Puppet because of my familiarity with it.
|
||||
|
||||
This is only one way of doing this, and I make no claims to being the
|
||||
most adept Puppet user in the world. All I hope to achieve is that
|
||||
someone finds this useful. Courteous comments welcome.
|
||||
Loading…
Add table
Add a link
Reference in a new issue