182 lines
6.8 KiB
Markdown
182 lines
6.8 KiB
Markdown
---
|
|
date: 2013-06-23
|
|
title: Automated GlusterFS
|
|
category: devops
|
|
---
|
|
|
|
{% img right
|
|
[https://www.hastexo.com/system/files/imagecache/sidebar/20120221105324808-f2df3ea3e3aeab8\\\_250\\\_0.png](https://www.hastexo.com/system/files/imagecache/sidebar/20120221105324808-f2df3ea3e3aeab8\_250\_0.png)
|
|
%} As I promised on Twitter, this is how I automate a GlusterFS
|
|
deployment. I\'m making a few assumptions here:
|
|
|
|
- I am using CentOS 6, so should work on RHEL 6 and Scientific Linux 6
|
|
too. Others may work, but YMMV.
|
|
- As I use XFS, RHEL users will need the *Scalable Storage* option.
|
|
Ext4 will work, but XFS is recommended.
|
|
- That you have a way of automating your base OS installation. My
|
|
personal preference is to use
|
|
[Razor](https://github.com/puppetlabs/Razor).
|
|
- You have a system with at least a complete spare disk dedicated to a
|
|
GlusterFS brick. That is the best way to run GlusterFS anyway.
|
|
- You have 2 nodes and want to replicate the data
|
|
- You have a simple setup with only a single network, because I am
|
|
being lazy. As a proof-of concept this is fine. Modifying this for
|
|
second network is quite easy, just change the IP address in you use.
|
|
|
|
{% img
|
|
<https://docs.google.com/drawings/d/1XA7GH3a4BL1uszFXrSsZjysi59Iinh-0RmhqdDbt7QQ/pub?w=673&h=315>
|
|
\'simple gluster architecture\' %}
|
|
|
|
The diagram above shows the basic layout of what to start from in terms
|
|
of hardware. In terms of software, you just need a basic CentOS 6
|
|
install and to have Puppet working.
|
|
|
|
I use a pair of Puppet modules (both in the Forge):
|
|
[thias/glusterfs](https://forge.puppetlabs.com/thias/glusterfs) and
|
|
[puppetlabs/lvm](https://forge.puppetlabs.com/puppetlabs/lvm). The
|
|
GlusterFS module CAN do the LVM config, but that strikes me as not the
|
|
best idea. The UNIX philosophy of \"do one job well\" holds up for
|
|
Puppet modules as well. You will also need my
|
|
[yumrepos](https://github.com/chriscowley/puppet-yumrepos) module.
|
|
|
|
Clone those 3 modules into your modules directory:
|
|
|
|
cd /etc/puppet/
|
|
git clone git://github.com/chriscowley/puppet-yumrepos.git modules/yumrepos
|
|
puppet module install puppetlabs/lvm --version 0.1.2
|
|
puppet module install thias/glusterfs --version 0.0.3
|
|
|
|
I have specified the versions as that is what was the latest at the time
|
|
of writing. You should be able to take the latest as well, but comment
|
|
with any differences if any. That gives the core of what you need so you
|
|
can now move on to you `nodes.pp`.
|
|
|
|
class basenode {
|
|
class { 'yumrepos': }
|
|
class { 'yumrepos::epel': }
|
|
}
|
|
|
|
class glusternode {
|
|
class { 'basenode': }
|
|
class { 'yumrepos::gluster': }
|
|
|
|
volume_group { "vg0":
|
|
ensure => present,
|
|
physical_volumes => "/dev/sdb",
|
|
require => Physical_volume["/dev/sdb"]
|
|
}
|
|
physical_volume { "/dev/sdb":
|
|
ensure => present
|
|
}
|
|
logical_volume { "gv0":
|
|
ensure => present,
|
|
require => Volume_group['vg0'],
|
|
volume_group => "vg0",
|
|
size => "7G",
|
|
}
|
|
file { [ '/export', '/export/gv0']:
|
|
seltype => 'usr_t',
|
|
ensure => directory,
|
|
}
|
|
package { 'xfsprogs': ensure => installed
|
|
}
|
|
filesystem { "/dev/vg0/gv0":
|
|
ensure => present,
|
|
fs_type => "xfs",
|
|
options => "-i size=512",
|
|
require => [Package['xfsprogs'], Logical_volume['gv0'] ],
|
|
}
|
|
|
|
mount { '/export/gv0':
|
|
device => '/dev/vg0/gv0',
|
|
fstype => 'xfs',
|
|
options => 'defaults',
|
|
ensure => mounted,
|
|
require => [ Filesystem['/dev/vg0/gv0'], File['/export/gv0'] ],
|
|
}
|
|
class { 'glusterfs::server':
|
|
peers => $::hostname ? {
|
|
'gluster1' => '192.168.1.38', # Note these are the IPs of the other nodes
|
|
'gluster2' => '192.168.1.84',
|
|
},
|
|
}
|
|
glusterfs::volume { 'gv0':
|
|
create_options => 'replica 2 192.168.1.38:/export/gv0 192.168.1.84:/export/gv0',
|
|
require => Mount['/export/gv0'],
|
|
}
|
|
}
|
|
|
|
node 'gluster1' {
|
|
include glusternode
|
|
file { '/var/www': ensure => directory }
|
|
glusterfs::mount { '/var/www':
|
|
device => $::hostname ? {
|
|
'gluster1' => '192.168.1.84:/gv0',
|
|
}
|
|
}
|
|
}
|
|
|
|
node 'gluster2' {
|
|
include glusternode
|
|
file { '/var/www': ensure => directory }
|
|
glusterfs::mount { '/var/www':
|
|
device => $::hostname ? {
|
|
'gluster2' => '192.168.1.38:/gv0',
|
|
}
|
|
}
|
|
}
|
|
|
|
What does all that do? Starting from the top:
|
|
|
|
- The `basenode` class does all your basic configuration across all
|
|
your hosts. Mine actually does a lot more, but these are the
|
|
relevant parts.
|
|
- The `glusternode` class is shared between all your GlusterFS nodes.
|
|
This is where all your Server configuration is.
|
|
- Configures LVM
|
|
- Defines the Volume Group \"vg0\" with the Physical Volume
|
|
`/dev/sdb`
|
|
- Creates a Logical Volume \"gv0\" for GlusterFS use and make it
|
|
7GB
|
|
- Configures the file system
|
|
- Creates the directory `/export/gv0`
|
|
- Formats the LV created previously with XFS (installs the package
|
|
if necessary)
|
|
- Mounts the LV at `/export/gv0`
|
|
|
|
This is now all ready for the GlusterFS module to do its stuff. All this
|
|
happens in those last two sections.
|
|
|
|
- The class `glusterfs::Server` sets up the peering between the two
|
|
hosts. This will actually generate a errors, but do not worry. This
|
|
because gluster1 successfully peers with gluster2. As a result
|
|
gluster2 fails to peer with gluster1 as they are already peered.
|
|
- Now `glusterfs::volume` creates a replicated volume, having first
|
|
ensured that the LV is mounted correctly.
|
|
- All this is then included in the node declarations for `gluster1`
|
|
and `gluster2`.
|
|
|
|
All that creates the server very nicely. It will need a few passes to
|
|
get everything in place, while giving a few red herring errors. It
|
|
should would however, all the errors are there in the README for the
|
|
GlusterFS module in PuppetForge, so do not panic.
|
|
|
|
A multi-petabyte scale-out storage system is pretty useless if the data
|
|
cannot be read by anything. So lets use those nodes and mount the
|
|
volume. This could also be a separate node (but once again I am being
|
|
lazy) the process will be exactly the same.
|
|
|
|
- Create a mount point for it ( \`file {\'/var/www\': ensure =\>
|
|
directory }
|
|
- Define your `glusterfs::mount` using any of the hosts in the
|
|
cluster.
|
|
|
|
Voila, that should all pull together and give you a fully automated
|
|
GlusterFS set up. The sort of scale that GlusterFS can reach makes this
|
|
sort of automation absolutely essential in my opinion. This should be
|
|
relatively easy to convert to Chef or Ansible, whatever takes your
|
|
fancy. I have just used Puppet because of my familiarity with it.
|
|
|
|
This is only one way of doing this, and I make no claims to being the
|
|
most adept Puppet user in the world. All I hope to achieve is that
|
|
someone finds this useful. Courteous comments welcome.
|