Resizing raid5 with mdadm, lvm and luks for encryption on debian wheezy

This guide will show you how to setup a simple mdadm, lvm test environment in which you can experiment with changes you want to execute on a real disk system.

Setup the test environment

You can of course use your own disks. In my case i did a trial run of the needed steps by using losetup to map a data file to a loop device.

  dd if=/dev/zero of=diska bs=1M count=300
  dd if=/dev/zero of=diskb bs=1M count=300
  dd if=/dev/zero of=diskc bs=1M count=300
  dd if=/dev/zero of=diskd bs=1M count=300
  losetup /dev/loop0 diska
  losetup /dev/loop1 diskb
  losetup /dev/loop2 diskc
  losetup /dev/loop3 diskd

  # Create raid 5 with one disk missing
  mdadm -Cv /dev/md2 -l5 -n3 /dev/loop0 /dev/loop1 missing

  # Add luks cryptolayer ontop
  cryptsetup luksFormat /dev/md2
  cryptsetup luksOpen /dev/md2 cryptotest

  # Add lvm structure 
  pvcreate /dev/mapper/cryptotest
  vgcreate testvg /dev/mapper/cryptotest
  lvcreate -l 100%FREE -n testlv testvg

  # Create filesystem
  mkfs.ext3  /dev/mapper/testvg-testlv
  mount /dev/mapper/testvg-testlv test/

You can now copy your data onto the raid 5 and when done add the missing disk to the final raid. You can of course use three disks from the start.

Add missing disk to the raid 5

  # Add missing device
  mdadm --add /dev/md2 /dev/loop2

Growing the raid 5 by adding a forth disk

Once your data has been copied you may want to add the now free disk to the raid of md2.

  # Add disk and resize md2
  mdadm --add /dev/md2 /dev/loop3
  mdadm --grow --raid-devices=4 /dev/md2

  # Check the sync process and the final result
  cat /proc/mdstat
  mdadm --detail /dev/md2
  umount /tmp/test/test

  # Resize the crypt layer and check the result
  cryptsetup resize cryptotest
  fdisk -l /dev/mapper/cryptotest

  # Resize physical volume and check the result
  pvresize /dev/mapper/cryptotest 
  pvdisplay

  # Resize logical volume and check result
  lvextend -l +100%FREE /dev/mapper/testvg-testlv
  lvdisplay 

  # Resize filesystem
  e2fsck -f /dev/mapper/testvg-testlv
  resize2fs /dev/mapper/testvg-testlv
  mount /dev/mapper/testvg-testlv test

Cleanup of the test environment

Of course you should not do these steps on the real raid. These steps are just needed when you want to cleanup your testenvironment.

  umount /tmp/test/test
  lvremove testvg testlv
  vgremove testvg
  pvremove  /dev/mapper/cryptotest 
  cryptsetup close /dev/mapper/cryptotest 
  mdadm --manage /dev/md2 --stop
  losetup -d /dev/loop0
  losetup -d /dev/loop1
  losetup -d /dev/loop2
  losetup -d /dev/loop3

MySQL 5.6 Debian Wheezy Installation

Just a small snippet on how to setup the MySQL 5.6 database on debian wheezy.

apt-get install libaio1
groupadd mysql
useradd -r -g mysql mysql
wget -O mysql-5.6.16-debian6.0-x86_64.deb http://downloads.mysql.com/archives/mysql-5.6/mysql-5.6.16-debian6.0-x86_64.deb
dpkg -i mysql-5.6.16-debian6.0-x86_64.deb
cd /usr/local
ln -s /opt/mysql/server-5.6 mysql
cd mysql
scripts/mysql_install_db --user=mysql
chown -R root .
chown -R mysql data
cp support-files/mysql.server /etc/init.d/mysql
mkdir -p /etc/mysql/conf.d/
cp my.cnf /etc/mysql/
echo $'!include /etc/mysql/my.cnf\n!includedir /etc/mysql/conf.d/' > my.cnf
update-rc.d mysql-5.7 defaults
service mysql start
./bin/mysql_secure_installation

High Performance MySQL Test Database Setup

Sometimes your unit tests have run against a real database server. This post will not discuss the question whether this is avoidable or not. Please take a look at embedded database server (e.g: HSQLDB for Java and mocking techniques e.g: mockito) if you want to remove this dependency from your test environment.

Instead this post will show how to setup a blazing fast MySQL Server against which you can run your unit tests.

The main performance bonus is gained by the innodb_flush_log_at_trx_commit setting. This setting will change the flush behavior for the mysql server. Additionally linux tmpfs will be used to put the mysql data files directly into ram.

The mysql-tmpfs-wrapper.sh script will be used to replace the /etc/init.d/mysql start script. Symlink this file to /etc/init.d/myql-tmpfs.sh and use the sysv-rc-conf tool to remove the autostart/stop of the original /etc/init.d/mysql script. I used insserv /etc/init.d/mysql-tmpfs.sh to enable autostart for my wrapper script.

The recreate_mysql_frombarecopy.sh will recreate the tmpfs ramdisk by using a bare copy of the mysql data directory.

Add this mountpoint to your /etc/fstab file to define the mysql data tmpfs. You can use the stat tool to check the uid and gid for your mysql data directory (e.g: stat /opt/mysql/server-5.6/data)

none            /opt/mysql/server-5.6/data  tmpfs   defaults,size=1000M,uid=999,gid=1000,mode=0700          0       0

I suggest to not directly modify the my.cnf file. Instead use the default my.cnf file and add a includedir section to include files in /etc/mysql/conf.d). This step is of course not needed when you use the original debian mysql server.

[mysqld]
bind-address = 0.0.0.0

thread_cache_size = 8
query_cache_size = 32M
thread_concurrency = 8
key_buffer_size = 256M
max_allowed_packet = 500M
table_open_cache = 512
sort_buffer_size = 256M
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M

innodb_use_native_aio = false
innodb_data_file_path = ibdata1:10M:autoextend:max:15360M
innodb_file_format = Barracuda
innodb_flush_method = O_DIRECT
innodb_write_io_threads = 64
innodb_read_io_threads = 64

#innodb_additional_mem_pool_size = 20M
innodb_buffer_pool_size = 2G
innodb_log_files_in_group = 2
innodb_log_buffer_size = 8M
innodb_log_file_size = 128M

innodb_flush_log_at_trx_commit = 2
innodb_lock_wait_timeout = 50

max_connect_errors = 1000000
max_connections = 350

The last important step is to create the bare copy. I suggest that you stripdown your mysql as much as possible.
Example:

  1. 1. Drop all databases except (mysql, perfomance_schema, information_schema)
  2. 2. /etc/init.d/mysql stop
  3. 3. Remove the ibdata and iblogfiles
  4. 4. /etc/init.d/mysql start
  5. 5. Mysql should work and should have recreated the ibdata and logfiles
  6. 6. /etc/init.d/mysql stop
  7. 7. cp -ra /opt/mysql/server-5.6/data /opt/mysql/server-5.6/databarecopy
  8. 8. rm -rf /opt/mysql/server-5.6/data/*
  9. 9. /etc/init.d/mysql-tmpfs.sh start

When put into lxc container a jenkins job can be created to restart the container on a regular basis and thus keep the mysql data directoy clean.

Happy testing!

Wordpress to Jekyll Migration

I just finished my migration from wordpress to jekyll.

Before you start

Make yourself familiar with the Jekyll liquid template info page and the Liquid for Designers wiki page.

It is also very helpful to take a look at the sources for one of the listed jekyll sites.

Splitting up your wordpress theme template

I save the index page into a dummy file and started to split the contents of that file up into manageable parts. At the end i got the following files:

  • head-includes.html – A include file that contains all the css and javascript includes for my blog
  • default.html – The default template file that contains the raw html skeleton for my blog. It also contained include tags for the head-includes.html.
  • post.html – The post template is a child of the default.html template. This template adds those html parts to the default.html that is needed to frame a post. It also contains various includes for widgets – such as the tag cloud widget and so on.

Exporting Wordpress Posts

$ ruby -rubygems -e 'require "jekyll/migrators/wordpress"; Jekyll::WordPress.process("database", "user", "pass", "127.0.0.1")'

I had to fix a lot of encoding issues in the generated post files. I used sed to correct those encoding issues. There might by way configuring the encoding format for the exporter.

Plugins

I added some plugins to handle youtube videos, adding code snippets etc. Please note that you mostly can’t use such plugins with github pages because github is only allowing building jekyll pages with safe plugins.

RSS Feed

  • /feed/index.html
    ---
    layout: nil
    ---
    <?xml version="1.0" encoding="utf-8"?>
    <feed xmlns="http://www.w3.org/2005/Atom">
     <title>Jotschi.de</title>
     <link href="http://www.jotschi.de/atom.xml" rel="self"/>
     <link href="http://www.jotschi.de/"/>
     <updated>{{ site.time | date_to_xmlschema }}</updated>
     <id>http://www.jotschi.de/</id>
     <author>
       <name>Jotschi</name>
       <email>webmaster@jotschi.de</email>
     </author>
     {% for post in site.posts %}
     <entry>
       <title>{{ post.title }}</title>
       <link href="http://www.jotschi.de{{ post.url }}"/>
       <updated>{{ post.date | date_to_xmlschema }}</updated>
       <id>http://www.jotschi.de{{ post.id }}</id>
       <content type="html">{{ post.content | xml_escape }}</content>
     </entry>
     {% endfor %}
    </feed>
    

Importing Comments to Disqus

I choose disqus to handle comments on my blog. I created a comments-widget.html file which i placed within my ___includes__ directory. That file contains the needed javascript that is used to embed disqus within my website.

It is possible to import your wordpress comments into your disqus account. I created a Intense Debate xml template and Wordpress export template.

  • IntenseDebate.xml
    ---
    layout: nil
    ---
    <?xml version="1.0"?>
    <output>
      {% for post in site.posts %}
      {% if post.comments.size > 0 %}
      <blogpost>
        <url>http://www.jotschi.de{{ post.url }}</url>
        <title>{{ post.title }}</title>
        <guid>http://www.jotschi.de{{ post.url }}</guid>
        <comments>
          {% for comment in post.comments %}
          <comment>
            <isAnon>1</isAnon>
            <score>1</score>
            <name>{{ comment.author | xml_escape }}</name>
            <email>{{ comment.author_email }}</email>
            <url>{{ comment.author_url }}</url>
            <ip>127.0.0.1</ip>
            <date>{{ comment.date }}</date>
            <gmt>{{ comment.date_gmt }}</gmt>
            <comment>{{ comment.content |xml_escape }}</comment>
          </comment>
         {% endfor %}
        </comments>
      </blogpost>
      {% endif %}
      {% endfor %}
    </output>
    
  • Wordpress.xml
    ---
    layout: nil
    ---
    <?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
      xmlns:content="http://purl.org/rss/1.0/modules/content/"
      xmlns:dsq="http://www.disqus.com/"
      xmlns:dc="http://purl.org/dc/elements/1.1/"
      xmlns:wp="http://wordpress.org/export/1.0/">
      <channel>
            {% for post in site.posts %}
      		{% if post.comments.size > 0 %}
    		<item>
    		  <title>{{ post.title }}</title>
    		  <link>http://www.jotschi.de{{ post.url }}</link>
    		  <content:encoded></content:encoded>
    		  <!-- value used within disqus_identifier; usually internal identifier of article -->
    		  <dsq:thread_identifier>237</dsq:thread_identifier>
    		  <wp:post_date_gmt>{{ comment.date_gmt }}</wp:post_date_gmt>
    		  <wp:comment_status>open</wp:comment_status>
    		  {% for comment in post.comments %}
    		  <wp:comment>
    			<!-- internal id of comment -->
    			<wp:comment_id>{{ comment.id }}</wp:comment_id>
    			<wp:comment_author>{{ comment.author | xml_escape }}</wp:comment_author>
    			<wp:comment_author_email>{{ comment.author_email }}</wp:comment_author_email>
    			<wp:comment_author_url>{{ comment.author_url }}</wp:comment_author_url>
    			<wp:comment_author_IP>127.0.0.1</wp:comment_author_IP>
    			<wp:comment_date_gmt>{{ comment.date_gmt }}</wp:comment_date_gmt>
    			<wp:comment_content>{{ comment.content | xml_escape }}</wp:comment_content>
    			<wp:comment_approved>1</wp:comment_approved>
    			<wp:comment_parent>0</wp:comment_parent>
    		</wp:comment>
    		{% endfor %}
    		</item>
    		{% endif %}
    		{% endfor %}
    </channel>
    </rss>
    

Somehow the generated IntenseDebate.xml file could not be successfully imported in disqus. I therefore created the Wordpress.xml export file. This file was successfully imported.

Search engine optimization

The sitemap.xml will help the search engine crawler to find your content as fast as possible.

  • sitemap.xml
    <?xml version="1.0"?>
    ---
    ---
    <?xml version="1.0" encoding="UTF-8"?>
    <?xml-stylesheet type="text/xsl" href="/sitemap.xsl"?>
    <urlset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd" xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">    
      <url>
        <loc>http://www.jotschi.de/</loc>
        <lastmod>{{ site.time | date_to_xmlschema }}</lastmod>
        <changefreq>daily</changefreq>
        <priority>1.0</priority>
      </url>
      {% for post in site.posts %}
      <url>
        <loc>http://www.jotschi.de{{ post.url }}/</loc>
        <lastmod>{{ post.date | date_to_xmlschema }}</lastmod>
        <changefreq>weekly</changefreq>
        <priority>0.8</priority>
      </url>
      {% endfor %}
    </urlset>
    
  • robots.xml
User-agent: *
Disallow:
Sitemap: http://www.jotschi.de/sitemap.xml

Google Search for Jekyll

I create a small template for a search widget. This template adds a search form that will use the google jsapi to provide search capability for my blog.

Handling wordpress urls

I wrote two templates that create a set of rewrite rules that can be used to redirect request to the old wordpress urls to the new jekyll urls.

Apache2

---
layout: nil
---
<IfModule mod_rewrite.c>
RewriteEngine on
{% for post in site.posts %}
{% if post.wordpress_url %}
# Rule for {{ post.title }}
RewriteCond %{QUERY_STRING} ^{{ post.wordpress_url }}$
RewriteRule ^$ http://www.jotschi.de{{ post.url }}? [R=302,NC,L]
{% endif %}
{% endfor %}
</IfModule>

Nginx

This template create a list of rewrite rules for nginx to handle wordpress shorturls so that those urls can be remapped for jekyll.

---
layout: nil
---
  {% for post in site.posts %}{% if post.wordpress_url %}
  # Rule for {{ post.title }}
  if ($args ~ "^p={{ post.wordpress_id }}$") {
    rewrite ^.*$ http://www.jotschi.de{{ post.url }}? permanent;
    break;
  }{% endif %}{%
   endfor %}

Include the generated file within a location / section of your nginx site configuration file like this:

location / {
  try_files $uri $uri/ /index.html;
  include /etc/nginx/sites-extra/wordpress-rewrites.conf;
}
Posted in Technik | Tagged with  |