The power and felixibility of OpenSSH tunneling in the cloud

If you didn't realise it, the secure shell that is used every day, all over the internet is written and managed by the OpenBSD project team, and we all owe them a huge debt of thanks for their efforts. It lets us communicate securely with our servers and other devices.

Well, one of the great features of ssh , is the ability to set up tunnels , that you can use to connect traffic across , and indeed reverse tunnels which can be very useful for connecting to remote servers.

As this tool has become so ubiquitous, there are other tools that have been built around it, and some examples of these are autossh, which allows you keep a tunnel monitored and open for when you need it. Also corkscrew , which allows you to connect to remote hosts by encapsulating you traffic in https packets, thus cutting holes through proxies.

When these are combined you have the ability to connect to remote hosts, hidden behind corporate firewalls and proxies, so that you can get some work done.

Here is an example. I have a servers than can connect to all the internal network, but you can't get to this server externally, but is has access to a proxy.

Firstly you set up it's ssh config file so that it can talk through the proxy, and then you create a reverse tunnel to a machine in the cloud. You then use autossh to manage that link, so that when you need to use it , it is available.

There are so many variables in this that covering them all is not feasible, but here is an example:

First, make sure that the ssh public keys for both servers have been put into the authorized_keys file in .ssh on the other machines. Swap public keys, in other words.

Configure you .ssh/config for the user to include the following parameters
TCPKeepAlive yes
ServerAliveInterval 270
ProxyCommand /usr/bin/corkscrew 192.168.2.1 8080 %h %p

Where 192.168.2.1 is the IP of your proxy and 8080 is the port it listens on

So when we make the connection to the cloud servers, it will use this encapsulation.

Now we create the tunnel

/usr/bin/autossh -M 29001 -f -nN -R 1102:localhost:10000 billblogs@cloud01.tangerine.co.uk

Ok, lets break the command down
autossh uses -M to specify it's keep alive port, and -f to place the job in the background.

ssh uses the -nN and -R to say this is going to be a reverse tunnel, and on the cloud server it will open port 1102 and connect it to port 10000 on the internal server running the ssh daemon that we start this job from. In other words, the sshd daemon on the internal server is listening on port 10000.

So, once this is all configured, you can go to your cloud server, connect on port 1102, and it will be as if you were sat at your desk, voila

Here is the command you would use to connect:

ssh -p 1102 billbloggs@localhost

NB You need to make these connections as secure as possible, and disable root logins and only allow keys.

The sad news of the death of Christopher Hitchens

The path to Atheism for most people on the planet is complex and fraught with a great deal of inner contemplation about the big issues that face us poor scared and frightened humans. It is with the help of great writers and thinkers that this process can be crystallized into your own detailed understanding and the ability to formulate your own view of these big questions.

Now, I can't say that Christopher Hitchens was the only or indeed main influence in my change of understanding. In fact he came very late to my party of transformation , which was far more deeply influenced initially by science and specifically the evolutionary writings of people like Darwin and Dawkins.

What he did bring once I discovered his work was a writer and essayist of such great conviction and clarity of thought, that it would be worth reading his work even if you were the most bigoted religious believer. The many articles and essays that covered an enormous array of topics and subjects. The realisation that this man was not just interested in a single topic , for which he has been grotesquely pidgin holed by some corners of the media, but was in fact deeply involved in many issues , of which religiosity was just one.

One thing I'm glad to say we had in common was our love of Evelyn Waugh, and thanks to his many books and essays on George Orwell, he has encouraged me to re-engage with a writer I loved when I was much younger. His insightful coverage of the many American presidential candidates and successful incumbents was always a delight to read. His characterisations of the current crop of extreme right wing religious Republican candidates was as humorous as it was salient.

I would also encourage others to seek out his many articles and essays from magazines like The Slate, Vanity Fair, The Statesman and The Atlantic to name but a few. Each one will question your beliefs and make you think about the topic being covered.

I certainly look forward to reading more of his works on Orwell, and I would suggest dear reader that it would be profitable to yourself to do the same.

Rest in peace Mr Hitchens, you will truly be missed.

Issue with git client and Ubuntu 11.10 through corporate proxy

If like me you use git for storing code and configuration files, then it can become a problem when using it through a corporate proxy if you have the wrong versions.

I found today while trying to do a git clone the following error:

emote HEAD refers to nonexistent ref, unable to checkout

After searching around the web I came across this post.

http://goo.gl/hWhXp

This does indeed work.

Basically the problem is that the version of libcurl3 included with Ubuntu 11.10 has problem authenticating through a proxy. You either have to use 11.04 or upgrade your libcurl3 to the version in the next release of Ubuntu(precise), which fixes the issue.

I hope this saves you some effort.

This link also offers URL's to grab the latest deb packages to install, and more back ground on the issue.

http://goo.gl/XIttB

How to add another hard disk to your Xen virtual machine

If like me you have many Xen virtual machines that have been running for a while, you will likely come across the problem of having to increase the disk space, as the services or jobs the virtual machine offers over time changes.

Now you could increase the size of the existing disk, but I always feel that offers a level of risk, which with adding a new disk and using LVM or a new mount goes away.

Ok, so firstly we need to create a new virtual disk on the physical harddisk with the following command:

dd if=/dev/zero of=vm01-disk2.img bs=1G count=50

This will give us 50GB of new space. You can of course change the size to what ever you require, if your physical disk space will allow it.

Then we need to edit the Xen virtual machine configuration file and add in the new disk.

On my virtual machine host this is in /etc/xen/vm01

The line that needs to be altered is obviously the disk line, like so

disk = [ "tap:aio:/opt/xendisks/vm01.img,xvda,w",\
"tap:aio:/opt/xendisks/vm01-disk2.img,xvdb,w" ]

NB Notice the slash which allows the configuration line to expand over two lines in the configuration file.

Once you have done this, you can use the virsh and xm commands to restart the virtual machine , so that the new disk is available inside the virtual machine.

Once your VM has re-booted, you will notice another disk, which you can then either add to your LVM configuration, or just format and mount in a new directory. Here is my new mount as an example.

/dev/xvdb 50G 19G 29G 39% /opt/tomcat

Using the lightweight Fedora LXDE desktop for snappier Virtual machines

While having a look at Fedora 16 beta within Virtualbox virtualization software, I decided to use a more lightweight desktop environment.

Over the last couple of years as hardware has improved the desktop systems that sit on top of X windows have started to become more complex and powerful, but as a side effect have become resource intensive, which is not ideal for virtual machines.

There are many great different types of lightweight desktop environments, but one that is now supported by all the main distributions is LXDE.

Fedora 16 installation was a breeze - though I recommend upgrading Virtualbox to version 4.1.4 before you start for better support.

The lighter desktop has indeed made a massive improvement over speed and repsonsiveness to the Gnome 3 shell I installed on the Fedora 15 image I tested.

The menus and layout my be a little legacy for some, and perhaps for everyday use, but when most of the time is spent in a terminal widow, that is not the end of the world.

Here is a snap of the initial machine once installed.





There are many new features in this release of Fedora, including the latest 3.1 kernel, deltarpms with presto , the use of systemd instead of the now very old SysV daemon management & control and the use of specifically locked ethernet hardware to configuration definitions - so you see p2p1 instead of eth0 for instance.

The final release is only a few days aways, and for people who like cutting edge open source software Fedora is a good choice.

Give it a try and see what you think.

The FUD used against Linux , really annoys me

If you are not familiar with the term FUD, it stands for Fear, Uncertainty and doubt, which are more commonly described as lies in technical circles. They are used by marketing people to stop people using others products, just in case.

Over the years I have seen this tactic used many times by large corporations to scare companies into buying their product's, but none have used it more often and more perniciously than Microsoft.

Well, they have tried everything against Linux, and have told lies by the bucket load, yet still the system gains market share, where today you can almost guarantee that if you go somewhere on the web, the server powering that site will be Linux.

I have also used it on my laptops for years, and this summer on holiday put it through it's paces , but taking it with me to do some work while on the road in Spain. It never missed a beat. Every little dusty side street eating house that offered WIFI, I could connect to, I could do whatever I needed, whenever I needed, without fail. The one device that did let me down on occasion was my iPad, rarely, but it did.

So the facts are Ubuntu Linux on my Dell laptop 100% success, Apple iPad 98% success.

I love my iPad, so I'm not making some cheap point about it , but a fact is a fact.

So the next time someone tells you something about Linux, do some research and find out the facts. Linux is not perfect, but then neither are any of the systems offered, but Linux is free and you can laways help make it better.

Less FUD, and more facts please.

Here is a link to try it for yourself.

Ubuntu Linux

DNSMASQ config on Ubuntu desktop for fast DNS lookups

Here is a quick rundown on how to use dnsmasq as a fast DNS cache. I was driven to set this up, as the DNS servers that my flat company supply are like treacle on a very cold day.

Help on installing DNSMASQ

Install the daemon, and then edit /etc/dnsmasq.conf to listen on port 127.0.0.1. You need to find the line that has #listen-address= and remove the hash and add the address.

Add this nameserver 127.0.0.1 into the top of /etc/resolv.conf file at the top, and restart the daemon. This is where Linux looks for it's DNS servers.

/etc/init.d/dnsmasq restart

Do a DNS lookup like this to test - dig google.com
Look at the time returned, and then do it again, and look at the time difference, a lot faster isn't it.

Advanced fun and games.

If you want to look at the contents of the cache that dnsmasq is curently storing (in memory)then you need add the -q parameter to the startup scipt, just tag it onto the end. The start-up script is in the /etc/init.d directory, and you need to scroll down to the start function. You will see a long list of parameters that get passed, just add the -q to the end.

Then restart the daemon.

To look at the cache we need to send the SIGUSR1 parameter to the kill command to dump the currently held cache into the /var/log/syslog file. Use the following command to do that.

kill -s USR1 1234 - for instance where 1234 is the PID number of the dnsmasq running daemon. You can find that with the ps -ef | grep dnsmasq command.

If you want to add static DNS entries into your cache, edit the /etc/hosts file and add your required , and often used DNS entries in there. These will be read when dnsmasq starts , and will be available immediately without reference to an upstream DNS resolver. So they will be extrememly quick, which is just what I wanted.

For a quick guide on DNS look here.

Quick guide to DNS