Using S3cmd to list the ACL's of all the files in an S3 bucket

In the absence of any replies from the S3cmd forum, I managed to use a command line hack to get the ACL status of all files in a bucket with this:

 I tried parsing the whole bucket with an "astrix" as an option , but that didn't work with version 1.0.1 or 1.0.5 of S3cmd. 

s3cmd -c ~/.s3cfg_uk2 ls s3://test-hubs/ | awk '{print $4}' | sed 's/s3\:\/\/test-hubs\///g' | xargs -I file s3cmd -c ~/.s3cfg_uk2 info s3://test-hubs/file 

 This is all on one line - and the xargs option is a capitol "i" and not an "el" as it appears here ;-) If anyone can see how to refactor this to make it more efficient be my guest, but it works.

Or indeed answer my original question with a snappy command line option to s3cmd ;-)


If you want to see if there is "anyone" access to a file you will see "anon" as the ACL setting , so you can search on that if you want to look for globally available files - which is what I wanted to do.

Using an encrypted git repository on Dropbox to store my puppet code in the Cloud.

Today, most technologists have said they love the cloud storage paradigm and to have files and configs at their finger tips , no matter where they are , or how they access them.

Dropbox was arguably the first, easiest and globally available cloud storage solution out their, and it's the one I continue to use today. Like all cloud solutions - apart from owncloud - they have one drawback, who else can look at your files? This tends to make users suspicious and cautious about what they are prepared to store in the cloud, me too. As I control various ssh keys with my puppet code, I certainly don't want people having access to these.

So the solution I found was Truecrypt - a great open source project that allows you to create encrypted  volumes that you can mount on all popular operating systems, including Linux, OSX and Microsoft. As I use all three of these systems at various times, it gives me great coverage. In the post PC era, it would be nice to see IOS and Android versions available in the future, but in all honesty I have not seen a real need yet.

My purposes only required a small amount of space, as code does not generally take up too much room, so I only made it 10MB. Your mileage may vary of course. There are good tutorials on the Truecrypt web site, and if you like what you see, then please donate to the project.

Once Truecrypt has been installed , created your volume and you have mounted it , you can now create your git repository using standard git commands like so:

git init --bare project.git

You can then take the project you have been working on your local hard drive, and commit it to your new git repo, like so

git remote add origin /media/truecrypt1/git/project.git
git push -u origin master

That's all there is too it, so now I have my puppet git repository where ever I go, and I can very quickly bring a machine up to my requirements with all the settings and packages I need to get work done with puppet and a few git commands. It literally saves hours of time, and I can rest easy in the knowledge it is safe in the cloud.

You can obviously use Truecrypt for storing all sorts of other things in the cloud, as I do. If you want to make sure only you can see sensitive data then this a good way to go.

The power and felixibility of OpenSSH tunneling in the cloud

If you didn't realise it, the secure shell that is used every day, all over the internet is written and managed by the OpenBSD project team, and we all owe them a huge debt of thanks for their efforts. It lets us communicate securely with our servers and other devices.

Well, one of the great features of ssh , is the ability to set up tunnels , that you can use to connect traffic across , and indeed reverse tunnels which can be very useful for connecting to remote servers.

As this tool has become so ubiquitous, there are other tools that have been built around it, and some examples of these are autossh, which allows you keep a tunnel monitored and open for when you need it. Also corkscrew , which allows you to connect to remote hosts by encapsulating you traffic in https packets, thus cutting holes through proxies.

When these are combined you have the ability to connect to remote hosts, hidden behind corporate firewalls and proxies, so that you can get some work done.

Here is an example. I have a servers than can connect to all the internal network, but you can't get to this server externally, but is has access to a proxy.

Firstly you set up it's ssh config file so that it can talk through the proxy, and then you create a reverse tunnel to a machine in the cloud. You then use autossh to manage that link, so that when you need to use it , it is available.

There are so many variables in this that covering them all is not feasible, but here is an example:

First, make sure that the ssh public keys for both servers have been put into the authorized_keys file in .ssh on the other machines. Swap public keys, in other words.

Configure you .ssh/config for the user to include the following parameters
TCPKeepAlive yes
ServerAliveInterval 270
ProxyCommand /usr/bin/corkscrew 192.168.2.1 8080 %h %p

Where 192.168.2.1 is the IP of your proxy and 8080 is the port it listens on

So when we make the connection to the cloud servers, it will use this encapsulation.

Now we create the tunnel

/usr/bin/autossh -M 29001 -f -nN -R 1102:localhost:10000 billblogs@cloud01.tangerine.co.uk

Ok, lets break the command down
autossh uses -M to specify it's keep alive port, and -f to place the job in the background.

ssh uses the -nN and -R to say this is going to be a reverse tunnel, and on the cloud server it will open port 1102 and connect it to port 10000 on the internal server running the ssh daemon that we start this job from. In other words, the sshd daemon on the internal server is listening on port 10000.

So, once this is all configured, you can go to your cloud server, connect on port 1102, and it will be as if you were sat at your desk, voila

Here is the command you would use to connect:

ssh -p 1102 billbloggs@localhost

NB You need to make these connections as secure as possible, and disable root logins and only allow keys.

How to add another hard disk to your Xen virtual machine

If like me you have many Xen virtual machines that have been running for a while, you will likely come across the problem of having to increase the disk space, as the services or jobs the virtual machine offers over time changes.

Now you could increase the size of the existing disk, but I always feel that offers a level of risk, which with adding a new disk and using LVM or a new mount goes away.

Ok, so firstly we need to create a new virtual disk on the physical harddisk with the following command:

dd if=/dev/zero of=vm01-disk2.img bs=1G count=50

This will give us 50GB of new space. You can of course change the size to what ever you require, if your physical disk space will allow it.

Then we need to edit the Xen virtual machine configuration file and add in the new disk.

On my virtual machine host this is in /etc/xen/vm01

The line that needs to be altered is obviously the disk line, like so

disk = [ "tap:aio:/opt/xendisks/vm01.img,xvda,w",\
"tap:aio:/opt/xendisks/vm01-disk2.img,xvdb,w" ]

NB Notice the slash which allows the configuration line to expand over two lines in the configuration file.

Once you have done this, you can use the virsh and xm commands to restart the virtual machine , so that the new disk is available inside the virtual machine.

Once your VM has re-booted, you will notice another disk, which you can then either add to your LVM configuration, or just format and mount in a new directory. Here is my new mount as an example.

/dev/xvdb 50G 19G 29G 39% /opt/tomcat

Is cloud computing really that new and original

I have been working in IT for more years than I care to remember, but it seems that if you wait around long enough, technologies that I have used before, suddenly become the flavour of the month and re-appear with smart presentations from slick young people in nicely tailored suits to tell us all how different it all is this time.

This happened with virtualization, where if you are brutally honest , the only key difference to what IBM were doing decades ago, was that the technologists got it to run on the Intel architecture, with all it's horribly legacy memory management et al.

So to the cloud, which in essence is the ability to leverage the power and storage of a remote machine to achieve the task's that you were used to doing locally, sound familiar, it certainly does to me. Literally decades ago people used to share resources on an IBM mainframe to run their payroll or business finance systems, which would be physically a long way away - people couldn't afford their own mainframes - and people would attach via modems and dumb terminals.

This is exactly what we are moving towards with the concept of the "Cloud is the machine". The local device can become significantly less powerful as all the real processing of the services is done remotely. This leads to much cheaper consumer devices, whether they be phones, iPads or netbooks. The services that people also want to use today are also becoming more and more central service orientated, things like YouTube, Twitter, Facebook, Blogging, Email, Instant Messaging etc.

I now their are many new technologies like HTML5 and improved Javascript performance that will improve the ability of the clients to do some work locally, but they are not going to be over tasked. The new breed of Intel Atom and ARM processors will be more than powerful enough to cope with the enhanced new GPU's that are appearing almost daily.

There will always be some tasks that are best suited to having a large powerful computer locally as their are many tasks that don't virtualize well, but we are talking about a small percentage for many users who today in 2010 consider the net to be the reason for owning a computer in the first place.

Cloud storage helps data access across local virtual machines

Another fantastic use for cloud based file storeage is to keep the information in sync between your main machine and all your virtual machines on that host. I have to keep a lot of documents around for the various projects I'm currently working on , and having to duplicate those across several virtual machines would be a complete pain in the backside. So the easy and simple way around this, while also making sure that they are constantly backed up, is to open either an Ubuntu one or Dropbox cloud account , and save your documents there.

I know several people may well advice you that you data isn't safe, or could be looked at by the company staff running the systems, but having read the security documents of these companies, and knowing the reliability of local storage, I believe the potential for undeisred access to your data is far out weighed by the massive advantages you get on the other side, of which this use is but one.

Once the documents are saved into the Ubuntu One or Dropbox folder, you can then share that folder with any virtual machine on your host as this folder is constantly available to you, even while off-line, which is great for working on the move.

I'm currently running the 64bit version of Virtualbox on my Dell XPS laptop, with Ubuntu 9.10 , and using the Dropbox and Ubuntu One cloud areas for my data. I then simple share these using the shared folder utility - see below - and can access my documents easily and securely from any of my many virtual machines which include:

Windows XP
Redhat Enterprise server - several
Fedora 11
Ubuntu 9.10 server - several
Windows 7 - toy, of no practical use




I also have to comment that I'm using Virtualbox considerably more these days on the laptop as it is just faster than VMware workstation and it's open source and free. I still keep VMware around for those virtual machines that need it, but all my new machines are now built with Virtualbox. I do fully expect VMware workstation to at some stage to be made free, as trying to compete with Virtualbox with just VMware player is not going to work as a strategy. VMware has all the other technologies knocked into a cocked hat when it comes to the server management and deployment, but the workstation has moved on.

Virtual Private clouds are the next logical step for everyone

I have been working with virtualization since I could first get my hands on the code and it has proved to be a great success in many , but not all, environments. The introduction a couple of years ago of Amazons EC3 cloud was a great first step, and you could see the logic for it's uses among many companies, but again not all.

With the announcement of Amazons new Virtual Private Cloud, I think a very large step is now being taken into the next stage of cloud based systems development.

I think this is nothing more than the realisation by Amazon, and I suspect other cloud service providers that many large companies were never, and I mean never , going to put their corporate systems into a global cloud, no matter how long you could talk on the security of separation.

Using this method, corporate IT managers can migrate certain systems into the cloud and see how they go, like all things with corporates, slowly ,slowly. Once they are happy that the world hasn't ended with moving the Intranet into this new cloud extension to their network, they could well be persuaded to start to move other services into this new area.Then five years from now, they will all be sitting at the bar at a conference telling people they were one of the first to see the light and they always new it would be the future ;-)

Have a look at the announcement here Amazons Virtual Private Cloud

Cloud based features in the Netgear 1100 NAS server

It is becoming obvious to everyone in IT that you can't ignore the cloud any more as it is appearing everywhere, especially in new appliances that are starting to offer additional features that allow you to backup to the cloud.

This rack mounted NAS box , uses Linux, and offers all the normal features you would expect, like NFS,CIFS,AFP etc. It also however adds in an additional feature, and that is an option to sign up with Netgear to backup you data to the cloud. Now I know about all the security issues that surround this debate with secure transfer and storage and who has control of the data once it is stored, but for many small/medium sized companies, this really does take away a lot of the pain of making sure your data is backed up and stored safely.

As a user of Dropbox already for backing up certain data, I can definitely see a use for this service, even if you don't want to put all of the companies most vital data into the cloud just yet.Is it any more risky than asking someone to do backups onto tape, and then safely store those offsite? How many people in a small/medium company regularly check the tapes or test the backups? I would suspect a small percentage, so not only does that take up a lot of time, it does not guarantee success. While sending your data to a professional company that specialises in data backup and restore would offer a decent alternative. The costs are also compelling when put up against the cost of tapes and off-site storage. The cost of fast,reliable tape units are expensive and the tapes to go in them are also not give-away.

Well I embraced the cloud years ago, and I have not had any ill affects as yet, but I watch server hard disks fail on a daily basis, and they really hurt customers. The watch word here is just be careful on the services you select, check exactly what you are getting and look at the small print. Always make a point about checking the security in transfer and storage and who has access to the data once it's in the cloud. As with all things in this area, it will eventually come down to your level of risk acceptance.

Mozilla Labs Weave project is really taking shape

What is it? A bolt on to firefox to store your bookmarks and other browser data in the cloud.

I have used a few of these cloud based bookmark systems in the past, with varying degrees of success, and I have just started testing the Mozilla labs Weave system and it is working very well indeed. I have it talking on three separate systems, including one virtual image and they are syncing perfectly. It means I no longer have to have a different set of bookmarks on different machines, which is not very cloud freindly. The performance is the best I have experienced, and it really happens in the background without you noticing. You can select what parts of your locally stored data you wish to send in to the cloud with a bolt onto the Firefox (3.5 is required) options/preferences window. I like the fact you can encrypt locally and then send the data.



It would be great and extremely useful if it opens its doors eventually to other browsers like Chromium and Opera, and hopefully this will happen. As they are all open source the opportunities are more obvious that with a closed source product like IE.

I think we will start to see a lot more tools and utilities like this moving forward as the cloud becomes the storage system of choice for your information. The data you send to the Mozilla servers is whatever you choose from the list of options, and it is encrypted with a passphrase only you know, so there can't be any data mining without you knowing it. Security is always key in the cloud and can not be ignored, so I think this system works well.

I would definitely recommend giving it a go.

Ubuntu getting it's head further into the clouds

Canonical have ramped up there commercial offerings to get businesses to move some or all of there server requirements into the clouds. They are now offering a two pronged attack, with the publicly available Amazon EC2 and now a more secure offering from Canonical based on Ubuntu Server 9.04 and Eucalyptus.

They have plenty of information on their offerings and how to do this for yourself.

Build your own cloud with Ubuntu server 9.04

Build your cloud using Amazons EC2 public offering

Floating with Abobe Air on Linux in the Cloud

As I'm sure you already know, cloud computing and new client software that will take advantage of our applications moving to the cloud are coming at us thick and fast. We have Google Gears offering us Google apps when we are not connected to the web, AJAX that is being used by just about every to offer richer applications. We have Mozilla Prism, which is based on their Webrunner technology and now Adobe Air. I also believe Microsoft have an offering with Silverlight, but as it is not cross platform I won't be able to use it. There is a Linux port called Moonlight, but I hate products that are not designed from the ground up to be cross platform, that is very anti web.

As a recent convert to using twitter (@codfather), I started to look around for clients that offered more functionality than just the bog standard web interface. I found a great new web application call tweetdeck. Now this requires Adobe Air to use, as it is cross platform development environment, very much like Java, but designed for the next generation of web applications.

Installing this onto Linux is very straight forward, the instructions on their web site are great, all you need to do is do the following:


1) Download the binary installation program to your home directory by clicking on the link
2) Open a terminal and make the binary file executable - good old Linux security
3) Run the Binary

nick@nick-laptop:~$ chmod +x AdobeAIRInstaller.bin
nick@nick-laptop:~$ ./AdobeAIRInstaller.bin


Once you have done this very simple operation, the installation process is done through a point and click installer.Once it has been installed, you will have two new application links in your Gnome Applications --> Accessories menu bar.

Installing applications couldn't be easier, you just identify an Adobe Air application you want to install in Firefox, and just click on it. They all have a .air extension, and firefox will tell you it's an Air application and will offer to open it with the Adobe Air installer. Give this a go by installing tweetdeck.

I have installed this on all my laptops and desktops to give it a good test and to try and shake down any issues. I'm currently using it on my Eeepc with Eeebuntu installed as I type this. Now a twitter friend of mine , @glynmoody, has found some potential memory hogging, but I haven't seen this as of yet.

I would definitely recommend giving it a go and trying out the many applications that are available for it on the Adobe Air web site , but also out on the web.

I suspect we have not seen all the potential that this new web development has to offer yet, but the initial products are good enough for us start enjoying them.