Using S3cmd to list the ACL's of all the files in an S3 bucket

In the absence of any replies from the S3cmd forum, I managed to use a command line hack to get the ACL status of all files in a bucket with this:

 I tried parsing the whole bucket with an "astrix" as an option , but that didn't work with version 1.0.1 or 1.0.5 of S3cmd. 

s3cmd -c ~/.s3cfg_uk2 ls s3://test-hubs/ | awk '{print $4}' | sed 's/s3\:\/\/test-hubs\///g' | xargs -I file s3cmd -c ~/.s3cfg_uk2 info s3://test-hubs/file 

 This is all on one line - and the xargs option is a capitol "i" and not an "el" as it appears here ;-) If anyone can see how to refactor this to make it more efficient be my guest, but it works.

Or indeed answer my original question with a snappy command line option to s3cmd ;-)


If you want to see if there is "anyone" access to a file you will see "anon" as the ACL setting , so you can search on that if you want to look for globally available files - which is what I wanted to do.

Using an encrypted git repository on Dropbox to store my puppet code in the Cloud.

Today, most technologists have said they love the cloud storage paradigm and to have files and configs at their finger tips , no matter where they are , or how they access them.

Dropbox was arguably the first, easiest and globally available cloud storage solution out their, and it's the one I continue to use today. Like all cloud solutions - apart from owncloud - they have one drawback, who else can look at your files? This tends to make users suspicious and cautious about what they are prepared to store in the cloud, me too. As I control various ssh keys with my puppet code, I certainly don't want people having access to these.

So the solution I found was Truecrypt - a great open source project that allows you to create encrypted  volumes that you can mount on all popular operating systems, including Linux, OSX and Microsoft. As I use all three of these systems at various times, it gives me great coverage. In the post PC era, it would be nice to see IOS and Android versions available in the future, but in all honesty I have not seen a real need yet.

My purposes only required a small amount of space, as code does not generally take up too much room, so I only made it 10MB. Your mileage may vary of course. There are good tutorials on the Truecrypt web site, and if you like what you see, then please donate to the project.

Once Truecrypt has been installed , created your volume and you have mounted it , you can now create your git repository using standard git commands like so:

git init --bare project.git

You can then take the project you have been working on your local hard drive, and commit it to your new git repo, like so

git remote add origin /media/truecrypt1/git/project.git
git push -u origin master

That's all there is too it, so now I have my puppet git repository where ever I go, and I can very quickly bring a machine up to my requirements with all the settings and packages I need to get work done with puppet and a few git commands. It literally saves hours of time, and I can rest easy in the knowledge it is safe in the cloud.

You can obviously use Truecrypt for storing all sorts of other things in the cloud, as I do. If you want to make sure only you can see sensitive data then this a good way to go.

How to add another hard disk to your Xen virtual machine

If like me you have many Xen virtual machines that have been running for a while, you will likely come across the problem of having to increase the disk space, as the services or jobs the virtual machine offers over time changes.

Now you could increase the size of the existing disk, but I always feel that offers a level of risk, which with adding a new disk and using LVM or a new mount goes away.

Ok, so firstly we need to create a new virtual disk on the physical harddisk with the following command:

dd if=/dev/zero of=vm01-disk2.img bs=1G count=50

This will give us 50GB of new space. You can of course change the size to what ever you require, if your physical disk space will allow it.

Then we need to edit the Xen virtual machine configuration file and add in the new disk.

On my virtual machine host this is in /etc/xen/vm01

The line that needs to be altered is obviously the disk line, like so

disk = [ "tap:aio:/opt/xendisks/vm01.img,xvda,w",\
"tap:aio:/opt/xendisks/vm01-disk2.img,xvdb,w" ]

NB Notice the slash which allows the configuration line to expand over two lines in the configuration file.

Once you have done this, you can use the virsh and xm commands to restart the virtual machine , so that the new disk is available inside the virtual machine.

Once your VM has re-booted, you will notice another disk, which you can then either add to your LVM configuration, or just format and mount in a new directory. Here is my new mount as an example.

/dev/xvdb 50G 19G 29G 39% /opt/tomcat

Cloud storage helps data access across local virtual machines

Another fantastic use for cloud based file storeage is to keep the information in sync between your main machine and all your virtual machines on that host. I have to keep a lot of documents around for the various projects I'm currently working on , and having to duplicate those across several virtual machines would be a complete pain in the backside. So the easy and simple way around this, while also making sure that they are constantly backed up, is to open either an Ubuntu one or Dropbox cloud account , and save your documents there.

I know several people may well advice you that you data isn't safe, or could be looked at by the company staff running the systems, but having read the security documents of these companies, and knowing the reliability of local storage, I believe the potential for undeisred access to your data is far out weighed by the massive advantages you get on the other side, of which this use is but one.

Once the documents are saved into the Ubuntu One or Dropbox folder, you can then share that folder with any virtual machine on your host as this folder is constantly available to you, even while off-line, which is great for working on the move.

I'm currently running the 64bit version of Virtualbox on my Dell XPS laptop, with Ubuntu 9.10 , and using the Dropbox and Ubuntu One cloud areas for my data. I then simple share these using the shared folder utility - see below - and can access my documents easily and securely from any of my many virtual machines which include:

Windows XP
Redhat Enterprise server - several
Fedora 11
Ubuntu 9.10 server - several
Windows 7 - toy, of no practical use




I also have to comment that I'm using Virtualbox considerably more these days on the laptop as it is just faster than VMware workstation and it's open source and free. I still keep VMware around for those virtual machines that need it, but all my new machines are now built with Virtualbox. I do fully expect VMware workstation to at some stage to be made free, as trying to compete with Virtualbox with just VMware player is not going to work as a strategy. VMware has all the other technologies knocked into a cocked hat when it comes to the server management and deployment, but the workstation has moved on.