Using S3cmd to list the ACL's of all the files in an S3 bucket

In the absence of any replies from the S3cmd forum, I managed to use a command line hack to get the ACL status of all files in a bucket with this:

 I tried parsing the whole bucket with an "astrix" as an option , but that didn't work with version 1.0.1 or 1.0.5 of S3cmd. 

s3cmd -c ~/.s3cfg_uk2 ls s3://test-hubs/ | awk '{print $4}' | sed 's/s3\:\/\/test-hubs\///g' | xargs -I file s3cmd -c ~/.s3cfg_uk2 info s3://test-hubs/file 

 This is all on one line - and the xargs option is a capitol "i" and not an "el" as it appears here ;-) If anyone can see how to refactor this to make it more efficient be my guest, but it works.

Or indeed answer my original question with a snappy command line option to s3cmd ;-)


If you want to see if there is "anyone" access to a file you will see "anon" as the ACL setting , so you can search on that if you want to look for globally available files - which is what I wanted to do.

Setting up the SSH daemon on your Netgear 1100 NAS server

After having installed one of these units for a small project , I wanted to get passed the web interface to have a look at the detailed error logs on this NAS unit.

I had a look on the Netgear forums and there are several add-ons you can apply to the unit to enhance or modify what it is capable of doing. As this is basically Linux running on Sparc, you will be able to do pretty much anything you like. It would appear that virtually every appliance these days runs Linux or BSD, and getting an SSH CLi onto these units gives you a lot more granular control.

When you have a look around the configuration on this unit, the company that actually built these devices before Netgear bought them - Infrant - has it's fingerprints all over the place. When you look at the processor for instance, it reports itself as an:

Infrant Technologies, INC - neon version: 0

When you dig around in the kernel libraries it becomes clear that this is a sparc CPU.

Anyway, back to the project in hand. You need to download these two files:

ToggleSSH_1.0.bin and
EnableRootSSH_1.0.bin

You then need to install them one at a time using the software update screen - below - and reboot the unit after each install. Why they are bin files is not obvious, as a simple script file would have been sufficient, but it may be to do with the extension install mechanism. It would appear all they do, is enable the SSH daemon in /etc/init.d and configure the file /etc/ssh/sshd_config to allow root access.



Once you have re-booted for the second time, point your favourite SSH client at the IP address of the NAS server and log in as root, like so:

SSH root@192.168.1.1 - or whatever your IP address is

voila!

Sun's open source storage solutions - loving aliteration

I attended the London OpenSolaris users group a few weeks ago and the main topic was their moves to include OpenSolaris into there new SAN storage solutions and how this would be open source and community driven. This appeals as a positively gauranteed future proofing of your storage, which for Enterprises and Governments can only be an exceptionally good thing. You are not tying your "Crown Jewels", the life blood of your organisation, to the proprietary whims of the current storage suppliers.

This I found very interesting, as the massive amount of money that certain large storage vendors extract from their entrapped customers can be truely eye watering, EMC2 and NetApp take a bow now.

It also started me to wonder about the future direction of OpenSolaris , as it will obviously suit some customers to continue their server solutions with Solaris rather than Linux, but it will also now be driven by the requirements to support Sun's SAN/NAS hardware products. I think the chances of it ever becoming a serious desktop operating system are shrinking into the distance, I know I have tried it.

Sun are certainly leveraging their ZFS and Dtrace in this new arena, and I have to say with some considerably success, if the figures being thrown around at the meeting were anything to go by. Dtrace is being used extensively with the storage network management tools, which are also open source and free with the product, and can offer a level of visibilty not seen on other products.This is a massive cost saving for all.

I know the buzz in the storage world at the moment is Flash Disk storage, and how this can be included in your storage hierarchy to improve performance and flexibility. I can see that Sun have this embedded into their latest offerings, and though we are at the early stages of this, it is obvious to me that this is the future direction of storage networks. This is not simply for performance , though that is compelling enough, it is for the datacentre space you will save and the amount of energy you will save. These last two points add up to some serious money savings going forward, and in these hard economic times that can't be ignored.

I think that having worked in this area for many years, that these new products from Sun truely are original and extremely useful, and you would be silly not to have them on your shortlist if you are looking for new/extra storage.

The one caveat is that currently they don't offer firbre channel until the end of this year, but with iSCSI performance I don't see this as a massive issue, but it could be for some.

If you are a Linux house you might worry about how this works within your infrastructure, but as the OpenSolaris can be as visible or invisible as you like, I see this as a benefit over other vendors proprietary offerings.

Go and have a look for yourself, and see what you think.

Suns Open Source Storage