I’ve been using Jungle Disk to do offsite backups of my data. Jungle Disk uses Amazon’s cheap online storage service, Amazon S3, to host backups. (15 cents a gig!) I don’t like Jungle Disk because it’s not open source and because the app is a little clunky, at least on Linux. I recently found that s3cmd could do an rsync-like sync of a directory. That’s cool, but it doesn’t do encrypted backups like Jungle Disk did. In this post, I’ll demonstrate how to make an encrypted backup of locally unencrypted data using EncFS, Amazon S3, and s3cmd.

Enter EncFS. EncFS transparently encrypts files with AES encryption from a FUSE mountpoint to a local directory. That means I could have an encrypted directory, like /home/user/encrypted, and a encfs mountpoint at /home/user/unencrypted. The unencrypted directory would contain all the plaintext (unencrypted) data, and the encrypted directory would contain a mirror of the unencrypted directory’s directory structure as well as all of the individual files, except that the file names and contents have been encrypted. (Note that this could be a disadvantage of EncFS depending on your needs: the files contents and filenames have been scrambled, but an attacker who has accessed your data still encypted can still see approximate file sizes, approximate file name lengths, and file attributes. Jungle Disk shares these disadvantages with its encryption.) More on EncFS here…

You might already see how EncFS can make it really easy to back up your encrypted data without any hassle, but what about if you already have a ton of unencrypted files which you don’t care to encrypt on your local disk? Well EncFS has a cool little “reverse” mode that lets you create an encrypted mountpoint from an unencrypted directory, suitable for rsyncing against, or in this case, for using s3cmd sync with.

How to do it

Before you get started, you have to have an Amazon S3 account. You can sign up here if you’re not signed up already. You should also have a modern Linux distro with FUSE, as well as encfs and the s3cmd utility. Now lets go to a terminal and configure s3cmd:

sharp@blue:~$ s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Secret Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: (just hit enter, if you want)
Path to GPG program [/usr/bin/gpg]: (hit enter)

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: Yes

New settings:
  Access Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  Secret Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y
Please wait...
Success. Your access key and secret key worked fine  

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/home/sharp/.s3cfg'

You may have noticed my access key and secret key blocked out with Xs. These are unique to your account and can be found at this page. Now that s3cmd is configured and working, we can make a bucket to keep our backup. (You can keep multiple backups per bucket.) Keep in mind that nobody else on S3 may be using the same bucket name, so you’ll have to pick one thats unique. This is because lots of S3 users make whatever content is in their buckets public (although the default is to keep it private.) So lets create our bucket:

sharp@blue:~$ s3cmd mb s3://sharpbackup
Bucket 'sharpbackup' created

Now we need a temporary directory to mount the encrypted filesystem on.

sharp@blue:~$ mkdir Music_enc

You might make this in /tmp, especially if you are scripting the process. In this example I’m trying to back up my music (which is in /home/sharp/Music, so I’ve given the mountpoint the name /home/sharp/Music_enc.) Now finally we can create our key and reverse mount this unencrypted directory to an encrypted mountpoint. Be sure to use the full path of both the directory you are backing up and the mountpoint.

sharp@blue:~$ encfs --reverse /home/sharp/Music /home/sharp/Music_enc
Creating new encrypted volume.
Please choose from one of the following options:
 enter "x" for expert configuration mode,
 enter "p" for pre-configured paranoia mode,
 anything else, or an empty line will select standard mode.
?> (press enter here)

Standard configuration selected.
--reverse specified, not using unique/chained IV

Configuration finished.  The filesystem to be created has
the following properties:
Filesystem cipher: "ssl/aes", version 2:1:1
Filename encoding: "nameio/block", version 3:0:1
Key Size: 192 bits
Block Size: 1024 bytes

Now you will need to enter a password for your filesystem.
You will need to remember this password, as there is absolutely
no recovery mechanism.  However, the password can be changed
later using encfsctl.

New Encfs Password: (enter password here)
Verify Encfs Password: (again...)

Now we’ll want to back up the EncFS config file. EncFS puts this file in the rootDir you specify. It contains the key used to decrypt the file system. The key itself is encrypted with your EncFS password, so if Mallory gets this file, he’ll still have to have your password. If you’re paranoid you can keep this file and put it somewhere safe, but since I’m not that paranoid about keeping my music unreadable, and because I don’t want to lose it (we are making a backup, after all,) I’ll put it in my bucket:

sharp@blue:~$ s3cmd put Music/.encfs6.xml s3://sharpbackup/music.xml
File 'Music/.encfs6.xml' stored as s3://sharpbackup/music.xml (911 bytes in 0.0 seconds, 3.28 MB/s) [1 of 1]

Now that file is safe, we can use s3cmd sync to sync all the encrypted versions of the files to the bucket.

sharp@blue:~$ s3cmd --delete-removed sync Music_enc/ s3://sharpbackup/music
Compiling list of local files...
Retrieving list of remote files...
Found 11 local files, 0 remote files
Verifying checksums...
Summary: 11 local files to upload, 0 remote files to delete
...

…and we’re done. If you stop this command and then start it again it will pick up where it left off. That’s actually true for any point in this process. You can even change files, add files or delete files, and s3cmd will only transfer the files it has to to make the backup up-to-date. This is the beauty of using EncFS with an rsync-like system. One last thing. When we’re done you should unmount the EncFS mountpoint:

sharp@blue:~$ fusermount -u Music_enc/

Restoring the backup

Now lets pretend our hard disk crashes, and we’ve lost all our data. We install Linux along with EncFS and s3cmd. At this point you could sync back all the data and use it like a regular EncFS folder. The problem is that we never intended for the data to be encrypted locally, and it would be a hassle to mount it as a regular EncFS folder and copy all the data out of there. Luckily we can reverse mount the same way we did before and sync all our music back. First, lets create our folders:

sharp@blue:~$ mkdir Music
sharp@blue:~$ mkdir Music_enc

Now we have to pull our config file back into the directory we want all of our files to go into:

sharp@blue:~$ s3cmd get s3://sharpbackup/music.xml Music/.encfs6.xml
Object s3://sharpbackup/music.xml saved as 'Music/.encfs6.xml' (911 bytes in 0.0 seconds, 1569.16 kB/s)

Now all we have to do is sync the encrypted files back into our Music_enc directory, and EncFS handles the rest:

sharp@blue:~$ s3cmd sync s3://sharpbackup/music Music_enc/
Retrieving list of remote files...
Compiling list of local files...
Found 11 remote files, 1 local files
Verifying checksums...
Summary: 11 remote files to download, 1 local files to delete
not-deleted 'UO5JPyI9Q3Q7hcnRW0kz8d6H'
...

sharp@blue:~$ cd Music
sharp@blue:~/Music$ ls
Minor Threat

sharp@blue:~$ fusermount -u Music_enc/

Final thoughts

• EncFS makes a ton of stuff like this really easy. You could do pretty much the same process with rsync and rsync.net. Or with rsync and another FUSE filesystem like sshfs or GmailFS, although I wouldn’t recommend the latter because Google looks down upon that sort of thing and is known to remove accounts that use tons of bandwidth. The upside to S3 is that it is cheap storage.
• This whole process can be easily scripted. I may (or may not) be releasing a script soon that just does this whole thing if you give it a directory you want to back up and a name of a bucket and prefix.
• Metadata (file size, file name size, attributes, etc) is still easy to see. The contents and file names may be encrypted, but it is not hard to figure out that a bunch of folders containing 10 or so files that are about 2-4 megs are folders containing music.
• Backing up folders already encrypted with EncFS is even easier. Just sync them.

So I decided to spruce it up somewhat and improve the atmosphere. To do this I would remove the pc from the environment entirely. Luckily my cupboard is directly above my garage where I have pleanty of room for computers and other assorted paraphenalia.

The goal decided, it was time to plan.

Schematic for wiring of a cupboard

Downstairs Cabling

With all the computer crap I do it’s necessary for me to be able to plug in extra computing gear as required. So I tried to make sure there would be enough ports to handly anything I could reasonable think of.

In addition to the cupboard wiring I also decided to network the house with Cat6. Just because I could.

To begin I ripped out the cupboard walls, replaced some joists then put all the cables in place.

All the cables in place

The left hand side shows all the cat6 from the rest of the house in place. There are 7 network ports around the house with an additional 4 to be put in the cupboard.

The gyprock sheets are in place

Putting the plaster up was the easy bit, finishing it off is a skill I do not inately possess. It’s somewhat hidden with the undercoat, but really I should have sanded it more.

Undercoat on

Due to delays in getting some of the cabling I wasn’t able to fully test it until after I had painted, but I couldn’t change my mind anyway so it’s a good thing it worked.

Yay! It Works!

The mostly finished product can be seen here

She Rides!

Still to come:

1. Wall mounts for the monitors.
2. Cleaning up the cables so you can’t see any.
3. Desk made out of something decent.
4. Set of speakers.
5. Top half of the cupboard (I have yet to decide what to do with it, but eventually it’ll all be ripped out and replaced as well).

Downstairs is similarly incomplete. I am going to make a server cupboard to hide everything in, but that can wait a bit longer.

Three different snake skins - peg for scale.

Picture doesn’t show it too clearly, but they are all from different snakes. The scales and size change in each one.

I already have RAM & SATA HDD so they are not included. Will probably get an SSD for the box at a later date.

Have narrowed it down to the following

Shiny Case

Remote - Logitech Harmony 785 Advanced Universal Remote

http://www.logitech.com/en-nz/remotes/universal_remotes/devices/370 U

Keyboard - Logitech diNovo Cordless Mini Keyboard

http://www.logitech.com/en-au/keyboards/keyboard/devices/3848 U

Case - Silverstone ML02B-MXR Slim MoDT Case

http://www.silverstonetek.com/products/p_spec.php?pno=ml02&area=usa U

Motherboard - Zotac IONITX-F-E ION N330 Dual Core PCIE

http://www.zotacusa.com/zotac-ionitx-f-e-atom-n330-1-6ghz-dual-core-mini-itx-intel-motherboard.html U

Optical Drive – SONY AD7590S SATA Internal Slim DVD Burner(OEM) U

Tuner - Leadtek PCI-E PxDVR3200H Hybrid TV Tuner HDTV/AC3

http://www.leadtek.com/eng/tv_tuner/overview.asp?lineid=6&pronameid=377 U

Total Price – $1013

Cheaper Alternative

Replace the Case/Motherboard/ODD & Tuner with a

Case - ASRock ION330 NV ION GBL DVDRW HDMI BK

http://www.asrock.com/nettop/spec/ion%20330.asp U

Tuner - KWorld TV Tuner DVB-T HDTV USB U

Brings the price down to $628.00

With a Hills Solar 22 tube solar tube collector and a 250L tank. Theoretically we should reduce our power usage by ~40% according to the literature. Only time will tell.

Installing the manifold.

The manifold installed.

Installing the tubes

Installing the tubes

The tubes

The tubes

The tubes

The tank arrives

Installing the tank

The finished tank

]]> http://blog.kayosdesign.com/2010/03/22/solar-hot-water/feed/ 0 http://blog.kayosdesign.com/2010/01/06/openfirejabberfreepbx-integration-2/ http://blog.kayosdesign.com/2010/01/06/openfirejabberfreepbx-integration-2/#comments Wed, 06 Jan 2010 06:55:06 +0000 Dave http://blog.kayosdesign.com/?p=133 Having played some more I’ve come up with the following solution – I’m not overly happy with it but it allows me to get what I wanted.
All of the following occurs in extensions_override_freepbx.conf
NOTE: This assumes you have gotten the Openfire Asterisk-IM plugin working and that you have jabber.conf configured correctly to talk to your openfire server.

Asterisk obeying Openfire Status

eg. When you mark yourself as away/DND inbound calls will be directed to Voicemail.

This inserts itself into the standard FreePBX call queue for internal calls. I haven’t got an inbound link setup yet so I haven’t figured out how to get that bit working.

The jabber-notify block must contain every extension in the system this is to work on. It could be altered to return to the normal call path if a desired extension isn’t in the list – but I haven’t implemented that.

Basically a call to an extension picks up the appropriate jabber information and runs the macro “reach_user_with_presence”. There is probably some way to hook this into a db/ldap server so you don’t have to enter everyone in manually.

This then checks the users jabber status and if they are not available it jumps to their voicemail. Else it returns to the normal call path.

[from-internal]
include => jabber-notify

[jabber-notify]
;include => macro-reach_user_with_presence
exten => 10,1,Macro(reach_user_with_presence,user10@jabber.kayosdesign.com,${EXTEN})
exten => 11,1,Macro(reach_user_with_presence,user20@jabber.kayosdesign.com,${EXTEN})
exten => 12,1,Macro(reach_user_with_presence,user30@jabber.kayosdesign.com,${EXTEN})
exten => 13,1,Macro(reach_user_with_presence,user40@jabber.kayosdesign.com,${EXTEN})

[from-internal-original]
include => from-internal-xfer
include => bad-number

[macro-reach_user_with_presence]
; ${ARG1} is a jabber address such as test@jabber.server
; ${ARG2} is the voicemail box of the user

exten => s,1,jabberstatus(asterisk,${ARG1},STATUS)
;presence in will be 1-6.
;In order : Online, Chatty, Away, XAway, DND, Offline
;If not in roster variable will = 7
exten => s,n,gotoif($[$[${STATUS}]<3]?available:unavailable) ;GotoIf(condition?label_if_true:label_if_false) ;exten => s,n(available),jabbersend(asterisk,${ARG2},"Call from ${CALLERID(name)} at number ${CALLERID(num)} on ${STRFTIME(,GMT-1,%A %B %d %G at %l:%M:%S %p)}")
exten => s,n(available),Goto(from-internal-original,${ARG2},1)
exten => s,n(unavailable),VoiceMail(${ARG2},u)
exten => s,1,Goto(from-internal-original,s,1)
exten => h,1,Macro(hangupcall)

Call groups being notified by jabber for inbound calls.

We utilise the Counterpath Bria softphone. It’s a bit stupid and does not follow ALERT notifications such as  different ring tones.  As such there is no way for a user to know if an inbound call is implicitely for them or if it is coming in on the call group.

As such I hooked Asterisk into jabber to broadcast a message to all people in the call group. I don’t particularly like this as you cannot create call groups dynamically using the FreePBX interface. If you do create one you MUST rebuild this section appropriately. Basically all it does is override the default settings and sends out an IM to everyone listed (you have to add each individual in the group to the ext-group section), after that it returns to the setup created by FreePBX.

[ext-group]
exten => 330,1,jabbersend(asterisk,user10@jabber.kayosdesign.com,"Group call")
exten => 330,n,jabbersend(asterisk,user20@jabber.kayosdesign.com,"Group call")

exten => 330,n,Goto(ext-group-original,330,1)

[ext-group-original]
exten => 330,1,Macro(user-callerid,)
exten => 330,n,GotoIf($["foo${BLKVM_OVERRIDE}" = "foo"]?skipdb)
exten => 330,n,GotoIf($["${DB(${BLKVM_OVERRIDE})}" = "TRUE"]?skipov)
exten => 330,n(skipdb),Set(__NODEST=)
exten => 330,n,Set(__BLKVM_OVERRIDE=BLKVM/${EXTEN}/${CHANNEL})
exten => 330,n,Set(__BLKVM_BASE=${EXTEN})
exten => 330,n,Set(DB(${BLKVM_OVERRIDE})=TRUE)
exten => 330,n(skipov),Set(RRNODEST=${NODEST})
exten => 330,n(skipvmblk),Set(__NODEST=${EXTEN})
exten => 330,n,Set(RecordMethod=Group)
exten => 330,n,Macro(record-enable,11-13,${RecordMethod})
exten => 330,n,Set(RingGroupMethod=ringall)
exten => 330,n,Set(__ALERT_INFO='\\;info=ring_two')

exten => 330,n(DIALGRP),Macro(dial,20,${DIAL_OPTIONS},11-13)
exten => 330,n,Set(RingGroupMethod=)
exten => 330,n,GotoIf($["foo${RRNODEST}" != "foo"]?nodest)
exten => 330,n,Set(__NODEST=)
exten => 330,n,dbDel(${BLKVM_OVERRIDE})
exten => 330,n,Hangup
exten => 330,n(nodest),Noop(SKIPPING DEST, CALL CAME FROM Q/RG: ${RRNODEST})

Below are the steps I used to get OpenLDAP working as an LDAP addressbook.

NOTE: Change the domain to something appropriate for you.

NOTE: This is not a step by step instruction manual. It’s for my edification so I can figure it out again later without perusing 30 odd different sites. It will require some thought and intelligence to use.

apt-get install slapd ldap-utils

This results in a basic setup of OpenLDAP with bugger all configured, even the old school slapd.conf is missing. You need to install all the extra schemas and set up passwords yourself as the installer does nothing at all.

Next install some schemas

ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/cosine.ldif
ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/inetorgperson.ldif
ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/nis.ldif

NIS can be left out if you don’t intend on using the LDAP server for authentication.

The following ldif contains all the configuration to get a new database up and running, stash it in a file somewhere.

If you are not using NIS then leave out the shadowLastChange attribute. Also set the passwords to whatever you want using the slappasswd tool, or you can probably type a password in cleartext.

###########################################################
# DATABASE SETUP
###########################################################
# Load modules for database type
dn: cn=module{0},cn=config
objectClass: olcModuleList
cn: module{0}
olcModulePath: /usr/lib/ldap
olcModuleLoad: {0}back_hdb

# Create directory database
dn: olcDatabase=hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: hdb
olcDbDirectory: /var/lib/ldap
olcSuffix: dc=kayosdesign,dc=com
olcRootDN: cn=admin,dc=kayosdesign,dc=com
olcRootPW: {SSHA}8QDckoodrIsXgv/BG43Hf5WAbgmzZYEf
olcAccess: {0}to attrs=userPassword,shadowLastChange by dn="cn=admin,dc=kayosdesign,dc=com" write by anonymous auth by self write by * none
olcAccess: {1}to dn.base="" by * read
olcAccess: {2}to * by dn="cn=admin,dc=kayosdesign,dc=com" write by * read
olcLastMod: TRUE
olcDbCheckpoint: 512 30
olcDbConfig: {0}set_cachesize 0 2097152 0
olcDbConfig: {1}set_lk_max_objects 1500
olcDbConfig: {2}set_lk_max_locks 1500
olcDbConfig: {3}set_lk_max_lockers 1500
olcDbIndex: uid pres,eq
olcDbIndex: cn,sn,mail pres,eq,approx,sub
olcDbIndex: objectClass eq

##########################################################
# DEFAULTS MODIFICATION
###########################################################
# Some of the defaults need to be modified in order to allow
# remote access to the LDAP config. Otherwise only root
# will have administrative access.
dn: cn=config
changetype: modify
delete: olcAuthzRegexp

dn: olcDatabase={-1}frontend,cn=config
changetype: modify
delete: olcAccess

dn: olcDatabase={0}config,cn=config
changetype: modify
add: olcRootPW
olcRootPW: {SSHA}8QDckoodrIsXgv/BG43Hf5WAbgmzZYEf

dn: olcDatabase={0}config,cn=config
changetype: modify
delete: olcAccess

Install this by using

ldapadd -Y EXTERNAL -H ldapi:/// -f /root/db.ldif

Now create an ldif for an admin user

# Root of the LDAP tree
dn: dc=kayosdesign,dc=com
objectClass: dcObject
objectClass: organization
o: kayosdesign.com
dc: kayosdesign
description: Tree Root

#LDAP admin
dn: cn=admin,dc=kayosdesign,dc=com
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: admin
userPassword: {SSHA}8QDckoodrIsXgv/BG43Hf5WAbgmzZYEf
description: LDAP administrator account

Insert this into the directory with the following

ldapadd -x -D cn=admin,dc=home,dc=com -W -f /root/base.ldif

You should now be able to search the directory

ldapsearch -xLLL -b cn=config -D cn=admin,cn=config -W
ldapsearch -xLLL -b cn=config -D cn=admin,cn=config -W olcDatabase={1}hdb
ldapsearch -xLLL -b dc=home,dc=com

Once the above is working you can now move on to creating the Addressbook.

Create yet another ldif file with data such as

# Addressbook branch
dn: ou=addressbook,dc=kayosdesign,dc=com
objectClass: organizationalUnit
ou: addressbook
description: LDAP Addressbook

# Addressbook entry
dn: cn=Test Account+mail=test@kayosdesign.com,ou=addressbook,dc=kayosdesign,dc=com
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
givenName: Test
sn: Account
cn: Test Account
mail: test@kayosdesign.com
homePhone: 1234567
telephoneNumber: 76543421
fax: 01928374
mobile: 44556677
street: 1 Street
l: MyTown
st: MyState
postalCode: 666
title: Grand Poo Bah
o: Snorks Anonymous

Insert the entry into the directory with the following

ldapadd -x -D cn=admin,dc=kayosdesign,dc=com -W -f /root/addressbook.ldif

You can now hook whatever addressbook system you want into the LDAP server using the BaseDN

ou=addressbook,dc=kayosdesign,dc=com

Below is a table showing all the different socketable gems that can be upgraded.

It is configured and working using the documentation provided by Openfire. No when someone receives/makes a call Openfire updates their IM status to “On the phone”.

However, it does not appear to be bi-directional eg. Setting jabber status to DND does not tell Asterisk to send your calls to voicemail.

Having played with the config I now have bi directional communication between asterisk/openfire working within the FreePBX framework.

The code below will check extensions to see what their status is in jabber before passing the call on to them.

If they are flagged as available, the server will send them an IM telling them about the incoming call & then pass the call through.

If they are not available, the server will direct the incoming call to voicemail.

editing extensions_custom.conf you need the following

[from-internal-custom]
include => macro-reach_user_with_presence
exten => 10,1,Macro(reach_user_with_presence,SIP/10,test@jabber.server,${EXTEN})
[macro-reach_user_with_presence]
; ${ARG1} is a destination such as SIP/10
; ${ARG2} is a jabber address such as test@jabber.server
; ${ARG3} is the voicemail box of the user

exten => s,1,jabberstatus(asterisk,${ARG2},STATUS)
;presence in will be 1-6.
;In order : Online, Chatty, Away, XAway, DND, Offline
;If not in roster variable will = 7
exten => s,n,gotoif($[$[${STATUS}]<3]?available:unavailable)
;GotoIf(condition?label_if_true:label_if_false)
exten => s,n(available),jabbersend(asterisk,${ARG2},"Call from ${CALLERID(name)} at number ${CALLERID(num)} on ${STRFTIME(,GMT-1,%A %B %d %G at %l:%M:%S %p)}")
exten => s,n,Goto(from-trunk,${ARG3},1)
exten => s,n(unavailable),VoiceMail(${ARG3},u)