Posts Tagged ‘Linux’

DIY Linux Transmission Server


One of my favourite things about Linux is its ability to transform a formerly useless elderly computer into something really useful indeed. This is the first in a series of posts covering some great projects to breathe new life into old machines using Linux.

What’s the Project?

BitTorrent is a fantastically useful distribution mechanism – when downloading updated Linux distro ISOs, it regularly exceeds the speeds available through centralised (HTTP/FTP) download repositories, and the sharing of bandwidth and ease of distribution are a great fit for online communities and fan groups (Zappateers, for example). Unfortunately, BitTorrent tends to be a total disk I/O hog, constantly reading and writing tiny chunks of file all over your disk. This causes big slowdowns for all other running programs which need access to your hard disk, not to mention the stress it puts on the drive itself.

We’re going to use an old computer, GNU/Linux and the Transmission BitTorrent client to offload this resource-intensive process to a networked machine, freeing up your main machine to get on with business unfettered. Sacrificing convenience, however, is not an option. Therefore, we’re going to implement the following convenience features:

  • Administering the client through a web browser
  • Sharing content over the local network
  • Creating a zero-setup process for starting new transfers


Schedule WordPress Backups Using Cron

I was contacted a couple of days back by a reader who wanted to add my super simple WordPress backup program to Cron, the Unix task scheduler, to create automated regular snapshots of his site. Backing up my website using the program is something I normally do by hand as part of my general system maintenance routine, but the program is easily incorporated in to Cron to run on a timer. Completely by coincidence, I suffered a WordPress plugin failure within a couple of days of creating the Cron job, and thanks to my now daily backups I was able to restore from a 10 hour old snapshot with the minimum of fuss. (more…)

TWiT Early Show Downloader

For me, TWiT network is one of the most valuable resources on the internet: their programming is always informative, slick and balanced, and the guests are always top-teir tech talkers.

As with most pod/netcasts broadcast from the States, the scheduling of live shows is always at pretty peculiar hours of the day thanks to the time difference, which means waiting for the edited version to be uploaded and published to the TWiT website.

Handily, the editors seem to work much quicker than the web folks over in Petaluma, CA, and the shows are normally uploaded quite a bit earlier than they are listed on the website. I’ve been using a little Python program which I wrote to download the shows before they are linked on the site, which you can grab using the instructions below. It uses cURL to download the files so it’ll work on Macintosh and Linux systems.


On Software Patents, Oracle & VisiCalc

The web is aflutter at the moment with discussion of the patent system and its relationship with the software industry in particular (the most thought-provoking piece on the subject so far comes from Nilay Patel and is fully worth the requisite time investment). Should we let people patent software? At what point does math become a patentable invention?

Like most people, I’m unversed on patent law, but I am a sometime software developer. Personally, my gut feeling is that if a piece of software is developed for an explicitly commercial application then it absolutely should be protected under law, particularly if the software effectively IS the business (Google, for example). Surely it isn’t right that I invent something, someone else brings an identical product or service to market and my company is forced out of business? But gut feeling seems to be what most pundits are running on, and is not necessarily the best way to approach the issue.

Current hysterics calling the system ineffective or broken should consider two software cases from near the dawn of our industry, over three decades ago. The first case is Software Arts’ VisiCalc, the first electronic spreadsheet program for personal computers. Although a new invention, the fundamental concept – an electronic spreadsheet – of VisiCalc was not patented, and within a couple of years of its release the market was populated with alternatives, most notably from Lotus and Microsoft. If the core idea behind VisiCalc had been patented, 1-2-3, Multiplan and later Excel would not have existed, or would have needed to invent a totally new spreadsheet metaphor to be permitted under law. But Software Arts did not patent the electronic spreadsheet, and consequently the only time they or VisiCalc are mentioned these days are as points in patent discussions.

Dan Bricklin, one of the co-creators of VisiCalc doesn’t argue for specific change, but does intimate that the patent system is not best suited to the software industry. However, in the same post he actually states that the lack of a VisiCalc monopoly (which would have been enabled by a patent of Software Arts’ take on the electronic spreadsheet) encouraged competition and improvement in quality of all the available software packages (Multiplan correcting the mistakes of 1-2-3, which improved on the shortcomings of VisiCalc etc…). If the core concept behind Archie had been protected by law at the beginning of the ’90s, Yahoo!, Google et al. perhaps wouldn’t exist. Because the overarching concept of a web search engine isn’t itself patented, but the effective and novel technical innovations that make a great search engine can be, we have multi-billion-dollar companies built on the foundations of innovation, protected by the patent system. Bricklin is correct to assert that the broad metaphor of an electronic spreadsheet doesn’t benefit from being patented, much as the fundamental idea of an internet search engine wasn’t. But the current patent system has demonstrably fostered protectable innovation in the case of Google – not so broken in that case, apparently.

One company famous, perhaps more so than any other, for developing, patenting and licensing high technology is IBM. In the 1970s, researchers at IBM developed the relational database model, a vastly superior means of storing and operating on data to the existing database paradigms on the market at the time. To protect the revenue streams provided by their existing database systems however, they chose not to immediately market products based on the new technique. Instead, one Lawrence J. Ellison and colleagues formed a company of their own to being a product to market based on an implementation of the SQL concept outlined in IBM’s research – the Oracle Database.

Oracle, the company, was started to market a software concept developed but not exploited by another organisation, and just like Microsoft with Multiplan and Lotus’ 1-2-3 they may not have invented the central idea but they have innovated on it and created a technologically progressive sector of the industry where there would instead have been a monopoly.

Those are the two cases which spring immediately to my mind when I meditate on this subject (again, not one which I am educated to consider from a legal standpoint). It’s difficult to separate my feelings as a developer – if I invented the RDBMS and Larry Ellison ate my lunch, I wouldn’t be very happy – from my thoughts as an observer – had IBM patented the RDBMS concept or SQL as an implementation thereof, this site wouldn’t be running (and the LAMP stack wouldn’t exist). It seems to me that if any changes need to be made, and again, I’m not a lawyer, they need to be made to what kind of patents you can be granted rather than whether you can patent software at all. Larry Page’s patent on PageRank doesn’t preclude anyone else from creating a search engine, but it isn’t allowed to work just like his does, which is what forms the core of his competitive advantage. And in Oracle’s ongoing complaint against Google and Android, Oracle shouldn’t be allowed to own the idea of a VM with JIT compilation, but they should be able to protect the innovations which they own to do with its operational techniques. The legal system would be best improved by stipulating the breadth and application of concepts eligible for patent protection, as well as those who are permitted to action upon them (no trolls, please).

Restore Hidden Files to OS X 10.7 Lion Finder

One step Lion takes in the direction of erasing the file system altogether is hiding the Library, System and other OS- and user-crucial directories in the Finder.

Obviously as power users we’re fully used to delving into the (already abstracted) file system of Mac OS X, and we want those folders back! Via this site, there is a Terminal command to restore access to these directories, but be warned that it comes with a caveat: by enabling access to these folders, you open up the whole filesystem including UNIX folders (etc, var…) and DS_Store files for view in the Finder.

In short, it’s way annoying, especially when browsing Linux volumes.

Irritating as having bits of the filesystem whipped away from prying eyes, I think I’m going to stick with Terminal for accessing those folders. TextWrangler can still browse them, and that’s enough for me. For those who want to see everything, and I mean everything:

$ defaults write AppleShowAllFiles YES

Super Simple Linux RAM Disk Manager – rdmanage

I’m currently developing a piece of server software which needs to be able to create, modify, serve up and dispose of many temporary files many times every second. Because a great number of I/O operations will be occurring, the program requires very rapid disk access – ideally, it would run on a RAID array or a solid-state disk. Because the server I’m using doesn’t have either, I’m using RAM disks as a substitute.

RAM disks take a portion of your system RAM memory and make it available for storage by mounting it in your file system. Using a RAM disk, you gain a working folder with speed way far in excess of any hard disk or even solid state drive. The usual rules of RAM apply – this is volatile memory, so anything stored on a RAM disk will be erased when the power is cut – but it’s perfect for temporary file storage.

I’ve written a very simple program for Linux called ‘rdmanage’ to help create and manage RAM disks easily at the command line.


There are three modes of operation for this program – create a new RAM disk, create a RAM disk from an existing folder, and remove existing RAM disk. As promised, the program is very simple to operate:

To create a RAM Disk:

$ ./rdmanage create [size in mb]

For example, to create a 100MB RAM disk:

$ ./rdmanage create 100

To create RAM Disk from an existing folder:

$ ./rdmanage from /path/to/folder

For example, to create a RAM disk from the folder ‘/home/randymarsh/sim’:

$ ./rdmanage from /home/randymarsh/sim

To remove an existing RAM Disk:

$ ./rdmanage remove


– When using the ‘from folder’ mode, the maximum size of the folder is 512MB

– You will need root permissions to mount the RAM drive

– The ‘remove’ mode will commit the state of the RAM drive to disk in a folder called ‘ramdisk.contents’

– All operations will occur within the current working directory (i.e. the folder you’re presently in). To check which folder you’re in, type ‘pwd’ at the command prompt.

– Depending on how you download the program, you might have to set the file as executable:

$ chmod +x rdmanage

– If you want to check the status of your drive, it will be listed when you use the ‘df’ command

– For program help, type:

$ ./rdmanage help


You can download the program by right-clicking here and selecting ‘Save As’/’Save Target As’, or if you would prefer to download straight to your Linux machine from the command line, use wget:

$ wget

Hopefully this program can save you a bit of time creating and managing your RAM disks. Enjoy!

Super Simple WordPress Backup Program

Whatever you do with your computer, it’s absolutely vital to back up your data. Never is this more true than when you’re administering a website, where creating backups should be a well organised and regular procedure.

This site is no exception to that rule, so to help me out with my daily backups I’ve written a small program in the Python programming language which automates the process of archiving daily snapshots of my site. For peace of mind, I still prefer to launch and organise backups myself rather than use completely automatic solutions; this program helps make that process much quicker.


You can download the program by right-clicking here and pressing ‘Save As’/’Save Target As’.

If you want to download straight to your Linux machine from the command line, use wget:

$ wget

The program is designed to run under Linux, but theoretically it will work on any Unix system (Mac OS X, Solaris etc…) with Python installed.


All you need to feed the program is the path to your WordPress installation (the directory containing your wp-****.php files). The database login credentials are borrowed from WordPress, and the backup is placed in a date-stamped folder for easy archiving.

Each backup contains a dump of your MySQL database and a tarball of your installation directory – everything you need to restore your site in case of a disaster. The backup will be created in the folder from which you start the program.

To launch the program:

$ ./wordpressbackup /path/to/wordpress/installation

For example, if your site is located in the directory ‘/home/randymarsh/web’, the command would be:

$ ./wordpressbackup /home/randymarsh/web

If you need help or more information, use the help & about functions:

$ ./wordpressbackup help

$ ./wordpressbackup about

Hopefully this little script can help out anyone else like me who only trusts a backup they’ve done by hand. Enjoy!


MacBook Pro 13″ Ubuntu Linux Drivers & Installation

Ubuntu Linux made its way onto my 13″ MacBook Pro this week via external storage. While installing, I was armed with just the laptop itself and a wireless-only internet connection, a real problem given that Ubuntu doesn’t detect the WLAN card out of the box.

I put together a disc to get the machine up and running which includes the DKMS package and the WLAN & nVidia display drivers. It’s not a full collection of files needed to get the laptop playing perfectly with Ubuntu, but it takes care of the most important core functionality.

You can get hold of the ISO by following this link, and I’d also highly recommend reading up on this page, although I’ve outlined and expanded on what I consider to be the most useful pieces of information from it below. I used Ubuntu 11.04 on a 13″ MacBook Pro 7,1 for everything in this post.


Install Linux to and Boot from a USB Drive on Your MacBook

I, like millions of others, use Mac OS X as my main operating system. It’s fast, reliable and secure, and the computers it runs on are undeniably the best designed and built machines available on the market. There are many options available to users who need the added flexibility of running Linux or Windows alongside OS X, perhaps through SSH or by using a Virtual Machine. Sometimes, though, you need a full, non-virtualised OS environment to work in, and while Boot Camp is great it’s not ideal for someone like me who rolls with a very fast, but very small, SSD boot drive.

In this tutorial, I’ll show you how to install Linux to any external USB device and boot your Apple computer from it. I’ll be working with Ubuntu 11.04 32-bit and a MacBook Pro running Mac OS X 10.6 Snow Leopard. Guides elsewhere online seem to only help you if you happen to already have a Linux box to work with – my tutorial only requires one Macintosh computer. All the software used herein is open source and free of charge.


Tutorial Series: Linux Command Line Basics

Last week, I published a seven part series of posts introducing some of the key concepts, commands and techniques of the Linux/Unix command line. Here’s an index of the topics covered to help you find what you’re after.