Setting Up a Headless VirtualBox VM in Ubuntu Server

In the past week, I’ve been asked to reinstall Windows XP onto a PC which has recently suffered a hard drive failure. One of the problems with the continuing use of this OS (which celebrates its tenth birthday this year) is its antique selection of included drivers and inability to load from anything except a floppy disk at install time – press F6, etc…

Thankfully, we have nLite, which helps the process of slipstreaming drivers written this century into an installation and creating custom media for loading the now ancient XP onto machines with such advanced features as…SATA.

This is exactly the issue I faced with this particular machine, but without a Windows machine on which to run nLite (which is sadly not available for Mac OS X or Linux) I was pretty stuck. For my own work, I never have any need to use Windows, however there is one killer application for Windows which means I do sometimes have use for it once or twice a year – repairing other Windows computers. A virtual machine is the perfect solution for this problem.

In this tutorial, I’ll show you how to set up VirtualBox on a headless Linux server and access the VM from any internet-connected computer.

(These instructions are written for Ubuntu Server 10.04, however they should work with other versions of Ubuntu and be similar to other distributions as well)


  • Oracle VirtualBox 4.0
  • Operating system installation media
  • Microsoft Remote Desktop for Mac (or equivalent)

Step 1: Installing VirtualBox

  1. Visit Oracle’s VirtualBox download page on your client machine. There is a wide range of supported Linux distributions and versions, so find the one you are running. Right-click the appropriate link and select ‘Copy Link Address’
  2. Open an SSH connection to your server and navigate to the folder you wish to download the installer to (for example /home/user/downloads).
  3. Use wget and the address you have copied to the clipboard to download the installation files from Oracle. For example:
  4. wget

  5. Once the download is complete, we can proceed to install VirtualBox. To accomplish this, we use dpkg. For example:
  6. sudo dpkg -i virtualbox-4.0_4.0.0-69151~Ubuntu~lucid_i386.deb

  7. After VirtualBox is installed, we need to install the Extensions Pack for version 4.0 in order to enable some of the more advanced functionality required. First, wget the package:
  8. wget

  9. Now we need to install the downloaded package. To do this, navigate to the download folder and use the following command:
  10. VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-4.0.0-69151.vbox-extpack

Step 2: Configuring the Host

  1. You will need to make sure your user account is in the group ‘vboxusers’. For example, if your username was ‘randymarsh’:
  2. sudo adduser randymarsh vboxusers

  3. The ‘vboxusers’ group will need to have the correct kernel permissions. To set this, you will need to edit ‘/etc/udev/rules.d/40-permissions.rules’ (using "sudo nano /etc/udev/rules.d/40-permissions.rules") and add the following line:

KERNEL=="vboxdrv", GROUP="vboxusers", MODE="0660"

Step 3: Configuring the Virtual Machine

  1. We’re now ready to configure the new VirtualBox VM. The following command uses ‘VBoxManage’ to create a new VM with the name ‘WinXP’ – this can be anything you like. The OS is of type ‘WindowsXP’ – you need to change this to match the type of the guest OS. Finally, the VM is registered with VirtualBox.
  2. VBoxManage createvm -name WinXP –ostype WindowsXP -register

  3. Next, we need to enter some configuration information for the new VM. In the following command, we use VBoxManage to modify the virtual machine ‘WinXP’. ‘–memory’ is allocated in megabytes, in this case 1024MB. PAE and ACPI are enabled, and the DVD drive is set as the first boot device. VirtualBox allows you to configure up to 9 virtual ethernet devices (nic1-9). In this command, we specify that ‘–nic1’ is going to operate in a bridge with the physical ethernet interface ‘eth1’. You will need to change ‘eth1’ to whatever the primary network device is on your machine (usually eth0).
  4. VBoxManage modifyvm WinXP --memory 1024 --pae on --acpi on --boot1 dvd --nic1 bridged --bridgeadapter1 eth1

  5. We now need to create an IDE channel to connect our virtual drives to.
  6. VBoxManage storagectl WinXP --name IDE0 --add ide

  7. Before attaching the virtual hard drive, we first need to create it. In this command, we are creating a .vdi file (virtual hard disk image) of size 12000MB. As usual, you will need to replace ‘WinXP’ with the name of your VM.
  8. VBoxManage createvdi -filename ~/.VirtualBox/drives/winxp.vdi -size 12000

  9. Okay, now we can attach the new drive image to the IDE channel we created in step 3. Again, alter ‘WinXP’ to match the name of your VM and the path to the .vdi image to match the path you set in step 4.
  10. VBoxManage storageattach WinXP --storagectl IDE0 --port 0 --device 0 --type hdd --medium ~/.VirtualBox/drives/winxp.vdi

  11. The second device we need attached to our IDE controller is the virtual DVD drive we will be using to install the operating system. We also need to specify the ISO image of our install media which is to be mounted – in this example it is /home/randymarsh/windowsxp.iso.
  12. VBoxManage storageattach WinXP --storagectl IDE0 --port 1 --device 0 --type dvddrive --medium /home/randymarsh/windowsxp.iso

  13. Finally, we need to configure VRDE. VRDE is the protocol VirtualBox uses to forward the display of a VM running on a headless system to a GUI-inclined client machine. It is compatible with Microsoft’s RDP remote desktop protocol, however rather than using Windows’ (or other guest OS’) RDP server, VirtualBox takes care of things, which means you can forward the virtual BIOS, boot sequence and, crucially, installation process to a client machine before it has booted to OS. The following configuration uses no security authentication and the default RDP port of 3389. This is fine if you’re going to use the VM casually and only very occasionally, but will not do for servers or anything ‘mission-critical’.
  14. VBoxManage modifyvm WinXP --vrde on --vrdeauthtype null --vrdeport 3389

Step 4: Running and Connecting to the Virtual Machine

  1. The VM is now fully configured, VRDE is enabled and it’s time to fire it up for the first time. At the command prompt, enter the following to start the machine:
  2. VBoxHeadless -startvm WinXP &

  3. The host will present you with a couple of lines of copyright information and confirmation that the VRDE server is listening on the assigned port, and then return the command line. It is now time to load your RDP software and connect to the VRDE server. As my client machine is running Mac OS X, I use Microsoft Remote Desktop Connection for Mac (available here). To connect using this software, input the IP address of the host server and press connect.
  4. If everything has gone to plan, you should be viewing the startup sequence of the virtual machine. The DVD drive should be configured as the first boot device, and you can now proceed to install an OS. Remember to install Oracle’s VirtualBox Guest Additions (VBoxGuestAdditions) to get full functionality and optimal performance from your VM after OS installation is complete.

After all this, you should have a very useful VM at your disposal. And remember, with port forwarding correctly configured you will be able to access the machine from any computer with an RDP client!

5 Responses to “Setting Up a Headless VirtualBox VM in Ubuntu Server”

  1. […] Para instalar VirtualBox no tuve mucho problema pero
    para configurarlo sin un entorno grafico (Ubuntu Server) si se puso
    algo complicado. Para este fin seguí esta pequeña guia: link

  2. […] wrote a post at the beginning of this year detailing how to set up a remotely accessible cloud-y virtual machine […]

  3. […] go server-side. If you’ve never set up a remote headless VirtualBox machine before, check out my guide on the subject for help if things become […]

  4. […] – whether you want to try out something beta, something old, run apps across platforms or in the cloud, VMs are a killer part of any power user’s […]

  5. Brenda says:

    Absolutely awesome! Many years ago we used to run a Windows-based answering machine for our home office. Then I switched to Mac and the alternatives were few and far between. We did the mgetty option for several years, but the issue there is lack of support for common hardware and when it breaks, it’s difficult to get the thing back up and running whereas the Windows app we used was flawless.

    I’ve almost got Windows installed in my VM… and if I can get my three modems connected, this will be pure gold.

    Thanks so much for the detailed instructions. Just a word of caution to others… copy/paste on some of the commands didn’t work for me. The –‘s were getting converted to some smart dash that made some of the commands fail.

Leave a Reply