Xen Project Q&A Forum: First Line Help for Simple Questions

This is your chance to ask questions and provide answers about basic use of the Xen Project software. For debugging problems and for more complex issues, consider using the xen-users mailing list instead. You can find information about xen-users under "HELP | Mailing Lists" in the navigation bar above.

HI All,

I have successfully (I think) performed a Bare Metal install of a small Ubuntu 13.04 Dom0 on to legacy hardware and am trying to create a 12.10 quantal DomU into a separate Volume Group named 'TestMachine' using 'xen-create-image'.

When I try to create the running machine I get the following error in the bootloader log file:

[Errno 28] No space left on device.
Error writing temporary copy of ramdisk

Some details:

xl-list - produces:
Domain-0 ID=0 Mem=256 VCPUs=1 State=r-----

I created the DomU image with the following command:

xen-create-image --arch-i386

This seemed to go smoothly. The log file did not seem to contain any bad errors. However, when I try to start the DomU with the command:

xl create /etc/xen/newmachine.cfg

The log file contained the error message I noted above.

lvdisplay TestMachine lists two logical volumes named:

LV Path: /dev/TestMachine/newmachine-swap
LV Name: newmachine-swap
VG Name: TestMachine
LV Write Access: read/write
LV Status: available
LV Size: 192.00 MiB

LV Path: /dev/TestMachine/newmachine-disk
LV Name: newmachine-disk
VG Name: TestMachine
LG Write Access: read/write
LV Status: available
LV Size: 3.62 GiB

What am I doing wrong?
What should I be checking beforehand?
Any suggestions?

Thanks in advance...

Accepted Answer

Monday, July 08 2013, 05:55 AM - #permalink
After spending some time digging, I discovered that 'pygrub' wants to create the initial ramdisk (and other things I suspect) in the default working directory: '/var/run/xend/boot'.

On my system, this directory did not exist.

By creating the above directory OR supplying the bootloader argument '--output-directory' in the configuration file so that it appears something like:

bootloader = "pygrub"
bootloader_arg = ['--output-directory=/some/directory/with/write/access']

# Rest of configuration follows...

The machine was able to boot normally and present me with a normal command prompt.

Problem solved!


I did some subsequent Root Cause Analysis and thought I would share this for anyone encountering this in the future.

The machine I was installing Xen on is a legacy (circa 2000) machine and is limited in almost every resource - in particular the Dom0 has just 184M of memory.

When pygrub boots guest domain, it tries to copy the guest's vmlinuz and initrd.gz files out of the guest's /boot file system back into the Dom0's file space - specifically into the /run hierarchy.

Ubuntu mounts a tmpfs device on the /run mount point (through /proc/mounts - essentially a RAM disk) and the size is limited to approximately 10% of Dom0 system memory. On my system, the maximum size of the /run hierarchy was around 17MB. The entry in /proc/mounts looks like this:

tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=17560k,mode=755 0 0

When we look at the initrd.gz file for Ubuntu 12.04.2, it has a size of around 6MB and when expanded is upwards of 16MB. As you can guess, when attempting to launch a DomU in a Dom0 with pygrub in such limited resources, you will run out of space in the /run hierarchy when expanding initrd.gz resulting in the [Errno 28] error I quoted earlier.

You have two easy options (there may be more) to get your DomU booted:

1. Use the bootloader_arg solution I described above into some other directory in a file system that has plenty of space to hold the DomU's expanded initrd.gz.

2. Temporarily remount the tmpfs with a larger maximum size (/run will revert back to original size with next reboot) with the command:

mount –t tmpfs tmpfs /run –o size=20M,mode=1777,remount

You can also add this step into the rc.local script or somewhere else appropriate to make it a permanent occurrence. However, if you automatically start Xen domains at machine startup, the remount may not occur in time as rc.local script does not happen until the very end of the boot sequence.

Another more difficult (difficult in the sense that I don't know how to do it), is to modify how the kernel boots and constructs the /run file system. The advantage there is that it would happen prior to any user processes are kicked off and therefore in time for Xen domains to take advantage of the increased size.

Hope this helps...
The reply is currently minimized Show
Responses (3)
  • Accepted Answer

    Wednesday, July 03 2013, 08:34 PM - #permalink
    Have you verified that your LVM disk is not full? At a little under 4GB, it would be easy to fill it up depending on what you installed.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, July 04 2013, 09:06 PM - #permalink
    Thanks, this is a brand new PV guest immediately after installation of the Ubuntu distribution. I have not had time to do anything with it yet.

    What flashes by on the console is the Grub menu and then an immediate message as I described above.

    If there was no room for "writing the temporary copy of the ramdisk" I would have thought I would have gotten that during installation. I have tried bumping the system memory up to 256M but that has made no difference.

    For what is worth, here are the contents of my guest configuration file (excuse any typos I might make):

    bootloader = 'pygrub'
    name = 'testMachine'
    memory = '256'
    disk = ['/etc/xen/domUs/testMachine/disk1.img,,xvda,w'
    vif = ['bridge=xenbr0']
    vcpus = 1
    on_reboot = "restart"
    on_crash = "restart"
    The reply is currently minimized Show
  • Accepted Answer

    Adam Bolte
    Adam Bolte
    Tuesday, February 23 2016, 05:26 AM - #permalink
    I too just hit this on an older machine running Debian Wheezy after upgrading a guest to Jessie (which uses an initramfs ~3Mb larger in size on my amd64 architecture), so your post was super helpful.

    I'm not sure about Ubuntu, but in Debian (Wheezy and Jessie at least), the tmpfs man page says that the default max tmpfs size value of 10% for /run can be adjusted by setting something like
    in the /etc/default/tmpfs configuration file. This is sourced by /lib/init/tmpfs.sh, which in turn is sourced by /etc/init.d/mountkernfs.sh (part of the initscripts package for use by sysvinit).

    It's not my intention to "re-open" an old thread, but I just want to leave this note here in case anyone else also comes across the same issue in future.
    The reply is currently minimized Show
Your Reply