vmware-specific-specific 5.5.x and kernel 2.6.29-2.6.30
Posted on April 21st, 2009
You can follow any responses to this entry through the RSS 2.0 feed.
Once again, here’s a patch to have vmware 5.5.x (and vmware server 1.0.x AFAIK) working with the latest kernels 2.6.29 and 2.6.30.
Sorry for the delay 
As usual this patch only work on x86.
Since I didnt take the time to work around the un-exported init_mm symbol in the kernel, this time the patch require patching your kernel. Please follow the README included in the archive, or in simple words, just apply the 2.6.29-export-init_mm.patch file on your kernel source tree and recompile/reboot before upgrading vmware.
As usual, have fun ;p
Update:
Fixed missing commit_creds(), thanks dpalffy
Fixed memory leak
Works without modification on 2.6.30. The same symbol export patch can be used.
April 26th, 2009 at 11:31 pm
Thx for notifying me of the update
The patch isn’t necessary anymore for Jaunty (but a different one is required).
April 27th, 2009 at 6:15 pm
In vmmon_only/linux/driver.c: __LinuxDriver_Ioctl() you missed a commit_creds() I think, new is never commited.
April 27th, 2009 at 9:43 pm
works great with vmware-server 1.0.9 and kernel 2.6.29.1
big thanks for your work!
>>>cyber
April 29th, 2009 at 7:37 pm
Hi!
now I have got big problems … after about 4½ hours running one vmware my hostsystem starts to kill processes with oom …
I have got statistics with munin, I can see increasing slab_cache constantly.
do you think that your modules are affacted by a memory leak?
I’ve used with kernel 2.6.29 also modules from it-psycho.de – but I’ve got problems with them to virtualize a 2.6 vmware …
I hope you read my message …
kind regards, Alois
April 30th, 2009 at 3:20 am
I can second this. Do *NOT* use this patch on vmware 1.0.9.
It crashes your VM’s after some minutes or hours due to a memory leak.
April 30th, 2009 at 8:34 am
the memory leak is in vmmon module – perhaps in combination with virtual disc access (I emulate IDE) – as soon as I do much write / read access slap_cache increases.
kind regards, Alois
April 30th, 2009 at 10:56 am
Cyber, Jonas,
Have you tried the “-2″ version of this patch?
The vmmon diff isnt very big, I didn’t see anything obvious (beside that the patch isnt very very nice
It could be some error in the module that wasnt triggered with previous kernels as well.
April 30th, 2009 at 4:05 pm
Hi kang,
jes, this is version -2 of this patch.
with “slaptop” I can see kmalloc-128 increasing instantly – per second about 100k. perhaps this helps … thanks.
>>>cyber
April 30th, 2009 at 8:54 pm
hi kang,
with debug-output I’ve found the function which allocates more and more memory – can you imagine why?
thanks a lot for your help! (:
kind regards, Alois
vmmon-only/linux/hostif.c
void *
HostIF_AllocKernelMem(size_t size, int wired)
{
void * ptr = kmalloc(size, GFP_KERNEL);
printk(KERN_NOTICE “vmware-kmalloc3 %i\n”,size);
if (ptr==NULL) {
Warning(“HostIF_AllocKernelMem failed (size=%p)\n”, (void*)size);
}
return ptr;
}
April 30th, 2009 at 9:38 pm
that’s the memory allocating function (stating the obvious i know
Anyway, vmware wraps around it with this HostIF_AllocKernelMem, so, I don’t know which debug output you used, but maybe you have the call to the previous function? Unless something in kmalloc changed (or kmem_cache)
I will probably only be able to look it up on the next monday unless you have to function called and we get lucky. or maybe someone else will post a patch in between
May 4th, 2009 at 8:05 pm
hi kang,
I tried to put debug-output (debug-output from anove see the posted function – the printk(KERN_NOTICE “vmware-kmalloc3 %i\n”,size); writes debug-messages to klogd so I was able to detect that this function is affected) in the whole vmmon-source, where HostIF_AllocKernelMem is called.
I tried it with the method as above, bug as soon as I add similar lines the vmmon-module is not able to compile – I am sorry, I am not able to “speak” c … ):
it would be great if you can take a look at it!
kind regards, Alois
May 12th, 2009 at 1:49 am
HI KANG! nice site! xD
May 13th, 2009 at 11:14 am
Need some help
[root@sunjoy 2.6.29.2-52.fc10.x86_64]# patch -p1 -i /home/radhju/Desktop/downloads/vmware-update-2.6.29-5.5.9/2.6.29-export-init_mm.patch
can’t find file to patch at input line 5
Perhaps you used the wrong -p or –strip option?
The text leading up to this was:
————————–
|diff –git a/arch/x86/kernel/init_task.c b/arch/x86/kernel/init_task.c
|index 5b3ce79..3d01e47 100644
|— a/arch/x86/kernel/init_task.c
|+++ b/arch/x86/kernel/init_task.c
————————–
File to patch:
May 15th, 2009 at 10:32 am
hey sanjay,
you have to patch your linux kernel source (usually at /usr/src/linux) and there the file arch/x86/kernel/init_task.c
@kang: i’m daily visiting your page with hope you will find the memory leak … ):
>>>cyber
May 19th, 2009 at 3:12 pm
hi cyber, please download the new archive, this fixes the memory leak i believe you have, please tell me if it works now
May 27th, 2009 at 2:53 pm
Hi kang,
yeah, fixed. ((:
not memory does not increase any more! ((:
Thanks a lot for your professional work!!!
Great!
>>>cyber
June 13th, 2009 at 4:53 am
Any chance to get rid of the kenel patch?
June 21st, 2009 at 8:26 pm
I’m trying to get VMWare Workstation 5.5.9 running. I’m willing to attempt to patch the kernel and I think I have the source files but… ugh, I don’t have an init_task.c file. There’s an init_task.h but no reference to init_mm in it. What now?
I just installed Fedora 11, 2.6.29.4-167.fc11.x86_64.
June 22nd, 2009 at 12:29 am
Alright, I kept digging and discovered I didn’t have the kernel source installed. Found this article …
http://www.howtoforge.com/kernel_compilation_fedora_p2
… and VMWare works now! Thanks for the patch
June 23rd, 2009 at 3:26 pm
Thanks for this patch. It works fine for me but I have two questions on the code -
one relating to yours and second relating to the original vmware code :
1. in linux/driver_compat.h LinuxDriverLockedNoPage() (file added by http://www.insecure.ws)
after line
pg = fdata->pgoff;
need
pg = VMMON_MAP_OFFSET(pg); /* jel */
2. linux kernel #defines
#define VM_FAULT_SIGBUS 0×0002
but compat_mm.h has
#ifndef NOPAGE_SIGBUS
#define NOPAGE_SIGBUS (0)
#endif
and I believe vmware’s use of its NOPAGE_SIGBUS is same as linux kernel’s use of its VM_FAULT_SIGBUS
so shouldn’t the vmware define be
#define NOPAGE_SIGBUS (2)
September 28th, 2009 at 9:48 am
Hello,
I am having problems with this patch. I use 2.6.30.5-43 PAE kernel and I cannot install vmwares server ANY VERSION. I get this error, and I do not know if there is something I made wrong or the patch is not for my kernel. Thank you in advance.