Tuesday, September 05, 2006

Linux fails to check bounds...

While I was testing a program for limiting its data segment size on a Linux box, I found something very strange. I specified the limit to the data segment size for my process by 'ulimit -d '. But to my surprise the program's behaviour was very weird, it kept on running without any problems whereas it should have failed at some point. I tried this small test wherein I typed the following commands to see if bound checking is actually done. I did this on both Linux and Solaris machines and the output is as follows:

Linux:
> ulimit -d 1
> ls -la


This seems to run fine and display contents of the directory.


Solaris:
> ulimit -d 1
> ls -la


This did not work. 'ls -la' gave out an error, "Not enough memory".

This means that the limits set through ulimit have no effect so to say on Linux. This site has more details on why such behaviour: http://isec.pl/vulnerabilities/isec-0012-do_brk.txt

No comments: