It was really a pain for me to work on Oracle VM, It won't allow you to use mouse or do copy-paste. So I decided to connect my local VM via SSH( Putty).
Steps :
1. In your VM box, Goto Settings -> Network ->Adapter 1 and select "Bridged Adapter"
2. In your Ubuntu Machine, Install "openssh-server"
sudo apt-get install openssh-server
3. Reboot the VM and Run "ifconfig" command in terminal and get "inet addr" of "eth0".
4. Open putty, Enter the IP address(IP of your VM machine ) , Select port as "22" and connection type as "SSH" and click on open button.
5. Enter your credential and you are done :)