Project2S18

Levels of Immersion

In this project we are going to explore different levels of immersion with the Oculus Rift. Like in project 1, we're going to use the Oculus SDK, OpenGL and C++.

Starter Code

We recommend starting either with your project 1 code, or going back to the starter code. It's very compact but yet uses most of the API functions you will need.

Here is the minimal example .cpp file with comments on where the hooks are for the various functions required for this homework project. Note that this file has a main() function, so that it can be built as a console application, and will start up with a console window into which debug messages can be printed.

Project Description (100 Points)

You need to do the following things:

1. Modify the starter code (in which many cubes are rendered) to render just two cubes, one behind another along the user's line of sight.
2. Use texture mapping to display this calibration image on all faces of both cubes. The image needs to be right side up on the vertical faces, and make sure it's not mirrored. Each cube should be 20cm wide and its closest face should be about 30cm in front of the user. Here is source code to load the PPM file into memory so that you can apply it as a texture. Sample code for texture loading can be found in the OpenVR OpenGL example's function SetupTexturemaps().
3. Render this stereo panorama image around the user as a 10m wide sky box with the user at its center. The trick here is to render different textures on the sky box for each eye, which is necessary to achieve a 3D effect. Cycle with the 'X' button between showing the entire scene (cubes and sky box), just the sky box in stereo, and just the sky box in mono (i.e., one of the panorama images is rendered to both eyes).
4. Gradually vary the physical size of both cubes with the left thumb stick left/right. This means changing the width of the cubes from 30cm to smaller or larger values. Pushing down on the thumb stick should reset the cubes to their initial sizes. Support a range from 0.01m to 0.5m. Make sure the cubes' center points do not move when their size changes (i.e., scale the cubes about their centers).
5. Cycle between the following four modes with the 'A' button: 3D stereo, mono (the same image rendered on both eyes), left eye only (right eye black), right eye only (left eye black), inverted stereo (left eye image rendered to right eye and vice versa). Regardless of which mode is active, head tracking should work correctly, depending on which mode it's in as described below.
6. Cycle between different head tracking modes with the 'B' button: no tracking (position and orientation frozen to what they just were before the user pressed the button), orientation only (position frozen to what it just was before the mode was selected), position only (orientation frozen to what it just was), both (which is the normal tracking mode).
7. Gradually vary the interocular distance (IOD) with the right thumb stick left/right. Pushing down on the thumb stick should reset the IOD to the default. You'll have to learn about how the Oculus SDK specifies the IOD. They don't just use one number for the separation distance, but each eye has a 3D offset from a central point. Find out what these offsets are in the default case, and modify only their horizontal offsets to change the IOD, leaving the other two offsets untouched. Support an IOD range from -0.1m to +0.2m.

Notes:

• Cycling means that each time the respective button is pressed the viewing mode will change from one mode to the next, and eventually back to the first mode.
• The view in the Rift always needs to look like the control window on the screen: the render texture should never shift off the display panels in the Rift.

Notes on Panorama Rendering

There are six cube map images in PPM format in the ZIP file. Each is 2k x 2k pixels in size. The files are named nx, ny and nz for the negative x, y and z axis images. The positive axis files are named px, py and pz. Here is a downsized picture of the panorama image:

And this is how the cube map faces are labeled:

The panorama was shot with camera lenses parallel to one another, so the resulting cube maps will need to be separated by a human eye distance when rendered, i.e., their physical placement needs to be horizontally offset from each other for each eye. Support all of the functions you implemented for the calibration cube also for the sky box (A, B and X buttons, both thumb sticks).

Extra Credit (up to 10 points)

There are four options for extra credit. Note that the options above aren't cumulative. You can get points for only one of them.

1. Take two photos at an eye distance apart (about 65mm), and show each of them in one eye of the Rift without tracking. You should see a stereo image. Make it an alternate option to the Bear image when the 'X' button is pressed. (3 points)
2. Create your own monoscopic sky box: borrow the Samsung Gear 360 camera from the media lab, or use your cell phone's panorama function to capture a 360 degree panorama picture (or use Google's StreetView app, which is free for Android and iPhone). Process it into cube maps - this on-line conversion tool can do this for you. Texture the sky box with the resulting textures. Note you'll have to download each of the six textures separately. Make it an alternate option to the Bear image when the 'X' button is pressed. (5 points)
3. Monoscopic panoramic video: Download one of the 360 degree movies from https://www.videoblocks.com/videos/footage/360-files. Display it as a video texture in the Rift. For that you'll need to extract the image frames from the video, for instance with [www.ffmpeg.com FFMPEG]. Then you need to create a spherical triangle mesh with texture coordinates. Then you render each frame onto the inside of the sphere. Finally, you need to play back the video at the frame rate it was shot at, regardless of the frame rate the Rift renders at (usually 90 fps). Auto-repeat the movie once it ends. (8 points)
4. Stereoscopic panoramic video: Similar to Option 3, but now in stereo. Download a stereo panorama film from the internet (for instance this one which was shot with a GoPro rig). The video may have one eye in the upper half, one in the lower. Using FFMPEG as in Option 3, process the video for each eye separately and play it back in the Rift with head tracking so that you can look around the video as it's playing. Make it run at its original frame rage (fps), despite the Rift rendering at 90fps. (10 points)

--> More to follow...