Graphics Programming Support

From Immersive Visualization Lab Wiki
Revision as of 11:26, 23 February 2012 by Jschulze (Talk | contribs)

Jump to: navigation, search

Contents

Lab hardware

We are using CentOS 5.3 on most of our lab machines. The names of the lab machines you have an account on are:

  • flint.ucsd.edu: Intel Pentium D based Dell GX620 on 2nd floor
  • gneiss.ucsd.edu: Intel Pentium D based Dell GX620 on 2nd floor
  • sand.ucsd.edu: Intel Pentium D based Dell GX620 on 2nd floor
  • sessions.ucsd.edu: Intel Pentium D based Dell GX620 in room 6307
  • chert.ucsd.edu: on 2nd floor
  • tuff.ucsd.edu: on 2nd floor
  • pumice.ucsd.edu: on 2nd floor
  • shale.ucsd.edu: on 2nd floor
  • cwall-1.ucsd.edu: Dell XPS, drives upper half of c-wall in room 6307
  • cwall-2.ucsd.edu: Dell XPS, drives lower half of c-wall in room 6307
  • megapixel.ucsd.edu: in Auditorium control room

All these machines can be accessed from the public internet. You can log in with your username by using ssh. Example for user 'jschulze' logging into 'flint': ssh jschulze@flint.ucsd.edu

Account information

If you change your password on any of our lab machines make sure you change it on sessions first and tell an administrator so we can propagate the change to all lab machines.

In order to cross login between lab machines without a password you will need to create a DSA key pair. To generate this you run the command 'ssh-keygen -t dsa' on any lab machine and use the default file names, and do not enter a pass phrase. This should generate two files in your ~/.ssh/ directory: id_dsa and id_dsa.pub. The last step is to copy the file id_dsa.pub to new file authorized_keys, or append the contents of id_dsa.pub to authorized_keys if it already exists.

Intersection testing

If you have wondered how you can find out if the wand pointer intersects your objects, here is a template routine for it. You need to pass it the beginning and end of a line you're intersecting with, in world coordinates. The line will be starting from the hand position and extend along the Y axis. 

#include <osgUtil/IntersectVisitor>

class IsectInfo     // this is an optional class to illustrate the return values of the accept() function
{
  public:
      bool       found;              ///< false: no intersection found
      osg::Vec3  point;              ///< intersection point
      osg::Vec3  normal;             ///< intersection normal
      osg::Geode *geode;             ///< intersected Geode
};

void getObjectIntersection(osg::Node *root, osg::Vec3& wPointerStart, osg::Vec3& wPointerEnd, IsectInfo& isect)
{
    // Compute intersections of viewing ray with objects:
    osgUtil::IntersectVisitor iv;
    osg::ref_ptr<osg::LineSegment> testSegment = new osg::LineSegment();
    testSegment->set(wPointerStart, wPointerEnd);
    iv.addLineSegment(testSegment.get());
    iv.setTraversalMask(2);

    // Traverse the whole scenegraph.
    // Non-Interactive objects must have been marked with setNodeMask(~2):     root->accept(iv);
    isect.found = false;
    if (iv.hits())
    {
        osgUtil::IntersectVisitor::HitList& hitList = iv.getHitList(testSegment.get());
        if(!hitList.empty())
        {
            isect.point     = hitList.front().getWorldIntersectPoint();
            isect.normal    = hitList.front().getWorldIntersectNormal();
            isect.geode     = hitList.front()._geode.get();
            isect.found     = true;
        }
    }
}


Interaction handling

To register an interaction so that only your plugin uses the mouse pointer while a button on the wand is pressed, you want to use the TrackerButtonInteraction class. For sample usage see plugins/Volume. Here are the main calls you will need. Most of these functions go in the preFrame routine, unless otherwise noted:

  • Make sure you include at the top of your code: 
     
      #include <OpenVRUI/coTrackerButtonInteraction.h>
  • In the constructor you want to create your interaction for button A, which is the left wand button: 
          interaction = new coTrackerButtonInteraction(coInteraction::ButtonA,"MoveObject",coInteraction::Menu);
  • In the destructor you want to call: 
          delete interaction;
  • The code for handling the interaction needs to go in the preFrame() function. To register your interaction and thus disable button A interaction in all other plugins call the following function. Make sure that to call this function before other modules can register the interaction. In particular, this might mean that you need to register the interaction before a mouse button is pressed, for instance by registering it when intersecting with the object to interact with. 
          if(!interaction->registered)
      {
         coInteractionManager::the()->registerInteraction(interaction);
      }
  • To do something just once, after the interaction has just started: 
          if(interaction->wasStarted())
      {
      }
  • To do something every frame while the interaction is running: 
         if(interaction->isRunning())
     {
     }
  • To do something once at the end of the interaction: 
         if(interaction->wasStopped())
     {
     }
  • To unregister the interaction and free button A for other plugins: 
          if(interaction->registered && (interaction->getState()!=coInteraction::Active))
      {
         coInteractionManager::the()->unregisterInteraction(interaction);
      }

OSG Text

Make sure you #include <osgText/Text>. There is a good example for how osgText can be used in ~/covise/src/renderer/OpenCOVER/osgcaveui/Card.cpp. _highlight is the osg::Geode the text gets created for, createLabel() returns the Drawable with the text, _labelText is the text string, and osgText::readFontFile() reads the font.

How to Create a Rectangle with a Texture

The following code sample from osgcaveui/Card.cpp demonstrates how to create a rectangular geometry with a texture. 

Geometry* createIcon()
{
  Texture2D _icon = new Texture2D();
  Image* image = NULL;
  image = osgDB::readImageFile("image.jpg");  // make sure it's power of 2 edges
  if (image)
  {
    _icon->setImage(image);
  }
  else return NULL;

  Geometry* geom = new Geometry();

  Vec3Array* vertices = new Vec3Array(4);
  float marginX = (DEFAULT_CARD_WIDTH  - ICON_SIZE * DEFAULT_CARD_WIDTH) / 2.0;
  float marginY = marginX;
                                                  // bottom left
  (*vertices)[0].set(-DEFAULT_CARD_WIDTH / 2.0 + marginX, DEFAULT_CARD_HEIGHT /
2.0 - marginY - ICON_SIZE * DEFAULT_CARD_WIDTH, EPSILON_Z);
                                                  // bottom right
  (*vertices)[1].set( DEFAULT_CARD_WIDTH / 2.0 - marginX, DEFAULT_CARD_HEIGHT /
2.0 - marginY - ICON_SIZE * DEFAULT_CARD_WIDTH, EPSILON_Z);
                                                  // top right
  (*vertices)[2].set( DEFAULT_CARD_WIDTH / 2.0 - marginX, DEFAULT_CARD_HEIGHT /
2.0 - marginY, EPSILON_Z);
                                                  // top left
  (*vertices)[3].set(-DEFAULT_CARD_WIDTH / 2.0 + marginX, DEFAULT_CARD_HEIGHT /
2.0 - marginY, EPSILON_Z);
  geom->setVertexArray(vertices);

  Vec2Array* texcoords = new Vec2Array(4);
  (*texcoords)[0].set(0.0, 0.0);
  (*texcoords)[1].set(1.0, 0.0);
  (*texcoords)[2].set(1.0, 1.0);
  (*texcoords)[3].set(0.0, 1.0);
  geom->setTexCoordArray(0,texcoords);

  Vec3Array* normals = new Vec3Array(1);
  (*normals)[0].set(0.0f, 0.0f, 1.0f);
  geom->setNormalArray(normals);
  geom->setNormalBinding(Geometry::BIND_OVERALL);

  Vec4Array* colors = new Vec4Array(1);
  (*colors)[0].set(1.0, 1.0, 1.0, 1.0);
  geom->setColorArray(colors);
  geom->setColorBinding(Geometry::BIND_OVERALL);

  geom->addPrimitiveSet(new DrawArrays(PrimitiveSet::QUADS, 0, 4));

  // Texture:
  StateSet* stateset = geom->getOrCreateStateSet();
  stateset->setMode(GL_LIGHTING, StateAttribute::OFF);
  stateset->setRenderingHint(StateSet::TRANSPARENT_BIN);
  stateset->setTextureAttributeAndModes(0, _icon, StateAttribute::ON);

  return geom;
}

Color Specification: Materials

If your text or geometry doesn't show the color you gave it and is just white, or if you have to turn lighting off to see the color, the geometry is probably missing a material.

You can fix this by adding a material to your geode: 

// color is RGBA, A should always be 1 for opaque objects, other values are 0..1
// example: setColor(Vec4(1.0,0.0,0.0,1.0)) for a red object
setColor(Vec4 color)  
{
    StateSet* stateSet = geode->getOrCreateStateSet();
    Material* mat = new Material();
    mat->setColorMode(Material::AMBIENT_AND_DIFFUSE);
    mat->setDiffuse(Material::FRONT,color);
    mat->setSpecular(Material::FRONT,color);
    stateSet->setAttribute(mat);
    stateSet->setAttributeAndModes(mat, StateAttribute::ON);
}

Khanh Luc created http://ivl.calit2.net/wiki/index.php/MatEdit, a material editor GUI which will generate OSG source code. The executable is at /home/covise/covise/extern_libs/src/MatEdit/Release/MatEdit.

Load and display an image from disk

Here is sample code to create a geode (imageGeode) with an image which gets loaded from disk. OpenGL's size limitation for textures applies (usually 4096x4096 pixels). The image might have to have powers of two edges, but that limitation should not hold anymore on newer graphics cards. 

/** Loads image file into geode; returns NULL if image file cannot be loaded */
Geode* createImageGeode(const char* filename)
{
  // Create OSG image:
  Image* image = new Image();
  image = osgDB::readImageFile(filename);

  // Create OSG texture:
  if (image)
  {
    imageTexture = new Texture2D();
    imageTexture->setImage(image);
  }
  else 
  {
    std::cerr << "Cannot load image file " << filename << std::endl;
    delete image;
    return NULL;
  }

  // Create OSG geode:
  imageGeode = new Geode();
  imageGeode->addDrawable(createImageGeometry());
  return imageGeode;
}

/** Used by createImageGeode() */
Geometry* createImageGeometry()
{
  const float WIDTH  = 3.0f;
  const float HEIGHT = 2.0f;
  Geometry* geom = new Geometry();

  // Create vertices:
  Vec3Array* vertices = new Vec3Array(4);
  (*vertices)[0].set(-WIDTH / 2.0, -HEIGHT / 2.0, 0); // bottom left
  (*vertices)[1].set( WIDTH / 2.0, -HEIGHT / 2.0, 0); // bottom right
  (*vertices)[2].set( WIDTH / 2.0, HEIGHT / 2.0, 0); // top right
  (*vertices)[3].set(-WIDTH / 2.0, HEIGHT / 2.0, 0); // top left
  geom->setVertexArray(vertices);

  // Create texture coordinates for image texture:
  Vec2Array* texcoords = new Vec2Array(4);
  (*texcoords)[0].set(0.0, 0.0);
  (*texcoords)[1].set(1.0, 0.0);
  (*texcoords)[2].set(1.0, 1.0);
  (*texcoords)[3].set(0.0, 1.0);
  geom->setTexCoordArray(0,texcoords);

  // Create normals:
  Vec3Array* normals = new Vec3Array(1);
  (*normals)[0].set(0.0f, 0.0f, 1.0f);
  geom->setNormalArray(normals);
  geom->setNormalBinding(Geometry::BIND_OVERALL);

  // Create colors:
  Vec4Array* colors = new Vec4Array(1);
  (*colors)[0].set(1.0, 1.0, 1.0, 1.0);
  geom->setColorArray(colors);
  geom->setColorBinding(Geometry::BIND_OVERALL);

  geom->addPrimitiveSet(new DrawArrays(PrimitiveSet::QUADS, 0, 4));

  // Set texture parameters:
  StateSet* stateset = geom->getOrCreateStateSet();
  stateset->setMode(GL_LIGHTING, StateAttribute::OFF);  // make texture visible independent of lighting
  stateset->setRenderingHint(StateSet::TRANSPARENT_BIN);  // only required for translucent images
  stateset->setTextureAttributeAndModes(0, imageTexture, StateAttribute::ON);

  return geom;
}


Backtrack all nodes up to top of scene graph

osg::NodePath path = getNodePath();
ref_ptr<osg::StateSet> state = new osg::StateSet;
for (osg::NodePath::iterator it = path.begin(); it != path.end(); ++it)
{
  if ((*it)->getStateSet())
  {
    state->merge((*it)->getStateSet());
  }
}


Taking a screenshot from the command line

  • run application on visualtest02 and bring up desired image, freeze head tracking
  • log on to coutsound
  • Make sure screenshot is taken of visualtest02: setenv DISPLAY :0.0
  • Take screenshot: xwd -root -out <screenshot_filename.xwd>
  • Convert image file to TIFF: convert <screenshot_filename.xwd> <screenshot_filename.tif>


Return a ref_ptr from a function

Here is a safe way to return a ref_ptr type from a function. 

osg::ref_ptr<osg::Group> makeGroup(...Some arguments..) 
{
  osg::ref_ptr<osg::MatrixTransform> mt=new MatrixTransform();
  // ...some operations...
  return mt.get();
}

Also check out this link to learn more about how to use ref_ptr: http://donburns.net/OSG/Articles/RefPointers/RefPointers.html



Moving an object with the pointer

Here is some sample code to move an object. object2w is the object's transformation matrix in world space. lastWand2w and wand2w are the wand matrices from the previous and current frames, respectively, from cover->getPointer(). 

void move(Matrix& lastWand2w, Matrix& wand2w)
{
    // Compute difference matrix between last and current wand:
    Matrix invLastWand2w = Matrix::inverse(lastWand2w);
    Matrix wDiff = invLastWand2w * wand2w;

    // Perform move:
    _node->setMatrix(object2w * wDiff);
}

Using Shared Memory

A great tutorial page is at: http://www.ecst.csuchico.edu/~beej/guide/ipc/shmem.html

Find out which node a plugin is running on

The following routine works on our Varrier system to find out the host number, starting with 0. The node names are vellum1-10, vellum2-10, etc. 

int getNodeIndex()
{
  char hostname[256];
  gethostname(hostname, sizeof(hostname));
  int node;
  sscanf(hostname, "vellum%d-10", &node);  // this needs to be adjusted to naming convention
  return (node-1);  // names start with 1
}

For the StarCave: 

int getStarCaveNodeIndex()
{
  char hostname[256];
  gethostname(hostname, sizeof(hostname));
  int row,column;
  sscanf(hostname, "tile-%d-%d.local", &column,&row);
  return (column*3 + row);  // names start with 0, goes up to 14
}

Hide or select different pointer

The pointer is by default a line coming out of the users's hand held device. Icons like an airplane, a steering wheel, slippers, or a magnifying glass indicate the current navigation mode. This is the mode the left mouse button will use. To hide the pointer, remove all geodes which are part of the pointer: 

  while(VRSceneGraph::instance()->getHandTransform()->getNumChildren())
    VRSceneGraph::instance()->getHandTransform()->removeChild(m_handScaledTransform->getChild(0));

To select a different pre-defined pointer: 

  VRSceneGraph::instance()->setPointerType(<new_type>);

Hide the mouse cursor

  osgViewer::Viewer::Windows windows;
  viewer.getWindows(windows);
  for(osgViewer::Viewer::Windows::iterator itr = windows.begin(); itr != windows.end(); ++itr)
  {
    (*itr)->useCursor(false);
  }

Keyboard Input

This is what you need to do to process keyboard events in a plugin: Define a function in your main class with a sig like void key (int type, int keySym, int mod). In the area of your code where you define coVRInit, coVRDelete, coVRPreFrame, etc. place a hook like this: 

void coVRKey(int type, int keySym, int mod) 
{
  plugin->key(type, keySym, mod);
}

The function is called when a key is pressed or released. type indicates if the key has just been pressed or released (values 6 and 7), keySym identifies the key, and mod is normally 0.

Fast Shader-Based Spheres

Use the class coSphere. A single instance of the class suffices. Things you need to do outside of having a valid COVISEDIR environment variable:

  • Set the radius. It is initialized to NULL so it wont have a valid radius until it is set.
  • Set coordinates. Like the radius, it is initialized to NULL so coordinates must be set.
  • Add the geode to a group node.

Here is a piece of sample code: 

void drawSphere(osg::Group* root)
{
	coSphere *drawable = new coSphere();

	drawable->setNumberOfSpheres(1);
	float radius[] = {0.5f};
	float coordsX[] = {0.0f};
	float coordsY[] = {0.0f};
	float coordsZ[] = {0.0f};
	drawable->updateRadii(radius);
	drawable->updateCoords(coordsX,coordsY,coordsZ);

	drawable->setColor(0,0.0f, 1.0f, 0.0f, 1.0f); // set color of 0th sphere

	osg::Geode *geode = new osg::Geode();
	root->addChild(geode);
	geode->addDrawable(drawable);
}

Andrew Note: It seems to be using the first light source that is enabled to draw the spheres. I will try to add the ability to set the light source soon.


Sound Effects with the Audio Server

Documentation for the Audio Server is at: [[1]]

COVISE's VRML reader supports audio nodes and has built in playback functions for the Audio Server.

The main driver file is 
covise/src/kernel/vrml97/vrml/PlayerAServer.cpp

If vrml97.pro won't link and complains about missing jpeg functions, add the following lines to .cshrc: 

  setenv JPEG_LIBS -ljpeg
  setenv JPEG_INCPATH /usr/include

Covise Animation Manager

Covise's Animation Manager provides a simple way to mark time. It allows the user to navigate through a series of frames either automatically at a variable frame-rate, or manually by stepping forwards or backwards.

First, include the following line of code: 

  #include <kernel/coVRAnimationManager.h>

Here is a simple example of an Animation Manager setup. 

  coVRAnimationManager::instance()->setAnimationSpeed(int framerate);  //Set the default frame-rate for playback
  coVRAnimationManager::instance()->enableAnimation(bool play);        //Set the animation to play/pause by default
  coVRAnimationManager::instance()->setAnimationFrame(int frame);      //Set the first frame
  coVRAnimationManager::instance()->showAnimMenu(bool on);             //Add the "Animation" SubMenu to the Opencover Main Menu, or remove it
  coVRAnimationManager::instance()->setNumTimesteps(int steps, this);  //Set the total number of frames to cycle through

This can be done at initialization or at any point during program execution.

The Animation Manager has been setup and can now provide information to the rest of your program. Calling 

  coVRAnimationManager::instance()->getAnimationFrame();

Will return the current frame number. This frame number will automatically loop back to zero after it reaches the value provided to setNumTimesteps.

Finally, you can jump to a specific frame by calling 

  coVRAnimationManager::instance()->setAnimationFrame(int framenumber);


Blackmagic Intensity Pro Capture Card Support

This section explains how to get the Intensity Pro card from Blackmagic working on a CentOS system.

  • 1) Physically install the card into the system.
  • 2) Install the latest x86_64 .rpm Intensity drivers from the Blackmagic Website
  • 3) Make sure the permissions for the driver, located at /dev/blackmagic/card0, are set to 777.
  • 4) Run the BlackmagicControlPanel application and make sure the changes you make are saved. (You might have to log in as the root user to do this).
  • 5) Download the Blackmagic SDK.
  • 6) Hook a video source to the video input HDMI slot of the card. Find the "Capture" sample code in the SDK and run with each possible video input type until you find one that consistently captures input.
  • 7) Use the Capture code in your own plugin to get live captured data. (Note: Images come in encoded in the YUV422 format)

Simplifying Geometry and Optimizing with OSG

In order to improve the performance of your plug-in, you may wish to run an optimizer on it. The OpenSceneGraph Optimizer can be called on any osg::Node, and it will apply the optimization to the node and its subgraph. The Optimizer can perform the following operations: (Source)

  • FLATTEN_STATIC_TRANSFORMS - Flatten Static Transform nodes by applying their transform to the geometry on the leaves of the scene graph, and then removing the now redundant transforms.
  • REMOVE_REDUNDANT_NODES - Remove redundant nodes, such as groups with one single child.
  • REMOVE_LOADED_PROXY_NODES - Remove loaded proxy nodes.
  • COMBINE_ADJACENT_LODS - Optimize the LOD groups, by combining adjacent LOD's which have complementary ranges.
  • SHARE_DUPLICATE_STATE - Optimize State in the scene graph by removing duplicate state, replacing it with shared instances, both for StateAttributes, and whole StateSets.
  • MERGE_GEOMETRY - Not documented.
  • CHECK_GEOMETRY - Not documented.
  • SPATIALIZE_GROUPS - Spatialize scene into a balanced quad/oct tree.
  • COPY_SHARED_NODES - Copy any shared subgraphs, enabling flattening of static transforms.
  • TRISTRIP_GEOMETRY - Not documented.
  • TESSELLATE_GEOMETRY - Tessellate all geodes, to remove POLYGONS.
  • OPTIMIZE_TEXTURE_SETTINGS - Optimize texture usage in the scene graph by combining textures into texture atlas. Use of texture atlas cuts down on the number of separate states in the scene, reducing state changes and improving the chances of use larger batches of geometry.
  • MERGE_GEODES - Combine geodes.
  • FLATTEN_BILLBOARDS - Flatten MatrixTransform/Billboard pairs.
  • TEXTURE_ATLAS_BUILDER - Texture Atlas Builder creates a set of textures/images which each contain multiple images.
  • STATIC_OBJECT_DETECTION - Optimize the setting of StateSet and Geometry objects in scene so that they have a STATIC DataVariance when they don't have any callbacks associated with them.
  • FLATTEN_STATIC_TRANSFORMS_DUPLICATING_SHARED_SUBGRAPHS - Remove static transforms from the scene graph, pushing down the transforms to the geometry leaves of the scene graph. Any subgraphs that are shared between different transforms of duplicated and flatten individually.
  • ALL_OPTIMIZATIONS - Performs all of the optimizations listed above
  • DEFAULT_OPTIMIZATIONS - Performs all of the default optimizations.

Sample Code: 

#include <osgUtil/Optimizer>
osgUtil::Optimizer optimizer;
optimizer.optimize(pyramidGeode, osgUtil::Optimizer::ALL_OPTIMIZATIONS);

You can also use the Simplifier to reduce the number of triangles in an osg::Geometry node. The Simplifier will do its best to maintain the shape of the geometry while reducing the number of triangles.

Sample Code: 

#include <osgUtil/Simplifier>
osg::Geometry* geometry;
//some code...
osgUtil::Simplifier simple;
simple.setSampleRatio(0.7f); //reduces the number of triangles by 30% 
geometry->accept(simple)

Problems When Mixing OpenGL With OSG

OSG allows OpenGL code to be called within a Drawable's DrawImplementation.

Problem: OpenGL geometry gets culled when it should be displayed.

Resolution: The problem is most likely that the OpenGL geometry does not have a proper OSG bounding box. Every time a custom Drawable is created with custom OpenGL code, the bounding box of this Drawable needs to be carefully adjusted to contain the entire geometry generated by the OpenGL commands.


Problem: Textures do not show up or get corrupted, even in the menu.

Resolution: This is likely an OSG state issue. Whenever the OpenGL state gets changed by custom OpenGL code outside of OSG commands, it needs to carefully be restored before control is returned to OSG. In practice this means that at the beginning of the drawImplementation in which OpenGL code is executed, the OpenGL state needs to be saved, and at the end it needs to be restored. If any parts of the state are not restored correctly, the observed problems may be observed.


Problem: Textures and other OpenGL context specific objects do not show up on all screens.

Resolution: This can happen if custom OpenGL does not pay attention to the context ID. All user created OpenGL objects(textures, VBOs, etc.) need to be generated and have their data loaded separately in each context. For a custom OSG drawable you will get a call to the drawImplementation function from each render context each frame. You can find what context you are in by using the getContextID function in the RenderInfo object passed into the drawImplementation function. Also note that the draw call can potentially be multi-threaded.

Useful Links

  • osgWorks adds useful functionality to OpenSceneGraph
Retrieved from "http://ivl.calit2.net/wiki/index.php?title=Graphics_Programming_Support&oldid=4617"