User Tools

Site Tools


tutorials:legacy:python_panda3d:interactive_bitmaps

This binding has been deprecated - please see the most recent release notes for more information.

Interactive Bitmaps

Introduction

In this tutorial, we further explore Gestureworks Core and the Python bindings by consuming gesture events obtained from Gestureworks Core and manipulating on-screen images represented as OnscreenImages within Panda3D. This tutorial also introduces image manipulation in Panda3D such as rotation and scaling, values for which you will obtain from Gestureworks. For this tutorial you will need the Gestureworks Core multitouch framework; a free trial is available.

In this application, the user is able to drag the objects (the yellow Gestureworks logos) around the screen, scale them using the familiar “pinch”-style gesture, and rotate them using the n-finger rotate gesture.

Download the code for all of the Python & Panda3D multitouch tutorials here: tutorials_python_panda3d.zip

Python 3.1 386x281.png


Requirements

Estimated time to complete: 45 minutes

This tutorial assumes that you have completed the steps found in Python & Panda3D: Getting Started I (Hello World) and Python & Panda3D: Getting Started II (Hello Multitouch). You should have a fully configured Eclipse project ready for use and should be familiar with consuming GestureWorks point event data. If you have have not yet done so, please follow both Tutorial #1 to prepare an Eclipse project and Tutorial #2 to become familiar with consuming GestureWorks touch event data.

In addition to the above, this tutorial expects the student to have a basic understanding of the Python language, the Panda3D framework, and is familiar with object-oriented programming concepts.


Process Overview

Process Details

1. Create a new Eclipse - Panda3D project

For this tutorial it is recommended that you create a new, empty Panda3D project with the gwc_python module using the procedure outlined in Python & Panda3D: Getting Started I (Hello World). You may also use the project created in Python & Panda3D: Getting Started II (Hello Multitouch) but you will need to remove or comment out some unnecessary code.

2. Add the GestureWorks Logo Images to the Project

For this application, two identical images will be used as the textures that will appear on-screen. Using the method described in tutorial #2 for adding resources to the project, add the following folder and all contents to the project:

 \GestureWorksCore\bindings\python\Panda3D\tutorials\03_BitmanManipulation\media

3. Import Required Modules

Again, we'll create a new file called main.py and import the following:

from gwc_python.core import GestureWorksCore
from gwc_python.GWCUtils import TOUCHADDED, TOUCHREMOVED, TOUCHUPDATE, rotateAboutCenter
 
from direct.gui.OnscreenImage import OnscreenImage
from direct.showbase.ShowBase import ShowBase
from direct.task import Task
from pandac.PandaModules import LineSegs
from pandac.PandaModules import deg2Rad
from pandac.PandaModules import NodePath
from pandac.PandaModules import Vec3
from panda3d.core import TransparencyAttrib
from panda3d.core import TextNode
 
from math import radians

4. Set screen dimensions

We want to set parameters for our screen dimensions (for screen height and width). Adjust the values to match your environment.

SCREEN_WIDTH = 1920
SCREEN_HEIGHT = 1080

5. Create TouchContainer Class

We’ll need to create a simple class to hold our touch objects.

class TouchObject():
   pass

6. Create Panda3D App

Just as before, we’ll need a class that inherits the Panda3D ShowBase class for our application to run. We’ll start with the constructor and build method.

   def __init__(self, gw):
       self.gw = gw
       self.touch_points = {}
       self.touch_objects = {}
 
       ShowBase.__init__(self)
       self.build()
       # schedule our processing cycle
       self.taskMgr.add(self.updateGestureWorks, "updateGestureWorksTask")

This is very similar to the previous tutorial, we just have a new dict for holding touch objects and touch points.The build method will require more explanation this time.

   def build(self):
       if not self.gw.registerWindow('Panda'):
           print('Unable to register touch window')
           exit()
 
       # Set background image
       OnscreenImage(parent=render2d, image="media/Logo_gestureworks_core_1920x1080_white.png",pos=(0,0,0))
 
       # Create our touch objects
       container_0 = TouchObject()
       container_0.name = 'object_0'
       container_0.picture = OnscreenImage(image='media/logo.png',pos=(-1,0,0),scale=0.25)
       container_0.scale = 0.25
       self.touch_objects.update({container_0.name: container_0})
 
       container_1 = TouchObject()
       container_1.name = 'object_1'
       container_1.picture = OnscreenImage(image='media/logo.png',pos=(1,0,0),scale=0.25)
       container_1.scale = 0.25
       self.touch_objects.update({container_1.name: container_1})
 
       for name in self.touch_objects:
           # Tell GestureWorks about our touch object and add gestures to it
           self.gw.registerTouchObject(name)
           self.gw.addGesture(name, 'n-drag')
           self.gw.addGesture(name, 'n-rotate')
           self.gw.addGesture(name, 'n-scale')

After we register our touch window with GestureWorks, we set the background image of our screen. Then we create our 2 touch objects, setting their image to the yellow GestureWorks logo provided with the tutorial code. The important code is at the end of the loop. Here, we are registering each touch object with GestureWorks using a unique string identifier that we keep as the key in our touch_objects dict. Then we add three gestures to each touch object: n-drag, n-rotate, n-scale. For a more detailed explanation of this, please see the GestureML wiki.

7. Initialize GestureWorks And Load GML

Again, we load and initialize GestureWorks the same way as before.

if __name__ == '__main__':
   # Initialize GestureWorksCore with the location of the library
   gw = GestureWorksCore('C:\\path\\to\\GestureWorksCore\\GestureWorksCore32.dll')
   if not gw.loaded_dll:
       print 'Unable to load GestureWorksCore'
       exit()
 
   try:
       # Load a basic GML file
       gw.loadGML('\\gml_path\\basic_manipulation.gml')
   except WindowsError, e:
       print 'Unable to load GML'
       exit()  
 
   gw.initializeGestureWorks(SCREEN_WIDTH, SCREEN_HEIGHT)
   app = MultitouchApp(gw)
   app.run()

Except, this time, we also load GML since we want GestureWorks to process gestures as well as basic touch input. You can either add the basic_manipulation.gml file to your project as an imported resource or use the path to its location in the GestureWorks installation directory directly. Again, please be sure that you are loading the correct GestureWorksCore DLL (GestureWorksCore32.dll or GestureWorksCore64.dll) for your system.

8. Create updateGestureworks callback

Just like last time, we'll need to create a callback for our scheduled timer event to call.

   def updateGestureworks(self, task):
       self.gw.processFrame()
       point_events = gw.consumePointEvents()
       gesture_events = gw.consumeGestureEvents()
       self.processTouchEvents(point_events)
       self.processGestureEvents(gesture_events)
 
       return Task.cont

We can consume gesture events just like we consume point events. The GestureEvent structure is also defined in GWCUtils and we will receive a list of gesture events when we consume them.

9. Process Touch Events

Again, we’ll create a method for processing touch events, translate the coordinates of our touch points and check the status of each event. Only in this tutorial, we need to check if new touch points collide with any of our touch objects. If they do, we want to let GestureWorks know about it so that it can add the touch point to the object and check for gesture events involving the new touch point.

   def processTouchEvents(self, touches):
       for touch in touches:
           # We need to convert Gestureworks coordinates to Panda3D coordinates
           touch_x = float((touch.position.x - SCREEN_WIDTH/2) / SCREEN_WIDTH) * 4
           touch_y = float((SCREEN_HEIGHT/2 - touch.position.y) / SCREEN_HEIGHT) * 2
 
           if touch.status == TOUCHADDED:
               obj = self.hitTest(touch_x, touch_y)
               if obj:
                   self.gw.addTouchPoint(obj.name, touch.point_id)
           elif touch.status == TOUCHREMOVED:
               # Handle touch removed
               pass
           elif touch.status == TOUCHUPDATE:
               # Handle touch updates
               pass

We don’t need to worry about any touch event statuses other than TOUCHADDED, but we’ve included the others here for illustration.

10. Hit Test And Associate Touch Points With Touch Objects

Hit testing is pretty straight forward in this example. We iterate through all of our touch objects, rotate the touch point about the object’s center based on the object’s current rotation and check to see if the point lies within the object’s bounding box. We return a reference to the touch object on the first detection of a collision. Since this is meant to be a tutorial on how to use GestureWorks and not on two dimensional transformations, we provide a function to rotate a point about another arbitrary point (rotateAboutCenter) and leave it to the reader to investigate these concepts further.

   def hitTest(self, x, y):
       for obj in self.touch_objects.values():
           (local_x, local_y) = rotateAboutCenter(x, y, obj.picture.getX(), obj.picture.getZ(), radians(-obj.picture.getR()))
           if (local_x > (obj.picture.getX() - obj.scale) and local_x < (obj.picture.getX() + obj.scale) ):
               if (local_y > (obj.picture.getZ() - obj.scale) and local_y < (obj.picture.getZ() + obj.scale) ):
                   return obj

Remember that we need to account for the difference between Panda3D coordinates and GestureWorks coordinates.

11. Process Gesture Events

Processing gesture events is just as simple as processing touch events. You saw in step 8 that we are now calling consumeGestureEvents in our update loop after we ask GestureWorks to process a frame. Now we need a method to process those gesture events.

   def processGestureEvents(self, gesture_events):
       for e in gesture_events:
           obj = self.touch_objects[e.target]
           {'n-drag': self.handleDrag,
            'n-rotate': self.handleRotate,
            'n-scale': self.handleScale}[e.gesture_id](obj, e)

We just iterate through each event and check its gesture_id property. These IDs will correspond to the gesture names that we registered on each touch object at the end of step 6. The names of the gestures are defined in the GML file that we loaded when we initialized GestureWorks. Here we simply call a handler method with the touch object and gesture event as parameters based on which gesture we received. GestureWorks lets us know which object the gesture applies to via the target field of the gesture event. This name will correspond to the string we used to register the touch object in step 6.

For the most part, relevant gesture event data is stored in the values member of the event object. This member is a map of strings to floats. The string used to address any value will be based on that attribute's name in the GML (please see the GestureML wiki for more information). For our example, these attribute names can be viewed in the basic_manipulations.gml file we loaded previously. It is also important to note that these values are generally expressed as deltas; that is, they represent differentials between the current state and the previous state; eg. drag_dx and drag_dy each represent the change in position between this frame and the last.

12. Overview Of Object Transformations

Drag and scale gestures are pretty straight forward and handled in similar ways. GestureWorks gives us values for most gestures as delta values based on frame boundaries so we can use the values directly as incrementers.

   def handleDrag(self, obj, gesture_event):
       # We need to convert Gestureworks coordinates to Panda3D coordinates
       dx = gesture_event.values['drag_dx'] / SCREEN_WIDTH * 4
       dy = gesture_event.values['drag_dy'] / SCREEN_HEIGHT * 2
       obj.picture.setX(obj.picture.getX() + dx)
       obj.picture.setZ(obj.picture.getZ() - dy)
 
       # Make sure the objects stay on the screen
       obj.picture.setX(max(min(obj.picture.getX(), 1.90), -1.90))
       obj.picture.setZ(max(min(obj.picture.getZ(), 1), -1))

Note that we must compensate for the difference in coordinate systems. The last thing we do is just check the final position of the object to make sure it stays on our screen. Scale is handled in much the same way.

   def handleScale(self, obj, gesture_event):
       dsx = gesture_event.values['scale_dsx']/4
       obj.scale += dsx
       obj.scale = max(min(obj.scale, 1.5), .125)
       obj.picture.setScale(obj.scale)

Rotation is only slightly more complicated because we want to rotate around the center of the cluster of touch points activating the gesture. This gives a more realistic feel than simply rotating the object around its center. Again, the rotateAboutCenter is provided in GWCUtils for this purpose.

   def handleRotate(self, obj, gesture_event):
       theta = gesture_event.values['rotate_dtheta']
       obj.picture.setR(obj.picture.getR() + theta)
       if gesture_event.n:
           # We need to convert Gestureworks coordinates to Panda3D coordinates
           gesture_x = float((gesture_event.x - SCREEN_WIDTH/2) / SCREEN_WIDTH) * 4
           gesture_y = float((SCREEN_HEIGHT/2 - gesture_event.y) / SCREEN_HEIGHT) * 2
           (new_x,new_y) = rotateAboutCenter(obj.picture.getX(), obj.picture.getZ(), gesture_x, gesture_y, radians(-theta))
           obj.picture.setX(new_x)
           obj.picture.setZ(new_y)

We rotate the object around it’s center and then rotate the object around the center of our gesture event. Note that we only do the second rotation if there are actual touch points activating the gesture. When an inertial filter is active, it is possible for the event to be triggered when there are no active touch points. In this case, we only want to rotate the object about its own center. For more information on gesture filters, please see the GestureML wiki. One final important note to make is that GestureWorks gives us rotation values in degrees but our helper function expects radians.


Review

In this tutorial we expanded on the knowledge gained in Python & Panda3D: Getting Started II (Hello Multitouch) by manipulating on-screen objects using gesture data obtained from GestureWorks. This tutorial covers a number of concepts that are central to the usage of GestureWorks:

  • Registering touch objects
  • Adding gestures to the registered touch objects
  • Manipulating Panda3D object instances based on data obtained from GestureWorks

A principal concept that was briefly touched upon in step #11 above is that due to the language- and framework-independent nature of GestureWorks, it is the programmer's responsibility to associate the local implementation of an object with an object as tracked by GestureWorks. To review: GestureWorks doesn't know anything about the TouchContainer class that we defined, but we've registered each object with GestureWorks using the object’s name property as an identifier; we then manipulate our TouchContainers based on the data contained in the matching GestureEvent (the GestureEvent whose Target property matches our TouchContainer's name property).


Continuing Education

It is the intention of this and the other GestureWorks Core tutorials to get the programmer started using the GestureWorks Core framework, and is by no means an exhaustive explanation of all of the features of GestureWorks Core; in fact, we’ve only barely scratched the surface!


For more information on GestureWorks Core and GestureML, please visit the following sites:


Previous tutorial: Python & Panda3D: Getting Started II (Hello Multitouch)

tutorials/legacy/python_panda3d/interactive_bitmaps.txt · Last modified: 2019/01/21 19:24 (external edit)