[[tutorials:legacy:python_kivy|{{ :kivy-icon-60.png }}]]

This binding has been deprecated - please see the most recent [[:release_notes| release notes]] for more information.

====== Interactive Bitmaps ====== ===== Introduction ===== In this tutorial, we further explore Gestureworks Core and the Python bindings by consuming gesture events obtained from Gestureworks Core and manipulating on-screen images represented as Scatter objects within Kivy. This tutorial also introduces Scatter manipulation in Kivy such as rotation and scaling, values for which you will obtain from Gestureworks. For this tutorial you will need the [[http://gestureworks.com/|Gestureworks Core]] multitouch framework; a [[http://files.gestureworks.com/downloads/Core/Trial/GestureworksCoreTrialSetup.exe|free trial]] is available. In this application, the user is able to drag the objects (the yellow Gestureworks logos) around the screen, scale them using the familiar "pinch"-style gesture, and rotate them using the n-finger rotate gesture. Download the code for all of the Python & Kivy multitouch tutorials here: [[http://files.gestureworks.com/tutorials/tutorials_python_kivy.zip|tutorials_python_kivy.zip]] {{:tutorials:legacy:python_kivy:python_3.1_386x281.png?nolink|Python 3.1 386x281.png}} ---- ===== Requirements ===== Estimated time to complete: **45 minutes** This tutorial assumes that you have completed the steps found in [[tutorials:legacy:python_kivy:getting_started_1_hello_world|Python & Kivy: Getting Started I (Hello World)]] and [[tutorials:legacy:python_kivy:getting_started_2_hello_multitouch|Python & Kivy: Getting Started II (Hello Multitouch)]]. You should have a fully configured Eclipse project ready for use and should be familiar with consuming Gestureworks point event data. If you have have not yet done so, please follow both Tutorial #1 to prepare an Eclipse project and Tutorial #2 to become familiar with consuming Gestureworks touch event data. In addition to the above, this tutorial expects the student to have a basic understanding of the Python language, the Kivy framework, and is familiar with object-oriented programming concepts. ---- ===== Process Overview ===== - [[tutorials:legacy:python_kivy:interactive_bitmaps#create_a_new_eclipse_-_kivy_project|Create a new Eclipse - Kivy project]] - [[tutorials:legacy:python_kivy:interactive_bitmaps#add_the_gestureworks_logo_images_to_the_project|Add the GestureWorks logo images to the project]] - [[tutorials:legacy:python_kivy:interactive_bitmaps#create_a_kivy_language_file|Create a Kivy Language file]] - [[tutorials:legacy:python_kivy:interactive_bitmaps#import_required_modules|Import required modules]] - [[tutorials:legacy:python_kivy:interactive_bitmaps#create_touchcontainer_class|Create TouchContainer class]] - [[tutorials:legacy:python_kivy:interactive_bitmaps#create_kivy_app|Create Kivy app]] - [[tutorials:legacy:python_kivy:interactive_bitmaps#initialize_gestureworks_and_load_gml|Initialize GestureWorks and load GML]] - [[tutorials:legacy:python_kivy:interactive_bitmaps#create_updategestureworks_callback|Create updateGestureWorks callback]] - [[tutorials:legacy:python_kivy:interactive_bitmaps#process_touch_events|Process touch events]] - [[tutorials:legacy:python_kivy:interactive_bitmaps#hit_test_and_associate_touch_points_with_touch_objects|Hit test and associate touch points with touch objects]] - [[tutorials:legacy:python_kivy:interactive_bitmaps#process_gesture_events|Process gesture events]] - [[tutorials:legacy:python_kivy:interactive_bitmaps#overview_of_object_transformations|Overview of object transformations]] ---- ===== Process Details ===== ==== 1. Create A New Eclipse - Kivy Project ==== For this tutorial it is recommended that you create a new, empty Kivy project with the gwc_python module using the procedure outlined in [[tutorials:legacy:python_kivy:getting_started_1_hello_world|Python & Kivy: Getting Started I (Hello World)]]. You may also use the project created in [[tutorials:legacy:python_kivy:getting_started_2_hello_multitouch|Python & Kivy: Getting Started II (Hello Multitouch)]] but you will need to remove or comment out some unnecessary code. ==== 2. Add the Gestureworks Logo Images to the Project ==== For this application, two identical images will be used as the textures for the Scatter objects that will appear on-screen. Using the method described in tutorial #2 for adding resources to the project, add the following folder and all contents to the project: \GestureWorksCore\bindings\python\kivy\media ==== 3. Create a Kivy Language File ==== Our Kivy file from the previous tutorial just needs a few minor additions for this application. We’ll also call this file “multitouch.kv”. #:kivy 1.5.0 #:import kivy kivy #:import win kivy.core.window FloatLayout: canvas: Color: rgb: 1, 1, 1 Rectangle: source: 'media/Logo_gestureworks_core_1920x1080.png' size: self.size on_size: self.center = win.Window.center size_hint: None, None ==== 4. Import Required Modules ==== Again, we'll create a new file called “main.py” and import the following: from gwc_python.core import GestureWorksCore from gwc_python.GWCUtils import TOUCHADDED, TOUCHREMOVED, TOUCHUPDATE, rotateAboutCenter from kivy.app import App from kivy.clock import Clock from kivy.uix.scatter import Scatter from kivy.uix.image import Image from kivy.factory import Factory from math import radians ==== 5. Create TouchContainer Class ==== We’ll need to create a simple class to hold our touch objects. This will inherit from the Kivy Scatter class and simply override the built in touch event handler. class TouchObject(Scatter): def on_touch_move(self, touch): return Factory.register('TouchObject', TouchObject) ==== 6. Create Kivy App ==== Just as before, we’ll need a class that inherits the Kivy App class for our application to run. We’ll start with the constructor and build method. class MultitouchApp(App): def __init__(self, gw): self.gw = gw self.touch_points = {} self.touch_objects = {} self.max_objects = 2 self.has_received_gesture = False # schedule our processing cycle Clock.schedule_interval(self.updateGestureWorks, 1./60) super(MultitouchApp, self).__init__() This is very similar to the previous tutorial, we just have a new dict for holding touch objects, a max_objects variable and a state variable has_received_gesture that we’ll use for aesthetic purposes later on. The build method will require more explanation this time. def build(self): # Register our app window with GestureWorks if not self.gw.registerWindow('Kivy'): print('Unable to register touch window') exit() for i in range(0, self.max_objects): container = TouchContainer() container.name = 'object_{}'.format(i) print container.name container.add_widget(Image(source='media/logo.png')) container.scale = 2 self.root.add_widget(container) self.touch_objects.update({container.name: container}) # Tell Gestureworks about our touch object and add gestures to it self.gw.registerTouchObject(container.name) self.gw.addGesture(container.name, 'n-drag') self.gw.addGesture(container.name, 'n-rotate') self.gw.addGesture(container.name, 'n-scale') After we register our touch window with GestureWorks, we create each of our touch objects setting their image to the yellow GestureWorks logo provided with the tutorial code. The important code is at the end of the loop. Here, we are registering each touch object with GestureWorks using a unique string identifier that we keep as the key in our touch_objects dict. Then we add three gestures to each touch object: n-drag, n-rotate, n-scale. For a more detailed explanation of this, please see the [[http://gestureml.org/|GestureML wiki]]. ==== 7. Initialize Gestureworks And Load GML ==== Again, we load and initialize Gestureworks the same way as before. if __name__ == '__main__': # Initialize GestureWorksCore with the location of the library gw = GestureWorksCore('C:\\path\\to\\GestureworksCore\\GestureworksCore32.dll') if not gw.loaded_dll: print 'Unable to load GestureworksCore' exit() try: # Load a basic GML file gw.loadGML('\\gml_path\\basic_manipulation.gml') except WindowsError, e: print 'Unable to load GML' exit() gw.initializeGestureWorks(1920, 1080) app = MultitouchApp(gw) app.run() Except, this time, we also load GML since we want GestureWorks to process gestures as well as basic touch input. You can either add the basic_manipulation GML file to your project as an imported resource or use the path to it’s location in the GestureWorks installation directory directly. Again, please be sure that you are loading the correct DLL for your system. ==== 8. Create updateGestureWorks callback ==== Just like last time, we’ll need to create a callback for our scheduled timer event to call. def updateGestureWorks(self, *args): self.gw.processFrame() point_events = gw.consumePointEvents() gesture_events = gw.consumeGestureEvents() if len(gesture_events) != 0: self.has_received_gesture = True; self.processTouchEvents(point_events) self.processGestureEvents(gesture_events) if not self.has_received_gesture: buf = self.root.width / (len(self.touch_objects) + 1) for obj in self.touch_objects.values(): obj.center = (buf, self.root.height / 2) buf += buf We can consume gesture events just like we consume point events. The GestureEvent structure is also defined in GWCUtils and we will receive a list of gesture events when we consume them. The final bit of code is only run before any gesture events have been processed and is only there to make sure that we draw our two logos in the same place. ==== 9. Process Touch Events ==== Again, we’ll create a method for processing touch events, translate the y-coordinates of our touch points and check the status of each event. Only in this tutorial, we need to check if new touch points collide with any of our touch objects. If they do, we want to let GestureWorks know about it so that it can add the touch point to the object and check for gesture events involving the new touch point. def processTouchEvents(self, touches): for touch in touches: # We need to convert kivy y coordinate to Gestureworks y coordinate touch_y = self.root.height - touch.position.y touch_x = touch.position.x if touch.status == TOUCHADDED: obj = self.hitTest(touch_x, touch_y) if obj: self.gw.addTouchPoint(obj.name, touch.point_id) elif touch.status == TOUCHREMOVED: # Handle touch removed pass elif touch.status == TOUCHUPDATE: # Handle touch updates pass We don’t need to worry about any touch event statuses other than TOUCHADDED, but we’ve included the others here for illustration. ==== 10. Hit Test And Associate Touch Points With Touch Objects ==== Hit testing is pretty straight forward in this example. We iterate through all of our touch objects, rotate the touch point about the object’s center based on the object’s current rotation and check to see if the point lies within the object’s bounding box. We return a reference to the touch object on the first detection of a collision. Since this is meant to be a tutorial on how to use Gestureworks and not on two dimensional transformations, we provide a function to rotate a point about another arbitrary point (rotateAboutCenter) and leave it to the reader to investigate these concepts further. def hitTest(self, x, y): for obj in self.touch_objects.values(): (local_x, local_y) = rotateAboutCenter(x, y, obj.center_x, obj.center_y, radians(-obj.rotation)) if (abs(local_x - obj.center_x) <= obj.width*obj.scale/2) and (abs(local_y - obj.center_y) <= obj.height*obj.scale/2): return obj Remember that we need to account for the difference between Kivy coordinates and GestureWorks coordinates. ==== 11. Process Gesture Events ==== Processing gesture events is just as simple as processing touch events. You saw in step 8 that we are now calling consumeGestureEvents in our update loop after we ask GestureWorks to process a frame. Now we need a method to process those gesture events. def processGestureEvents(self, gesture_events): for e in gesture_events: obj = self.touch_objects[e.target] {'n-drag': self.handleDrag, 'n-rotate': self.handleRotate, 'n-scale': self.handleScale}[e.gesture_id](obj, e) We just iterate through each event and check its gesture_id property. These IDs will correspond to the gesture names that we registered on each touch object at the end of step 6. The names of the gestures are defined in the GML file that we loaded when we initialized GestureWorks. Here we simply call a handler method with the touch object and gesture event as parameters based on which gesture we received. Gestureworks lets us know which object the gesture applies to via the target field of the gesture event. This name will correspond to the string we used to register the touch object in step 6. * For the most part, relevant gesture event data is stored in the values member of the event object. This member is a map of strings to floats. The string used to address any value will be based on that attribute’s name in the GML (please see the [[http://gestureml.org/|GestureML wiki]] for more information). For our example, these attribute names can be viewed in the basic_manipulations.gml file we loaded previously. * It is also important to note that these values are generally expressed as deltas; that is, they represent differentials between the current state and the previous state; eg. drag_dx and drag_dy each represent the change in position between this frame and the last. ==== 12. Overview Of Object Transformations ==== Drag and scale gestures are pretty straight forward and handled in similar ways. GestureWorks gives us values for most gestures as delta values based on frame boundaries so we can use the values directly as incrementers. def handleDrag(self, obj, gesture_event): obj.center_x += gesture_event.values['drag_dx'] obj.center_y -= gesture_event.values['drag_dy'] # Make sure the objects stay on the screen obj.center_x = max(min(obj.center_x, self.root.width), 0) obj.center_y = max(min(obj.center_y, self.root.height), 0) Note that we are subtracting the “drag_dy” value of the gesture to compensate for the difference in coordinate systems. The last thing we do is just check the final position of the object to make sure it stays on our screen. Scale is handled in much the same way. def handleScale(self, obj, gesture_event): dsx = gesture_event.values['scale_dsx'] obj.scale += dsx obj.scale = max(min(obj.scale, 6), .5) Rotation is only slightly more complicated because we want to rotate around the center of the cluster of touch points activating the gesture. This gives a more realistic feel than simply rotating the object around its center. Again, the rotateAboutCenter is provided in GWCUtils for this purpose. def handleRotate(self, obj, gesture_event): theta = gesture_event.values['rotate_dtheta'] obj.rotation -= theta if gesture_event.n: obj.center = rotateAboutCenter(obj.center_x, obj.center_y, gesture_event.x, self.root.height - gesture_event.y, radians(-theta)) We rotate the object around it’s center and then rotate the object around the center of our gesture event. Note that we only do the second rotation if there are actual touch points activating the gesture. When an inertial filter is active, it is possible for the event to be triggered when there are no active touch points. In this case, we only want to rotate the object about its own center. For more information on gesture filters, please see the [[http://gestureml.org/|GestureML wiki]]. One final important note to make is that GestureWorks gives us rotation values in degrees but our helper function expects radians. ---- ===== Review ===== In this tutorial we expanded on the knowledge gained in Tutorial #2 by manipulating on-screen objects using gesture data obtained from Gestureworks. This tutorial covers a number of concepts that are central to the usage of Gestureworks: * Registering touch objects * Adding gestures to the registered touch objects * Manipulating Kivy object instances based on data obtained from Gestureworks A principal concept that was briefly touched upon in [[tutorials:legacy:python_kivy:interactive_bitmaps#process_gesture_events|step #11]] above is that due to the language- and framework-independent nature of Gestureworks, it is the programmer’s responsibility to associate the local implementation of an object with an object as tracked by Gestureworks. To review: Gestureworks doesn’t know anything about the TouchContainer class that we defined, but we’ve registered each object with Gestureworks using the object’s name property as an identifier; we then manipulate our TouchContainers based on the data contained in the matching GestureEvent (the GestureEvent whose Target property matches our TouchContainer’s name property). ===== Continuing Education ===== It is the intention of this and the other [[:tutorials|Tutorials]] and [[:tutorials:legacy|Legacy Tutorials]] to get the programmer started using the GestureWorks core framework, and is by no means an exhaustive explanation of all of the features of GestureWorks Core; in fact, we’ve only barely scratched the surface! For more information on GestureWorks Core, GestureML, and CreativeML, please visit the following sites: * [[http://gestureworks.com|GestureWorks]] * [[http://gestureml.org|GestureML.org]] * [[http://creativeml.org|CreativeML.org]] ---- Previous tutorial: [[tutorials:legacy:python_kivy:getting_started_2_hello_multitouch|Python: Getting Started II (Hello Multitouch)]]