Table of Contents
Interactive Bitmaps
Introduction
In this tutorial, we further explore GestureWorks Core and the Java bindings by consuming gesture events obtained from GestureWorks Core and manipulating on-screen images represented as Graphics2D objects within Java. This tutorial also introduces Graphics2D manipulation such as dragging, rotation and scaling, values for which you will obtain from GestureWorks. For this tutorial you will need the Gestureworks Core multitouch framework; a free trial is available.
In this application, the user is able to drag the objects (the blue GestureWorks logos) around the screen, scale them using the familiar “pinch”-style gesture, and rotate them using the n-finger rotate gesture.
Download the code for all of the Java & Java 2D multitouch tutorials here: tutorials_java_java2d.zip
Requirements
Estimated time to complete: 45 minutes
This tutorial assumes that you have completed the steps found in Java: Getting Started I (Hello World) and Java: Getting Started II (Hello Multitouch). You should have a fully configured NetBeans project ready for use and should be familiar with consuming GestureWorks point event data. If you have have not yet done so, please follow both Java: Getting Started I (Hello World) to prepare a NetBeans project and Java: Getting Started II (Hello Multitouch) to become familiar with consuming GestureWorks touch event data.
In addition to the above, this tutorial expects the student to have a basic understanding of the Java language, the Java2D framework, and is familiar with object-oriented programming concepts.
Process Overview
Process Detail
1. Create a new Java project with required dependencies
For this tutorial it is recommended that you create a new NetBeans project with the required dependencies using the procedure outlined in tutorial #1. In addition to the dependencies added in the first tutorial, the resulting project from tutorial #2 should be included as a reference. This will allow us to reuse the GestureWorks initialization and frame processing from the TouchWindow class.
- Create a new Java Application project called GestureWorksTutorial3
- Add the following dependencies:
- gwc_java project
- The project from tutorial#2
- jna jar
- resources folder
2. Create touch object
We need to create a TouchObject component to load the image and to handle hit testing (see section 6 for hit testing details) and object transformations. Similar to the TouchPoint graphics from Tutorial#2, this object will extend the java.awt.Component class. The constructor for this object will require an image source path and a name to register the object with GestureWorks.
private BufferedImage bi;
public TouchObject(String imagePath, String name)
{
this.name = name;
try{
URL imageSrc = ((new File(imagePath))).toURI().toURL();
bi = ImageIO.read(imageSrc);
width = bi.getWidth();
height = bi.getHeight();
setSize((int)width, (int)height);
}catch (IOException e){
System.err.println("Image could not be read");
System.exit(1);
}
}
The paint method manages all transformations through an AffineTransform object. The idea is to declare global transformation properties then call this method every time one of these properties is modified.
@Override
public void paint(Graphics g) {
super.paint(g);
Graphics2D g2d = (Graphics2D)g;
g2d.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
drawX = GWCUtils.getDrawingCoordX(width, height, x, y, (float) Math.toRadians(rotation), scale);
drawY = GWCUtils.getDrawingCoordY(width, height, x, y, (float) Math.toRadians(rotation), scale);
AffineTransform transform = new AffineTransform();
transform.translate(drawX, drawY);
transform.rotate(Math.toRadians(rotation));
transform.scale(scale, scale);
g2d.drawImage(bi, transform, null);
}
3. Create gesture window
Now we need to create a new TouchWindow object to display the bitmaps and to translate gesture events into interactions.
Create a new Java class called GestureWindow that extends TouchWindow
public class GestureWindow extends TouchWindow{
public GestureWindow(String name)
{
super(name);
}
@Override
public void processPointEvents(ArrayList<PointEvent> events) {}
@Override
public void processGestureEvents(ArrayList<GestureEvent> events) {}
}
Inside the constructor, generate the following mapping to assign handler methods to the gesture events. This map will be used to invoke the correct handler when processing specific events.
gestureEventHandlers = new LinkedHashMap<>();
gestureEventHandlers.put("n-drag", getClass().getMethod("handleDrag", GestureEvent.class));
gestureEventHandlers.put("n-rotate", getClass().getMethod("handleRotate", GestureEvent.class));
gestureEventHandlers.put("n-scale", getClass().getMethod("handleScale", GestureEvent.class));
Now we’ll add our object registration method which instantiates the TouchObject, adds it to the window, registers the object and its listeners with GestureWorks.
public void addTouchObject(String path, String name)
{
TouchObject bm = new TouchObject(path, name);
touchObjects.put(name, bm);
add(bm);
validate();
gwCore.registerTouchObject(name);
for(String type: gestureEventHandlers.keySet()) {
gwCore.addGesture(name, type);
}
}
4. Process Touch Events
Again, we’ll create a method for processing touch events and check the status of each event. Only in this tutorial, we need to check if new touch points collide with any of our touch objects. If they do, we want to let GestureWorks know about it so that it can add the touch point to the object and check for gesture events involving the new touch point.
@Override
public void processPointEvents(ArrayList<PointEvent> events)
{
for(PointEvent event: events)
{
switch(event.status)
{
case Status.TOUCHADDED:
for(String name: touchObjects.keySet())
{
if(touchObjects.get(name).isHit(event.position.x, event.position.y))
gwCore.addTouchPoint(name, event.point_id);
}
break;
case Status.TOUCHUPDATE:
break;
case Status.TOUCHREMOVED:
break;
default:
break;
}
}
}
We don’t need to worry about any touch event statuses other than TOUCHADDED, but we’ve included the others here for illustration.
5. Hit Test And Associate Touch Points With Touch Objects
Hit testing is pretty straight forward in this example. We iterate through all of our touch objects, rotate the touch point about the object’s center based on the object’s current rotation and check to see if the point lies within the object’s bounding box. We return a reference to the touch object on the first detection of a collision. Since this is meant to be a tutorial on how to use GestureWorks and not on two dimensional transformations, we provide a function to rotate a point about another arbitrary point (rotateAboutCenter) and leave it to the reader to investigate these concepts further.
public boolean isHit(float x, float y)
{
float localX = GWCUtils.rotateAboutCenterX(x, y, this.x, this.y, -(float) Math.toRadians(rotation));
float localY = GWCUtils.rotateAboutCenterY(x, y, this.x, this.y, -(float) Math.toRadians(rotation));
boolean inWidth = Math.abs(localX - this.x) <= (width*scale)/2;
boolean inHeight = Math.abs(localY - this.y) <= (height*scale)/2;
return inWidth && inHeight;
}
6. Process Gesture Events
Processing gesture events is just as simple as processing touch events. Now that we are assigning touch points to touch objects, the processGestureEvents method can be implemented.
@Override
public void processGestureEvents(ArrayList<GestureEvent> events)
{
for(GestureEvent event: events)
{
try{
gestureEventHandlers.get(event.gesture_id).invoke(this, event);
}catch(IllegalAccessException | IllegalArgumentException |
InvocationTargetException e){
e.printStackTrace();
}
}
}
We just iterate through each event and check its gesture_id property. These IDs will correspond to the gesture names that we registered on each touch object at the end of step 4. The names of the gestures are defined in the GML file that we loaded when we initialized GestureWorks. Here we simply call a handler method with the touch object and gesture event as parameters based on which gesture we received.
GestureWorks lets us know which object the gesture applies to via the target field of the gesture event. This name will correspond to the string we used to register the touch object in step 3.
- For the most part, relevant gesture event data is stored in the values member of the event object. This member is a map of strings to floats. The string used to address any value will be based on that attribute’s name in the GML (see GestureML.org for more information). For our example, these attribute names can be viewed in the basic_manipulations.gml file we loaded previously.
- It is also important to note that these values are generally expressed as deltas; that is, they represent differentials between the current state and the previous state; eg. drag_dx and drag_dy each represent the change in position between this frame and the last.
7. Overview Of Object Transformations
Drag and scale gestures are pretty straight forward and handled in similar ways. GestureWorks gives us values for most gestures as delta values based on frame boundaries so we can use the values directly as incrementers.
//GestureWindow code
public void handleDrag(GestureEvent event)
{
TouchObject to = touchObjects.get(event.target);
to.addX(event.values.getValue("drag_dx"));
to.addY(event.values.getValue("drag_dy"));
}
//TouchObject code
public void addX(float x)
{
this.x += x;
repaint();
}
public void addY(float y)
{
this.y += y;
repaint();
}
Scale is handled in much the same way.
//GestureWindow code
public void handleScale(GestureEvent event)
{
TouchObject to = touchObjects.get(event.target);
to.addScale(event.values.getValue("scale_dsx"));
}
//TouchObject code
public void addScale(float scale)
{
this.scale += scale;
repaint();
}
Rotation is only slightly more complicated because we want to rotate around the center of the cluster of touch points activating the gesture. This gives a more realistic feel than simply rotating the object around its center. Notice the rotateX and rotateY attributes represent the center point of the cluster. See paint method from step 3 for rotation transformation.
//GestureWindow code
public void handleRotate(GestureEvent event)
{
TouchObject to = touchObjects.get(event.target);
to.addRotation(event.values.getValue("rotate_dtheta"));
to.setRotateX(event.x);
to.setRotateY(event.y);
}
//TouchObject code
public void addRotation(float rotation)
{
this.rotation +=rotation;
repaint();
}
public void setRotateX(float x)
{
rotateX = x;
}
public void setRotateY(float y)
{
rotateY = y;
}
We rotate the object around it’s center and then rotate the object around the center of our gesture event. Note that we only do the second rotation if there are actual touch points activating the gesture. When an inertial filter is active, it is possible for the event to be triggered when there are no active touch points. In this case, we only want to rotate the object about its own center. For more information on gesture filters, please see the GML overview. One final important note to make is that GestureWorks gives us rotation values in degrees but our helper function expects radians.
8. Compile and run
To test your GestureWindow and TouchObjects, add the following code to the project’s main class and hit F6 to run the application.
GestureWindow window = new GestureWindow("Interactive Bitmaps");
window.addTouchObject("resources/Processing.png", "java_img");
window.addTouchObject("resources/Processing.png", "java_img2");
The application should display a window titled “Interactive Bitmaps”. The TouchObject(s) can be dragged, rotated, and scaled.
Review
In this tutorial we expanded on the knowledge gained in Tutorial #2 by manipulating on-screen objects using gesture data obtained from GestureWorks. This tutorial covers a number of concepts that are central to the usage of GestureWorks:
- Registering touch objects
- Adding gestures to the registered touch objects
- Manipulating Java’s Graphic2D object instances based on data obtained from GestureWorks
A principal concept is that due to the language- and framework-independent nature of GestureWorks, it is the programmer’s responsibility to associate the local implementation of an object with an object as tracked by GestureWorks. To review: GestureWorks doesn’t know anything about the TouchObject class that we defined, but we’ve registered each object with GestureWorks using the object’s name property as an identifier; we then manipulate our TouchObject based on the data contained in the matching GestureEvent (the GestureEvent whose target property matches our TouchObject’s name property).
Continuing Education
It is the intention of this and the other Gestureworks Core Tutorials to get the programmer started using the GestureWorks core framework, and is by no means an exhaustive explanation of all of the features of Gestureworks Core; in fact, we’ve only barely scratched the surface!
For more information on GestureWorks Core, GestureML, and CreativeML, please visit the following sites:
Previous tutorial: Java: Getting Started II (Hello Multitouch)