Multi-Touch Interaction in Spring

[bilhamil]

Hi Everyone,
I posted a while back asking for some help on a technical issue working with multitouch events in spring. Anyway, I've made some progress. I posted a youtube video of some of what I have been working on so far, which is mostly camera control and button pushing (through LuaUI). I'm not a Spring pro player and some of the interaction is not completely fleshed out, so the video is kind of rough. I know that multi-touch is not really something that everyone has (or even a large number of people have) right now, but I feel like there are lot of cool things that could be done with it in this kind of system.
From what I can tell there has already been a lot of work done to create nice interactions in the engine and the mods. So thanks for all of your great work in that regard. So I guess I'm looking on feedback on what you guys think, also if anyone lives in Texas I would appreciate any volunteer user study participants (I'm a human computer interaction PhD student).

[KaiserJ]

man...

that looks so freaking cool. i live far away, and don't have a touchscreen monitor, so i can't help in any way.

but i will be the first of many to give kudos. controls look good; i would love to try playing like this

[Pxtl]

That's sexy as hell. Watched it with the sound off...

So, to get it - the left hand touches buttons and does all the buttons and the shift-command, and right hand clicks... I like the big shift key.

I could see toggling between the unit-build-lists and the orders panel being annoying - Zero-K gets away with that because the orders panel is usually circumvented by the hotkeys.

Does right hand have a way to do left/right clicks? I can't see if there are buttons on your stylus.

[KaiserJ]

I like the big shift key

i didnt even see this the first time through, now the mechanics make a lot more sense

[bilhamil]

Yeah modifiers are kind of a necessity. It seemed like for the time being that shift was the most essential for doing queuing. Eventually I think I want to move away from the keyboard semantics, could think of using number of fingers in some space as a specific modifier, but for now it's cheapest to emulate keyboard modifiers.

There are barrel buttons on the pen, but they are really uncomfortable to activate, which makes right clicking a real pain. Will probably want to do more gesture based interactions in the long run I think. ZeroK's marking menu is an interesting start, but would want to incorporate other orders besides building most likely.

[Pxtl]

Oh? I've used the stylus-buttons on the Wacom Bamboo and found they're okay as long as you're not looking for precision on *where* the button was clicked. So popping out the rclick context menu was never really bad.

Having the stylus-button function as "cancel" for issuing orders would be good. Having it function as rclick for using default commands would be bad.

Really, that's going to be your biggest challeng - the fact that in every RTS, the mouse functions for both
a) Selecting units
b) Issue default command

And both of those modes need to be accessed *really* fast.

"number of fingers in a specific space as modifiers"

Oh, like 1-finger-shift is shift, 2-finger shift is ctrl, 4-finger shift is alt, and then do binary combinations? So for ctrl-shift-alt you need to put 7 fingers on the shift block.

This is why I'm not an interaction designer.

[Echo419]

itd be considerably easier if you also worked up some kind of full UI for this.

qeueing multiple units would be dragging the icon from one location to another, just drag a number at a time. (more fingers on screen = more units?)

[marciolino]

Awesome work! Cant wait to see more demos

[knorke]

that was interessting, thanks for sharing

[Tobi]

Wow, very cool!

[bilhamil]

Oh? I've used the stylus-buttons on the Wacom Bamboo and found they're okay as long as you're not looking for precision on *where* the button was clicked. So popping out the rclick context menu was never really bad.

Messed around with the stylus some more, I think you're right about the buttons, hard to precisely click, but the larger of the two buttons is definitely easy to activate.

Thinking maybe use one of those for shift or context menu. If I do end up sort of doing a gesture based pen system (i.e. draw an x for attack, or a chevron for attack move, circle for guard, etc) then it might be good to have unit deselect on hand since it'll be hard to deselect by clicking on empty space then.

[MidKnight]

Man, that's sweet. Seeing ZK look like that makes me feel warm and fuzzy inside.

It must be super-cool to control Spring via a touch interface.

[luckywaldo7]

That looks ridiculously awesome. Kudos for all your work!

[bobthedinosaur]

any one remember that voice activated command RTS game: end war?
yeah me neither...

[luckywaldo7]

http://www.youtube.com/watch?v=cq2Ng14dV6E

Nicely done!

[smoth]

very cool

[abma]

nice! just one question: why did you use the 0.82 branch for developing?

[Gota]

First of all,awesome setup you got there.

now for Suggestion:

It all seemed very cool but also very clumsy and unfit for playing a fast, intense game.

#1 IMO you need first of all to disallow any sort of perspective changing cause its not really needed for gameplay and you dont want people to do it by accident.

#2 I think the mini map needs to be on the bottom left and a bit bigger so you can easily press on it with your finger and have the screen center on that point on the map.
so you got one finger touch instead of multiple slow hand waving to move there(finger dragging on the minimap should also be allowed)...

Some Problems:

how to execute fast zooming in and out.
Seems impossible to have free fast zoom in and out that is as stable as the mouse wheel.
how to simulate shift and control buttons?
This includes deselecting units from a group,adding units to a group...

Raw ideas:

*single units gets selected by one finger touch and deselected by one finger touch.
When u got several units selected and you finger one of those selected units it deselects but the rest stay selected.

*Same with adding selections;if you got units selected fingering more units adds them to the selection and does not deselect anything.
To deselect double tap finger on empty space.

*zooming in and out.
max zoom out needs to be restricted(based on map size).
you should have probably 3 fixed zoom levels.
2 based on map size and a third constant.
One zoom is a steady max zoom out(probably smaller than entire map since its rare that you really need to see the entire map..)
one zoom is middle zoom out based on map size but not in a linear progression so it can be usefull even on the very big maps(the bigger the map the smaller the zoom out bonus).
Third zoom is a static zoom distance that allows easy finger micro management (based on tests).
the zooming in and out of these 3 levels will be done using the same finger movement you use now only having 3 zoom levels will stabilize it , not requiring godly finesse with finger movement since it is harder to do as accurately as controlling the mousewheel)

*selection zooming.
When you have more than one unit selected a button should appear and when you press it the screen zooms in on these selected units, this is for easier management of selected units and deselecting from the group fast.
Pressing this button again zooms you out, back to you former zoom level.

[Saktoth]

Amazing, great to see Zero-K being used in this way. This is what open source is all about, kudos.

There are a lot of things you can do to the interface to break away from traditional RTS design. The first and obvious one is that you do not actually need two mouse buttons. You cannot select enemy units, and the context command for allied units is 'guard', which is only useful for constructor assisting, which can be implemented in another way. This means selection and orders can be done using only one button. Box selection and 'Deselect Everything' are then the only sticking points. Box select can probably be solved by having a lasso select as in the second video (and starting on a unit) and deselect isnt really required for anything other than ensuring you dont give accidental commands- this can be fixed by just selecting a unit like a wind or mex which doesnt accept commands anyway.

Two buttons are not required for construction either, finishing a build queue can be done by depressing shift, or possibly by giving a 'null' radial menu order.

Anyway, good work, really cool stuff.

[jseah]

Here's another suggestion:
When making a build queue, you can allow the dragging of the unit/building icon from the list to the queue.

Eg. in the video, when you had the gunship fac selected, you could just touch-drag the icon for a brawler into the queue at the bottom and have the fac build it.
Similarly, for buildings, drag the building icon off the panel and it turns into a building plan, press the button for lclick and drag to draw a line of buildings, hold shift to queue, etc.

With some practice and UI streamlining, this could become faster than the standard mouse+keyboard controls.