Plugin API: asynchronous code through alternate event loop

Plugin API: asynchronous code through alternate event loop

SpringRTS Perl Autohost for Dedicated Server

Moderators: Moderators, Lobby Developers, SPADS AutoHost

Post Reply
User avatar
Nemo
Spring 1944 Developer
Posts: 1376
Joined: 30 Jan 2005, 19:44

Plugin API: asynchronous code through alternate event loop

Post by Nemo »

still hacking SPADS plugins, still having a blast. just now I made something like this (except more awesome and related to S44, rather than web scraping):

Code: Select all

package PluginWithAsyncWebRequest;
use Mojo::IOLoop;
use Mojo::UserAgent;

# <normal callbacks that SPADS plugins require>

my $ua = Mojo::UserAgent->new;

sub eventLoop {
  Mojo::IOLoop->one_tick;
}

sub new {
  my $class = shift;

  addSpadsCommandHandler(spring_news => sub {
    my $source = shift;
    my $user = shift;
    $ua->get("springrts.com" => sub {
      my ($ua, $tx) = @_;
      my $headline = $tx->res->dom('div.newsheadline a')->first->text;
      sayPrivate($user, "the current headline is: $headline");
    });
  }});

  bless({}, $class);
}

Mojo::IOLoop->start unless Mojo::IOLoop->is_running;
1;
so, yeah. async web requests by hitching a ride on the SPADS event loop. weeee!

because Mojolicious is the bomb, I could even serve web routes from within a spads plugin, but that's probably better left to an external service.
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: SPADS AutoHost

Post by bibim »

Nemo wrote:

Code: Select all

sub eventLoop {
  Mojo::IOLoop->one_tick;
}
I'm not a Mojolicious expert, but it seems one_tick will block (running Mojo's event loop) until a Mojo event occurs?
As SPADS eventLoop() callback shouldn't block (it would prevent SPADS' own event loop to work correctly), wouldn't it be better to add a very short Mojo timer to prevent blocking for too long, as explained in Mojo::IOLoop::one_tick doc?
Nemo wrote:

Code: Select all

Mojo::IOLoop->start unless Mojo::IOLoop->is_running;
1;
Doesn't this start the never-returning Mojo event loop (blocking SPADS' one)?
User avatar
Nemo
Spring 1944 Developer
Posts: 1376
Joined: 30 Jan 2005, 19:44

Re: SPADS AutoHost

Post by Nemo »

bibim wrote:
Nemo wrote:

Code: Select all

sub eventLoop {
  Mojo::IOLoop->one_tick;
}
I'm not a Mojolicious expert, but it seems one_tick will block (running Mojo's event loop) until a Mojo event occurs?
As SPADS eventLoop() callback shouldn't block (it would prevent SPADS' own event loop to work correctly), wouldn't it be better to add a very short Mojo timer to prevent blocking for too long, as explained in Mojo::IOLoop::one_tick doc?
Nemo wrote:

Code: Select all

Mojo::IOLoop->start unless Mojo::IOLoop->is_running;
1;
Doesn't this start the never-returning Mojo event loop (blocking SPADS' one)?
Yes, you are quite correct. Apparently it worked due to a specific behavior of EV: it stops the event loop when no events are being watched. Since I hadn't registered any routes to serve (which would need the event loop to always run), there were no events registered yet when that line ran, and it shut down. The above code also works with that line removed.

I checked repeatedly to make sure that SPADS wasn't getting blocked while long-running async web requests were going (ie, the backend that I called was just sleep-ing for 10 seconds, but SPADS was still responsive). I'll have to dig more to understand why.

What do you think the Right Way (tm) would be to handle this sort of thing? Telling both SPADS and Mojo to use a parent event loop, like EV?
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: SPADS AutoHost

Post by bibim »

Nemo wrote:What do you think the Right Way (tm) would be to handle this sort of thing? Telling both SPADS and Mojo to use a parent event loop, like EV?
Personally I really like the AnyEvent approach, which makes it possible to write event-driven code without subscribing to a specific event system (but unfortunately there is some controversy about the author of this module...).

So if SPADS was using AnyEvent instead of implementing its own event loop, it would already be directly compatible with Mojo::IOLoop.

But I wanted SPADS to require as few dependencies as possible, which means using only CORE Perl modules. That's why I implemented my own (very simplistic) event loop for SPADS.

From there in our specific case I see 2 solutions: either it is possible to make Mojo::IOLoop work just by calling a non-blocking function regularly at each iteration of SPADS' event loop, i.e. in the eventLoop() callback (for exemple one_tick + very short timer as I said previously), either SPADS event loop needs a rewrite so that it automatically use AnyEvent if available.
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: SPADS AutoHost

Post by bibim »

Nemo wrote:I checked repeatedly to make sure that SPADS wasn't getting blocked while long-running async web requests were going (ie, the backend that I called was just sleep-ing for 10 seconds, but SPADS was still responsive). I'll have to dig more to understand why.
Did you find anything regarding this?

Btw I just added Socket management functions to the plugin API, so that SPADS plugins can now create and use any sockets asynchronously through SPADS event loop.
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: Plugin API: asynchronous code through alternate event loop

Post by bibim »

FYI, I'm in the process of integrating AnyEvent compatibility into SPADS core.

So basically, if you have the AnyEvent and EV modules installed, in next SPADS major version you should be able to integrate code using Mojo::IOLoop directly in your plugin without worrying about the event loop (no need to call anything in SPADS "eventLoop" callback, just call the Mojo::IOLoop API functions directly).
User avatar
Nemo
Spring 1944 Developer
Posts: 1376
Joined: 30 Jan 2005, 19:44

Re: Plugin API: asynchronous code through alternate event loop

Post by Nemo »

Woooohoo!

You're the man, bibim :)
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: Plugin API: asynchronous code through alternate event loop

Post by bibim »

After several refactoring, SPADS event loop has been externalized from SPADS core into a new module: SimpleEvent. This new module brings AnyEvent support, but this is disabled by default until next major SPADS release.

You can still unlash AnyEvent power in SPADS 0.11.35 or later by adding a specific command line parameter "eventModel=AnyEvent" when launching SPADS, for example:

Code: Select all

perl spads.pl etc/spads.conf eventModel=AnyEvent
Of course, you need the AnyEvent perl module installed for this. Also, if you plan to use this with a plugin using Mojo::IOLoop, I guess you need to use the EV backend, which means you need to have a "use EV" statement in your plugin code.
User avatar
Nemo
Spring 1944 Developer
Posts: 1376
Joined: 30 Jan 2005, 19:44

Re: Plugin API: asynchronous code through alternate event loop

Post by Nemo »

bibim wrote:You can still unlash AnyEvent power
Ultimate power! :D

(this is incredibly awesome, and I am hyped)
User avatar
Nemo
Spring 1944 Developer
Posts: 1376
Joined: 30 Jan 2005, 19:44

Re: Plugin API: asynchronous code through alternate event loop

Post by Nemo »

As expected, this is the bomb. Just played around enough to convince myself that it worked, and indeed, now we can trivially host a web server inside the autohost in a plugin :)

Why is this cool, people may wonder?

Well, imagine:

Code: Select all

POST spring1944.org/game/4/team/3/units
{
  unitname: gertigerii,
  count: 5,
}
-- inject new units into a running game! Synchronously! Using only dedicated spring, not headless!

Or similarly:

Code: Select all

GET spring1944.org/game/5/team/3/units_killed
{
  teams: {
   0: 0,
   1: 423,
   2: 53
  }
}
get information out of a running game in real time, push -or- pull! (or even do something fancy like use a websocket, and stream this to the browser...)
User avatar
bibim
Lobby Developer
Posts: 952
Joined: 06 Dec 2007, 11:12

Re: Plugin API: asynchronous code through alternate event loop

Post by bibim »

Nice to see you enjoy AnyEvent support :)

Indeed, it allows lots of functionalities to be easily added in SPADS plugins, as one can see just from the >500 AnyEvent related module available on CPAN (let alone other event loops such as Mojo::IOLoop which are also compatible through the EV backend).

However, while AnyEvent - especially when using the EV backend - is very fast and efficient by design, be aware that SPADS core hasn't been optimized to run performance critical tasks (mainly because it wasn't needed for a basic autohost application, and also because I wanted maximum compatibility and minimum requirements on host system). For example it doesn't use async IO, and only some long jobs are actually performed asynchronously (currently even unitsync calls are performed synchronously, which is something I should change I guess). [*][/b][/color]

Basically SPADS currently offers the versatility of AnyEvent, but not the full efficiency theoretically possible with it. If you're not running performance critical tasks in plugins such as HTTP servers handling thousands or requests by second, it shouldn't be a problem though.

[*] Calls to unitsync library are now performed asynchronously since SPADS 0.12.1[/b][/color]
Post Reply

Return to “SPADS AutoHost”