Release Early, Release Often.

[Argh]

Ok, people. In my opinion... the current release policy needs to be revised, based on what we've learned from the development of 0.77b1.

What do I think we've learned, mainly the hard way? Well, here are my conclusions, as the non-engine coder who sat observing most of what happened in SVN over the last 10.5 months:

1. Giant commits are very bad. They can break the entire engine, bitrot easily... and should not be allowed. Seriously. Just make smaller patches, and test with current SVN head before forwarding for commits. I know that this is going to cause a lot of argument and finger-pointing, but I'd like to say that we were all guilty, to some extent or another, of letting this happen this time. Were some of the contents of these giant commits well worth their price, when they were finally debugged? Yes. I'm mainly arguing that they should have been more gradual in nature whenever possible. The number of "ambitious goals" achieved in 0.77 was huge... imho, many of them could have been broken up into smaller segments and gradually digested... and if release early, release often had been the norm, they would have been tested by the public... and fixed sooner. A "keep working on it in the dark" policy is a very bad one for Spring- it's a model from the commercial world that I do not think we should attempt to emulate. Instead, see the players in the same light as the QA testers hired by major firms. Relying on the few guys like me who keep an eye on things and are testing "outside the box" is not a good policy. What happens, for example, when I leave, or get busier?

2. If it wasn't for Bibim, we'd have been totally screwed, imho. If it wasn't for him providing binaries that allowed for at least *some* QA oversight, then we'd be in a really terrible situation right now. People were building various things that ran fine on their hardware, without much testing from anybody. We should all thank Bibim very much, for keeping us from having a disastrous release. Seriously... we owe him a lot. From what I observed, it appears that the debugging builds really helped there at the end- I'm not qualified to judge that, but that's what it looked like, to me.

3. Just looking through the SVN, it looks like there were a record number of commits where one person corrected the work of another. While I praise everybody for helping out and trying their best to keep the engine stable... I also get the distinct impression that in several instances, we had people working at cross-purposes. When that happens, a little public (non-flame) discussion would be good, to arrive at a strategy. Yes, this will slow things down a bit sometimes, but if there's a genuine conflict of interests, I think that things should be slowed down. Most of the time, on the major issues, such as SMT, I think that you guys did a very good job on this, I'm just saying that it should be a goal- "stealthy coding" or "ninja commits" is a very bad idea, even if the code itself is well-formed and commented adequately.

4. The no-end-in-sight policy discouraged other game developers from keeping abreast of what was going on, until the very end... and as a result, from what I can see, a lot of games are not functioning correctly as of release. When I read game developers saying, only one month ago, "hai, I heard u might be releasing soonish, what's happened"... that's a VERY BAD SIGN

5. Because we ended up relying on yourselves, Satirik, myself (and my beta-testers, who luckily were willing to be crazy guinea pigs) to perform QA, we undoubtedly missed a lot of bugs. I've tested P.U.R.E. in 0.77b2... it's fine. However, I was working within that environment the whole time, so I was as guilty of tunnel-vision as anybody else was. It worked fine for me... and I darn well didn't have time to test for BA or any other mods. I should have made time, and would have done so if I'd had a little advance notice before the release, but I did not.

6. Before a release... I think it would be good to give a public notice via the News, say three days, so that anybody who wants to test a SVN build can do so and report bugs. It shouldn't happen like this, where (at least here, on my end), we're cruising along, and I can see that a lot of little bugs have been fixed really, really fast... then OMG, we have a release.

*****************************************************
I expect the usual round of "what do you know" posts. Fine, I'm not really qualified to have an opinion about the engineering side- all I did was test and observe, and occasionally kick somebody's buttocks in public, to get a few things fixed.

But nobody should argue that a nearly 11-month cycle where nearly half of that time was spent in Limbo, due to major patches that went awry in various ways... was a paradigm we should repeat. As fine as this release is, and it *is* something you guys should all be proud of... the process by which it was made was not, in my opinion, a good one.

If you don't agree with my conclusions, please take a whack at looking at the problems of process we had here, and provide solutions. Don't waste my time yelling at me, basically- if I'm wrong, and none of what I said seems to pin it down correctly, fine, but these are my critiques from where I sat on this. Just explain your version, so that we're having a dialogue, not a flame war, which would be utterly pointless.

Lastly... no finger pointing. No bad actors were involved- this wasn't "somebody's fault". So far as I can tell, everybody did their best to get everything working, and in the end, it all worked out, and you guys built something that is so much more advanced that 0.76 that it will take months, if not a year, for everybody to fully appreciate it. As much of the new stuff as I tried to shoehorn into P.U.R.E., I cannot claim that I even scratched the surface.

So don't see this as a slam, but as an opportunity to review what happened, and why, and whether it could have been better.

Oh... and one last thing... I have a crazy idea. What if there was some way to put the latest commits and descriptions into RSS, and show it here on the Forum somewhere? I hate having to manually page through the SVN, reading through each commit note... and it would make the connection between Joe Game Designer and Joe Engine Coder a lot stronger, imo. Just a crazy idea, but I think that while the current SVN is "transparent" in terms of non-engineer access, it takes a lot of work to keep up and it could be displayed to stakeholders in a better way.

[smoth]

disagreement.

[Argh]

No offense, Smoth, but given that you were AWOL through most of this, I can hardly be expected to give a hoot whether you agree or not

And I am citing a relatively well-accepted policy concept here. Not something I just made up because I thought it sounded cool

Tobi has said repeatedly that that was the goal. I am saying, "let's turn that into something that isn't a slogan, because taking 11 months on a build is bad policy".

Given that we're probably going to need a third release within a week or two, to address the serious bugs that got left in, largely because of the QA tunnel vision that occurred (which I am to blame for, if anybody is), I think it's high time to examine new models.

[smoth]

So because I have a life with a girlfriend and friends, because I was sick for a month I cannot say that I think release early and release often si a bad thing?

Spring cannot afford to treat it's players like betatesters, this isn't microsoft.

Developing things takes time and pushing out a potentialy unstable build is stupid.

Spring is being done by volunteers and I am sure they want to get the releases out of the door as well. However, we cannot ask the devs to give us a release time. that sort of hurry up and release mentality is one of the key failures of the software development industry.

Commit early and commit often, save early save often. Yeah I am fine with these and the EARLY part of your statement is where it all falls apart. Feel free to dismiss me for being AWOL, I have experience and a formal education so what would I know? I am not an industry veteran but somethings are stupid even to a school boy.

There was a time when the waterfall model was considered a good idea. Some people hold on to that idea.

I merely posted that I disagreed because you didn't want an argument, you didn't want people saying shup noob. I told you before, I am tire of arguing with you. Instead you want some drawn out time consuming debate which gets us no where. Saying I disagree is fine, I have just as much at stake with each release as you do, hell I want to say more, I have put a lot of time into gundam. I am sorry that you want to have some drawn out discussion that wants to get us nowhere while you pop off about shit that you have only read in some article, if I want to spend hours reading/listening to shit I can join the monthly tools conference at the office or read all the comany news letters or even better start working with a consultant.

is that enough TEXT for you? I thought "disagreemen"t was a much nicer post.

[Argh]

Yes, that was more useful. Now I can say with complete certainty that I disagree with your premise, which is based largely on the fact you did not spend any serious time in the trenches.

I think that the players should be a part of the testing and revision process. I don't think there's any excuse for not doing that. Who's going to find all of the bugs- a couple of guys testing their own stuff in relative isolation, or a larger number of eyes? If nothing else, the entire process of making P.U.R.E. has taught me that the more eyes I get on it, the more accurate the reports get, and the more actual bugs / annoyances get addressed. This isn't different.

Spring has a small playerbase, but that's irrelevant.

It's an engine. If people are having trouble with a new version, they can stick with an older one. It's hardly the end of the world. Spring should not exist merely to keep people playing BA at all hours, and while stability should always be a primary goal, the best way to ensure that stability is to make sure that the product is getting bashed around by a lot of people. We've tried limited-beta runs around here, and they've been useful, but I don't think they're adequate.

The more I think of it, my "crazy idea" about an easy-to-read feed display of commits seems like a good idea, too. I know it's probably not in the cards, but... I feel pretty strongly that if it was made a bit easier to read through, more people would, uh, find that time, and then actively test SVN revisions.

And if a bad release means that we annoy a few people, but rapidly improve the engine, and respond to our "customers", I can't see anything really wrong with that.

Lastly, this debate is long overdue. Your statement that this is just some words, when you couldn't find the time for action, is duly noted.

[FoeOfTheBee]

All hail the troll king.

Releasing a little more often shouldn't be controversial. It's easier on the coders and easier on the code to do things incrementally.

If you do things incrementally you need to test more often, but the results of the test are less likely to be catastrophic. The likely result is that Spring works more of the time for more people, because even if it breaks more often, the fixes will be faster.

Bandwidth isn't so valuable that downloading a few more releases is a big imposition on the players.

The coders will ultimately decide, and I expect they'll release more often not because of anything in the forums, but because it works better for them.

[smoth]

Tell me about your time in the trenches old one, I must learn about your years of experience and all of your studies argh.

[Argh]

Well, in my years and years of working with various groups of people on the Internet on game projects where we all worked for free, and the commercial game development I've done (very limited, I've published exactly one game, ever, and obviously it didn't make me rich)...

...I can sum it all up in one sentence:

"The higher the level of transparency between producers and customers, the better the final result."



For those who want that sentence explained a bit more, here's the TL:DNR version. Sorry, it really is quite long. What the hell, I haven't done any serious philosophical stuff here for awhile, other than the GPL debates and talking about P.U.R.E.... which is all very stimulating, to be sure, but doesn't really get to the heart of things.

Failure on software projects of sufficient complexity, in my experience, is usually caused by coders burning out because they're working in the dark without a lot of feedback. Over and over again, I've seen the change in the psychology of people working on stuff when they have feedback, and it's always positive. The coders may not see it that way, at the time. Nobody likes being told that their hard work isn't perfect. But it works out.

If you'd like to explore that proposal of fact further, I can probably give us a dozen examples or more off the top of my head. However, I personally think that's a given, tbh.

I take it that Smoth thinks that the coders should take as long as they feel like, without any deadline pressures. My feeling on that is that we've just seen what happens, and it's not a good model for the future. It's better for people to have to commit smaller chunks of something, unless it's giant, like zerver's SMT work.

That, if we go look at it, has been a pretty good model of transparency. Zerver has been willing to discuss his implementations and the problems with it quite openly, even though sometimes it's been a little prickly.

And that's good practice. Early releases would get more people involved in that feedback loop, basically, and it would not be a bad thing. No QA guy, no matter how good they are, can possibly see every angle. I missed a lot, Satirik missed a lot, my beta-testers and the early players of P.U.R.E. missed a lot... and so did the developers.

You seem to have a very weird idea that developers are godlike beings who are so far above the rest of us mortals that they don't need a second opinion, Smoth. Maybe it's because you're still so new to actual development... or maybe it's just a blind spot, I dunno.

In my experience, developers are human, and make mistakes. And they aren't gods, they're just people with specific talents, like any other specialist.

If you can accept this as part of your worldview, then you understand why I believe that openness, transparency, and having as many eyes on the ball as possible are a good thing.

I trust the devs to know their software engineering, by all means- I rarely even bother reading their source, unless it's super-important. Usually, when I do, it's to get them thinking, not because I actually think I'm going to solve the problem. That said, they're fallible, and usually the point of failure is in their inability to see outside their box.

As a game developer, I'm quite familiar with that particular problem... and I know that the best way to fix it is to get people to sit down and beat the snot out of it. All I'm advocating is that we have an open mind about what causes the failure of the process... and what might improve it.

Perhaps there are other ways:

1. Make Bibim's builds available from a link on the front page.

2. Make an RSS feed on the website that tells any visitor that the coders are busy and that Things Are Happening.

3. Create some sort of "beta-tester acknowledgment" program, that actively awards people for testing SVN builds and submitting feedback, by crediting them, like I did with P.U.R.E.

There are lots of possible ideas that could be explored. I just think that the most obvious solution is simply to release more often, and trust to user feedback to self-correct the system.

At any rate, I'm sure that all of the smart people around here can come up with solutions that work for them, these are just some random ideas to improve transparency and get coders the feedback they need.

[KDR_11k]

Other projects can release as often as they want because users can choose to upgrade or not and it doesn't matter when a new version is broken, you can roll back. With Spring everybody needs the same version.

[imbaczek]

we need a lobby infrastructure that can support multiple engine versions without pain and/or confusion; IIRC uberlobby supports that server-side.

[hoijui]

we need a lobby infrastructure that can support multiple engine versions without pain and/or confusion; IIRC uberlobby supports that server-side.

+1

[AF]

Saying don't do huge commits si fine but nobody will listen

For example trepan likes to work by making a humongous gigantic set of changes on his own for a few weeks then merge it into trunk and fix the hiccups. Its very annoying and people have tried to persuade him not to but he ignores it all, nonetheless he gets the work done.

[jK]

Saying don't do huge commits si fine but nobody will listen

For example trepan likes to work by making a humongous gigantic set of changes on his own for a few weeks then merge it into trunk and fix the hiccups. Its very annoying and people have tried to persuade him not to but he ignores it all, nonetheless he gets the work done.

<trepan> not really doing spring dev anymore

:'<

[AF]

Well youll have to get to work then doing his chores! *cracks whip*

[bibim]

Thanks Argh.
But you know, I don't think it would have changed that much things if I hadn't provided BuildServ. Someone else would just have provided a similar service (at least uploading up to date binaries somewhere from time to time), or Tobi/Fnordia would just have fixed buildbot uploads. Either way we would have easily found a way to cope with it. I just tried to help a little with what I had so devs could focus more on Spring itself.

[overkill]

Imo, the players shouldnt EVER be exposed to a potentially unstable game released, but should have easy acces to test things so there would be more testers, and the community wouldnt have to put up with unstable versions being released every month.

I do agree on HUDGE commits being reduced to small steps, and also this is most likely my last post in this topic, I cant stand being involved in retarded internet arguements.

[Tobi]

+1 to release early release often. Reality proves it's often hard to do though, but maybe with release branch as we tried again now it could be improved a bit.

Personally I think 2-3 months per release would be near ideal for Spring.

Didn't read the rest yet, too much text for now

[el_matarife]

Perhaps there are other ways:

1. Make Bibim's builds available from a link on the front page.

2. Make an RSS feed on the website that tells any visitor that the coders are busy and that Things Are Happening.

3. Create some sort of "beta-tester acknowledgment" program, that actively awards people for testing SVN builds and submitting feedback, by crediting them, like I did with P.U.R.E.

These are good suggestions, but I'd add a 4th suggestion that we go ahead and schedule multiplayer beta testing sessions ahead of time, by specifying that they occur on the first and third Friday of every month, for instance, or something similar. A day before release we specify a revision for the test by tagging it on SVN similar to a release to test so everyone can get installed, and we make sure there's a forums thread or news post or wiki entry where everyone can keep up to date. Also, possibly announce it to everyone on the lobby an hour before it starts.

Personally I think 2-3 months per release would be near ideal for Spring.

I think a six month schedule may make sense, we could release in September and March, so that we'd be out in the month before release of a bunch of Linux / BSD / other OS distros. It would give us plenty of time to make sure we can get in all those release, and we'd probably have time to get a hotfix patch in with them as well.

[Pxtl]

Smoth, "release early, release often" doesn't mean release buggy, it means write a piece of code, test it, and release it, rather than code-code-code-code-code-test-test-test-test-test release.

Also, "never go dark" - this is like release early, release often for the individual contributors. Large commits are indisputably a BAD thing. Commits should be as small as possible.

[Masure]

There is a difference between feature release and bugfix release.

When you talk about an ideal 3 month cycle, you mean feature release ?

I don't think dev have to release new features that often. I'd rather like many bugfix releases to have a fully playable game.

The 0.76b2 release has been running for a year with annoying bugs (decals, high pings on start) that could have been fixed with a bugfix release which is not a pain compared to a feature release.