Free Professional Assistance with finding bugs in the source

Robert.Finking · Post by **Robert.Finking** » 29 Mar 2006, 12:02

Hi Guys,

The company I work for is currently looking to test out various professional software tools to automate the finding of bugs in source code. In order to assess these tools side by side we want to make sure they all get used on the same bit of source code. Therefore I intend to point all the tool vendors at the TA Spring source code and get them to analyse it and report back to me all the defects they find.

I'm happy to pass on to you the output of each of the tools (probably upload to File Universe). This should result in the removal of large numbers of "crashbugs" , "memory leaks", "unitialised memory reads" etc. from the code.

My concern is that building the TA source is slightly problematic, as witnessed by the "Building spring (mingw and visual studio 7/8)" sticky thread. In order for both you and I to get the most out of this free analysis, is it possible to come up with a single clear set of instructions for:

1. How to obtain the source
2. How to build the source

Clearly, for the trial to be useful we need all vendors to be using the same version of the source. It's no good if they all get out "the latest", because they'll end up with different versions depending on when they get hold of it. The ideal would be if somebody could prepare a tarball or zip file with all the source in, accompanied with build instructions that have been tested and are known to work.

I hope this is a possibility - having looked at the Spring source code a while back, I think it could probably do with some automated analysis!

FWIW the tools that we're talking about here cost up to $50,000 each, still it's the results that count.

Regards

Robert Finking

Post by **jcnossen** » 29 Mar 2006, 12:28

Sounds really cool. I didn't know such tools were already this advanced!

I can build a single package containing everything for VS7 (source + libraries + gamedata).
If you want a different compiler then someone else has to do it...

Robert.Finking · Post by **Robert.Finking** » 29 Mar 2006, 12:56

Hi Zaphod,

That would be brilliant, thanks. Part of what I'm trying to assess is just how advanced these tools are. The marketing blurb sounds great, but I'm interested to see what happens when the rubber hits the road. I've read a couple of reviews where these tools have been applied to things like the Linux kernel or the source code for CVS and have been pretty effective, so I guess they must be fairly good.

A single file with everything needed for VS 7 should be fine. I'm hoping to get things kicked off with the tool vendors by the end of this week - does it sound doable to have it by then?

Regards

Robert Finking

BvDorp · Post by **BvDorp** » 29 Mar 2006, 14:06

Whoa, this should be fun!

Robert, can you tell the name of the software you're planning to use?

E: So, there's multiple tools involved, for the purpose of testing them? Awesome! I see the need for 1 robust package. Hope this can be done?

Post by **jcnossen** » 29 Mar 2006, 15:34

A single file with everything needed for VS 7 should be fine. I'm hoping to get things kicked off with the tool vendors by the end of this week - does it sound doable to have it by then?

That's ok.

Do these tools actually analyse the codepaths themself or is it required to run the code and make sure all the parts are executed?

Especially the global AIs are known to crash eventually, but because you need to play for a while to make them crash it's very hard and timeconsuming to debug them. It would be really cool if those were included in the test. I'll put 2 AIs in the package (JCAI and NTAI), because those are in the svn repository as well.

AF · Post by AF » 29 Mar 2006, 20:38

In that case I'll email you my latest source zaphod as it contains a few bugfixes (when you hand a command to spring that ahs an empty parameter list when it should have a full list it crashes, this only occurs in spring 0.7 though adn was fine in 0.6(I dont udnerstand why its happening though, I push_back() soemthing into the param list but param.empty() still evaluates as true, that and sometimes NTAI crashes with a memory access violation accessing an iterator even though I'm not messing with pointers and have taken great pains to avoid numerous pitfalls fromt he past))

Robert.Finking · Post by **Robert.Finking** » 30 Mar 2006, 09:25

Hi Zaphod

Zaphod wrote: Do these tools actually analyse the codepaths themself or is it required to run the code and make sure all the parts are executed?

Especially the global AIs are known to crash eventually, but because you need to play for a while to make them crash it's very hard and timeconsuming to debug them. It would be really cool if those were included in the test. I'll put 2 AIs in the package (JCAI and NTAI), because those are in the svn repository as well.

These are static analysis tools, so no they don't require the code to run. They do need to be able to "build" the code though, in order to analyse them.

Hopefully they should pick out some of those time consuming bugs for you =) At least some of these tools do full path analysis (incredlible though it may seem). Sure go ahead and stick the AI code in - the more the merrier =) Since you have a pluggable kind of architecture there, that would be useful from my point of view too.

Thanks again for your support.

Regards

Robert Finking

Post by **jcnossen** » 30 Mar 2006, 13:57

Here is the package I made (Compressed with 7zip: 36 MB):
http://spring.clan-sy.com/dl/taspring_r1083.exe

It includes sources (from svn revision 1083), binary build, gamedata. Redundant/old parts of the source code which are in svn are not included.

- Build output files go the game directory, it is set up like that in the rts project file.

- spring.exe is the starting point for single-player testing. The other exe in the game directory (TASClient) is a game lobby client written in delphi, so I'm not sure what you want to do with it. I have put it in for the sake of having a complete package.

- Go to rts/build/vstudio7 for the Visual Studio 2003 project files

- The AI project files are in ai/global/ntai and ai/global/jcai, and copy their DLLs automatically to /game/aidll/globalai/

There are also "group AIs", which are AI plugins that can be attached to units. These are much simpler and can be left out of the bug finding if that is convenient.

BvDorp · Post by **BvDorp** » 30 Mar 2006, 18:34

y don't u add aai and otai, since they might use the global AI in different ways. Also, could help those AI developers some :)

Tobi · Post by **Tobi** » 30 Mar 2006, 20:00

AAI is closed source at the moment if I recall correctly

AF · Post by AF » 30 Mar 2006, 20:22

Zaphod does that include the source I pm'ed you (uploaded to FU and quickly sent pm at last minute before I got kicked off comp yesterday)

Post by **jcnossen** » 30 Mar 2006, 20:58

Yes. Only I deleted the vs8 project files for clarity.

Robert.Finking · Post by **Robert.Finking** » 30 Mar 2006, 23:27

Hi Zaphod,

Zaphod wrote:Here is the package I made (Compressed with 7zip: 36 MB):
http://spring.clan-sy.com/dl/taspring_r1083.exe

Much appreciated. Using a given SVN version is an excellent idea.

I intend to send out the invite to all the vendors tomorrow - I've been in touch with most of them already on an informal level. If you want me to CC or BCC the mail to you send me a PM with your mail address.

I'll be refering the vendors to this board/thread for support getting the build going if that's OK?

Thanks again for all your help

Regards

Robert Finking

Min3mat · Post by **Min3mat** » 31 Mar 2006, 00:35

damn thats so cool! maybe Zaphod should get a free vsn for letting u use the sacred spring code

lol its open source =)

Robert.Finking · Post by **Robert.Finking** » 05 Apr 2006, 15:31

Hi,

I've had positive responses back from most of the tool vendors saying that they'll go ahead and analyse the TA Spring source code. I've asked them to submit their results to me by the 19th April.

I am going on leave for Easter tomorrow, so I won't be able to answer any of the vendor's questions myself. I've pointed them at this thread and this forum for support, so keep your eyes peeled for support requests.

Regards

Robert Finking

Robert.Finking · Post by **Robert.Finking** » 05 Apr 2006, 17:17

Hi there,

Amazingly one of the vendors has got back to me already with a summary of their initial analysis. It's a 2MB PDF file, which can be downloaded from here:
http://www.fileuniverse.com/?p=showitem&ID=2854

Note this is summary information only at this stage. The detailed bug reports should follow when I get back from leave.

Happy reading

Robert Finking

SwiftSpear · Post by **SwiftSpear** » 05 Apr 2006, 18:40

Wow, that thing is intense, good luck zaphod

Post by **jcnossen** » 05 Apr 2006, 19:13

That looks very promising, I'm looking forward to the full report :)

Argh · Post by **Argh** » 05 Apr 2006, 19:26

Interesting. I really enjoyed staring at the cluster... I didn't realize so much of Spring was held together that tightly hehe...

AF · Post by AF » 05 Apr 2006, 20:13

hmm no groupAI's/JCAI/NTAI data....

However I am adamant that the problem that makes the following crash spring 0.7 but not 0.67 is showing up

Code: Select all

Command c;
c.id=CMD_ATTACK;
aicallback->giveorder(unit,&c);

Spring RTS Engine

Free Professional Assistance with finding bugs in the source

Free Professional Assistance with finding bugs in the source

Static analysis

Thanks

On Leave

Initial results summary