2025-08-23 21:05 CEST
  

View Issue Details [ Jump to Notes ] [ Issue History ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0005405Spring engineGeneralpublic2016-11-28 10:112017-07-03 23:52
ReporterGoogle_Frog 
Assigned ToKloot 
PrioritynormalSeveritymajorReproducibilityalways
StatusresolvedResolutionfixed 
Product Version103.0 +git 
Target VersionFixed in Version 
Summary0005405: Performance regression since 103.0
DescriptionThe performance of 103.0.1-298-gaf1edb1 is noticeably worse than 103.0. Many players, including myself, have noticed this an commented on it since 103.0. Zero-K used 103.0.1-134-g819e5b4 for a while and the performance was a bit worse there as well.

The framerate is noticeably lower in medium sized games. Unit counts of 500 to 800 used to run smoother than they now do. My benchmarks are not good at detecting the difference between the engines except that it takes significantly longer to fast forward through the first eight minutes of my test case. This could be a symptom of the problem.

Example benchmarks here: https://docs.google.com/spreadsheets/d/1L_RadeoBEUqd45QkNE8qrY9itqJQtq4x0nXKOeK3NII/edit#gid=2078734332

If this widget loads all models then the difference in fast forward time is not due to the model loading change: https://github.com/ZeroK-RTS/Benchmarks/blob/master/games/caiFight_2016_11.sdd/LuaUI/Widgets/dbg_load_all_models.lua
TagsNo tags attached.
Checked infolog.txt for Errors
Attached Files

-Relationships
+Relationships

-Notes

~0016915

hokomoko (developer)

2016-11-29 04:23

 Looking during games, it seemed that lups is a main perf sink. I don't have anything smart to say regarding whether it changed in any way since 103.0

~0016917

Google_Frog (reporter)

2016-11-29 12:29

 Did the LUS handling of nanospray change between 103.0 and now? There was a change but I am not sure how old it is.

~0016918

hokomoko (developer)

2016-11-29 15:02

 Not that I know

~0017014

Google_Frog (reporter)

2017-01-03 06:19

 I have another benchmark comparing 103.0, 103-298 and 103-439. In it, 103-298 does slightly worse than 103.0 and 103-439 somehow has a bimodal distribution.

Plots: http://imgur.com/a/l2Q4s
Data: https://docs.google.com/spreadsheets/d/1ddQdofgAYj15COatiLDSrthQNf9dptoSVFcJIoe1lkI/edit?usp=sharing

~0017016

Kloot (developer)

2017-01-03 11:17

Last edited: 2017-01-03 12:04

View 2 revisions

 I would not call it bimodal without two distinct peaks. 103.0.1-439 has clearly just shifted a portion of the fps_15_20 and fps_20_25 buckets into fps_25_30 (which is an interesting performance improvement), widening but lowering [15,25] into a plateau.

These graphs also do not offer evidence of a notable regression between 103.0 and 103.0.1-298, or at least nothing to warrant a severity level of "major".

~0017017

Google_Frog (reporter)

2017-01-03 12:10

 The shift in the average over 15 games of 103-439 is because four of the games ran very well while the rest ran worse than previous versions. Look at the second image of the album. I'll need to do more to determine the difference between these two groups of games.

The regression from 103.0.1-298 to 103.0 is not fully shown in the data.

~0017032

Google_Frog (reporter)

2017-01-13 07:34

 Here is a dense benchmark between 103 and 103-134: https://docs.google.com/spreadsheets/d/1OAiozfyH6cgZwGiTy6Iizzz5NfLnUefzotBASk7526Y/edit#gid=597029653

The speedy_dt regression first shows up in 103.0.1-103-g741aa39 which has the following corresponding commit: https://github.com/spring/spring/commit/741aa393454a738d57e0c04f224108040bfaaa07

The unbuilt commit prior to it has this commit which does not seem large enough to be the culprit: https://github.com/spring/spring/commit/543efbd55c81579a678ebadec4d86466ee4b129a

~0017033

hokomoko (developer)

2017-01-13 08:29

 While the speedy_dt does indeed look weird-ish, the FPS seem to tell a different story.
I highly doubt changing a bunch of strings caused a huge regression, but we can easily test this.

~0017034

hokomoko (developer)

2017-01-13 08:39

 I've forced a build in a new branch with no timers, which should appear here in a few minutes:
https://springrts.com/dl/buildbot/default/no_timers

please test it vs. latest develop (preferrably use the same demo, so results are consistent - it shouldn't be an issue since sync is preserved)

~0017035

Kloot (developer)

2017-01-13 13:18

Last edited: 2017-01-13 13:32

View 3 revisions

 scoped timers lock a mutex on destruction so they don't come for free (disabling them when not in alt+B mode crossed my mind a few times), but *can not* have caused a regression. I would sooner suspect bbe039495975f23478266ba7fb5b1acd50f173b5 plus 828cd1f4eb5e889f466132e88197908d497cd054.

741aa393454a738d57e0c04f224108040bfaaa07 looks mostly harmless.

~0017036

hokomoko (developer)

2017-01-13 13:47

 There's a test showing boost::mutex to be clearly inferior to spring::mutex, so I highly doubt it

~0017037

Google_Frog (reporter)

2017-01-13 14:09

 Quick benchmark results: https://docs.google.com/spreadsheets/d/1s44WFatmnUpX_EMjdbEtM1I6Rg1xh0Y-wK5IIcxeyww/edit?usp=sharing

Removing the timers restores speedy_dt to 103.0 values. I'll run a longer benchmark overnight to gather enough data for a significant comparison of framerate.

~0017038

Google_Frog (reporter)

2017-01-14 02:44

 Here is a longer benchmark: https://docs.google.com/spreadsheets/d/1O5-o8znSHLthoH0y9kindxku59ILeFmffxJwXgdm-74/edit?usp=sharing

There are 15 runs of each of 103-298, 103-489 and 103-490. There are also five runs of 103-489 and 103-490 on the same replay.

I think there is some improvement in 103-490 but 103-298 is still doing better. Here is a plot: http://i.imgur.com/vHvxms7.jpg

~0017039

Kloot (developer)

2017-01-14 13:31

 298 is doing better despite containing a claimed regression that was fixed (surpressed, really) in 490? I think you'll agree that makes no sense.

~0017040

hokomoko (developer)

2017-01-14 13:37

 There are two different issues here, speedy_dt (fast forward time) which was supposedly suppressed/fixed, and FPS which was barely affected by 490 (and had nothing to do with the timers commits, as I said earlier).

~0017041

Google_Frog (reporter)

2017-01-14 13:53

 Sorry, misnamed ticket. Rename it to "Performance regressions since 103.0".

~0017055

Google_Frog (reporter)

2017-01-16 23:30

 Here is a benchmark of some versions between 103-298 and 103-489. It took a few nights of false starts because many intermediate versions don't run or crash on exit. https://docs.google.com/spreadsheets/d/1sZRlPIxsBg2XYXozYCOQGJOjuSvBVqB0xrSHcQn_og8/edit?usp=sharing

hokomoko wanted Spring.SendCommands("debuginfo profiling") so I added it to the end of the benchmark, after the minute of frame gaps are taken. I'll attach the infologs.

~0017056

Kloot (developer)

2017-01-17 01:54

Last edited: 2017-01-17 02:40

View 4 revisions

 going by the 5 averaged w_update_dt values, there is one degradation between 345-g62b3749 and 390-ga808bf1 (0.052 -> 0.059dt, 19.15 -> 16.92fps) and another between 390-ga808bf1 and 439-g438f3d3 (0.059 -> 0.066dt, 16.92 -> 15.21fps) of equal magnitude, while all changes in the [298, 345] and [439, 489] ranges had no significant effect.

the first can be explained by a fix related to the UHM which removed a broken (since LosHandler was refactored) optimization *or* by a bunch of container swaps (PathCache and DecalHandler stand out), the second could be either due to switching ThreadPool to use a LF queue (barely included in 439) or several more container replacements (less likely because none of those were individually high-impact). I'll create two test branches reverting the UHM and PFS edits to start with.

of note: both 345-g62b3749 and 390-ga808bf1 fall in the ThreadPool-being-busted range where tasks could simply be dropped before finishing or without ever executing. the last valid "old" ThreadPool build would be ed95df3c36 and is only a few commits ahead of 298.

~0017057

Kloot (developer)

2017-01-17 02:54

Last edited: 2017-01-17 02:55

View 2 revisions

 please test these builds when they become available at https://springrts.com/dl/buildbot/default/ under mantis5405*:

g2914ab9a (UHM)
g2dc4fadd (PC)
g130b499d (PE)

~0017058

Google_Frog (reporter)

2017-01-17 03:18

 Thanks, will these builds sync with develop 103.0.1-509-gb514b5b? If so I could use the same replays on each of them for more reliability.

~0017059

Kloot (developer)

2017-01-17 11:21

Last edited: 2017-01-17 11:26

View 2 revisions

 g2914ab9a and g2dc4fadd should sync, g130b499d won't.

for g130b499d you also need to delete your path cache.

~0017060

Google_Frog (reporter)

2017-01-17 13:22

 I benchmarked earlier today, before reading this, so there are some issues. Here is the data https://docs.google.com/spreadsheets/d/1T8p4Uy6Rf7MMZx4VRKMzcUNbo8oCY3_FuV-OS7G3QuY/edit?usp=sharing

All my previous tests are with 32 bit Spring but only 64 bit builds were available. The framerate numbers appear to be higher but there still the regression between 298 and 509 (I used the 64 bit versions for these two versions as comparison).

I did not delete path cache for any of the runs. The benchmarks use the same configuration directory so they share path caches. This might have broken something.

The results seem to say that 103.0.1-510-g2dc4fad (the first path cache test) is the cause of the regression. I'm going to run a longer benchmark overnight with independent configuration directories and cleared path caches to make sure. If 32 builds are available I will use them for the sake of consistency.

~0017061

Google_Frog (reporter)

2017-01-17 15:21

 I think the different branches already caused some independence between my path caches. http://i.imgur.com/dOXwDKe.png

~0017062

Kloot (developer)

2017-01-17 16:08

Last edited: 2017-01-17 16:10

View 2 revisions

 g130b499d and g2dc4fadd are on the same branch.

32-bit builds are now available, and there is a candidate fix on develop (sync-compatible with 509-gb514b5b) under 511-g12f211a8/.

~0017063

Google_Frog (reporter)

2017-01-18 01:14

 Here is my benchmark from last nigh: https://docs.google.com/spreadsheets/d/17Phny9I5fFy8cDNfxZH70uZRoIPbZYm5kKG3kO4VlNw/edit?usp=sharing

103.0.1-511-g130b499 has the best framerate but this might be caused by fewer units on the map. The average unit count is 30 less than in other runs so something significant changed in the runs.

Here are the average FPS buckets http://i.imgur.com/OTiYRkE.jpg

~0017066

Kloot (developer)

2017-01-18 12:48

Last edited: 2017-01-18 12:49

View 2 revisions

 g130b499 (PE) effectively reintroduces 5411 so units will take less optimal paths which leads to different game outcomes. it would however be more logical to attribute regained (298-level) perf to that than to g2dc4fadd (PC), but any hard conclusions should be reserved until the effect of g12f211a8 on 509 is measured.

g2914ab9a (UHM) can be ruled out.

~0017069

Google_Frog (reporter)

2017-01-19 01:24

 Benchmark comparing 103.0.1-511-g12f211a and 103.0.1-509-gb514b5b https://docs.google.com/spreadsheets/d/1pR1l3JofNzvA8IFBDL-QxA_V8N1zj_MrAZ4IVle8hvI/edit?usp=sharing

There seems to be an improvement. Here are the average FPS buckets visualized: http://i.imgur.com/ZTcifEy.jpg

~0017070

Google_Frog (reporter)

2017-01-19 14:27

 Here is a run which was supposed to compare 103.0.1-511-g12f211a and 103.0.1-298-gaf1edb1: https://docs.google.com/spreadsheets/d/1IzGCPyT8ESOtmsNtVMJ7P2p8idW9keShVou2mzY5c2I/edit?usp=sharing

I accidentally used 64 bit 103.0.1-298-gaf1edb1 and 32 bit 103.0.1-511-g12f211a so I need to retest. One interesting thing is that the number of units at the start of the benchmarks differ by 25.

~0017071

Kloot (developer)

2017-01-19 15:36

Last edited: 2017-01-19 15:40

View 2 revisions

 An average 0.8fps absolute (5% relative) improvement over 509 is far from bad, though not enough.

There is some low-hanging optimization fruit left in the PFS but reclaiming 4fps while also keeping the 5411 fix may not be possible.

~0017072

hokomoko (developer)

2017-01-19 16:02

Last edited: 2017-01-19 16:36

View 2 revisions

 https://github.com/spring/spring/commit/e41d9be271b9fd8cb0726c33c96d4f2c1a04910d#commitcomment-20539160
^ is worth testing. I believe the added access will hurt less than the added states expanded


EDIT: or not. well, worst case I'll try making my own PF :P

~0017073

Kloot (developer)

2017-01-19 18:49

Last edited: 2017-01-19 18:57

View 2 revisions

 new benchmarking candidate versus 511-g12f211a: https://springrts.com/dl/buildbot/default/develop/103.0.1-514-gfa03978/

this one also requires playtesting (if it meets its performance target) due to the PE blocksize increase.

~0017074

hokomoko (developer)

2017-01-19 18:58

 The regular benchmark doesn't give a good measurement of pathdfinding as it's done on CCR

~0017075

Google_Frog (reporter)

2017-01-20 00:44

 Here is a benchmark for 103.0.1-511-g12f211a and 103.0.1-298-gaf1edb1 and 32 bit vs 63 bit: https://docs.google.com/spreadsheets/d/1vKK2cmOL2p4aX2qxFL-WYexEmly3xmWUcVIIJqwrXk0/edit?usp=sharing

FPS buckets: http://i.imgur.com/xwvFO4p.jpg

~0017076

Google_Frog (reporter)

2017-01-20 04:06

 I ran a quick benchmark for 103.0.1-511-g12f211a and 103.0.1-514-gfa03978. There seems to be no effect: https://docs.google.com/spreadsheets/d/1BQ4n6pCpbzyxSXqm_3qWvTTF8az9GAdt9ACWs2jxJkc/edit?usp=sharing

FPS buckets: http://i.imgur.com/X9yIFQx.jpg

~0017079

Kloot (developer)

2017-01-20 17:04

 103.0.1-511-g12f211a versus 103.0.1-298-gaf1edb1 is inconsistent (64-bit 511 gets close to 298, 32-bit 511 falls off a cliff) so I'm taking that one with a grain of salt.

there won't be [m]any new builds for a while.

~0017080

Google_Frog (reporter)

2017-01-21 00:39

 Are you going to revert https://github.com/spring/spring/commit/fa039784ac501a7b0c6082ea6527250667272484 since it probably impacts pathing quality?

Here is a dense run between 390 and 439. https://docs.google.com/spreadsheets/d/1Kh6JiDNmAZ49IYbA6HcjymwVDXj9fkTAdh2tDdoltLk/edit?usp=sharing

~0017081

Kloot (developer)

2017-01-21 01:32

Last edited: 2017-01-21 01:35

View 3 revisions

 not directly, fa039784 will be subsumed by a more comprehensive scheme I have in the works.

apparently 416-428 compensated for whatever caused the drop in 390-402 (no obvious candidate, maybe 1f4db15), neat.

~0017102

Google_Frog (reporter)

2017-01-22 02:54

 Here is a dense benchmark from 103.0.1-390-ga808bf1 to 103.0.1-402-g1f4db15: https://docs.google.com/spreadsheets/d/1zZC8fYbbuS8lXWevyKeVw478UpgqOCSGiFF25pUo6zo/edit?usp=sharing

The 0.9 FPS drop from 390 to 402 has reduced to a drop of 0.1 which just goes to show that the data is highly noisy. I made a chart comparing FPS to units_start over all runs to check whether the FPS change was just correlated with the number of units running around. Here it is: http://i.imgur.com/e6E9upA.jpg

FPS buckets: http://i.imgur.com/DTjvUF5.jpg

If this data is to be believed then one of these two commits caused a decline in performance from 103.0.1-390-ga808bf1 to 103.0.1-392-g315ed0a:
 * https://github.com/spring/spring/commit/315ed0a311ea3668c9d5893fc84fdbae98c92aa0 (possible? As far as I know our normalmapping CUS gadget could be affected by this).
 * https://github.com/spring/spring/commit/486014134fc22989130fc85b35744cb67f1405e9 (I don't know how much of the restructuring of LOS update is actually enabled).

The data would also have us believe there is a performance decline from 103.0.1-399-gf1a9225 to 103.0.1-402-g1f4db15. Looking at the commits it is surely https://github.com/spring/spring/commit/1f4db15421b5954baf0a5abb50339392d1e44c3b. The benchmark doesn't even feature any Cannon-using units with flightTime set.

The data also says that doing 's/map/vector/' is a great idea.

I worry that noise will make any dense run say that performance goes up and down from version to version. However, the previous follow investigation yielded some consistent gains so I think these are worth testing.

~0017107

Kloot (developer)

2017-01-22 13:32

Last edited: 2017-01-22 13:38

View 2 revisions

 315ed0a311 can be sped up a bit, I took the lazy approach there.

486014134f is a no-op with respect to the staggered updates part.

1f4db15421 should cause a minor decline related to the number and radius of terrain-deforming explosions that take place across the map.

since at least ~1fps can be hidden within the noise (and averaging over many more runs to remove it would make benchmarking infeasible), taking that value as a variance bound seems useful to avoid chasing phantoms.

~0017111

Google_Frog (reporter)

2017-01-23 13:35

 I reram 103-390 to 103-402 and I think much of what we see is flunctuation. https://docs.google.com/spreadsheets/d/148BjR_SQTE7s7a51rBw4U7lBCfsjrJ3LKf0LgGPZtDU/edit?usp=sharing

I applied 103.0.1-519-g91b0000 to the ZK hosts for feedback/crash testing. Players commented that Spring freezes a bit at the start of the game so I have made a 4 second benchmark that replicates this.

~0017112

Google_Frog (reporter)

2017-01-23 13:46

 Benchmark commit (sorry, some other stuff was included): https://github.com/ZeroK-RTS/Benchmarks/commit/fe782b8f2da0ddda456007d04d57eaf9655190b1

This is what running the benchmark looks like: https://www.youtube.com/watch?v=hJaLr7VDVJA

~0017113

Google_Frog (reporter)

2017-01-23 14:15

 I've run the benchmark and it shows that https://github.com/spring/spring/commit/5a97ea9cef3e3aa629e1acd1e38e3bfe8371e315 (or possibly its redux https://github.com/spring/spring/commit/681fe52e02679ca4ecae43afb75e1dc1dc16f65d) causes the gap between frames when spawning 12 commanders to increase from around 0.2 seconds to 2 seconds.

See w_update_dt(max) here: https://docs.google.com/spreadsheets/d/1pHZ9IEtNQTolKgvmYgo1ZvRNR2XbSb876_wqV_gDwfc/edit?usp=sharing

~0017114

Google_Frog (reporter)

2017-01-23 14:40

 The effect (2s frame gap) is lost when the commander is replaced with armwar, a unit that has less lua attached to it than a commander.

Removing all gadgets and widgets and running with a commander retains the effect.

Replacing the model of armwar with that of the commander and running the benchmark with armwar retains the effect.

~0017115

Google_Frog (reporter)

2017-01-23 14:46

 The commander uses the model commsupport.s3o and armwar uses the model spherewarrior.s3o.
 * commsupport.s3o uses two 512x512 dds textures. Has 28 pieces.
 * spherewarrior.s3o uses two 1024x1024 dds textures. Has 22 pieces.

~0017116

Google_Frog (reporter)

2017-01-23 14:49

 Deleting all pieces, except the empty piece base, from commsupport.s3o retains the effect.

~0017117

Google_Frog (reporter)

2017-01-23 14:57

 I've got a bit mixed up, will recheck the last comment and present something a bit more organized later.

~0017119

Kloot (developer)

2017-01-23 15:44

 Possibly threadpool task distribution is still suboptimal.

Spring would suffer from this the most with tasks that require heavy disk IO, like loading models and textures. I also noticed /give all is horrifically slow in ZK compared to BAR.

~0017120

Google_Frog (reporter)

2017-01-23 16:00

 I've cleared up what is happening. The last four comments can be safely ignored. I was able to replicate the issue by spawning a single constructor. The issue goes away if the buildlist of the constructor is removed. This makes me think that spawning a constructor loads something related to the units it can build and that this loading has become slow. If this is the case it could be worthwhile to check other types of resources as well.

Here is a benchmark https://docs.google.com/spreadsheets/d/1KfHuDswecK2RqFTGJPlq4b6U9g4eL7Yrqmjxx0GqCFY/edit?usp=sharing

The benchmarks used are here: https://github.com/ZeroK-RTS/Benchmarks/commit/5ad7f28da4e4bfec219b99e4dbd590cda7c4abd2

~0017121

Google_Frog (reporter)

2017-01-23 16:02

 Many BAR models share the same texture.

~0017122

hokomoko (developer)

2017-01-23 16:03

 This points to model preloading which was changed to use the threadpool, something should make sure that the main thread takes no tasks if there is no "for_mt" etc. active.

~0017125

Kloot (developer)

2017-01-23 22:58

Last edited: 2017-01-23 23:00

View 2 revisions

 The spike in GF's MultiPlop actually gets created by for_mt (which forces WaitForFinished), not model loading. Compiling without threadpool or never spawning any threads makes it disappear.

~0017126

hokomoko (developer)

2017-01-23 23:15

Last edited: 2017-01-23 23:18

View 2 revisions

 https://github.com/spring/spring/blob/develop/rts/Rendering/Models/IModelParser.cpp#L198

Model preloading doesn't happen when threadpool is off...

Is it possible that the first WaitForFinished blocks the execution until all tasks are finished and not just the for_mt one?

~0017127

Kloot (developer)

2017-01-23 23:43

Last edited: 2017-01-23 23:51

View 4 revisions

 A for_mt task can end up being queued after preloading tasks.

Naturally, WaitForFinished(for_mt) will then block until they're out of the way.

tldr: truly asynchronous tasks should have their own private queue(s).

~0017128

hokomoko (developer)

2017-01-23 23:54

Last edited: 2017-01-23 23:54

View 2 revisions

 so they should probably just not use the threadpool (which was the original implementation)

~0017129

Kloot (developer)

2017-01-24 00:11

 let's just say threadpool was not ready for primetime and the decision to use it based on incomplete knowledge.

in any case it's a small step now and I would prefer Spring's threading to be as centralized as possible.

~0017198

Google_Frog (reporter)

2017-02-04 14:42

 Here is a /debuginfo profiling since hokomoko requested it earlier.

~0017931

Kloot (developer)

2017-07-03 23:52

 I think this is closable now, perf has been generally restored.
+Notes

-Issue History
Date Modified Username Field Change
2016-11-28 10:11 Google_Frog New Issue
2016-11-29 04:23 hokomoko Note Added: 0016915
2016-11-29 12:29 Google_Frog Note Added: 0016917
2016-11-29 15:02 hokomoko Note Added: 0016918
2017-01-03 06:19 Google_Frog Note Added: 0017014
2017-01-03 11:17 Kloot Note Added: 0017016
2017-01-03 12:04 Kloot Note Edited: 0017016 View Revisions
2017-01-03 12:10 Google_Frog Note Added: 0017017
2017-01-13 07:34 Google_Frog Note Added: 0017032
2017-01-13 08:29 hokomoko Note Added: 0017033
2017-01-13 08:39 hokomoko Note Added: 0017034
2017-01-13 13:18 Kloot Note Added: 0017035
2017-01-13 13:28 Kloot Note Edited: 0017035 View Revisions
2017-01-13 13:32 Kloot Note Edited: 0017035 View Revisions
2017-01-13 13:47 hokomoko Note Added: 0017036
2017-01-13 14:09 Google_Frog Note Added: 0017037
2017-01-14 02:44 Google_Frog Note Added: 0017038
2017-01-14 13:31 Kloot Note Added: 0017039
2017-01-14 13:37 hokomoko Note Added: 0017040
2017-01-14 13:53 Google_Frog Note Added: 0017041
2017-01-16 23:30 Google_Frog Note Added: 0017055
2017-01-16 23:30 Google_Frog File Added: infologbatch_298_489.zip
2017-01-17 01:54 Kloot Note Added: 0017056
2017-01-17 01:55 Kloot Note Edited: 0017056 View Revisions
2017-01-17 02:00 Kloot Note Edited: 0017056 View Revisions
2017-01-17 02:40 Kloot Note Edited: 0017056 View Revisions
2017-01-17 02:54 Kloot Note Added: 0017057
2017-01-17 02:55 Kloot Note Edited: 0017057 View Revisions
2017-01-17 03:18 Google_Frog Note Added: 0017058
2017-01-17 11:21 Kloot Note Added: 0017059
2017-01-17 11:26 Kloot Note Edited: 0017059 View Revisions
2017-01-17 13:22 Google_Frog Note Added: 0017060
2017-01-17 15:21 Google_Frog Note Added: 0017061
2017-01-17 16:08 Kloot Note Added: 0017062
2017-01-17 16:10 Kloot Note Edited: 0017062 View Revisions
2017-01-18 01:14 Google_Frog Note Added: 0017063
2017-01-18 12:48 Kloot Note Added: 0017066
2017-01-18 12:49 Kloot Note Edited: 0017066 View Revisions
2017-01-19 01:24 Google_Frog Note Added: 0017069
2017-01-19 14:27 Google_Frog Note Added: 0017070
2017-01-19 15:36 Kloot Note Added: 0017071
2017-01-19 15:40 Kloot Note Edited: 0017071 View Revisions
2017-01-19 16:02 hokomoko Note Added: 0017072
2017-01-19 16:36 hokomoko Note Edited: 0017072 View Revisions
2017-01-19 18:49 Kloot Note Added: 0017073
2017-01-19 18:57 Kloot Note Edited: 0017073 View Revisions
2017-01-19 18:58 hokomoko Note Added: 0017074
2017-01-20 00:44 Google_Frog Note Added: 0017075
2017-01-20 04:06 Google_Frog Note Added: 0017076
2017-01-20 17:04 Kloot Note Added: 0017079
2017-01-20 17:04 Kloot Status new => acknowledged
2017-01-21 00:39 Google_Frog Note Added: 0017080
2017-01-21 01:32 Kloot Note Added: 0017081
2017-01-21 01:34 Kloot Note Edited: 0017081 View Revisions
2017-01-21 01:35 Kloot Note Edited: 0017081 View Revisions
2017-01-22 02:54 Google_Frog Note Added: 0017102
2017-01-22 13:32 Kloot Note Added: 0017107
2017-01-22 13:38 Kloot Note Edited: 0017107 View Revisions
2017-01-23 13:35 Google_Frog Note Added: 0017111
2017-01-23 13:46 Google_Frog Note Added: 0017112
2017-01-23 14:15 Google_Frog Note Added: 0017113
2017-01-23 14:40 Google_Frog Note Added: 0017114
2017-01-23 14:46 Google_Frog Note Added: 0017115
2017-01-23 14:49 Google_Frog Note Added: 0017116
2017-01-23 14:57 Google_Frog Note Added: 0017117
2017-01-23 15:44 Kloot Note Added: 0017119
2017-01-23 16:00 Google_Frog Note Added: 0017120
2017-01-23 16:02 Google_Frog Note Added: 0017121
2017-01-23 16:03 hokomoko Note Added: 0017122
2017-01-23 22:58 Kloot Note Added: 0017125
2017-01-23 23:00 Kloot Note Edited: 0017125 View Revisions
2017-01-23 23:15 hokomoko Note Added: 0017126
2017-01-23 23:18 hokomoko Note Edited: 0017126 View Revisions
2017-01-23 23:43 Kloot Note Added: 0017127
2017-01-23 23:44 Kloot Note Edited: 0017127 View Revisions
2017-01-23 23:45 Kloot Note Edited: 0017127 View Revisions
2017-01-23 23:51 Kloot Note Edited: 0017127 View Revisions
2017-01-23 23:54 hokomoko Note Added: 0017128
2017-01-23 23:54 hokomoko Note Edited: 0017128 View Revisions
2017-01-24 00:11 Kloot Note Added: 0017129
2017-02-04 14:42 Google_Frog File Added: profiling103.0.1-570-gcf3f016.txt
2017-02-04 14:42 Google_Frog Note Added: 0017198
2017-07-03 23:52 Kloot Assigned To => Kloot
2017-07-03 23:52 Kloot Status acknowledged => resolved
2017-07-03 23:52 Kloot Resolution open => fixed
2017-07-03 23:52 Kloot Note Added: 0017931
+Issue History