Regression testing

[GenghisXerxes]

Having been a player for some time now I have noticed that with each release we sometimes see the re-emergence of an old issue that has been resolved at some point in the past. I assume that each mod or bug fix is properly tested prior to release and will therefore have an associated test script. So what I suggest is the following:
- Inno build a fixed test environment that it can refresh to a known state,
- they ask players if they would like to participate in the Regression test BEFORE the software is released,
- any players who want to participate are emailed a test script (including relevant user id and password etc) and then conduct the test in the test environment
- results of the test are emailed back to Inno.
Clearly players will need to be given an incentive for doing this. Diamonds is the obvious one but equally Forge Point Packs, goods, coins, supplies and/or premium buildings may also generate some interest. Each test script would need to be ranked by difficulty (for example, simple, moderate, difficult, complex) with the level of reward appropriate to the difficulty.
This is a relatively straightforward process to put in place and one that will improve the quality of the product and the player experience.

 

[DeletedUser6065]

' . . . I have noticed that with each release we sometimes see the re-emergence of an old issue that has been resolved at some point in the past. I assume that each mod or bug fix is properly tested prior to release . . .'.

And this is where GenghisXerxes met his Waterloo.
. . . mk

 

[DeletedUser109510]

You would think that regression testing was something that happened with each release. I'd fire any programmer of mine that did not do this.

 

[DeletedUser107476]

You would think that was what beta was for when they release there first.
This has been proven to not be the case as feedback often includes bugs etc but it is still released without fixing them.
So regression testing will not happen.

 

[DeletedUser720]

Maybe they should make a sandbox environment, where people can test more than on beta. The main difference vs beta would be that this world would have it's speed increased (to avoid long waiting times when testing) and maybe (at the risk of sounding greedy) a large amount of diamonds given each time a new feature is released.

 

[DeletedUser107476]

Maybe they should make a sandbox environment, where people can test more than on beta. The main difference vs beta would be that this world would have it's speed increased (to avoid long waiting times when testing) and maybe (at the risk of sounding greedy) a large amount of diamonds given each time a new feature is released.

A sandbox would not really work. What inno want to test, is when the coding is being used by multiple users 4k plus and what it does to to old coding when being used.
A sandbox is normally restricted down in users and are normally created to test one piece of coding and how it affects the software.
It could be used at start but you would still need a beta world to truly test it.
What inno really needs to do is listen to feedback from beta then re-run the software when bugs are removed and re-test on beta.
If they won't listen to feedback from beta do you truly think they will listen to feedback from a sandbox?

 

[DeletedUser102348]

They already run missions on Beta as well as bug reporting.

 

[GenghisXerxes]

OK - I appreciate the comments but let me try to explain a little more.
Assume we have a bug - BUG X - which is something to do with Stealth Tanks.
The programmers come along, develop a fix for BUG X, a test script is produced, it is tested OK and included for release in RELEASE N.
Then assume we have a new bug - BUG Y - in RELEASE N related to the way Rail Guns work.
The programmers come along, develop a fix for BUG Y, a test script is produced, it is tested OK and included for release in RELEASE N+1.
Does anyone re-run the test script for BUG X to ensure it has not been impacted by the fix for BUG Y?

 

[DeletedUser109510]

OK - I appreciate the comments but let me try to explain a little more.
Assume we have a bug - BUG X - which is something to do with Stealth Tanks.
The programmers come along, develop a fix for BUG X, a test script is produced, it is tested OK and included for release in RELEASE N.
Then assume we have a new bug - BUG Y - in RELEASE N related to the way Rail Guns work.
The programmers come along, develop a fix for BUG Y, a test script is produced, it is tested OK and included for release in RELEASE N+1.
Does anyone re-run the test script for BUG X to ensure it has not been impacted by the fix for BUG Y?

Exactly

 

[DeletedUser107476]

OK - I appreciate the comments but let me try to explain a little more.
Assume we have a bug - BUG X - which is something to do with Stealth Tanks.
The programmers come along, develop a fix for BUG X, a test script is produced, it is tested OK and included for release in RELEASE N.
Then assume we have a new bug - BUG Y - in RELEASE N related to the way Rail Guns work.
The programmers come along, develop a fix for BUG Y, a test script is produced, it is tested OK and included for release in RELEASE N+1.
Does anyone re-run the test script for BUG X to ensure it has not been impacted by the fix for BUG Y?

So you would expect a new release every what year? No new builds, no new features etc
To do what you want they would have to play test the new release for 2 weeks. Play test everything else too. If a bug is found then play test that bug and make sure it does not impact on all previous fixes. Then try play testing it again. And it goes on and on.

I know of games that are on consoles that have been play tested all the way through development, then before release. Yet they still have bugs and need patches to fix them.

So if you want to wait a year or two for each new feature then your idea may work.

Bugs should be patched as they are found. Whether that is in development or once released does not matter as long as the response time is good. This is where inno could improve, in fixing bugs faster.

 

[DeletedUser107476]

Personally I would prefer Inno to cut back on events with just 4 for now: Winter, Spring, Summer and Autumn.
This would allow them to fix current issues within the game and concentrate on their development era wise.
Then they could add other events once they have the game running the way it should.

I would also like them to use beta as it should be used. Test a new feature and listen to feedback. If players are not happy with it, don't release to main. If bugs found try and fix at that stage and try it again on beta. The rush from beta to main is unnecessary and should not be happening.

 

[DeletedUser108240]

Such "Old Bugs coming back" errors are usually caused by lack of a good and useable versioning system.

When the original error was released some undisciplined programmer kept the sourcecode file (with the error) on his PC, instead of placing it back in the versioning system and deleting it from his PC.
And then he continued working on that file without fetching it from the versioning system first (so he could get the corrected version), thus re-introducing the error again. and potentially again, etc...

I remember an online "Game of Thrones" game that did this so consistently that the game just died. Everybody left. So Cheer up. Inno is not the worst programming house ;-)).

 

[DeletedUser6838]

Personally I would prefer Inno to cut back on events with just 4 for now: Winter, Spring, Summer and Autumn.
This would allow them to fix current issues within the game and concentrate on their development era wise.
Then they could add other events once they have the game running the way it should.

Doesn't really matter, those 4 are yeah sort of "big" events, but the rest are pretty much copy/paste from each other just with different text and images, quests are almost identical - complete 24 hour supply work ina production building x times, scout a new province, acquire x sectors, etc.

Also, i don't think there is a need for such control on previous bugs on every update, cause mostly when a update happens, there appears new bugs more than old ones resurfacing, so using the bug reporting is more efficient in my opinion.