Regarding the open beta tester initiative
March 24, 2021, 06:48:16 PM Posted by Suki on March 24, 2021, 06:48:16 PM in Regarding the open beta tester initiative | 3 CommentsFirst of all we want to extent a big thank you to all the people who participated in the initiative, we really appreciate the response we had and the collaboration everyone showed towards it. It really helped us to not only find and patch bugs but also served as a big motivator to keep on working on it!
We know the resulted release isn't perfect and there are still issues pending to be corrected, those are noted and scheduled onto our release cycle, however, I do want to point out that 2.0.18 was prob the biggest patch we ever released and having it go as smooth as humanly possible is a big win for everyone involved, needs to be celebrated but it also needs to be examined to look for improvement areas.
Having saying that here's some numbers to help understand the impact the initiative had on development:
- 5x more beta testers joined the group, almost 90% of them remained active during the beta testing phase.
- 2 Months and a few days was the duration of the program, from announcing to releasing. This includes the release process as well, actual beta testing time was close to a month and a half.
- 2 days average response from devs, this means whenever a bug was reported the devs response to confirm it and release new betas was 2 days average, kudos to shawnb61 for the excellent work and dedication!
- 15 bug reports, including other sources, we had a total of 15 bug reports, nearly 80% of them were confirmed.
- 127 commits were added to the codebase since 2.0.17
- 67 files were modified by 8 contributors
- A few rants. Kudos to SleePy and CoreISP for enduring my tantrums!
Improvement areas
- Better release focus. Having a set of things to look at or test really helped us but we can do better by segment each item on a the list and expand them on possible use cases.
- Better overall integration. Out pending issues have to do with not properly pay attention to all the corners of the system, we should create a list of things to look at while testing.
- Communication. Improve it for all the parties involved.
So again a big thank you and see you on the next testing round!
We know the resulted release isn't perfect and there are still issues pending to be corrected, those are noted and scheduled onto our release cycle, however, I do want to point out that 2.0.18 was prob the biggest patch we ever released and having it go as smooth as humanly possible is a big win for everyone involved, needs to be celebrated but it also needs to be examined to look for improvement areas.
Having saying that here's some numbers to help understand the impact the initiative had on development:
- 5x more beta testers joined the group, almost 90% of them remained active during the beta testing phase.
- 2 Months and a few days was the duration of the program, from announcing to releasing. This includes the release process as well, actual beta testing time was close to a month and a half.
- 2 days average response from devs, this means whenever a bug was reported the devs response to confirm it and release new betas was 2 days average, kudos to shawnb61 for the excellent work and dedication!
- 15 bug reports, including other sources, we had a total of 15 bug reports, nearly 80% of them were confirmed.
- 127 commits were added to the codebase since 2.0.17
- 67 files were modified by 8 contributors
- A few rants. Kudos to SleePy and CoreISP for enduring my tantrums!
Improvement areas
- Better release focus. Having a set of things to look at or test really helped us but we can do better by segment each item on a the list and expand them on possible use cases.
- Better overall integration. Out pending issues have to do with not properly pay attention to all the corners of the system, we should create a list of things to look at while testing.
- Communication. Improve it for all the parties involved.
So again a big thank you and see you on the next testing round!
Comments
Cool beans!
With just looking at the code does not fix it, also every machine can fail but a good hammer will solve it, right? 
Congratulations for a great job!