Summary
Following the upgrade of the statement generator in Mabdeck (see release notes for further information), we have been advised of an issue preventing the production of statements.
See below for the cases affected. If you are aware of any other problems relating to raising statements using Mabdeck, please contact our support team, and we will provide you with guidance.
Please follow this service incident to receive automatic updates as we continue to investigate and implement fixes.
For more information regarding Billing Profiles, please see the dedicated section in our help centre.
09/06/2021 - 17:00 GMT
Incident Report - Post Mortem - please see below
02/06/2021 - 22:00 GMT
Deployment complete.
02/06/2021 - 13:15 GMT
Deployment confirmed to be completed overnight.
02/06/2021 - 09:15 GMT
All testing complete, awaiting deployment confirmation.
17:30 GMT
Test cases continuing to be confirmed and release will be undertaken once all test cases are passed.
15:30 GMT
Critical issues resolved, currently undergoing testing.
13:00 GMT
Majority of issues are resolved. Remaining issues are underway and we expect them to be resolved today, as a permanent resolution.
11:30 GMT
We are working towards a permanent resolution.
09:45 GMT
Service disruption confirmed and issue identified. Resources allocated to resolve issues highlighted. Update on timeframe to be provided in due course.
Decision not to roll-back previous release due to implications on current transactions and statements generated and issued since roll-out.
08:30 GMT
We are currently investigating reports of issues in Statement Generation, as detailed in the summary. This relates to a previous release.
Additional Information
| Incident Description | Status | Notes |
|
Balance-brought-forward placeholder is not rendering the correct value on the statement |
Resolved | |
|
Billing periods are incorrect- this is usually related to a change of tenancy |
Resolved | |
| Address lines do not render correctly on the statement | Resolved | |
| Multiple Tariff names, and charges are populated within the same cells within the table detailing the breakdown of charges for a particular meter | Resolved |
This is related to the statement template layout. The solution is to bill for different tariff periods separately. If you would prefer to amend your statement template, please contact the support team if you would like some further instruction. |
| Statements are stuck at Ready-to-Publish/Ready-to-Review | Resolved | |
| Error message received on statement generation "Generation could not start, please check charges, supply points and meters are attached" even when meters/supplypoints/tariffs are in place | Resolved |
POST-MORTEM
Published - 9th June 2021
Summary
On the 24th of May a new release of software was made to upgrade the statement generation functionality of Mabdeck, including the introduction of two new sections relating to billing; the billing runs page and the billing profiles wizard. A number of issues were experienced by some user that prevented the completion of the full billing cycle.
Services affected
Statement generation
Statement rendering
Statement approvals
Impacted users
A small proportion of customers experienced interruption to their normal use of Mabdeck.
Incident description
Pre-release
Prior to release, the new version of Mabdeck statement generation functionality underwent our usual process of testing between 9th of February 2021 and the 19th of May 2021
Beta-testing
Prior to general release, a selection of Mabdeck clients were provided access to the new system. A series of live, production billing runs were undertaken during the beta-testing phase.
General release
The general release of the new functionality was rolled out on the 24th of May 2021.
A number of issues were experienced. On 1st June 2021, a service incident was announced based on a potential issue with the generation engine. This was found to be limited to a small subset of statements.
the fix was deployed as a planned release at 22:00 GMT on 2nd June 2021.
Root cause(s)
The issues experienced with the release affected subsets of statements generates, not the core generation.
the vast majority of issues related to the transition of accounts and statements across the old and new statement generator engines, where there was an insufficient number of fringe test cases.
Several of the issues related to specific statement templates, which were caused by human error during the initial deployment transition. This affected less than 0.3% of templates.
Incident response actions
During the incident, development resources were diverted to investigating and rectifying the issues that had been identified.
Our support team proactively dealt with queries and updated clients and an incident was notified though help.mabdeck.com, tracking the technical progress of the response.
Post-incident response actions
Following the incident, a full investigation has been undertaken to identify and review the above details, as well as identify any changes that are required in processes, technology, training or other areas.
Lessons learned and changes implemented
The review of this service incident has highlighted a number of changes that will be made, or have already been made.
General development processes
In order to improve our general development processes, the following changes have been identified:
- The frequency of releases will be increased.
- Development standards are to be updated.
- Development documentation will include all related system subcomponents.
Testing
Testing was highlighted as a significant contributor to the incident and the following changes have been identified:
- Expansion of self-test against acceptance criteria - already implemented.
- Test cases for all experienced issues have been created for regression test purposes - already implemented.
- Integration testing will be further developed.
- Testing data will be expanded to include further fringe cases.
- A new test pack will be created.
Bug handling
Bug handling was not identified as a particular issue, however some improvement opportunities were identified:
- An improved process for service incidents has been developed.
- A new process identified to identify the same issue from multiple sources.
Deployments
The specific issues relating to this incident have been resolved, as well as identifying some additional changes:
- Deployment process updated to resolve errors - already implemented
- Hotfix policy updated-already implemented
- Wider team training in deployment processes and procedures.
The above changes that still need to be implemented will be monitored weekly until completed.