On March 26th, 2019 LeafData suffered yet another major data-access/security issue. Retail class licensees were able to modify records of inventory from the supply-side licensee they purchased materials from. This issue became known as LW4821.
On March 26th we had scores of customers contacting our support line reporting that inventory had re-appeared. Searching through our logs we identified that tens of thousands of records had been modified within the last few hours. Reviewing the logs we could see that the modifications were done by LeafData Global User IDs that were prevously unknown to us. We were able to quickly narrow down the pattern of data-mangling to see that it only affected inventory that had been sold to retailers.
Naturally, we promptly contact the WSLCB with our concernts. At this point it was difficult to determine if this error was simply a bug in LeafData, or if there was intentional malice.
A conversation with staff from both the WSLCB and MJ Freeway identified that it was not malicious. A third party software integrator had recently made some changes to how inventory is received. During the receipt of materials rather than using the received inventory ID, they were using the shipped inventory ID. This is in combination with a recent re-factor of code (and obvious lack of testing) to this component by LeafData. In turn, LeafData handled these requests by modifing records of inventory that were not owned by the licensee making the request Read that again, LeafData was allowing License-A to modify records owned by License-B. If a malicious actor had discovered this bug the outcome would have been much worse.
Together the WSLCB and MJ Freeway reached out these licensees, and their service provider to determine a fix. Initially the idea to block this service provider from the system was floated! We took the opportunity to state we did not believe that to be a good course of action, it would have negative affects across the entire cannabis industry in Washington state.
A few days later at an integrators meeting (the WSLCB holds them every other week (mostly)) the issue was discussed again. Over 290,000 records had been identified as affected by this bug. Sadly, LeafData did not have the capability or capacity to repair these data issues. The duty to repair this massive data corruption was left to the licensees and their software providers.
It took us few hours to write a proper script to analyse the corrupt data for clients. I’m sure it took other providers a similar amount of time. It also took some time for these scripts to run; making many requests to LeafData frequently results in 502 and 530 level HTTP errors. However, after some hours of processing it appears that all is well.
This is yet another example of the dangers of closed-source systems. With an open-source code base, the odds are orders of magnitude better that bugs will be identified before they end up in production. Additionally, had LeafData had test-cases this bug could have been caught there. Since it wasn’t, we can surmise that either test cases are incomplete or non-existant.
All together this issue caused, at least, 100s of wasted hours across the entire cannabis industry in Washington State