MJ Freeway/LeafData Update to 1.37.5 - Post Mortem
Attempted to document as well as possible. Most of us are small shops, don't have BigCo like resources to fight-fires, take dilligent notes, participate in meetings and run communications. This post-mortem is also very brief -- as there are still lingering issues. All times are America/Los_Angeles.
July 13 at 23:00
LeafData goes offline, as scheduled. This freezes all supply side work. Every licensee is dependent on this centralized system to generate unique identifiers and to electroniclly transfer regulated material for B2B sales. Harvesting, processing, packaging, and laboratory testing are all frozen. B2B Sales work cannot be processed. This outage is scheduled to last until July 16 at 08:00
The bonus, is that retail sales are not affected. They don't need to be filed in real-time and they are not dependent on unique identifiers for next steps in the regulated system. These licensees can "simply" (but still inconvient) queue up these regulated transactions.
July 14, 15
LeafData offline. Plant processing, Packging, B2B Sales are all frozen because everything is dependent on these unique identifiers at every step of the process. B2C sales continue but B2B sales are still frozen, retailers are not able to acquire new product.
July 16 06:00
LeafData appears online. In a later post-mortem meeting we learn that "we immediately started seeting problems".
For supply-side licensees the problems are severe. Many of the "fixes" provied by MJ Freeway are blocking B2B sales. We discover that millions of records have been scrambled
For the data nerds this means that dependent linked attributes (ie: foreign keys) now point to new objects, not owned by the primary record owner. All these values need to be reset to correct, last-known-good values.
July 17, 18
New Licenses were observed in the License List, which were called WAATTESTED -- to be the owner of the WAATTESTE [sic] type lab results. These licenses were also recognizable because their address was in Nevada.
July 19 at 14:00
Release 1.37.5 Pesticide Result Instructions. Every single Lab Result has to be updated in the system to fill the value "0" (zero) into every field. Due to a bug in data-processing any non-compatible results will be marked as failure.
July 20 - Saturday
Duplicated license data appeared in the system for a short period. Identified by licenses over MM26N (2831). This has since been corrected.
The MJ Freeway upgrade hell-week is over.