How BONG Works with CCRS
One of the needs of cannabis software is an interface to various back-end service platforms such as compliance reporting engines (CRE). In Washington the system is called (CCRS)[/blog/tags/ccrs]. It's a very simiplistic CSV upload based system, attempting to support INSERT, UPDATE and DELETE operations. Any error responses are sent out via email and, as designed, no response at all means success.
However, this system is not very robust, error reporting is spottty and known to even fail to report -- leaving folks completely in the dark. BONG serves as a processing layer for this system providing two key features for data integrity. First, we keep a mirror record of data-objects that should be in the system. Second we capture detailed logs of what was sent and what replies, if any, were received.
And, since BONG follows the standard OpenTHC API, it's trivial to integrate.
Mirror Records
For each record that must be sent up to a CRE, including CCRS, BONG will track a mirror record. This record holds both core and meta data for the objects (see the OpenTHC API for more information). For CCRS this includes Section, Variety, Product, Crop, Inventory and B2B Transactions.
CCRS operates in an asynchronous fashion; so the BONG record is updated and then in batch fashion they are processed into CSV and uploaded. Once (or if) a response is received from CCRS the record is updated to indicate such.
This allows us to see each (and every) record that we think should be in BONG -- and the current status. It also makes it much simpler to re-attempt any data updates.
Request Logging
Once these the data upload CSV files are generated they are processed into CCRS. A copy of the uploaded data, and the response page are captured. Timestamps are included for these steps.
A unique approach that OpenTHC takes is to include a "-canary-" record in each upload. In this way, every upload we send must be acknowledged by the LCB -- even if the reply is just that canary line. Another "trick" is to execute a second INSERT -- which will give a duplicate error to confirm the data is properly stored.
After some time an email reply may (most of the time) will come in from CCRS. It will be either some failure message, our special canary-failure or a Manifest PDF document.
These incoming emails and files are processed by BONG, linked back to their specific uploads and the results are stored in this log. When uploads are processed, any of the linked mirror-ojects are updated as well; with either a success (eg: double-insert) or failure. This allows the down-stream systems (ie: a seed-to-sale application) to be aware of the data status.
More Information
All of the code that powers BONG is available on github.com/openthc/bong. It has a dependency on github.com/openthc/cre-adapter for some lower-level CCRS data handling. If you're building your own systems, or just curious on dealing with CCRS check it out.