Custom data processing¶
Some units require performing data processing after readings are collected, and we provide default algorithms for all the data we collect. Iin the default scenario, for those sensors that need it (i.e. electrochemical sensors), raw data is processed outside of the units in a periodic way by
scdata. However, you might need to perform custom data processing algorithms on your data. You can do so by simply accessing the data on our API, and then working with it offline, or you can set up an automatic data processing workflow. To summarise, there are two ways to have a custom data processing:
- Set up a development environment and following the example on data processing
- Follow the guide below for automatic processing
Automatic data processing¶
Data is processed in a recurrent manner whenever there is
postprocessing field in the device data tree. You can check it by visiting
<device-id> is the number that follows in the url
How to do it?
Visit these instructions to make sure your
postprocessing is safely stored.
postprocessing field is composed of several subfields being the most important one for the automatic data processing the
hardware_url should point to a valid json file that contains the necessary information for the data processing. This json file is customisable, but should contain at least the fields defined in the calibration guide. In order to customize how our data is processed, we will need to modify the
blueprint_url and, if we have specific calibration needs, the
versions field. Once these changes are done, this hardware json should go in the hardware folder or any other place that can host it.
blueprint_url should point to a json that defines which postprocessing you want to apply for each channel, which functions and parameters it has, and how the new channels are called (don't worry, there are a lot of examples in the blueprints folder). In order for your data to be processed automatically, you will need to provide that blueprint to
scdata. For this, you can host it somewhere, or you can simply make a pull request to the scdata github repository, having the new json in the
blueprints folder. Finally, you will then need to modify
hardware json and update the blueprint field of your kit, with the new
blueprint_url. See below to understand how this blueprint works and how to write custom fuctions.
In case your kit has specific calibration data needs, you will also need to modify the
versions field. This field is a list which contains the history of the kit in terms of calibration (calibrations can change with time):
Each of the
ids field make reference to specific calibration ids for each timespan, where the
to fields are the dates during which those ids where valid. This method ensures allows to change sensors with time, and also considers that normally sensors that have specific calibrations use a serial number or similar to identify them.
ids field, the keys (
AS_49_10, ... in the example above), define the type of sensor id (
AS is an Alphasense sensor,
PT is a PT100 temperature probe) and potentially other useful identifiers, such ADC addresses and channels (i.e.
AS_48_32 defines an Alphasense sensor connected to the ADC in address 0x48 and channel 3 for the working electrode and channel 2 for the auxiliary electrode). Other sensor types would need to defined and we can support in the implementation process. Finally, the actual sensor id, and the calibration data associated with it, is stored in another json, this time in the calibrations folder. Similarly to the blueprints above, a simple pull request would work.
Defining custom functions¶
If you want to define custom processing functions, make a Pull Request to the repository of scdata to integrate your functions. Otherwise, you can make a fork and install the package in
editable mode. You can then add the functions in the device processing folder of your own. We are available at email@example.com for more support.