[vz-dev] API

[Justin Otherguy]

Hi Bart,

and: hello gonium - welcome! :-)

Am 19.07.2010 um 23:38 schrieb Bart Van Der Meerssche:
> On 07/18/2010 09:11 PM, Justin Otherguy wrote:
>> Am 13.07.2010 um 17:33 schrieb Bart Van Der Meerssche:
>> 
>>> icarus75 at nimbus:~$ curl -k -v -X GET -H "Accept: application/json" -H "X-Version: 1.0" -H "X-Token: d8a8ab8893ea73f768b66b45234b5c3a" "https://api.flukso.net/sensor/c1411c6b4f9910bbbab09f145f8533b9?interval=hour&unit=watt"
>> 
>> so, basically we have two parts:
>> - header info: mime-type + protocol version 1) + access token
>> - request paramaters:
>>   - identification of the sensor and controller:
>>     - Flukso:
>>       - "sensor" (references the ucid, correct?)
>>       - there seems to be no keyword to reference the controller / flukso meter, yet - correct?
>>     - VZ:
>>       - "ucid" identifies the sensor ->  I suggest switching to "sensor" here; like flukso and more readable than "ucid"
>>       - "uuid" identifies the controller ->  I suggest switching to "meter" here; this is more readable than "uuid" and more precise than "user id"
> 
> What's in a name? :-)
...just hollow words... ;-)

Seriously: I'm German, so we'll need to sort out thing, right?! ;-)

> We use sensor id's and device id's. The latter are the equivalent of 
> your controller/uuid's. You should indeed not call them user id's as 
> users might have multiple devices in use.
well - the data type is called "uuid" (there is a psql data type with that name and there even is an RFC for the format: http://tools.ietf.org/html/rfc4122); we're not going to associate that to users -> so that's fine, too.
for the moment we'll go with uuids for the sensors - every sensor will have it's own uuid.
Steffen came up with a pretty brilliant concept of "virtual groups". On the server side these sensor ids can be associated to groups. These groups are being referenced by uuids again. So, to request a graph showing the data of 3 or 10 sensors you associate these sensors to a group and use the group's uuid to request all associated data.
Every sensor can be in several groups at the same time. Say, we have 2 controllers and both controllers have 3 power sensors and one temperature sensor. We could have one group with all 6 power sensors; we could have a second group with the 2 temperature sensors and we could have a third group with all 8 sensors. Referring to the group you can always retrieve the desired data using a single uuid in the request.
The concept would also allow for comparison of different consumers or ... or ...
I'm quite impressed by the idea.

> So we associate sensors to 
> devices and devices to users. We plan to offer an anonymous service with 
> device/sensor auto-registration. If you would then like to 
> visualize/share your data within Flukso, you would need to create an 
> account and associate your devices with your account.
maybe this is the same idea and only just now I understood ;-)

> The device id is not required for fetching data from the server. A 
> sensor id suffices.
-> perfect.

I had a loooong phone conversation with Steffen on monday (hello Steffen, how's the ear?).

>> speaking of AVR net io: we'll have to find out - I don't know yet; EEPROM would be the place, I suppose (RAM: doesn't survive a reboot; SD: in most cases not available);
>> Flukso: shouldn't be a big deal :-)
> 
> Flukso uses a power-loss detection circuit which uses the AVR's analog 
> comparator. The latter triggers an interrupt the moment the rail voltage 
> drops below a certain threshold. The interrupt then writes all meter 
> values to EEPROM. We use an elco and a diode to store energy and prevent 
> leakage to the router board in the event of a power loss. By being 
> parsimonious with the AVR's energy consumption, we can still write +16 
> bytes to EEPROM the moment a power loss occurs.
cool; if I didn't know better I would have guessed you have an electronic engineering background ;-)

>>   - "unit":
>>     I just understood the different approaches we had; so far we (VZ) are transmitting pulse counts that are calculated into consumptions later on; that's why we need to know the meter's resolution on the server side; I can't see any disadvantage of switching to the flukso approach here: transmitting the consumption together with the unit. Can someone else? Otherwise: let's get rid of ours!
> 
> Euh, actually we use a quite similar approach for logging. We just 
> normalize the pulses on the sensor board. So for analog (clamps) or 
> digital (pulses) inputs, a pulse = 1Wh. From this point onwards we work 
> with the meter value.
> 
> For the local JSON/REST interface on the Fluksometer, we calculate the 
> power every second in the AVR and update the JSON array accordingly: pop 
> the oldest value and push the newest one.
> 
> The 'unit' I was referring to was for fetching data from the server. You 
> can have it in watt, kWh/year, eur/year, etc.
yep, got the two APIs (frontend/backend and controller/backend) mixed up. Again.

I keep loosing track, so mainly for myself:
- we're (mainly) discussing the frontend/backend API here
- the controller/backend API (currently XMLRPC for flukso, will be JSON/REST as well at some stage) will be a second step - not our focus at the moment

>>   - "interval":
>>     we don't have that at all so far; what's the idea behind transmitting that as well?
> 
> This is again for fetching data from the server. It makes for a 
> straightforward call: when iterval=day, then the Flukso server will 
> provide you with a JSON array containing a day's worth of data in 15 min 
> slots, so 96 data points. If you make the same call again an hour later, 
> you don't need to specify the time. It's again the data between 
> 'now-24h' and 'now'. Pretty much the way RRDtool allows you to fetch 
> data from an RRD.
got it. Yet again - mixed up...

>> Bart: do you have more parameters planned so far? If not, I suggest we try to agree on the ones present so far and then move on to the additional ones we find necessary, alright?
> 
> I'm thinking of having a sort of 'detail=y' option which would give you 
> all parameters related to the sensor, like:
> - type (eg electricity)
> - function (eg main, solar, ...)
> - last metervalue
> - etc.
ok

>> remarks:
>> 1) in our api reference [1] we currently have the version information as one of the regular request parameters; I don't mind switching to that variant; we'll be using http headers anyway
> 
> Actually, we allow for the version and token to be sent either as a 
> header field, or as part of the query parameters.
oh, that's good. I like the idea of providing both variants - at least for now.

>> 2) I never liked the idea of using uuids to identify sensor as I always supposed this would mean that you would have to include all sensor uuids when you wanted to retrieve all sensor data for one controller; in that case I would have preferred to only transmit one uuid (the controller one); having uuids for the controller _and_ the sensors seem to give us the best of both worlds ->  I like that
> 
> Enter Avahi/Bonjour. We'll just use this mDNS service (thanks gonium!) 
> on the Fluksometer to advertise the available sensors to computers, 
> smartphones, Chumbies in the LAN. No need for writing down/copying all 
> those sensor id's. We use the Bonjourfoxy FF plugin in our demo setup to 
> auto-detect the sensors and present them to the JS code. The JS code can 
> then form its JSON/REST call based on the information deliverd by Bonjour.
> 
> Just think about this: What's the difference between auto-detecting a 
> printer or a sensor in your LAN?
> 
> With HTML5 you've got this local storage option. So your smartphone 
> would only have to associate to your wireless LAN once to discover all 
> the sensors and store them. It could then fetch data from the web server 
> when on the road. All of this without having to copy/paste a single <uuid>.
> 
> Note: There still remains some fair amount of coding and testing to get 
> this feature working properly. But I think it's an awesome one.
yep, I remember you got me on this one a few weeks back, already. That sounds nice.

>> I'm not sure about the syntax; to me it seems best practice would be to have all parameters/values separated by slashes and not to use the "?" and "&" at all - am I wrong? Is this defined at all?
> 
> I made the same 'mistake' in my first API attempt. Use clean url's for 
> people. Machines are much better off with the explicit queries. They are 
> much more flexible since they are not fixed to a specific location 
> within your URL. What will happen when you have e.g. two optional 
> parameters in your API request? The slashes won't cut it.
just to be sure, I got you right on this one:
you prefer using only a few fixed parameters as part of the path and having the rest as part of the "query string" (i.e. separated by the "?"), right?

that takes me to my next question: 
how do you take care of the url paths on the server side?
If I'm not mistaken you actually do create dirs/files for each sensor on the flukso, right?
Is that the way you do it on flukso.net as well?

> open:elektro.camp(<<2010>>).
> 
> Quiz: Guess which language the barcamp title refers to. First one to 
> guess correctly will get ein Helles from me during the camp. Barcamp 
> organizers cannot participate in this contest.
damn, I'm too late for this one ;-)

A word to Mathias' concerns:
no one intends to break anything ;-)

Seriously:
I'd like to use the flukso frontend API without modification for volkszaehler; that's why I try to understand it. So far it seems to me that there's no need to change anything. We might find some additional parameters to extend the API - but that wouldn't break compatibility, then. 
To my understanding this is even clearer for the flukso/backend API: a change (xmlrpc -> json/rest) is coming up; the current API has to be supported for quite a long while after the new API is available. We're not interfering with that at all as we (volkszaehler) will not be implementing the xmlrpc interface but json/rest here as well from the start.


Regards, J.