Uploaded image for project: 'Spring XD'
  1. Spring XD
  2. XD-2233

REST representation of an aggregate-counter can lead to mixed up output

    XMLWordPrintable

    Details

    • Type: Story
    • Status: To Do
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: Waiting for Triage
    • Component/s: Analytics, REST
    • Labels:
      None
    • Story Points:
      8
    • Rank (Obsolete):
      46135

      Description

      The current representation of REST resources of time-series data (e.g. aggregate counter) can lead to problems in consuming applications.

      Despite the time series data provided by the "counts" data structure is logically ordered by key (timestamps) it doesn't guarantee an ordering, since many consuming applications interpret JSON data as an unordered map like data structure.

      Because of this consuming applications have to apply special ordering / transformation logic to get the data in an chronologically ordered fashion. It would be helpful if one could configure the rendering of the time series data, e.g. as a list of json object like:

      {
       "ts":"Sun Oct 12 23:10:23 CEST 2014"
       ,"value": 42
      } 
      

      Where ts denotes the timestamp and value denotes the value. It would also be helpful if one could adjust the date format either with a pattern or a well known date format like, c.f. ISO 8601.

      I attached a python example for this that demonstrates the problem.

      Steps to reproduce:
      Create stream

      xd:>stream create test --definition "http | filter --expression=payload.contains('pivotal') | log"
      

      Create tap on stream with aggregate-counter

      xd:>stream create test_tap --definition "tap:stream:test.filter > aggregate-counter"
      

      Post some http data

      xd:>http post --data "Hello pivotal data labs"
      #...some more data...
      

      Display aggregate counter

      xd:>aggregate-counter display test_tap --resolution minute
        AggregateCounter=test_tap
        -----------------------------  -  -----
        TIME                           -  COUNT
        Sun Oct 12 23:10:23 CEST 2014  |  0
        Sun Oct 12 23:11:23 CEST 2014  |  0
        Sun Oct 12 23:12:23 CEST 2014  |  0
        Sun Oct 12 23:13:23 CEST 2014  |  0
        ...
        Mon Oct 13 00:02:23 CEST 2014  |  0
        Mon Oct 13 00:03:23 CEST 2014  |  0
        Mon Oct 13 00:04:23 CEST 2014  |  0
        Mon Oct 13 00:05:23 CEST 2014  |  0
        Mon Oct 13 00:06:23 CEST 2014  |  0
        Mon Oct 13 00:07:23 CEST 2014  |  0
        Mon Oct 13 00:08:23 CEST 2014  |  3
        Mon Oct 13 00:09:23 CEST 2014  |  1
      

      Install python Requests library (REST support)

      pip install Requests
      

      Start a python console (or IPythonNotebook) and run the following program:

      import requests
      import json
      
      res = requests.get("http://localhost:9393/metrics/aggregate-counters/test_tap?resolution=minute")
      status_object = json.loads(res.content)
      print(json.dumps(status_object, indent=4))
      

      The above program should result in a similar output, but as one can see, due to pythons interpretation of the JSON object as a dict, the order of the keys in the output got mixed up.

      Instead of showing the counts ....3 and then 1 ... as in the example above. This is just one example of how the current representation of the rest resource could lead to problems in consuming applications.

      {
          "counts": {
              "2014-10-12T21:26:28.553Z": 0,
              "2014-10-12T22:15:28.553Z": 0,
              "2014-10-12T22:22:28.553Z": 0,
              "2014-10-12T21:49:28.553Z": 0,
              "2014-10-12T21:35:28.553Z": 0,
              "2014-10-12T21:47:28.553Z": 0,
              "2014-10-12T22:18:28.553Z": 0,
              "2014-10-12T22:12:28.553Z": 0,
              "2014-10-12T22:16:28.553Z": 0,
              "2014-10-12T21:57:28.553Z": 0,
              "2014-10-12T21:28:28.553Z": 0,
              "2014-10-12T22:08:28.553Z": 3,
              "2014-10-12T21:40:28.553Z": 0,
              "2014-10-12T22:06:28.553Z": 0,
              "2014-10-12T21:27:28.553Z": 0,
              "2014-10-12T21:52:28.553Z": 0,
              "2014-10-12T22:11:28.553Z": 0,
              "2014-10-12T22:05:28.553Z": 0,
              "2014-10-12T21:29:28.553Z": 0,
              "2014-10-12T21:24:28.553Z": 0,
              "2014-10-12T21:56:28.553Z": 0,
              "2014-10-12T21:43:28.553Z": 0,
              "2014-10-12T22:00:28.553Z": 0,
              "2014-10-12T22:10:28.553Z": 0,
              "2014-10-12T21:58:28.553Z": 0,
              "2014-10-12T22:21:28.553Z": 0,
              "2014-10-12T21:32:28.553Z": 0,
              "2014-10-12T21:46:28.553Z": 0,
              "2014-10-12T22:04:28.553Z": 0,
              "2014-10-12T22:02:28.553Z": 0,
              "2014-10-12T21:51:28.553Z": 0,
              "2014-10-12T21:38:28.553Z": 0,
              "2014-10-12T21:31:28.553Z": 0,
              "2014-10-12T22:20:28.553Z": 0,
              "2014-10-12T21:54:28.553Z": 0,
              "2014-10-12T22:07:28.553Z": 0,
              "2014-10-12T22:03:28.553Z": 0,
              "2014-10-12T21:34:28.553Z": 0,
              "2014-10-12T22:09:28.553Z": 1,
              "2014-10-12T21:44:28.553Z": 0,
              "2014-10-12T22:17:28.553Z": 0,
              "2014-10-12T21:53:28.553Z": 0,
              "2014-10-12T22:19:28.553Z": 0,
              "2014-10-12T21:30:28.553Z": 0,
              "2014-10-12T22:23:28.553Z": 0,
              "2014-10-12T21:36:28.553Z": 0,
              "2014-10-12T21:41:28.553Z": 0,
              "2014-10-12T22:13:28.553Z": 0,
              "2014-10-12T21:59:28.553Z": 0,
              "2014-10-12T22:01:28.553Z": 0,
              "2014-10-12T21:33:28.553Z": 0,
              "2014-10-12T21:45:28.553Z": 0,
              "2014-10-12T21:39:28.553Z": 0,
              "2014-10-12T21:50:28.553Z": 0,
              "2014-10-12T21:37:28.553Z": 0,
              "2014-10-12T22:14:28.553Z": 0,
              "2014-10-12T21:25:28.553Z": 0,
              "2014-10-12T21:55:28.553Z": 0,
              "2014-10-12T21:42:28.553Z": 0,
              "2014-10-12T21:48:28.553Z": 0
          },
          "name": "test_tap",
          "links": [
              {
                  "href": "http://localhost:9393/metrics/aggregate-counters/test_tap",
                  "rel": "self"
              }
          ]
      }
      

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            thomasd Thomas Darimont
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated: