Streaming data with HTTP/1.1 Transfer-Encoding: Chunked

I've considered this before, but never really had a need for it.  In working on an AI training data project this (sending data in chunks) was touted to make a demo look "better" - data is available in chunks and this can used to make it look like the AI is typing the response.

In the end it was not used for that, but I did realise that I COULD used this to send more frequent position updates in FollowMe - so ineffect this is a PoC for my OWN application.

Before writing any code

Add A and AAAA records for a subdomain api.paul.sullivan.za.org and get certificates for that domain

apt-get install certbot

I already have a nodejs environment that I use on a VM.

Server Side Code

      res.setHeader('Content-Type', 'application/json; charset=UTF-8');
      res.setHeader('Cache-Control', '  no-cache');
      res.setHeader('Access-Control-Allow-Origin', 'https://paul.sullivan.za.org');
      res.setHeader('Access-Control-Allow-Headers', '*');
      res.setHeader('Transfer-Encoding', 'chunked');

      let intrvl = setInterval(function(){
        if (!running) {
          running = true;
          getRandomCoords().then(function(r) {
            running = false;
            if (r.length == 0) {
              clearInterval(inerv);
              res.end('{"r": "nodata"}');
            } else {
              s = JSON.stringify(r[0]);
              res.write(s);
            }
          });
        }
      }, 1000);


      let tmout = setTimeout(function(){
        clearInterval(intrvl);
        res.end('{"r": "done"}');
      }, 30000);

      req.on('close', function(err) {
        clearInterval(intrvl);
        clearTimeout(tmout);
        res.end('{"r": "done"}');
      });

This deals with CORS. And I found the "charset=UTF-8" was necessary.

The getRandomCoords function "polls" the database for some random data to emulate new data begin available.  I'm not streaming end-to-end yet - source data is still out into a DB table at 10s intervals

I did run into a timezone issue with mysql2 timestamp dates from the DB server (in UTC) where being returned an hour out to the App server (in BST) this was fixed by forcing:

timezone: '+00:00'

If the client closed the page (TCP RST) or reloads it is important to clear the Interval and Timeout or they will keep running - consuming reasource and sending it nowhere.

Client Side Code

To see if the client side is receiving data:

setInterval(checkXMLHttpRequest, 2000);

function checkXMLHttpRequest() {
          if ((xhr.readyState === 0) || ((o.r == "done") && (xhr.readyState === 4))) {
               xhr.abort();
               start = 0;
               xhr = new XMLHttpRequest();
               xhr.addEventListener("progress", handleEvent);
               xhr.open("GET", "https://api.paul.sullivan.za.org/FollowMe", true);
               xhr.send();
          }
}

if the HttpRequest is not running or the (finite) stream of server data is complete then a new request is made. 

Notes & Caveats

1. The responseText property grows and grows so I've set up a finite amout of time the server will send data.
2. I've noticed that if this page is loaded by browsers on PCs with certain security software the request comes from 161.69.71.25 (not the device IP).  The user agent presented is the one from the browser, but the CORS check is unable to complete.