Streaming Data over REST/HTTP with ASP.NET Core-Alfus Inigo Jaganathan

Background

While working on a project requirement, to stream file content to requesting clients, we came across several solutions, of course, the one came up first was gRPC over HTTP/2. Other options were to use signalR, Websockets, Kafka, Steeltoe Streams, etc.

However, we got into a situation where we weren’t able to continue with gRPC over HTTP/2, due to certain limitations on the hosting platform we are currently at. Now we were left over with the other solutions, mentioned above.

Meanwhile, I thought of exploring other options which can satisfy the requirement, but with minimal implementation overhead. There comes a thought of using REST API, where I started creating a quick sample application to try it out. Fortunately, after some exploration and trials, I was able to successfully stream the contents over REST API to the client application.

Now that I got is done successfully, thought of sharing it to the community, so that it might be useful for handling such scenarios, incase needed.

Let’s create it now

Create a Server Web API app using dotnet new webapi and then create a Controller with the name of your choice, I used StreamController in my sample. Also, add the below Action Method, which sends the stream of Guids as many as asked by the requester via count argument. Here, the cancellation token does the purpose of graceful handling on client cancellation events.

[HttpGet("{count}")]
public async Task GetAsync(int count, CancellationToken cancellationToken = default)
{
    try
    {
        this.Response.Headers.Add(HeaderNames.ContentType, "text/event-stream");
        using var outputStreamWriter = new StreamWriter(this.Response.Body);

        await foreach (var item in GetData(count))
        {
            cancellationToken.ThrowIfCancellationRequested();
            logger.LogInformation($"Streaming {item}");

            await outputStreamWriter.WriteLineAsync(item);
            await outputStreamWriter.FlushAsync();
        }
    }
    catch (OperationCanceledException)
    {
        logger.LogWarning($"Operation Cancelled");
    }
}

The simple idea here is to leverage the Response.Body which itself is a stream, through which we can send the data.

Another thing to do is, to add Synchronous IO support from the web server, which we can accomplish using below code.

webBuilder.UseKestrel(options=>
{
    options.AllowSynchronousIO = true;
});

Below is the method which mimics the data preparation (generation of Guids), which can be replaced by anything as needed. Task.Delay added just to create a realtime feel when running this sample app.

private async IAsyncEnumerable<string> GetData(int count)
{
    for (int i = 0; i < count; i++)
    {
        await Task.Delay(new Random().Next(100, 1500));
        yield return await Task.FromResult(Guid.NewGuid().ToString());
    }
}

Now you can simply test it via browser using https://localhost:7001/Stream/10000, where Stream is the controller name (in my sample case) and 10000 is the record count.

You should be able to see the data being streamed, as it is being processed in the server. You can also create a simple console app to test it out, which you can implement using below code. You can also find sample code in the github repo.

Using HttpClient

If you are using HttpClient, below are the 2 methods of implementation in a streaming client.

1. Streaming without http response check

using var httpClient = new HttpClient();
var stream = await httpClient.GetStreamAsync("https://localhost:7001/streams/300", token);
try
{
    using var reader = new StreamReader(stream);
    while (!reader.EndOfStream)
    {
        var line = await reader.ReadLineAsync();
        Console.Out.WriteLine(line);
    }
}
catch (Exception ex)
{
    Console.Error.WriteLine(ex.ToString());
}

2. Streaming after http response check, which helps in error handling

using var httpClient = new HttpClient();
try
{
    var response = await httpClient.GetAsync("https://localhost:7001/streams/300", HttpCompletionOption.ResponseHeadersRead, cancellationToken);

    if (response.IsSuccessStatusCode)
    {
        using var stream = await response.Content.ReadAsStreamAsync();
        using var reader = new StreamReader(stream);
        while (!reader.EndOfStream)
        {
            var line = await reader.ReadLineAsync();
            Console.Out.WriteLine(line);
        }
    }
    else
    {
        //handle response errors
    }
}
catch (Exception ex)
{
    Console.Error.WriteLine(ex.ToString());
}

Code Reference

Find the sample application here in Github

Hope you had fun coding! Sharing is caring!

Streaming Data over REST/HTTP with ASP.NET Core

A Guide to Sending Data Streams in ASP.NET Core