Skip to main content
Version: 2.x

Streams

A Redis stream is an append-only log of entries, each made of an ID and a set of field-value pairs. This page covers the 32 stream commands on Redis, from appending and reading entries to consumer groups, claiming and trimming.

Stream IDs have the form <milliseconds>-<sequence>. Pass "*" to Redis#xAdd to let the server generate the ID, and use "-" and "+" as the smallest and largest possible IDs in range queries.

Every command here takes the key, group, consumer and ID as values with a ZIO Schema Schema, which are encoded by the CodecSupplier. In the examples they are plain Strings.

Appending Entriesโ€‹

Redis#xAdd appends an entry made of one or more field-value pairs. The result type is the type of the generated ID, so we finish with returning[A], and the result is None when noMakeStream is set and the stream does not exist:

import zio._
import zio.redis._

def addClick(redis: Redis): IO[RedisError, Option[String]] =
redis.xAdd("clicks", "*")("page" -> "/home", "user" -> "42").returning[String]

The following table lists the commands that append entries:

CommandPurpose
Redis#xAddAppends an entry
Redis#xAddWithMaxLenAppends an entry and trims the stream to at most count entries
Redis#xAddWithMinIdAppends an entry and evicts entries with an ID lower than minId

Redis#xAddWithMaxLen and Redis#xAddWithMinId take the same approximate and limit parameters, described under Trimming Streams. The following example keeps roughly the latest 1000 entries:

import zio._
import zio.redis._

def addCapped(redis: Redis): IO[RedisError, Option[String]] =
redis
.xAddWithMaxLen("clicks", "*", count = 1000L, approximate = true)("page" -> "/home")
.returning[String]

Reading Entriesโ€‹

Range queries return the entries between two IDs, inclusive. They return StreamEntries[I, K, V], which is a Chunk[StreamEntry[I, K, V]], where StreamEntry holds an id and a fields map:

CommandPurpose
Redis#xRangeReturns entries from start to end, optionally count
Redis#xRevRangeReturns entries from end down to start, optionally count
Redis#xLenReturns the number of entries
Redis#xDelDeletes entries by ID and returns how many were deleted

Note that Redis#xRevRange takes end before start. The following example reads the ten oldest entries:

import zio._
import zio.redis._

def firstTen(redis: Redis): IO[RedisError, StreamEntries[String, String, String]] =
redis.xRange("clicks", "-", "+", 10L).returning[String, String]

Redis#xRead reads from one or more streams, starting after the given ID in each. Pass "0-0" to read from the beginning, or "$" together with block to wait for entries that arrive later. It returns StreamChunks[SK, I, K, V], a Chunk[StreamChunk] with the stream name and its entries:

import zio._
import zio.redis._

def waitForNew(redis: Redis): IO[RedisError, StreamChunks[String, String, String, String]] =
redis.xRead(count = Some(10L), block = Some(5.seconds))("clicks" -> "$").returning[String, String]

Capped Streamsโ€‹

A stream grows without bound unless it is capped. The trimming commands and options use CappedStreamType internally, with the cases MaxLenApprox, MaxLenExact, MinIdApprox and MinIdExact. Public methods expose the same choice through plain parameters.

CommandTrims by
Redis#xTrimWithMaxLenNumber of entries (threshold)
Redis#xTrimWithMinIdSmallest ID to keep (threshold)

Both commands take approximate: Boolean = false and limit: Option[Long] = None. With approximate = true, Redis may keep slightly more entries than requested, which is cheaper, and limit bounds the work done per call. Both return the number of entries removed:

import zio._
import zio.redis._

def trim(redis: Redis): IO[RedisError, Long] =
redis.xTrimWithMaxLen("clicks", threshold = 10000L, approximate = true, limit = Some(100L))

Consumer Groupsโ€‹

A consumer group lets several consumers share the work of reading one stream. Redis delivers each entry to one consumer of the group and remembers it in the group's pending entries list until the consumer acknowledges it.

The following table lists the commands that manage groups and consumers:

CommandPurpose
Redis#xGroupCreateCreates a group that starts after the given ID
Redis#xGroupCreateLastEntryCreates a group that starts after the current last entry
Redis#xGroupSetIdMoves the group's last delivered ID
Redis#xGroupSetIdLastEntryMoves the group's last delivered ID to the last entry
Redis#xGroupDestroyDeletes a group and returns whether it existed
Redis#xGroupCreateConsumerCreates a consumer and returns whether it was created
Redis#xGroupDelConsumerDeletes a consumer and returns how many pending entries it had

Redis#xGroupCreate and Redis#xGroupCreateLastEntry accept mkStream: Boolean = false, which creates the stream when it does not exist, and entriesRead: Option[Long] = None. The *LastEntry variants pass "$" for you. The following example creates a group on a new stream and registers a consumer:

import zio._
import zio.redis._

def setUp(redis: Redis): IO[RedisError, Boolean] =
redis.xGroupCreate("orders", "billing", "0-0", mkStream = true) *>
redis.xGroupCreateConsumer("orders", "billing", "worker-1")

The Stream Reading Flowโ€‹

The usual flow with a consumer group has four steps. A producer appends entries with Redis#xAdd. A consumer calls Redis#xReadGroup with the ID ">", which asks for entries never delivered to any consumer of the group. After handling an entry, the consumer calls Redis#xAck. Entries that were never acknowledged are recovered with Redis#xPending, Redis#xClaim or Redis#xAutoClaim.

Redis#xReadGroup takes count, block and noAck. With noAck = true entries are not added to the pending list, so no acknowledgement is needed. If you pass an ID such as "0" instead of ">", Redis returns the consumer's own pending entries.

Redis#xAck takes a group and one or more IDs and returns how many entries were acknowledged. The following example reads a batch, handles each entry and acknowledges it:

import zio._
import zio.redis._

def consume(redis: Redis): IO[RedisError, Unit] =
for {
chunks <- redis
.xReadGroup("billing", "worker-1", count = Some(10L), block = Some(2.seconds))("orders" -> ">")
.returning[String, String]
_ <- ZIO.foreachDiscard(chunks.flatMap(_.entries)) { entry =>
ZIO.debug(s"handling ${entry.id}: ${entry.fields}") *>
redis.xAck("orders", "billing", entry.id)
}
} yield ()

Recovering Pending Entriesโ€‹

When a consumer crashes, its entries stay in the pending list. The following table lists the commands that inspect and take over pending entries:

CommandPurpose
Redis#xPending (group only)Returns a PendingInfo summary
Redis#xPending (with range)Returns a Chunk[PendingMessage] for a range of IDs
Redis#xClaimTransfers named entries to a consumer and returns them
Redis#xClaimWithJustIdSame, but returns only the IDs
Redis#xAutoClaimScans the pending list and claims idle entries, returning them
Redis#xAutoClaimWithJustIdSame, but returns only the IDs

PendingInfo holds total, the first and last pending IDs and a consumers map of pending counts. Each PendingMessage holds id, owner, lastDelivered and counter. The ranged Redis#xPending also accepts consumer: Option[SC] and idle: Option[Duration] to filter the result:

import zio._
import zio.redis._

def inspect(redis: Redis): IO[RedisError, (PendingInfo, Chunk[PendingMessage])] =
for {
summary <- redis.xPending("orders", "billing")
details <- redis.xPending("orders", "billing", "-", "+", 10L, Some("worker-1"), Some(1.minute))
} yield (summary, details)

Redis#xClaim takes the minIdleTime an entry must have exceeded, plus the optional idle, time, retryCount, force and lastId parameters that set the new delivery state. Redis#xAutoClaim takes minIdleTime and an optional count, and the start ID in the second parameter list. It returns a StreamClaimedEntries with streamId, the entries and the deletedIds. The WithJustId variant returns a StreamClaimedIdChunk with streamId, claimedIds and deletedIds. Use the streamId as the next start:

import zio._
import zio.redis._

def takeOver(redis: Redis): IO[RedisError, StreamClaimedEntries[String, String, String, String]] =
redis
.xAutoClaim("orders", "billing", "worker-2", 5.minutes, count = Some(Count(10L)))("0-0")
.returning[String, String, String]

Stream Informationโ€‹

The info commands return typed summaries instead of raw replies:

CommandReturns
Redis#xInfoStreamStreamInfo[I, K, V] with length, last ID, first and last entry
Redis#xInfoStreamFullFullStreamInfo[I, K, V] with entries, groups and consumers
Redis#xInfoGroupsChunk[StreamGroupsInfo]
Redis#xInfoConsumersChunk[StreamConsumersInfo]

Redis#xInfoStreamFull has an overload that takes count: Long to limit the number of entries and pending records reported. The FullStreamInfo and its ConsumerGroups, Consumers, GroupPel and ConsumerPel types live in StreamInfoWithFull. The following example prints the lag of each group:

import zio._
import zio.redis._

def lags(redis: Redis): IO[RedisError, Chunk[(String, Long)]] =
redis.xInfoGroups("orders").map(_.map(g => g.name -> g.lag))

Option Typesโ€‹

The following table summarizes the types from options/Streams.scala that appear in command signatures and results:

TypeUsed by
StreamEntry[I, K, V]id and fields: Map[K, V] of one entry
StreamChunk[N, I, K, V]Entries of one stream read by xRead
StreamClaimedEntries[SI, I, K, V]Result of xAutoClaim
StreamClaimedIdChunk[SI, I]Result of xAutoClaimWithJustId
PendingInfo, PendingMessageResults of xPending
StreamInfo, StreamGroupsInfo, StreamConsumersInfoResults of the xInfo* commands

XGroupCommand, WithForce, WithJustId, MkStream, NoMkStream, NoAck, LastId and WithEntriesRead encode the command arguments for you and are not needed in application code.

  • Lists offer a simpler queue when consumer groups are not needed.
  • Codecs explains how keys, fields and values are encoded.
  • Error Handling lists the errors raised, such as NOGROUP when a group does not exist.

ZIO Chat

Hi there! ๐Ÿ‘‹ I'm ZIO Chat, your friendly neighborhood ZIO expert. I'm here to help you navigate the world of ZIO. Whether you're a seasoned pro or just starting out, fire away with your ZIO questions. I will hit you back with the best insights I've got. Let's dive into the world of ZIO together!